SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A simple approach is to apply an activation function with a nonnegative triple derivative. An integral network that uses this activation and has nonnegative linear layer weights satisfies the condition. We call this approach "Constrained Triple AutoInt". However, the output of an integral network can grow very quickly with large input, and the gradients are likely to explode during training. Moreover, the non-negative constraint on the influence function only requires  $\frac{\partial F_{\theta}}{\partial\mathbf{s}\partial t}$  to be positive. But an activation function with a nonnegative triple derivative would also enforce other partial derivatives to be positive. Such constraints are overly restrictive for STPPs, whose partial derivatives  $\frac{\partial F_{\theta}}{\partial\mathbf{s}\partial\mathbf{s}}$  and  $\frac{\partial F_{\theta}}{\partial t\partial t}$  can both be negative when the intensity is well-defined.
ProdNet. We propose a different solution to enforce the 3D non-negative constraint called ProdNet. Specifically, we decompose the influence function  $f_{\theta}(s_1,s_2,t):\mathbb{R}^3\to \mathbb{R}$  as  $f_{\theta}^{1}(s_{1})f_{\theta}^{2}(s_{2})f_{\theta}^{3}(t)$ , the product of three  $\mathbb{R}\rightarrow \mathbb{R}$  AutoInt derivative networks. The triple antiderivative of the influence
Figure 3: Comparing the ground truth conditional intensity  $\lambda^{*}(\mathbf{s},t)$  with the learned intensity on the ST Hawkes Dataset 1 and ST Self-Correcting Dataset 3. Top row: Ground truth. Second row: Our AutoSTPP. Rest of the rows: Baselines. The crosses on top represent past events. Larger crosses indicate more recent events.
function is then  $F_{\theta}^{1}(s_{1})F_{\theta}^{2}(s_{2})F_{\theta}^{3}(t)$ , the product of their respective integral networks. Then we can apply 1D non-negative constraint to each of the derivative networks. The other partial derivatives are not constrained because  $F_{\theta}^{1}, F_{\theta}^{2}, F_{\theta}^{3}$  can be negative.
One limitation of such decomposition is that it can only learn the joint density of marginally independent distribution. We circumvent this issue by parameterizing the influence function as the sum of  $N$  ProdNet,  $\sum_{i=1}^{N} f_{\theta,i}^{1}(s_{1}) f_{\theta,i}^{2}(s_{2}) f_{\theta,i}^{3}(t)$ . The formulation is no longer marginally independent since it is not multiplicative decomposable. Figure 2 shows that the sum of two ProdNets is already sufficient to approximate a function that cannot be written as the sum of products of positive functions. In contrast, the constrained triple AutoInt's intensity is convex everywhere and fails to fit  $f$ .
Increasing the number of ProdNet improves AutoInt's flexibility at the cost of time and memory. We perform an ablation study of the number of ProdNet in Appendix F.
3.7 Model Training
Given the integral network  $F_{\theta}(t,\mathbf{h}) \coloneqq \int_{\mathcal{S}} F_{\theta}(\mathbf{s},t,\mathbf{h})$  and the derivative network approximating the influence function  $f_{\theta} = \frac{\partial F_{\theta}}{\partial t\partial\mathbf{s}}$ , the log-likelihood of an event sequence  $\mathcal{H}_n = \{(s_1,t_1),\dots ,(s_n,t_n)\}$  observed in time interval  $[0,T]$  with respect to the model is
$$
\mathcal {L} (\mathcal {H} _ {n}) = \sum_ {i = 1} ^ {n} \log \left(\sum_ {j = 1} ^ {i - 1} f _ {\theta} (\mathbf {s} _ {i} - \mathbf {s} _ {j}, t _ {i} - t _ {j}, \mathbf {h} _ {i})\right) - \sum_ {i = 1} ^ {n} \left(F _ {\theta} (T - t _ {i}, \mathbf {h} _ {n}) - F _ {\theta} (0, \mathbf {h} _ {i})\right)
$$
Obtaining the above from Equation 1 is straightforward by the Fundamental Theorem of Calculus.  $F_{\theta}$  is evaluated using the Divergence theorem. We can learn the parameters  $\theta$  in both networks by maximizing the log-likelihood function. In experiments, we parametrize  $f_{\theta}$  with two AutoInt networks  $L$  and  $M$ . Each network is a sum of  $N$  ProdNets. That is,
$$
f _ {\theta} (\mathbf {s}, t) := \sum_ {i = 1} ^ {N} f _ {\theta_ {i}} (\mathbf {s}, t) = 3 \sum_ {i = 1} ^ {N} \prod_ {x \in \{s _ {1}, s _ {2}, t \}} \left[ \frac {\delta^ {3} \int M _ {i} d x}{\delta x ^ {3}} - \frac {\delta^ {3} \int L _ {i} d x}{\delta x ^ {3}} \right]
$$
We name our method Automatic Spatiotemporal Point Process (AutoSTPP).
We compare the performances of different neural STPPs using synthetic and real-world benchmark data. For synthetic data, our goal is to validate our AutoSTPP can accurately recover complex
Table 1: Test log likelihood (LL) and Hellinger distance of distribution (HD) on synthetic data (LL higher is better, HD lower is better). Comparison between AutoSTPP, NSTPP, Monte Carlo STPP, on synthetic datasets from two types of spatiotemporal point processes.
Spatiotemporal Hawkes process	Spatiotemporal Self Correcting process
DS1	DS2	DS3	DS1	DS2	DS3
LL	HD	LL	HD	LL	HD	LL	HD	LL	HD	LL	HD
NSTPP	-5.3110	0.5341	-4.8564	0.5849	-3.7366	0.1498	-2.0759	0.5426	-2.3612	0.3933	-3.0599	0.3097
DSTPP	-3.8240	0.0033	-3.1142	0.4920	-3.6327	0.0908	-1.2248	0.2348	-1.4915	0.1813	-1.3927	0.2075
Monte Carlo STPP	-4.0066	0.3198	-3.2778	0.3780	-3.7704	0.2587	-1.0317	0.1224	-1.3681	0.1163	-1.4439	0.2334
AutoSTPP	-3.9548	0.3018	-2.5304	0.1891	-3.7700	0.1495	-1.0269	0.1216	-1.3657	0.1119	-1.3979	0.2181
intensity functions. Additionally, we show that the errors resulting from numerical integration lead to a higher variance in the learned intensity than closed-form integration. We show that our model performs better or on par with the state-of-the-art methods for real-world data.
Synthetic Datasets. We follow the experiment design of Zhou et al. [2022] to validate that our method can accurately recover the true intensity functions of complex STPPs. We use six synthetic point process datasets simulated using Ogata's thinning algorithm [Chen, 2016], see Appendix B for details. The first three datasets were based on spatiotemporal Hawkes processes, while the remaining three were based on spatiotemporal self-correcting processes. Each dataset spans a time range of [0, 10000) and is generated using a fixed set of parameters. Each dataset was divided into a training, validation, and testing set in an  $8:1:1$  ratio based on the time range.
Spatiotemporal Hawkes process (STH). A spatiotemporal Hawkes process, also known as a self-exciting process, posits that every past event exerts an additive, positive, and spatially local influence on future events. This pattern is commonly observed in social media and earthquakes. The process is characterized by the intensity function [Reinhart, 2018]:
$$
\lambda^ {*} (\mathbf {s}, t) := \mu g _ {0} (\mathbf {s}) + \sum_ {i: t _ {i} <   t} g _ {1} (t, t _ {i}) g _ {2} (\mathbf {s}, \mathbf {s} _ {i}): \mu > 0. \tag {5}
$$
$g_{0}$  represents the density of the background event distribution over  $S$ .  $g_{2}$  represents the density of the event influence distribution centered at  $\mathbf{s}_i$  and over  $S$ .  $g_{1}$  describes each event  $t_i$ 's influence decay over time. We implement  $g_{0}$  and  $g_{2}$  as Gaussian densities and  $g_{2}$  as exponential decay functions.
Spatiotemporal Self-Correcting process (STSC). A spatiotemporal Self-Correcting process assumes that the background intensity always increases between the event arrivals. Each event discretely reduces the intensity in the vicinity. The STSC is often used for modeling events with regular intervals, such as animal feeding times. It is characterized by:
$$
\lambda^ {*} (\mathbf {s}, t) = \mu \exp \left(g _ {0} (\mathbf {s}) \beta t - \sum_ {i: t _ {i} <   t} \alpha g _ {2} (\mathbf {s}, \mathbf {s} _ {i})\right): \alpha , \beta , \mu > 0 \tag {6}
$$
$g_{0}(\mathbf{s})$  again represents the density of the background event distribution.  $g_{2}(\mathbf{s},\mathbf{s}_{i})$  represents the density of the negative event influence centered at  $\mathbf{s}_i$ .
See the Appendix B for the simulation parameters of the six synthetic datasets.
Real-world Datasets. We follow the experiment design of Chen et al. [2020] and use two of the real-world datasets, Earthquake Japan and COVID New Jersey. The first dataset includes information on the times and locations of all earthquakes in Japan between 1990 and 2020, with magnitudes of at least 2.5. This dataset includes 1050 sequences over a  $[0,30)$  time range and is split with a ratio of  $950:50:50$ . The second dataset is published by The New York Times and describes COVID-19 cases in New Jersey at the county level. This dataset includes 1650 sequences over a  $[0,7)$  time range and is split with a ratio of  $1450:100:100$ .
Evaluation Metrics. For real-world datasets, we report the average test log-likelihood (LL) of events. For synthetic datasets, the ground truth intensities are available, so we report the test log-likelihood
(LL) and the time-average Hellinger distance between the learned conditional spatial distribution  $f^{*}(\mathbf{s}|t)$  and the ground truth distribution. The distributions are estimated as multinomial distributions  $P = \{p_i,\dots,p_k\}$  and  $Q = \{q_{i},\dots,q_{k}\}$  at  $k$  discretized grid points. The Hellinger distance is then calculated as  $H(P,Q) = \frac{1}{\sqrt{2}}\sqrt{\sum_{i = 1}^{k}\left(\sqrt{p_i} - \sqrt{q_i}\right)^2}$
Baselines. We compare with two state-of-the-art neural STPP models, NSTPP [Chen et al., 2020] and Deep-STPP [Zhou et al., 2022]. We also design another baseline, Monte Carlo STPP, which uses the same underlying model as AutoSTPP but applies numerical integration instead of AutoInt to calculate the loss. For a fair comparison, Monte Carlo STPP models the influence  $f_{\theta}^{*}(s,t)$  as a multi-layer perceptron instead of a derivative network. This numerical baseline aims to demonstrate the benefit of automatic integration.
Exact Likelihood. Not all baseline methods have exact likelihood available. NSTPP [Chen et al., 2020] uses a numerical Neural-ODE solver. DeepSTPP [Zhou et al., 2022] introduces a VAE that optimizes a lower bound of likelihood. Monte Carlo STPP estimates the triple integral of intensity by Monte Carlo integration. As such, the results presented in Table 1 are estimated for these baseline models, whereas for our model, the results reflect the true likelihood.
Advantage of AutoInt. We visualize and compare two sample sets of intensities from STH Dataset 1 and STSC Dataset 3 in Figure 3. While the influence function approximator in Monte Carlo STPP can theoretically approximate more functions than Auto-STPP, the intensity it learns is "flatter" than the intensity learned by Auto-STPP. This flatness indicates a lack of information regarding future event locations.
Figure 4: Forward mixed  $(d / dx_{1}dx_{2}\dots dx_{k})$  partial derivative computation average speed comparison between our efficient implementation and PyTorch naive AutoGrad, for a two-layer MLP.
The primary reason behind this flatness can be traced back to numerical errors inherent in the Monte Carlo method. The Monte Carlo integration performs poorly for sharply localized integrants because its samples are homogeneous over the phase space. The intensity of ST-Hawkes is a localized function; it is close to zero in most of the places but high near some event locations. As a result, Monte Carlo STPP can hardly recover the sharpness of the ground truth intensity. NSTPP also uses Monte Carlo integration and suffers the same drawback on STH Dataset 1. In contrast, AutoSTPP evaluates the integral with closed-form integration and alleviates this issue.
Synthetic Datasets Results. Table 1 compares the test LL and the Hellinger distance between AutoSTPP and the baseline models on the six synthetic datasets. For the STSC datasets, we can see that AutoSTPP accurately recovers the intensity functions compared to other models. In Figure 3, AutoSTPP is the only model whose peak intensity location is always the same as the ground truth. DeepSTPP does not perform well in learning the dynamics of the STSC dataset; it struggles to align the peak and tends to learn the flat intensity as overly sharp. The peak intensity location of NSTPP is also biased.
Table 2: Test log likelihood (LL) comparison for space and time on real-world benchmark data, mean and standard deviation over three runs.
LL	COVID-19 NY	Earthquake JP
NSTPP	2.5566±0.0447	-4.4949±0.1172
DSTPP	2.3433±0.0109	-3.9852±0.0129
MonteSTPP	2.1070±0.0342	-3.6085±0.0436
AutoSTPP	2.6243±0.5905	-3.5948±0.0025
For the STHP datasets, DeepSTPP has a clear advantage because it uses Gaussian kernels to approximate the Gaussian ground truth. In Table 1, AutoSTPP outperforms all other models except DeepSTPP, and its performance is comparable. Figure 3 shows that Monte Carlo STPP and NSTPP
can only learn an unimodal function, whereas AutoSTPP can capture multi-modal behavior in the ground truth, especially the small bumps near the mode.
Real-world Datasets Results. Table 2 compares the test LL of AutoSTPP against the baseline models on the earthquakes and COVID datasets. Our model demonstrates superior performance, outperforming all the state-of-the-art methods. One should note that while Monte Carlo STPP shows performance comparable to ours on the Earthquake JP dataset, it falls short when applied to the COVID-19 NY dataset. We attribute this discrepancy to the large low-population-density areas in the COVID-19 NY data, which causes higher numerical error in integration.
4.3 Computational Efficiency
Figure 4 visualizes the benefit of using our implementation of AutoInt instead of the PyTorch naive implementation. More visualizations of the forward and backward computation times can be found in Appendix A. We can see that our implementation can be extended to compute any order of partial derivative. It is significantly faster than the naive autograd. In our AutoSTPP, we calculate the intensity using the product of three first-order derivatives. Our implementation would lead to a speedup of up to  $68\%$  for computing each first-order derivative.
We propose Automatic Integration for neural spatiotemporal point process models (AutoSTPP) using a dual network approach. AutoSTPP can efficiently compute the exact likelihood of any sophisticated intensity.
We validate the effectiveness of our method using synthetic data and real-world datasets and demonstrate that it significantly outperforms other point process models with numerical integration when the ground truth intensity function is localized.
However, like any approach, ours is not without its limitations. While AutoSTPP excels in computing the likelihood, sampling from AutoSTPP is computationally expensive as we only have the expression of the probability density. Closed-form computation of expectations is also not possible; Knowing the form of  $\int \lambda (t)$ , calculating  $\int t\lambda (t)$  is still intractable.
Our work presents a new paradigm for learning continuous-time dynamics. Currently, our neural process model takes the form of Hawkes processes (self-exciting) but cannot handle the discrete decreases of intensity after events due to the difficulty of integration. Future work includes relaxing the form of the intensity network with advanced integration techniques. Another interesting direction is to increase the approximation ability of the product network.
This work was supported in part by U. S. Army Research Office under Army-ECASE award W911NF07-R-0003-03, the U.S. Department Of Energy, Office of Science, IARPA HAYSTAC Program, NSF Grants #2205093, #2146343, and #2134274.
Ryan Prescott Adams, Iain Murray, and David JC MacKay. Tractable nonparametric bayesian inference in poisson processes with gaussian process intensities. In Proceedings of the 26th annual international conference on machine learning, pages 9-16, 2009.
Anders Brix and Peter J Diggle. Spatiotemporal prediction for log-gaussian cox processes. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(4):823-841, 2001.
Anders Brix and Jesper Moller. Space-time multi type log gaussian cox processes with a view to modelling weeds. Scandinavian Journal of Statistics, 28(3):471-488, 2001.
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. arXiv preprint arXiv:2011.04583, 2020.
Utkarsh Upadhyay, Abir De, and Manuel Gomez-Rodriguez. Deep reinforcement learning of marked temporal point processes. arXiv preprint arXiv:1805.09360, 2018.
Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Le Song, and Hongyuan Zha. Wasserstein learning of deep generative point process models. arXiv preprint arXiv:1705.08051, 2017.
Junchi Yan, Xin Liu, Liangliang Shi, Changsheng Li, and Hongyuan Zha. Improving maximum likelihood estimation of temporal point process via discriminative and adversarial learning. In IJCAI, pages 2948-2954, 2018.
Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes process. In International Conference on Machine Learning, pages 11183-11193. PMLR, 2020.
Zihao Zhou and Rose Yu. Automatic integration for fast and interpretable neural point process. In Learning for Dynamics and Control Conference. PMLR, 2023.
Zihao Zhou, Xingyi Yang, Ryan Rossi, Handong Zhao, and Rose Yu. Neural point process for learning spatiotemporal event dynamics. In Learning for Dynamics and Control Conference, pages 777-789. PMLR, 2022.
A More Implementation Benchmarks
Figure 5: Forward mixed  $(d / dx_{1}dx_{2}\dots dx_{k})$  partial derivative computation speed, 3 layers MLP
Figure 6: Forward mixed  $(d / dx_{1}dx_{2}\dots dx_{k})$  partial derivative computation speed, 4 layers MLP
As the number of MLP layers increases, the lower-order derivative computation becomes faster (relative to PyTorch naive implementation), whereas the higher-order derivative computation becomes slower. This performance pattern is because our implementation uses Python for loop. Our approach is faster than the baseline in the majority of cases.
Figure 7: Forward + Backward mixed  $(d / dx_{1}dx_{2}\dots dx_{k})$  partial derivative computation speed, 3 layers MLP
Figure 8: Forward + Backward univariate  $(d / dx_{1}dx_{1}\dots dx_{1})$  partial derivative computation speed, 3 layers MLP
As for both forward and backward computation times, our implementation still consistently surpasses the baseline in terms of speed. Notably, our method is even more efficient when calculating univariate partial derivatives than mixed partial derivatives. This advantage is primarily due to the reduced number of iterations required by the Python for-loop in the case of univariate derivatives.
B STSC and ST-Hawkes Introduction and Simulation Parameters
We use the same parameters as Zhou et al. [2022].
The STSCP's and the STHP's kernels  $g_0(\mathbf{s})$  and  $g_2(\mathbf{s},\mathbf{s}_j)$  are prespecified to be Gaussian:
$$
g _ {0} (\mathbf {s}) := \frac {1}{2 \pi} | \Sigma_ {g 0} | ^ {- \frac {1}{2}} \exp \left(- \frac {1}{2} (\mathbf {s} - [ 0, 0 ]) \Sigma_ {g 0} ^ {- 1} (\mathbf {s} - [ 0, 0 ]) ^ {T}\right)
$$
$$
g _ {2} (\mathbf {s}, \mathbf {s} _ {j}) := \frac {1}{2 \pi} | \Sigma_ {g 2} | ^ {- \frac {1}{2}} \exp \left(- \frac {1}{2} (\mathbf {s} - \mathbf {s} _ {j}) \Sigma_ {g 2} ^ {- 1} (\mathbf {s} - \mathbf {s} _ {j}) ^ {T}\right)
$$
The STSCP is defined on  $S = [0,1] \times [0,1]$ , while the STHP is defined on  $S = \mathbb{R}^2$ . The STSCP's kernel functions are normalized according to their cumulative probability on  $S$ . Table 4 shows the
simulation parameters. We discretized the STSCP's spatial domain as a  $101 \times 101$  grid during the simulation.
Table 3: Parameter settings for the synthetic dataset
α	β	μ	Σg0	Σg2
ST-Hawkes	DS1	.5	1	.2	[.2 0; 0 .2]	[0.5 0; 0 0.5]
DS2	.5	.6	.15	[5 0; 0 5]	[.1 0; 0 .1]
DS3	.3	2	1	[1 0; 0 1]	[.1 0; 0 .1]
ST-Self Correcting	DS1	.2	.2	1	[1 0; 0 1]	[0.85 0; 0 0.85]
DS2	.3	.2	1	[.4 0; 0 .4]	[.3 0; 0 .3]
DS3	.4	.2	1	[.25 0; 0 .25]	[.2 0; 0 .2]
Each dataset is a single, long sequence that spans over 10,000 time units. We divide each dataset into 50 sequences, each spanning 200 time units. We use 40 sequences for training, 5 for validation, and 5 for testing. Here's a summary of the total number of events found in each dataset:
Table 4: Number of events in each synthetic dataset
number of events
ST-Hawkes	DS1	3983
DS2	9017
DS3	11693
ST-Self Correcting	DS1	10002
DS2	6668
DS3	5004
The prediction task applies sliding windows to each of the datasets. We try to use historical events to predict the likelihood of the next event in the same sequence.
C Model Setup Details
We detail the specific hyperparameter settings in Table 5. Except for the learning rate, the same set of parameters was applied across all datasets. Despite varying datasets, This consistency in performance demonstrates our model's robustness to hyperparameters.
Name	Value	Description
Optimizer	Adam	-
Learning rate	-	Depends on dataset, [0.0002, 0.004]
Momentum	0.9	Adam momentum
Epoch	50 / 100	50 for synthetic dataset and 100 for real-world dataset
Batch size	128	-
Activation	tanh	Activation function in L and M (intensity parameter networks)
N	2 / 10	Number of product nets to sum in L and M
2 for synthetic dataset and 10 for real-world dataset
bias	true	L and M use bias in their linear layers
Table 5: Hyperparameter settings for training AutoSTPP on all datasets.
D Forward-pass Algorithm for Automatic Integration
Function: dnforward  $(f, n, x, \mathrm{dims})$ , partition  $(n, k)$  finds all k-subset partitions of  $n$
Data:  $n$ , dimension of  $f(x)$ ,  $x$ , a tensor of shape (batch, dim),
dims, list of dimensions to derive, layers, composite functions in  $f$
Result:  $d^{\mathrm{dims}}f / dx^{\mathrm{dims}}$
Initialize dictionary dnf, mapping from dims to  $d^{\mathrm{dim}}f / dx^{\mathrm{dim}}$  , empty list pd
if  $|dims| = 1$  then
Precompute  $f(x)$    
pd  $\leftarrow \delta_{i,\mathrm{dim}}|_{i\in [1,n]}$    
else for subdims  $\in$  combination(dims, len(dims)-1) do | Precompute dnforward(f, n, x, subdims) end   
pd  $\leftarrow 0$
end for layer  $\in$  layers do
if layer is linear then pd append last  $\mathrm{pd}\times W^T$ $W$  is the linear weight
else if layer is activation then if  $|dims| = 1$  then termsum  $\leftarrow$  last pd  $\times 1$
else termsum  $\leftarrow 0$  for order  $\in 0,\dots ,|$  dims|do if order  $= 0$  then term  $\leftarrow$  last pd else term  $\leftarrow 0$  for part  $\in$  partition(dims,order  $+1$  )do temp  $\leftarrow 1$  for subdims  $\in$  part do temp  $\leftarrow$  temp  $\times$  dnforward  $(f,n,x,$  subdims)(precomputed) term  $\leftarrow$  term  $^+$  temp end termsum  $\leftarrow$  termsum  $^+$  term end end termsum  $\leftarrow$  termsum  $\times$  layer  $(n)(f)$  end end end append termsum   
end return last pd
E Universal Approximation Theorem for Derivative Network
Consider an AutoInt integral network with the form
$$
g (x) = C \cdot (\sigma \circ (A \cdot x + b)), \qquad A \subseteq \mathbb {R} ^ {k \times n}, b \subseteq \mathbb {R} ^ {k}, C \subseteq \mathbb {R} ^ {k},
$$
where  $\sigma$  denotes a  $\mathbb{R} \to \mathbb{R}$  continuous non-polynomial function applied elementwise to each input dimension.
The derivative network thus takes the form
$$
g ^ {\prime} (x) = C \cdot \left(\sigma^ {\prime} \circ (A \cdot x + b) \circ A _ {c o l}\right),
$$
where  $A_{col} \subseteq \mathbb{R}^k$  is a column of  $A$  that corresponds to the deriving dimension.
Recall the universal approximation theorem [Daniels and Velikova, 2010], which says for every compact  $K \subseteq \mathbb{R}^n$  and  $f \in C(K, \mathbb{R}), \varepsilon > 0$ , there exist  $A, b, C$  such that
$$
\sup  _ {x \in K} \| f (x) - g (x) \| <   \varepsilon
$$
Proposition E.1. (Universal Approximation Theorem for Derivative Network) for every compact  $K \subseteq \mathbb{R}^n$  and  $f \in C(K, \mathbb{R}), \varepsilon > 0$ , there exists  $A \in \mathbb{R}^{k \times n}, b \in \mathbb{R}^k, C \in \mathbb{R}^k, \beta \in \mathbb{R}$  such that
$$
g (x) := C \cdot (\sigma \circ (A x + b)) - \beta x
$$
$$
\sup  _ {x \in K} \| f (x) - g ^ {\prime} (x) \| <   \varepsilon
$$
Proof. Given the mapping  $f$ , by UAT, there exists  $A, b, C$  that approximate  $f(x)$ . Construct  $\tilde{C} \in \mathbb{R}^k$  and  $\beta \in \mathbb{R}$ , such that