SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zihao Zhou
Department of Computer Science  
University of California, San Diego
La Jolla, CA 92092
ziz244@ucsd.edu
Rose Yu
roseyu@ucsd.edu
Learning continuous-time point processes is essential to many discrete event forecasting tasks. However, integration poses a major challenge, particularly for spatiotemporal point processes (STPPs), as it involves calculating the likelihood through triple integrals over space and time. Existing methods for integrating STPP either assume a parametric form of the intensity function, which lacks flexibility; or approximating the intensity with Monte Carlo sampling, which introduces numerical errors. Recent work by Omi et al. [2019] proposes a dual network approach for efficient integration of flexible intensity function. However, their method only focuses on the 1D temporal point process. In this paper, we introduce a novel paradigm: AutoSTPP (Automatic Integration for Spatiotemporal Neural Point Processes) that extends the dual network approach to 3D STPP. While previous work provides a foundation, its direct extension overly restricts the intensity function and leads to computational challenges. In response, we introduce a decomposable parametrization for the integral network using ProdNet. This approach, leveraging the product of simplified univariate graphs, effectively sidesteps the computational complexities inherent in multivariate computational graphs. We prove the consistency of AutoSTPP and validate it on synthetic data and benchmark real-world datasets. AutoSTPP shows a significant advantage in recovering complex intensity functions from irregular spatiotemporal events, particularly when the intensity is sharply localized. Our code is open-source at https://github.com/Rose-STM-Lab/AutoSTPP.
Spatiotemporal point process (STPP) [Daley and Vere-Jones, 2007, Reinhart, 2018] is a continuous time stochastic process for modeling irregularly sampled events over space and time. STPPs are particularly well-suited for modeling epidemic outbreaks, ride-sharing trips, and earthquake occurrences. A central concept in STPP is the intensity function, which captures the expected rates of events occurrence. Specifically, given the event sequence over space and time  $\mathcal{H}_t = \{\left(\mathbf{s}_1,t_1\right),\dots ,\left(\mathbf{s}_n,t_n\right)\}_{t_n\leq t}$ , the joint log-likelihood of the observed events is:
$$
\log p \left(\mathcal {H} _ {t}\right) = \sum_ {i = 1} ^ {n} \log \lambda^ {*} \left(\mathbf {s} _ {i}, t _ {i}\right) - \int_ {\mathcal {S}} \int_ {0} ^ {t} \lambda^ {*} (\mathbf {u}, \tau) d \mathbf {u} d \tau \tag {1}
$$
$\lambda^{*}$  is the optimal intensity,  $S$  the spatial domain,  $\mathbf{u}$  and  $\tau$  the space and time, and  $t$  the time range.
Learning STPP requires multivariate integrals of the intensity function, which is numerically challenging. Traditional methods often assume a parametric form of the intensity function, such as the integral
can have a closed-form solution Daley and Vere-Jones [2007]. But this also limits the expressive power of the model in describing complex spatiotemporal patterns.
Others propose to parameterize the model using neural ODE models Chen et al. [2020] and Monte Carlo sampling, but their computation is costly for high-dimensional functions. Recently, work by Zhou et al. [2022] proposes a nonparametric STPP approach. They use kernel density estimation for the intensity and model the parameters of the kernels with a deep generative model. Still, their method heavily depends on the number of background data points chosen as a hyper-parameter. Too few background points cause the intensity function to be an inflexible Gaussian mixture, while too many background points may cause overfitting on event arrival.
To reduce computational cost while maintaining expressive power, we propose a novel automatic integration scheme based on dual networks to learn STPPs efficiently. Our framework models intensity function as the sum of background and influence functions. Instead of relying on closed-form integration or Monte Carlo sampling, we directly approximate the integral of the influence function with a deep neural network (DNN). Taking the partial derivative of a DNN results in a new computational graph that shares the same parameters; see Figure 1.
First, we construct an integral network whose derivative is the intensity function. Then, we train the network parameters by maximizing the data likelihood formulated in terms of the linear combinations of the integral networks. Finally, we reassemble the parameters of the integral network to obtain the intensity. This approach leads to the exact intensity function and its antiderivative without restricting its parametric forms, thereby making the integration process "automatic".
Our approach bears a resemblance with the fully NN approach by Omi et al. [2019] for 1D temporal point processes. There, automatic integration can be easily implemented by imposing monotonicity constraints on the integral network [Lindell et al., 2021, Li et al., 2019]. However, due to triple integration in STPP, imposing monotonicity constraints significantly hurdles the expressivity of the integral network, leading to inaccurate intensity. As the experiments show, extending Fully NN to 3D cannot learn complex spatiotemporal intensity functions. Instead, we propose a decomposable parametrization for the integral network that bypasses this restriction.
Figure 1: Illustration of AutoSTPP.  $W$  denotes the linear layer's weight.  $\sigma$  is the nonlinear activation function. Left shows the intensity network approximating  $\lambda(\mathbf{s}, t)$ , and right is the integral network that computes  $\int_{\mathbf{s}} \int_{t} \lambda$ . The two networks share the same parameters.
Our approach can efficiently compute the exact likelihood of any continuous influence function. We validate our approach using synthetic spatiotemporal point processes with complex intensity functions. We also demonstrate the superior performance on several real-world discrete event datasets. Compared to FullyNN by Omi et al. [2019], our approach presents a more general higher-order automatic integration scheme and is more effective in learning complex intensity functions. Also, the probability density of Omi et al. [2019] is ill-defined as its intensity integral does not diverge to infinity [Shchur et al., 2019]. We fix the issue by adding a constant background intensity  $\mu$ .
To summarize, our contributions include:
- We propose the first deep learning framework AutoSTPP to speed up spatiotemporal point process learning with automatic integration. We use dual networks and enforce the nonnegativity of the intensity via a monotone integral network.
- We show that our automatic integration scheme empirically learns intensity functions more accurately than other integration approaches.
- We prove that the derivative network of AutoSTPP is a universal approximator of continuous functions and, therefore, is a consistent estimator under mild assumptions.
- We demonstrate that AutoSTPP can recover complex influence functions from synthetic data, enjoys high training efficiency and model interpretability, and outperforms the state-of-the-art methods on benchmark real-world data.
Parametrizing Point Process. Fitting traditional STPP, such as the spatiotemporal Hawkes process, to data points with parametric models can perform poorly if the model is misspecified. To address this issue, statisticians have extensively studied semi- and non-parametric inference for STPP. Early works like Brix and Moller [2001], Brix and Diggle [2001] usually rely on Log-Gaussian Cox processes as a backbone and Epanechnikov kernels as estimators of the pair correlation function. Adams et al. [2009] propose a non-parametric approach that allows the generation of exact Poisson data from a random intensity drawn from a Gaussian Process, thus avoiding finite-dimensional approximation. These Cox process models are scalable but assume a continuous intensity change over time.
Recently, neural point processes (NPPs) that combine point processes with neural networks have received considerable attention [Yan et al., 2018, Upadhyay et al., 2018, Huang et al., 2019, Shang and Sun, 2019, Zhang et al., 2020]. Under this framework, models focus more on approximating a discrete set of intensities before and after each event. The continuous intensity comes from interpolating the intensities between discrete events. For example, [Du et al., 2016] uses an RNN to generate intensities after each event. [Mei and Eisner, 2016] proposes a novel RNN architecture that generates intensities at both ends of each inter-event interval. Other works consider alternative training schema: Xiao et al. [2017] used Wasserstein distance, Guo et al. [2018] introduced noise-contrastive estimation, and Li et al. [2018] leveraged reinforcement learning. While these NPP models are more expressive than the traditional point process models, they still assume simple (continuous, usually monotonous) inter-event intensity changes and only focus on temporal point processes.
Neural STPP [Chen et al., 2020, Zhou et al., 2022] further generalizes NPP to spatiotemporal data. They use a non-negative activation function to map the hidden states to a scalar, i.e., the temporal intensity immediately after an event, and a conditional spatial distribution. The change of intensity between events is represented by a decay function or a Neural ODE. The conditional spatial distribution is represented by a kernel mixture or a normalizing flow. Nevertheless, all models assume a continuous transformation of the intensity function and have limited expressivity.
In the context of temporal point processes (TPP), closely related approaches are Omi et al. [2019] and Zhou and Yu [2023]. Both propose using a Deep Neural Network (DNN) to parameterize the integral of an intensity function. The work by Omi et al. [2019] offers a more flexible formulation yet does not incorporate any specific prior assumptions about the form of the intensity changes. On the other hand, the approach by Zhou and Yu [2023] is capable of capturing more sophisticated influence functions, and it is this work that our research builds upon. However, both of these studies focus on the easier problem of learning the derivative with respect to time alone, thereby neglecting the rich amount of other features that may associated with the timestamps.
Integration Methods. Integration methods are largely ignored in NPP literature but are central to a model's ability to capture the complex dynamics of a system. Existing works either use an intensity function with an elementary integral [Du et al., 2016] or Monte Carlo integration [Mei and Eisner, 2016]. However, we will see in the experiment section that the choice of integration method has a non-trivial effect on the model performance.
Integration is generally more complicated than differentiation, which can be mechanically solved using the chain rule. Most integration rules, e.g., integration by parts and change of variables, transform an antiderivative to another that is not necessarily easier. Elementary antiderivative only exists for a small set of functions, but not for simple composite functions such as  $\exp (x^2)$  [Dunham, 2018]. The Risch algorithm can determine such elementary antiderivative [Risch, 1969, 1970] but has never been fully implemented due to its complexity. The most commonly used integration methods are still numerical: Newton-Cotes Methods, Romberg Integration, Quadrature, and Monte Carlo integration [Davis and Rabinowitz, 2007].
Several recent works leverage automatic differentiation to speedup integration, a paradigm known as Automatic Integration (AutoInt). Liu [2020] proposes integrating the Taylor polynomial using the derivatives from Automatic Differentiation (AutoDiff). It requires partitioning of the integral limits
and choosing the order of Taylor approximation. Though it makes use of AutoDiff, the integration procedure involves a trade-off between runtime and accuracy and is numerical in nature. Li et al. [2019] and Lindell et al. [2021] proposed a dual network approach, which we will discuss in detail in Section 3. This approach guarantees a closed-form integral and is efficient.
We first review the background of Spatiotemporal Point Processes. Then, we introduce the AutoInt technique, which is interpretable and flexible. Lastly, we consider applying the AutoInt technique to the 3D integration in the spatiotemporal point process.
3.1 Spatiotemporal Point Process
A spatiotemporal point process (STPP) generalizes TPP to model the number of events  $N(S \times (a, b))$  that occurred in the Cartesian product of the spatial domain  $S \subseteq \mathbb{R}^d$  ( $d$  is the spatial dimensionality) and the time interval  $(a, b]$ . It is characterized by a non-negative space-time intensity function given the event history  $\mathcal{H}_t := \{(\mathbf{s}_1, t_1), \dots, (\mathbf{s}_n, t_n)\}_{t_n \leq t}$ ,
$$
\lambda^ {*} (\mathbf {s}, t) := \lim  _ {\Delta \mathbf {s} \rightarrow 0, \Delta t \rightarrow 0} \frac {\mathbb {E} [ N (B (\mathbf {s} , \Delta \mathbf {s}) \times (t , t + \Delta t)) | \mathcal {H} _ {t} ]}{B (\mathbf {s} , \Delta \mathbf {s}) \Delta t} \tag {2}
$$
which is the probability of finding an event in an infinitesimal time interval  $(t, t + \Delta t]$  and an infinitesimal spatial ball  $S = B(\mathbf{s}, \Delta \mathbf{s})$  centered at location  $\mathbf{s}$ . Alternatively, an STPP can be seen as a TPP with a conditional spatial distribution  $f^{*}(\mathbf{s}|t)$ , such that  $\lambda^{*}(\mathbf{s}, t) = \lambda^{*}(t)f^{*}(\mathbf{s}|t)$ .
3.2 AutoInt Point Process
Consider the following NPP model that generalizes the spatiotemporal Hawkes process:
$$
\lambda^ {*} (\mathbf {s}, t) = \mu + \sum_ {t _ {i} <   t} f _ {\theta} ^ {+} \left(\mathbf {s} - \mathbf {s} _ {i}, t - t _ {i}, \mathcal {H} \left(t _ {i}\right)\right). \tag {3}
$$
Here  $\mu$  is the base intensity.  $f_{\theta}^{+}$  is a positive scalar function that takes space, time, and representations of event history  $\mathcal{H}(\mathbf{s}_i,t_i)$  as inputs. Each  $f_{\theta}^{+}$  is approximated by a DNN.
The two main benefits of such design are flexibility and interpretability.  $f_{\theta}$  is a neural network that can model complex inter-event change. The additive form allows the decomposition of the intensity function for event influence analysis. We extend this model into the spatiotemporal domain,
3.3 Automatic Integration (AutoInt)
One advantage of the neural STPP model in Equation 3 is that we can instantiate automatic integration (AutoInt) and calculate the volume integral  $\int_{t=a}^{b} f_{\theta}(\mathbf{s}, t, \mathbf{h}) := F_{\theta}(b, \mathbf{h}) - F_{\theta}(a, \mathbf{h})$ , where  $\mathbf{h}$  is the latent representation of the event history until  $t$  generated by a deep sequence model. AutoInt first constructs the integral network  $F_{\theta}$  and then reorganizes the computational graph of  $F_{\theta}$  to form the integrant, the derivative network  $f_{\theta}$ . The two networks thus share the same set of parameters  $\theta$ .
Specifically, let  $\mathbf{x} \coloneqq \mathbf{s} \oplus t \oplus \mathbf{h}$ , we approximate the integral of the intensity function with a DNN of the following form:
$$
F _ {\theta} (\mathbf {x}) = \mathbf {W} _ {n} \dots \left(\mathbf {W} _ {3} \sigma \left(\mathbf {W} _ {2} \sigma \left(\mathbf {W} _ {1} \mathbf {x}\right)\right)\right),
$$
where  $n$  is the number of layers,  $\mathbf{W}_k: \mathbb{R}^{M_k} \mapsto \mathbb{R}^{N_k}$  denotes the weight of the  $k$ -th linear layer of the neural network and  $\sigma$  denotes the elementwise nonlinearity.  $M_k$  and  $N_k$  are the input and output dimension for the  $k$ -th layer. Hence, the set of parameters in this neural network is  $\theta = \{\mathbf{W}_k \in \mathbb{R}^{M_k \times N_k}, \forall k\}$ .
The derivative network  $f_{\theta}$  is the partial derivative of the integral network  $F_{\theta}$ . As long as the activation function is differentiable everywhere, one can compute the intensity recursively,
$$
f _ {\theta} (\mathbf {x}) := \frac {\partial F _ {\theta}}{\partial t} (\mathbf {x}) = \mathbf {W} _ {n} \sigma^ {\prime} \left(\mathbf {W} _ {n - 1} \sigma \left(\mathbf {W} _ {n - 2} \dots \left(\mathbf {W} _ {1} \mathbf {x}\right)\right)\right) \dots \circ \mathbf {W} _ {2} \sigma^ {\prime} \left(\mathbf {W} _ {1} \mathbf {x}\right) \circ \mathbf {W} _ {1 1},
$$
where  $\circ$  indicates the Hadamard product, and  $\mathbf{W}_{11}$  is the first column of  $\mathbf{W}_1$ , i.e.,
$$
\mathbf {W} _ {1} := \left[ \begin{array}{c c c c} \mathbf {W} _ {1 1} & \mathbf {W} _ {1 2} & \ldots & \mathbf {W} _ {1, M _ {1}} \end{array} \right]
$$
Computing  $f_{\theta}(\mathbf{x})$  involves many repeated operations. For example, the result of  $\mathbf{W}_1\mathbf{x}$  is used for compute both  $\sigma (\mathbf{W}_1\mathbf{x})$  and  $\sigma^{\prime}(\mathbf{W}_1\mathbf{x})$ , see Figure 1. We have developed a program that harnesses the power of dynamical programming to compute derivatives efficiently with AutoDiff. See Appendix D for the detailed algorithm.
3.4 AutoInt Point Processes as Consistent Estimators
We show that the universal approximation theorem (UAT) holds for derivative networks. This theorem signifies that, given a sufficient number of hidden units, derivative networks can theoretically approximate any continuous functions, no matter how complex. Therefore, using derivative networks does not limit the range of influence functions that can be approximated.
Proposition 3.1 (Universal Approximation Theorem for Derivative Networks). The set of derivative networks corresponding to two-layer feedforward integral networks is dense in  $C(\mathbb{R})$  with respect to the uniform norm.
For a detailed proof, see Appendix E. With UAT, it is clear that under some mild assumptions, AutoInt Point Processes are consistent estimators of point processes that take the form of Equation 5.
Proposition 3.2 (Consistency of AutoInt Point Process). Under the assumption that the ground truth point process is stationary, ergodic, absolutely continuous and predictable, if the ground truth influence function is truncated (i.e.,  $\exists C$ ,  $f(t) = 0 \forall t > c$ ), the maximum likelihood estimator  $f_{\theta}$  converges to the true influence function  $f$  in probability as  $T \to \infty$ .
Our model belongs to the class of linear self-excitation processes, whose maximum likelihood estimator properties were analyzed by Ogata et al. [1978]. Under the assumptions above, two conditions are needed for the proof of consistency:
Assumption 3.3. (Consistency Conditions) For any  $\theta \in \Theta$  there is a neighbourhood  $U$  of  $\theta$  such that
1.  $\sup_{\theta^{\prime}\in U}|\lambda_{\theta^{\prime}}(t,\omega) - \lambda_{\theta^{\prime}}^{*}(t,\omega)|\to 0$  in probability as  $t\to \infty$
2.  $\sup_{\theta' \in U} |\log \lambda_{\theta^*}'(t, \omega)|$  has, for some  $\alpha > 0$ , finite  $(2 + \alpha)$  th moment uniform bounded with respect to  $t$ .
The first condition is satisfied by UAT. The second condition depends on the rate of decrease of the influence tail and is satisfied by truncation. In our experiments, we truncated the history by only including the influences of the previous 20 events. If the ground truth influence function decays over the entire time domain, our estimator may exhibit negligible bias.
3.5 3D Automatic Integration
AutoInt gives us an efficient way to calculate a line integral over one axis. However, for spatiotemporal point process models, we need to calculate the triple integral of the intensity function over space and time. Since we cannot evaluate the integral network with an input of infinity, we assume the spatial domain to be a rectangle, such that the triple integral is over a cuboid. We then convert the triple integral to line integrals using Divergence and Green's theorem [Marsden and Tromba, 2003].
Define  $\mathbf{s} \coloneqq (x, y)$  and  $t \coloneqq z$ , we model the spatiotemporal influence  $f_{\theta}^{*}(x, y, z)$  as
$$
f _ {\theta} ^ {*} (\mathbf {s}, t) = \frac {\partial P}{\partial x} + \frac {\partial Q}{\partial y} + \frac {\partial R}{\partial z} \tag {4}
$$
The Divergence theorem relates volume and surface integrals. It states that for any  $P(x,y,z), Q(x,y,z), R(x,y,z)$  that is differentiable over the cuboid  $\Omega$ , we have
$$
\iiint_ {\Omega} \left(\frac {\partial P}{\partial x} + \frac {\partial Q}{\partial y} + \frac {\partial R}{\partial z}\right) d v = \iint_ {\Sigma} P d y d z + Q d z d x + R d x d y,
$$
where  $\Sigma$  are the six rectangles that enclose  $\Omega$ .
Figure 2: Comparison of fitting a nonnegative function  $f(x,y,z) = \sin (x)\cos (y)\sin (z) + 1$ . Our proposed AutoInt ProdNet approach can well approximate the ground truth function. In contrast, imposing the constraint through activation function with a nonnegative triple derivative "Constrained Triple AutoInt" fails due to overly stringent constraint.
z=1.14
Using  $R$  as an example, we begin with two neural network approximators,  $L_{R}$  and  $M_{R}$ . We initialize the two approximators as first-order derivative networks, whose corresponding integral networks are  $\int L_{R} dx$  and  $\int M_{R} dy$ . We use the  $xy$  partial derivatives of the two networks to model  $R$ , such that  $R := \frac{\partial M_{R}}{\partial x} - \frac{\partial L_{R}}{\partial y}$ . Then the  $z$  partial derivative  $\frac{\partial R}{\partial z}$  is  $\frac{\partial^2 M_{R}}{\partial x \partial z} - \frac{\partial^2 L_{R}}{\partial y \partial z} = \frac{\partial^3 \int M_{R} dy}{\partial x \partial y \partial z} - \frac{\partial^3 \int L_{R} dx}{\partial x \partial y \partial z}$ . We can see that  $\frac{\partial R}{\partial z}$  can be exactly evaluated as the third derivatives of two integral networks. In the same manner, we can model  $P$  and  $Q$  such that the intensity is parametrized by six networks:  $L_{R}, L_{Q}, L_{P}, M_{R}, M_{Q}$ , and  $M_{P}$ .
We use neural network pairs  $\{L, M\}$  to parametrize each intensity term in Equation 4 according to Green's theorem. It states that for any  $L$  and  $M$  differentiable over the rectangle  $D$ ,
$$
\iint_ {D} \left(\frac {\partial L}{\partial x} - \frac {\partial M}{\partial y}\right) d x d y = \oint_ {L ^ {+}} (L d x + M d y),
$$
where  $L^{+}$  is the counterclockwise path that encloses  $D$ .  $\int Rdx dy$  is then  $\oint L_{R}dx + M_{R}dy$ , and can be exactly evaluated using the integral networks.
In practice, we observe that the six networks' parametrization of intensity can be simplified.  $L_{R}$ ,  $L_{Q}$ ,  $L_{P}$  can share the same set of weights, and similarly for  $M_{R}$ ,  $M_{Q}$ ,  $M_{P}$ . The influence is essentially parametrized by two different neural networks,  $L$  and  $M$ . That is,
$$
f _ {\theta} (t, \mathbf {h}) := \left(\frac {\delta^ {3} \int M d y}{\delta x \delta y \delta z} - \frac {\delta^ {3} \int L d x}{\delta x \delta y \delta z}\right) + \left(\frac {\delta^ {3} \int M d z}{\delta x \delta y \delta z} - \frac {\delta^ {3} \int L d y}{\delta x \delta y \delta z}\right) + \left(\frac {\delta^ {3} \int M d x}{\delta x \delta y \delta z} - \frac {\delta^ {3} \int L d z}{\delta x \delta y \delta z}\right)
$$
3.6 Imposing the 3D Non-negativity Constraint
For the NPP model in Equation 3, the derivative network  $f_{\theta}$  needs to be non-negative. Imposing the non-negativity constraint for 3D AutoInt is a challenging task. It implies that the integral network  $F_{\theta}$  always has a nonnegative triple derivative.