Title: Automatic Integration for Spatiotemporal Neural Point Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: spatiotemporal modeling

neural point processes

integration method
Abstract: Learning continuous-time point processes is essential to many discrete event forecasting tasks. However, integration poses a major challenge, particularly for spatiotemporal point processes (STPPs), as it involves calculating the likelihood through triple integrals over space and time. Existing methods for integrating STPP either assume a parametric form of the intensity function, which lacks flexibility; or approximating the intensity with Monte Carlo sampling, which introduces numerical errors. Recent work by Omi et al. proposes a dual network approach for efficient integration of flexible intensity function. However, their method only focuses on the 1D temporal point process. In this paper, we introduce a novel paradigm: `Auto-STPP` (Automatic Integration for Spatiotemporal Neural Point Processes) that extends the dual network approach to 3D STPP. While previous work provides a foundation, its direct extension overly restricts the intensity function and leads to computational challenges. In response, we introduce a decomposable parametrization for the integral network using ProdNet. This approach, leveraging the product of simplified univariate graphs, effectively sidesteps the computational complexities inherent in multivariate computational graphs. We prove the consistency of `Auto-STPP` and validate it on synthetic data and benchmark real-world datasets. `Auto-STPP` shows a significant advantage in recovering complex intensity functions from irregular spatiotemporal events, particularly when the intensity is sharply localized. Our code is open-source at https://github.com/Rose-STL-Lab/AutoSTPP.

Automatic Integration for Spatiotemporal Neural Point Processes

Abstract

Learning continuous-time point processes is essential to many discrete event forecasting tasks. However, integration poses a major challenge, particularly for spatiotemporal point processes (STPPs), as it involves calculating the likelihood through triple integrals over space and time. Existing methods for integrating STPP either assume a parametric form of the intensity function, which lacks flexibility; or approximating the intensity with Monte Carlo sampling, which introduces numerical errors. Recent work by Omi et al. [2019] proposes a dual network approach for efficient integration of flexible intensity function. However, their method only focuses on the 1D temporal point process. In this paper, we introduce a novel paradigm: AutoSTPP (Automatic Integration for Spatiotemporal Neural Point Processes) that extends the dual network approach to 3D STPP. While previous work provides a foundation, its direct extension overly restricts the intensity function and leads to computational challenges. In response, we introduce a decomposable parametrization for the integral network using ProdNet. This approach, leveraging the product of simplified univariate graphs, effectively sidesteps the computational complexities inherent in multivariate computational graphs. We prove the consistency of AutoSTPP and validate it on synthetic data and benchmark real-world datasets. AutoSTPP shows a significant advantage in recovering complex intensity functions from irregular spatiotemporal events, particularly when the intensity is sharply localized. Our code is open-source at https://github.com/Rose-STL-Lab/AutoSTPP .

1 Introduction

Spatiotemporal point process (STPP) [Daley and Vere-Jones, 2007, Reinhart, 2018] is a continuous time stochastic process for modeling irregularly sampled events over space and time. STPPs are particularly well-suited for modeling epidemic outbreaks, ride-sharing trips, and earthquake occurrences. A central concept in STPP is the intensity function , which captures the expected rates of events occurrence. Specifically, given the event sequence over space and time H t = { ( s 1 , t 1 ) , . . . , ( s n , t n ) } t n ≤ t , the joint log-likelihood of the observed events is:

λ ∗ is the optimal intensity, S the spatial domain, u and τ the space and time, and t the time range.

Learning STPP requires multivariate integrals of the intensity function, which is numerically challenging. Traditional methods often assume a parametric form of the intensity function, such as the integral

can have a closed-form solution Daley and Vere-Jones [2007]. But this also limits the expressive power of the model in describing complex spatiotemporal patterns.

Others propose to parameterize the model using neural ODE models Chen et al. [2020] and Monte Carlo sampling, but their computation is costly for high-dimensional functions. Recently, work by Zhou et al. [2022] proposes a nonparametric STPP approach. They use kernel density estimation for the intensity and model the parameters of the kernels with a deep generative model. Still, their method heavily depends on the number of background data points chosen as a hyper-parameter. Too few background points cause the intensity function to be an inflexible Gaussian mixture, while too many background points may cause overfitting on event arrival.

To reduce computational cost while maintaining expressive power, we propose a novel automatic integration scheme based on dual networks to learn STPPs efficiently. Our framework models intensity function as the sum of background and influence functions. Instead of relying on closedform integration or Monte Carlo sampling, we directly approximate the integral of the influence function with a deep neural network (DNN). Taking the partial derivative of a DNN results in a new computational graph that shares the same parameters; see Figure 1.

First, we construct an integral network whose derivative is the intensity function. Then, we train the network parameters by maximizing the data likelihood formulated in terms of the linear combinations of the integral networks. Finally, we reassemble the parameters of the integral network to obtain the intensity. This approach leads to the exact intensity function and its antiderivative without restricting its parametric forms, thereby making the integration process 'automatic'.

Our approach bears a resemblance with the fully NNapproach by Omi et al. [2019] for 1D temporal point processes. There, automatic integration can be easily implemented by imposing monotonicity constraints on the integral network [Lindell et al., 2021, Li et al., 2019]. However, due to triple integration in STPP, imposing monotonicity constraints significantly hurdles the expressivity of the integral network, leading to inaccurate intensity. As the experiments show, extending Fully NN to 3D cannot learn complex spatiotemporal intensity functions. Instead, we propose a decomposable parametrization for the integral network that bypasses this restriction.

入(s,t)dsdt

↑

Derivative

Network

D

Integral

入(s, t)

W1

W1

M

Network

OM

Wo

Wo

X

X

Time

Our approach can efficiently compute the exact likelihood of any continuous influence function. We validate our approach using synthetic spatiotemporal point processes with complex intensity functions. We also demonstrate the superior performance on several real-world discrete event datasets. Compared to FullyNN by Omi et al. [2019], our approach presents a more general higher-order automatic integration scheme and is more effective in learning complex intensity functions. Also, the probability density of Omi et al. [2019] is ill-defined as its intensity integral does not diverge to infinity [Shchur et al., 2019]. We fix the issue by adding a constant background intensity µ .

To summarize, our contributions include:

We propose the first deep learning framework AutoSTPP to speed up spatiotemporal point process learning with automatic integration. We use dual networks and enforce the nonnegativity of the intensity via a monotone integral network.

We show that our automatic integration scheme empirically learns intensity functions more accurately than other integration approaches.

We prove that the derivative network of AutoSTPP is a universal approximator of continuous functions and, therefore, is a consistent estimator under mild assumptions.

We demonstrate that AutoSTPP can recover complex influence functions from synthetic data, enjoys high training efficiency and model interpretability, and outperforms the state-ofthe-art methods on benchmark real-world data.

2 Related Work

Parametrizing Point Process. Fitting traditional STPP, such as the spatiotemporal Hawkes process, to data points with parametric models can perform poorly if the model is misspecified. To address this issue, statisticians have extensively studied semi- and non-parametric inference for STPP. Early works like Brix and Moller [2001], Brix and Diggle [2001] usually rely on Log-Gaussian Cox processes as a backbone and Epanechnikov kernels as estimators of the pair correlation function. Adams et al. [2009] propose a non-parametric approach that allows the generation of exact Poisson data from a random intensity drawn from a Gaussian Process, thus avoiding finite-dimensional approximation. These Cox process models are scalable but assume a continuous intensity change over time.

Recently, neural point processes (NPPs) that combine point processes with neural networks have received considerable attention [Yan et al., 2018, Upadhyay et al., 2018, Huang et al., 2019, Shang and Sun, 2019, Zhang et al., 2020]. Under this framework, models focus more on approximating a discrete set of intensities before and after each event. The continuous intensity comes from interpolating the intensities between discrete events. For example, [Du et al., 2016] uses an RNN to generate intensities after each event. [Mei and Eisner, 2016] proposes a novel RNN architecture that generates intensities at both ends of each inter-event interval. Other works consider alternative training schema: Xiao et al. [2017] used Wasserstein distance, Guo et al. [2018] introduced noise-contrastive estimation, and Li et al. [2018] leveraged reinforcement learning. While these NPP models are more expressive than the traditional point process models, they still assume simple (continuous, usually monotonous) inter-event intensity changes and only focus on temporal point processes.

Neural STPP [Chen et al., 2020, Zhou et al., 2022] further generalizes NPP to spatiotemporal data. They use a non-negative activation function to map the hidden states to a scalar, i.e., the temporal intensity immediately after an event, and a conditional spatial distribution. The change of intensity between events is represented by a decay function or a Neural ODE. The conditional spatial distribution is represented by a kernel mixture or a normalizing flow. Nevertheless, all models assume a continuous transformation of the intensity function and have limited expressivity.

In the context of temporal point processes (TPP), closely related approaches are Omi et al. [2019] and Zhou and Yu [2023]. Both propose using a Deep Neural Network (DNN) to parameterize the integral of an intensity function. The work by Omi et al. [2019] offers a more flexible formulation yet does not incorporate any specific prior assumptions about the form of the intensity changes. On the other hand, the approach by Zhou and Yu [2023] is capable of capturing more sophisticated influence functions, and it is this work that our research builds upon. However, both of these studies focus on the easier problem of learning the derivative with respect to time alone, thereby neglecting the rich amount of other features that may associated with the timestamps.

Integration Methods. Integration methods are largely ignored in NPP literature but are central to a model's ability to capture the complex dynamics of a system. Existing works either use an intensity function with an elementary integral [Du et al., 2016] or Monte Carlo integration [Mei and Eisner, 2016]. However, we will see in the experiment section that the choice of integration method has a non-trivial effect on the model performance.

Integration is generally more complicated than differentiation, which can be mechanically solved using the chain rule. Most integration rules, e.g., integration by parts and change of variables, transform an antiderivative to another that is not necessarily easier. Elementary antiderivative only exists for a small set of functions, but not for simple composite functions such as exp( x 2 ) [Dunham, 2018]. The Risch algorithm can determine such elementary antiderivative [Risch, 1969, 1970] but has never been fully implemented due to its complexity. The most commonly used integration methods are still numerical: Newton-Cotes Methods, Romberg Integration, Quadrature, and Monte Carlo integration [Davis and Rabinowitz, 2007].

Several recent works leverage automatic differentiation to speedup integration, a paradigm known as Automatic Integration (AutoInt). Liu [2020] proposes integrating the Taylor polynomial using the derivatives from Automatic Differentiation (AutoDiff). It requires partitioning of the integral limits

and choosing the order of Taylor approximation. Though it makes use of AutoDiff, the integration procedure involves a trade-off between runtime and accuracy and is numerical in nature. Li et al. [2019] and Lindell et al. [2021] proposed a dual network approach, which we will discuss in detail in Section 3. This approach guarantees a closed-form integral and is efficient.

3 Methodology

We first review the background of Spatiotemporal Point Processes. Then, we introduce the AutoInt technique, which is interpretable and flexible. Lastly, we consider applying the AutoInt technique to the 3D integration in the spatiotemporal point process.

3.1 Spatiotemporal Point Process

A spatiotemporal point process (STPP) generalizes TPP to model the number of events N ( S × ( a, b )) that occurred in the Cartesian product of the spatial domain S ⊆ R d ( d is the spatial dimensionality) and the time interval ( a, b ] . It is characterized by a non-negative space-time intensity function given the event history H t := { ( s 1 , t 1 ) , . . . , ( s n , t n ) } t n ≤ t ,

which is the probability of finding an event in an infinitesimal time interval ( t, t + ∆ t ] and an infinitesimal spatial ball S = B ( s , ∆ s ) centered at location s . Alternatively, an STPP can be seen as a TPP with a conditional spatial distribution f ∗ ( s | t ) , such that λ ∗ ( s , t ) = λ ∗ ( t ) f ∗ ( s | t ) .

3.2 AutoInt Point Process

Consider the following NPP model that generalizes the spatiotemporal Hawkes process:

Here µ is the base intensity. f + θ is a positive scalar function that takes space, time, and representations of event history H ( s i , t i ) as inputs. Each f + θ is approximated by a DNN.

The two main benefits of such design are flexibility and interpretability. f θ is a neural network that can model complex inter-event change. The additive form allows the decomposition of the intensity function for event influence analysis. We extend this model into the spatiotemporal domain,

3.3 Automatic Integration (AutoInt)

One advantage of the neural STPP model in Equation 3 is that we can instantiate automatic integration (AutoInt) and calculate the volume integral ´ b t = a f θ ( s , t, h ) := F θ ( b, h ) -F θ ( a, h ) , where h is the latent representation of the event history until t generated by a deep sequence model. AutoInt first constructs the integral network F θ and then reorganizes the computational graph of F θ to form the integrant, the derivative network f θ . The two networks thus share the same set of parameters θ .

Specifically, let x := s ⊕ t ⊕ h , we approximate the integral of the intensity function with a DNN of the following form:

where n is the number of layers, W k : R M k ↦→ R N k denotes the weight of the k -th linear layer of the neural network and σ denotes the elementwise nonlinearity. M k and N k are the input and output dimension for the k -th layer. Hence, the set of parameters in this neural network is θ = { W k ∈ R M k × N k , ∀ k } .

The derivative network f θ is the partial derivative of the integral network F θ . As long as the activation function is differentiable everywhere, one can compute the intensity recursively,

where ◦ indicates the Hadamard product, and W 11 is the first column of W 1 , i.e.,

Computing f θ ( x ) involves many repeated operations. For example, the result of W 1 x is used for compute both σ ( W 1 x ) and σ ′ ( W 1 x ) , see Figure 1. We have developed a program that harnesses the power of dynamical programming to compute derivatives efficiently with AutoDiff. See Appendix D for the detailed algorithm.

3.4 AutoInt Point Processes as Consistent Estimators

We show that the universal approximation theorem (UAT) holds for derivative networks. This theorem signifies that, given a sufficient number of hidden units, derivative networks can theoretically approximate any continuous functions, no matter how complex. Therefore, using derivative networks does not limit the range of influence functions that can be approximated.

Proposition 3.1 (Universal Approximation Theorem for Derivative Networks) . The set of derivative networks corresponding to two-layer feedforward integral networks is dense in C ( R ) with respect to the uniform norm.

For a detailed proof, see Appendix E. With UAT, it is clear that under some mild assumptions, AutoInt Point Processes are consistent estimators of point processes that take the form of Equation 5.

Proposition 3.2 (Consistency of AutoInt Point Process) . Under the assumption that the ground truth point process is stationary, ergodic, absolutely continuous and predictable, if the ground truth influence function is truncated (i.e., ∃ C, f ( t ) = 0 ∀ t > c ), the maximum likelihood estimator f θ converges to the true influence function f in probability as T →∞ .

Our model belongs to the class of linear self-excitation processes, whose maximum likelihood estimator properties were analyzed by Ogata et al. [1978]. Under the assumptions above, two conditions are needed for the proof of consistency:

Assumption 3.3. (Consistency Conditions) For any θ ∈ Θ there is a neighbourhood U of θ such that

sup θ ′ ∈ U | λ θ ′ ( t, ω ) -λ ∗ θ ′ ( t, ω ) |→ 0 in probability as t →∞ ,

sup θ ′ ∈ U | log λ ∗ θ ∗ ( t, ω ) | has, for some α > 0 , finite (2 + α ) th moment uniform bounded with respect to t .

The first condition is satisfied by UAT. The second condition depends on the rate of decrease of the influence tail and is satisfied by truncation. In our experiments, we truncated the history by only including the influences of the previous 20 events. If the ground truth influence function decays over the entire time domain, our estimator may exhibit negligible bias.

3.5 3D Automatic Integration

AutoInt gives us an efficient way to calculate a line integral over one axis. However, for spatiotemporal point process models, we need to calculate the triple integral of the intensity function over space and time. Since we cannot evaluate the integral network with an input of infinity, we assume the spatial domain to be a rectangle, such that the triple integral is over a cuboid. We then convert the triple integral to line integrals using Divergence and Green's theorem [Marsden and Tromba, 2003].

Define s := ( x, y ) and t := z , we model the spatiotemporal influence f ∗ θ ( x, y, z ) as

The Divergence theorem relates volume and surface integrals. It states that for any P ( x, y, z ) , Q ( x, y, z ) , R ( x, y, z ) that is differentiable over the cuboid Ω , we have

where Σ are the six rectangles that enclose Ω .

Ground Truth

Constrained Triple Autolnt

+

f

X

X

X

Z=1.14

Using R as an example, we begin with two neural network approximators, L R and M R . We initialize the two approximators as first-order derivative networks, whose corresponding integral networks are ´ L R dx and ´ M R dy . We use the xy partial derivatives of the two networks to model R , such that R := ∂M R ∂x -∂L R ∂y . Then the z partial derivative ∂R ∂z is ∂ 2 M R ∂x∂z -∂ 2 L R ∂y∂z =

∂ 3 ´ M R dy ∂x∂y∂z -∂ 3 ´ L R dx ∂x∂y∂z . We can see that ∂R ∂z can be exactly evaluated as the third derivatives of two integral networks. In the same manner, we can model P and Q such that the intensity is parametrized by six networks: L R , L Q , L P , M R , M Q , and M P .

We use neural network pairs { L, M } to parametrize each intensity term in Equation 4 according to Green's theorem. It states that for any L and M differentiable over the rectangle D ,

where L + is the counterclockwise path that encloses D . ´ Rdxdy is then ¸ L R dx + M R dy , and can be exactly evaluated using the integral networks.

In practice, we observe that the six networks' parametrization of intensity can be simplified. L R , L Q , L P can share the same set of weights, and similarly for M R , M Q , M P . The influence is essentially parametrized by two different neural networks, L and M . That is,

3.6 Imposing the 3D Non-negativity Constraint

For the NPP model in Equation 3, the derivative network f θ needs to be non-negative. Imposing the non-negativity constraint for 3D AutoInt is a challenging task. It implies that the integral network F θ always has a nonnegative triple derivative.

A simple approach is to apply an activation function with a nonnegative triple derivative. An integral network that uses this activation and has nonnegative linear layer weights satisfies the condition. We call this approach 'Constrained Triple AutoInt'. However, the output of an integral network can grow very quickly with large input, and the gradients are likely to explode during training. Moreover, the ∂F θ

non-negative constraint on the influence function only requires ∂ s ∂t to be positive. But an activation function with a nonnegative triple derivative would also enforce other partial derivatives to be positive. Such constraints are overly restrictive for STPPs, whose partial derivatives ∂F θ ∂ s ∂ s and ∂F θ ∂t∂t can both be negative when the intensity is well-defined.

ProdNet. We propose a different solution to enforce the 3D non-negative constraint called ProdNet . Specifically, we decompose the influence function f θ ( s 1 , s 2 , t ) : R 3 → R as f 1 θ ( s 1 ) f 2 θ ( s 2 ) f 3 θ ( t ) , the product of three R → R AutoInt derivative networks. The triple antiderivative of the influence

SpatialTemporal Hawkes(STH)

Spatial TemporalSelf-Correcting(STSC)

3.0

0.3

X

GT

2.0

0.2

1.0

0.1

3.0

Auto

0.2

0.3

2.0

1.0

0.1-

Monte

3.0

0.3-

0.2

Carlo

1.0

2.0

0.1.

3.0

0.3

Deep

2.0

入0.2

0.1

1.0

3.0

0.3

2.0

0.2

NSTPP

1.0

0.1

-2-1012

y

Q

2

2

X

0

2

-2

,0

-2-10

X

1

2

2

0.2A,6.810

X"

X

.810

140

150

160

170

75

80

85

06

time

time

function is then F 1 θ ( s 1 ) F 2 θ ( s 2 ) F 3 θ ( t ) , the product of their respective integral networks. Then we can apply 1D non-negative constraint to each of the derivative networks. The other partial derivatives are not constrained because F 1 θ , F 2 θ , F 3 θ can be negative.

One limitation of such decomposition is that it can only learn the joint density of marginally independent distribution. We circumvent this issue by parameterizing the influence function as the sum of N ProdNet, ∑ N i =1 f 1 θ,i ( s 1 ) f 2 θ,i ( s 2 ) f 3 θ,i ( t ) . The formulation is no longer marginally independent since it is not multiplicative decomposable. Figure 2 shows that the sum of two ProdNets is already sufficient to approximate a function that cannot be written as the sum of products of positive functions. In contrast, the constrained triple AutoInt's intensity is convex everywhere and fails to fit f .

Increasing the number of ProdNet improves AutoInt's flexibility at the cost of time and memory. We perform an ablation study of the number of ProdNet in Appendix F.

3.7 Model Training

Given the integral network F θ ( t, h ) := ´ S F θ ( s , t, h ) and the derivative network approximating the influence function f θ = ∂F θ ∂t∂ s , the log-likelihood of an event sequence H n = { ( s 1 , t 1 ) , · · · , ( s n , t n ) } observed in time interval [0 , T ] with respect to the model is

Obtaining the above from Equation 1 is straightforward by the Fundamental Theorem of Calculus. F θ is evaluated using the Divergence theorem. We can learn the parameters θ in both networks by maximizing the log-likelihood function. In experiments, we parametrize f θ with two AutoInt networks L and M . Each network is a sum of N ProdNets. That is,

We name our method Automatic Spatiotemporal Point Process ( AutoSTPP ).

4 Experiments

We compare the performances of different neural STPPs using synthetic and real-world benchmark data. For synthetic data, our goal is to validate our AutoSTPP can accurately recover complex

intensity functions. Additionally, we show that the errors resulting from numerical integration lead to a higher variance in the learned intensity than closed-form integration. We show that our model performs better or on par with the state-of-the-art methods for real-world data.

4.1 Experimental Setup

Synthetic Datasets. We follow the experiment design of Zhou et al. [2022] to validate that our method can accurately recover the true intensity functions of complex STPPs. We use six synthetic point process datasets simulated using Ogata's thinning algorithm [Chen, 2016], see Appendix B for details. The first three datasets were based on spatiotemporal Hawkes processes, while the remaining three were based on spatiotemporal self-correcting processes. Each dataset spans a time range of [0 , 10000) and is generated using a fixed set of parameters. Each dataset was divided into a training, validation, and testing set in an 8 : 1 : 1 ratio based on the time range.

Spatiotemporal Hawkes process (STH). A spatiotemporal Hawkes process, also known as a self-exciting process, posits that every past event exerts an additive, positive, and spatially local influence on future events. This pattern is commonly observed in social media and earthquakes. The process is characterized by the intensity function [Reinhart, 2018]:

g 0 represents the density of the background event distribution over S . g 2 represents the density of the event influence distribution centered at s i and over S . g 1 describes each event t i 's influence decay over time. We implement g 0 and g 2 as Gaussian densities and g 2 as exponential decay functions.

Spatiotemporal Self-Correcting process (STSC). A spatiotemporal Self-Correcting process assumes that the background intensity always increases between the event arrivals. Each event discretely reduces the intensity in the vicinity. The STSC is often used for modeling events with regular intervals, such as animal feeding times. It is characterized by:

g 0 ( s ) again represents the density of the background event distribution. g 2 ( s , s i ) represents the density of the negative event influence centered at s i .

See the Appendix B for the simulation parameters of the six synthetic datasets.

Real-world Datasets. We follow the experiment design of Chen et al. [2020] and use two of the real-world datasets, Earthquake Japan and COVID New Jersey . The first dataset includes information on the times and locations of all earthquakes in Japan between 1990 and 2020, with magnitudes of at least 2.5. This dataset includes 1050 sequences over a [0 , 30) time range and is split with a ratio of 950 : 50 : 50 . The second dataset is published by The New York Times and describes COVID-19 cases in New Jersey at the county level. This dataset includes 1650 sequences over a [0 , 7) time range and is split with a ratio of 1450 : 100 : 100 .

Evaluation Metrics. For real-world datasets, we report the average test log-likelihood (LL) of events. For synthetic datasets, the ground truth intensities are available, so we report the test log-likelihood

(LL) and the time-average Hellinger distance between the learned conditional spatial distribution f ∗ ( s | t ) and the ground truth distribution. The distributions are estimated as multinomial distributions P = { p i , ..., p k } and Q = { q i , ..., q k } at k discretized grid points. The Hellinger distance is then calculated as H ( P, Q ) = 1 √ 2 √ ∑ k i =1 ( √ p i - √ q i ) 2

Baselines. We compare with two state-of-the-art neural STPP models, NSTPP [Chen et al., 2020] and Deep-STPP [Zhou et al., 2022]. We also design another baseline, Monte Carlo STPP, which uses the same underlying model as AutoSTPP but applies numerical integration instead of AutoInt to calculate the loss. For a fair comparison, Monte Carlo STPP models the influence f ∗ θ ( s, t ) as a multi-layer perceptron instead of a derivative network. This numerical baseline aims to demonstrate the benefit of automatic integration.

4.2 Results and Discussion

Exact Likelihood. Not all baseline methods have exact likelihood available. NSTPP [Chen et al., 2020] uses a numerical Neural-ODE solver. DeepSTPP [Zhou et al., 2022] introduces a VAE that optimizes a lower bound of likelihood. Monte Carlo STPP estimates the triple integral of intensity by Monte Carlo integration. As such, the results presented in Table 1 are estimated for these baseline models, whereas for our model, the results reflect the true likelihood.

Advantage of AutoInt. Wevisualize and compare two sample sets of intensities from STH Dataset 1 and STSC Dataset 3 in Figure 3. While the influence function approximator in Monte Carlo STPP can theoretically approximate more functions than Auto-STPP, the intensity it learns is 'flatter' than the intensity learned by Auto-STPP. This flatness indicates a lack of information regarding future event locations.

Comparing baseline and efficient impl of AutoInt,mixed derivative,forward

baseline

ours

0.8

relative time

0.6

0.4

0.2

0

1

23

¥45

6

orderofderivative

The primary reason behind this flatness can be traced back to numerical errors inherent in the Monte Carlo method. The Monte Carlo integration performs poorly for sharply localized integrants because its samples are homogeneous over the phase space. The intensity of ST-Hawkes is a localized function; it is close to zero in most of the places but high near some event locations. As a result, Monte Carlo STPP can hardly recover the sharpness of the ground truth intensity. NSTPP also uses Monte Carlo integration and suffers the same drawback on STH Dataset 1. In contrast, AutoSTPP evaluates the integral with closed-form integration and alleviates this issue.

Synthetic Datasets Results. Table 1 compares the test LL and the Hellinger distance between AutoSTPP and the baseline models on the six synthetic datasets. For the STSC datasets, we can see that AutoSTPP accurately recovers the intensity functions compared to other models. In Figure 3, AutoSTPP is the only model whose peak intensity location is always the same as the ground truth. DeepSTPP does not perform well in learning the dynamics of the STSC dataset; it struggles to align the peak and tends to learn the flat intensity as overly sharp. The peak intensity location of NSTPP is also biased.

For the STHP datasets, DeepSTPP has a clear advantage because it uses Gaussian kernels to approximate the Gaussian ground truth. In Table 1, AutoSTPP outperforms all other models except DeepSTPP, and its performance is comparable. Figure 3 shows that Monte Carlo STPP and NSTPP

can only learn an unimodal function, whereas AutoSTPP can capture multi-modal behavior in the ground truth, especially the small bumps near the mode.

Real-world Datasets Results. Table 2 compares the test LL of AutoSTPP against the baseline models on the earthquakes and COVID datasets. Our model demonstrates superior performance, outperforming all the state-of-the-art methods. One should note that while Monte Carlo STPP shows performance comparable to ours on the Earthquake JP dataset, it falls short when applied to the COVID-19 NY dataset. We attribute this discrepancy to the large low-population-density areas in the COVID-19 NY data, which causes higher numerical error in integration.

4.3 Computational Efficiency

Figure 4 visualizes the benefit of using our implementation of AutoInt instead of the PyTorch naive implementation. More visualizations of the forward and backward computation times can be found in Appendix A. We can see that our implementation can be extended to compute any order of partial derivative. It is significantly faster than the naive autograd. In our AutoSTPP, we calculate the intensity using the product of three first-order derivatives. Our implementation would lead to a speedup of up to 68% for computing each first-order derivative.

5 Conclusion

We propose Automatic Integration for neural spatiotemporal point process models ( AutoSTPP ) using a dual network approach. AutoSTPP can efficiently compute the exact likelihood of any sophisticated intensity.

We validate the effectiveness of our method using synthetic data and real-world datasets and demonstrate that it significantly outperforms other point process models with numerical integration when the ground truth intensity function is localized.

However, like any approach, ours is not without its limitations. While AutoSTPP excels in computing the likelihood, sampling from AutoSTPP is computationally expensive as we only have the expression of the probability density. Closed-form computation of expectations is also not possible; Knowing the form of ´ λ ( t ) , calculating ´ tλ ( t ) is still intractable.

Our work presents a new paradigm for learning continuous-time dynamics. Currently, our neural process model takes the form of Hawkes processes (self-exciting) but cannot handle the discrete decreases of intensity after events due to the difficulty of integration. Future work includes relaxing the form of the intensity network with advanced integration techniques. Another interesting direction is to increase the approximation ability of the product network.

References

Ryan Prescott Adams, Iain Murray, and David JC MacKay. Tractable nonparametric bayesian inference in poisson processes with gaussian process intensities. In Proceedings of the 26th annual international conference on machine learning , pages 9-16, 2009.

Anders Brix and Peter J Diggle. Spatiotemporal prediction for log-gaussian cox processes. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 63(4):823-841, 2001.

Anders Brix and Jesper Moller. Space-time multi type log gaussian cox processes with a view to modelling weeds. Scandinavian Journal of Statistics , 28(3):471-488, 2001.

Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes. arXiv preprint arXiv:2011.04583 , 2020.

Yuanda Chen. Thinning algorithms for simulating point processes. Florida State University, Tallahassee, FL , 2016.

Daryl J Daley and David Vere-Jones. An introduction to the theory of point processes: volume II: general theory and structure . Springer Science & Business Media, 2007.

Hennie Daniels and Marina Velikova. Monotone and partially monotone neural networks. IEEE Transactions on Neural Networks , 21(6):906-917, 2010.

Philip J Davis and Philip Rabinowitz. Methods of numerical integration . Courier Corporation, 2007.

Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1555-1564, 2016.

William Dunham. The calculus gallery . Princeton University Press, 2018.

Ruocheng Guo, Jundong Li, and Huan Liu. Initiator: Noise-contrastive estimation for marked temporal point process. In IJCAI , pages 2191-2197, 2018.

Hengguan Huang, Hao Wang, and Brian Mak. Recurrent poisson process unit for speech recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 6538-6545, 2019.

Haibin Li, Yangtian Li, and Shangjie Li. Dual neural network method for solving multiple definite integrals. Neural computation , 31(1):208-232, 2019.

Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Le Song. Learning temporal point processes via reinforcement learning. arXiv preprint arXiv:1811.05016 , 2018.

David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14556-14565, 2021.

Keqin Liu. Automatic integration. arXiv e-prints , pages arXiv-2006, 2020.

Jerrold E. Marsden and Anthony Tromba. Vector Calculus . Macmillan, August 2003. ISBN 978-0-7167-4992-9. Google-Books-ID: LiRLJf2m_dwC.

Hongyuan Mei and Jason Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. arXiv preprint arXiv:1612.09328 , 2016.

Yosihiko Ogata et al. The asymptotic behaviour of maximum likelihood estimators for stationary point processes. Annals of the Institute of Statistical Mathematics , 30(1):243-261, 1978.

Takahiro Omi, Naonori Ueda, and Kazuyuki Aihara. Fully neural network based model for general temporal point processes. arXiv preprint arXiv:1905.09690 , 2019.

Alex Reinhart. A review of self-exciting spatio-temporal point processes and their applications. Statistical Science , 33(3):299-318, 2018.

Robert H Risch. The problem of integration in finite terms. Transactions of the American Mathematical Society , 139:167-189, 1969.

Robert H Risch. The solution of the problem of integration in finite terms. Bulletin of the American Mathematical Society , 76(3):605-608, 1970.

Jin Shang and Mingxuan Sun. Geometric hawkes processes with graph convolutional recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 4878-4885, 2019.

Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. arXiv preprint arXiv:1909.12127 , 2019.

Utkarsh Upadhyay, Abir De, and Manuel Gomez-Rodriguez. Deep reinforcement learning of marked temporal point processes. arXiv preprint arXiv:1805.09360 , 2018.

Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Le Song, and Hongyuan Zha. Wasserstein learning of deep generative point process models. arXiv preprint arXiv:1705.08051 , 2017.

Junchi Yan, Xin Liu, Liangliang Shi, Changsheng Li, and Hongyuan Zha. Improving maximum likelihood estimation of temporal point process via discriminative and adversarial learning. In IJCAI , pages 2948-2954, 2018.

Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes process. In International Conference on Machine Learning , pages 11183-11193. PMLR, 2020.

Zihao Zhou and Rose Yu. Automatic integration for fast and interpretable neural point process. In Learning for Dynamics and Control Conference . PMLR, 2023.

Zihao Zhou, Xingyi Yang, Ryan Rossi, Handong Zhao, and Rose Yu. Neural point process for learning spatiotemporal event dynamics. In Learning for Dynamics and Control Conference , pages 777-789. PMLR, 2022.

A More Implementation Benchmarks

Comparing baseline and efficient impl

ofAutoInt,mixedderivative,forward

baseline

ours

0.8

relativetime

0.6

0.4

0.2

0

1

2345

6

/

orderofderivative

Comparingbaseline andefficientimpl

ofAutoInt，mixedderivative,forward

1.2

baseline

ours

time

0.8

relative

0.6

0.4

0.2

1

23

4

5

6

7

orderofderivative

As the number of MLP layers increases, the lower-order derivative computation becomes faster (relative to PyTorch naive implementation), whereas the higher-order derivative computation becomes slower. This performance pattern is because our implementation uses Python for loop. Our approach is faster than the baseline in the majority of cases.

Comparingbaselineandefficientimpl

ofAutoInt,mixed derivative,backward

baseline

ours

relativetime

0.8

0.6

0.4

0.2

0

1

23

4

5

6

/

orderofderivative

Comparingbaselineandefficientimpl

ofAutoInt,univariatederivative,backward

baseline

ours

0.8

relativetime

0.6

0.4

0.2

Q

1

2

4

5

6

/

orderofderivative

As for both forward and backward computation times, our implementation still consistently surpasses the baseline in terms of speed. Notably, our method is even more efficient when calculating univariate partial derivatives than mixed partial derivatives. This advantage is primarily due to the reduced number of iterations required by the Python for-loop in the case of univariate derivatives.

B STSC and ST-Hawkes Introduction and Simulation Parameters

We use the same parameters as Zhou et al. [2022].

The STSCP's and the STHP's kernels g 0 ( s ) and g 2 ( s , s j ) are prespecified to be Gaussian:

The STSCP is defined on S = [0 , 1] × [0 , 1] , while the STHP is defined on S = R 2 . The STSCP's kernel functions are normalized according to their cumulative probability on S . Table 4 shows the

simulation parameters. We discretized the STSCP's spatial domain as a 101 × 101 grid during the simulation.

Each dataset is a single, long sequence that spans over 10,000 time units. We divide each dataset into 50 sequences, each spanning 200 time units. We use 40 sequences for training, 5 for validation, and 5 for testing. Here's a summary of the total number of events found in each dataset:

The prediction task applies sliding windows to each of the datasets. We try to use historical events to predict the likelihood of the next event in the same sequence.

C Model Setup Details

We detail the specific hyperparameter settings in Table 5. Except for the learning rate, the same set of parameters was applied across all datasets. Despite varying datasets, This consistency in performance demonstrates our model's robustness to hyperparameters.

D Forward-pass Algorithm for Automatic Integration

Function: dnforward ( f, n, x, dims ) , partition ( n, k ) finds all k-subset partitions of n

return last pd

E Universal Approximation Theorem for Derivative Network

Consider an AutoInt integral network with the form

where σ denotes a R → R continuous non-polynomial function applied elementwise to each input dimension.

The derivative network thus takes the form

where A col ⊆ R k is a column of A that corresponds to the deriving dimension.

Recall the universal approximation theorem [Daniels and Velikova, 2010], which says for every compact K ⊆ R n and f ∈ C ( K, R ) , ε > 0 , there exist A,b, C such that

Proposition E.1. (Universal Approximation Theorem for Derivative Network) for every compact K ⊆ R n and f ∈ C ( K, R ) , ε > 0 , there exists A ∈ R k × n , b ∈ R k , C ∈ R k , β ∈ R such that

Proof. Given the mapping f , by UAT, there exists A,b, C that approximate f ( x ) . Construct ˜ C ∈ R k and β ∈ R , such that

̸

Then,

Note that ˜ C · ( σ ′ ◦ ( A · x + b ) ◦ A col ) -β = d dx col ( ˜ C · ( σ ◦ ( Ax + b )) -βx ) , which is the derivative net of a two-layer feedforward integral network.

F Relationship between Number of ProdNets and Model Expressivity

metric.name="mse"

COLORS

0.02

run.~.n_prodnet

10

の)

0.01

8

7

6

0.006

5

0.005

4

0.004

3

0.003

0.002

CHARTS

#

metric.name

1

"mse"

0.001

0.0006

Steps

0.0005

500

1000

1500

2000

2399

We applied ten summations of positive ProdNets to fit the non-multiplicative-decomposable function sin( x ) cos( y ) sin( z ) + 1 .

Each of these ProdNets consists of three MLP components, each with two hidden layers with 128 dimensions. All models underwent training with a consistent learning rate set at 0.005.

Our results, shown in Figure 9, indicate that increasing the number of ProdNets generally improves the model's performance in fitting the non-decomposable function. The model with the best MSE

uses 10 ProdNets, while the model with 2 ProdNets had the second-worst performance. This result is intuitively sensible, as more linear terms are typically required to express an arbitrary function precisely. However, we observed that employing more ProdNets does not always lead to better performance, as demonstrated by the model with 8 ProdNets.

Peer reviews:
comments: SUMMARY:
In this paper, the authors propose a new method for spatiotemporal neural point processes (STPP). A key innovation of their method is that they introduced AutoInt for efficient integration of the intensity function of STPP. Using the divergence theorem and Green’s theorem, they convert the original triple integral to line integrals which AutoInt can efficiently calculate. They also impose 3D non-negativity constraint on the integral network. Experimental results show efficacy of the proposed method

STRENGTHS:
1. Applying AutoInt to solve the STPP problem looks pretty novel.
2. Converting the original triple integral to line integrals using the divergence theorem and Green’s theorem and imposing non-negative constraint on the integral network also have strong methodology contribution.

WEAKNESSES:
1. The paper misses an important reference which also uses AutoInt for point processes: Automatic Integration for Fast and Interpretable Neural Point Processes. So the authors are encouraged to clarify contributions of this paper over the published paper above.

QUESTIONS:
1. It’s not clear why we cannot use AutoInt to do triple automatic integration on the intensity function directly. 
2. Compared with traditional methods on integral calculation, e.g. Monte Carlo, what are the advantages of the proposed method from the viewpoint of computational complexity?
3. To train AutoInt (or corresponding duo neural networks), are there any minimum requirement of the number of events (training samples)?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel framework to extend the automatic integration of conditional intensity function to spatio-temporal point processes, aiming to overcome the computational challenges of triple integration. Specifically, the proposed model adopts an integral network and converts the triple integral into line integrals using the Divergence theorem and Green’s theorem. The authors use a summation of ProdNet to handle the non-negativity of the conditional intensity function (derivative network) instead of directly imposing a non-negative constraint on the model parameters, which enhances the model expressiveness. The consistency, validity, and computational efficiency of the AutoInt STPP are comprehensively demonstrated.\n\nSTRENGTHS:\n1. The proposed framework based on auto-integration is valuable for spatio-temporal point process modeling. It significantly improves the model flexibility and accuracy compared to the close-form or monte-carlo based integration model.\n2. The natural constraint of the spatio-temporal point processes (e.g., non-negativity of the conditional intensity function) is well and not overly satisfied. The triple integral has also been handled in a novel way.\n3. The comprehensive experimental results on both synthetic and real-world data, including the ablation study, well support the claimed advantage of the proposed model.\n4. The consistency of the AutoInt PP estimator is proved under mild constraints.\n\nWEAKNESSES:\n1. Although it does not impact the overall goodness of the paper, I have to admit that the presentation of Sec. 3.5 is not straightforward. The purpose and advantage of using line integral to handle the triple integral are not clear to me. It would be better if the authors could further explain it with several high-level sentences.\n2. More experiments on the spatio-temporal Hawkes process with a non-separable influence function can be included. For example, the model performance on synthetic data generated by the ETAS [1], which is a widely-used STPP model in real-world practice.\n3. I would say the performance advantage of AutoSTPP is not significant or highlighted. For example, it seems like the DSTPP has a similar performance to AutoSTPP on synthetic data, and so does MonteSTPP if the standard deviation is considered on real data.\n\n\n---\n[1] Space-Time Point-Process Models for Earthquake Occurrences. Yosihiko Ogata. Annals of the Institute of Statistical Mathematics, 1998, vol. 50, issue 2, 379-402\n\nQUESTIONS:\n1. Forgive me if I understand wrong, but is there an integral of f over S missed in the integral at line 151?\n2. I would like to know how N (in ProdNet) is chosen in experiments, and if there is a significant performance difference with different N.\n3. It would be great if the authors could explain the advantages of AutoSTPP over DSTPP.\n4. The computational efficiency of the derivative is great, however, the readers or practitioners would like to know the comparison of the computational efficiency of the model with other baselines.\n\nLIMITATIONS:\n1. Modeling performance and computational efficiency can be further explained or demonstrated.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new approach to address the challenge of integrating spatiotemporal point processes in a flexible and efficient manner. The existing methods either lack flexibility or introduce numerical errors. The authors propose AutoSTPP as a novel paradigm that overcomes this limitation and showcases its advantages in recovering complex intensity functions from irregular spatiotemporal events. Its effectiveness is validated through experiments on synthetic data and benchmark real-world datasets.\n\nSTRENGTHS:\n1. Clear presentation. The writing is clear and concise.\n2. It provides the proof of consistency for AutoSTPP, which enhances the credibility of the proposed method.\n3. Comprehensive experiments. The authors evaluate the proposed approach on both synthetic datasets and real-world datasets.\n\nWEAKNESSES:\n1. An essential related work is missing: Automatic Integration for Fast and Interpretable Neural Point Processes [1]. \n2. Despite the solution for STPP is very interesting, the proposed technique lacks novelty. Zhou et al.[1] has already proposed a very similar framework for TPPs. The major different part lies in the data dimension. I don’t think just extending 1D to 3D is qualified for an accepted NIPS paper.  \n\n[1] Zhou, Zihao, and Rose Yu. \"Automatic Integration for Fast and Interpretable Neural Point Processes.\" Learning for Dynamics and Control Conference. PMLR, 2023.\n\n## After Rebuttal\n\nThe authors' response has addressed most of my concerns. Therefore, I decide to raise the score to 6 (Weak accept).\n\nHowever, I highly suggest authors carefully discuss links between their work and Zhou et al. [1].\n\nMoreover, recently developed diffusion (or score-based) models seem like a promising workaround for handling likelihood computation in the neural point process. Authors should discuss the following related works that have been published recently.\n- Li, Zichong, et al. \"SMURF-THP: Score Matching-based UnceRtainty quantiFication for Transformer Hawkes Process.\" in ICML 2023.\n- Yuan, Yuan, et al. \"Spatio-temporal Diffusion Point Processes.\" in KDD 2023.\n\nQUESTIONS:\nWhat’s the key contribution of this paper compared to [Zhou et al 2023]?  The authors should carefully differentiate their approach from it.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper is proposed an automatic integration scheme for efficient learning for spatiotemporal process models, where the corresponding function is approximated with a deep neural network. A priori, the method can be used to compute the likelihood of any intensity, even for sophisticated ones. The method is validated in both synthetic and real datasets, where the proposed method obtains better results than those provided by state-of-the-art approaches.\n\nSTRENGTHS:\nThe paper is well written and clear enough. In general, the authors provide most of the details, explanations and assumptions to understand their approach and the corresponding contribution. To be honest, I do not have many issues with the current submission. \n \nThe method shows a clear advance in terms of accuracy in comparison with imposing the constraint through activation function with a nonnegative triple derivative. AutoSTPP performance is very competitive with respect to previous neural STPPs. \n \nBoth quantitative and qualitative experiments are provided in the paper. Both of them show the superiority of the method, validating all the claims in the paper. In particular, I like the experiments on synthetic data, as complex intensity functions are considered and the proposed method can capture them correctly. In any case, the real-world scenarios are also needed, and the set of experiments in this line is well executed. Moreover, the method clearly outperforms the rest of competing methods in these experiments.\n\nWEAKNESSES:\nSome parameters in Eq.(1) are not properly defined. See, u or \\tau, for example. I know many parameters are included in the paper so please, be careful with all of them. \n\nThe authors do not include a proper discussion about the limitations of the method as it is a bit superficial. Moreover, some failure cases could help the reader. To this end, including more complex intensity functions where the method obtains a worse performance could greatly improve the paper. Thanks to that, we can see the real limitations of the method. \n\nI am not sure why the authors only use two real-world datasets of Chen et al. 2020, as more cases were considered there. At least, that could be briefly commented.\n\nQUESTIONS:\nAs the proposed method is available for learning continuous-time dynamics, I think the authors could consider more challenging scenarios where the applicability is especially relevant in science and technology, such as fluids or continuous mechanics. I would like to see at least a discussion about this type of scenario. \n \nAblation studies by considering the parameters in table 3. The authors should indicate how the best parameters were selected, and if any variation is given in the parameter tuning for both synthetic and real experiments.\n\nLIMITATIONS:\nThe authors do not include a proper discussion about the limitations of the method as it is a bit superficial. Moreover, some failure cases could help the reader. To this end, including more complex intensity functions where the method obtains a worse performance could greatly improve the paper. Thanks to that, we can see the real limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}