Neural Point Process for Forecasting Spatiotemporal Events

1 . Introduction . Accurate modeling of spatiotemporal event dynamics is fundamentally important for disaster response ( Veen and Schoenberg , 2008 ) , logistic optimization ( Safikhani et al. , 2018 ) and social media analysis ( Liang et al. , 2019 ) . Compared to other sequence data such as texts or time series , spatiotemporal events occur irregularly with uneven time and space intervals . Discrete-time deep dynamics models such as recurrent neural networks ( RNNs ) ( Hochreiter and Schmidhuber , 1997 ; Chung et al. , 2014 ) assume events to be evenly sampled . Interpolating an irregular sampled sequence into a regular sequence can introduce significant biases ( Rehfeld et al. , 2011 ) . Furthermore , event sequences contain strong spatiotemporal dependencies . The rate of an event depends on the preceding events , as well as the events geographically correlated to it . Spatiotemporal point processes ( STPP ) ( Daley and Vere-Jones , 2007 ; Reinhart et al. , 2018 ) provides the statistical framework for modeling continuous-time event dynamics . As shown in Figure 1 , given the history of events sequence , STPP estimates the intensity function that is evolv- © 2022 Z. Zhou , X. Yang , R. Rossi , H. Zhao & R. Yu . ing in space and time . However , traditional statistical methods for estimating STPPs often require strong modeling assumptions , feature engineering , and can be computationally expensive . Machine learning community is observing a growing interest in continuous-time deep dynamics models that can handle irregular time intervals . For example , Neural ODE ( Chen et al. , 2018 ) parametrizes the hidden states in an RNN with an ODE . Shukla and Marlin ( 2018 ) uses a separate network to interpolates between reference time points . Neural temporal point process ( TPP ) ( Mei and Eisner , 2017 ; Zhang et al. , 2020 ; Zuo et al. , 2020 ) is an exciting area that combines fundamental concepts from temporal point processes with deep learning to model continuous-time event sequences , see a recent review on neural TPP ( Shchur et al. , 2021 ) . However , most of the existing models only focus on temporal dynamics without considering spatial modeling . In the real world , while time is a unidirectional process ( arrow of time ) , space extends in multiple directions . This fundamental difference from TPP makes it nontrivial to design a unified STPP model . The naive approach to approximate the intensity function by a deep neural network would lead to intractable integral computation for likelihood . Prior research such as Du et al . ( 2016 ) discretizes the space as “ markers ” and use marked TPP to classify the events . This approach can not produce the space-time intensity function . Okawa et al . ( 2019 ) models the spatiotemporal density using a mixture of symmetric kernels , which ignores the unidirectional property of time . Chen et al . ( 2021 ) proposes to model temporal intensity and spatial density separately with neural ODE , which is computational expensive . We propose a simple yet efficient approach to learn STPP . Our model , Deep Spatiotemporal Point Process ( DeepSTPP ) marries the principles of spatiotemporal point processes with deep learning . We take a non-parametric approach and model the space-time intensity function as mixture of kernels . The parameters of the intensity function are governed by a latent stochastic process no sampling which captures the uncertainty of the event sequence . The latent process is then inferred via amortized variational inference . That is , we draw a sample from the variational distribution for every event . We use a Transformer network to parametrize the variational distribution conditioned on the previous events . Compared with existing approaches , our model is non-parametric , hence does not make assumptions on the parametric form of the distribution . Our approach learns the space-time intensity function jointly without requiring separate models for time-intensity function and spatial density as in Chen et al . ( 2021 ) . Our model is probabilistic by nature and can describe various uncertainties in the data . More importantly , our model enjoys closed form integration , making it feasible for processing large-scale event datasets . To summarize , our work makes the following key contributions : • Deep Spatiotemporal Point Process . We propose a novel Deep Point Process model for forecasting unevenly sampled spatiotemporal events . It integrates deep learning with spatiotemporal point processes to learn continuous space-time dynamics . • Neural Latent Process . We model the space-time intensity function using a nonparametric approach , governed by a latent stochastic process . We use amortized variational inference to perform inference on the latent process conditioned on the previous events . • Effectiveness . We demonstrate our model using many synthetic and real-world spatiotemporal event forecasting tasks , where it achieves superior performance in accuracy and efficiency . We also derive and implement efficient algorithms for simulating STPPs . 2 . Methodology . We first introduce the background of spatiotemporal point process , and then describe our approach to learn the underlying spatiotemporal event dynamics . 2.1 . Background on Spatiotemporal Point Process . Spatiotemporal Point Process . Spatiotemporal point process ( STPP ) models the number of events N ( S × ( a , b ) ) that occurred in the Cartesian product of the spatial domain S ⊆ R2 and the time interval ( a , b ] . It is characterized by a non-negative space-time intensity function given the history Ht : = { ( s1 , t1 ) , . . . , ( sn , tn ) } tn≤t : λ∗ ( s , t ) : = lim ∆s→0 , ∆t→0 E [ N ( B ( s , ∆s ) × ( t , t+∆t ) ) |Ht ] B ( s , ∆s ) ∆t ( 1 ) which is the probability of finding an event in an infinitesimal time interval ( t , t + ∆t ] and an infinitesimal spatial ball S = B ( s , ∆s ) centered at location s. Example 1 : Spatiotemporal Hawkes process ( STH ) . Spatiotemporal Hawkes ( or self-exciting ) process assumes every past event has an additive , positive , decaying , and spatially local influence over future events . Such a pattern resembles neuronal firing and earthquakes . It is characterized by the following intensity function ( Reinhart et al. , 2018 ) : λ∗ ( s , t ) : = µg0 ( s ) + ∑ i : ti < t g1 ( t , ti ) g2 ( s , si ) : µ > 0 ( 2 ) where g0 ( s ) is the probability density of a distribution over S , g1 is the triggering kernel and is often implemented as the exponential decay function , g1 ( ∆t ) : = α exp ( −β∆t ) : α , β > 0 , and g2 ( s , si ) is the density of an unimodal distribution over S centered at si . Example 2 : Spatiotemporal Self-Correcting process ( STSC ) . Self-correcting spatiotemporal point process Isham and Westcott ( 1979 ) assumes that the background intensity increases with a varying speed at different locations , and the arrival of each event reduces the intensity nearby . STSC can model certain regular event sequences , such as an alternating home-to-work travel sequence . It has the following intensity function : λ∗ ( s , t ) = µ exp ( g0 ( s ) βt− ∑ i : ti < t αg2 ( s , si ) ) : α , β , µ > 0 ( 3 ) Here g0 ( s ) is the density of a distribution over S , and g2 ( s , si ) is the density of an unimodal distribution over S centered at location si . Maximum likelihood Estimation . Given a history of n events Ht , the joint log-likelihood function of the observed events for STPP is as follows : log p ( Ht ) = n∑ i=1 log λ∗ ( si , ti ) − ∫ S ∫ t 0 λ∗ ( u , τ ) dudτ ( 4 ) Here , the space-time intensity function λ∗ ( s , t ) plays a central role . Maximum likelihood estimation seeks the optimal λ∗ ( s , t ) from data that optimizes Eqn . ( 4 ) . Predictive distribution . Denote the probability density function ( PDF ) for STPP as f ( s , t|Ht ) which represents the conditional probability that next event will occur at location s and time t , given the history . The PDF is closely related to the intensity function : f ( s , t|Ht ) = λ∗ ( s , t ) 1− F ∗ ( s , t|Ht ) = λ∗ ( s , t ) exp ( − ∫ S ∫ t tn λ∗ ( u , τ ) dτdu ) ( 5 ) where F is the cumulative distribution function ( CDF ) , see derivations in Appendix A.1 . This means the intensity function specifies the expected number of events in a region conditional on the past . The predicted time of the next event is the expected value of the predictive distribution for time f⋆ ( t ) in the entire spatial domain : E [ tn+1|Ht ] = ∫ ∞ tn t ∫ S f∗ ( s , t ) dsdt = ∫ ∞ tn t exp ( − ∫ t tn λ∗ ( τ ) dτ ) λ∗ ( t ) dsdt Similarly , the predicted location of the next event evaluates to : E [ sn+1|Ht ] = ∫ S s ∫ ∞ tn f∗ ( s , t ) dtds = ∫ ∞ tn exp ( − ∫ t tn λ∗ ( τ ) dτ ) ∫ S sλ∗ ( s , t ) dsdt Unfortunately , Eqn . ( 4 ) is generally intractable . It requires either strong modeling assumptions or expensive Monte Carlo sampling . We propose the Deep STPP model to simplify the learning . 2.2 . Deep Spatiotemporal Point Process ( DSTPP ) . We propose DeepSTPP , a simple and efficient approach for learning the space-time event dynamics . Our model ( 1 ) introduces a latent process to capture the uncertainty ( 2 ) parametrizes the latent process with deep neural networks to increase model expressivity and ( 3 ) approximates the intensity function with a set of spatial and temporal kernel functions . Neural latent process . Given a sequence of n event , we wish to model the conditional density of observing the next event given the history f ( s , t|Ht ) . We introduce a latent process to capture the uncertainty of the event history and infer the latent process with armotized variational inference . The latent process dictates the parameters in the space-time intensity function . We sample from the latent process using the re-parameterization trick Kingma and Welling ( 2013 ) . As shown in Figure 2 , given the event sequence Ht = { ( s1 , t1 ) , . . . , ( sn , tn ) } tn≤t , we encode the entire sequence into the high-dimensional embedding . We use positional encoding to encode the sequence order . To capture the stochasticity in the temporal dynamics , we introduce a latent process z = ( z1 , · · · , zn ) for the entire sequence . We assume the latent process follows a multivariate Gaussian at each time step : zi ∼ qϕ ( zi|Ht ) = N ( µ , Diag ( σ ) ) ( 6 ) where the mean µ and covariance Diag ( σ ) are the outputs of the embedding neural network . In our implementation , we found using a Transformer Vaswani et al . ( 2017 ) with sinusoidal positional encoding to be beneficial . The positions to be encoded are the normalized event time instead of the index number , to account for the unequal time interval . Recently , Zuo et al . ( 2020 ) also demonstrated that Transformer enjoys better performance for learning the intensity in temporal point processes . Non-parametric model . We take a non-parameteric approach to model the space-time intensity function λ∗ ( s , t ) as : λ∗ ( s , t|z ) = n+J∑ i=1 wiks ( s , si ; γi ) kt ( t , ti ; βi ) ( 7 ) Here wi ( z ) , γi ( z ) , βi ( z ) are the parameters for each event that is conditioned on the latent process . Specifically , wi represents the non-negative intensity magnitude , implemented with a soft-plus activation function . ks ( · , · ) and kt ( · , · ) are the spatial and temporal kernel functions , respectively . For both kernel functions , we parametrize them as a normalized RBF kernel : ks ( s , si ) = α −1 exp ( − γi∥s− si∥ ) , kt ( t , ti ) = exp ( − βi∥t− ti∥ ) ( 8 ) where the bandwidth parameter γi controls an event ’ s influence over the spatial domain . The parameter βi is the decay rate that represents the event ’ s influence over time . α = ∫ S exp ( −γi∥s−si∥ ) ds is the normalization constant . We use a decoder network to generate the parameters { wi , γi , βi } given z separately , shown in Figure 2 . Each decoder is a 4-layer feed-forward network . We use a softplus activation function to ensure wi and γi are positive . The decay rate βi can be any number , such that an event could have constant or increasing triggering intensity over time . In addition to n historical events , we also randomly sample J representative points from the spatial domain to approximate the background intensity . This is to account for the influence from unobserved events in the background , with varying rates at different absolution locations . The inclusion of these representative points can approximate this background distribution . The model design in ( 7 ) enjoys a closed form integration , which gives the conditional PDF as : f ( s , t|Ht , z ) = λ∗ ( s , t|z ) exp ( − n+J∑ i=1 wi βi [ kt ( tn , ti ) − kt ( t , ti ) ] ) ( 9 ) See the derivation details in Appendix A.2 . DeepSTPP circumvents the integration of the intensity function and enjoys fast inference in forecasting future events . In contrast , NSTPP Chen et al . ( 2021 ) is relatively inefficient as its ODE solver also requires additional numerical integration . Parameter learning . Due to the latent process , the posterior becomes intractable . Instead , we use amortized inference by optimizing the evidence lower bound ( ELBO ) of the likelihood . In particular , given event historyHt , the conditional log-likelihood of the next event is : log p ( s , t|Ht ) ≥ log pθ ( s , t|Ht , z ) + KL ( qϕ ( z|Ht ) ||p ( z ) ) ( 10 ) = log λ∗ ( s , t|z ) − ∫ t tn λ∗ ( τ ) dτ + KL ( q||p ) ( 11 ) where ϕ represents the parameters of the encoder network and θ are the parameters of the decoder network . p ( z ) is the prior distribution , which we assume to be Gaussian . KL ( ·||· ) is the Kullback–Leibler divergence between two distributions . We can optimize the objective function in Eqn . ( 11 ) w.r.t . the parameters ϕ and θ using back-propagation .

Summary: This work studies the DNN-based spatiotemporal point process model. It points out the drawback of most existing DNN-based point process models: incapability to incorporate the spatio information. Although in statistics, the spatiotemporal point process is capable of capturing events in continuous space and time, such methods are computation expensive. The theoretical analysis is provided, and experimental comparisons are conducted on synthetic and real data.