Neural Spatio-Temporal Point Processes
Authors: Ricky T. Q. Chen
Brandon Amos
Maximilian Nickel
Venue: ICLR 2021 Poster (2021)
Keywords: point processes
normalizing flows
differential equations

Abstract: We propose a new class of parameterizations for spatio-temporal point processes which leverage Neural ODEs as a computational method and enable flexible, high-fidelity models of discrete events that are localized in continuous time and space. Central to our approach is a combination of continuous-time neural networks with two novel neural architectures, \ie, Jump and Attentive Continuous-time Normalizing Flows. This approach allows us to learn complex distributions for both the spatial and temporal domain and to condition non-trivially on the observed event history. We validate our models on data sets from a wide variety of contexts such as seismology, epidemiology, urban mobility, and neuroscience.

Published as a conference paper at ICLR 2021
NEURAL SPATIO-TEMPORAL POINT PROCESSES
Ricky T. Q. Chen
Brandon Amos, Maximilian Nickel Facebook Al Research {bda, maxn 8 fb. com
University of Toronto; Vector Institute rtgichenêcs. toronto.cdu
ABSTRACT
We propose new class of parameterizations for spatio-temporal point processes which leverage Neural ODEs as a computational method and enable flexible, high- fidelity models of discrete events that are localized in continuous time and space. Central to our approach a combination of continuous-time neural networks with two novel neural architectures, L.e., Jump and Attentive Continuous- time Normalizing Flows. This approach allows us to learn complex distributions for both the spatial and temporal domain and to condition non-trivially on the observed event history. We validate our models on data sets from wide variety of contexts such as seismology, epidemiology, urban mobility, and neuroscience.
INTRODUCTION
Modeling discrete events that are localized in continuous time and space an important task in many scientific fields and applications. Spatio-temporal point processes (STPPs) are versatile and principled framework for modeling such event data and have, consequently, found many applications in diverse range of fields. This includes, for instance, modeling earthquakes and aftershocks (Ogata, 1988; 1998), the occurrence and propagation of wildfires (Hering et al., 2009), epidemics and infectious discases (Meyer et al., 2012; Schoenberg al., 2019), urban mobility (Du et al., 2016), the spread of invasive species (Balderama et al., 2012), and brain activity (Tagliazucchi et al., 2012).
It is of great interest in all of these areas to learn high-fidelity models which can jointly capture spatial and temporal de- pendencies and their propagation effects. However, existing parameterizations of STPPs are strongly restricted in this re- gard due computational considerations: In its general form, STPPs require solving multivariate integrals for computing likelihood values and thus have primarily been studied within the context of different approximations and model restric- tions. This includes, for instance, restricting the model class to parameterizations with known closed-form solutions (c.g., exponential Hawkes processes (Ozaki, 1979)), to restrict de- pendencies between the spatial and temporal domain (e.g., independent and unpredictable marks (Daley & Vere-Jones, 2003)), discretize continuous time and space (Ogata, 1998). These restrictions and approximations to mis-specified models and loss of cntemmatiio-mmtivatete the development of neural temporal point processes such as Neural Hawkes Processes (Mei & Eisner, 2017) and Neural Jump SDEs (Jia & Benson, 2019). While these methods are more flexible, they can still require approximations such as Monte-Carlo sampling of the likelihood (Mei & Eisner, 2017; Nickel Le, 2020) and, most importantly, only model re- stricted spatial distributions (Jia & Benson, 2019).
5
0
can lead
Color is used to denote p(z]t), which can be evaluated for Neu- ral STPPs. After observing an event in one mode, the model instantancously updated as it strongly expects an event in the next mode. After period of no ob- servations, the model smoothly reverts back to the marginal distribution.
Figure 1:
To overcome these issues, we propose a new class of parameterizations for spatio-temporal point processes which leverage Neural ODEs as computational method and allows us to define flexible,
Work done while Facehook AI Research.

---

Published as a conference paper at ICLR 2021
high-fidelity models for spatio-temporal event data. We build upon ideas of Neural Jump SDEs (Jia & Benson, 2019) and Continuous-time Normalizing Flows (CNFs; Chen et al. 2018; Grathwohl et al. 2019; Mathieu & Nickel 2020) to learn parametric models of spatial (or mark') distributions that are defined continuously in time. Normalizing flows are known to be flexible universal density estimators (e.g. Huang al. 2018 8: 2020; Teshima et al. 2020; Kong & Chaudhuri 2020) while retaining computational tractability. As such, our approach allows the computation of exact likelihood values even for highly complex spatio-temporal distributions, and our models create smoothly changing spatial distributions that naturally benefits spatio-temporal modeling. Central to our approach, are two novel neural architectures based on CNFs using either discontinuous jumps in distribution or self-attention- condition spatial distributions on the event history. To the best of our knowledge this is the first method that combines the flexibility neural TPPs with the ability to learn high-fideling models of continuous marks that can have complex dependencies on the event history. In addition to our modeling contributions, we also construct five new pre-proces processee data sets for benchmarking spatio-temporal event models.
2 BACKGROUND
In the following, we give brief overview of two core frameworks which our method builds upon, i... spatio-temporal point processes and continuous-time normalizing flows.
Event Modeling with Point Processes Spatio-temporal point processes concerned with mod- eling sequences of random events in continuous space and time (Moller & Waagepetersen, 2003; Baddeley et al., 2007). Let H {(t,, a=) denote the sequence of event times t; and their associated locations Ø, Rd the number of events n being also random. Additionally, let He {(4i, 2;) 4 < 1, 4 E H} denote the history of events predating time I. A spatio-temporal point process is then fully characterized by its conditional intensity function
P Pt. E [t, f, At). x4 Bax, Ax) H₂)
A(t, x Hi)
lim A40,A
(I)
[B Arr Av
where (xx, Ar) denotes a ball centered at æ E 12d and with radius Ax. The only condition that A(t, x H.) 2 0 and need not be normalized. Given i - previous events, the conditional intensity function describes therefore the instantaneous probability of the i-th event occurring at and location x. In the following. we will use the common star superscript shorthand X Af, = A(t,, x (H)) to denote conditional dependence on the history. The joint log- likelihoodd of observing H. within a time interval of [0, T] is then given by (Daley & Vere-Jones, 2003, Proposition 7.3.III)
log p (H) = Σ log X (tt, - 1/¹ J X² x)) dædr.
(2)
Training general STPPs with maximum likelihood difficult as eq. (2) requires solving a multivariate integral. This need to compute integrals has driven research to focus around the use of kernel density estimators (KDE) with exponential kernels that have known anti -erivveris (Reinhart et al., 2018).
Continuous-time Normalizing Flows Normalizing flows (Dinh al., 2014; 2016; Rezende & Mohamed, 2015) is a class of density models that describe flexible distributions by parameterizing an invertible transformation from simpler base distribution, which enables exact computation of the probability of the transformed distribution, without any unknown normalization constants.
Given random variable Zy with known distribution p(xg) and invertible transformation F(z). the transformed variable F(xo) is a random variable with probability distribution function that satisfies
log ((x))) logp(m)) log det 0F Ör (mr)
There have been many advances in parameterizing F with flexible neural networks that also allow for cheap evaluations of eq. (3). We focus our attention on Continuous- time Normalizing Flows (CNFs), which parameterizes this transformation with a Neural ODE (Chen al., 2018). CNFs
We regard any marked temporal point process with continuous marks as spatio-temporal point process.
2

---

Published as a conference paper at ICLR 2021
define infinite set of distributions the real line that vary smoothly across time, and will be our core component for modeling events the spatial domain.
Let p(zo) be the base distribution". We then parameterize instantaneous change in the form of an ordinary differential equation (ODE), dæ: ft, æz), where the subscript denotes dependence t. This function can be parameterized using any Lipschitz-continun sample Zo from the base distribution, let æt be the solution of the initial value problem at time t, Lee it is from trajectory that passes through æn at time O and satisfies the ODE da./dt = f. We can express the value of the solution at time /
neural network. Conditioned on a
23 Zo + S x(1,x,))T.
(4)
The distribution of æ, then also continuously changes t. through the following equation,
log ((g]())))) = loo p(0)) - / ( 0f 0x
(T. (T. dr.
(5)
In practice, eq. (4) and eq. (5) are solved together from to i, as eq. (5) alone is not an ordinary differential equation but the combination of æ; and log p(x;) is. The trace of the Jacobian of (T, ax)) can be estimated using a Monte Carlo estimate of the identity (Skilling, 1989; Hutchinson, 1990), tr(A) Bo,,N((,1)) Av). This estimator relies only vector-Jacobian product, which can be efficiently computed modern automatic differentiation and deep learning frameworks. This has been used (Grathwohl et al., 2019) scale CNFs to higher dimensions using Monte Carlo estimate of the log likelihood objective,
log p(x))t) = log p(zo) Be [ [' Or (T, x. )u dr
which, even if only one sample of is used, is still amenable training with stochastic gradient descent. Gradients with respect to any parameters in f can be computed with constant memory by solving an adjoint ODE in reverse-time as described in Chen al. (2018).
NEURAL TEMPORA POINT PROCESSES
We are interested in modeling high- fichffidliity distributions in continuous time and space that can be event information. For this purpose,
updated based parameterize a STPP by combining ideas from Neural Jump SDEs and Continuous Normalizing Flows create highly flexible models that still allow exact likelihood computation.
the Neural ODE framework to
We first (re- )introdut necessary notation. Let {44 (() denote sequence of event times E [0, 7) and locations a, and the use of subscripting with t; will be useful later in the continuous modeling framework. Following Daley & Vere-Jones (2003), we decompose the conditional intensity function as
(4) Rdd The superscript indicates an association with the i-th event,
xt, (²) p" (x
where (t) is the ground intensity of the temporal process and where p"(x () is the conditional density a mark æ given H4. The star superscript used as again shorthand denote dependence eq. (7) allows now to simplify the log- ligeliilddood function
on the history. Since Squ p" (x of the joint process from eq. (2), such that
logr(H() = Σ log (²) 1000'''''0 x() dr
n
log (a
temporal log. likelihood
spatial likelibodo
"This he any distribution that easy sample from and evaluate log-likelihond for. We the typical choice of standard Normal in our experiments. "This initial value problem has unique solution if is the Picard-I Lnndelöf theorem.
i Lipschitz-conttiuuuus along the trajectory to ze by
3

---

Published as a conference paper at ICLR 2021
Furthermore, based eq. (7), can derive separate models for the ground intensity and conditional mark density which will be jointly conditioned on a continuous-timg following, we will first describe how construct a latent dynamics model, which
hidden state with jumps. In the
compute the ground intensity X (1). We will then propose three novel CNF-based approaches for modeling the conditional mark density p" (x:(t. We will first describe an unconditional model, which is already a strong baseline when spatial event distributions only follow temporal patterns and there is little to no correlation between the spatial observations. We then devise two new methods of conditioning on the event history H: one explicitly modeling instantaneous changes in distribution, and another that uses an attention mechanism which is more amenable parallelism.
Latent Dynamics and Ground Intensity For the temporal variables {tt}. parameterize the inten- sity function using hidden state dynamics with jumps, similar to the work of Jia & Benson (2019). Specifically, we evolve continuous-tims hidden state h and set
(t) = (x,) (Ground intensity)
(9)
where gA is neural network with a softplus nonlinearity applied to the output, to ensure the intensity positive. We then capture conditional dependencies through the use of a continuously changing state f, with instantaneous updates when conditioned on an event.
The architecture is analogous recurrent neural network with continuous-time hidden state (Mei & Eisner, 2017; Che et al., 2018; Rubanova et al.. 2019) modeled by Neural ODE. This provides us with a vector representation h; at every time value that acts as both summary of the history of events and as a predictor of future behavior. Instantaneous updates to h, allow to incorporate abrupt changes to the hidden state that are triggered by observed events. This mechanism is important for modeling point processes and allows past events to influence future dynamics in discontinuous way (e.g., modeling immediate shocks to a system).
We use S to model the continuous change in the form of an ODE and ga. to model instantaneous changes based on an observed event.
his ho dh, fi, tt)
(An initial hidden state)
(10)
between event times event times 4 The use of e to portray that h; is a cåglåd function, i.e. left-continuous with right limits, with discontinuous jump modeled by gh.
(Continuous evolution)
(11)
di
lim heste = gh (Li he, 20
(Instantaneous updates)
(12)
The parameterization of continuous- time hidden states in the form of eqs. (10) to (12) has been used for time series modeling (Rubanova et al., 2019; De Brouwer et al., 2019) as well as TPPs (Jia & Benson, 2019). We parameterize fn, as standard multi-layer fully connected neural network, and use the GRU update (Cho et al., 2014) to parameterize gh. was done in Rubanova et al. (2019).
Time-varying CNF The first model we consider is a straightforward application of the CNF to time-variable observations. Assuming that the spatial distribution is independent of prior events,
log (x (1)))) = log p( ( (f. log p(=))) (*) 1." tr ( (r. ("
(r, dr
(13)
where xt is the solution of the ODE f with initial value x (6) the observed event location, at t, the observed event time. The spatial distribution of an event modeled by a Time-varying CNF changes with respect to the time it occurs. Some spatio-temporal data sets exhibit mostly temporal patterns and little to no dependence
previous events in the spatial domain, which would make time-varying CNF a good fit. Nevertheless, this model lacks the ability to capture spatial propagation effects, as does not condition previous event observations.
major benefit of this model is the ability to evaluate the joint log-likelihood fully in parallel across events, since there of only allow a scalar terminal time. Thus, to solve all n integrals in eq. (13) with a single call to an ODE solver, we can simply reparameterize all integrals with consistent dummy variable and track
dependencies between events. Most modern ODE solvers that
aware
4

---

Published as a conference paper at ICLR 2021
the terminal time in the state (see Appendix F for detailed explanation). Intuitively, the idea is that we can reparameterize ODEs that are on t E ty] into an ODE on 8 E [0, 1] using the change of variables f/e, (or st,) and scaling the output of by t.. The joint ODE is then
ast1,
(1)
(1)
d
L' ((s,A,)ds
which gives
(14)
ds
tnf (strr (())
2 (n))
as'
f(x,A.)
Ao
Thus the full trajectories between O to to for all events can be computed in parallel using this augmented ODE by simply integrating once from s 0
Jump CNF For the second model, we condition the dynamics defining the continuous normalizing flow on the hidden state h. allowing the normalizing flow to update its distribution based on changes For this purpose, we define continuous-time spatial distributions by making again use of two components: (i) a continuous-timg normalizing flow that evolves the distribution continuously, and (ii) a standard (discrete-time) flow model that changes the distribution instantaneously after conditioning on new events. As normalizing flows parameterize distributions through transformations of the samples, these continuous- and discrete-time transformations are composable in a straightforward manner and are end-to-en differentiable.
The generative process of a single event in Jump CNF is given by:
Zo ~ p(x0)
(An initial distribution)
(15)
dx; = (1, Rtt he)
between event times
(Continuous evolution)
(16)
di
lim tt = Ir (1i, Fti h4,) c-3
at event times
(Instantaneous updates)
(17)
The initial distribution can be parameterized by normalizing flow. In practice, we set a base distribution at negative time value and model p(z.) using the same CNF parameterized by fz. The instantaneous updates (or jumps) describe conditional updates in distribution after each new event has been observed. This conditioning on h2 required for the continuous and instantaneous updates to depend on the history of observations. Otherwise, Jump CNF would only be able to model the marginal distribution and behave similarly to a time-varying CNF. solve for h, alongside æt.
The final probability of an event e; some / > in after observing events is given by the sum of changes according to continuous- and discrete-time normalizing flows.
logp" (xx(t)= = log p(z))
(f((T, It, hT) Ox
dr log det Oga(tir Ft, he,)
Σ
+
Ox
E.
(18)
Change density up last
(T dr
Ox
Change in density from last event to
As the instantaneous updates must be applied sequentially Jump CNF, we can only compute the integrals in eq. (18) one at time. As such, the number of initial value problems scales linearly with the number of events in the history because the ODE solver must be restarted between each instantaneous update account for the discontinuous change to state. This incurs a substantial cost when the number of events is large.
To design spatial model with conditional dependencies that alleviates the compu- tational issues of Jump CNFs and can be computed in parallel, make of efficient attention mechanisms based on the Transformer architecture (Vaswani et al., 2017). Denoting only the spatial variables for simplicity, each conditional distribution log p(æ,, (z,, He,) can be modeled by CNF that
Attentive CNF
5

---

Published as a conference paper at ICLR 2021
X/
Seguevce
x/21
Seguence Serpince
x x/2
x x
Jump CNF
Attentive CNF
Figure 2: Visualization of the sampling paths of Neural STPP models for 1-D spatio-temporal data set where [44]2= are event times. The Jump CNF uses instantaneous jumps to update its distribution based on newly observed events while the Attentive CNF depends continuously on the sampling paths of prior events. We additionally visualize a second sequence for the Attentive CNF where the random base samples [== are the same in sequence 1. Bven so, the sampling paths different because the first event is different, effectively leading to different conditional spatial distributions. See Figure for visualizations of the learned density.
depends on the sample path of prior events. Specifically, we take the dummy-variable reparameteriza- tion of eq. (14) and modify it so that the i-th event depends on all previous events using Transformer architecture for f.
t1f stl, (2' (1()) pi (im tef st2, fac ( ) (hr, 12=1
(1)
a
(2)
d
x:a
famm-
(19)
ds
(n) tnf stn: fas (8) Ma (he, 1=
With this formulation, the trajectory of (9) depends continuously on the trajectory of ay for all j < i and the hidden states h prior to the i-th event. Similar to eq. (14), an Attention CNF can now solve for the trajectories of all events in parallel while simultaneously depending non-trivially on H.
(3)
To parameterize famm, we use an embedding layer followed by two multihead attention (MHA) blocks and an output layer to map back into the input space. We use the Lipschirz-continue multihead atten- tion from Kim et al. (2020) as they recently showed that the dot product multihead attention (Vaswani et al., 2017) is not Lipschitz-contiin continuous and thus may be ill-suited for parameterizing ODEs.
Low-variance Log-tikelihood Estimation The vari- ance of the Hutchinson stochastic trace estimator in eq. (6) grows with the squared Frobenius norm of the Jacobian, Eig [35/a=]3 (Hutchinson, 1990). For at- tentive CNFs, can remove some of the non-diagonal elements of the Jacobian and achieve a lower variance estimator. The attention mechanism creates block- triangular Jacobian, where each block corresponds to one event, but the elements outside of the block- diagonal are solely due to the multihead attention. By detaching the gradient connections between different events in the MHA blocks, we can create surrogate Jacobian matrix that do not contain cross-event partial derivatives. This effectively allows us to apply Hutchinson's 's estimator thus has the same expected variance trace estimator. The procedure consists of selectively removing partial derivatives and is straightforward but notationally cumbersome; the interested reader can find the details in Appendix
Naivo Hutchinson
1.68
Low-van Hutchinson
Villcction
1.66
1.64
1.62
1,60
2000
4000
6000
8000
10000
Trainnn iboration
Figure 3: Lower variance estimates of log- likelihood allows training better Attentive CNFs.
matrix that has the same diagonal elements as the Jacobian af/ax-an and has zeros outside of the block-diagonal, leading to lower
This is similar in spirit Chen & Duvenaud (2019) but instead of constructing a neural network that specifically allows cheap removal of partial derivatives, we make use of the fact that multihead attention already allows cheap removal of (cross-event) partial derivatives.
An ablation experiment is shown in Figure 3 for training on the PINWHEEL data set, where the lower variance estimates (and gradients) ultimate led to faster convergence and better converged models.

---

Published as a conference paper at ICLR 2021
4 RELATED WORK
Neural Temporal Point Processes Modeling real-world data using restricted models such as Exponential Hawkes Processes (Ozaki, 1979) may lead to poor results due to model mis-specification. While this has led to many works on improving the Hawkes process (e.g. Linderman & Adams 2014; Li & Zha 2014; Zhao al. 2015; Farajtabar et al. 2017; & Ke 2020; Nickel & Le 2020), recent works have begun to explore neural network parameterizations of TPPs. A common approach is to use recurrent neural networks accumulate the event history in a latent state from which the intensity value can then be derived. Models of this form include, for instance, Recurrent Marked Temporal Point Processes (RMTPPs; Du al. 2016) and Neural Hawkes Processes (NHPs; Mei & Eisner 30. In contrast to our approach, these methods can not compute the exact likelihood of the model and have to resort to Monte-Carlo sampling for its approximation. However, this approach is especially problematic for commonly occurring clustered and bursty event sequences as it either requires a very high sampling rate or ignores important temporal dependencies (Nickel & Le, 2020). To overcome this issue, Jia & Benson (2019) proposed Neural Jump SDEs which extend the Neural ODE framework and allow to compute the exact likelihood for neural TPPs, up to numerical errors. This method is closely related to approach and we build on its ideas to compute the ground intensity of the STPP. However, current Neural Jump SDEs aas well as NHPs and RMTPPs- not well-suited for modeling complex continuous mark distributions as they are restricted to methods such as Gaussian mixture models in the spatial domain. Finally, Shchur et al. (2019) and Mchrasa al. (2019) considered combining TPPs with flexible likelihood-basel models, however for different purposes as in our case, i.e., for intensity-free learning of only temporal point processes.
Continuous Normalizing Flows The ability to describe an infinite number of distributions with Continuous Normalizing Flow has been used by few recent works. Some works in computer graphics have used the interpolation effect of CNFs to model transformations of point clouds (Yang et al., 2019; Rempe et al., 2020; Li et al., 2020). CNFs have also been used in sequential latent variable models (Deng et al., 2020; Rempe et al., 2020). However, such works do not align the "time" axis of the CNF with the temporal axis of observations, and do not train observations more than one value of "time" in the CNF. In contrast, we align the time axis of the CNF with the time of the observations, directly using its ability model distributions on real-valued axis. A closely related application of CNFs to spatio-temporal data was done by Tong et al. (2020), who modeled the distribution of cells in a developing human embryo system at five fixed time values. In contrast to this, we extend to applications where observations are made at arbitrary time values, jointly modeling space and time within the spatio-temporal point process framework. Furthermore, Mathieu & Nickel (2020); Lou et al. (2020) recently proposed extensions of CNFs to Riemannian manifolds. For our proposed approach, this is especially interesting in the context of earth and climate science, as allows us to model STPPs on the sphere simply by replacing the CNF with its Riemannian equivalent.
5 EXPERIMENTS
Data Sets Many collected data can be represented within the framework of spatio-temporal events. We pre-process data from open sources and make them suitable for spatio-temporal event modeling. Each sequence in these data sets can contain up to thousands of variables, all the while having a large variance in sequence lengths. Varying across a wide range of domains, the data sets we consider are: carthquakes, pandemic spread, consumer demand for a bike sharing app, and high-amplitude brain signals from fMRI scans. We briefly describe these data sets here; further details, pre-processing steps, and data set diagnostics can be found in Appendix C. Code for preprocessing and training are open sourced at ht tps: / /github....... com/ facebookresearchh. wtwprrccchhuurr_rs_eeeee
This is a synthetic data set with multimodal and non-Gaussian spatial distributions designed to test the ability to capture drastic changes due to event history (see fig. 5). The data set consists of 10 clusters which form a pinwheel structure. Events are sampled from a multivariate Hawkes process such that events from one cluster will increase the probability of observing events in the next cluster in a clock- wise rotation. Number of events per sequences ranges between 4 to 108.
PINWIIEEL
EARTHQUAKES For modeling earthquakes and aftershocks, we gathered location and time of all carthquakes in Japan from 1990 to 2020 with magnitude of at least 2.5 from the U.S. Geological Survey (2020). Number of events per sequences ranges between 18 to 543.
7

---

Published as a conference paper at ICLR 2021
(a)
(b)
(c)
(d)
(E)
(1)
(
(h)
Figure 5: Evolution of spatial densities PINWHEEL data. top: Attentive CNF. bottom: Jump CNF. (a) Before
observing any events at t=0, the distribution is across all clusters. (h-f) Each event increases the probability observing future event from the subsequent cluster clock-wise ordering. (g-h) After a period of no new events, the distribution smoothly returns back to the initial distribution (a).
Jump CNF
Conditionnl KDE
(a)
(b)
(c)
(d)
Figure Snapshots of conditional spatial distributions modeled by the Jump CNF (top) and conditional kemel density estimator (KDE; bottom). (a) Distribution before any events at t=0. (b-d) The Jump CNF's distributions concentrate around tectonic plate boundaries where carthquakes and aftershocks gather, whereas the KDE must use large variance in order to capture propagation of aftershocks in multiple directions.
COVID-19 19 CASES We use data released publicly by The New York Times (2020) on daily COVID-19 cases in New Jersey state. The data is aggregated the county level, which we dequantize uniformly across the county. Number of events per sequences ranges between to 323.
BOLD5000 This consists of MMRI scans as participants are given visual stimuli (Chang et al., 2019). We convert brain responses into spatio-temporal events following the z-score thresholding approach in Tagliazucchi et al. (2012; 2016). Number of events per sequences ranges between 6 to 1741.
In addition to these datasets, we also report in Appendix A results for CITIBIKE, a data set consisting of rental events from bike sharing service in New York City.
Baselines To evaluate the capability of proposed models, compare against commonly-used baselines and state-of-the-art models. In some settings, ground intensity and conditional mark density are independent of each other and we can freely combine different baselines for the temporal and spatial domains. As temporal baselines, we use a homogencous Poisson process, a self-correction process, Hawkes process, and the Neural Hawkes Process, which were trained using their officially released code. As spatial baselines, we use conditional kernel density estimator (KDE) with learned parameters, where p(ær [f) is essentially modeled as history-dependent Gaussian mixture model (see Appendix B), as well as the Time-varying CNF. In addition, we also compare to our implementation of Neural Jump SDEs (Jia & Benson, 2019) where the spatial distribution is a Gaussian mixture model. GRU-based continuous-timg timme hidden states for fair comparison,
We use the same architecture as we found the simpler parameterization Jia & Benson (2019) to be numerically unstable for large number of events. Range of hyperparameter values are outlined in Appendix D.
8

---

Published as a conference paper at ICLR 2021
Pinwheel
Earthquakes JP
COVID-19 NJ
BOLD5000
Model
Temporal
Spatial
Temporal
Spatial
Temporal
Spatial
Temporal
Spatial
Poisson Process
-0. go
-0.11 1amm
0.878_0066
0.8622ne
Self-correcing Process
-2.117
-7.051
10.053
6.4 6.470
Hawkes Process
0.276
0.1 paaee
2.092.tom
2.860
Noural Hawkes Process
00.023
0.198
2.229 poors
3. 3.080
Conditional KDE
-2.988m0000
=2.259
-2.583=000
-,00
Time-varying CNF
-2.185nnoom
-1s459a00s
-2.002.orem
- 1,846_nav
Noural Jump SDE (GRL)
+0.006
-2.077 -wuD0
0.1 =G
-1.652
2.251 -tose
-20214
5.675
0,743
Jump CNF
0.027 s0.00
-1.562:0o1s
0.1 :
-1.007
2.242
=0900
5.536
i.246
Attentive CNF
0.034
-1.572=nome
0. 0.204
-1.237 20.0
2.258 -1 2600
5.842
1.252
Table I: Log- likeliiid per event on held-out test data (higher is better). Standard devs. estimated over 3 runs.
Results & Analyses The results of our evaluation are shown in table 1. We highlight all results where the intervals containing one standard deviation away from the mean overlap.
Across all data sets, the Time-varying CNF outperforms the conditional KDE baseline despite not being conditional on history. This suggests that the overall spatial distribution is rather complex. We also see from Figure 7 that Gaussian clusters tend to compensate for far-reaching events by learning a larger band-width whereas a flexible CNF can easily model multi-modal event propagation.
The Jump and Attentive CNF models achieve better log-likelihoods than the Time-varying CNF, suggesting prediction in these data sets benefit from modeling dependence on event history.
For COVID-19, 19, the self-exciting Hawkes process is strong baseline which aligns with similar results for other infectious diseases (Park et al., 2019), but Neural STPPs can achieve substantially better spatial likelihoods. Overall, NHP is competitive with the Neural Jump SDE; however, it tends to fall short of the Attentive CNF which jointly models spatial and temporal variables.
In closer comparison to the temporal likelihood of Neural Jump SDEs (Jia & Benson, 2019), we find that overly-restricted spatial models can negatively affect the temporal model since both domains are tightly coupled. Since our realization of Neural Jump SDEs and our STPPs use the same underlying architecture to model the temporal domain, the temporal likelihood values are often close. However, there is still a statistically significant difference between Neural STPP models and Neural Jump SDEs even for the temporal likelioil on all data sets.
Finally, we note that the results of the Jump and Attentive CNFs typically close. The attentive model generally achieves better temporal ligeliielsooods while maintaining competitive spatial log- liliiiiiihoods. This difference is likely due to the Attentive CNF's ability to attend to all previous events, while the Jump CNF has to compress all history information inside the hidden state at the time of event. The Attentive CNF also enjoys substantially faster computations (see Appendix A).
6 CONCLUSION
To learn high-fidelity models of stochastic events occurring in continuous space and time, we have proposed new class of parameterizations for ppatio-temporg point processes, Our approach combines ideas of Neural Jump SDEs with Continuous Normalizing Flows and allows to retain the flexibility of neural temporal point processes while enabling highly expressive models of continuous marks. We leverage Neural ODEs as a computational method that allows computing, up to negligible numerical error, the likelihood of the joint model, and we show that our approach achieves of the-art performance on spatio-temp meatio-temporl datasets collected from a wide range of domains.
A promising area for future work are applications of our method in earth and climate science which often are concerned with modeling highly complex ppatio-temporal data. In this context, the use of Riemannian CNFs (Mathieu& & Nickel, 2020; Lou et al.. 2020; Falorsi & Forré, 2020) is especially interesting as allows us to model Neural STPPs on manifolds (e.g. the earth's 's surface) by simply replacing the CNF in models with Riemannian counterpart.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We acknowledge the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing the core of tools that enabled this work, including PyTorch (Paszke et al., 2019), torchdiffeg (Chen, 2018), fairseq (Ott et al., 2019), Jupyter (Kluyver et al., 2016), Matplotlib (Hunter, 2007), seaborn (Waskom al., 2018), Cython (Behnel al., 2011), numpy (Oliphant, 2006; Van Der Walt et al., 10, pandas (McKinney, 2012), and SciPy (Jones et al., 2014).
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 2016.
Adrian Baddeley, Imre Bárány, and Rolf Schneider. Spatial point processes and their applications. Stochastic Geometry: Lectures Given at the CIME Summer School Held in Martina Franca, Italy, September 13-18, 2004, pp. 1-75, 2007.
Farvin Balderama, Frederic Paik Schoenberg, Erin Murray, and Philip W Rundel. Application of branching models in the study of invasive species. Journal of the American Statistical Association, 107(498):467-4444 476, 2012.
Stefan Behnel, Robert Bradshaw, Craig Citro, Lisandro Dalcin, Dag Sverre Seljebotn, and Kurt Smith. Cython: The best of both worlds. Computing in Science & Engineering, 13(2):31-39 2011.
Nadine Chang. John A Pyles, Austin Marcus, Abhinav Gupta, Michael J Tarr, and Elissa M Aminoff. BOLD5000, a public fMRI dataset while viewing 5000 visual images. Scientific data, 6(1):1-18, 2019.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Ricky T. Q. Chen. tor forchdiffec:
torchdiffeq. 2018. URL https: com/ www./o/o//tttttttoteeeeoooooooee
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. In Advances in neural information processing systems, pp. 6571-658. 2018.
Ricky TQ Chen and David Duvenaud. Neural networks with cheap differential operators. In Advances in Neural Information Processing Systems. pp. 9961-9971, 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
Daryl Daley and David Vere-Jones. An introduction to the theory point processes, volume 1: Elementary theory and methods. Verlag New York Berlin Heidelberg: Springer, 2003.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. GRU-ODE- Continuous modeling of sporadically obserred time series. In Advances in Neural Information Processing Systems, pp- 7390, 2019.
Ruizhi Deng, Bo Chang, Marcus A Brubaker, Greg Mori, and Andreas Lehrmann. Modeling continuous stochastic processes with dynamic normalizing flows. arXiv preprint Ni0:22002.10511 2020.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 4rXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. 2016.
10

---

Published as a conference paper at ICLR 2021
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodrigee and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp- 1555-1564, 2016.
Luca Falorsi and Patrick Forré. Neural ordinary differential equations on manifolds. arXiv preprint arXiv:2006. 201.:20006.06663, 06663,
2020.
Mehrdad Farajtabar, Jiachen Yang. Xiaojing Ye, Huan Xu, Rakshit Trivedi, Elias Khalil, Shuang Li, Le Song, and Hongyuan Zha. Fake news mitigation via point process based intervention. arXiv preprint arXiv: arXiv:1703.000 100.00073..77.... 2017.
Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam M Oberman. How train your neural ode. arXiv preprint wxXiv:200..02798, 2020.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference Learning Representations, 2019.
Amanda S Hering, Cynthia L Bell, and Mare G Genton. Modeling spatio-temporal wildfire ignition point patterns. Envirommental and Ecological Statistics, 16(2):225-250, 2009.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087, 2018.
Chin- Wei Huang, Laurent Dinh, and Aaron Courville. Solving ode with universal flows: Approxima- tion theory for flow-based models. In ICLR 2020 Workshop Integration of Deep Neural Models and Differential Equations, 2020.
John Hunter. Matplotlib: A 2d graphics environment. Computing in science engineering, 9()):
90, 2007.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Communications in SaatisticsSSmmllant and Computation, 19(2):433 450, 1990.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. In Advances in Neural Information Processing Systems, pp. 9847-9858, 2019.
Eric Jones, Travis Oliphant, and Pearu Peterson. {SciPy): Open source scientific tools for (Python]. 2014.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The Lipschitz constant of self-attention. arXiv preprint arXiv:2006.047/1 2020.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions. In Advances in neural information processing systems, pp. 10215-10224, 2018.
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter notebooks-a publishing format for reproducible computational workflows. In ELPUB, pp. 87-90, 2016.
Zhifeng Kong and Kamalika Chaudhuri. The expressive power of a class of normalizing flow models. arXiv preprini 20t:::2000000999 2020.
Liangda Li and Hongyuan Zha. Learning parametric models for social infectivity in multi multi-dimensional hawkes processes. In Proceedings of the Twenty-Eights AAA/ Conference on Artificial Intelligence,
pp. 101-107, 2014.
Tianbo Li and Yiping Ke. TWeedie-hawkes processes: Interpreting the phenomena of outbreaks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4699-4706, 2020.
Yang Li, Haidong Yi, Christopher M Bender, Siyuan Shan, and Junier B Oliva. Exchangeable neural ode for modeling. arXiv preprint 0rXiv:2008.02676 2020.

---

Published as a conference paper at ICLR 2021
Scott Linderman and Ryan Adams. Discovering latent network structure in point process data. In International Conference on Machine Learning. pp. 1413-1421, 2014.
Aaron Lou, Derek Lim, Isay Katsman, Leo Huang, Qingxuan Jiang, Ser-Nam Lim, and Christopher De Sa. Neural manifold ordinary differential equations. arXiv preprint arXiv:2006. 10254, 2020.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. arXiv preprint arXiv:2006. 10605, 2020.
Wes McKinney. Python for data analysis: Data wrangling with Pandas, NumPy, and IPython. O'Reilly Media, Inc.", 2012.
Nazanin Mehrasa, Ruizhi Deng, Mohamed Osama Ahmed, Bo Chang, Jiawei He, Thibaut Durand, Mareus Brubaker, and Greg Mori. Point process flows. arXiv preprint arXiv:1900.08281, 2019.
Hongyuan Mei and Jason Eisner. The Neural Hawkes Process: A neurally self-modulating multivari- ate point process. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp- 6754-6764, 2017.
Sebastian Meyer, Johannes Elias, and Michael Höhle. space-time conditional intensity model for invasive meningococcal disease occurrence. Biometrics, 68(2):607-616, 2012.
Jesper Moller and Rasmus Plenge Waagepetersen. Statistical inference and simulation for spatial point processes. CRC Press, 2003.
Maximilian Nickel and Matthew Le. Learning multivariate hawkes processes at scale. arXiv preprint 10xiv:2002./2501,, 2020.
Yosihiko Ogata. Statistical models for earthquake occurrences and residual analysis for point processes. Journal of the American Statistical association, 83(401):9-22 1988.
Yosihiko Ogata. Space-time point-process models for earthquake occurrences. Annals of the Institute of Statistical Mathematics, 50(2):379-40) 1998.
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Travis E Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3): 10-20, 2007.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrationss), pp. 48-53, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. N..11653//1/NN14400 URL https: www. ac aewweb. org/ anthology/
N1 N19-4009
T. Ozaki. Maximum likelihood estimation of Hawkes' self-exciting point processes. Annals of the Institute of Statistical Mathematics, 31(1):145-1555 Dec 1979. ISSN 1572-9052. doi: 10. 1001/ bf02480272.
Junhyung Park, Adam W Chaffee, Ryan Harrigan, and Frederic Paik Schoenberg, A non- paramaract Hawkes model of the spread of ebola in west africa. 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, pp. 8026-8037, 2019.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint 1rX10:1710.05941, 2017.
Alex Reinhart et al. A review of self-exciting spatio-temporal point processes and their applications. Statistical Science, 3393):299-318, 2018.
12

---

Published as a conference paper at ICLR 2021
Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojcie, Srinath Sridhar, and Leonidas J Guibas. CaSPR: Learning canonical spatiotemporal point cloud representations. arXiv preprint (rXi1:2008.02792, 2020.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning, pp. 1530-1538, 1538, 2015.
Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud. Latent ODEs for ireeularly--sappled time series. arXiv preprint 1907 039007 2019.
Frederie Paik Schoenberg, Mare Hoffmann, and Ryan Harrigan. A recursive point process model for infectious diseases. Annals of the Institute of Statistical Mathematics, 7165):271-1287,, 2019,
Oleksandr Shchur, Marin Bilos, and Stephan Günnemann. Intensity-free learning temporal point processes. arXiv preprint arXiv: urXiv:199. 12127, 2019.
John Skilling. The eigenvalues of mega-dimensional matrices. In Maximum Entropy and Bayesian Methods, pp. 466.466.77 Springer, 1989.
Enzo Tagliazucchi, Pablo Balenzuela, Daniel Fraiman, and Dante R Chialvo. Criticality in large-scale brain fMRI dynamies unveiled by novel point process analysis. Frontiers in physiology, 3:15, 2012.
Enzo Tagliazuechi, Michael Siniatchkin, Helmut Laufs, and Dante R Chialvo. The voxel-wise functional connectome be efficiently derived from co-activations in sparse spatio-temporal point-process. Frontiers in neuroscience, 10:381, 2016.
Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling- based invertible neural networks are universal diffeomorphism approximators. arXiv preprint arxiv:2006. |469, 2020.
The New York Times. Coronavirus (Covid-19) Data in the United States, 2020. URL https: //github... com/ wwtom//ytiieee/eeeomooo caannnessovvid-1---eee datta.
Alexander Tong, Jessie Huang, Guy Wolf, David van Dijk, and Smita Krishnaswamy. Trajecto- ryNet: A dynamic optimal transport network for modeling cellular dynamics. arXiv preprint MxXi:20002.04611, 2020.
U.S. Geological Survey. Earthquake Catalogue (accessed August 21, 2020), 2020. URL https: // //earthquake usgs. gov / watttaatttuaaeeeeeeeeaaa search/.
Stefan Van Der Walt, Chris Colbert, and Gael Varoquaux. The numpy array: structure for efficient numerical computation. Computing in Science & Engineering, 13(2):22, 2011.
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Michael Waskom, Olga Botvinnik, Drew Kane, Paul Hobson, Joel Ostblom, Saulius Lukauskas, David C Gemperline, Tom Augspurger, Yaroslav Halchenko, John B. Cole, Warmenhoven, Julian de Ruiter, Cameron Pye, Stephan Hoyer, Jake Vanderplas, Santi Villalba, Gero Kunter, Eric Quintero, Pete Bachant, Marcel Martin, Kyle Meyer, Alistair Miles, Yoav Ram. Thomas Brunner, Tal Yarkoni, Mike Lee Williams, Constantine Evans, Clark Fitzgerald, Brian, and Adel Qalich. mwaskomm/seaoom: v0.9.0 (july 2018), July 2018. URL https:://oi.ont 5281/zenodo 1313201.
Guandao Yang, Xun Huang. Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. PointFlow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE International Conference on Computer Vision, pp. 14450, 2019.
Qingyuan Zhao, Murat A Erdogdu, Hera Y He, Anand Rajaraman, and Jure Leskovec. Seismie: A self-exeiting point process model for predicting tweet popularity. In Proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pp. 1513-1522, 2015.
13

---

Published as a conference paper at ICLR 2021
A
ADDITIONAL RESULTS AND FIGURES
Citibike
3.2
Model
Temporal
Spatial
Jump CNF
3.0
Attentive CNF
Poisson Process
0.609
Self-correciig Process
2.8
Hawkes Process
1.062
Train
Neural Hawkes Process
1.030
2.6
Conditional KDE
-288561o00
2.4
Time-varying CNF
-2.132+603
Neural Jump SDR
1.092.nom -
2.2
100000
200000
300000
400000
Jump CNF Attentive CNF
1.105
-2.155+00s
Wallolock time (seconds)
1.1 1.112 -2.0951000m
Table Log- ligllikelhood values on held-out test data for an urban mobility data set.
Figure Runtime comparison of Jump and Attentive CNF.
VMM
(0)
X
A (')
Jump CNF
Attentive CNF
Figure Both Jump CNF and Attentive CNF capable modeling different the spatial distributions based on event history, so the appearance synthetic I-D data set similar to PINWHEEL, except we use mixture of three Gaussians. Each event increases the likelihood of events for the cluster to the right.
new event effectively shifts distribution instantaneously. Shown on
Figure 10: Learned attention weights random event sequences.
14

---

Published as a conference paper at ICLR 2021
B BASELINE
Our self-excitation baseline uses Hawkes process model the temporal variable, then uses Gaussian mixture model to describe the spatial distribution conditioned on history of events. This corresponds to the following likelihood decomposition
n
log p(gptt....... 21, ..... Ln. log p t1.............. 4-11
24
(20)
Note that f; does not depend on the spatial variables associated with previous events. This dependence structure allows the usage of simple temporal point processes to model t,, e.g. a Hawkes process, since temporal variables do not depend on the spatial information. The spatial distribution conditions all past events well the current time of occurance.
Our baseline model assumes a simple Gaussian conditional model, that new events are likely to appear previous events.
expllt, - tz)/T}
pla, t1.................. 44-1, 21 11-1) = Σ (xz(o))) aj E expl t,//
j=1
(21)
This parameteric model has two learnable parameters: o² and r, which control the rate of decay in the spatial and temporal domains, respectively.
However, this Gaussian spatial model assumes events are propagated in all directions equally and can only model local self-excitation behavior. These assumptions are often used for simplifications are generally incorrect for many spatio-temporal data. To name few, earthquakes occur more frequently along boundaries of tectonic plates, epidemies propagate along traffic routes, taxi demands saturate locally and change as customers move around.
C PRE-PROCESSING STEPS FOR EACH DATA SET
We sample from a multivariate Hawkes process with 10 dimensions. We turn this into continuous spatial variables by assigning each dimension cluster from "pinwheel" distribution, and sample from the corresponding cluster for each event. Number of events per sequences ranges between 4 108.
PINWHEEL
EARTHQUAKES For modeling earthquakes and aftershocks, we gathered location and time of all earthquakes in Japan from 1990 to 2020 with magnitude of at least 2.5 from the U.S. Geological Survey (2020). Starting from January 01, 1990, we created sequences with gap of 7 days. Each sequence was of length 30 days. We ensured there no contamination between train/val/test sets by removing intermediate sequences. We removed earthquakes from 2010 November to 2011 December, as these sequences were too long and only served as outliers in the data. This resulted in 950 training sequences, 50 validation sequences, and 50 test sequences. Number of events per sequence ranges between 18 to 543.
COVID- 19 CASES We use data released publicly by The New York Times (2020) on daily COVID-19 cases in the New Jersey state, from March to July of 2020. The data aggregated at the county level, which we dequantize uniformly across the county. We also dequantize the temporal axis by assigning new cases uniformly within the day. Starting at March 15, and every 3 days, we took a day length sequence. For each sequence, we sampled each event with a probability of 0.01. This was done 50 times per sequence. We ensured there was no contamination between train/val/test sets by removing intermediate sequences. This resulted in 1450 training sequences, 100 validation sequences, and 100 test sequences. Number of events per sequence ranges between 3 to 323.
CITIBIKE Citibike is a bike sharing service in New York City. We treat the start of each trip as an event, and use the data from April to August of 2019. We split into sequences length I day starting at 5:00am of each day. For each sequence, subsampled with probability of 0.005 per event, 20 times. This resulted 2440 training sequences. 300 validation sequences, and 320 test sequences. Number of events per sequence ranges between 9 to 231.
15

---

Published as a conference paper at ICLR 2021
200
250
Frequency
150
Frequency
200
150
100
100
50
0
50
100
150
200
250
300
350
0
50
100
150
200
250
300
per
per Sequence
Earthquakes JP
Covid-19 NJ
400
Frequency
200
Frequency
300
200
100
0
50
100
150
200
0
0
200
400
600
800
1000
1200
per Sequence
per Sequence
Citibike NY
BOLD5000
Figure 11: Histograms of the number of events per sequence in each processed data set.
BOLD5000 This consists of IMRI scans of four participants as they given visual stimuli (Chang et al., 2019). We use the sessions single patient and for each run, we split into 3 sequences, treated individually. We converted brain responses into spatio-temporal events following the z-score thresholding approach in Tagliazucchi et al. 2016 Equation (2). We used threshold of = 6.0. We split the data into 1050 training sequences, 150 val sequences, 220 test sequences. Number of events per sequence ranges between to 1741.
Each data set contains sequences with highly variable number of events, with varying degrees of dependence between events, making them difficult to model with traditional point process models. We plot histograms showing the number events per sequence in Figure 11.
D HYPERPARAMETERS CHOSEN AND TESTED
For the time-varying, jump, and attentive CNF models, parameterized the CNF drift multilayer perceptron (MLP) with dimensions [d 64 - 64 - 64 - d]. where is the number of spatial variables. We swept over activation functions between using softplus time-dependent Swish (Ramachandran et al., 2017).
TimeDDependennSwish(t..
ho @ 2)
(22)
where is the logistic sigmoid function, 3 is the Hadamard (clement -iiee) product, and 3 R + R.,, MLP with widths [1 64 dg] where d, is the dimension of z, using the softplus activation function. We ultimately decided on using the time-dependent Swish for all experiments.
We swept over the MLP for defining fn for the continuous-time hidden state in eq. (1I) using hidden widths of [8 20). [32 32), [64 64). [32 - 32 32], and [64 - 64 - 64). The majority of models used 32- 32 as provided enough flexibility while remaining easy solve. We used the softplus activation function. We tried MIP for parameterizing the instantaneous change in eq. (12); however, was too unstable for long sequences. We therefore switched to the GRU parameterization, which takes input (new event), the hidden state at the time of event, and outputs new hidden state.
We regularized the Le norm of the hidden state drift with strength of lc-4, chosen from {0, le-4, le-3, le-2]. We optionally used optimal transport-inspired regularization from Finlay et al. (2020), which adds a Frobenius norm regularization to the gradient of the drift in addition to the L2 norm regularization, the CNF models with strength of {0, le-4, 1e-3, 1e-2). The varyingg and Attentive CNF models did not require regularization and were mostly kept at 0, but the Jump CNF models benefited from some amount of regularization avoid numerical instability.
16

---

Published as a conference paper at ICLR 2021
To model non non-trivial spatial distribution for the entire data interval, we shift the data interval to start at f = for all CNF models. Thus the interval used for parameterizing the CNF is [2, T + 2]. Generally, the "time" variable is dummy one; can place the base distribution at any time, and we can choose any interval on the real line to be the data interval; this does not limit the model in any way.
For the Jump CNF, we used a composition of radial flows (Rezende & Mohamed, 2015) to parame- terize the instantaneous updates in eq. (17). All parameters of the radial flows were parameterized to be the output of a MLP that takes as input the hidden state at the time of the event (before the hidden state is updated based on the current event). The radial flows were initialized in such a way that the log determinant is near zero.
For the Attentive CNF, the drift function consists of
Time-dependent MLP(d 64 64) - 2x Muutiheaa/AAtttnioonnn + Time-dependent MLP(644 - 64 - d)
where the Time-dependent MLPs make use of the TimeDependenttwish As was done in Vaswani et al. (2017), the multihead attention is used within a residual branch, except we swapped LayerNorm (Ba et al., 2016) with ActNorm (Kingma & Dhariwal, 2018) as LayerNorm has an unbounded Lipschitz and can be ill-suited for use in ODEs. We tested both standard multihead attention (Vaswani et al., 2017) and the Lipschitz multihead attention (Kim et al., 2020). The Lipschitz multihead attention typically produced similar validation NLLL as the standard multihead attention but were more stable on multiple occassions. We therefore kept the Lipschitz multihead attention for all experiments. We additionally, use an auxiliary (non-attentive, simply with the two multihead attention layers removed) CNF to map from 0 (i.e. the time of base distribution) the beginning of the data interval.
We initialized all Neural ODEs (for the hidden state and CNFs) with zero drift by initializing the weights and biases of the final layer zero
The log- bog-likelihood values reported are after the spatial variables have been standardized using the empirical mean and standard deviation from the training set.
We train and test on log-liielhhoo (in nats) per event, which normalizes eq. (8) of each sequence by the number of events.
All integrals were solved using Chen (2018) to within a relative and absolute tolerance of 1E-4 or 1E-6, chosen based preliminary testing for convergence and stability.
Our implementation of the Neural Jump SDE shares the same continuous-tima hidden state parame- terization but uses mixture of Gaussians as the spatial model. We used 5 mixtures, and MLP that maps from the hidden state to the parameters of this mixture of Gaussians (the means, log standard deviations, and mixture coefficients).
E REMOVING CROSS-EVENT PARTIAL DERIVATIVES
This results in a lower-variance gradient estimator for training, and allows parallel computation of conditional log probabilities at test time.
We first summarily describe the attention mechanism. For an input X E Raxd representing the
sequence of variables {a!0) (0)
,..., ....,,,.)) _.(")) } at some value of s. this attention mechanism creates logits P E R=x= and values V E Raxd both dependent on X, such that the output is
O softmax(P)) V.
(23)
where the softmax is taken over each row of P. The output is then added to X as a residual connection. The multihead attention computes P in way such that P.j depends on X; and Xj. and V/2 depends on X, This is true for both the vanilla MHA (Vaswani et al., 2017) and the L2 MHA (Kim et al., 2020). For our use case, Pss set to inf for j > i as we don't want to attend to future events.
We retain only the block diagonal gradients where each block contains variables corresponding to one event. This is equivalent to removing all the cross-event dependencies.
80: = S.. ox;
0V,
+ VT OS OP:
VT as oP.,
(24)
0x:
OP, 0x,
OP., 0X;
17

---

Published as a conference paper at ICLR 2021
F PARALLEL SOLVING OF MULTIPLE ODES WITH VARYING INTERVALS
da Our numerical ODE solvers integrate a single ODE system di f: IR+dd + Rd, on single fixed interval [tstart, tend]. We can express the inputs and outputs of an ODE solver with
f(t, x.), where I E Rd and
Mond
ODESolve f, Estort: tend) 2 20 +
f(t,z(t)) dt I(tena).
(25)
Pstart
where up is vector containing the initial state the initial time
Multiple ODEs Now suppose have set of M systems (i.e. dem fm for 1,..., M) that we would like to solve. If all systems had the same initial time and desired output time of tstart and Lend, we can readily create joint system
at
æ1
f1(t,21)
dæjoint dt
T joint
that follows
(26)
JM
[M((,M))
Solving this joint system can be done in parallel with single call to ODESOlve
Fjoint (tz) CDESo seesoovv(mpp) ve(rojoint fjoint; tu, (1)
(27)
which computes Im (t1) for all m = ,,.... M. This is the standard method used for solving a batch of Neural ODEs.
Adding dependencies is straightforward Note that extending this further, the ODE systems do need be independent. They can depend on other variables the same concurrent time value because the joint system is still an ordinary differential equation. For instance, we can have
dx dt fm (t, 21...., Tm)
(28)
where fm Id -
R.d. Each system now partial differential equation, but the joint system Tjoint is still an ODE and can be solved with one call to ODESolve.
Varied time intervals Now suppose each system has a different time interval that we want to solve. Different initial times and different end times. Let's denote the start and end time for the m-th system as fim) and ((m) respectively. We can construct dummy variable that always integrates from 1, and perform change of variables (reparameterization) transform every system to use this dummy variable.
As a concrete example of this reparameterzation procedure, consider just one system sxt) with drift function f(t, æ) that we want to integrate from tstart to tend with the initial value zo. We can transform ztt) using the relation Fend Istard
t-tutart or equivalently
t= s(tnd tstart) + tstarts
(29)
into a solution ā(s) on the unit interval [0, 1] such that
*(s) xx( r(s(tend tstart) + Estart)
(30)
The drift function for then follows as (ss) e dī(s) dx(t) ds
dt
dl
ds
(31)
f(t,x(())
t=s(tend tsa (Nend tstars)
= =(((tend - tstars) + tetart, (()) (tend - tstart)
Now since #(0) v(tstarr)) and ((1) xltena), the following equivalent
s(land) ODE.ol ve f.0, 0, ODESol ve J. Iistarts bend)
(32)
18

---

Published as a conference paper at ICLR 2021
Putting it all together
Let (s) be the reparameterized solution for 2m (t) such that
(s) Im (s
start)
(m)
+ Estart)
(33)
We can then solve for all M systems, with different varying time intervals, using
31 :
(1)
jojoitt ds
joint
that follows
(34)
JM
(M((s
Solving this system
yields Em (1) F.m (1 (L)
(m) Assuming 'start = 0 for all m = I..... M in order to reduce notational complexity, we can write this joint system terms of the original systems as
fi st (stend a(s) tend
dā joint ds
(35)
fi (st)na)((s) 2(s) tend
This is the joint system written in equation 14. The joint system in equation 19 adds dependence between the M systems but can still be solved with a single ODESolve.
19


Reviews:
[{"review_id": "k4KSafGcdGI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is generally well written. \n\nSection 3 is a little confusing as it's not readily clear which part of the model is using flows. Or what are the flow parameters that needed to be estimated.  How  will the neural network structure look like? How is it trained?\n\nOne might wonder why a flow based model is used among these many deep generative models? Where does this invertibility help out?\n\nThe GRU model is not well elaborated and is a little unclear.\n\nIt would have been interesting to see where the attention model usually attends to. Either in a real world data set or in simple intuitive toy tasks.\n\nEquation 14 and 19 are hard to follow. It's good to elaborate on them. At least in an appendix section due to space limitations.\n\nWill the log-likelihood sufficient for evaluation? Why not more intuitive tasks like event or time predictions? More explanation and justification would have been helpful.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "djNCj7E3Oqb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel deep approach to the learning of spatio-temporal point processes via normalizing flows. Overall I think this is a good paper, presenting many interesting ideas and may impact further research on point processes.  The combination of flow-based network structure and the probabilistic model--point process should make sense. The formulation and presentation are good, which makes the paper easy to follow. However, there are a couple of questions for the authors to further address:\n1. It seems that the proposed model contains a jump CNF for the mark probability $p^*(\\boldsymbol{x}_t|t)$. I'm not sure if it really makes sense that the probability of mark has a \"jump\" over time. Here such formulation seems to be problematic. It is straightforward for the ground intensity function to consider the discontinuity at the point when an event occurs, as it represents a (self-exiting/inhibitive) temporal point process. The features are often assumed to be homogenous over time. It would be better if the authors can justify such a formulation.\n2. It seems intractable for the model to predict the next event. To compute the arrival time and mark for the next event, one should consider integrals with respect to $\\lambda$ and $p$, which looks quite complex when they involve ODEs. \n3. The authors are advised to further illustrate the attentive CNF. This seems to be a very interesting topic, but the authors only give a brief introduction in very short content. It is not very clear how the authors incorporate the attention mechanism to CNF. \n4. The experiment looks somehow weak.  First, the authors criticize that KDE has a large entropy (variance?). However, the variance of KDE depends on the kernel bandwidth, it is not fair to judge based on just one prefixed kernel bandwidth. Second, the authors seem to miss a couple of baselines that deal with the same task in the experiment. The NHP and the RMTPP are also able to model the spatial-temporal point process if the losses are change to the metrics on Euclidean space. Besides, please consider \nLi, L., & Zha, H. (2014). Learning parametric models for social infectivity in multi-dimensional Hawkes processes. AAAI 2014.\nLi, T., & Ke, Y. (2020). Tweedie-Hawkes Processes: Interpreting the Phenomena of Outbreaks. AAAI 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NqwXVem8OKg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a neural-ODE-based point process for spatio-temporal data. Under the general framework, three particular variants are proposed: they handle data with different characteristics and have different computational efficiency. \n\nPros: \n\nThe idea is interestingly novel. \n\nThe proposed model architectures are all carefully thought through: each model component is well-motivated, being supported by convincing justification. \n\nThe presentation is clear; some technical parts are enjoyable to read. \n\nThe empirical results on log-likelihood comparison look compelling. \n\nCons: \n\nThe experiments are somewhat weak: this is the main reason I didn’t give a higher score. \n\nFor temporal comparison, there isn’t any neural baseline model. \n\nThere isn’t any prediction accuracy comparison. \n\nEmpirical analysis is very limited (maybe cuz of limited experiments conducted). \n\nQuestions: \n\nThe use of * is really non-standard: in statistics and machine learning, * usually denotes somewhat ground-truth. Can authors think of another notation to omit $H$? Or maybe $H$ can be kept since the single-column format is spacious enough to host long equations? \n\nEqn-(8): why not $\\log p(x | t)$? Is this a typo? \n\nRNN (particularly, LSTM) with continuous-time hidden state was proposed by Mei and Eisner 2017, earlier than the cited Rubanova et al. 2019. \nMoreover, the math properties described by eqn-(10--12) also hold for Mei & Eisner 2017. \nCan authors appropriately acknowledge these connections? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "I5Mj9xZGkog", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work investigates a new class of parameterizations for spatio-temporal point processes\nwhich uses Neural ODE to enable flexible, high-fidelity models of discrete events that are localized in continuous time and space. \n\nStrengths: this work is essentially an extension of the Neural Jump SDEs [Jia & Benson (2019)] where the temporal dynamics is modeled as an ODE and the spatial pdf is modeled as a history dependent Gaussian mixture distribution. In this work, the spatial pdf is further extended to an ODE based dynamics. For this purpose, three different continuous normalizing flow models are proposed (Time-Varying CNF, Jump CNF, Attentive CNF). Also, a large number of experiments are conducted and baselines are compared to validate the conclusion.\n\nI recommend rejection at the current stage for the reasons below.\n\nWeakness: A major concern is, if my understanding is right, every mark x^(i) is modeled as an ODE of x^(i)_t on [0, t_i] in the in Time-Varying CNF and Attentive CNF, so there are N (the number of points) ODEs in the model. This setup is problematic because any points except the 1st are impossible to happen at time 0, so they impossibly possess a mark x^(i) at time 0 (in fact, any time before t_(i-1) is impossible). A more reasonable way to characterize the dynamics of x^(i) is to model the ODE on [t_(i-1), t_i] which is used in the Jump CNF. I understand this setup contributes to the parallel computation with the reparameterization trick. In fact, this is reason why both Time-Varying CNF and Attentive CNF can be computed in parallel, but Jump CNF cannot. The Attentive CNF can be seen as a generalized version of Time-Varying CNF due to the introduction of history dependence, but the Jump CNF is a different model as stated above. \nAlso, the jump CNF can model the abrupt change of the spatial pdf but the Time-Varying CNF and Attentive CNF cannot. Theoretically speaking, the jump CNF should have a more powerful fitting capability (assuming other parts are same) compared with those two models. Why does the Attentive CNF model achieve a better or close performance than jump CNF in most experiments? Does that mean the dynamics in most datasets have no discontinuity? Maybe a simple synthetic experiment with discontinuity in dynamics can help prove this. \n\nSome specific concerns: some synthetic data experiments with specific setup (e.g. discontinuity) are needed to give a deep understanding of the two proposed spatial CNF models.\n\nTypo: or-->of, the second line from the bottom in the first page.\n 0-->1, the second line of Eq.(19). ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a model for spatiotemporal point processes using neural ODEs. Some technical innovations are introduced to allow the conditional intensity to change discontinuously in response to new events. Likewise, the spatial intensity is expanded upon that proposed in prior work on neural SDEs. Reviewers were generally positive about the contributions and the empirical assessments, and the authors made substantial improvements during the discussion phase.
Decision: Accept (Poster)