Neural multi-event forecasting on spatio-temporal point processes using probabilistically enriched transformers
Authors: Negar Erfanian
Santiago Segarra
Maarten V. de Hoop
Venue: Submitted to ICLR 2023 (2023)
Keywords: Stochastic Point Processes
Multi-event Prediction
Transformers
Normalizing Flows
Hawkes Process
Deep Learning
Generative Models

Abstract: Predicting discrete events in time and space has many scientific applications, such as predicting hazardous earthquakes and outbreaks of infectious diseases. History-dependent spatio-temporal Hawkes processes are often used to mathematically model these point events. However, previous approaches have faced numerous challenges, particularly when attempting to forecast multiple future events. In this work, we propose a new neural architecture for multi-event forecasting of spatio-temporal point processes, utilizing transformers, augmented with normalizing flows and probabilistic layers. Our network makes batched predictions of complex history-dependent spatio-temporal distributions of future discrete events, achieving state-of-the-art performance on a variety of benchmark datasets including the South California Earthquakes, Citibike, Covid19, and Hawkes synthetic Pinwheel datasets. More generally, we illustrate how our network can be applied to any dataset of discrete events with associated markers, even when no underlying physics is known.

Under review as a conference paper at ICLR 2023
NEURAL MULTI-EVENT FORECASTING ON TEMPOOLT POINT PROCESSES USING PROBABILISTICALLY ENRICHED TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Predicting discrete events in time and space has many scientific applications, such predicting hazardous earthquakes and outbreaks of infectious diseases. History- dependent spatio-temporal Hawkes processes are often used to mathematically model these point events. However, previous approaches have faced numerous challenges, particularly when attempting to forecast one or multiple future events. In this work, propose a new neural architecture for multi-event forecasting of spatio-temporal point processes, utilizing transformers, augmented with normal- izing flows and probabilistic layers. Our network makes batched predictions of complex hispory-dependent spatio-temporal distributions of future discrete events, achieving state-of-the-an art performance on a variety of benchmark datasets includ- ing the South California Earthquakes, Citibike, Covid-19, and Hawkes synthetic pinwheel datasets. More generally, we illustrate how our network can be applied to any dataset of discrete events with associated markers, even when underlying physics known.
INTRODUCTION
Predicting the occurrence of discrete events in time and space has been the focus of many scientific
studies and applications. Problems such as predicting earthquake hazards (Ogata) 1998; Chen et al.)
2020), infectious diseases over a population (Meyer al. 2012 Schoenberg al. 2019). mobility and traffic in cities (Du et al., 2016), and brain neuronal spikes
(Perkel et al.) [967) fall under this category and have gained quite some interest. Over the years, many works used the history de- pendent spatio-temporal Hawkes process Ozaki 1979 Ogata 1988
1998, Ogata & Zhuang (0066] Ogata et al. 1993 Nandan et al. 2017: Sornette Werner 2005 Zhuang 2012 Helmstetter & Sornette, 2003 Bansal & Ogata (013) to model these point events. The stochasticity and the ex- citatory destorr-dependency of the Hawkes process are modeled by a conditional intensity function that varies in time and space. Assuming events and their associated markers (t, x, M) collected in the history He- = {(t,, X,, M,) t4 < t, =i==0 1::1 n), the Hawkes intensity function is defined as (x),
x = lim
no(x) + Σ go - 4, x 1x--xc x,, M.), SE
(I)
Á [B(x, Ax)
where Pxr,ax (t, x(1);) denotes the history-dependamt probability of having an event in small time interval 1 1 + At) and a small ball Bxx, Ax) centered at x E Rd with the radius of Ax E Rd, and He denotes all events happening up to but not including time The functions μı and Jo represent the (parametric) background intensity function and spatio-temporal kernel, respectively, forming X(t, x H.)
variety of different parametric forms for both μi and Js have been pro- posed (Ogatal 1998). Despite their use across different domains, Hawkes processes have several limitations. First, one has to predetermine parametric forms for μ4 and gg. This limits the expres- sive power of the model and, more importantly, might lead to mismatch between the proposed model and one that can better represent the observed data. Second, (T) imposes similar behavior across all events over time meaning that every preceding event has the same form of impact on fu- ture events' occurrence. As an example, in the classical Hawkes process with a temporal decaying kernel ( - t₂)= = exp(- (t--tt))), ) the same parameter (3 is applied to all events via their corre- sponding time differences. Third, the intensity function A(t, x (H.) can be used to predict only one

---

Under review as a conference paper at ICLR 2023
step ahead via, e.g., the commonly-ure ussd first-order moment (Rasmussen, 2011 Snyder & Miller 2012)
given by
E[(n+11 He,,] 10 "/y X(1,
(-Jx J.
H(1,x|H(.) exp
M(u, v|H,, )dudv dxdl,
(2)
in
J X(t, ((,,) (-/ 1|x,+1 H,,]= / x tn.
I. M(u, v/H )dddd) dtdxx )
where solving the integral in continuous high-dimensional space could be computationally expen- sive, or inaccurate in the case of using sampling methods such as Monte Carlo sampling (Hastings) 1970). Fourth, although (2) makes it feasible predict the next event, the multi-event prediction demands sequential usage of the first-order moment, where at each step the history is updated via the most recent predicted event. This makes the problem of multi-event prediction challenging and prone to error accumulation.
To alleviate some of the mentioned shortcomings, previous works have proposed learning the inten- sity function A(1, x H))) directly from data using neural networks. The early studies in this regard (Du et al., 2016 Mei & Eisner 2017 Xiao al. 2017) attempted to learn A(1, x H2)) using a re-
current neural network (RNN) and variants thereof. In a recent work, Chen et al. (2020) designed a neural ordinary differential equation (ODE) architecture that can learn a continuous intensity func- tion. This is attained by combining jump and attentive continuous normalizing flows (CNFs) (Chen) et al. 2018 for space, and an RNN architecture for time. Despite being data-driven, these works do not tackle the last three drawbacks mentioned above. Moreover, due to the memory limitations of RNNs, these works are incapable of unveiling long-term dependencies. On the other hand, RNNs and their variants suffer from vanishing and exploding gradients, especially when applied to long sequential data. Furthermore, sequential training of both RNNs and jumpCNFs as proposed byen jet al. (2018) results in slow and computationally expensive training process. Some other recent works (Zuo et al. 2020 Zhou et al. 2022). proposed using attention mechanisms (already present in the attentiveCNN F in Chen et al. (2020)) alleviate the second drawback mentioned earlier. The capability of discovering long-term dependencies while achieving fast training (via parallelization) another benefit of this line of work. However, the last two drawbacks are still present in those approaches, hindering their utility for multi-event forecasting. Directly learning the temporal dis- tribution using the variational autoencoders (VAEs) (Kingma & Welling, 2014) augmented with CNFs (Chen et a 2018) as proposed by Mehrasa et al. (2019). or utilizing the deep sigmoidal flow (DSF) (Huang et 2018) and the sum of squares (SOS) polynomial flow (Jaini et al. 2019) as proposed by Shchhur et al. (2019) could tackle the first and third limitations. However, the second and last mentioned shortcomings remained to be solved. In another recent work, Zhu et al. (202I) proposed learning the Hawkes intensity function based on Gaussian diffusion kernels while using imitation learning as a flexible model-fitting approach. However, their method could not tackle any of the aforementioned shortcomings. Therefore, in the present work, we introduce novel neural multi-event forecasting while addressing
network that is capable of simultaneous spatio-tempore all the above-mentioned shortcomings. To the best of our knowledge, this is the first work propos- ing a data-driven multi-event forecasting network for stochastic point processes. Our architecture augments the encoder and decoder blocks of transformer architecture (Vaswani et al., 2017) from natural language processing (NLP) with probabilistic and bijective layers. This network design pro- vides batch of rich multi-event spatio-tempore distributions associated with multiple future events. We propose to directly learn/predict these distributions (without the need to learn the intensity func- tion as opposed to the previous works), using self seff-super learning (Liu et al., (02T), marking a further point of departure with respect to existing work. The rest of this paper is organized as follows. In Section we provide background knowledge on attention mechanisms and normalizing flows, since these used as constituent blocks in solution. Next, in Section 3 we provide formal definition of the problem to be solved along with our proposed neural multi-event forecasting network. Further, in Section 4 we evaluate the performance of our network on variety of datasets and compare it with baseline models. Finally, in Section 5 we finish the paper with conclusions and future goals. Our contributions can be summarized as follows:
We introduce a neural architecture that is capable of simultancous multi-event forecasting of the time and location of discrete events in continuous time and space.
We compare the performance of our solution with state-of of-te----the-rr models through extensive experiments on variety of datasets that represent stochastic point events.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND
2.1 ATTENTION AND TRANSFORMERS
Attention blocks play a critical role in the transformer architecture (aswann et al. 2017). Here we introduce their fundamental pieces, already specializing them to the problem hand. Let {R: X,, M,]li=:nn be a sequence
(column) vectors K; E Rd+2 representing n discrete events formed by concatenating the associated time R and markers X; R2 and M; R. The marker x; denotes the location of the event at time I; in a d dimensional space, and M; can be used to encode any other information of interest about the event, such the magnitude of an earthquake or the biker's age. Even though we represent here M, as scalar, these markers can also be higher dimensional. Moreover, the events K, are temporally sorted such that & < tj for < j. Using learnable linear transformations, we form query, key and value vectors (denoted by qi k;, and Vi, respectively) for each event R; as
qi W KE
WEKis Ki; k;
V; = W. Kig
where W, (d+2) WA (d+x()+ W. Rd.x(d42)) For a given event K2, we build the matrix K (3) = k2 ki), which contains as columns the key vectors of all the events up to (and containing) event K; and, similarly, the matrix VG) = [VIV2... vz] values. Based on this notation, compute the hidden representation h, of event K;
K TT9 %
h; V(t) softmax
(5)
dd
Note that in (5) we are generating h, by linearly combining the columns of V(a), i.e. the values of all events preceding (and including) KG. The weights in this linear combination are given by softmax applied to the inner products between the keys of every event and the query specific of KG- In this sense, h, contains a weighted combination of the values of all preceding events, where these values and the weights in the combination can be learned by tuning the transformations W, g. WE. and W..
2.2 NORMALIZING FLOWS
A normalizing flow is transformation of simple probability distribution complex one using a sequence of differentiable and invertible bijective maps (Kobyzev et al. 2020). Assume that k E Rdd has complex unknown distribution Px- Having z E R as a random sample drawn from simple known distribution P= (e.g., normal distribution). we could use sequence of invertible and differentiable functions F F1 o F2 ..... Fs to map z into is according to = F(z). Therefore, using the change of variables formula we have that
Ps (z)
Px (F()))))ee((((
(6)
where (z)(D)(()) denotes the determinant of the Jacobian matrix of F(z) and accounts for the volume change in the density as we move from the simple distribution to the complex one. Equa- tion (6) makes it feasible to have explicit form for the complex underlying distribution P.. and we can effectively sample from it by generating samples from pz and transforming those via F.
NEURAL MULTI-EVENT FORECASTING
Given history of events, we want determine the probability distribution (in time and space) of the next L events. Formally, we want to solve the following problem.
Problem Given the event history Ht, {(4,, M,)
n). estimate the probability
distributions {pr(ti,> x HH H.)+ of the next L events.
We refer to the above problem as batched or multi-event prediction because given the history up to event n we want to estimate the ppatio-tempora probability densities of a batch of L events shown by pt in Problem 1. Note that despite using the word batched. these distributions do not need to be independent of each other. We present here a data-driven solution our problem where, during training phase, get to observe several sequences of length + L. of events of interest.
3

---

Under review as a conference paper at ICLR 2023
We propose transformer-basee architecture trained in self-supervised manner to extract the rela- tions between historical events. Furthermore, we enhance the transformer block with probabilistic and bijective layers construct history-depender spatio-temporal distributions from the hidden states of the transformer. Since pr(1,, xi H+,) = pr (x1 H,, (t)))((4)HH (H, ), we separately predict two batched history-dependent temporal and spatial distributions. Figure shows a high-level view of our proposed network. Out of the n + L events in the training sequences, are used as the history information and the inputs the encoder, whereas the remaining are the inputs to the decoder in the training phase, and our goal (consistent with Problem 1) is to predict their spatio-tempora distributions the test phase.
3.1 NETWORK ARCHITECTURE
As shown in Figure D the transformer consists of two main blocks known the encoder and the decoder. The encoder is fed with input sequence batches, where each sequence has events with their associated markers (4,, X,, M,). Using multi-head attention blocks (Vaswani al. (2017). we extract representations for all n events that capture history dependency. More precisely, via (5). we extract the hidden representation h,, for time and hx; for space, separately, which will be further used part of the inputs to the decoder block.
ti
hgg
Multi-variant Gauasian distribution
he hx
has
hidden states
distribution
(Zx) Pi
Dense
k Kx
Dense
Pt
Encoder
Layer
Decoder
Normalizing
Normalizing Flow Softsign
Normalizing RealNVP
Train
Normalizing
Ve XX
mutti-baa
Pi (x) (xi tı, hx,)
Valdariin
Pi
(t)
(tr; (ti; ht)
mutti- nuutt--heed
self seff-attention
self-attention
with mask
pi (xilt,, H()
Dense
Batch of outputs
pt (til
t. X,, M₁
41 X4, M
Inputs
Undertying
i= 1::
1=n+l:nt2
distribution
Figure 1: The framework of our proposed architecture. Our network is composed of an encoder, a decoder, and two parallel batched probabilistic layers each followed by bijective layers known normalizing flows. The input to the decoder is used to learn the output of the bijective layers using log-likelihood maximization in the training phase. In the phase, sequence of all zeros input the decoder and the output of the bijective layers are the predicted batched probability distributions, i.c. our proposed solution Problem 1.
The decoder contains two attention blocks that extract hidden representations of the output data (events n. + 1 through + T.) during the training phase. We denote these representation.] by {he, hx The first attention block in the decoder captures history dependency within the subsequence of events between time + 1 and n + L. We denote the outputs of this first block by 9: and 9x, since these are used queries for the second multi-head attention block the decoder; see Figure [ The hidden states (hr, hx; } from the encoder are used as both the key and the value inputs to the second attention block in the decoder layer. In this way, the history dependency between the input (from event 1 to n) and output (from event n + 1 to L) events is encoded in the output hidden representations of the decoder, i.c., in { hx 11 Despite the typical imple-
1l=nt1-
L
Notice that, as a convention, we tend to use subindex é (as in h.)) to refer to the events in the interval [1,... ... n and subindex (as in fv for events in the interval {n + 1,..., n + L). Also we use bold notation to show vector form of varibale, meaning h4; depicts dimensional hidden states, whereas he, stands for dimensions of higher than
4

---

Under review as a conference paper at ICLR 2023
mentation of the transformer decoder using look-ahead masks, due to multi-event objective remove this mask in the second attention block of the decoder. is worth mentioning that, during testing, the time and markers for the events + through n + L are imput zero, that infor- mation is used for prediction beyond H,,,, thus abiding by the constraint of Problem 1. Finally, normalizing, residual, and dense layers used in both the encoder and the decoder blocks for dimension matching and enhancing the training.
In our next step, we separately inject {he, lim't1 and {hx 18+1. into two separate batched prob- abilistic layers (Dürr et al.) 2020). In essence, for the time variable 4, we define a trainable map going from he, into the parameter of an exponential distribution. We denote this distribu- tion by Pi (())(
( .;; :,g)). Similarly, for the spatial variable Xt, define a trainable map going from (hx, (t) into the parameters of a multivariate Gaussian distribution. We denote this distribu-
tion by Pi (xx) (-; 42, hxx). We learned the parameters of these distributions independently for ev- ery E {n + I,...,n n + L.}. However, it should be noticed that the hidden states (h+,) n++ and (hx, [( 1 already encode spatio-temporal dependencies, so that the parameters learned for the probability distributions of different events already capture the correlations with other events across time. Also notice that the parameters of the spatial distribution depend the time instant & since we want to estimate the joint probability density over time and space (see Problem 1) as pt(1, x1 (H.)) = pr (xi it. He, )r (4/ H.,)
Although the probabilistic layers have a critical role in forming the underlying batched distributions, they can only take simple and tractable forms (in our case, exponential and multivariate Gaussian), which are not expressive enough to capture the true underlying distributions of the times and loca- tions of future events. Thus,
of bijective layers, known as normalizing flows (Kobyzev al.] 2020), are used to transform them into richer and more complex batched forms. As discussed in Section 2.2
use (6) to address the transformation of distributions pi p(*t) and p(²*) into more complex forms. We use the exponential and multivariate normal distributions as the base distri- butions pj pf(+)) and (xxx) because they have been used extensively parametric kernels to form the Hawkes intensity function in T (Ogata 1998 1988). Moreover, in Appendix AA.7 we have shown that the histograms depicting the space and time distributions in various datasets not too far from being a multivariate Gaussian distribution for space and an exponential for consecutive time inter- vals. In this sense, normalizing flows only need to learn to further adjust these distributions to the observed data. More precisely, we find continuous and invertible functions Fi and F22 such that, for
every I, these functions transform samples from (as) and pi (2x) (denoted by 24, and 2x;) into t and x2 as in
t = Fz()) Xi = (2xi))
(7)
Using (6) we can relate the known expressions of the base distributions with the distributions of t and Xt using the Jacobians of Fi and F2, respectively. We denote these learned distributions by pi
(t) (-; hx.) and pj ( hx In terms of parametric functional forms, we use a softsign bi-
jector for Fi and RealNVP bijector (Dinh et al. 2016) for Fg. To train these bijections, we seek to maximize the ligelikel of the true events {4, x1 x under the learned distributions
Pi (1) (-) (-,,), and pi
(x) (-: hx,). More details about the training of our network will be discussed in Section 4.3 and Appendix [..2] Going back to Problem 1, by maximizing the log-likelihood
network in end-to-end fashion such that (1) (t1;)e)) z (,,t((H(,) and
are training
Pi (xr; 4t, lxx)) s (x Hr,), ). where, we recall, Hr, contains the history of events up to (but not including) event l. From here we can obtain
(x)
(t x H.,) (x((1 Hy (tt
for 1,
L.
(8)
Finally, be recalling that during testing every event beyond is completed via zero padding, in (8) we are effectively obtaining a distribution for time and space of every event I in the range + to n + given H. Ho as we aimed for in Problem
Note that during the test phase we only feed the decoder with 4 0 when modeling pf (-; hv,); However, we use the true known time values li as inputs to the probablistic layers when modeling Pi (9x) (13 hxg)) ) in both train and test phases. More details are provided in Appendix A.2
5

---

Under review as a conference paper at ICLR 2023
3.2 ADDRESSING THE LIMITATIONS OF EXISTING APPROACHES
As previewed in Section I our architecture in Figure 1 was inspired by the need of addressing four shortcomings present in classical Hawkes processes and partially shared by more modern alterna- tives. Here, we explicitly discuss how the different constituent blocks in our architecture address these deficiencies. First, we do not depend on prespecified kernels model the probability densi- ties of time and location. Notice that, although we adopt exponential and Gaussian distributions for the latent variables 2,, and 7x, as further described in Appendix I these are later transformed via normalizing flows to better represent the observed data. Second, the attention mechanisms in both the encoder and decoder blocks of the transformer can learn heterogeneous effects between events across time. Third. the explicit forms of pi" (tt: ha,) and p¹ xxx) (xx; (x1; t,, hx,) obtained via normaliz- ing flows facilitate the fast sampling of future events via (8) without having to rely on integration as in (2). Fourth, the incorporation of the decoder block of the transformer enables multi-event prediction bypassing the need of sequential one-by-one event prediction. To be more precise, the use of our decoder block implies that, during training, we are incorporating the future information (he, hx, Ji=n+1 along with their dependence on the encoded history (he, hx; Yi= This makes our training procedure self-supervised rather than unsupervised, unlike all previous works in the area. In essence, during training. the network learns to fit the distribution of multiple future events jointly. Hence, during testing, we can zero-pad the input to the decoder and still obtain an estimation of the spatio-temporal probability densities of several events in the future.
4 EXPERIMENTS
4.1
DATA
We used a variety of synthetic and real-world datasets representing point processes with discrete events. We briefly introduce these datasets here; more details can be found in Appendix AA. In all datasets, we collected sequences of 500 events with an overlap of 498 events. We used n = 497 events in each sequence as the inputs and the final L events as the outputs that we want to predict during the testing phase. Train, validation, and test sets are formed according to the 80% - 14% - 6% split rule after shuffling the formed sequences.
South California Earthquakes. Earthquake events from 2008 to 2016 in South California (Ross) jtt al., 2019) of magnitude at least 2.5. The event description includes time, location in dimensions, magnitude and consecutive events' time intervals as event markers.
Citibike. Rental events from a bike sharing service in New York City (Amazonawstripdata The event description includes starting time of the biking, location in 2 dimensions, the biker's 's birth year, and consecutive rentals' time intervals as event markers.
Covid-19. Daily Covid-19 cases in different states of the United States (The-New York- Times). The event description includes day of catching Covid-19, 19, location in 2 dimensions, the number cases on that day, and consecutive events' time intervals as event markers.
Pinwheel. Hawkes pinwheel dataset introduced in Chen et al. (2020). Hawkes time instances were simulated using the thinning algorithm introduced in Ogata (1981) and assigned to cluster-based pinwheel distribution. For simplicity, we assigned the same magnitude to all spatial points sampled
from the formed distribution when forming the data sequences.
4.2 BASELINES
We consider two categories of baseline methods, namely, those that try to predict the time and those that try to predict the location of the next event. Time category models are used learn the intensity function from input times and we predict the expected time of multiple events in the future by sequentially utilizing (2) These models include the Hawkes process, self-correcting point process (Isham & Westcoti, 1979), and the homogeneous Poisson process (Pasupathy) 2010). Note that the sequential prediction of the future time-points for more than one event, in cases of the Hawkes and self-correcting intensities, relies a redefined history that incorporates the very last predicted event time in the previous step. We also use space models to sequentially learn the space distributions of multiple events in the future. In this regard, we use the conditional Gaussian mixture model (GMM) (Murphy) 20122, Bishop & Nasrabadi] 2006) to lear the space distribution, where the Gaussian kernel parameters learned from the historical events.

---

Under review as a conference paper at ICLR 2023
4.3 NETWORK TRAIN AND TEST
Training starts by passing batches of input and output sequences to the encoder and the decoder. respectively. The encoder outputs also passed to the decoder earlier explained in Section 3.1 We used inputs and outputs with all the existing markers for each dataset as mentioned in Section 4.1
list of the hyperparameters used in this work is shown in Table 3 in Appendix A.7 We define the loss as minimizing the negative log-likelihne -igliicl
12 I.
loss
Σ 1=n+1
log 1 (ti:
Σ
log
(xi: &1, h
1=
Thus, we are effectively learning the coefficients the transformer blocks - which affect the hidden representations he, and hx, and the coefficients in the probabilistic and bijective layers which parameterize p) and pj to maximize the likelihood of generating the observed data.
The baseline methods are trained encode the history of past events and sequentially predict the expected next event based on ( Therefore, in each step we compute the expected time t, and location & of event I and want these be close to the true values I and x, before updating the history for the next prediction step. To satisfy this condition, during the training phase, we add regularizers the negative log-lik likelihhod in
loss
log (t(
Σ Ix H A1
Σ 1mm+1-1
t -illl
X2
Σ 1m
|xx|
xx 112
(10)
where Pe(- Pe(-(Ht;) and Px( -1t, H1 are the learned probability densities for the bascline method at hand, and A and 22 encode the relative weights of the regularizers. To more precise, if a method only predicts the time, only predicts the location we use the second and fourth terms as the loss. For a fair comparison between the loss of our network log-likelihood for events n + through n + L for all the tested models. We train our network using the Adam optimizer with an initial learning rate of 0.001 and scheduled learning rate rule. We used droupout layers to reduce overfitting and ran 1000 epochs using batches of size 32
only the first and third terms in (10) the loss and if method
the baselines, during the testing phase report the negative
Table 1: Results representing the negative log likelihood for the events in the range n++ to n L for the learned distributions (less is better). + indicate the standard deviation of the loss among all test hatch sequences.
Earthquake
Citibike
Covid-19
Pinwheel
Homo-Poisson (p)"
6.91
6.83
6.324..0
6.82
Hawkes (t)
0.42+0.09
-0.53
-3.7940.01
-3.98
Self-correcting (p) (2)
NAN
1.95
4.6.0.02
2.03
Conditional-GMM (p(x)
4.88+0.29
3.05
3.48
4.38
Our network (4)
0.33
-6.279
-5.87
1.27
Our network (p)
2.03
-3.2840.14
2.18
1.57
Our network (p) (1,x))
2.36.10.38
-9.561...18
-3.7
2.84
4.4 RESULTS
Figure shows the predicted batched spatial and temporal distributions for the next three events on the South California Earthquake data (Ross et al. 2019) using proposed architecture. On the top right figure, we took the average of 1000 samples taken from the predicted batched history dependent time distributions to forecast the time of future events shown by the red stars. The linear
*Code available supplementary material and will he publicly released github if the paper is accepted.

---

Under review as a conference paper at ICLR 2023
300
340
1.0
18
30
Figure 2: Visualizations of the learned history-dependent batched spatial and temporal distributions for the last three events of an unseen earthquake sequence using proposed architecture. The red stars the top figures the simultaneous prediction of the true events' time, shown by green circles, where took an average between 1000 samples taken from the output predicted time distributions. The bottom figures depict the predicted batched spatial distribution associated with the last three events in 3 dimensions where the z-axis is along events" depth. The 2-d representation of the last three events' predicted spatial distributions mapped to the events' depth are shown the middle plots where except for the one the left, show the differences in the predicted distributions. The black stars the spatial distributions represent the last 50 events of the input sequence as the history information.
trend is shown emphasize that our network is capable of predicting time events with different in-between time-intervals. The bottom figures show the 3-dimensional representation of batched history-dependent multi-modal predicted densities along the main faults, prone earthquake occur- rence. In the 3-d visualization we show three slices along the z-axis, representing the lowest, highest, and the depth associated with the true future event, shown by the green circle. In the middle, we show the 2-dimensional density visualizations that are associated with the true event's depth shown in the 3-d figures below them. Note that, while the bottom figures represent the predicted densities, in the middle figures we have the predicted density on the left (i.e.. for the first predicted event) and the other two middle figures depict the consecutive predicted density differences for the two subse- quent events. This is done to highlight the performance of network in predicting simultancous but different multi-event densities. Each predicted space density is associated with the assigned events' predicted time shown above it. Despite seeing only slight differences between the predicted space densities, regions that are most expected to have the next event are still well-recognized. Note that while the same history is encoded for all the three future events, the differences between the predicted batched densities are highly influenced by the decoder layer which incorporates future information that was available during the training phase. Considering that consecutive outputs in most sequences occur very close to each other in time and space, the decoder layer is trained to represent the same behaviour in the test phase, predicting batched densities with slight but informa- tive differences. The high bias on the upper left of the space density maps is caused by the highly earthquake- prone regions/faults in South California.
Figure 3 on the other hand, represents the sequential prediction performance of the baseline models on the same dataset used in Figure 2 The Hawkes model and the conditional GMM model are used to learn the time and space distributions, respectively, where we used (2) to sequentially predict the time and location of the next event. Comparing this result with Figure we see the better perfor- mance of network in predicting the time and space distribution of future events carthquake data. Moreover, the conditional GMM model incapable of predicting a complex and well-defined density along fault lines representing earthquake- prone regions. More details on how perform
8

---

Under review as a conference paper at ICLR 2023
300
340
100
Figure 3: Same visualizations as Figure for the same data using Hawkes model for time, augmented with conditional-GMM for space. Here the prediction in time and space is performed sequentially using by updating the history each prediction step.
sequential prediction is given in Appendix A.5 The black stars on spatial density plots in Figures 2 a show the last 50 out of the 497 historical events that are used as the encoder inputs.
The results shown in Table for all the experiments indicate that, except for the Hawkes time pin- wheel synthetic dataset, network outperforms all the baseline models. Due very high accumulation associated with the sequential prediction using the conditional GMM model, we used the true rather than the predicted values at each prediction step update the history. More visual- ization results are given in Appendix A.4 Since the times associated with the pinwheel dataset are synthetically simulated from the Hawkes model, Hawkes is better candidate for forecasting the ex- pected time distribution for this dataset as shown in Table [ Among the given baseline models, the self-correcting and homogeneous Poisson are performing the worst, where the self-correcting with the inhibitory assumption behind history dependency never converged on the earthquake dataset. While all the provided baseline models assume certain parametric forms for the space or time dis- tributions, our proposed network is applicable to any discrete type dataset even in the absence of underlying physics or known models that can guide the right choice of a apatio-temporal kernel.
5 CONCLUSION
We have proposed a novel probabilistic approach by enhancing a transformer architecture to con- ditionally learn the spatio-temporal distribution of multiple stochastic discrete events. Our network combines the transformer model and normalizing flows augmented with batched probabilistic layers simultaneously learn the underlying distributions of multiple future events using self-supervised learning. The attention blocks of the transformer architecture assign score-basee history dependency among events, which can capture both excitatory and inhibitory behaviors while being parallel and fast to train. We show that our approach achieves state-of-ther of-tt--rtttttttt performance on different spatio- temporal datasets collected from a wide range of domains.
For future work, we are interested in using other available sources of data as proposed by the work of Okawa et al. (2022) such as GPS and beneath-earth images to propose a context-aware batched spatio-temporal distribution forecasting tool, enhancing the performance of our network on the stochastic point processes. Another interesting direction is to use the diffusion model (Song et al. 2020; Ho et al., 2020 Nichol & Dhariwal 2021) instead of normalizing flows generative models of learning the probability distributions. We hope that by adding gradual noise in the reverse diffusion denoising steps, we might better learn regions associated with events.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Amazonawstripdatt Citibike data in the united states, 2021. URL sttpp://ss3 aaa.ss....aaaaa........ com/ com/tripdata/index
AR Bansal and Y Ogata. non-stationary epidemic type aftershock sequence model for seismicity prior to the december 26, 2004 m 9.1 sumatra-andaman islands mega-earthquake. Journal of Geophysical Research: Solid Earth, 18(())616-662, 2013.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Feng Chen and Thomas Stindl. RHawkes: Renewal Hawkes Process, 2022. URL https:// cran. wttppp/jeet.ooo/pcccaaaeeeeeeeee
package version 1.0.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differ- ential equations. arXiv preprint arXiv: 07366, 2018.
Ricky TQ Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes.
arXiv preprint arXiv:201 1.04583, 2020.
Joshua V Dillon, lan Langmore. Dustin Tran, Eugene Brevdo, Srinivas Vasudevan, Dave Moore, Brian Patton, Alex Alemi, Matt Hoffmman, and Rif A Saurous. Tensorflow distributions. arXiv preprint arXiv:1711.10660 2017.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprint arXiv:1410.8514 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv
preprint arXiv: 008803, 2016.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Godez-Rodriguez, and Song- Recurrent marked temporal point processes: Embedding event history to vector. In P Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1555-1564, 2016.
Oliver Düm. Beate Sick, and Elvis Murina. Probabilistic Deep Learning: With Python, Keras and FensorFlow Probability. Manning Publications, 2020.
Beno Gutenberg and Charles F Richter. Frequency of earthquakes in california. Bulletin of the Seismological society of America, 34(4):185-188, 1944.
W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. 1970.
Agnes Helmstetter and Didier Sornette. Predictability in the epidemic-type aftershock sequence model of interacting triggered seismicity. Journal of Geophysical Research: Solid Earth, 108 (B10), 2003.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087. PMIR, 2018
Valerie Isham and Mark Westcott. A self-correcting point process. Stochastic processes and their applications, 8(3):335-347) 1979.
Priyank Jaini, Kira Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In International Conference Machine Learning, pp. 3009-3018, PMLR, 2019.
DP Kingma and M Welling. Auto-encoding variational bayes. in international conference learn- ing representations (iclr). 2014.
Leon Knopoff. The magnitude distribution of declustered carthquakes in southern california. Pro- ceedings of the National Academy of Sciences, 97(22):11880- 1884, 2000.
10

---

Under review as a conference paper at ICLR 2023
Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43 (11):3964-9979 2020.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data En- gineering, 2021.
Nazanin Mehrasa, Ruizhi Deng. Mohamed Osama Ahmed, Bo Chang, Jiawei He, Thibaut Durand, Marcus Brubaker, and Greg Mori. Point process flows. arXiv preprint 1rXiv:/910.00220 2019.
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multi- variate point process. Advances neural information processing systems, 30, 2017.
Sebastian Meyer, Johannes Elias, and Michael Höhle. A space-time conditional intensity model for invasive meningococcal disease occurrence. Biometrics, 68(2):607-616, 2012.
Kevin P Murphy. Machine learning: probabilistic perspective. MIT press, 2012.
Shyam Nandan, Guy Ouillon, Stefan Wiemer, and Didier Sornette. Objective estimation of spatially variable parameters of epidemic type aftershock sequence model: Application to California. Jour- nal of Geophysical Research: Solid Earth, 122(7):5118-514) 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models.
In International Conference on Machine Learning, pp. 8162-8171, PMLR, 2021.
Yosihiko Ogata. On lewis' simulation method for point processes. IEEE transactions on information theory, 27(1))23-31, 1981
Yosihiko Ogata. Statistical models for earthquake occurrences and residual analysis for point pro- cesses. Journal of the American Staristical association, 83(401):9 1988.
Yosihiko Ogata. Space-time point-process models for earthquake occurrences. Annals of the Insti- tute of Statistical Mathematics, 50(2):379 402, 1998.
Yosihiko Ogata and Jiancang Zhuang. Space-time etas models and an improved extension. Tectono- physics, 413(1-2):13-23, 2006.
Yosihiko Ogata, Ritsuko Matsu' ura, and Koichi Katsura. Fast likelihood computation of epidemic type sfiershock-sequence model. Geophysical research letters, 20(19):2143-211 2244241111111,, 1993.
Maya Okawa, Tomoharu Iwata, Yusuke Tanaka, Takeshi Kurashima, Hiroyuki Toda, and Hisashi Kashima. Context-aware spatio-temporal event prediction via convolutional hawkes processes. Machine Learning, pp. 1-22, 2022.
Tohru Ozaki. Maximum likelihood estimation of hawkes' self-exciting point processes. Annals of the Institute of Statistical Mathematics, 31(1):145-155 1979.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
Raghu Pasupathy. Generating homogeneous poisson processes. Wiley encyclopedia of operations research and management science, 2010.
Donald H Perkel, George L Gerstein, and George P Moore. Neuronal spike trains and stochastic point processes: li. simultaneous spike trains. Biophysical journal, 7(4):419 440, 1967.
Jakob Gulddahl Rasmussen. Temporal point processes: the conditional intensity function. Lecture Notes, Jan, 2011.
DA Rhoades. Estimation of the gutenberg-richter relation allowing for individual carthquake mag- nitude uncertainties. Tectonophysics, 258(1-4):71-83, 1996.
Zachary E Ross, Daniel T Trugman, Egill Hauksson, and Peter M Shearer. Searching for hidden earthquakes in southern California. Science, 364(6444):767-771, 2019.
11

---

Under review as a conference paper at ICLR 2023
Frederic Paik Schoenberg, Mare Hoffmann, and Ryan Harrigan. A recursive point process model for infectious diseases. Annals of the Institute of Statistical Mathematics, (tntattii,,,())))).. 7165):1271 2019.
Oleksandr Shchur, Marin Bilos, and Stephan Günnemann, Intensity- learning temporal point 12127, 2019.
processes. arXiv preprint
Donald L Snyder and Michael I Miller. Random point processes in time and space. Springer Science & Business Media, 2012.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprini arXiv:2010.02502. 2020.
Didier Sornette and Maximilian Werner. Constraints the size of the smallest triggering earth- quake from the epidemic-type aftershock sequence model, båth's law, and observed aftershock sequences. Journal of Geophysical Research: Solid Earth, 110(B8), 2005.
The-New York-ToTsee Coronavirus (covid-19) data in the united states, 2021. URL https://
github.
Stefano Tinti and Francesco Mulargia. Effects of magnitude uncertainties on estimating the param- eters in the gutenberg-richter frequency-magnien law. Bulletin of the Seismological Society of America, 75(6):681-116 1985.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: http:::/pprcceedass..eeu 7dee htteeeeeel1ooo50c11ccccc3333 4e Paper.
Shuai Xiao, Junchi Yan, Xiaokang Yang, Hongyuan Zha, and Stephen Chu. Modeling the intensity function of point process via recurrent neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Zihao Zhou, Xingyi Yang, Ryan Rossi, Handong Zhao, and Rose Yu. Neural point process for learning spatiotemporal event dynamics. Learning for Dynamics and Control Conference, pp- 777-789. PMLR, 2022.
Shixiang Zhu, Shuang Li, Zhigang Peng, and Yao Xie. Imitation learning of neural point processes. IEEE Transactions on Knowledge and Data Engineering, 2021.
Jiancang Zhuang. Long-term earthquake forecasts based on the epidemic-type aftershock sequence (ctas) model for short-term clustering. Research in Geophysics, 2(1))c8 c8, 2012.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In International conference on machine learning, pp. 11692-11702. PMLR, 2020.
A APPENDIX
A.1 FURTHER DETAILS NETWORK ARCHITECTURE
showed the usage of two individual batched probabilistic layers that sep- In+T. arately fed with (hg J and {(bx 41) 1=nt1 where {hu b are the outputs of the decoder block and 1a+L are the known time values. We denoted the outputs of these layers by Pt (.)) he,) and pl (2x) (-:t,, f, hx,) as the conditional exponential and multivariate Gaussian distribu- 1n+L
In Section 3.1
1n+L To further tions, respectively. whose parameters are learned from (h,, Ji=++1 and {(h,, tx) 1=n+1 expand the mathematical description of these layers' outputs we have
12

---

Under review as a conference paper at ICLR 2023
Pi (. (-; :h)) exp(-
1
31 B1
Pi (ex) (1;41, (x,))x = ((2*) Mane Eil) à exp( (2x) - 11) " 1 -m)),,
(11)
where Be is learned as a function of hep and {Ei, pr are learned as functions of { hx J. pi (+t) (-; :,,) and pi ((=))( (.: hxg) in (11 are individually followed by a softsign bijector Fj and
a RealNVP bijector F2 (Dinh et al. 2016), respectively, as two separate bijective layers to model
the desired batched conditional temporal and spatial distributions p)
(0) (-; (-,)), and pj x lxx))
Using 66), we have
( (. (1)) = (..n)
x (-5 hx,,),,,) pi (/)))) (-; (-: 4,, hx, [dde(((
(12)
Notice that the usage of the softsign bijective function Fi)) - W after the exponential prob- abilistic layer restricts the outputs between O and 1, i.e. F1 (0, - (0, 1). This is consistent with the fact that we always normalize our time inputs (t,, f2) to stay between 0 and 1, so that our time intervals also always stay between 0 and 1. Therefore, this bijective map (Dillon et al. 2017) will ensure that the outputs from the exponential probabilistic layer will stay in our desired range. Moreover, as shown in the time-interval histograms associated with all datasets in Figure 12 all fol- low an approximate exponential behavior. This indicates that we do not require much flexibility in choosing the temporal bijective layer. Also, note that since time is 1 -ddimensional. it is not applicable to utilize the well-known bijective functions, such as the RealNVP (Dinh et al., (016). NICE Dinh et al. (2014), the Masked Autoregressive normalizing flow (Papamakarios et al. 2017). to model the batched temporal distribution, mainly because these flow architectures are designed for dimensions higher than 1.
A.2 DETAILS ON NETWORK TRAIN AND TEST
As pointed out carlier, the training and testing phases of our network come with slight differences which we better clarify in this section. As shown in Figure [ during training, both batches of inputs {(t,,,, M, 1==1 and outputs {tt, x,, MM) are fed into the encoder and the decoder, respec- tively. Using (9) the goal is to learn a parametric description of a batched probability distribution
that seeks to explain future events in only time and space, meaning (2) and (x) respectively. The
reason behind considering only these markers (time and space) is that these are the main markers
covered by the original Hawkes process shown in (T. Learning batched distributions associated with other markers may need the contribution of models other than Hawkes. As an example, the Bitenberg-Richter (GR) frequency-magninee Knopoti 2000 Tinti & Mulargia 1985 Rhoades 1996) is needed to be considered when modeling/ learning the batched magnitude distribution PM, associated with future earthquake events. Also no- tice that, as indicated by the Hawkes intensity function in (I). using other markers M as historical information has no contradiction with learning only the batched spatio-temporal distributions.
law in earth science (Gutenberg & Richter 1944
On the other hand, during testing, we only use the input batches {(4,, xi, M to feed the en- coder, whereas batch of zeros as outputs, meaning {(t,, x,, MM) M = O, are used to feed the decoder. We expect that in the testing phase, the trained network could predict the batched spatio- temporal distributions p) and y.x²) associated with the future L events, by only relying on the historical information {(t,, x,, M,) This might cause confusion about passing zero queries to the second attention layer of the decoder, due to using zero inputs. However, this can be addressed by relying on the biases associated with the decoder block, which were tuned in the training phase. Another question that might be raised due to feeding zero inputs to the decoder in the testing phase, is how the decoder can distinguish between different inputs. Notice that we used positional em- bedding (PE) proposed by Vaswani al. (2017) in both the training and testing phases. However, the decoder is also fed by the output from the encoder, which is computed during testing using the true history [f2, X,, .M,)) M, that we use for prediction. Consequently, the decoder can distinguish between different event histories via the output of the decoder in addition to having the preserved positions using PF.
13

---

Under review as a conference paper at ICLR 2023
As illustrated in Figure 1 designed our model to jointly predict both time and location distri- butions by first drawing time sample & from the learned pi and then fixing and using as an input to the batched spatial probabilistic layer, to model/predict pf (-; :f hx)). ). Since baselines are for either time prediction or location prediction, we decided to compare them with our method
in such way that we simultaneously learn/predict both Pi ( (. hv.) and p, (-; ;44 hxx): where for (x) (-; tt, hx,) we assume that the time t is known and given. In this way, the error is comparable with those that predict solely location since we controlling for the error that could be introduced by a wrong time prediction 4
Pi
A.3 ABLATION STUDY
To further leverage the importance of each layer/block used in our proposed model, in addition to what we described in Sections 3.I a.Iandd3.2 we also performed two ablation studies on our network using all the available datasets.
Table 2: Results representing the negative log-likelihood for events range n + 1 to n for learned distributions including ablation studies (less is better). + indicates the standard deviation of the loss among all test batch sequences.
Earthquake
Citibike
Covid-19
Pinwheel
Our network (p)
(4)
0.31
-6.279 10.123
-5.87
1.27 1..26
Our network (p) (x))
2.03
-3.28
2.18
1.57+0.09
Our network (p) (xx))
2.36
-9.56
13.7
2.84+00.19
History independent (p)
(5)
1.26
6.3
-5.910.22
1.27 10.27
History independent (p)
2.49
-3.27
1.6440.05
7. 7.06+0.03
History independent (p) (1xx)
3.75
-..58
-4.2640.00
8.334
No decoder (p)
(1)
0.
6.07640.1
-5.41+0.12
1. 1.295
No decoder (p) (x))
2.21.1..3 10.3
-3.0
2.2
0.61 10.06
No decoder (p) (1)0))
2.58
-9.17
-3.21
1.9
A.3.1 HISTORY INDEPENDENCY
In this study, we assume that p\ and Pi are completely independent of the history, meaning that
(t)
have pi (-344, 4 hx,) (xx) (-;4) and pi (6) (-; (,;h,,) (t) For this matter we provide batches of
zero inputs to the encoder {tt,, x,, x, H 0 in both the training and testing phase while keeping
all other steps exactly before. Results for this study on different datasets are given in Table 2
South California Earthquakes: As shown by Table 2 removing history dependency from our network causes worse performance of multi-event forecasting in both time and space for this dataset. This is in line with the well-know
sequence (ETAS) model (Ogata 1998) suggesting long-term and long-range dependencies in both time and space between seismic events.
Citibike: Removing history dependency on this data, did not highly affect the results. Our interpre- tation of this dataset is that many other factors other than the previous bikers' markers included in this dataset might influence the time location of riding a bike. Therefore, removing history de- pendency does not highly influence forecasting the time or location of multiple bikers in the future when no other influencing factors considered.
Covid-19:
We realized that removing the history dependency did not affect time distribution fore- casting, but instead enhanced predicting space distributions. Despite the disease being very tagious, the space distribution forecasting results are not surprising at all. This is due to the fact that we taking into account the long-range history dependency over many different states of the
14

---

Under review as a conference paper at ICLR 2023
United States. Unless consider trips between states, the patients' time and location in one state might even provide false information influencing the chances of others getting the disease in another state. Therefore, removing this information might even enhance predicting space distributions. Our interpretation of very little changes in forecasting time distribution in the case of having no historical information is that covid-19 is among the fastest disease to eatch, therefore not very long-term de- pendency in time is needed to forecast the time of catching the disease. We guess that the short-term dependency captured by the decoder during training enough to prediet when the next person will catch the disease.
Pinwheel: As shown on Figure|I] the sequences of pinwheel events are formed in a clock wise way. Therefore. historical positional and temporal information should have big affect on predicting the location of next events the future. This in line with the results shown in Table 2 indicating how removing historical information worsens spatial multi-event distribution forecasting. On the other hand, despite using synthetic Hawkes time (Chen & Stind! (022) associated with pinwheel events, we saw no change in time forecasting when removing history dependency. For this outcome, we infer that the thining process (Ogata et al. 1993) used to extract the times (Chen & Stindl 2022) assigned to the events does not consider long-term term and long-raud range dependency. Therefore, the short- term history dependency that exists among the output events x(), ) that is captured during training by the decoder layer, suffices predicting multiple events' time distributions with no reliance on further look onto the history.
A.3.2 No DECODER
Here, no longer feed {(4,, X1, M)) the decoder during the training phase. This is sim- ilar to the test phase of our model where we input {[ti, x1, M = 0 to the already trained 1
network. However, in this study we input {(t;, x1, M)) ing phases. Results in Table indicate that except for the pinwheels dataset's spatial distribution forecasting, removing the decoder does not significantly change the prediction results. Nonethe- less, results associated with the full network are slightly better. As earlier mentioned in Section 3.2 during training, the network learns to fit the distribution of multiple future events jointly, in a self- supervised setting. Hence, during testing, zero-padding the input to the decoder still obtains an estimation of the spatio-temporal probability densities of several events in the future. This happens due to the reliance of the network on the historical input data, as well as the learned bias associated with the decoder layer. However, we accept that the learned bias might only provide little history dependency information between output events, especially in the case of small number of events to be predicted (L = 3 in our case). Since the setup when having no decoder differs from the main setup of our proposed network only in terms of the learned bias associated with the decoder layer, do not expect to see much difference in the results. However, we suspect that having larger output sequences will highlight this difference between the two settings. In that case, could consider a larger distance in time and space among the output events, which results in differently tuning the decoder layers' bias during the training phase.
0 during both training and test-
In the case of the pinwheel dataset, we realized that removing the decoder enhanced the multi-event spatial distribution forecasting results. This is when the batched temporal distribution forecasting results are negatively affected, In line with the outcomes of the previous ablation study, we inferred that the pinwheel events are influenced massively by the long-term history dependency in space. Therefore, removing the short-term dependency among the output events will allow the events to fo- cus more on long-range effects of the historical data. However, the results associated with predicting the temporal distributions got worse. As expected from the outcome of the study in Appendix 1.3.1 this might be due to the importance of short-term dependency between the time of the events, mod- eled by the thining process (Ogata et al. 1993).
A.3.3 CONCLUSION ON ABLATION STUDY
Through these two studies and what we discussed in the main body of this paper, we showed the critical role of each layer forming our network. Mainly, we observed that each layer is designed for specific task, driven by the physics behind the formation of the used dataset. Unlike other methods that solely rely on the Hawkes process, our network is designed to fully consider any sort of short- or long-term history dependencies desired by the physics behind data.
15

---

Under review as a conference paper at ICLR 2023
In general, from these studies, we con- cluded that the performance of our pro- posed network is highly data-driven. Predicting batched distributions asso- ciated with datasets that are formed based on both long- and short-range his- tory dependencies, such the earth- quake dataset, demands the usage of both the encoder and decoder blocks of our model.
nodecoder
training loss
network
historyindependent
re
200
400
When events are more influenced by
nodecoder
short-term dependencies, or there's s a
validation loss
10
network
historyindependent
lack of the presence of other factors in-
fluencing the events' occurrence, the en-
coder might negatively affect the pre-
diction results. This was shown the
outcome of the first study on the Covid-
300
19 dataset. The reverse applies events dependent on long-range history depen- dencies, such the pinwheel and earth- quake datasets. There might be cases where removing the encoder has no in- fluence on the prediction results, due to no significant history dependency, such as the Citibike dataset.
number of epochs
Figure 4: Comparing the performance of the proposed network with two ablation studies applied the earthquake data. Both re- moving decoder and the history dependency worsen the predi- tion results. This highlights the importance of both long- and short- range history dependencies among scismic events.
On the other hand, capturing the short-term dependencies, mainly done by the decoder layer, is important when events are also influenced by these dependencies. This conclusion is based on the outcomes of applying the second ablation study to many datasets, such as the earthquake dataset, Citibike, Covid-19, and the temporal events forming the Pinwheel dataset.
Since all datasets used in this work, except for the pinwheel dataset, represent real-world phenom- ena, we believe no single model can be used to learn/predict distributions associated with them. Therefore, it is critical to have a network that can be driven by the data's underlying physics, whilst being capable simultaneous multi-event prediction.
A.4 MORE RESULTS AND ANALYSIS
To show the performance our network on synthetic Hawkes time simulated data vs. the baseline model, we used the pinwheel dataset introduced in Section 4.1 where the events are distributed with less bias towards a specific region. As depicted Figure on top, not only the history de- pendent batched spatial distributions for the next three events well predicted using our network, but also regions prone to events occurring further away in time, shown in a clock-wise pattern in Figure [I] are well predicted. On the other hand, the conditional GMM model was not successful in extracting the complex distribution associated with next events' occurrences. Note that using the conditional GMM model to perform sequential next event prediction, at each step, we incorporate the true event shown by the green circle, rather than the newly predicted event shown by the red star, redefine the history for next step prediction. This due to the high error accumulation caused by using the sequentially predicted events.
Figure T shows the predicted time of the next three events applying our proposed network and the time Hawkes model to the simulated Hawkes pinwheel dataset. While the result on the top figure are indicative of simultaneous multiple future time event prediction, the bottom figures depict utilizing (2 scquentially while updating the history via the most recently predicted event. These results are in agreement with Table D indicating the outperformance of using the Hawkes model for this dataset. As mentioned earlier, this is due to the fact that the time assigned are simulated by performing thinning algorithm on a time Hawkes process. As mentioned earlier, the expected times extracted by applying our network to this and all other datasets are measured
the synthetic Hawkes dataset
16

---

Under review as a conference paper at ICLR 2023
30
00040
0,0400
20-
24-
0:022
210035
98
110930
Our Network
55
0.0255
0.0025
0.015
0.0015
50-
60
0:0010
0,0013
92
09036
0.0005
-10
-1.
0.0000
0,000
1.000
0.10
20
24
0.00
conditional GMM
15-
9.50
3.5
008
15
0086
10
300
1.0
05
0,500
0.5
000
0.5
0.06
0:375
0.5
0.00
004
005
-05
0.325
0.03
0.08
-15
-0.
00
-01 08
-3n -10 0.
Figure 5: Visualizations of the Iearned history dependent batched spatial distributions for the last three events of simulated Hawkes pinwheel sequence using proposed architecture conditional GMM. The first figure the left hand side depicts the predicted distribution associated with the first event, where the middle right
figures only depict consecutive differences hetween the predicted densities. The green circle shows the true event that is expected to occur. The black stars represent the last 50 events of the input sequence as history information. The red star on bottom figures shows the sequentially predicted next event using (2) which needs be injected back the history for the next step prediction. Due high accumulation, we instead use the green circle to update history on bottom figures.
by taking the average of 1000 samples taken from the batched history dependent predicted time distributions.
852.5
Our Network
800
True time Predicted time Inear
600
852.0
851.5
851.0
100
200
300
400
500
0.0
0.5
1.0
1.5
2.0
851.75
Hawkes model
800
True time Predicted time Inear
851.25
851.00
100
200
300
400
500
0.0
0.5
1.0
1.5
2.0
true (green circles) vs. predicted stars) time of the last three events an earthquake sequence using our model on top and the time Hawkes model on the bottom. The results on top are generated simultaneously, whereas the bottom results are predicted sequentially via updating history based on the most recently predicted event time. The linear line is indicative of predicting same time-intervals hetween events.
Figure
17

---

Under review as a conference paper at ICLR 2023
-1
Figure 7: Visualizations of the learned history dependent hatched spatial distributions for the last three events of unseen carthquake sequence using proposed architecture. The bottom figures depict the predicted batched spatial distribution associated with the three events in 3 dimensions where the events" depth. The 2-d representation events' depth are shown on the top plots where except for the one on the right, we show the differences in the predicted distributions. The black stars the spatial distributions represent the last 50 events of the input sequence the history information, where in this figure the history events very much aligned along the main fault in South California.
is along
the last three events' predicted spatial distributions mapped to
Figure 8: Same visualization Figure 7 on another earthquake sequence length 500 where the history events, shown by black stars, are more distributed along all fault lines rather than just one.
A.5 BASELINES
We use baselines to separately model the time and spatial events distributions, conditioned on the history. Earlier in (10). we showed that the loss defined for the benchmarks have some differences with the loss we defined for our proposed architecture in the training phase. As already discussed the main body of the paper, the baseline models only provide the expected next event using (2). which unlike our proposed method, gives the predicted event explicitly rather than distribution. For further prediction in the future, the already predicted event has to incorporated in the history, which will accordingly update the intensity function X(t, H2,) well the space distribution p(x; t., H,,) to be used for further predictions. To have precise updated history, we need our predicted points be close the true values as possible. Therefore, expand (10) provide
18

---

Under review as a conference paper at ICLR 2023
decoderr Layer
Attention heads 1-3
44:00
Encoder-decoder Layer
Attention heads 1-3
30
Encoder-decoder Layer
Attention heads 1-3
Figure 9: Visualization of the first 3 out of 6 encoder-decodee attention layers, where we show 3 out of 6 heads per layer. These figures are the results of applying represent historical events, whereas the y axis show the three expected events in the future. The colors show how weak or strong the past events have an influence on the future events' occurrence.
proposed network to the pinwheel dataset. The axis
further detail as
n
n
loss
lo (t. (t.)
Σ
log px (xilt,, He,)
(13)
negative log-likelihood encoding history data
log (tt(i(H))))
Σ
log Px Px th, Ha)
b=++1
1=nt1
negative log. likelihood of predicted events canditioned the updated history
n+L
a+L
Σ 11 L X2
Σ
X - 811l2,
1=++1
1=++1
restricting the prodicted values match the true values
where H., is the updated history including the most recent predicted event (f, x
Conditional Gaussian mixture model: We implemented two forms for the Conditional Gaussian mixture models the baselines to learn the spatial distributions p(x,/4;, Hr, )i=in mentioned in
19

---

Under review as a conference paper at ICLR 2023
Section 4.2 In the first method compute the pairwise Gaussian log-likelihood among all x; and X, associated with events 1 - I I n, as well as the pairwise log-timedecay At.j. We minimize the loss associated with adding up the pairwise Gaussian log-likelihood and oog-timedecay to learn the history dependent Gaussian mixture model.
The second method is the general K-cluster Gaussian mixture model introduced in Murphy (0022) Bishop & Nasrabadi (2006), where the learned parameters are functions of the historical events markers. Having mixture of K Gaussian densities
1 exp( exp((=((xi, uu) 2 "xie px)), 15kSK (14) |E
11 K (2x)
where ny represents the number of samples associated with cluster k, we learn A4 and Σg from {[4, Xx, M,)) as the time, space and the extra marker associated with all events assigned to cluster k. In this case, pxxilu, Σ) in (14), denotes the conditional spatial probability associated with event X|=++1-
Although the second method is capable of conditionally being dependent any other markers associated with events, the first method had better performance on all datasets. Therefore, this work we only reported the results based the first method, where both implementations given in our publicly available code.
A.6 PREPROCESSING DATA SEQUENCES
South California Earthquakes. We earthquake discrete events distributed South of Cali- fornia with latitude, longitude and depth. The distribution of all events along the main faults as well four sequences of 500-Iength size, projected on 2-d map of South California faults, are shown in Figure [] Since we don't t work with in-between distances, there is no need to convert to the Cartesian coordinate system. After forming sequences as mentioned in Section 4.1
we normalize
Figure 10: Visualization of distribution of discrete earthquake events on the South California main faults cov- ering events from 2008 to 2016.
the space and magnitude of each sequence with respect to the mean and standard deviation gath- ered from all events. The bias caused by normalization towards more earthquake prone regions as previously shown in Figure 2 remained to be cased in later work as discussed in Section [
Citibike. We use the data from April to August of 2019. No preprocessing was performed for this dataset. This resulted in 3500 training sequences, 900 validation sequences, and 600 test sequences of same length.
Covid-19. We use the data from March of 2020 to May of 2022. No preprocessing performed for this dataset. This resulted in 1400 training sequences, 360 validation sequences, and 240 test sequences of same length.
20

---

Under review as a conference paper at ICLR 2023
Pinwheel. We sample from multivariate process with 15 clusters, each containing 150 events. Due to the space Hawkes synthetic dataset not being available, we used the same spatial distribution provided in Chen et al. (2020) with the assigned Hawkes timings using the RHawkes package Chen & Stind] (002)). Since we have sequences of 500 events, each sequence will contain few number of clusters shown in Figure [T] This resulted in 613 training sequences, 157 validation sequences, and 106 test sequences of same length.
Pinwheel
Eatthquake
Figure 11: Top: Pinwheel synthetic dataset where simulated Hawkes times are assigned to the events of clusters in clockwise way. These timings simulated using thinning algorithm Ogata 1981). Bottom: South
California earthquakes distributed over the main faults.
A.7 PATEMRRRRAM
We have set the number of attention layers in both the encoder and the decoder 6, whereas each layer contains 6 heads individually. The reason behind these choices are what proposed by the main transformer network proposed in NLP Vaswani al. (2017), where they suggest to choose between 6 to attention heads and layers. Before inputting any of the sequences to the encoder and the decoder, we first used Multi-layer perceptron (MLP) layers with ELU activation functions, to embed to 64 dimensional space. The latent representations h,, hx, depicted in Figure I are in a 64 dimensional space, whereas we used another set of MLP layers with ELU activation functions to transfer back he,, and hx, to the same dimension f,, and x₁ before injecting them into the probabilistic layers. This was due to the use of bijective layers (normalizing flows) following the probabilistic layers only being capable of mapping the same dimensional space.
Table 3: Hyper- parammtarresters and layer-types used our model both training and test.
parameter
value
parameter
value
Epochs
1k
Batch-size
32
Sequence-length
500
learning-rate (starting)
1e-3
Input-length
497
Output-length
3
Attention layers
Attention heads
6
Marker-embedding dim
64
Dropout-rate
0. -~~~ 0.2
A1 (for L1 norm)
0.1
Ag (for L2 norm)
0.1
probabilistic layer
type
bijective layer
type
time space
exponential multi-variate Gaussian
time
softsign RealNVP
space
21

---

Under review as a conference paper at ICLR 2023
Earthquake
300
Time
Latitude
Longitude
Depth
Extra
Time intervals
111000
L20000
Citibike
X.300
40/0
200600
Covid19
Pinwheel
Time
Latitude
Longitude
Extra
Time intervals
Figure 12: Histograms of different markers associated with different datasets used in this work. The "extra marker's associated with Earthquake, Citibike, Covid-19, and pinwheel datasets the magnitude, birth year, number cases, and all-one intensity, respectively.
we used multi-variate Gaussian and exponential probabilistic layers for space and time, respectively as considered by the Hawkes model. The markers histograms shown by Figure 12 as well indicate the usefulness of these choices for our probabilistic layers. We also tried modeling the time distribution using Gaussian probabilistic layers which led to bad results.
As mentioned earlier in Section 3.1
For the bijective flow layers we tried using Real et al. (2016) and the Masked Autoregres- sive flows Papamakarios et al. (2017) for the space distributions, and we tried softplus and softsign flows for time. Our results indicate that the RealNVP and softplus are capable of better modeling the expected space and time batched distributions. Note that both the RealNVP and Masked Au- toregressive flows are only applicable to data with dimension of higher than 1, and therefore are not good candidates for transforming the time batched distributions.
To overcome the overfitting when training our model, we used dropout layers with rates listed as (0.1, 0.2). For the earthquake and Hawkes pinwheel synthetie data, both 0.1 and 0. 0.15 were
22

---

Under review as a conference paper at ICLR 2023
good options whereas for the Citibike and Covid-19 19 we had to increase the rate to 0.2 to reduce the overfitting.
All the mentioned hyperparameters can be set differently in network using argparsing once the code publicly available. Our network can be applied to any discrete form data with variable dimension assigned markers for future use.
*Code is available supplementary material and will he publicly released github if the paper is accepted.
23


Reviews:
[{"review_id": "yi2TWL8H3I", "reviewer": "Reviewer_q56W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper presents a transformer-based architecture to handle multi-event forecasting in spatio-temporal domains. The proposed architecture allows for multiple events forecast by formulating problem as a self-supervised problem. ", "summary_of_the_review": "In general, the model and problem proposed in the paper are quite novel and of great importance. Authors have done a great job of explaining each individual parts of the model and its specific utility in solving the drawbacks from the previous work. Hence, I think this work deserves to be published. ", "strength_and_weaknesses": "Strengths\n* Architecture proposed is quite novel in sense of being applied to multiple event prediction task\n* All different parts of the model are well motivated and solve a purpose instead of a giant network being embedded for sake of doing deep learning. \n* Problem being attacked is important and has been something people in filed have mostly sidelined and been happy with doing sequential short predcitions. \n* well written and easy to follow paper. \n\nWeakness\n* The baselines in experiments consider only Hawkes alternatives. I would encourage authors to at least present a comparison using other methods in SI. \n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ozlzw8qOLsD", "reviewer": "Reviewer_boMt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of simultaneously predicting times and location of *multiple* (fixed number of) future events with a neural spatio-temporal point process model.\nThe proposed model is based on an encoder-decoder transformer architecture, and the distribution of the future events is modeled by ", "summary_of_the_review": "Several important aspects of the proposed approach need to be described more clearly. A more thorough comparison to the baselines and an ablation study would be really beneficial to justify the utility of the proposed model.", "strength_and_weaknesses": "**Strengths:**\n- The paper introduces an interesting variation of the prediction problem for spatio-temporal event data, where the goal is to predict multiple events simultaneously. A thorough analysis of the limitations of existing approaches that could be used in this task is provided. This is an important task and it would be important for practical applications of spatio-temporal point process models.\n- The visualizations in section 4 are well done and provide an insight into the model's performance.\n\n**Weaknesses:**\n\n1. My main concern is the clarity regarding the proposed model architecture & the training procedure. Here are some questions that need to be clarified in the updated version:\n    - During training the decoder takes the observed $\\\\{(t_l, x_l)\\\\}_\\{l=n+1\\}^\\{n+L\\}$ as input, and tries to predict the exact same values  $\\\\{(t_l, x_l)\\\\}_\\{l=n+1\\}^\\{n+L\\}$  as output. Intuitively, it seems that the decoder should learn to perfectly predict these values / the log-likelihood should tend to infinity as the model trains. Why is this not the case?\n    - It is mentioned that at test time the inputs to the decoder are replaced with all zeros.\n        - In this case, how can the decoder distinguish between different inputs? Is some positional encoding used?\n        - If $t_l = 0$ is fed into the model, this means that the distribution $p(x_l | t_l)$ will effectively become independent of $t_l$. What is the purpose of modeling the conditional distribution at training time, if at test time the time and location are always independent?\n    -  The softsign function $f(z) = \\frac{z}{|z| + 1}$ has domain/codomain $f: (-\\infty, \\infty) \\to (-1, 1)$. Restricting to positive values only (as $z_{t_l}$ comes from the exponential distribution), we get a function $f: (0, \\infty) \\to (0, 1)$. However, we need both the domain and the codomain here to be $(0, \\infty)$ since we're modeling the inter-event times or the arrival times. In other words, the used function is not a bijector so it does not define a valid normalizing flow transformation. Is there a mistake in my reasoning?\n        - As a side note, the softsign function has no learnable parameters, so it's unclear how exactly it increases the flexibility of the distribution $p(t_l)$.\n\n2. Baselines: Currently, the experiments in section 4 only include very simple baseline models (homogenous Poisson, self-exciting/self-correcting processes for time & gaussian mixture for locations). There exist other approaches that definitely could be compared against:\n\n    - Neural spatio-temporal point process model from Chen et al. (https://arxiv.org/abs/2011.04583)\n    - Other neural TPP models for predicting the event time (many of which are cited in the paper)\n\n    Figure 1 also suggests that the proposed model takes into account the magnitudes $M_l$ of the events, but it seems that none of the baselines have access to this information. If the magnitudes are indeed taken into account, it's important to also compare to the [ETAS model](https://www.ism.ac.jp/editsec/aism/pdf/050_2_0379.pdf) from seismology (using the power-law kernel and, very importantly, taking into account the contribution of the magnitudes on productivity)\n\n3. Lack of ablations: The proposed model consists of multiple components, but no experiments are performed to confirm that each of these components is actually beneficial. To demonstrate that the transformer architecture leads to a meaningful improvement, it would be necessary to compare with:\n    - $p(x_l)$ and $p(t)$ completely independent of the history, but parametrized exactly like in the paper (i.e., exponential + softsign for $t_l$, gaussian + real NVP for $x_l$)\n    - Encoder only (without feeding $t_l$, $x_l$, $M_l$ to the decoder, i.e. just mapping $h_{t_i}$ and $h_{x_i}$ to the parameters of $p(t_l)$ and $p(x_l)$). Intuitively, it seems that this option will work just as well as the model with the decoder, since in the full model the decoder anyway doesn't receive any input at prediction time anyway.\n\n\n**Minor comments**\n- The assumption that the future events are conditionally independent given the history (as defined in Problem 1) seems a bit restrictive: for example, if the next earthquake has a large magnitude, then the follow-up events will happen in quick succession. Lifting the conditional independence assumption could be an interesting avenue to explore.\n- Potentially missing references\n    - Neural spatio-temporal point processes: https://arxiv.org/abs/1906.05467,\n    - Modeling TPPs with normalizing flows: https://arxiv.org/abs/1910.08281 and https://arxiv.org/abs/1909.12127", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n-W5ZmTCPGy", "reviewer": "Reviewer_mvAP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\n\nThis paper proposes a neural architecture, combining the transformer model and normalizing flows, for multi-event forecasting of the time and location of discrete events. A variety of real spatio-temporal datasets are used to validate the state-of-the-art performance of the proposed method.\n", "summary_of_the_review": "Overall this paper is well-written. This work proposes to combine transformer and normalizing flows to predict future multi-events under the spatio-temporal Hawkes models. Numerical results are provided to demonstrate the good performance in real data examples. However, considering that both transformer and normalizing flows are not new and they have been already been used in point processes, the contribution of this work seems incremental. Moreover, more clarification on numerical results is also needed.\n", "strength_and_weaknesses": "Strength:\n- A new architecture consists of transformer and normalizing flows, to capture the spatio-temporal distribution of multiple stochastic discrete events jointly, conditioning on historical events.\n- extensive numerical experiments to demonstrate the performance of the proposed method.\n\nWeaknesses:\n- the contribution is incremental, both the transformer and normalizing flows have been used for point processes in the literature, e.g, [1] \n- the numerical experiments are not very clearly explained, and more clarification is needed\n\n[1] Mehrasa, N., Deng, R., Ahmed, M. O., Chang, B., He, J., Durand, T., ... & Mori, G. (2019). Point process flows. arXiv preprint arXiv:1910.08281.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Q7HYSRe9Bd", "reviewer": "Reviewer_yeaA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper leverages the sequence-to-sequence (seq2seq) model in NLP for multi-event prediction in the Hawkes process. The authors call the task \nthe batched prediction. The authors employ\nthe encoder-decoder architecture that incorporates spatial-temporal aspects. They leverage the transformer architecture, where two embedding vectors capture the temporal event history and spatial dependency. They seem to have used an interesting normalization flow model to handle the temporal development of probability distribution.", "summary_of_the_review": "This paper is unique in its sharp contrast between readable narratives and hard-to-follow descriptions of the probabilistic model. Because of the very definition of the intensity function (the probability density of the next event), multi-event prediction generally requires Monte Carlo integration. The situation is similar to that of multi-ahead prediction in forecasting. See, e.g., https://ieeexplore.ieee.org/document/8594894 \n\nThe proposed framework seems to have solved that issue by introducing the normalized flow approach combined with the seq2seq transformer model. I wish I could give a much better rating, but it is hard for me to endorse the approach due to the massive lack of formal descriptions.  \n", "strength_and_weaknesses": "**Strengths**\n\n- Proposes a new multi-event prediction framework. \n- Seems to have address the probabilistic nature (i.e., properties that require Monte Carlo or its variant) of multi-event prediction.\n- Incorporated a new normalized flow approach.\n \n\n**Weakness**\n\n- Some of the claims in the introduction seem misleading. The classical Hawkes process has a decay function, which introduces a significant distinction over the event instances. I'm not sure if \"(1) imposes similar behavior across all events over time meaning that every preceding event has the same form of impact on future events' occurrence\" is a scientifically correct statement.\n\n-  The proposed probabilistic models are not clearly defined. Most of the descriptions are very high-level and qualitative. As a result, it is hard to verify methodological novelty and reproduce/extend their results. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject