On the Predictive Accuracy of Neural Temporal Point Process Models for Continuous-time Event Data
Authors: Tanguy Bosser
Souhaib Ben Taieb
Venue: Accepted by TMLR ()

Abstract: Temporal Point Processes (TPPs) serve as the standard mathematical framework for modeling asynchronous event sequences in continuous time. However, classical TPP models are often constrained by strong assumptions, limiting their ability to capture complex real-world event dynamics. To overcome this limitation, researchers have proposed Neural TPPs, which leverage neural network parametrizations to offer more flexible and efficient modeling. While recent studies demonstrate the effectiveness of Neural TPPs, they often lack a unified setup, relying on different baselines, datasets, and experimental configurations. This makes it challenging to identify the key factors driving improvements in predictive accuracy, hindering research progress. To bridge this gap, we present a comprehensive large-scale experimental study that systematically evaluates the predictive accuracy of state-of-the-art neural TPP models. Our study encompasses multiple real-world and synthetic event sequence datasets, following a carefully designed unified setup. We thoroughly investigate the influence of major architectural components such as event encoding, history encoder, and decoder parametrization on both time and mark prediction tasks. Additionally, we delve into the less explored area of probabilistic calibration for neural TPP models. By analyzing our results, we draw insightful conclusions regarding the significance of history size and the impact of architectural components on predictive accuracy. Furthermore, we shed light on the miscalibration of mark distributions in neural TPP models. Our study aims to provide valuable insights into the performance and characteristics of neural TPP models, contributing to a better understanding of their strengths and limitations.

Published in Transactions Machine Learning Research (06/2023)
On the Predictive Accuracy of Neural Temporal Point Process Models for Continuous-time Event Data
Tanguy Bosser
tanguy, www.errummnsssc....oo
Department of Computer Science University of Mons
Souhaib Ben Taieb
souhaib
Department of Computer Science University of Mons
Reviewed OpenReview: https: https:// o/perevvee. net/ forum? id= 30SISBQPrM
Abstract
Temporal Point Processes (TPPs) serve as the standard mathematical framework for modeling asynchronous event sequences in continuous time. However, classical TPP models are often constrained by strong assumptions, limiting their ability to capture complex real-world event dynamics. To overcome this limitation, re- searchers have proposed Neural TPPs, which leverage neural network parametriza- tions to offer more flexible and efficient modeling, While recent studies demonstrate the effectivenoss of Neural TPPs, they often lack a unified setup, relying on differ- ent baselines, datasets, and experimental configurations. This makes challenging to identify the key factors driving improvements predictive accuracy, hinder- ing research progress. To bridge this gap, we present a comprehensive large-scale experimental study that systematically evaluates the predictive accuracy of state- of-the-art neural TPP models. Our study encompasses multiple real-world and synthetic event sequence datasets, following a carcfully designed unified setup. We thoroughly investigate the influence of major architectural components such as event encoding, history encoder, and decoder parametrization on both time and mark prediction tasks. Additionally, we delve into the less explored area of prob- abilistic calibration for neural TPP models. By analyzing our results, we draw insightful conclusions regarding the significance of history size and the impact of architectural components predictive accuracy. Furthermore, the miscalibration of mark distributions in neural TPP models. Our study aims to provide valuable insights into the performance and characteristics of neural TPP models, contributing to a better understanding of their strengths and limitations.
shed light
1 Introduction
From human social activity natural phenomena, the evolution of system of interest can often be characterized by sequence of discrcte events occurring at irregular time intervals. Online shop-

---

Published in Transactions Machine Learning Research (06/2023)
ping activity (Cai et al. 2018 earthquake occurrences (Ogata 1998). measurement of electronic health records (Wang ct al. 2016), and users activity on social media (Farajtabar et al. 2015) are typical examples where such sequences are frequently encountered. Given sequence of observed historical events, crucial challenge in numerous applications is to predict the timing of future events. Additionally, in cases where events are assigned a label, referred to as marks, it is necessary to also predict the type of event that is likely to occur. It is reasonable to assume that events are interdependent and that the future evolution of a system directly influenced by past occurrences. For example, an individual might be inclined to purchase a particular item at a specific time on an e-commerce platform solely because a previous purchase made necessary. Hence, modeling the intricate dynamics of event occurrences becomes crucial in predicting future events based on past observations.
Drawing upon solid theoretical foundations, the framework of Temporal Point Processes (TPPs) (Daley & Vere-Jones 2007) has established itself as a suitable choice for modeling these sequences of asynchronous and timo-dependent data. A TPP is fully characterized by conditional intensity function, which provides the instantancous unit rate of event arrivals based on the process history [Rasmussen] 2018). For event sequence modeling in a variety of domains, including finance (Hawkes 2018), crime [Egesdal et al. 2010), or epidemiology (Rizoiu et al., 2018) large number of earlier works examined classical parametric forms of TPPs, such as the Hawkes process (Hawkes 1971
However, these classical parameterizations have faced criticism due to their limited flexibility in capturing complex event dynamics (Mei & Eisner 2016 To address this, doep learning techniques have been introduced into the TPP literature, enabling more flexible models capable of capturing more complex temporal dependencies. Examples include recurrent neural networks (RNNs) (Mei & Eisner 2016; [hhchur et al. 2019) and self-attention mechanisms (Zhang et al. 2020; Zuo et al.,
Enguchard et al.) 2020). Since its introduction by Du et al. (2016), the field of neural TPPs has experienced rapid development, with the emergence of numerous novel architectures ures and applications (Shchur et al. 2021
2020
Given a sequence of events, neural TPP models typically involve a combination of three main architectural components: 1) an event encoder, creating fixed-sized embedding for each event in the sequence. 2) history encoder, generating sunmary of the history from past events' emboddings, and 3) decoder parametrizing function that fully characterizes the distribution of future events' arrival times and marks. Among other possibilities, improvements with respect to existing baselines are obtained by proposing alternatives to either of these components. For instance, one can replace an RNN-based history encoder with a self-attentive one, or choose to parametrize a certain TPP function that leads to useful properties, such as reduced computational costs or closed-form sampling- However, as pointed out by [Shchur et al. (202T), " "new architectures often change all these components at once, which makes it hard to pinpoint the source of empirical gains".. Moreover, the baselines against which newly proposed architecture is compared, as well as the datasets employed and the experimental setups, often differ from paper to paper, which renders a fair comparison even harder.
Additionally, modeling marked TPPs from data is a challenging problem from a statistical perspec- tive in the sense that it requires joint modeling of discrete distributions over marks and contimuous distributions over time. In practice, a model's ability to estimate the joint distribution of future arrival times and marks often solely evaluated on its performance with respect to the negative log-likelihood (NLL). However, reporting a single NLL value encompasses the contributions of both arrival time and mark distributions, making it hard to evaluate model performance with respect to
2

---

Published in Transactions Machine Learning Research (06/2023)
each prediction task separately. Moreover, while the NLL enables the comparison of different base- lines, it provides limited understanding of the distribution shapes and the allocation of probability mass across their respective domains. Furthermore, while probabilistic calibration is desirable property that any competent or ideal predictive distribution should possess (Dheur & Ben Taieb 2023 Guo et al., (017), the assessment of calibration in TPP models has been overlooked by the neural TPP community, both for the predictive distributions of time and marks.
In this paper, our objective to address the aforementioned concerns by presenting the following contributions:
We perform a large-scale experimental study to assess the prodictive accuracy of statc-of- the-art neural TPP models on 15 real-world event. sequence datasets in a carefully designed unified setup. Our study also includes classical parametric TPP models as well as synthetic datasets. In particular, we study the influence of each major architectural component (event encoding, history encoder, decoder parametrization) for both time and mark prediction tasks. To the best of our knowledge, this is the largest comprehensive study for neural TPPs to date. Our study is fully reproducible and implemented in a common code base
We assess the probabilistic calibration of neural TPP models, both for the time and mark predictive distributions. To this end, we employ standard metrics and tools borrowed from the forecasting literature, namely the probabilistic calibration error and reliability diagrams. Probabilistic calibration is a desirable property that any competent or ideal predictive distribution should possess. Yet, to the best of our knowledge, this has been generally overlooked by the neural TPP community.
Among other findings, we found that neural TPP models often do not fully leverage the complete information contained in all historical events. In fact, relying solely on a subset of the most recent observed occurrences can yield comparable performance to encoding the entire historical context. Furthermore, we demonstrate the high scnsitivity of various decoder parameterizations to the event. encoder, highlighting the significant gains pre- dictive accuracy that can be achieved through appropriate selection. In addition, while the distribution of arrival times is generally well-calibrated, our rescarch reveals that classical parametric baselines exhibit better calibration of mark distributions compared to neural TPP models. Lastly, our study shows that several commonly used event sequence datasets within the TPP literature may not be suitable for accurately benchmarking neural TPP baselines.
2 Background and Notations
Marked Temporal Point Processes are stochastic processes whose realizations consist in a sequence of n discrete events S = {(t,, ki), (t., k,)) observed within a fixed window [0, 7] with > 0. Each event E = (l,, k;) in the sequence represents an arrival time 4 for the 7th event, satisfying t1 ... t, T, and associated with mark h; belonging to the mark space K. The mark space K. can either be discrete, such as K. {1,.... K) (Du et al. 2016), or continuous, Lee. K = R. Note that within this definition, the number of events itself is random variable.
https://github.comm http////aaggyyaaaee////t.t.ttoo.
3

---

Published in Transactions Machine Learning Research (06/2023)
In the case of an unmarked scenario, events are solely characterized by their arrival times, i.e. = f........).. Without loss of generality, we will primarily focus on the marked setting for the remainder of this section, exclusively considering univariate discrete marks throughout the paper. A realization of marked TPP can also be represented using a counting process N(t) = Σk=1 (t), where N&(t) = H(4,, k;) E S/4 < t)] is the number of events with mark k that have occurred prior to time
Let 6;-1 = (4:-1, 1;) be the last observed event. The occurrence of the next event & in (f--1, oo can be characterized by the conditional joint distribution of arrival-times and marks f(t, k(H)), where H, {(tz, k2) S t2 < t denotes the observed process' history. The conditional joint dis- tribution be decomposed as f(t, k(H,) f(t)H))p(kile He), where f(t)He) corresponds to the conditional density of time while pkk]t, H4) is the probability of observing mark k, conditional on both and He. Given the conditional cumulative distribution of arrival-times F(4HH), a marked TPP can be equivalently characterized by the mark-wise conditional intensity function (MCIF) Au (t),,,,, i.c:
(x(1))))) ((t,
(1)
F())))))
Furthermore, for > 11-1, we have Rasmussen 2018
f(t, k k(s))
((t(H))p(klt, H.) 1 F((
(2)
E (t + dt) N2(O11H t & [1--..8]
(3)
dt
where X(t[7)) = = (t((())) the ground conditional intensity function of the process (GCIF).
In essence, the MCTF gives the expected instantaneous occurrence rate of event of mark k E K. Furthermore, as heuristic interpretation, the MCIF can be seen the instantaneous "probability" of observing the next event of mark l in an infinitesimal interval around t, conditional on the past of the process up to but not including l:
dt P[ts E [t,t di), k|H4,. tu & [t..1.
(4)
In the following, we will employ the notation '*' Daley & Vere-Jones (2007) to indicate dependence on Hi, i.e. (g(t) = As (t)HH)) From the MCIF, we can define the cumulative MCIF,
M - L" Ag(s)ds,
(5)
allowing us to define the conditional joint densily time and mark k;
f" X Σ AZ(1))
(6)
The proof of 66 can be found in Appendix L Additionally, note that a realization of a marked
TPP can be cquivalently represented by the sequence {(T1, k1), (Tn, k,)} where 71 = f1 and 7; = I; 4-1, for i> 1, are the inter-arrioal limes associated to the events. We will use both notations
interchangeably throughout the paper. Lastly, all expressions will be defined for 4 S t, unless stated otherwisc.
4

---

Published in Transactions Machine Learning Research (06/2023)
Parametrization of TPP models and Learning. Defining a marked TPP model usually involves specifying parametric form for any of the functions in 2 5 or 66. provided that the chosen parametrization defines valid joint distribution of arrival times and marks, (*, (). In this regard, a distinct set of requirements need to be fulfilled depending on the chosen function. We consider exclusively the setting of non-terminating TPP, meaning that a future event will arrive eventually with probability one. For a non-tert non-terminating TPP, if one decides to parametrize the MCTF, then the following conditions must be satisfied Rasmussen 2018):
Rt: (t(t) 2 0.
R lim J. Ng(s)ds = cc.
Alternatively, parametrizing the conditional joint density implics:
Rf: f(t,k) 0.
R/: S. f(t
Specifying parametrization of the cumulative MCIF involves four constraints Enguehard et al. 20201:
R Ag(t) >
RA: lim Ag(t) oc.
1-000
Rº: A JM (s)dds O.
R dAz(t)/tt 0.
where conditions Ri and R result from (t(t) 2 O, while R directly follows R2. In the setting of terminating TPP, where there is a probability æ that the process stops after the last observed event, 122 R2 and Rg are no longer satisfied.
common example of parametrization is the homogeneous Poisson process (Daley & Vere-Jones De et al. 2019) that defines constant MCIF:
2007
(t) = 2 0,
(7)
and hence implicitly assumes an exponential ial distribution with rate = k= for the inter-arrival times T:
(T) = exp(-HT).
Another example is the well-known Hawkes process Hawkes 1971 the MCIF to account for the influence of previous events on the process' dynamics:
Liniger 2009 which defines
K
(t(1) = μl +
Σ Σ
Φx (9--()) 4;),
(9)
#=1 ((AA)))))))
where &u RR + R+ is the so-called triggering kernel. For instance, choosing the exponential kernel (t - tx) = as exp( - (t) t4 2 0] with 0g, > and BE. > 0 allows to explicitly model self-excitation dynamics, for which the occurrence of an event increases the probability of observing future events, in positive, additive and exponentially decaying fashion.
Let 0 be the of learnable parameters for a valid parametrization of marked TPP function. As example, for the Hawkes process in 9 we have 0 {11, a, B3, where μ E RKK and a, K
5

---

Published in Transactions Machine Learning Research (06/2023)
The most common approach to learning is achieved by maximum likelihood estimation, i.e. by minimizing the negative log-likelihood (NLL). Given paramctric form of () 0), f* k; 0) or (t; 0) for all & E K, and sequence S of n events observed on [0. T7, the NLL objective writes:
L(0:S)= =- Σ
log Xi, (4::0) - A (f:: + (T;
(10)
==1
[log f (t,; 0) + log (ki 0) + (T;
(11)
where the term AT Σx= Ag(T;0) accounts for the fact that no event has been observed in the interval (Ln, T).
Prediction tasks with marked TPPs. The expression 11 presented above highlights that learning marked TPP model from an event sequence involves two main estimation tasks: (T1) estimating the conditional density of arrival times f (t), and (T2) estimating the conditional dis- tribution of marks p" (k))t). Once we have estimates for (t) and p" (ki]t), we can effectively address queries such as: When is the next event likely to occur? What will be the type of the nexl event, given that occurs at a certain time t? How long until on event of type k occurs?
Additionally, by utilizing f (t: (t: and p" (klt; 0), we can compute estimate for the next expected arrival time (T3) as:
i= Elt
(t;
(12)
1--1
and estimate the mark of the event at time t (T4) as:
Ag 0) argmax (t(t;
k = argmax (kltt; = argmax
(13)
REK (1;
KEK
REK.
In this paper, our focus will be exclusively on T1, which corresponds to the lime prediction task, as well as T2 and T4, which are referred to as the mark prediction tasks. We will not generate point estimates for the next expected arrival time (T3) since most of the considered models do not allow for estimating (12) in closed form. To maintain clarity in notation, we will omit the explicit dependency of the parametrizations O throughout the remainder of the paper.
3 Neural Temporal Point Processes
While simple parametric forms of marked TPPs, such as the Hawkes process, have been used to model event sequences in variety of real-world scenarios, they have been criticized for their lack of flexibility. For instance, events may inhibit the occurrences of future events rather than excite them, or their influence might not be strictly additive. While previous works proposed modifications of the Hawkes process to encompass such complementary effects, e.g. excitation and inhibition (Chen et al. 2019 Costa et al.) 2020 Duval et al. 2022), others relied on the expressiveness of neural networks to capture more complex event dynamics Mei & Eisner (2016). The resulting framework, called Neural TPPs, defines models that can be characterized by three major components (Shchur
et al. 2021 Recall that S fer,... e, where e, (tz, kx). A neural TPP model involves:
6

---

Published in Transactions Machine Learning Research (06/2023)
1. An event encoder which, for each et S, generates fixed size embedding e e Ri;
2. A history encoder, which for each 6 E S, generates a fixed size history embedding h4 E Rd from past event representations (el,..,e-1))
3. A decoder, which given query time and its associated embedding et for t--1 St t, parametrizes function characterizing the TPP (i.e. X()),, (1),, (*, (1,):)) using et, and/or h4
The general modeling pipeline is shown in Figure 1 In the following, we will give more details about each of its major components.
3.1
Event encoding
The task of event encoding consists in generating representation E Rd for cach event in S, to be fed to the history encoder, and ually to the decoder, al a later stage. This lask essentially boils down to finding an embedding ef E Rd for the conter-Jarrvee time 4, and an embedding e E Rd for the associated mark k; The event embodding e, finally obtained through some combination of et and e e.g. via concatenation:
E;
(14)
Encoding the (inter-)arrival times. A straightforward approach to obtain ef is to select the raw inter-arrival times as time embeddings e = T or their logarithms ef = log т (Omi et al.
2019 Shchur al. 2019 Mei & Eisner 2016 Du al. 2016). Inspired by the positional encoding 2017) and their extension to temporal data (Kazemi
of Transformer architectures (Vaswani al.
al. 2019). other works exploit more expressive representations by encoding the arrival times vectors of sinusoidals (Enguehard et al. 2020):
d
e =
+ sin (azts) @ cos (ayti), 1=0
(15)
2i where og x 1000 * and @ is the concatenation operator. Variants of sinusoidal encoding can also be found in Zhang et al. (2020) and Zuo et al. (2020). Alternatively, learnable embeddings can be obtained by feeding the inter-arrival times fully connected layer (FC) (Enguchard al. 2020
e = FC,),,
(16)
where FC : R Rd
Encoding the mark: When marks are available, the most common approach to obtain the mark
embeddings is achieved by specifying a learnable embedding matrix E* E Rd x (Du et al.] 2016
Mei & Eisner 2016 Shchur et al. 2019 Zhang al., 2020 Zuo al. 2020 Enguehard
al.
2020 Given the one-hot encoding k; E {0, 1), of mark k; its embedding e E R. is retrieved
e =
(17)
NNote that e, used to designate event, while the bold notation e, is used for the representation.
7

---

Published in Transactions Machine Learning Research (06/2023)
p" (ka
t1 A1
1gg A2
fg, kg
e22
e3
DEC(e' h4)
A
B C D [A,
K
s(-)
g(-)
(T(
h h2 h3 h4
Figure 1: General workflow of neural paramctrizations for marked TPPs. Events inter-arrival times тj and marks k; are encoded into a vector ei, which is in turn used to generate the history embeddings h, through some form of rego-seeeesiive mechanisms g(.) (e.g. GRU). A query-time embodding et and the history embeddings are then fed the decoder to estimate the conditional density (T++1) and conditional distribution p" (k4411/441) of the next event.
3.2 History encoding
The general principle behind history encoding in the context of neural TPP approaches is to struct fixed-size embedding h, E Rd for event e; from its sequence of past events representations }, using some form of auto-regressive mechanism or set aggregator. Naturally, the main goal when constructing h, is to capture as closely as possible relevant patterns in the process history, order to accurately estimate the distributions arrival times and marks of future events.
Recurrent architectures. natural choice to handle ordered sequences of tokens is achieved LSTM GRU (Shchur et al. 2019 Du et al. 2016 Mei & Eisner 2016). In this context, starting from initial state h1 initialized randomly, the history embedding h; of an event e, is constructed by sequentially updating the history embeddings at the previous time steps by using the next event's representation:
by employing form of recurrent architecture, such
h2 ==g(h,,e)))
h; ei-1),
(18)
where h is the initial state and g Rd Rd. refers to the update function of the chosen recurrent layer.
Self-attention encoders. As an alternative to recurrent architectures, the self-attention mecha- nism of Transformers Vaswani et al. 2017) computes the h,'s 's for each independently as follows

---

Published in Transactions Machine Learning Research (06/2023)
ZZhang al. 2020 Zuo al. 2020
Enguehard al. 2020):
h, SA(q, K,, Vi) - W208 h + +b,) + b2,
(19)
q K V
h. Softmax
(20)
where SA(.) refers to the self-attention mechanism between a query vector q: a key matrix K and a value matrix V, defined as
qi = Wo K, = Wx(ee,.... V, Wylei,... e,-l)
(21)
where Wo. W E xdd, Wy E Rdaxd, W2, W1 E Rdxdd, b1, by E Rdh and øR the ReLU activation function. In essence, the history embedding h; is therefore constructed as a weighted sum of the representations of the past events, where the attention weights are obtained by measuring similarity score (e.g. dot product) between the projection of e;-1, and the projections of 01-1)
As pointed out by Zuo et al. (2020), recurrent architectures may fail to capture long-term depen- dencies and are difficult to train due to vanishing and exploding gradients. Moreover, due to their inherently sequential nature, recurrett models forbid parallel processing and are usually trained using truncated backpropagation through time, which only returns an approximation of the true gradients. Conversely, architectures based on a self-attention mechanism can compute the history embeddings h; for each i in parallel, leading improved computational efficiency. However, since each h; depends
all events preceding et, computing such embeddings for all L events of quence scales in Ol2)) time for architectures based on a self-attentin mechanism, while it scales in O(L) time for their recurrent counterparts (Shchur et al. 2021
3.3 Decoders
As mentioned in Section 2 fully characterizing marked TPP can be achieved by parametrixing either of the functions (t), (t) or (kk,,),, as any can be uniquely retrieved from the others. While being mathematically cquivalent, a specific choice among these functions Icads to particular advantages, challenges, and constraints, which will be discussed below. Nonetheless, given the encoding query event e e= (t, k) with i > and history embedding h,, the parametrization is almost systematically carried out by neural network. We will consider the following state-of- the-art decoders in our study:
The Exponential Constant decoder (EC) (Upadhyay al. 2018 MCIF between two events using a feed-forward network on h4 as in 22
parametrizes a constant
The MLP decoder (MLP/MC) (Enguehard et al. 2020 parametrizes the MCIF as in 26
using a feed-forward network on e and h4-
The Self- Delf-tittntiio decoder (SA/MC) (Enguehard et al. in (28h by letting e attend to all (hi..... h
2020 parametrizes the MCIF as
The Neural Hawkes decoder (NH) (Mei & Eisner 2016 parametrizes the MCIF as in 34 using a set of contimuous-time LSTM equations.

---

Published in Transactions Machine Learning Research (06/2023)
The RMTPP decoder (RMTPP) (Du al., 2016) separately parametrizes the GCIF as in (30,, and mark distribution independent of time given the history in (31
The LogNormMix decoder (LNM) (Shchur et al. 2019) separately parametrizes the con- ditional density of inter-arrival times as a mixture log-normals as (44) and the mark distribution similarly to RMTPP.
The LogNorm decoder (LN) is defined similarly to LNM, but with single mixture com- ponent.
The FullyNN decoder (FNN) (Omi et al 2019 parametrizes the cumulative MCIF in using a feod-forward network on e and h,-
52
The Cumulative Self-Attention decoder (SA/CM) (Enguchard et al. 2020) parametrizes the cumulative MCIF in (56) by letting attend to all (hi:-.,,/:
A Hawkes decoder (Hawkes) with exponential kernels, which parametrizes the MCIF as in
A simple Poisson decoder (Poisson), which parametrizes constant MCIF as in
In the following, we will describe and discuss cach of the above docoders, except the Poisson and Hawkes decoders which have already been introduced in Section
EC decoder. The MCIF is assumed to be constant between two events and is parametrized using a feed-forward network on the history embedding, i.c.
x²() = X os.. (w) R Wh:; bi) + be),
(22)
where W E Rd xdy and 6 E Rd,, and we Rd and by are mark-specific weights and biases, respectively. osS. is mark-specific Softplus activation function:
5x )
(s = sslog (1+exp
_
(23)
with 8% E IR ensuring R.. The MCIF being indopendent of time between two events, its cumulative is given by
(t(() (t (t))
(24)
and the distribution of inter-arrival times is exponential with rate A ER= A
=K=
(())= exp(-X'r).
(25)
MLP /MC decoder. Although the EC decoder can capture more general patterns than the Hawkes decoder, it does not allow evolving dynamics between consecutive events. To eircumvent this limitation, the MLP MCC /C decoder takes as inpul the concatenation of e (which is a function of t) and h. In contrast to the EC decoder, the MCIF can vary between consecutive events:
(t) 14 +os (w) (W W (W, e²1 b.) + bb).
(26)
10

---

Published in Transactions Machine Learning Research (06/2023)
where μ4 R. ensures that R2 is met. The Softplus activation forbids the computation of the cumulative MCIF in closed form, requiring numerical integration techniques, such Monte Carlo, to approximate its expression:
t- 1-44-1 n2
AZ
(27)
T1g
where 8j 1( and n, is the number of Monte Carlo samples.
SA/MC decoder. The MCIF is parametrized by re-employing the same set of equations (0 with the only difference being that the query vector is constructed using e'. By doing so, this allows the query event, e to attend to previous event. representations in Hi:
x²(1) po + to
(w) ( K;, V.) + bb),
(28)
q = Woe', K, Wx (xihi,....hb) Wv = V; Vi[hi,.... h+],
(29)
where We E R4x4, WK Rd, and Wv Rdxxd, d, being the output dimension of the attention mechanism. Similarly to the MLP/MC decoder, the cumulative MCIF must be approximated by mumerical integration techniques, c.g. as in 27
RMTPP decoder. Instead of specifying the MCIF, the RMTPP decoder separately parametrizes the GCIF (t) and the mark distribution p" (k) as:
(1) = exp w 4-11) + (w²) h; + 11)
(30)
(k( Softmax(W"") h: h2:+00 +b
(31)
where w' E R,, wh E Rda E R. 4 E Rxdd, and b E R2 The exponential transformation in (30 along with the positivity of w', ensure that R, and RS are met. By assuming the distribution the time given the history of the process, the RMTPP decoder
marks to be independent makes a strong simplifying assumption, which has been criticized in previous works (Enguehard et al. 2020). However, the exponential in 30) allows to directly compute the cumulative GCIF in closed form:
(1) we (1 + (w²")Th; + exp((w")TTh; 1h;++1) b))
(32)
Moreover, as pointed out by Shehur et al. (2019), the RMTPP decoder defines a Gompertz distri- bution Wienke 2010) on the arrival times, with a density given by
(T) Bn exp (BT - n exp(Br) + n),
(33)
with shape exp((w) (w+b)))+ and scale 3 = w².
NH decoder. The MCIF is parametrized using a fully-connected layer on a history embedding (t) that is allowed to vary between consecutive events, i.e:
(t) Js.k (ww)) hn(t))
(34)
In contrast to classical discrete-time LSTM, which would only update the history embedding h when the event at time t; is observed, contimuous-time LSTM allows the history embedding to
11

---

Published in Transactions Machine Learning Research (06/2023)
exponentially decay in the interval [t:-1, ti], making it a function of time. Once (t,, k2) observed, the contimuous-time LSTM will apply discrete update to (t,),), using a modified set of discrete- Specifically, the parametrization of the MCIF can be summarized
time LSTM equations 37 43
as
(t) = o, @ (2c()))(t)) -1), c(t) = c + (c - c.Jepp( 6,(t - -111))
(35)
(36)
i ss W + U,h;-1 b,).
(37)
C; f. ( 2,
+
(41)
S (W pki + U,h;-1 by),
(38)
é S: c-1 + + 00
(42)
= 20s W k U,h;-1(t)) (t;) + b.)
1, (39)
6; 0s (aak + Uph;-1 (t:) + ba),
= -ss (W Wo + (14, b.),
(40)
(43)
where k E [0 is the one-hot encoding of mark ki-1, wi Rd W. Ws, W., W.. Wa E RdakK, U. Uy, U., U., Ua E Rd.xd,, and b, by, b.. b., ba E Rdh as: is the Sigmoid activation function, and as is the unmarked formulation of the Softplus activation, i.e. as(x) = s log + exp (z/s)) with s E R,. Here also, one must rely on numerical integration techniques to estimate the cunulative MCIF.
LNM decoder. The conditional density of inter-arrival times is parametrized as mixture of log-normal distributions, i.::
M Σ
(log Mm )2
(T()=
Pm
exp
(44)
TOm v 2m
20
where Pm = Softmax (W,h; + bp).,. corresponds to the probability that 7; was generated by the th mixture component, while pmm = (W,h; + by)m; om exp(WWor + bz)m are the mean and standard deviation of the m" mixture component, respectively. Wp. W,,, W E Mxd, and bp: by.. b, E RMM MM being the number of mixture components. Defined similarly to equation (31 the mark distribution p² (*k is assumed be conditionally independent of the inter-arrival times given the history of the process. Although not available in closed-form, the cumulative distribution of mixture of log-normale can be approximated with high precision (Abramowitz & Stegun
(aaamaxx,,h;;
1965
Σ erf log --)m) } )}
(T(T)=
(45)
Om 2
where erf(z) = So - ds is the Gaussian error function. From (T),,, we can retrieve the cumulative GCIF
A'(1== = -log (1-
(46)
FNN decoder. A major bottleneck of models parametrizing the MCIF resides in its cunulative not always being available in closed-form, thus requiring expensive numerical integration techniques.
An clegant way of addressing this challenge is to directly parametrize the cumulative MCIF, from
12

---

Published in Transactions Machine Learning Research (06/2023)
which (t) can be easily retrieved through differentiation. The original definition of Fully involved parameterizing A² (t) using fully-connected network that operated on both the inter- arrival times and the history embeddings. In the context of marked setting, this can be expressed as follows:
(1) = JsSk (w (tanh(w"(tt (t- 4-11) + hh + +b + bk),
(47)
where each weight is constrained to be positive, ensuring condition R and R Lc. wy E R din
w' E IR4- Wh E Rdnxd E rdt and by R.. However, as pointed out by Shchur et al. 2019 equation fails to satisty Rd:
Ai((i-1) = os (w) (tanh (ataanh((",, + b.) + bx) > O,
(48)
which in tur yields F*(r = 0) > 0. In other terms, the original FNN model attributes non-zero probability mass to null inter-arrival times. The original formulation also fails to satisfy Rg due to the saturation of the tanh activation function:
din
lim (t) = os 8-00
Σ
wkid + by < c0.
(49)
To prevent both these shortcomings, Enguehard et al. (2020) proposed to replace the tanh activation function with a Gumbel-Softplur activation:
(GS = [1- (1 axeex(())))
[[ (z))
(50)
with o% E R+. The Gumbel-Softplus activation function being non-saturating (i.c. lim ( = oo), using it as a replacement for the lanh activation function in 52 indeed satisfies -0000
Rg. Finally, RA can be satisfied by defining (t)
(1) (1) Gj(t-1),
(51)
- oss ( ( (Wie' wit,; + bi) bu),
(52)
which defines the generalized corrected version of FullyNN. Note that (1) takes encoding of a query time e' with W E Rd. However, to ensure that R remains satisfied, el must be monotonic in its input. Therefore, when parametrizing a curulattve decoder, the temporal event encoding in [15 cannot be used, and the weights of encoding (16) must be constrained to be positive. From the cumulative MCTF, (g()) can finally be retrieved through differentiation:
input any
d x²(1) di AZ.)).
(53)
SA/CM decoder. The SA/CM decoder shares a similar set of equations as in (88 to parametrize the cumulative MCIF. However, alike FullyNN, several modifications of the latter are required ensure that the decoder meets the various constraints imposed by (g()) First, the Softmax activation is replaced by Sigmoid to satisfy R Then, the ReLU activation is replaced by Sofbbe-SSftplus, which prevents
& (AI(1)
(1)(1) 0.
(54)
di
dt
13

---

Published in Transactions Machine Learning Research (06/2023)
In other terms, a cumulative model defined with a ReLU activation is equivalent the EC decoder in 22 Enguchard et al., 2020). Morcover, given the saturation of the Sigmoid function, term pa (4) with PA E R4+ is added to the cumulative MCIF to satisty R². Note that including
this term is equivalent to adding le directly to the MCTF. Finally, the Cumulative Self-Attention decoder is given by
Aī(t) = (t Gi(ti-1), (1( = (1 1-1) + oss (w) (SA (q,K,, K,, V.) bk): SA(q, V,) = W20Gs ( + bi) + b2,
(55)
(56)
(57)
q K'K ddz
2, Sigmoid
V.
where WI, W2 E R4 Xd,, b1, b2 E R4, and where each entry of WK. Wv, Wo, we and by is now constrained to be positive. The query keys K,, and values V, are given in 29 As before, the MCIF can be retrieved through differentiation.
4 Related Work
Neural Temporal Point Processes. Simple parametric forms of TPP models, such as the self- exciting Hawkes process (Hawkes 1971), or the self-correcting process (Isham & Westcott 1979 rely on strong modeling assumptions, which inherently limits their fiexibility. To capture complex dynamics of real-world processes, the ML community eventually turned to the latest advances in neural modcling and proposed new TPP models based various neural-network architectures. For instance, Du et al. (2016) proposed to model the inter-arrival time distribution distribution with a history encoding based on a discrete-time RNN. On similar line of work, Mei & Eisner (2016) extended this idea. by parametrizing the MCIF using a modified contimuous-time RNN that allows the history to evolve between consecutive events. Given their huge success as sequence encoders in NI.P, Zuo et al. (2020) and [Zhang al. (2020) employed a self-attention mechanism encode the process history. Since differentiation is often easier to carry out than integration, FullyNN (Omi et al.)
Gompertz
2019) instead proposed to dircctly paramctrize the cummlative GCIF using leed-forward nelwork, avoiding expensive numerical integration techniques. Inspired by FullyNN, Engucharr et al. 2020) corrected and extended the FullyNN model to the marked case, and proposed a generic self-attention decoder that can be employed to parametrize the cumulative MCIF, but also the MCIF itself. Instead of modeling the (cumulative) MCIF, Xiao et al. (2017b) directly modeled the conditional density of inter-arrival times with a Gaussian distribution, while Shchur et al. (2019) relies on a mixture of log-normals whose parameters are obtained from an RNNN- encoded history. To further alleviale the strong assumption of independence of times and marks given the history, Waghmare et al. 2022 proposed to model conditional mixtures of log normals for each mark separately. Finally, Ben Taich (2022) proposed to parametrize the conditional quantile function using recurrent neural splines, enabling analytical sampling of inter-arrival times. Alter- natives to NLL optimization techniques for training neural TPPs include 3 VAF objectives [Boyd
et al.) 2020). CRPS (Ben Taieb 2022), reinforcement learning (De et al. 2019h, noise contrastive estimation Guoo al.]|2018 Mci et al.] 2020) and adversarial learning (Xiao ct al.] 2017a 2018 For surveys recent advances in neural TPP modeling, refer to Shehhr al. (2021) and Yan (2019)
14

---

Published in Transactions Machine Learning Research (06/2023)
Experimental studies. Most papers in the neural TPP literature mainly focused on proposing methodological improvements for modeling streams of event data, often inspired by contemporary advances in neural network representation learning. While these contributions are paramount to driving future progress in the field of TPP, few studies have been carried out to identify real sources of empirical gains across neural architectures and highlight future interesting research directions. The empirical study of Lin et al. (2021) is the closest to our work. While they also compared the impact of various history encoders and decoders, they did not discuss the influence of different event encoding mechanisms. However, we found that specific choices for this architectural component can lead drastic performance gains. Acditionally, they did not include simple parametric models to their baselines, such the Hawkes decoder. Considering these models in experimental study allows however to fairly evaluate the true gains brought by neural architectures. We also assess the calibration of neural and non-neural TPP models on the distribution of arrival times and marks, and our experiments are conducted across a wider range of real-world datasets. Finally, our results are supported by rigorous statistical tests. Lin et al. (2022) also conducted empirical comparisons the context of neural TPPs but their attention was focused on deep generative models which are orthogonal to our work.
5 Experimental study of neural TPP models
We carry out a large-scale experimental study to assess the predictive accuracy of state-of-the-art neural TPP models 15 real-world event sequence datascts in carefully designed unified setup. We also consider classical parametric TPP models, such as Hawkes and Poisson processes, well as synthetic Hawkes datasets. In particular, we study the influence of each major architectural component (event encoding, history encoder, decoder parametrization) for both time and mark prediction tasks. The next section summarizes the variations of the architectural components we have considered in our experimental study. Section 5.2 presents the datasets including summary statistics and pre-processing steps. Section 5.3 describes our experimental setup. Finally, Section 5.4 presents the evaluation metrics and statistical tools used to assess the accuracy of the considered models, and Section 5.5 describes the procedure employed to aggregate the results across multiple datasets.
5.1 Models
To case the understanding of the following sections, brief summary of the different architectures considered in our experiments is presented below. Table summarizes the correspondence between all variations of event encoding, history encoder, and decoder, and their mathematical expressions.
Event encoding mechanisms. For the event encoding mechanism, we consider the raw inter- arrival times T (TO), the logarithms of inter-arrival times (LTO), ) a temporal encoding of arrival times (TEM), and learnable encoding of inter-arrival times (LE). Additionally, we include all their variants resulting from the concatenation of the mark embeddings ex, i.e. CONCAT (TO and e²), LCONCAT (LTO an e*). TEMWL (TEM and e*), and LEWL (LE and e*).
History encoders. To encode process' history, we employ GRU, and self-attention mech- anism (SA). Additionally, we consider constant history encoder (CONS), which systematically outputs h = ld. Note that a decoder equipped with this encoder parametrizes a function in-
15

---

Published in Transactions Machine Learning Research (06/2023)
Table 1: Summary of the various architectures components considered in the comparative study.
Component
Name
Acronym
Parametrization
Times
TO
C2 T2
Log- sig-ttines
LTO
log Tr
Concatenate
CONCAT
Cj
14
Event encoding
Log-concatenste
LCONCAT 6, log
71, ef
Temporal
TEM
e,
88
Temporal with labels
TEMWL
e, as in
115
e
88 in
E; 88 in
14
Learnable
LE
in
Learnable with labels
LEWL
16
6,
14
GRU
GRU(e1
61-1
Encoder
Self-attention
SA
hi as in
Constant
CONS
h;j 1dy
Component
Acronym
Parametrizatioo
Closed-form
Exponeutial
EC
MLP
MLP/MC
(C) 8s
in
FullyNN
FNN
( as
/
LogNormMix
LNM
(T)
(44
(R)
in
/
LogNorm
IN
(T)
44p with M
/
Decoder
RMTPP
RMTPP
(1)
30
p
Neural Hawkes
NH
(7,(0) as in
34
Self-attention
SA/MC
(g(() as in
28
X
Cumulative Self-actention
SA/CM
(A) 56
Hawkes
Hawkes
(2
Polsson
Polsson
(2 as In
dependent of the process' history and reduces essentially to renewal process Lindqvist et al. 2003).
Decoders. The decoders considered in this experimental study can be classified on the basis of the function that they parametrize, as well as on the assumptions they make regarding the distribution of inter-arrival times and marks. As described in Section 3.3 decoders that parametrize the MCIF the EC, MLP/MC SA/MC, NH, Hawkes and Poisson decoders. On the other hand, decoders that directly parametrize the cumulative MCIF are the FNN and SA/CM decoders. Finally, RMTPP separately parametrizes the GCIF and the distribution of marks, while LNM and its single mixture version, LN, separately parametrize the density of arrivalrrival times and the distribution marks. For these last three decoders, the distribution of marks assumed to be independent of the time, conditional on the history of the process.
Although not required define valid parametrization of marked TPP, we also consider for completeness the sctting where constant bascline intensity term μ4 (B) is added to the MCIF of the EC and RMTPP decoders. We define model as specific choice of event encoding mechanism, history encoder, and decoder, e.g. GRU-MLP CETTMMPP/MMCTTE MC-TO corresponds to model using the GRU encoder to build h, the MLP/MC decoder to parametrizes (t), and where the events are encoding using the TO event encoding. While most cases, any variation of a component can be seamlessly associated with any other variation of other components, some combinations are either impossible or meaningless. Indeed, all cumulative decoders (SA/CM and FullyNN) cannot be trained with
16

---

Published in Transactions Machine Learning Research (06/2023)
the TEM or TEMWL event encodings, as both would violate the monotonicity constraint of the cumulative MCIF. Morcover, as all event encodings are irrelevant with respect to the CONS history encoder, all CONS-EC models are equivalent. For the NH decoder, stick to the original model definition, it can be hardly disentangled into different components. A complete list of the considered combinations is given in Table 14 in Appendix C
We further classify the different models into three categories: parametric, semi-pone parmmerraametrc,, and non- parametric. Parametric TPP models include classical (i.e. non-neural) architectures, characterized by strong modeling assumptions and hence, low flexibility. We consider the Hawkes and Poisson decoders parametric baselines. On the other hand, cemi-parametric models include architectures that still make assumptions regarding the distribution of anter-arrival times, but their parameters are obtained from the output of a neural network. All models equipped with LNM, LN, RMTPP, or EC decoders are deemed SA/CM, and NH, are considered non -on-aaraetrric models.
All remaining baselines, Lee. FNN, MLP/MC, SA/MC,
5.2 Datasets
Real-world datasets. A total of 15 real-world datasets containing sequences of various lengths are used in our experiments, among which possess marked events. brief description is presented below, while their general statistics are summarized Tables and
Marked Datasets
LastFM Kumar et al. 2019 Records of users listening to songs. Each sequence corresponds to user, and the artist of the song is the mark.
MOOC Kumar al. 2019): Records of student's actions on an online course
system. Each sequence corresponds to a user, and the type of action is the mark.
Wikipedia + Kumar et al. 2019): Records of Wikipedia pages' edits. Each se-
quence corresponds to a Wikipedia page, and marks relate to the user that edited the corresponding page.
MIMIC2 (Du et al.
Clinical records of patients of intensive care unit for seven years. Each sequence corresponds to a patient, and marks describe the type of discase.
2016)
Records of users' actions on the open-source platform Each sequence corresponds to user, and the marks describe the action performed (Watch, Star, Fork, Push, Issue, Comment Issue, Pull Request, Commit).
Github Trivedi et al.
2018
Github during the year 2013.
Stack Overflow d Du et al. 2016 Records of the time users received specific badge the question-answering website Stack Overflow. Each sequence corresponds to user, and the mark is the badge received.
Retweets Mei & Eisner 2016): Streams retweet events following the creation of an original tweet. Each sequence corresponds a tweet, and the category (i.c. 'small", "medium', 'large") " to which a relweeter belongs corresponds to the mark.
htw.:///ww....bbooooeeoeee/eeeee Phttps: htttpp///gttuubcomm/ http://///ggghbb...o///////e//e/
17

---

Min Len. Published in Transactions Machine Learning Research (06/2023)
Table 2: Marked datasets statistics after pre-processing. "Sequences (%)* and 'Events (%)) re- spectively correspond to the number sequences and events that remain in the dataset after the pre-processing step. "MSL" the mean sequence length.
Sequences Sequences (%)
Events Events (%)
MSL Max Len.
Min Len. Marks
Wikipedia
390
0.59
30472
0.19
51.6
1163
50
MOOC
7047
1.00
351160
0.89
49.8
416
50
LastFM
856
0.92
193441
0.15
226.0
6396
50
MIMIC2
509
0.84
1812
0.68
3.0
32
43
Github
173
1.00
20657
1.00
119.4
4698
Stack Overflow
7959
1.00
569688
0.99
71.6
735
22
Retweets
24000
1.00
2610102
1.00
264
Table 3: Unmarked datasets statistics after pre-processing. "MSL" is the mean sequence length.
Sequences
Events
MSL
Max Len.
Reddit Subs.
1094
1235128
1129.0
2658
362
Reddit Ask Comments
1335
400933
295.9
2137
4
Taxi
182
17904
98.4
140
12
Twitter
1804
29862
16.6
169
2
Yelp Toronto
300
215146
717.2
2868
424
Yelp Airport
319
9716
30.5
55
9
Yelp Mississauga
319
17621
55.2
107
3
PUBG
3001
229703
76.5
26
Unmarked Datasets
Twitter Shchur et al.
2020
Records of tweets made by user over several years. Records of players' death in the online game PUBG.
PUBG Shchur et al. 2020)
m
Each sequence corresponds to a user.
Yelp Airport
Mississauga (hhhhur et al. 2020 and Toronto Shchur et al. Records of users' reviews the platform Yelp for the McCarran airport, Toronto city and Mississauga city respectively. Each sequence corresponds to a user.
2019
Reddit Ask Comments i Shchur et al. 2020) Records of comments reply to Reddit threads within 24hrs of the original post submission. Each sequence corre- sponds to thread.
Reddit Subs Shehur et al. 2020 Records of submissions to a political Red-Re the period from 01.01.201
to 12.12.2 Each sequence corresponds a 24hrs
window.
Taxi
Shchur et al. 2020
Records of taxi pick-ups in the South of Manhattan.
18

---

Published in Transactions Machine Learning Research (06/2023)
Pre-processing. Some marked datasets, such as Wikipedia and LastFM, originally presented a very large amount of marks, which distributions turn out to be highly spread across their respoctive domains. This observation raises two issues: (1) Some marks are therefore highly under-represented,, rendering the task of learning their underlying distribution difficult, and (2), as each mark is associated with an MCIF, the computational requirements drastically increase with the number of marks, which even more exacerbated when Monte Carlo samples need to be drawn. With the incentive to avoid either of these two bottlenecks, each marked dataset is filtered to only contain events belonging to the 50 most represented marks. The resulting sequenees containing less than two events are further removed from the dataset, which makes the number of distinet marks in MIMIC2 drop from 75 to 43. Finally, to avoid numerical instabilities, the arrival times of events are scaled in the interval [0,10]. Specifically, we compute si,ealed = 10t://max: where max is the largest observed timestamp in the dataset. Unmarked datasets do not go through any processing steps, at the exception of the scaling and removal of sequences containing less than two events.
As observed in Tables 2 and g the considcred datasets are relatively diverse in their charactoristics. Indeed, some datasets, such as Yelp Toronto or LastFM, possess a relatively short number of very long sequences, while others, such Twitter or MOOC, show the exact opposite characteristics. Figure 2 shows the distributions of the inter-arrival times logarithms across all sequences for all real-world datasets, as well as the mark distribution across all sequences for marked datasets. As observed, the distribution of the pooled (log) inter-arrival times differ significantly from one dataset to the other, in some cases presenting characteristics such as multimodality (LastFM, MOOC, Wikipedia, Yelp Toronto, PUBG) or large variance (MOOC, Wikipedia, Github, Yelp Toronto). The distribution of pooled marks also shows different characteristics. While the marks look evenly spread across their domains LastFM, MOOC, and Wikipedia, their distribution appears sharper on Github, MIMIC2, Stack Overftow, and Retweets. Such diversity should be empirically beneficial, as it would allow us to assess the models' performance across a wider range of real-life applicalions ions. In Figures| 12 13 and 14 of Appendix D we show the distribution of inter-arrival times and marks for some randomly sampled sequences in each dataset.
Synthetic datasets. We generate synthetic dataset from the multidimensional Hawkes process with the exponential kernel as in g with the following parameter values:
0.2
0.25
0.13
0.13
0.13
0.13
4.1
0.5
0.5
0.5
0.5
0.6
0.13
0.35
0.13
0.13
0.13
0.5
2.5
0.5
0.5
0.5
1
0.1
0.13
0.13
0.2
0.13
0.13
B
0.5
0.5
6.2
0.5
0.5
0.7
0.13
0.13
0.13
0.3
0.13
0.5
0.5
0.5
4.9
0.5
0.9
0.13
0.13
0.13
0.13
0.25
0.5
0.5
0.5
0.5
4.1
where the matrix a was scaled to have a spectral radius of approximately 0.8, guaranteeing station- arity of the process (Bacry et al.) 2020). The process essentially corresponds to a marked process with K = marks, from which we simulate distinct datascts of 1000 sequences using the library lick Bacry al. 2018
5.3 Experimental Setup
For each real-world and synthetic datasel, we randomly split. the sequences into train/ hat//vaaaaaaiaoneeeee splits of sizes 60%/20% 22t222%%%2% respectively. The models are trained to minimize the NLL in (I1) on the training sequences using mini-batch gradient descent (Suvrit et al. 2011). The NLL
19

---

Published in Transactions Machine Learning Research (06/2023)
MODC
Wikipedl
MIHIC2
Retweets
Toronto
Tex ttor
Tpp Wiss ssauga
Heddi Sube.
tel Airport
Reddi Comments
Figure 2: Distribution of log (top left) and mark distribution (top right) for marked datasets, after pre-processing. For unmarked datascts, only the distribution of log (bottom) is reported.
the validation sequences is evaluated at each epoch, and the training procedure is interrupted if the number of epochs reaches 500, or if no improvement is observed for 20 consecutive epochs. In the latter case, the model's paramcters are reverted their state of lowest validation loss. For all models, optimization carried out using the Adam optimizer (Kingına & Ba, 2014) with initial learning rate of 10-3 If no improvement in validation loss is observed for 5 consecutive epochs, the learning rate is divided by factor of 2, and training continues. We repeat this protocol 5 times using different random train/ haln//aaliaaiiontteee splits.
We conduct experiments with different values of event encoding dimension, specifically (4,8, 16, 32), as well as varying the number of hidden units for fully-connected layers in {8, 32). For models utilizing GRU, SA encoder, and SA/MC decoder, we explore different numbers of hidden units in (8, 16, 32, 64), and consider one two layers. In the case of SA encoder and SA/MC de- two heads. Additionally, the number of mixtures in LNM is explored in (8, 16, 32, 64). To determine the model'ss hyperparameters, we follow a specific procedure. For each model and dataset split, we randomly select five hyperparameter configuratinns. The model is trained using each of these configurations, and we select the configuration with the lowest validation loss. These five best configurations (which may differ depending on the split) are then evaluated on the respective test set. Finally, we report the average test metrics as described in the following section.
coder, we consider
20

---

Published in Transactions Machine Learning Research (06/2023)
5.4 Evaluation metrics
We consider range of metrics assess the performance of the model in terms of the time prediction task, which involves estimating (T(,,, and the mark prediction task, which involves estimating p" (k](t) and predicting the mark of the next event. As discussed in Section 2 reporting single NLL metric gathers the contributions of both inter-arrival times and marks, and in effect, obscures how a model actually performs on fitting the two distributions separately. Consequently, we split the NLL into NLL-T and NLL-M terms, highlighting the contribution of each term to the total NLL metric. Given a set of test sequences S (Si, .t.....,;; where each sequence St contains oy events, the average test NLL is given by
C(0,5) ΣΣ
L m4
log (1),, +
-
Σ
L
T1
lo p (kr,sle,si 0).
(58)
NLL-T
NLL-M
On the other hand, model's ability to predict the next event's mark is measured with the F1- score.
Calibration. TPP models are probabilistic models which estimate predictive distributions over arrival times and marks. While achieving a low out-of-sample NLL score is crucial, it also important ensure that the predictive distributions are well-calibrated. Calibration, in the context of forecasting theory, refers the statistical consistency between the predicted distribution and the observed outcomes (Gnciting et al. 2007). Formally, model that outputs a predictive CDF F*(r)
(which may be retrieved from (T) (1) or (*)) is (unconditionaall)) probabilistically calibrated
if (Dawid 1984 Kuleshov et al.
2018):
p. Vp E [0. 1].
(59)
where the probability is taken over and He. For example, if predictive distribution is well- calibrated, it means that a 90% prediction interval for inter-arrival times would, on average, contains the observed inter-arrival times 90% of the time. Similarly, in the context of mark prediction, probabilistic calibration is defined as (Guo et al. 2017):
pkk ((((tt=))) = E E [0,
(60)
where i and i the predicted and true mark at time respectively. Intuitively, when we say model is calibrated with respect to the mark distribution, it mcans that 90% of the predictions made with a confidence level of 0.9 should match the observed mark, on average.
We measure probabilistic calibration for the arrival time distribution using the Probabilistic Calibration Error (PCE), defined as (Dhour & Ben Taicb 2023 g:
M Σ m=1 ==1
1[F S pm
PCE = M
Pm
(61)
where Pm = m M E [0, 1] are specific probability levels, n= Ele ni, and where we set M = 50. For the mark distribution, we report the Expected Calibration Error (ECE) (Naeini et al., 2015
21

---

Published in Transactions Machine Learning Research (06/2023)
defined as:
M
ECE-
1 M
Σ ==1
aco(BB) - conf((B))
(62)
where each prediction k is assigned to the jth out of J bins (obtained by discretizing the interval [0, 1] in J equal-size bins) if p" (k)t) lies in the interval 1- 3]. By is the set of predictions that fell within bin j, acc(BB) corresponds to the model's accuracy within bin j. and conf( B2) is the average confidence level of all predictions that fell within bin j. We set J = 10, and lower PCE and ECE are better.
Reliability diagrams. disadvantage with PCE and ECE metries is that information regarding the calibration error at individual probability levels Pi...., or within individual bins Bi..., B is lost. Reliability diagrams visual tools that can be used to assess the probabilistic calibration of a model at fine-grained level for both continuous and discrete distributions. For the distribution of inter-arrival times, reliability diagram obtained by plotting the empirical CDF Ei=1 [[F"))) in
"
61 against all probability levels Pm- For the distribution of marks, it is obtained by plotting aco(B)) against conf( Bj) for all Bj. In both cases, probabilistically calibrated model should align with the diagonal line, and any significant deviation from it corresponds to miscalibration (Gneiting
et al.
2007 Guo et al. 2017]-
Statistical comparisons. We further conduct statistical pairwise comparisons between all de- coders, for each metric separately. First, Friedman test (Friedman) 1937 1940) employed to assess of at least one statistical difference among all decoders. If the null hypothesis is rejected at the = 0.05 significance level, proceed with comparing each decoder against cach other, using Holm's posthoc test (Holm) 1979) to account for multiple hypothesis lesting. The outcome of the pairwise comparisons is displayed on critical difference (CD) diagrams, which show the average
rank of a model on a motric of interest across all datascts, as well as groups of models that are not
statistically different from one another at a given significance level. Refer to Demšar (2006) García
& Herrera (2008) for additional details.
5.5 Results Aggregation
Given the high number of variations per model's component, the number of possible combinations renders the comparison of all individual models across every dataset unmanageable. To overcome this challenge, we aggregate the model results for each metric separately across all datasets as follows. When comparing different event encodings, we first group all models that equipped with specific decoder variation (e.g. MLP/MC). Then, among this decoder group, we further group all models by event encoding variations (e.g. TO-Any history Mccoder-MP/MMC compute the average score of that event sicoding-decoder group on a given dataset with respect to each metric. Finally, for each metric, we rank this encoding-decoder group against other encoding- decoder groups based on their average score on the same dataset. We apply this operation for each dataset separately and report the average and median scores, as well as the average rank of that component variation group across all datasets.
We then
Moreover, as the scale of the NLLs (NLL-T and NLL-M) vary significantly from one dataset to another, we standardize their values on each dataset separately prior to applying the aggregation
22

---

Published in Transactions Machine Learning Research (06/2023)
procedure above. For each model, we compute its standardized NLL (-T or -M) as
NLL mi
(63)
IQR
where 2 is the NLL score of model m dataset d, while and IQR are the median and inter-quartile ranges of NLL scores for all models on dataset d, respectively.
6 Results and Discussion
Tables and 5 present the average results across all marked datascts for different variations of decoders and event encoding/ /histoyy encoders, respectively. Table G displays the results for the combination of architectural components that achieved the lowest NLITT and NLL-M on average across all marked datasets and decoders. Appendix [ provides the same results for unmarked datasets, while Appendix B includes additional raw metrics and standard errors for cach datasct. In the tables, the "Mean" and 'Median" columns represent the average and median aggregated scores, respectively. The 'Rank' column indicates the average rank across all marked datasets, explained in the previous section. For the subsequent discussion, will focus on the 'Mean' column.
We would like to note that while these tables include all marked datasets, we found that certain datasets (MIMIC2, Stack Overflow, Taxi, Reddit Subs, Reddit Comments, Yelp Toronto, and Yelp Mississauga) may not be suitable for benchmarking neural TPP models, as most decoders achieve competitive performance on them. We urge researchers to exercise caution when using these datasets in future studics, and will discuss this concern later in this section. For completeness, have included the results of the aggregal ion procedure with these datasets excluded in Appendix and we found no significant differences in the results.
Analysis of the event encoding. Comparing the results in Table 4 we aim to provide answers to the two following questions: (1) Are vectorial representations of time betler for estimating the inter-arrival time distribution? (2) Do we need encode post mark occurrences to better model the
distribution of future marks?
(1) We observe that vectorial representations of time (i.e. TEM and LE in opposition to TO and LTO) improve the NLL-T and PCE scores for the EC and both SA decoders. Given that a model equipped with the EC decoder only uses the event encoding mechanism at the history encoding stage, this finding suggests that GRU and SA encoders rely on expressive transformations time to capture patterns of event occurrences. However, we observed that the GRU encoder is rather stable with respect to the time encoding employed, while the performance of the SA encoder drastically decreases with TO or LTO encodings. Added to the fact. that both SA decoders only perform well when using TEM and LE, conclude that self-attention mechanisms must be combined with vectorial representations of time to yicld good performance in the context of neural TPP models.
Furthermore, a log transformation of the inter-arrival times (i.e. LTO) improves performance on the same metrics for the RMTPP, FNN and MLP/ MLP/MC decoders. Using the LTO encoding in combination with the RMTPP decoder effectively defines an inverse Weibull distribution for the inter-arrival times (Kleiber & Kotz 2003). It. appears to be better fit. for the data than the Gompertz distribution from the original formulation of RMTPP
23

---

Published in Transactions Machine Learning Research (06/2023)
Moreover, for FNN and MLP/ /MC decoders, we found that the last Softplus activation prevents the gradient of the GCIF to take large values for very short inter-arrival times. However, in many of the considered datasets, most events occur in packs during extremely short time spans, requiring the GCIF to change quickly between two events. The LTO encoding in combination with the Softplus activation allows steeper gradients for short inter-arrival times and thus enables rapid changes in the GCIF. As a result, the FNN and MLP/MC decoders can reach lower NLL-T with the LTO encoding.
MLP/MC
(2) Including a mark representation in the event encoding generally improves performance in terms of NLL-T and PCE when moving from TO to CONCAT and from LTO to LCONCAT. On the one hand, this observation suggests that information contained in previous marks does help the model to better estimate the arrival times of future events. However, we observe that in most cases, TEMWL and LEWL encodings show higher NLL-T compared lo their TE and LE counterparis. Therefore, relevant information contained in previous marks appears less readily exploitable by the models when the mark embedding is concatenatod to a vectorial representation of time.
All decoders improve substantially with respect to the NLL-M, ECE, and F1-score metrics when the mark is included in the event encoding. While expected, this finding confirms that expressive representations of past marks are paramount to predicting future marks.
Analysis of the history encoder. From the results of Table 5 we observe that models equipped with a GRU history encoder yield overall better performance with respect to all time and mark- specific metrics compared to ones equipped with SA encoder. While self-attention mechanisms have gained increasing popularity since their introduction by (Vaswani et al., 2017] for sequence modeling tasks, we found that they are on average less suited than their RNN counterparts in the context of TPPs. Specifically, the GRU encoder is more stable with respect to the choice of event encoding mechanism, while the SA encoder requires vectorial event representation to achieve good performance. Furthermore, the constant (CONS) history-independent encoder systematically achieves the worst results with respoct to all metrics. This observation confirms the common as- sumption that future event occurrences associated with real-world TPPs indeed depend on past arrival times and marks, and hence better predictive accuracy can be achieved with better repre- sentations of the observed history.
Analysis of the decoder. On Table 6 report for each decoder separately the combination that yielded the best performance with respect to the NL.L-T (top rows) and the NL.L-M (bottom rows), on average across all marked datasets. While we previously explained that some variations of event encoding and history encoders worked on average better for a given decoder, it does not necessarily mean that the best combination includes that variation. Moreover, we find that a single combination does not perform cqually well on both metrics separately. In the following, we will thus focus our discussion on the lop-row models for time-related metrics (NLL-T, PCE), and on bottom-row models for the mark-related ones (NLL-M, ECE, F1-score).
As first observation, we note that LNM achieves the lowest NLL-T, outperforming all other basc- lines. The difference in performance with the LN decoder further suggests that the assumption of log-normality for the distribution of the inter-arrival times is not a sufficient inductive bias by itself and that the additional flexibility granted by the mixture necessary to achieve a lower NLL-T, Additionally, most neural baselines achieve on average lower NLL-T than their parametric coun- terparts, indicating that high-capacity models are more amenable to capturing complex patterns in real-world event data.
24

---

Published in Transactions Machine Learning Research (06/2023)
Table 4: Average and median scores, as well as average ranks per decoder and variation of event encoding, for marked datasets. Best results are highlighted bold. Among others, our key insights include that (1) the SA/MC and SA/CM decoders achieve significantly lower NLL-T and PCE when combined with vectorial representations time (i.e. TEM and LE) compared to the TO and LTO encodings, (2) in comparison the TO encoding, LTO significantly improves performance on the NLL-T and PCE for the RMTPP, FNN and MLP/Me /MC docoders, and (3) including a represcntation of past observed marks when encoding the history (CONCAT, TEMWL, LEWL) is paramount to achieve good performance with respect to the NLL-M, ECE and Fl-score for all decoders. Refer to text. for more dctails, and to Section 5.5 for details on the aggrogation procodure.
Marked Datacets
NLL-T
PCE Moan Mediaa Rank
NLL-M
BCE Mean Mediau Roak
Fi-score
Moun Medion Rack
Moan Mediou Rank
Moan Median Rook
EC-TO
0.59
0.58
8.5
0.2
0.23
6.25
0.19
0.21
6.5
0.44
0.44
7.12
0.19
0.17
8. 6.88
EC-LTO
0.7
0.57
7.26
0.2
0.23
6.38
0.15
0.17
8.5
0.44
0.45
7.0
0.19
0.17
7.25
EC-CONCAT
0.13
0.14
2.5
0.17
0.16
2.25
-0.51
-0.75
2.75
0.34
0.31
1.88
0.3
0.28
2.12
RG-LCONCAT
0.36
0:28
4.25
0.18
0-2
4.75
-0.45
-0.68
3.3
0.34
0.3
2.75
0.28
0.27
2.88
EC-TEM
0.2
0.17
4.38
0.18
0.15
4.3
0.18
0-2
6.35
0-42
0.42
8.12
0:2
0.18
8.
EC-TEMWL
1.46
0.3
4.26
0.19
0.17
4.59
-0.67
-0.78
2.0
0.88
0.27
2.38
0.3
0.29
1.88
EC-LE
0.05
0.11
3.0
0.18
0.15
2:88
0.15
0.13
5.38
0.41
0.2
0.15
5.75
BO-LEWL
0.17
0.18
3.58
0.15
4.12
33
-0.58
4.0
0.35
0. 36
0.29
0.28
8.38
LSM-TO
CONCAT
LNMELCNNCAT
LNM-TEM
-0.87
4.76
4.5
0.03
0.05
LNM-TRMWW.
-0.53
-0.87
4.76
0.03
4.12
89
-1.49
9.25
2.12
0.36
25
LNMLL
-1.11
-0.78
5.12
0.02
0.02
6.12
0.0
0.02
6.12
0.42
0.44
0.12
0.2
0.18
6.88
LNM-LEWL
-0.95
-0.92
4.5
0.02
0.01
4.25
-1.99
-1.15
8.5
0.3
0.28
8.38
0.32
0.29
3.98
FNN-TO
0.96
0.68
4.5
0.21
0.28
4.5
0.3
0.34
4.5
0.46
0.47
8.25
0.17
0.17
5.25
ENN-LTO
-0.32
-0.31
2.0
0.03
0.02
2.12
0.05
0.03
3-75
0.4
0.
2.25
0:21
0.19
2.62
PNN-CONCAT
0.86
0.45
4.38
0.21
0:27
4.5
0.00
0.02
2:88
0:43
86
3.8
0.19
0.10
3.0
PNN-LCONCAT
-0.57
-0.66
1.0
0.03
1.02
-0.99
-0.08
2.12
0.31
1.78
0.28
0.23
1.38
PNN-LR
0.87
0.53
4.25
3-75
0.27
0.31
4.12
4.25
0.17
0.18
1.62
FNN-LEWL
MLP/MC-TO
MCC
MLP MC- CONCAT
0.0
MILP/MC-L.CONCE
MNC-LOONCAT
-0.34
-0.28
2.25
2.0
-0.19
-0.42
3.95
0.29
28
2.30
0.98
0.28
0.01
-0.01
0.13
0.25
0.23
0.51
0.18
PX//MC-TEMWL
0.20
0.3G
7.20
0.18
0.17
7.62
-0.46
-0.45
2.62
0.34
0.38
2.73
0.3
0.29
2.0
MLP/MC-LE
.0.2
-0.19
3.12
0.18
0.11
3.12
0.37
0.22
0.62
0.38
0.3T
6.0
0.23
0.19
5.0
MLP/MC-LEWL
-0.15
-0.16
3.0
0.18
0.11
3.63
-0.42
-0.35
2.88
0.29
0.26
2.38
0.3
0.29
2.26
RETPP-TO
0.16
0.29
6.76
0.16
0.16
6.75
0.11
0.12
6.25
0.42
0.48
7.12
0.19
0.18
7.12
RMTFF-LTO
-0.19
-0.10
4.02
0.06
0.05
3:25
0.09
0.07
6.25
0.42
0.42
7.0
0.19
0.15
0.98
RXCTPE-COE CONCAT
-0.05
4.12
-1.28
2:5
0.35
0.33
2.5
SMEEP-LCONCES
-0.42
-0.34
2.0
85
-1.81
0.21
2:12
0.37
0.34
RMIPP.TEM.
0.01
RMTPP-LEWL -0.03
3.6
3A/CM-TO
10.78
0.12
0.11
4.12
1.14
0.33
4.75
5.122
0.14
4.75
SA SA/CM-LTO
20.39
-0.01
3.0
0.05
2.75
0.68
0.11
4.0
0.43
0.44
2.88
0.18
CONCCCA
-0.08
-0.09
3.58
0.07
0.05
3.25
0.6
0.16
4.0
0.44
0.44
3.38
0.17
0.17
CA/CM-LCONGA
20.53
0.05
4.12
0.19
0.05
3.62
0.61
0.06
2.5
0:43
0.45
3.25
0.18
0.11
3.0
SA 3A/CM-LL
-0.37
-0.4
2.0
0.05
0.04
2.38
0.09
0.07
3.0
0.42
0.44
3.62
0.19
0.17
4.19
BA/CSL-LEWL
2.14
-0.07
3.76
0.09
0.08
4.55
-0.0
-0.01
2.75
0.41
0.44
2.76
0.2
0.16
3.26
8A/MO-TO
1.22
1.0
7.38
0.28
0.31
6.75
0.24
0.81
5.12
0.47
0.47
7.12
0.17
0.16
7.6
SA/MC-LTO
1.07
0.82
5.76
5.88
0:28
0.4
0.47
6.0
0.18
SA/MC- CACCCCOCATT
0.84
0.85
0.5
6.25
0.2
TMG-TRMWL
SA/MC-LB
-0.62
8A/MC-LEWL
-0.48
-0.45
2.39
1.88
0.25
25

---

Published in Transactions Machine Learning Research (06/2023)
Table 5: Average and median scores, well as average ranks per decoder and variation of history encoder, for marked datasets. Best results are highlighted in bold. The key insight parsed from this Table is that models equipped with a GRU encoder (i.e. GRU-s) show overall improved performance with respect all metrics compared ones equipped with a self-attention encoder (i.e. SA-*). Refer text for more details, and Section 5.5 for details on the aggregation procedure.
Dutasets
Mediaan Median Rank
Median Bank
Menn Median Ronk
Median
CONS-EC
1.74
1.34
3.0
0.3
8:36
2.83
0.63
0.56
2.67
0.46
0.49
3.0
0.14
0.05
3.0
0:65
2.0
0.25
0.27
2:17
-0.04
2.0
0.43
0.42
2.0
0.22
0.23
1.0
-0.17
0.39
0.38
1.17
SA-LNM
CRU-LNM
0.35
1.17
CON8-FNN
1.02
0.72
3.0
0.2
0.24
2.67
0.25
2.83
0.46
0.48
2.83
0.17
0.31
2.5
SA-FNN
0.78
0.65
1.83
0.19
0.23
2.0
0.05
0.06
2.17
0.45
0.45
1.83
0,18
0.11
2.0
GRU-ENN
0.52
0.18
1:17
0.18
0.21
1.88
-0.17
-0.04
1.0
0.42
0.44
1.33
0.18
0.11
1.8
CONS-MLP/MC
0.5
0.52
3.0
0.19
0.23
2.83
0-4
0-3
2:335
0.39
0.4
2.0
0.22
0.17
2:33
SA-MLP/MC
0.18
0.13
2.0
0.27
0.21
2.0
0.04
-0.02
2.85
0.35
0.39
2.67
0.22
0.17
2:39
GRU-MLP.MC
-0.15
-0.14
1.0
0.16
0.188
1-17
00.05
-0.20
1.88
0.35
0.37
1:33
0.23
0.18
1.98
CONS-RMTPP
0.21
2.83
0.47
0.48
SA-RMTPP
CONS-SA/CM
SA-SA/CM
0.08
0.08
0.12
0.44
0.44
1,83
1.17
GRU-SA/CM
0.08
1.33
0.12
1.83
0.44
0.44
2.67
CONS-SA/MC
0.44
0.2
3.0
0.18
0.19
2.83
0.43
0.28
2.07
0.41
0.41
2.38
0.2
0.18
2.0
8A-AA/MC 0.07
0.08
1.67
0.10
0.16
1.83
-0.00
-0.10
1.6
0.99
0.41
1.07
0.10
0.12
2.33
GRU-SA/MC
0.05
-0.04
1,55
0.38
0.16
1.33
-0.3
-0.16
1.85
0.39
0.42
2.0
0.2
0.12
1.47
Figure 3 shows (t) (orange) and (*( (green) between two events in test sequence in LastFM, for the combinations of Table G that performed the best on the NLL-T. The greater the gap between (t) and f*(),, the higher the cumulative GCIF between two events. We observe that most decoders learn to assign high probability to very low inter-arrival times, which allows them to reach low NLL-T on datascts where events are highly clustered, such as LastFM. This behavior is only possible if the GCIF is allowed to vary rapidly once a new event is observed. As discussed previously, for decoders employing Softplus activation, such as MLP/MC FNN, or SA/CM, a steeper gradient for the GCIF can be obtained low inter-arrival times by using the LTO or LCONCAT encodings.
With respect to the mark prediction task, we find that the Hawkes decoder achieves the overall best results, outperforming all baselines in terms of NLL-M, ECE, and F1-score. While the Hawkes decoder did not perform favorably on time-related metrics, its clear superiority compared to more complex models on the mark prediction task rather intriguing. Likewise, the RMTPP and LN(M) decoders also show competitive performance on these metrics, despite their simplifying assumption of marks being independent of the time given the history of the process. The overall superiority of parametric and paramerreetticc architectures on the mark prediction task suggests that non- parametric docoders may actually suffer from their high flexibility, making them hard to optimize in practice. This particularly stands out for the NH decoder which performs extremely badly on all metrics, only marginally outperforming the Poison decoder. Nonetheless, although LN, LNM,
26

---

Published in Transactions Machine Learning Research (06/2023)
Table 6: Average and median scores, and average ranks of the best combinations per decoder on the
NLL-T (top rows) and NLL-M (bottom row) across all marked datasets. Best results are highlighted bold.
Marked Datameta
NEATT
PCR
NLL-M
P1-ecore
Median Rank
Moan
Mediau Raak
Rank
Mean Median
GRU-EC-LB
-0.22
-0.02
7.75
D-17
0.14
8-12
-0.1
-0.03
6.5
0.38
0.41
7.0
0.21
0.18
7.38
GRU-LNME-TO
-1.44
-0.93
1.5
0.02
0.01
1.88
-0.08
-0.07
6.3
0.41
0.42
6.75
0:21
0.18
7-12
GRU-LN-LEWL
-0.64
-0.06
6.25
0.07
0.04
6.76
-1.56
-1.38
2.76
0.28
0.28
4.5
0.33
0.35
2.88
GRU-FNN- DRU-FNN-LCONCAT
-0.6
-0.71
3.38
0.02
0.02
2.25
-1.67
-1.0
3.5
0.29
0.36
4.5
0.27
0.23
4.12
CRU-MLP MC- OC-LLCNNAT
-0.42
-0.35
5.5
0.00
0.07
6.35
-0.47
-0.47
5.62
0.26
0.26
2.58
0.28
0.28
4.38
GRU-RMTE PP-LCCNCAT
4.75
6.04
0.03
-1.88
-1.99
2:38
0.22
2.88
0.4
0.35
GRC-SA/ CM-LE
-0.46
6.12
0.05
0.07
7.76
8.12
0.19
0.18
1.07
10.25
CRU-EC-TEMWE
0.18
0.18
7.85
0.18
0.18
8.82
+1.42
-1.1
5.0
0.29
0.24
5.75
0.3
0.3
6.12
GRU-LNM-CONCAA
-1.41
-1.0
1.12
0.02
0.01
1.62
-2.62
-1.78
2.5
0.28
0.18
3.88
0.38
0.34
3.0
GRU-LN- LAN-ONNCTT
-0.51
-0.57
4.25
0.07
0.04
4.85
-2.89
-1.81
3.12
0.25
0.26
4.12
0.33
0.34
4.88
GRU-FNN CRU--NN-LCONCAA CN-LCCNATT
-0.6
-0.71
2.75
D.02
0.02
2.25
-1.57
-1.0
5.25
0.28
0.35
4.88
0.27
0.23
5.12
MLC--LP/MGG-TEEE TEMIL
0.11
0.23
7.5
D-17
0.16
7-12
-0.73
-0.64
7-0
0.32
0.33
6.38
0.32
0.3
5.5
CRU-RMTPE TEMEL
-0.10
-0.11
5.62
0.14
0.11
6.02
-2.67
-1.85
2.88
0.2
0.16
2.58
0.42
0.42
2.62
GRC-SA JCM-LEWL
-0.23
.0.23
5.5
0.08
0.07
5.12
.0.15
-0.11
5.12
0.41
0.43
8.5
0.21
0.2
8.88
TRU-SA/MO-TEMME
-0.29
-0.3
4.88
0.11
0.09
5.25
+0.55
-0.55
7.38
0.31
0.31
6.25
0.28
0.26
6.5
RMTPP, and Hawkes do better than their non-parametric counterparts, find that all decoders perform rather poorly on the mark prediction task.
Figure&shows the CD diagrams between the average ranks of all decoders at the = 0.1 significance level, on each metric separately. The lower the rank (further to the left on the top horizontal axis), the better the performance decoders that are not significantly different from one another. As can be seen, there are clear differences between the models' average ranks (e.g. LNM on the NLL-T for marked datasets). The fact that Holm's posthoc tests do not allow us to conclude statistically significant differences between them can be explained by a large number of pairwise comparisons compared to the few available samples (datascts), which results in high adjusted p-values.
decoder with respect to that metric, while a bold black line groups
On the calibration of TPP models. Figure shows the reliability diagrams for the inter-arrival time predictive distributions associated with the combinations presented in Table & averaged over all marked datasets. We obscrve a good probabilistic calibration for the LNM, FNN, MLP/MG RMTPP, and SA/CM decoders, as demonst.ratd by their curves closely matching the diagonal. On the other hand, the other decoders present higher degrees of miscalibration. More precisely, their empirical CDF lays systematically above the diagonal line, which suggests that the predictive distributions attribute little probability mass to low inter-arrival times. This observation supports our argument about the behavior of (T) on Figure 3. i.c. the conditional density of calibrated models attributes most of the mass to short inter-arrival times.
Figure 6 shows the reliability diagrams for the mark distributions associated with the best- performing combinations (in terms of the NLL-M) in Table 0 aggregated over all marked datasets. Overall, we observe that the calibration of these decoders mirrors their performance on the NLL-M and F1-scores, i.e. better scores with respect to these metrics correspond to better calibration. Indeed, the Hawkes decoder is the best-calibrated model among our baselines, followed by RMTPP
27

---

Published in Transactions Machine Learning Research (06/2023)
GRU-EC
CRU-LNK1
GRU-LNM
GRU-FNN
2000
30
3000
1000
* ((t1))))
1500
20
2000
500
1000
10
1000
500
GRU- PMM
GRU AMTPP
GRU-SA/CM
GRU-SA/MC
1000
1000
750
3000
750
500
2000
2000
500
250
1000
250
Hawkes
Polsson
80-
100
t (R())))
75
60
50
25
20
0
Figure 3: Evolution of (²) (orange) and [(((H)) (green) between two events on the LastFM
dataset. Arrival times correspond to 2.0975 and 2. 2.0986, respectively.
and LNM. Nonetheless, we note that all decoders are usually over-confident in their predictions, as shown by the bin accuracies falling systematically below the diagonal line.
Analysis of the history sizc. When training neural TPP model, it is typically assumed that the complete history Hl;, spanning from the first to the last observed event, contains valuable information about the future dynamics of the process. In other words, the joint distribution of an event e, = (t,, k;) is commonly modeled as dependent on the entire history of the process up to es, denoted as f(e,)H,,) = f(eylei-....... e1). However, in certain real-world scenarios, it may be more appropriate to explicitly assume a q-order Markovian property, where only g of the last 1 events preceding e actually influence the distribution of eq.
When comparing the impact of different encoders, we have found that effectively encoding the process history is crucial for modeling the joint distribution. However, it does not provide insights into the number of past events that truly contain useful information. To address this question, we train models using variations of both the GRU and self-attention history encoders. These encoders generate a history embedding h; specifically for event e using H."' {(t,, E Hit, < tt, i- q S j < i), which includes at most q events preceding 6--
In Figure 7 we show the evolution of the NLI-T and NLL-M (when relevant), by varying the maximal history size to which the history encoder has access to during training. We report our observations for the GRU encoder operating on this fixed-size window (GRU-Fixed), but similar results were obtained for a fixed self-attentive encoder (SA-Fixed). Compared to a completely masked history, which equivalent training the model with the CONS encoder (i.e. q q= we observe a substantial improvement in NLL-T for most models when only the last event is made
28

---

Published in Transactions Machine Learning Research (06/2023)
10
GRILI NM-TO
GRIALNM-TT
GRU-SA/MC-LE
GRU-EC-LE
CRU-FA
CRU-SANCH-LL
HEMTPP-LCONCA
Hawkes
CONCAT
GRU-LN-LEWLL
GRU-SA/CM-LE
BRU-SA/MC-LE
CRU-MLP/MC-LCONCTT
GRU-MLYAAC-LCONCAII
GRU-LN-LEWL
(a) NLL-T.
(b) PCE.
GRU AMTPP-EMMLL GRU LNW CONCAT
Poisson
Howkes
Poisson
NH GRULSEEEELLEL
GRU AMTPP-TEMMLL GRU LNM CONCAT
GRU CONCAT
GRULSA/CMLEWL
GRU LEMWIL
SRUMSAMMTTEWWL GRU MLPIMCTEMEE
GRU-LN CONCAI
GRL- TEMWL GRU BIU-SA/MCTIO TE MWI
FNIN LCONICTT
GRU FRU-FICNLCCNCA LCONCCT GRU EC TEMVIL
Howkss
(c) NLL-M.
(d) ECE.
10
GRU-RMTPR-TEMWL + GRU-LNIN SHU--LNM-ONNAAI Hawkes
Poisson
NH
GRU.SAAEMMLLWI GRU-SAIMC-TEHMA GRU-EC-TEMIVL
GRU LN CONCAT
GRU-FNN-LCONCAT PMMCTEWAMM
(e) F1-score.
Figure 4: Critical Distance (CD) diagrams per metric the a 0.10 significance level for all decoders on marked datasets. The decoders' average ranks are displayed on top, and bold line joins the decoders' variations that not statistically different.
Observed Ooserveddrene frequency
LNM
FNN
SA/CM
RMTPP
SA/MC
1.0
0.88
0.6
0.2
0.0
0.2
0.4
0.6
08
1.0
0.0
0.2
04
0.6
088
1.0
00
0.2
04
0.6
0.8
1.0
0.0
0.2
0.4
0.6
088
1.0
0.0
0.2
0.4
0.6
0.8
3.0
Observed frequency
MLP/MC
Hawkes
EC
NH
Poisson
0.9
0.4
0.2
0.0
0.0
1.0
Predicted probability
Figure 5: Reliability diagrams the distribution of inter-arrival times for the decoder's combi- nations that performed the best on the NLLLT (top rows of Table 6) averaged over all marked datasets. The bold black line corresponds perfect marginal calibration.
29

---

Published in Transactions Machine Learning Research (06/2023)
Hawkes
RMTPP
LNM
MLP/MC
FNN
Accuracy
0.8
0.6
0.2
0.0
p% Samples
0.6
0.0
EC
SA/MC
SA/CM
NH
Poisson
1.0
Accuracy
0.6
0.4
0.2
0.0
p% Samples
0.4
0.2
0.0
Confidence
Figure 6: Reliability diagrams the distribution marks for varions decoders, averaged over all marked datasets. Front rows depict a decoder's average accuracy per bin, while bottom rows show the average proportion of samples falling per bin. Bins aligning with the bold black line corresponds perfect calibration.
available to the encoder. This indicates process' past does indeed contain useful information. However, observed on most datasets, additional context does not yield significant improvement, with an NLL-T that often quickly stabilizes as increases.
On the one hand, this finding suggests that real-world processes do possess Markovian property and that going far history does not bring valuable additional insight regarding the arrival time of the next event. On the other hand, it could also indicate that RNNs (and self-attention mechanisms) fail to capture dependencies among event occurrences in the context of TPPs and that more research is needed to design better alternatives to encode the history. In essence, masking part of the history translates to reduction in model complexity, making them effectively less prone to overfitting on the training sequences. Additionally, the EC decoder appears to be more impacted on the NLL-T as q increases than other baselines. Considering that this decoder can only leverage the history embedding to define the MCIF, we hypothesize that optimization may in this case force the encoder extract much information possible from the patterns of previous marks occurrences.
30

---

Published in Transactions Machine Learning Research (06/2023)
GRU-Fixed EC
GRU-Fixed FNN
GRU-Fixed RMTPP GRU-Fixed SA/CM
GRU-Fixed SA/MC
GRU-Fixed LNM
GRU-Fixed MLP/MC
LadFH
220
-420
650
-250
80%
-100
Time
Mark
Time
230
1.00
775
2000
130
140
-3200
135
100
170
History Size
History Size
History Size
History Size
Wilgedts
160
Time NLL
-150
Time NLL
Mark NL.
420
Mark
0.40
-200
200
-250
100
200
-200
12
History Size
History Size
History Size
History Size
Tutter
Sabe
-249
History Size
Hilstory Size
Hi5to 5ize
History S125
4000
-200
TE710 NLL
-000
Time
290
History Size
Hittory 3
History
History Stze
Figure 7: Evolution of models' performance with respect to the Time NLL, and the Mark NLL when available, as a function of the maximal number of events used to construct h& when using a GRU operating on a fixed-size window. 'F' refers to the unconstrained GRU, i.e. the encoder having access to the full history.
A similar statement can be made for the NLL-M. However, we find continuous improvement as g increases for FNN, RMTPP, and LNM on Github and LastFM. As LNM and RMTPP parametrize the mark distribution solely from ha they also require expressive representations of the history to perform well with respect to the NLL-M.
31

---

Published in Transactions Machine Learning Research (06/2023)
(a) Marked datasets.
LastFM
MOOC
Github
Wikipedia
MIMIC2
Retweets
Stack Overflow
-1000
-160
-300
-200
350
-200
-42
1230
-10
-490
-300
-07
-0
-1500
-100
-900
LNM EC
H FNN
LNM EC FNN
LNK BC H FNN
LNM EC
FNN
LNM EC
FVN
LNM EC
FAN
INM EC FNN
3000
300
200
900
75
325
8C0
#G
80
190
60
300
2.5
0
0.0
UNM
UNN
UN
PHN
(b) Unmarked datasets.
Taxi
!witter
Heddit Subs.
Reddit Ask.
PUBG
Yelp loronto
Yelp Mississauga
Yelp Airport
-106
-000
-20
-114
-290
PM
RIN
FAN
LNW
LMM
UNM
LNN EC FNN
Figure& 8: Performance OF LNM, EC, Hawkes (H), and SA/CM on the NLL-T and NLL-M for each dataset.
On the adequacy of TPPs datascts. A concern that is rarely addressod in the neural TPP literature relates the validity of the current benchmark datasets for neural TPP models. En- guehard et al. (2020) raised some concerns regarding MIMTC2 and Stack Overflow as the simple iideeennett EC decoder yielded competitive performance with more complex baselines on such datasets. Figure reports both NLL-T and NLL-M (when available) for LNM, EC, Hawkes and FNN e on all real-world datasets. We indeed observe that the EC decoder is com- petitive with LNM on the NLL-T and NLL-M for MIMIC2 and Stack Overflow, supporting their recommendation that future research should show a cerlain degree of caution when benchmarking new methods on these datasets. We further express additional concerns about Taxi, Reddit Subs and Reddit Ask Comments, Yclp Toronto, and Yelp Mississanga, parable performance. All remaining datasets appear to be appropriate benchmarks for evaluating neural TPP models, as higher variability is observed in the results.
which all decoders achieve com-
Computational times. On Table 7 we report the computing time (in seconds) for each decoder averaged over 10 epochs for a single forward and backward pass on the training sequences of the MOOC dataset (standard deviation is given in parenthesis). All decoders were equipped with the GRU history encoder and the TEMWL event encoding, when relevant (i.e. not for Hawkes, Poisson, and NH), with similar hyper- papaae-erameterss configurations. Additionally, the batch size was 32, and the models were trained on an NVIDIA RIX A5000. With respect to the results and mark prediction tasks, we find LNM to provide the best performance cattaaanneerrutiinnnn /runtime trade-off among all the decoders considered in the study. On the other hand, the NH decoder is extremely expensive to train, while generally performing worse than other neural baselines.
both the time
Here also, we employed the combinations on the top row of Table to compute the NLL-T on marked datascts, and the combinations of Table I0 to compute the NLL-I on unmarked datasets, The NLL-M was computed using the combination ions on the bottom row of Table
32

---

Published in Transactions Machine Learning Research (06/2023)
Table 7: Average execution time in seconds per decoder for a single forward and backward pass on the MOOC dataset.
EC
LNM
MLP/MC
FNN
RMTPP
SA/CM
SA/MC
Hawkes
Polsson
NH
1.21 (0.06) (0.06) 1.66 (0.07) (0.07) 1.32 (0.07) 4.01 (0.08) (0.06) 17.11 (0.27) (0.01) 234.13 (1.36)
7
Conclusion
We conducted large-scale empirical study of state-of-the-art neural TPP models using multiple real-world and synthetic event sequence datasets in a carefully designed unified setup. Specifically, we studied the influence of major architectural components on predictive accuracy and highlighted that some specific combinations of architectural components can lead to significant improvements for both time and mark prediction lasks. Moreover, we assessed the rarely discussed topic of probabilistic calibration for neural TPP models and found that the mark distributions are often poorly calibrated despite time distributions being well calibrated. Additionally, while recurrent encoders are better at capturing process' history, we found that solely encoding a few of the last observed events yielded comparable performance to embedding of the complete history for most datasets and decoders. Finally, confirmed the concerns of previous research regarding the commonly used datasets for benchmarking neural TPP models and raised concerns about others. We believe our findings will bring valuable insights to the neural TPP research community, and we hope will inspire future work.
Acknowledgement
We would like to thank Victor Dheur and Sukanya Patra from the Big Data & Machine Learning lab at UMONS for providing helpful critical feedback in earlier versions of this paper. This work supported by the ARC-21/25 UMONS3 Concerted Research Action funded by the Ministry of the French Community General Directorate for Non-compulsory Education and Scientific Research.
References
Milton Abramowitz and Irene A. Stegun. Handbook of mathematical functions: With formulas, graphs, and mathematical tables, 1965. Courier Corporation.
Emmanuel Bacry, Martin Bompaire, Philip Deegan, Stéphane Gaillas, and Søren V. Poulsen. tick: a python library for statistical learning, with an emphasis on hawkes processes and time-dependent models, 2018. 18:1-5, Journal of Machine Learning Research.
Emmanuel Bacry, Martin Bompaire, Stéphane Gaïffas, and Jean-François Muzy. Sparse and low- rank multivariate hawkes processes, 2020. 21:1817 1848, Journal of Machine Learning Research.
Souhaib Ben Taieb. Learning quantile functions for temporal point processes with recurrent neural splines, 2022. AISTATS.
Alex Boyd, Robert Bamler, Stephan Mandt, and Padhraic Smyth. User-dependent neural sequence models for continuous-time event data, 2020. Neurips.
33

---

Published in Transactions Machine Learning Research (06/2023)
Renqin Cai, Xueying Bai, Zhenrui Wang, Yuling Shi, Parikshit Sondhi, and Hongning Wang. Mod- cling sequential online interactive behaviors with temporal point process, 2018. CIKM.
Shizhe Chen, Ali Shojaie, Eric Shea-Brown, and Daniela Witten. The multivariate hawkes process in high dimensions: Beyond mutual excitation, 2019.
Manon Costa, Carl Graham, Laurence Marsalle, and Viet Chi Tran. Renewal in hawkes processes with selfexxitatiin and inhibition. 2020. 52(3):879-99) 915, Advances in Applied Probability.
Daryl Daley and David Vere-Jones. An introduction to the theory of point processes volume ii: General theory and structure, 2007.
A. P. Dawid. Present position and potential developments: Some personal views: Statistical theory: The prequential approach., 1984. Journal of the Royal Statistical Society. Series A, 147:278-292.
A. De, U. Upadhyay, and M. GonezzRodriguez. Temporal point processes, 2019. Notes for Human- Centered ML, Saarland University.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets., 2006. Journal of Machine Learning Research, 7:1-30.
Victor Dheur and Souhaib Ben Taieb. A large-scale study of probabilistic calibration in neural network regression, 2023. ICML.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez Rodriguz, and Le Song. Recurrent marked temporal point processes: Embedding event history vector, 2016. SIGKDD.
Céline Duval, Eric Luçon, and Christophe Pouzat. Interacting hawkes processes with multiplicative inhibition, 2022. 148:180-226, Stochastic Processes and their Applications.
Mike Egesdal, Chris Fathauer, Kym Louie, Jeremy Neuman, George Mohler, and Erik Lewis, 2010. SIAM Undergraduate Research Online.
Joseph Enguehard, Dan Busbridge, Adam Bozson, Claire Woodcock, and Nils Y. Hammerla. Neural temporal point processes for modelling electronic health records, 2020. ML4H.
Mehrdad Farajtabar, Yichen Wang, Manuel Gomez Rodriguez, Shuang Li, Hongyuan Zha, and Le Song. Coevolve: joint point process model for information diffusion and network co- evolution, 2015. Neurips.
Milton Friodman. The use ranks to avoid the assumption of normality implicit in the analysis of variance., 1937. Journal of the American Statistical Association, 32:675 701.
Milton Friedman. A comparison of alternative tests of significance for the prob lem of m rankings., 1940. Annals of Mathematical Statistics, 11:86-95 92.
Salvador Gareía and Francisco Herrera. An extension on "statistical comparisons of classifiers over multiple data sets" for all pairwise comparisons, 2008. Journal of Machine Learning Research, 9:2677-2694.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E. Raftery. Probabilistic forecasts, calibration and sharpness, 2007. Journal of the Royal Statistical Society: Scries B (Statistical Mothodology).
34

---

Published in Transactions Machine Learning Research (06/2023)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017. ICML.
Ruocheng Guo, Jundong Li, and Huan Liu. Initiator: Noise-contrastive estimation for marked lemporal point process; 2018. IJCAL.
Alan G Hawkes. Point spectra of some mu- tually exciting point processes, 1971. Journal of the Royal Statistical Society: Series B, 33(3).
Alan G. Hawkes. Hawkes processes and their applications to finance: review, 2018. Quantitative Finance, 18:2, 193-198.
Sture Holm. A simple sequentially rejective multiple test procedure., 1979. Scandinavian Journal of Statistics, 6:65-70.
Valerie Isham and Mark Westcott. A self-correcting point process., 1979. Stochastic Processes and Their Applications, 8(3):335-347,
Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. Time2vec: Learning a vector representation of time, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. ICLR.
Christian Kleiber and Samuel Kotz. Statistical size distributions in economics and actuarial sciences, 2003. Wiley, p. 179.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression, 2018. ICML.
Srijan Kumar, Xikun Zhang, and Jure Leskovec. Predicting dynamic embedding trajectory in temporal interaction networks. In Proceedings of the 25th ACM STGKDD international conference Knowledge discovery and data måning, 2019. KDD.
Haitao Lin, Cheng Tan, Lirong Wu, Zhangyang Gao, and Stan. Z. Li. An empirical study: Extensive deep temporal point process, 2021.
Haitao Lin, Lirong Wu, Guojiang Zhao, Pai Liu, and Stan Z. Li. Exploring generative neural temporal point process, 2022. TMLR.
B. H. Lindqvist, Elvebakk G., and K. Heggland. The trend-renewal process for statistical analysis of repairable systems., 2003. Technometrics 45, no. 1 (2003): 31-44.
Thomas Josef Liniger. Multivariate hawkes processes., 2009. Diss., Eidgenossische Technische Hochschule ETH Zurich.
Hongyuan Mei and Jason Fisner. The neural hawkes process: A neurally self-modulating multi- variate point process, 2016. Neurips.
Hongyuan Mei, Tom Wan, and Jason Fisner. Noise-contrastive estimation for multivariate point processes, 2020. Neurips.
35

---

Published in Transactions Machine Learning Research (06/2023)
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht, Obtaining well calibrated probabilities using bayesian binning, 2015. AAAI
Yosihiko Ogata. Space-time point-process models for carthquake occurrencos, 1998. Annals of the
Institute of Statistical Mathematics, 50(2):379 402.
Takahiro Omi, Naonori Uoda, and Kazuyuki Aihara. Fully neural network based model for general temporal point processes. 2019. Neurips.
Jakob Gulddahl Rasmussen. Lecture noles: Temporal point processes and the conditional intensity function, 2018
Marian-Andrei Rizoiu, Swapnil Mishra, Quyu Kong, Mark Carman, and Lexing Xie. Sir-hawkes: Linking epidemic models and hawkes processes model diffusions finite populations, 2018. Proceedings of the 2018 World Wide Web Conference on World Wide Web.
Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-froe learning of temporal point processes, 2019. ICLR.
Oleksandr Shehur, Nicholas Gao, Marin Biloš, and Stephan Günnemann. Fast and flexible temporal point processes with triangular maps, 2020. Neurips.
Oleksandr Shchur, Ali Caner Türkmen, Tim Januschowski, and Stephan Günnemann. Neural temporal point processes: A review, 2021. Proceedings of 13th Joint Conference on Artificial Intelligence.
Sra Suvrit, Nowozin Sebastian, and J. Wright Stephen. Optimization for machine learning, 2011. MIT Press.
Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Representation learn- ing over dynamic graphs, 2018. ICLR.
Utkarsh Upadhyay, Abir De, and Manuel Gomez-Rodrigun Deep reinforcement learning of marked temporal point processes, 2018. Neurips.
Ashish Vaswani, Noam Shazeor, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017. Neurips.
Govind Waghmare, Ankur Debnath, Siddhartha Asthana, and Aakarsh Malhotra. Modeling inter- dependence between time and mark in multivariate temporal point processes, 2022. CTKM.
Yichen Wang, Bo Xie, Nan Du, and Le Song. Isotonic hawkes processes, 2016. 33rd International Conference on Machine Learning.
Andreas Wienke. Frailty models in survival analysis, 2010. Chapman and Hall
Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Le Song, and Hongyuan Zha. Wasser- stein learning of deep generative point process models, 2017a. Neurips.
Shuai Xiao, Junchi Yan, Stephen M. Chu, Xiaokang Yang, and Hongyuan Zha Modeling the intensity function of point process via recurrent neural networks, 2017b. AAAI
36

---

Published in Transactions Machine Learning Research (06/2023)
Shuai Xiao, Honteng Xu, Junchi Yan, Mehrdad Farajtabar, Xiaokang Yang, Le Song, and Hongyuan Zha. Learning conditional generative models for temporal point processes, 2018. AAAL
Junchi Yan. Recent advance in temporal point process from machinc learning perspective, 2019.
Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive hawkes processes, 2020. ICML.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process, 2020. ICML.
37

---

Published in Transactions Machine Learning Research (06/2023)
A Results unmarked datasets.
We report the comparison of different. event encoding mechanisms and history encoders for marked datasets in Table and Table [ respectively, where also added the worst score of each component's & variation. The results of the combinations that performed best with respect to the NLL-T can be found in Table 10 Overall, our findings regarding unmarked datasets align with the ones described in Section 6 However, we note that most decoders show a lower PCE, and hence, improved calibration with respect to the distribution of inter-arrival time compared to marked datasets. The reliability diagrams displayed in Figure for unmarked datasets do indeed confirm this observation.
In Figure 10 we present the CD diagrams illustrating the relationships between all combinations of Table 10 for the NLL-T and PCE metrics. While there is a noticeable distinction in the average ranks among the decoders, the available data for marked datasets do not provide sufficient evidence to establish statistical differences between them at the a = 0.1 significance level.
Table 8: Average, median, and worst scores, as well as average ranks per decoder and variation event encoding, for unmarked datasets. Refer to Section 5.5 for details on the aggregation procedure. Best scores are highlighted bold.
Unmarked Dataauta
NLL.T Worst Runk
PCR
Moan
Worst Rauk
BO-TO
0:25
0.24
0.6
3.5
0.07
0.06
0:16
3.25
RC-ETO
0:25
0.26
0.63
3.5
0.07
0.06
D-16
3.62
EC-TEM
-0.21
-0.26
0.31
1.0
0.05
0.03
0.14
1.25
EC-LB
-0.99
-0.04
0.48
2.0
0.05
0.08
0.15
1.58
LNM-TO
-1.15
3.0
0.01
0.01
3.12
LNM-LTO
-1.17
-0.87
-0.25
3.36
0.01
0.03
2:25
FNN-TO
1.06
2.62
FNN-LTO
-0.25
1:09
1.25
1.0
FNX-LE
0.58
0.5
1.05
2.12
0.08
0.07
0.15
2.58
MLP/MG-TO
-0.09
0.0
0.1
3.75
0.03
0.04
D.06
3.62
MLP./MC-ITOQ
-0.14
-0.17
0.06
3.5
0.03
0.03
D.08
1.88
MLP/MCCTRM
-0.23
-0.21
-0.04
2.12
0.04
0.03
D.09
2.62
MLP/MC-LE
-0.3
-0.38
-0.04
1.62
0.08
0.02
0.07
1.88
-0.07
3.78
0.04
0.08
3.75
RMIPP-LTO
-0.21
-0.13
0.2
2.75
0.02
0.03
2.0
RMIPP-TEM
-0.37
-0.92
1.25
0.03
0.07
1.88
SA/CM-TO
4.19
2.38
SA/CM-LTO
4.93
2.12
2.25
SA/OM-LE
-0.18
1.5
0.02
0.01
1.12
BA/MC-TO
0.78
0.96
1.17
3.6
0.09
0.08
0.16
3.58
5A/MG-LTO
0.65
0.69
1.08
3.5
0.09
0.1
D.15
3.58
5A/MO-TEM
-0.42
-0.39
-0.18
1.88
0.02
0.01
0.07
1.82
BA/MC-LE
-0.5
-0.49
0.17
1.12
0.03
0.02
0.08
1.62
38

---

Published in Transactions Machine Learning Research (06/2023)
Table Average, median, and worst scores, as well as average ranks per decoder and variation of history encoder, for unmarked datasets. Refer to Section 5.5 for details on the aggregation procedure, Best scores are highlighted in bold.
Unmarked Datacets
NLL.T
CONS-EC
1.03
0.57
2.14
3.0
0.1
0.00
0.2
3.0
SA-BO
0.48
0.64
0.36
2.0
0.08
0.07
0.17
2.0
GRU-EO
-0.35
-0.4
0.25
1.0
0.04
0.02
0.14
1.0
CONS-LNM
0.01
GRU-LNM
CONS-FNN
SA-PNK
GRU-ENN 0.25
0.18
0.94
1.12
0.06
0.08
0.11
2.25
CONS-MLP /MC
0.48
0.5
0.74
2.88
0.06
0.06
0.11
2.88
SA-MLP/MMC
0:15
0.17
0.33
2.12
0.05
0.01
0.09
2.0
GRO-MLI//CC
-0.49
-0.47
-0.33
1.0
0.03
0.02
0.06
1.12
CONS- RMTPP
0.31
0.26
0.78
2.88
0.04
0.08
0.00
2.38
SA-RMTPP
0.0g
0.01
0.28
9.12
0.84
0.03
0.07
CRU-RMCTFP
-0.33
GRU-SA/CM
CONS-SA/MC
0.06
SA-8A/MC
0.17
0.21
0.84
1.75
0.08
0.0T
0.08
1.5
CRU-SA/MC
0.1
0.14
0.38
1.88
0.06
0.07
0.08
1.88
Table 10: Average, median, worst scores, and average ranks of the best combinations per decoder on the NLL-T across all unmarked datasets. Best results are highlighted in bold.
Unmarkod Datarets
NLL-T
PCB
Mean Medion Worst Rank
Mean Median Worst Roak
GRU-EC BIU-EO-TEM
5.5
7.12
PERU-LNM-TEE TEM
-1.95
-0.95
-0.62
1.25
0.01
0.01
0.01
2.12
CRU-LNKI-TERE
-0.25
-0.25
0.41
7.0
0.05
0.05
0.07
7.75
PIU--NN-LTO- + nI
-0-15
-9.58
1:08
4.12
0.01
0.01
0.05
3:36
GRU-MLP GRI-MLP/MM-IIOO
-0.57
-0.56
-0.44
4:25
0.02
0.01
0.05
5.5
GRI-NMTOP-LIOO
-0.6
-0.33
-0.28
4:25
0.01
0.01
0.03
4.12
SA-S//OM-LE
.0.23
.0.15
0.0
7.35
0.02
0.01
0.04
4.5
GRU-SA SAUGSS//M-LL
.0.56
.0.55
0.4
4.12
0.03
0.02
0.1
5.35
Hawlen
-0.22
7.12
Poirea
11.0
10.0
39

---

Published in Transactions Machine Learning Research (06/2023)
Observed frequency
EC
LNM
FNN
MLP/MC
RMTPP
0.8
0.4
0.2
0.0
0.0
0.4
08
0.0
0.2
0.4
0.6
0.8
1.0
00
0.2
0.
0.6
1.0
0.2
04
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Observed Trequency
SA/CM
SA/MC
Hawkes
NH
Poisson
1.0
0.9
0.5
0.4
0.0
Predicted probability
Figure 9: Reliability diagrams the distribution of anter-arrival times for the decoder's combina- tions that performed the best on the NLL-T averaged over all unmarked datasets. The bold black line corresponds perfect marginal calibration.
10
8 9
GRU-LNM-TEM GRU-FNN LTO GRU-SA/MC-LE
Poisson
Deisson
NI SA-SA/C-LLE Howkes GRU-LN-TEM
GRULM P/ME-LTO
SA-SACCALLE GRU SA/MC LE
GAL-EC-TEM GRU MLP/MC.LOO
GRU RMU LTO GRU-EC-TEM B
Hewkes
(a) NLL-T.
(b) PCE.
Figure 10: Critical Distance (CD) diagrams per metric at the 0.10 significance level for all decoders on unmarked datasets. The decoders' average ranks are displayed on top, and bold line joins the decoders' variations that are not statistically different.
40

---

Published in Transactions Machine Learning Research (06/2023)
B Additional results
In Table 11 we report the results with respect to all time-related metrics (NIII-T, PCE) for all models of Table 6 on marked datasets, while results with respect to marked related metrics (NLL- M. ECE, F1-score) are given in Table 12 In turn, results with respect to time-related metrics for models of Table TO on unmarked datasets are reported in Table 13 Finally, we report in Figure
the standardized NLL-T and NLL-M for all models of Table i for all marked datasets.
Table 11: Results with respect to the NLI.T and PCE for the combinations of models presented in Table 6 on all marked datasets. Standard error across all splits is reported in parenthesis.
NLL-T
LasPM
MOOC
Wilkipeda
Gitbub
MTMIC2
Haw
Retwoets
GRU-LNM-TO
-1363-43
(39.
-275
(3-83)
(0)844)
-378 D9
(37.75)
(0:35)
-75
(1.01)
788-11
(2.32)
-11.777
(14.25) 25)
GRU-LN-TO
-1551.97
(60 26)
-278
(3-b5)
-22.77
-380
(67.05)
(0.07)
-78
(0:75)
-44224
(1:42)
-882.47
(2:47)
GRU CONCAAT
-1355.58
(58
37)
(0:11)
(0.8)
(1:41)
(3.63)
(1:3)
ORE-RMPPEE CONTPP-L.COM
CRU-SA LMC-LE
-1349.8
CRU-MLP PRU-MML/MC--LCOM MCCNCCC
-1336.38
(56.01) -270
-16..5
(67
877
-077.8
Hawkes
-1189.48 (55.2)
[3
352
(85.92)
-306
(T)
(0:24) -78.66
(0.82)
-83.18
(1.4)
-354.5 (2.15)
GRU-EC-LE
-1102.45 (49.96) -105
115377
(14.4)
-290.58
37)
0.91 (0.05) -78.00 (0.75)
-82.38 (1:34)
-546.95 (2.86)
NH
-788.83 (33.0) -70.84
(1:23)
(12 -102.72
-188 80
(20. 30)
1:73 (0.111 -72.64 (0.67)
-79.79 (1.08)
-256.29 (4.38)
Pritson
-747:35 (46.13)
-51.74
(1)
-57 32
(16.3) 363
+1.99.5
(26.06)
9 96 (0:12)
-71.6
(0:7)
-73.6
(1:13)
-231.51
(0:5)
-1368.78
(59. 77)
-250
(3.06)
-26..79
(37 88)
-367
41 (56.36)
0.60 (0:22) 76.67
(1.18)
-56
(8.03)
-621.33
65)
GRU- LN. CONCAT
-1348.27 (57 20)
-200 (3.15)
-22..31
(31 13) 367
41
(56.36)
0.97 (0.05) -78 (0.76)
-51 42 (1.84)
-563.40 (2:36)
GRU-S A /CM-L EWL
-1274.49
(50 07)
268 28 (6.07)
174.11
(20
336
(51 14)
1.5 (0.18) 777
(0.65)
-0.23 (1.5)
-55 (0.08)
RMTPP TEMWL
(52
2022.8
136
(5144)
(0:14)
(0.79)
522.06 (1.36)
-565.49 (3-72)
CRU-SA CRU-SA/MC- TEMWL
-1243.21
(47.
-257
(3:25) 188
81)
-322
(0:11) -78.04 (0.73)
-88.24 (1:29)
-669.54
/MC-LWW.
(0:79)
GRO-RO-T
(37.
(1
(1.03)
(11.16)
(51.2)
(0:11)
(1:42)
PCE
LasFM
MOOC
Wikepedta
Gathub
MIMIC2
Hawkes
80
Retweets
CRU-LNM-TOO
GRU-LN-TO
0.03
0.04 (0.0)
0.28 [0. 05)
0.05 (0.0)
(0.01)
0.03 (0.0)
0.05 (0.9)
0.01 (0.0)
GRI PRI--NN-JI WWW---N---CCCAN
0.02 (0.0)
0.02 (0.0)
0.07 (0.01)
0.02 (0.0)
0.05 (0.0)
0.0 (0.0)
0.0 (0.0)
0.01 (0.0)
CRU-SA /CM-LE
0.05 (0.01)
0.02 (0.0)
0.15 (0.06)
0.09 (0.01)
0.05 (0.01)
0.0 (0.0)
0.0 (0.0)
0.01 (0.0)
GRU-RMTPP- LCONCAT
0.06 (0.0)
0.07 (0.0)
0.1 (0.01)
0.02 (0.0)
0.04 (0.0)
0.01 (0.0)
0.01 (9.0)
0.03 (0.0)
GRU-SA /MC-LE
0.05 (0.01)
0:13 (0.0)
0.24 (0. 04)
0.12 (0.01)
0.04 (0.01)
0.01 (0.0)
0.01 (9.0)
0:01 (0.0)
GRU-MLP LCCLCCAAAT
0.05 (0.01)
0.15 (0.01)
0.27 (0.03)
0.14 (0.01)
0.05 (0.01)
0.01 (0.01
0.01 (0.0)
0.02 (0.0)
Hawkea
0.21 (0.0)
0.2 (0.0)
0.19 (0.01)
0.15 (0.01)
0.09 (0.0)
0.0 (0.0)
0.01 (0.0)
0.83 (0.0)
CRU-EC-LE NH
0.27 (0.0)
0.37 (0.01)
0.36 (0.01)
0,19 (0.01)
0.08 (0.01)
0.02 (0.0)
0.02 (0.0)
0.05 (0.0)
0:35 (0.01)
0.4 (0.0)
0.35 (0.01)
0.3 (0.02)
0.07 (0.0)
0.04 (0.0)
0.05 (0.0)
0.36 (0.0)
Poizson
0.38 (0.01)
0.4 (0.0)
0.54 (0.01)
0.33 (0.02)
0.07 (0.0)
0.04 (0.0)
0.04 (0.0)
0.36 (0.0)
(0.01)
(0:01)
(0.01)
(0.013
SIRI-LN-CONGAT
(0.01)
(0-0)
CRU-SA CLLL-S//MM-LEWE
(0.02)
0.05
0.01 (0.0)
RTTPP- TTEWL
(0.01)
(0.01)
0.03
(0.0)
0.04
GRU-SA A. /MC- TEMWL
0.17 (0.01)
0.19 (0.01)
0.33 (0.01)
0.15 (0.01)
0.05 (0.0)
0.01 (0.0)
0.01 (0.0)
0.03 (0.0)
GRU-MLP/ MCCLEWW
0.21 (0.01)
0.24 (0.01)
0.33 (0.01)
0.19 (0.01)
0.05 (0.0)
0.01 (0.0)
0.01 (0.0)
0.02 (0.0)
GRO-EC-TERE TEMWL
0.29 (0.01)
0.39 (0.0)
0.35 (0.01)
0.19 (0.01)
0.08 (0.0)
0.02 (0.0)
0.02 (0.0)
0.12 (0.01)
41

---

Published in Transactions Machine Learning Research (06/2023)
Table 12: Results with respect the NLL-M, ECE and Fl-score for the combinations of models presented in Table G on all marked datasets. Standard error across all splits is reported in paren- thesis.
NLL-M
LartFM
MOOC
Wikipedta
Githab
MIMIC
Hawiina
80
Retweets
CRU-LNA-CONCAT
(22.43)
92:47
(1.85)
250
(21.41)
124.
(18:24)
2.61 (0.14)
110.15 (0.6)
107
(0.6)
CRU-FNN-LCOO CONCCCCA
791.0 (31.28) 103.16 (4.0)
346.0 (47.58)
128.
(19.06)
(0:15)
110.72
(0.01)
109
(6.87)
23)
(1.87)
GRU-RRMTPP-LE
(40.33)
GRU-SA /MC-
869.55 (30.36)
(3.28)
407
(74.55)
156
(22.56)
4.50
111.71
(0.54)
117
(0.01)
GRU-MLP/ P/C-LOTOO
875.01
(31.01) 163.62
(4.1)
476 (136.36
166.5 51 (23.07)
111.67 (0.54)
118 33
(2:63)
Hawwkes
5144-13
(16.18) 112.14
26:
144 79 (11.88)
122.44 (35.5)
12:84 (0.23)
110.53
114
89 (0-75)
50
(0.14)
CRU-EC-TO
870.76 (29.85) 152.98
(1.71) 250.00 (41.0)
154.62 (21.3)
4.4 (0.16)
111.00 (0.56) 120
26 (6.09)
89.96 (0.25)
NH
869.56 56 (30.47) 184.50 (2.36)
272 06 (21.64)
101.84
(21.50)
4.60 (0.11)
111.99 (0.54)
191.2
(0.74)
91.1
(0.18)
Potzzon
873.82
(30.24)
188.13
(2:31)
534.95
(151
168. 37
(21.04)
11:25 (0:48)
113.48
(0.54)
152 93
(0:74)
92.11 (0:15)
Hawkes TRU-RMMTP-LLCCONE +
6144-13 (16.19)
112.14
(1.26)
144.79 (11.80)
122.44 (15.5)
12.84 (0:25)
110.53
(0.54)
114
(0.75)
80.5 (0.14)
722.89 (18.13)
66:32 (0:95)
401.
(117
123 67 (16.18)
3.75 (0:25)
110.31
(0 (8)58) 108 45
(0.89)
(0:21
GRU-INM-CONCAA
762
002-0311 (22.43)
82.47 (1.85)
258 19 (21.41)
120 34 (18:21)
2:51 (0:14)
130 15 (0.6)
107
(0.6)
83)
CRU-MLP MCC-CCONCAT
(22.15)
(5.34)
620.5 (162. 31]
147.
(21.76)
3.00 (0:34)
110.55
(0 (0:40)
(0.9)
(0.3)
CRU-FNN- --LNNCCT
791.0 (8.28)
103.16 (4.0)
340.0 (47.58)
128.
(19.00)
(0:15)
110.72
100
(6.87)
GRU-EC-
830.42 (83.2)
(1:65)
(38.42)
(31.71)
(0:18)
110.06
10
(0:9)
(3.0)
BCB
LasiFM
MOOC
Wikkipedia
Cithab
MIMIC2
Huwkeu
80
Retweeka
GRU-LNM-TO
0.5
(0.01)
0.41
0.44 (0.0)
0.28 (0.02)
0.25
GRU-LN-TO
0.5 (0.0)
0.45 (0.01)
0.5 (0.0)
0.42 (0.01)
0.4 (0.02)
0.44 (0.0)
0.28 (0.01)
0.23 (0.02)
GRU-PYN-LCONCAL
0.43 (0.05)
0.31 (0.04)
0.5. (0.01)
0.15 (6.0)
0.46 (0.0)
0.39 (0.01)
0.08 (0.0)
0.06 (0.0)
GRU-SA /CM--LP
0.5 (0.0)
D:45 (0.0)
0.5 (0:0)
0.43 (0:01)
0.45 (0.0)
0.44 (0.0)
0.26 (0.04)
0.01 (0.03
GRU GRU-RMTPP-- LCONCC-
0.43 (0.03)
0.05 (0.01)
0.36 (0.00 (0.06)
0.15 (0:01)
D:16 (0.02)
0.4 (0.0)
0.15 (0.03)
0.08 (0.02)
GRO-SA /MC-LE
0.5 (0.0)
0.42 (0.01)
0:49 (0.0)
0.22 (0-02)
0:33 (0.01)
0.44 (0.0)
0.15 (0.01)
0.27 (0.02)
CRU-MLP CAMMMPPMM... MC-LCCONCTT
0.42 (0.02)
0.21 (0.04)
0.47 (0.02 (0.02)
0.14 (0 08)
0.32 (0.03)
0.44 (0.01)
0.08 (0.01)
0.06 (0.0)
Hawkes
0.03 (0.0)
0.14 (0.0)
0.11 03)
0.07 02)
0.16 (0.01)
0:34 (0.01)
0.00 (0.02)
0.19 (0.0)
GRU-BC-LE NE
0.5 (0.0)
(0.01)
0.48 (0.01)
0.31
0:38 (0.0)
0.44 (0.0)
0.23 (0.02)
0.38 (0.01)
0.5 (0.0)
0.5 (0.0)
(0.0)
0.47
0.46 (0.0)
(0.0)
0.48 (0.0)
0.45 (0.01)
Poisson
0.6 (0.0)
0.5 (0.0)
(0.0)
0.49 (0.01)
0.47 (0.0)
(0.0)
0.48 (0.0)
(0.0)
CRU-LNM-CONCAT
(0.07)
GRU-LN- CONCAATT
(0.02)
0.46
0.31
0.2 05)
0.39
0.07 (0.01)
GRU-SA DLSSAA/CMEEEE
0.5 (0.0)
(0.03)
0.43
0.39
(0.0)
0.34
0.23
GRU RMTTP--PPPPP -T TEMWLL CRU-SA/MC /L/C/CCTEEEE -EC-TEWWL
(0.02)
(0.01)
0.34
0.18
0.1
(0.0)
0.13 (0.09)
0.07
0.49 (0.01)
0.36 (0.02)
0.36 (0.06)
0.26 (0.01)
0.27 (0.02)
0.44 (0.0)
0.08 (0.04)
0.28 (0.08)
CRU-MLP/ LEWL
0.4 (0.05)
0.23 (0.01)
0.44 (0.05)
0.22 (0.02)
0.15 (0.03)
0.44 (0.0)
0.03 (0.0)
0.17 (0.04)
GRU-EC-
GRU-RC-TEMWE TEMWI.
0.48 (0.01)
0:21 (0.03)
0:49 (0.0)
0.27 (0.04)
0.17 (0.02)
0.4 (0.0)
0.21 (0.01)
0.12 (0.02)
Fl-xcor
LarFFM
MOOC
Wikeipedia
Githab
MIMIC2
GRU-LNM--T
0.01 (0.0)
0.06 (0.01)
0.01 (0.0)
0.4 (0.08)
0.22 (0.02)
0.14 (0.01)
0.28 (0.01)
0.54 (0.01)
GRU-LN-TO
0.01 (0.0)
0.06 (0.01)
0.01 (0.0)
0.4 (0.03)
0.23 (0.02)
0.15 (0.01)
0.28 (0.0)
0.55 (0.0)
0.02 (0.01)
0.23 (0.04)
0.02 (0:01)
0.48
0:22 (0.02)
0:23 (0.0)
0.33 (0.0)
(0.0)
GRU-BRUSS
ERU-RMTPP-LCOOS
GRU-SA /CM-LE
0.0 (0.0)
0.0 (0.01)
0.38 (6.01)
0.04 (0.0)
0.31 (0.11)
(0.0)
0.52 (0.01)
0.4 (0.03)
0:74 (0.01)
0:22 (0.02)
0:13 (0.01)
0:25 (0.01)
0.33 (0.0)
0.28 (0.0)
0.4h (0.04)
(0.0)
GRU- MC-LCONNAT
(0.0)
(0.0)
(0.0)
(0.03)
0.24 (0:03)
0:16 (0.01)
(0.0
Hawkes
0.3
0.06
CRU-BC-LE NH
0.0
0.06
0.0
0.01
(0.08)
(0.0)
0.26 (0.0
Potsson
0.0
0.01 (0.0)
0.01 (b.0)
0.4 (0.02)
0.2 (0.02)
0.11 (0.0)
0.28
0.33
GRU-LNM-CONCAN
0.1 (0.03)
0.36 (0:01)
0.22 (0-1)
0.47 (0 03)
0.64 (0.01)
0:25 (0.0)
0.33 (0 00
0.58 (0.01)
GRU-L GRU-EN-CONNE N-CONCAT
0.1 (0.02)
0.36 (0.01)
0.02 (0:01)
0.47 (0 03)
0.53 (0.09)
0:26 (0.0)
0.32 (0.0) 03
0.6 (0.0)
GRU-SA /CM-LEWT
0.0 (0.0)
0.05 (0.01)
0.0 (0:0)
0.4 (0.03)
0:25 (0.04)
0.34 (0.0)
0.28 (0.01)
0.58 (0.01)
CRU-RMTPP TE-TEW
0.11 (0.01)
0.39 (0.0)
0.46 00)
0.48 02)
0.78 (0.03)
0:27 (0.0)
0.32 (0.0)
0.59 (0.0)
GRU-SA BA/MC-TTEMVV
0.01 (0.0)
0.08 (0.02)
0.1 (0.04)
0.4 (0.08)
0.57 (0.02)
0.19 (0.02)
0.32 (0.0)
0.56 (0.0)
GRD-MLP/ MCWL
0.02 (0.01)
(0.01)
0.07 (0.04)
0.42 02)
0.59 (0.02)
0.18 (0.01)
0.53 (0.0)
0.57 (0.01)
GRU-EC- TEMWL
0.02 (0.0)
0.28 (0.01)
0.01 (0.0)
0.41 (0.03)
0.53 (0.03)
(0.0)
0.32 (0.0)
0.58 (0.01)
42

---

Published in Transactions Machine Learning Research (06/2023)
Table 13: Results with respect the NLL-T and PCE for the combinations of models presented in Table TO on all unmarked datasets. Standard error across all splits is reported in parenthesis.
NLL-T
Taxi
Twister
Reddit Subs
Reddit Ask
PUBG
Yolp T.
Yelp A.
Yelp M.
(3.91)
(24.0)
-917.45
(13.1)
-2803
(85.00
(0.18)
(0.76)
GRO-FNN-LTO
GRU CRU-MLP/MM LLP/MO-LTO
-136.25
-4403.41
-021.5
(12.77)
-130.44
32
(0.79)
RMTPP-LTO
-136.15
-4407.55
-010.06 (12.81)
-107
-58-31
SA-SA /OM-LE
(1.132
-10.54 (0.51)
-4320.9 (23.17)
-856.8 (12.31)
-108.51 (1:31)
-2739
+5.99 (0:15)
GRU-SA/MCCLLE
-124.53
-12.11 (0.5)
-4887 88 (19.64)
922
(13.07)
-109.3 (0.9)
-2001.79 79 (92.0)
+6.53 (0.23)
-57.55 (0.84)
Hawkea
1134.09 (3.98)
-7.62 (0.51)
4336.22 (23.74)
-919.01 (12.92)
-102 02 (0.2)
-2778.63 (87.1)
-4.62 (0.18) -53.33 (0.8)
NH
-18.21 (24.44)
-1.54 (0.22)
-4158.41
74)
-711.22 (11.7)
-91.95 (0.14)
-2081.81
(291 26)
-.22 (0.14)
-99.19 (0.6)
Possson
-41.71 (1.02)
5.18 (0.11)
-3706.3
(17.61)
-676.53
(30.2)
-35.11 (0.15)
-675.76
(17:22)
-2.11 (9.1)
-25.2 (0.29)
Reddit Subs
Reddit Ask
Yelp
Yelp
CRU-EC-TEM
0.01 (0.0)
8.18 (0.0)
0.01 (0.0)
0.02 (0.0)
0.02 (0.0)
0.05 (0.0)
0.02 (0.0)
0.06 (0.0)
(0.0)
9.01 (0.0)
0.01 (0.0)
(0.0)
0.01 (0.0)
0.01 (0.0)
0:01 (0.0)
0.01 (0.0)
GRU-FNN-LTO
(0.0)
8.81 (0.0)
(0.0)
0.0 (6.0)
0.05 (0.0)
(0.0)
0.01 (0.0)
CRU-SA/MC-LL
00.00.08)
Howken
(0.0)
NH
(0.06)
Poteoon
(0.0)
0.2
0.27 (0.0)
0.25
0.07 (0.0)
0.47 (0.0)
0.12
0.27 (0.0)
43

---

Published in Transactions Machine Learning Research (06/2023)
10.0
LastFN
MOOC
Wklpedia
Github
HHNIZ
8.0
Slack
Rethweets
6.0
NLLT
4.0
2.0
0.0
0000
-2.0
Hawkes
LNM
RMTPP
MLP/MC
FNN
SA/MC
SA/CM
NH
EC
6.0
4.0
2.0
NLL-M
0.0
-2.0
Witteeda
Gthub
HBMC2
Stack
Retweets
Hawkes
LNM
RMTPP
EC
MLP/MC
FNN SA/MC SA/CM
NH Poisson
Standardized NLL-T and NLL-M for various decoders per dataset. The NLL-T was computed using the top row models of Table 5 while the NLL-M was computed using the bottom row models of Table Lower NLL-T and NLL-M is better.
Figure 11:
44

---

Published in Transactions Machine Learning Research (06/2023)
C TPP models considered
Table 14:
Combinations included in the experimental study. "Unmarked' refers to whether the method is adapted for unmarked datasets. For EC, LNM and RMTPP, the setting where a baseline intensity term (B) is also considered.
CONS
CONS RC
7O CONDAT
CONS MELP/WE TO CONS MLLP JMC CORCINT
LODNCAT
LTO LCONCAT
GRU-EC-4TOO LCC-CCCLLCNCAT
CONS-36 /MC-TEMM
TEXEL
SOXS-MLP//DL TEMELL
CRU
TEM
GRU-BC-TEX TEET
LEWL
CONS MLP MAC-LLEEW
CONS ALL MMC//MC-LB
LE
MIP/MMC
LTO
GRU MILP /MC LTD
TO CONCAT
SA RC TO EA-BC-DONCEE CONCAT BA-EC-LCONCAT 2 SA TRAIIT
GRU
LODKCAT
GRU MLP /LLP/MC LOOKDAT
TEXEW
GRL-MLEY/SSCE SEC-TDAWW
LCONGAT
GRU-NLLD/MCCLE
CRU
LEWE.
TRADET
SA-RCLLK RA-KC-LEWL
LIEW
TO DA-MLP/KG-TO
CONCAT
SA-MEE MLL//MC CONCAT
TIMM
SA-MLI /NC-TIMM
EA- TEMWE
TO CONCAT LOONCAT
GHU-LNMI-TO
LE
EA-MLE /MCALE
GIEE-LRSS-LTO
SA-MLI /AEE-LIWW WL
CRU CONCAT
GIUU
GRU
GRU ENMUTEM
CONS /OM TO
LN3E
TEMIN
GRU-L89E TENSIL
LTO LOOKCAT
CONSS-A JON-ETO
LEWL
CRU-LNN--LEW
CONS
DONS-SA/CNS /CA LOOKCAT DONS-BA LEWS-BAUUN-LEWE
LEWE
LTO
EA-LNAL-ITO
SAIENS LOUNCAT
SAJCM GRT ETO GRI 2A/OM CONCAT
SA
LCONCAT
ENM
LTO CONCAT
TENWE
TAM TRAWWT
LE
SA-LNI-LE
LIE
GRE-SA/CHHLL
CONS
CONSRNTTP
TO
CA-BA/CCS-TO PA-RAUUN-CONNAT
TO
SA
CONUAT LOOKCAT
BA-SA/CSE LOOKCAT
SA-EA CONCAC
ETO
ORU- GRU-EMITPP-12E
SA-SA/CM-LB SA SA,CM LEWLL
CRE CONCAT
L.WEE
GRU
LCONCAT
TEM
GHU-UAFTTT-TEE
RMPTP
LE
GAU-RSCIPO-LE
LZO
CONG-SA AES-SAAAROO FACEEETO
CUNS
LOOSCAT
CONS R MOC LOORDAT
TO
PA-RMTPIITTO
TEMOL
CONS-SA TMC-TTEEEE
CONCAT
SA-EMEPP CONCOTT
LEWE
DONS- LED.LEWW
LCONCAT
SA-RXTPP LOONGAT
TRMDL
TO CONCAT
GEL--SA MACSTO
LEW
SA/MC
GBU-EA/MCC MICCONNAT
GRC
TIRM
GRI &A 2A/MP TRM
LTO LCONCAT
CON
LE LEWE
CRE-SA /MCSLB GRU-SA/AMC-LEN
CONS-PNN-CO DONS PNN-LCONDAT
CONS-FNN-LB
SA-TA SECSIIU LOOKCAT
LTO
FNX
TO
GRU PXEE TO GRI PNN CONDAT GRU-TEN-LLE GBU-7SS-LERR
SA
SA SAMC LOOKCAT SA /MC TEMEE SA-SA/MES-LEWL
CRU
CONCAT LD LEWL
TRMWI.
LEWE
Neuna Howken
EA-PX--O SA.AA AA-P
UTO LCONCAT
Howler
Howlees
SA
Poisson
Poison
LRWI
SA PAOE LEWL
45

---

Published in Transactions Machine Learning Research (06/2023)
D
Sequence distribution plots
LastFM
-20
-15
-10
-5
0
10
MOOC
-25
-20
-15
-10
-5
10
15
Wikipedia
-17.5
-12.5
-7.5
-2.5
2.5
7.5
Github
40
-30
-20
100
0
10
20
Figure 12: Distribution of log t. (left) and mark distribution (right) for 10 randomly sampled sequences in LastFM, MOOC, Wikipedia, and Github, after preprocessing.
46

---

Published in Transactions Machine Learning Research (06/2023)
MIMIC2
-10
-8
6
-4
-2
0
2
4
6
8
Stack O.
-25
-20
-15
-10
-5
5
Retweets
-T.511:....100075-5--222000 2.5 5.0 7.5
Distribution of log (left) and mark distribution (right) for 10 randomly sampled Figure 13:
sequences in MIMIC2, Stack Overflow, and Retweets, after preprocessing.
47

---

Published in Transactions Machine Learning Research (06/2023)
Taxi
Twitter
-12
-10
-8
-2
0
2
4
12
10
8
-6
-4
-2
0
2
4
6
Reddit Subs.
Reddit Comments
-12
-10
-8
-6
-4
-2
0
2
12
-10
-8
6
-2
PUBG
Yelp Toronto
-8
-6
4
-2
2
4
22.5
-17.5
-12.5
77.5
-2.5
2.5
Yelp Airport
Yelp Mississauga
-12
-10
-8
-6
-4
-2
0
2 4 6
12
10
-8
-6
-4
-2
0
2
Figure 14: Distribution of log T, for 10 randomly sampled sequences in unmarked datasets, after preproccssing.
48

---

Published in Transactions Machine Learning Research (06/2023)
E Proofs
1) Proof of Equation 6 in Section 2
By definition of (1)
Rasmussen
2018 have:
(((1)
(²)
(64)
1- (t)
log (1 - (t))
(65)
dt
Integrating both sides from 4-11 to t, we get
(t) = S" X7(s)ds f" I
-log (1-og(1-22) F*
(66)
-1oog(1 F* (1)) + log
1 1-F"(1i))
(67)
where (t = O results from the point process being simple, i.c. two events cannot occur simultaneously. Rearranging the terms, we find:
= -1xx) = -1 exp (- -
A200))=
(68)
k=
Difterentiating with respect to gives
d xt))))X())spp((
f*())
(69)
dt
Given that x²(1) = (1))((())))) we finally find
(tt = (*)(pp (2())epp( (-)
(70)
49

---

Published in Transactions Machine Learning Research (06/2023)
F Results on relevant datasets only
The discussion in Section highlighted that some datasets (MIMIC2, Stack Overflow, Taxi, Reddit Subs, Reddit Ask Comments, Yelp Toronto and Yelp Mississauga) might potentially be inappoppi- ate for benchmarking neural TPP models. For completeness, we report in Tables
15 16
1
and 20 the results of the aggregation procedure discussed in Section 5.5 without these borderling inadequate datasets included. We found no significance differences with respect to the conclusions of Section 0
Table 15: Average and median scores, and average ranks of the best combinations per decoder on the NLL-T (top rows) and NLL-M (bottom row) across all relevant marked datasets. Best results are highlighted in bold.
Marked Datasets
NLL-T
PCB
NLL-M
ECE
F Flascore
Moan Median Rank
Mean Median Rank
Mean Mediam Ronk
Menn Median Roak
Moan Median Rank
SET-BC-LCONCAA
4.38
0.27
CRU-LNM-COOD CONCAT
-0.86
-0.84
0.02
0.01
1.5
-2.97
-1.76
2.33
0.26
0.27
3.53
0.34
0.31
2.67
GRU- LN-LTO
-0.41
-0.63
4.17
0.06
0.05
5.0
-0.28
40.33
7.17
0.21
0.14
7.5
CRU-PNN-LEO
.0.62
-0.7
3.17
0.02
0.02
1.33
-0.09
-0.14
83
0.39
0.42
6.53
0.21
0.14
6.33
GRU-MEP PEMP/EP/R MC- LOONGAT
-0.41
-0.35
5.67
0.11
0.11
7.0
-0.58
-0.47
5.5
0.29
0.32
3.5
0.25
0.23
4.33
GRU GRU-RM SEEMPPP--CCNCA
-0.59
-0.3
1.67
0.05
0.04
4.67
-1.91
-1.98
67
0.25
0.26
2.67
0.35
0.31
2.17
CRU-SA/CM-LL
-0.33
-0.89
6.38
0.06
0.04
4.0
0.11
0.11
8.17
0.45
0.45
8.38
0.17
0.08
9.0
CRU-SA SAU-SA MC-LE
.0.5
-0.45
4.17
0.00
0.08
5.5
0.1
04
8.17
0.39
0.43
7.33
0.19
0.1
7.38
Hawkes
0.5
.0.15
7.17
0.13
0.17
6.67
-6.27
-1 58
3.
0.15
0.13
2.0
0.44
0.42
2.0
Possua
1.51
10.5
1.39
1:28
0.14
0.08
10.98
0.14
GRD-RC-TRMWW.
GRU-LNM- AT
CRU-LN-CONGHN CONCAT
-0.57
4.5
GRU-PN PNN- -X-LOONCAT
-0.02
1.88
+1.98
4.67
0.27
GRU-MLP 'MC- TEMWL
0.18
0.24
7.83
0.22
0.25
8.17
-0.77
-0.84
T.17
0.95
0.39
6.17
0.27
0.28
6.33
B
-0.54
-0.49
3.93
0.05
0.04
4.0
-3.28
-2.0
8.17
0.2
0.16
2.67
0.57
0.38
2.17
DRU-SA/CM-LEWC
-0.25
-0.23
5.83
0.1
0.00
6.17
-0.21
-0.17
8.0
0.42
0.44
8.6
0.19
0.1
9.0
BRU-SA/MC-TERAS FAAWL
-0.26
-0.3
5.67
0.34
0.15
6.0
-0.39
46
7.67
0.36
0.36
6.83
0.22
0.15
7.17
50

---

Published in Transactions Machine Learning Research (06/2023)
Table 16: Average and median scores, as well as average ranks per decoder and variation of event encoding, for relevant marked datasets. Refer to Scction 5.5 for details on the aggregation procedure. Best results highlighted in bold.
Dataneta
Mean Median Rank
Median Rank
Mean Median Rank
Mean Median Ronk
Mean Median Rank
0:83
0.38
7.17
6.0
0:35
8:19
7-17
8:17
2.5
EC-TEM
EC-TEMWL
0.3
RC-LR
0.2
0.16
3.0
0.24
2.0
0.43
0.45
5.5
0.19
0.08
EC-LEWL.
0.26
3.83
0.23
0.25
4.67
-0.15
-0.45
0.39
0.41
4.17
0.24
3.67
LNM-TO
-0.55
-0.57
6.5
0.02
0.02
5.0
0.03
0.11
6.83
0.45
0.46
6.88
0.17
0.08
7.6
LNM-LTO
-0.56
-0.3 59
6.17
0.09
0.02
5.5
0.0
0.06
6.5
0:45
0 66
7:17
0.17
0.1
6.67
LNM- CONCAT
-0.8
-0.5
1.07
0.02
0.02
4.0
-2.48
-1.47
1.8
0.28
0.26
2.0
0.35
0.35
1.6
LNM-LCONCAT
-0.79
-0.58
2.0
0.02
0.01
2.38
-1.90
-1.18
8-0
0.91
0.38
2.88
0.20
0.27
8.38
LNM-TEM
-0.74
-0.78
4.6
0.02
0.01
4.0
-0.0
0.05
6.5
0.44
0.
G.O.
0.19
0.1
6.88
LNM-TEMWL
-0.7
4.0
-2.17
-1.49
0.3
1.88
0.32
-0.71
-0.77
5.67
6.83
06
-0.02
6.0
0.19
FNN-LTO
FNX- CON-CONCAT
1.1
4.39
FNN- LCONCAT
-0.59
-0.88
0.02
0.02
1.33
-1.25
-1.17
1.67
0.35
1.17
0.25
0.22
FNK-LE
1.14
0.72
4.33
0.27
0.32
4.17
0.3
0.34
4.0
0.68
4.0
0.15
0.08
4.87
ENN-LEWL
1.19
0.65
4.83
0.27
0.32
4.58
0.24
0.34
0.47
0.47
4.6
0.15
0.08
MLP/MC-TO
0.23
0.22
6.67
0.18
0.25
5.83
0.34
0.333
6:33
0-43
0.43
6.67
0.18
0.08
7.33
MLP/MC-LTO
-0.99
-0.17
3.33
0.11
0.1
1.67
0.5
0.25
7:17
0-41
0.45
6.33
0.19
0.1
83
MLP/ P/MC/ CONCAT
0.03
0.06
4.58
0.2
0.22
0.88
03
-0.15
8.17
0.87
0. 89
8.0
0.24
0.22
88
MLP MLP/MC-LL NCCLLCNNCT
-0.38
-0.28
1.88
0.11
0.1
1.07
-0.17
-0.43
8.17
0-38
0.94
2.67
0.24
0.21
MLP/MC- 0.04
0.03
5.67
6.0
0.28
0.18
6.88
0.19
MLP/MC-TENT
0.24
0.32
2.6
0.28
-0.13
0.22
MLP.MMGLLWW
TOTPP-TO
0.23
6.58
0.00
RMTPP-LTO
-0.25
-0.42
0.05
0.03
AATPP-COOCAT
0.0
0.06
4.33
-1.66
RMTPP-LCONCEE LCONCAT
-0.52
-0.47
1.6
0.05
0.06
1.0
-1.85
-1.57
3.0
0.26
0.27
1.88
0.34
0.52
1.07
RMISP-TEM
0.03
0.04
5.93
0.19
0.22
5.83
0.08
0.05
6.67
0.43
0.44
6.6
0.19
0.11
0.98
MITED-TRMWL
0.08
0.12
1.83
0.18
0.22
5.83
-2.28
-1.12
1.67
0.27
0. 0.29
2.17
0.35
0.37
1.83
RMTPE-LE
0.01
0.02
4.17
0.19
0.22
4.3
0.07
0.03
6.0
0-45
0.44
5.87
0.19
0.11
8.0
RXTPP-LEEWL
0.05
0.12
4.83
0.19
0.23
5.5
-1.24
-1.03
4.0
0.32
0.35
3.07
0.28
0.28
4.0
SA/CM-TO
.0.05
4.67
4.17
0.75
0.92
5.0
0.14
8.33
BATCM-LTO
-0.21
2.67
0.10
SA CONCAT
-0.25
0.2
8.6
SA/CM- CONNCCA
BA/CMA BA/CM-LE
BA/CM-LEWL
SA/MG-TO
1.4
1.0
6.67
0.22
7.0
0.14
0.06
MA/C--TOO
1.21
0.82
0.28
6.0
0.25
0.4
0.06
BA/MC-LCONCAA
BA/MC- CONCAT
BA/MC-CONCAT
0.02
0.03
0.85
0.74
6.67
5.17
0.24
0.24
0.31
0.3
6.39
6.5
0.03
0.1
0.15
0.11
5.17
0.0
0.44
0.42
0.48
0.49
6.88
5.0
0.18
0.18
0.06
0.06
5. 5.88
5.67
5A/MC-TEM
-0.35
-0.3
3.33
0.15
0.14
3.33
0.18
0.05
5.0
0:41
0.44
4.33
0.18
0.08
4.17
BA/MC-TEMWL
-0.25
-0.26
4.17
0.14
0.14
4.17
-0.32
-0.27
2.17
0.37
0.38
2.0
0.22
0.14
1.07
SA/MC-LE
-0.48
-0.42
1.8
0.00
0.00
1.38
0.12
-0.03
4.67
0.4
0.43
3.07
0.19
0.09
3.6
SA/MG-LERWI
-0.45
-0.45
0.1
1.67
-0.1
-0.13
0.37
1.88
0.3
9.1
1.67
51

---

Published in Transactions Machine Learning Research (06/2023)
Table 17: Average and median scores, as well as average ranks per decoder and variation of event encoding, for relevant marked datasets. Refer Section 5.5 for details on the aggregation procedure. Best results are highlighted in bold.
NLL-T
NLL-M
BCB
Mean Median Rank
Mean Median Rank
Mean Mudian Bank
Menn Medman Raak
Mean Median Rank
CONS-EC GRU-BC
2.83
0.46
CONS-LNM
SA-LNM
0.09
2.17
-0.8
0.39
1.50
1.53
GRU-LNM
-0.79
0.01
1.5
-0.89
1.17
0.35
0.37
1.17
1.17
CONE-FNN
0.94
0.61
3.0
0.2
0.23
2.67
0.76
0.41
3.0
0.47
0.48
2.67
0.16
0.07
3.0
SA-FNN
0.68
0.38
1.83
0.19
0.22
2.0
0.15
0.16
2.0
0.45
0.45
1.98
0.17
0.11
1.53
GRU-FNN
0.31
0.32
1.17
0.18
0.21
1.35
-0.18
-0.08
1.0
0.43
0.45
1.5
0.18
0.1
1.17
CONS-MIPP/MC
0-43
0.18
2.0
0:88
0.58
0-44
0.46
3.0
SA-MLP/MC
0.08
0-15
0.17
0.21
0-13
2.0
0.39
0.4
GRU-MLP/MC
-0.12
1.33
0.37
0.199
1.57
00.8
-0.23
1.33
0.58
0.37
1.0
0.17
1.17
SA-RMTPP
GRU-RNTPP
CONS-SA/CM -0.02
-0.01
0.08
0.1
2.17
0.75
2.87
0.45
0.45
2.0
SA-SA/CM
-0.14
0.D8
2.17
0.29
0.45
0.45
1,83
1.57
GRU-SA/CM
-0.16
-0.07
2.0
0.07
0.07
1.07
0.31
0.1
1.6
0.45
0.40
2.17
0.17
0.1
2.8
CONS-SA/MC
0.5
0.52
3.0
0.22
0-95
2.83
0.76
0.64
3.0
0-46
0-47
3.0
0.16
0.07
3.0
SA-8A/MC
0.20
0.23
1.07
0.10
0.21
1.83
0.1
0.13
1.58
0.42
0.48
2.0
0.18
0.08
1.5
CRU-SA/MC
0.28
0.22
1.88
0.39
0.21
1.88
0.02
0.07
1.17
0.42
0.45
1.0
0.18
0.08
1.6
52

---

Published in Transactions Machine Learning Research (06/2023)
Table 18: Average, median, worst scores, and average ranks of the best combinations per decoder on the NLL-T across all relevant unmarked datasets. Best results are highlighted in bold.
Unmarked Datasels
NLL-T
PCE
Median Worst
Mean Median Woret
GRU-BO-TEM
-0.21
-6.25
0.16
1.0
0.08
0.02
0.13
GRU-LNM-TER.
-3.7
-1.03
-0.53
1.47
0.01
0.01
0.01
1.67
GRU-LN-LE
-0.25
-0.45
0:47
6.0
0.02
0.02
0.04
6.67
GRU-PNN-IITO
-1.03
-0.79
-0.88
4.0
0.91
0.01
0.91
2.33
GRC-MLP /MC-LTO
-0.57
-0.87
-9.5
4.67
0.02
0.01
0.08
6.67
GRU-RMPPP TEM
.0.51
-0.54
.0.36
5.97
0.02
0.01
0.06
5.0
BAU-8A/CM-TT
.0.71
.0.56
0.73
5.67
0.02
0.01
0.
6.0
ERC-SA/MC-LE
-0.81
-0.87
0.01
0.01
Harriaa
0.01
0.01
Table 19: Average, median, and worst scores, as well as average ranks per decoder and variation of event encoding, for relevant unmarked datasets. Refer to Section 5.5 for details on the aggregation procedure. Best scores are highlighted bold.
Unmarked Dutaneta
NLL-T
PCR
Mean Medtan Warst Rank
Worst Rank
RO-TO
0:36
0:35
0.6
3.33
0.08
0.08
D:16
2.67
RC-LTO
0:38
0:31
0.61
3.67
0.08
0.01
D.16
4.0
RC-TRM
-0.03
-0.13
0.31
1.0
0.06
0.03
0.14
1.0
EC-LB
0.17
0.04
0.46
2.0
0.07
0.08
0.15
2:38
LNM-TO
-2.19
.0.73
0.01
0.01
LNM-LTO
-2.18
-0.72
2.39
-2.04
-0.82
1.88
1.07
ENM-LE
FNN-TO
0.64
2.38
2.38
PNN-UTO
1.0
1.0
PNX-LR
2.07
0.08
2.67
MLP ULP/MC-TO
-0.03
0.01
0.1
3.38
0.04
0.08
0.08
3.58
MLP.MMC-LTO
-0.21
-0.17
0.06
2.33
0.02
0.03
0.03
1,33
MLP/MMO MLP/MC-TEM
-0.12
-0.14
-0.04
2.33
0.04
0.03
D.09
3.0
MLP/MC-LE -0.31
-0.39
-0.12
2.0
0.05
0.02
0.06
2.39
RMTPP-TO
-0.11
-0.16
0.11
3.33
0.04
D.08
3.33
RMTPP-LTO
-0.15
0.02
0.08
2.58
RMTPP-TEM
-0.20
0.08
0.07
2.39
.0.24
SA/CM-TO
SA/CM-LTO
0.67
3.38
2.33
0.04
2.67
SA/OM-LE
-0.25
0.19
1.67
0.01
0.01
1.0
BA/MC-TO
0.76
0.72
0.96
4.0
0.08
0.07
0.15
4.0
SA/MC-LTO
0.61
0.42
0.80
3.0
0.07
0.06
0.14
3.0
SA/MO-TEM
-0.57
-0.65
-0.39
2.0
0.01
0.03
0.02
1.0
SA/MC-LE
-0.69
-0.8
-0.45
1.0
0.01
0.01
0.02
2.0
53

---

Published in Transactions Machine Learning Research (06/2023)
Table 20: Average, median, and worst scores, as well as average ranks per decoder and variation of history encoder, for relevant unmarked datasets. Refer to Section 5.5 for details on the aggregation procedure. Best scores are highlighted in bold.
Unmarked
NLL-T
Mean Median Wores Rank
Mean Mudian Worst Rank
CONS-EC GRU-EC
CONS-LNM
SA-LNM
+1.94
-0.64
-0.94
GRU-INM
-3.01
-0.93
0.01
0.01
CONS-FNN
0.61
0.37
1.26
2.67
0.06
0.08
0.13
2.0
SA-NNN
0.15
0.10
0.34
2.0
0.05
0.03
0.1
1.33
CRU-FNN
0.05
0.14
0.17
1.35
0.06
0.03
0.11
2.87
CONS-MLP/MCC
0.51
0.03
0.07
2.67
SA-MLP/MC
0.33
2:33
0.04
0.03
0.06
GRU-MLP/MC
-0.44 -0.41
-0.38
0.03
0.02
0.05
SA-RMITP
CRU-RMTTF
CONS-SA/CM
0.02
0.04
SA-SA/CM
0.01
-0.43
0.02
1.33
GRU-SA/CM
0.04
-0.01
1.54
1.88
0.03
0.01
0.05
2.0
CONS-SA/MC
0.54
0.73
0.50
3.0
0.05
0.04
0.09
2.67
SA-8A/MC
0.05
-0.08
0.27
2.0
0.05
0.08
0.08
1.33
CRU-SA/MC
-0.06
-0.10
0.25
1,0
0.06
0.04
0.08
2.0
54

---

Published in Transactions Machine Learning Research (06/2023)
G Critical distance diagrams for event encodings and history encoders
EC-CONCAT FCJI EC-LEWL EC-LCON
EC-LTO FC-O EC-TEM EC-TEWWL
ECTEMNAL EC-CONCAT
DC-TO
EC-TEM
LEWI EC-TEM
TEMWL EC-LCONCAT
EC-LCONCAI EC-LEWI
EC-LTO
DC-LE
LAM LON-CCONCAT LNM-CONCAT INM-TEMMN LNM LEWL
LNMILE LNM-LTO INM-TO UNM TEM
LNM CONCAT LIIM-TEMIWW INNLICONCAT LNM LEWL
UIMTO
UNM-TTM
UNN-EWW LNMATEM
LNIALLE DEMTEMMWL
INN-TO
UNM
IANAGUUNCA FANALE
INN-LWW CONCAT
INHAGUNCAT FAN-LE
INN-LUPI FAN-TO
FNN-LCONCCT FNN-CONCAT FNN LEWIL
FRN-TO FNN-LE FNN LTO
MUPHMC-TO MESHACEEE
MEPINCCICC MLNOC-LE
MURNICOO
MURNC-CONCAT NIPME--FONEETT
MLAMAC-TEMAL
NLP-MC-LTO NUPMC-TO NIRMMSTTM
MEAMOAE
RMTPP PNTPP LE
INTONCCCNNCA
INTII-TEMMAL INTERICCOCAA ANTR--LEW
RMTRRAAL IMTERPALOO RMT-P-TEM
RMTPP TEM EMTPPATO
RATPR
CONER-CONEAT
SACMALE
SMENTO
SANTALL
SATIN-FWW
CATI-CCNNCT SACIGLEW
GATTATO
SAICAATO
SMON-LCONERT
SAIEMITO
SACIKTTO
SANENCOONCT
SAIMC-LE SANNCTTM SNNC-TEMWIL
SAMO-II CANG-GMMAT SANCLTO SUNCLUCONCT
SAMIC-LE
SARICIDD
SARKCTENVIL SANN LEWIL
SAMO
MAUNC-LEDNEAT SAINC-LT
SANG LOONCA"
SAMOCTEM
SEWN
SANC CONCAT
SAIM TO
(L) NLL-T.
(2) PCE.
NLL-M.
Figure 15: Critical Distance (CD) diagrams for the NLL-T, PCE and NLL-M the 0.10 significance level for all event encoding-decode combinations of Table The combinations' average ranks are displayed top, and bold line joins the combinations that not statistically different.
55

---

Published in Transactions Machine Learning Research (06/2023)
EC-CONCAT EC-TEMWLL CONCAT EC-LEWL
EC-ITO EC-TO
EC-TEMVLL EC-CONCAT ECALEWL
EC-TO
EC-TD
EC-LE
LNM CONCAT NIM-F-MWI UNM LOONCAT LNM-LCWWL
LNM-LTT NM-TO UVM LE UNM-TEM
LNM-CONGAT INM-TFMMI LNM LCONCAT UNM-LOWL
LNMMTT NASTTO UNM (NM-TEM
INN-LDO DONCAL
PAN-OO
PAN-CCOINCA PAN-CONCAT
MURINC LCONCCT MURIMC-TEMUL
MLAINC M.. MLRNNCLTO
FALL MURMACLCCNIC
NLPIMC TEM
IMTRSIC INTI9-TEM AMTRPAEE
RNST INECCCCNCCA AETR-LLEWL
AMTRR-CONCAI
RMIPP-EEUL
RMV PI-LE
SAUCHATO SARTM-TOACAT
SAIEN-TO
SACMM LTD
SAICH LE SAVCW-OORCAT
SAINC TENWIL SAIMC LEW.
SAMMC TO SANNI CONCAT SANNC-LCONCAT
SAIMC TEMAL SAMK LEW SAMCCTEM
SAINC TO SAINC LT0 SAINC LONCAAT
SAIN-TTM
(1) ECE.
(2) F1-score.
Figure Critical Distance (CD) diagrams for the ECE and Fi-score the 0.10 significance level for all event encoding-decoder combinations of Table 4 The combinations' average ranks displayed top, and bold line joins the combinations that not statistically different.
56

---

Published in Transactions Machine Learning Research (06/2023)
7
GRU-EC SA-EC
CONS-EC
GRU-EC SA.EC
CONS-EC
GRU-EC
CONS-EC
SA-EC
GRU-LNM SA-LNM
CONS-LNM
GRU-LNM SA-LNM
CONS-LNM
GRU-LNM
CONS-LNM
SA-LNM
GRU-FNN SA-NNN
CONS-FNN
GRU-FNN SA-TNN
CONS-FAN
GRU-FNN
CONS-FNN
SA-NNN
ALRIMC SA-MI P/MC.
CONS MURIMC
GRU-NLP/MC CONS- MLPIIC
S4 MIF/MC
MLR/MC
PYMC
CRU-RMTPP
CONS -MMTPRMMPPPP
CRU-RNTPP
CONS-RMTE RMTPP
GRU-RMTPP
SA-SA/CM SA/CM
CONS-SACC
CRU-SAICM CONS-SAICM
SA-SACCM
GRU-SAICM SA-SAJCM
CONS-SAICM
GRU-NAIMC SAIMC
DONS-SAMMC
GRU-SAMUC SA-SA/MC
DONS-SAMC
GRI-SAJMC SA-SA/MC
DONS-SAMC
(1) NLL-T.
(2) PCE.
(3) NLL-M.
Figure 17: Critical Distance (CD) diagrams for the NLL-T, PCE and NLL-M at the a = 0.10 significance level for all event history encoder-decoder combinations of Table 5 The combinations' average ranks are displayed on top, and bold line joins the combinations that not statistically different.
57

---

Published in Transactions Machine Learning Research (06/2023)
GRU-EC- SA-EC
CONS-EC GRU-EC SA-EC
CONS-EC
GRU-LNM SA-LNM
CONS-LNM GRU-LNN SA-LNM
CONS.LNM
GRU-FNN SA-FNN
CONS-FNN SA.FNN
GRU-FNN
CONS-FNN
MIPPIMC SA. MLP-MMC
MIRPIM
GRU-M SA NLPNIM
CONS MUFMC
GRU-RMTPP
CONS-RMTIP
GRU-ANTFP
CONS-RMIPP
SA-SA/C
CONS-SAICM
GRU-SA/CM
GRU
SA-SAICM
CRU SAIM
CUNS SAMM
GRU-SAING SA-SA/MC
SAWC
SA-SA/MC
(1) BCE.
(2) F1-score.
Figure 18: Critical Distance (CD) diagrams for the ECE and Fi-score at the 0.10 significance level for all event history encoder-decoder combinations of Table 5 The combinations' average ranks are displayed on top, and a bold line joins the combinations that are not statistically different.
58


Reviews:
[{"review_id": "E0IIeXQDYB", "reviewer": "Reviewer_iQsN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n\nGiven the complexity of many temporal point process models, it is helpful to understand when different architectural components should be deployed, and when they should be avoided. \n\nWeaknesses\n---\n\nWhile the paper does provide an incredible amount of detail on empirical results (and what performs where), it is slightly lacking with regards to distilling insights into the various architectural components – particularly for counter-intuitive results, and best practices on what pipeline to deploy on a brand new dataset/problem. In addition, the paper also falls short of cleanly benchmarking predictive performance across neural TPPs, given the very dataset-specific outcomes, with no attempts to reconcile the underperformance on weaker datasets beyond saying that they should be avoided for neural TPP models. Main questions/improvements described below:\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper benchmarks a wide range of state-of-the-art neural temporal point process models on a variety of datasets for a mixture of different tasks – flagging which architecture combinations work well on specific datasets. Furthermore, an in-depth analysis on the calibration of Neural TPP models is performed, finding that mark distributions for most models are poorly calibrated.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\n1.\tDetailed clarification is required on the precise prediction problem that we’d like to solve, along with the specific benchmark metric we should be using to rank models on it. While marked TPPs can forecast time-to-event problems and dynamic classification problems separately, there are other model types which can be used to solve each separately as well (e.g. time-to-event models and standard classifiers). With regards to NLL metrics, how does this reflect the predictive accuracy of the model, and does this account for the sparsely observed labels (in time)? Assuming the goal is to forecast both jointly, is time prediction or event prediction more important?\n\n2.\tAssuming that performance does vary based on dataset type, which models are more suited to what dataset characteristics? Can this be backed up by a data analysis of the underlying datasets used?\n\n3.\tCounter-intuitive results – a) why do neural forecasting models underperform in some datasets? b) why do self-attention architectures underperform recurrent ones? Seems to run contrary to findings of Transformer Hawkers Process (ICML 2020) where recurrent marked TPPs underperforms THPs.\n\n4.\tRelated to the above, are 5 random search iterations enough for hyperparameter optimisation? Should learning rates be tuned and what regularisation (e.g. dropout) used on the models? How do results look if the settings from the original papers are used? \n\n5.\tI might be missing something, but are discrete event labels and inter-arrival times the only inputs into the model? \n", "broader_impact_concerns": "--", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ArMdxXb5Bu", "reviewer": "Reviewer_prAn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an outstanding work: the overview in sections 2 and 3 can serve as a definitive reference on the topic, the experimental evaluation raises the bar in terms of empirical rigor compared to existing works, and the findings of the paper have important implications for future research in the field of neural TPPs.\n\n- **Detailed survey**: The paper provides a unified description of the existing neural TPP architectures using consistent notation, which will be a useful reference for future work.\n- **Significance**: The paper sheds light on the importance of various design choices in neural TPP architectures. This is interesting from the research perspective, evaluating the progress of the field by comparing proposed methods in a unified setup and highlighting promising future directions. This can also open the door for new applications of neural TPPs, helping practitioners pick among the many existing models. The findings on the inadequate quality of many datasets currently used in the literature will hopefully lead to development of more challenging benchmarks inspired by real-world applications.\n* **Empirical rigor**: Multiple aspects of the experimental evaluation in the paper make it stand out among prior works on neural TPPs. These include:\n\t* Including various sanity-check baselines to provide a reference point for the performance of more sophisticated models\n\t* Performing visual analysis of the data (Figure 2) and individual model outputs (Figure 3) to better understand the reasons for good/bad performance of various models\n\t* Using a large number of datasets (15)\n\t* Using significance tests when reporting the results, instead of simply considering the mean score\n\t* Novel task - investigating the calibration of the probabilistic predictions produced by the models\n* **Presentation**: Despite its length, the paper is easy to follow and the concepts build on top of each other. The experimental results are presented clearly, and a large set of experiments are summarized with a few key findings.\n\nI haven't found major weaknesses in this paper, except several partially incorrect statements that need to be adjusted. These are listed below under \"requested changes\". One aspect of the evaluation procedure (normalization of the data) can have a significant effect on the results, but is not described clearly in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work performs a large-scale study of neural TPP models on various predictive tasks related to continuous-time event data. \n\nThe paper starts with an overview of the existing approaches, describing the important architectural choices when defining TPP models. These choices include representation of events, history encoding and time/mark decoders.\n\nNext, a large-scale evaluation on 15 real-world datasets is performed to investigate the effects of these architectural choices on the predictive performance. Besides the standard evaluation metrics such as time/mark negative log-likelihood (NLL), the models are evaluated based on the calibration of their predictions. Finally, additional analysis produces two surprising findings: (1) in many cases the models achieve competitive results even without considering most of the event history and (2) many of the benchmark datasets used in the literature can be equivalently using trivial baseline models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Clarify the scope & setting: \n\t1. Mention that the paper only considers categorical marks, while other choices are also possible.\n\t2. Only the problem of mark prediction *conditioned on time* is considered. This means, the evaluation does not consider the question \"What will be the type of the next event?\" (page 6), but rather \"What will be the type of the next event, given that it occurs at a certain time?\"\n2. Equation 6: The statement is only true for Poisson processes, where the compensator does not depend on the history. However, in the general case the right-hand side is ill-defined: the arrival times $t_1, ..., t_{i-1}$ are random variables, so we cannot express the expected # of events at time $t$ as a function that depends on the observed arrival times. If we decided to condition on the observed arrival times $t_1, ..., t_{i-1}$, we would end up with the following expression $$\\mathbb{E}[N(t) | t_1, ..., t_{i-1}] = i - 1 + \\Pr(t_i \\in [t_{i-1}, t] | t_1, ..., t_{i-1}),$$ which is not very insightful.\n4. The discussion in paragraph **Parametrization of TPP models and Learning** is only valid if we assume that we are dealing with non-terminating TPPs - for a terminating TPP assumptions $R_2^\\lambda$, $R_2^f$, $R_3^\\Lambda$ are violated. It's worth noting that this is the setting considered in the paper. We need to include the termination probability (i.e., violate these assumptions) if we model finite sequences of events on $[0, \\infty)$ (instead of a bounded interval $[0, T]$, as in the paper), since otherwise any finite sequence would have the NLL of $-\\infty$.\n5. Scaling of arrival times to the interval $[0, 10]$ in the experiments (page 18): Are the arrival times scaled individually for each sequence in the dataset? What if different sequences are observed on intervals of different length $T$? In such case, the chosen scaling procedure can distort information - for example, if all sequences were generated by the same renewal process on intervals of different length, the ground truth renewal process would not perfectly model the rescaled event sequences. The normalization procedure should be described in more detail.\n\n\nMinor comments:\n- Page 2: solemnly -> solely\n- Before Equation 15: inter-arrival times\n- Paragraph **Encoding the mark**: The two works mentioned at the start of the paragraph also use learned embeddings for each mark, and are equivalent to the remaining approaches discussed in the paragraph (since one-hot encoding followed by a linear layer is equivalent to an embedding obtained from a learnable embedding matrix).\n- Equation 18: The initial state $h_1$ is either fixed (e.g., all zeros) or is a learnable parameter, but usually doesn't have an associated distribution. If we assume such a distribution, we would need to perform poster inference over the state (e.g., with variational inference) and thus won't be able to compute the log-likelihood in a closed form.\n- Equations 23, 25, 26: Potentially use notation $\\lambda_k^*$ instead to highlight that the rate is constant over $[t_{i-1}, t]$.\n- Equation 27: Highlight that the state $\\boldsymbol{e}$ depends on the current time $t$.\n- Equation 28: $\\approx$ instead of =\n- Section 5.1: Why are \"raw inter- arrival times\" abbreviated to TO, and \"logarithms of inter-arrival times\" abbreviated to LTO?\n- Page 23: Potentially worth point out that the CONS encoder effectively reduces any neural TPP model to a renewal process.\n- Page 25, last paragraph: Does Hawkes outperform approaches due to the fact that it's the only decoder that does not assume conditional independence between time/type of the event?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BdPYlWpaN3", "reviewer": "Reviewer_ZZrh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths\n---------\n\n* comprehensive benchmark of recent neural TPP approaches, with reasonably thorough and systematic experimental practices\n* focus on calibration was welcome, I wish more works would assess this\n* use of Critical Distance diagrams was appreciated\n* qualitative evaluation in Fig 3 helps provide qualitative insight (fast rise in $\\lambda$ after an event indicates flexibility of neural models)\n\nWeaknesses\n----------\n\nI list short phrases about weaknesses here, with longer discussion further down this review form.\n\n- W1: Why include datasets you identify as borderline inadequate?\n\n- W2: Computational time tradeoffs not assessed at all\n\n- W3: Tough to parse main messages from Tables\n\n- W4: Presentation quality of math background (Sec 2) could be improved (see detailed comments below), to make more accessible to readers new to TPPs\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "\nThis work pursues a large-scale systematic evaluation of neural temporal point process (TPP) methods.\n\nThe major contributions seem to be\n\n1) a systematic evaluation\n\n- larger than previous works in terms of size (15 datasets)\n- covers calibration as well as likelihood performance\n- designed to pinpoint effect of each architectural component (event encoder, history encoder, decoder)\n\n2) several insights from the evaluation could inform future practice\n\n- several datasets studied here are discouraged as good benchmarks for neural TPPs due to small variability between small parametric and large neural models\n- fixed-context-length encoders of only the last few events do about as well as encoders of arbitrarily long history\n- mark distributions are not well calibrated by neural TPPs (basic parametric models do better)\n- self-attention needs to use vectorial representations of time for good performance\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Content Changes (critical)\n-----------------------------------\n\n## W1: Why include datasets you identify as borderline inadequate?\n\nI like the critical eye this work puts on existing datasets on page 31, where 7 of the 15 datasets (MIMIC2, Stack overflow, Taxi, Reddit Subs, and 3 more) are identified as potentially inadequate because \"all decoders achieve comparable performance\"\n\nHowever, this leaves me confused, because all the earlier tables about ranking methods used these datasets. \n\nI'm eager to hear from the authors: Why not take a stronger stance and remove these less-suitable datasets from your own experiments? Would any of the conclusions from earlier sections change?\n\nIn my mind, steering the community away from un-interesting datasets could be a valuable contribution here, but the fact that you've used these datasets in all experiments seems to perpetuate these datasets. A reader who isn't careful to read all the way thru page 31 would totally miss the recommendation to not use these.\n\n\n## W2: Computational time tradeoffs not assessed at all\n\nA big missed opportunity here is the lack of reporting of runtime or othe resources required for each method. Given that there isn't often a clear \"winner\" in terms of the critical distance diagrams (Fig 4), in practice many readers will want to know how to manage tradeoffs in terms of performance (likelihood/calibration) vs resources required (training time). \n\nI wonder if the authors could comment on this: can you say anything about which methods are recommended from a runtime perspective? \n\nTo be clear: I don't necessarily want you to go run more experiments, but I wonder if there's some insight you can extract from the experiments you already did that would be useful.\n\n\n## W3: Tough to parse main message from Tables\n\nI have to say I found Tables 4 and 5 rather tough to parse the main takeaway messages with so many methods listed\n\nI wonder if these can be punted to appendix, and the main paper can focus more on the same kind of information but converted to Critical Distance diagrams like in Fig 4.  Those seem much easier for readers to browse to get the main idea about which approaches are working truly better than others versus similar to others.\n\n\n## W4: Requested Presentations Changes (critical)\n\nRegarding W4, as someone quite familiar with probabilistic modeling in general, but who doesn't work directly with TPPs often, I felt the presentation in Sec 2 could be improved. \n\n#### 1) adjust text around Eq 1-3 to better introduce quantities like the MCIF $\\lambda(\\cdot)$ to unfamiliar readers. \n\nI suggest first introducing f and F from Eq 1 as a pdf and a CDF over times in interval $[t_{i-1}, \\infty]$, then explaining how the MCIF $\\lambda$ is defined in terms of these. The \"instantaneous probability\" aspect was confusing to me, because it seems at first from Eq 1 you can treat $\\lambda$ as a PDF, but then in requirement $R_2$ later we find it integrates to infinity, not one. I feel this integration requirement should be better explained. \n\n#### 2) typo in Eq 3, replace \"] t_i-1, t]\" with \"[ t_i-1, t]\"\n\n#### 3) adjust explanation around Eq 5 of the cumulative MCIF $\\Lambda$\n\nthe exp term here seems unmotivated, but perhaps at least a pointer to page 8 of the lecture notes of Rasmussen (2018) would clear things up, as you can define $\\lambda(t)$ as the negative derivative of the log of 1 -F wrt t.\n\n#### 4) clarify in Eq 6 the sum of $\\Lambda$ terms implied by the \"...\"\n\nwhat time values are provided, just t_1, t_2, ... t_i-1, and then t? the bounds of the sum are vague to me, perhaps because the first t has a subscript but last one doesn't\n\n\n\n\nMinor Presentation Changes (suggested)\n--------------------------------------\n\npage 2: \"solemnly evaluated\" should maybe be \"solely evaluated\"\n\npage 6: the footnote defining e_i is awkwardly placed (looks like a superscript of the $\\mathbb{R}$ term)\n\naround Eq 10-11, should be clear that in addition to a given sequence of n events, you need to specify the max time T of your observation window\n\ntext above Eq 12: should say \"the **expected** next arrival time is ...\"\n\nEq 13: is the denominator needed? \n\nFig 3: x-tick labels needed to help appreciate timescale of this plot\n\n", "broader_impact_concerns": "No concerns. I don't think a Broader Impact statement is necessary here, though there could be one.\n\nSome data analyzed here do originally arise from humans (e.g. the health records data in MIMIC2, the social media data in Reddit).\nHowever, these datasets seem to be suitably de-identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agreed that the paper is well written, and its claims empirically supported, and quite detailed.  Reviewers also agreed that this work is of interest to the TMLR audience, and that this work is a valuable survey of neural temporal point process methods -- we recommend it for a survey certification. 

A lingering criticism is the translation of empirical evaluations and comparison into insights about methodological choices.  Tables 4 and 5 remain difficult to parse -- perhaps the caption could highlight a few important points or insights that the results in the table reveal.
Decision: Accept (poster)