Decomposable Transformer Point Processes
Authors: Aristeidis Panos
Venue: NeurIPS 2024 poster (2024)
Keywords: point processes
Primary area: probabilistic_methods

Abstract: The standard paradigm of modeling marked point processes is by parameterizing the intensity function using an attention-based (Transformer-style) architecture. Despite the flexibility of these methods, their inference is based on the computationally intensive thinning algorithm. In this work, we propose a framework where the advantages of the attention-based architecture are maintained and the limitation of the thinning algorithm is circumvented. The framework depends on modeling the conditional distribution of inter-event times with a mixture of log-normals satisfying a Markov property and the conditional probability mass function for the marks with a Transformer-based architecture. The proposed method attains state-of-the-art performance in predicting the next event of a sequence given its history. The experiments also reveal the efficacy of the methods that do not rely on the thinning algorithm during inference over the ones they do. Finally, we test our method on the challenging long-horizon prediction task and find that it outperforms a baseline developed specifically for tackling this task; importantly, inference requires just a fraction of time compared to the thinning-based baseline.

Decomposable Transformer Point Processes
Aristeidis Panos
University of Cambridge ap23138cam.c ac.uk
Abstract
The standard paradigm of modeling marked point processes is by parameterizing the intensity function using an attention-based (Transformer-style)) architecture. Despite the flexibility of these methods, their inference is based on the computa- tionally intensive thinning algorithm. In this work, we propose a framework where the advantages of the attention- beceed architecture the thinning algorithm circumvented. The framework depends on modeling the conditional distribution of inter-event times with a mixture of log-normals satisfying a Markov property and the conditional probability mass function for the marks with Transformer-basee architecture. The proposed method attains state-of-tte of-tte-art performance in predicting the next event of a sequence given its history. The experiments also reveal the efficacy of the methods that do not rely on the thinning algorithm during inference over the ones they do. Finally, we test our method on the challenging long-horizon prediction task and find that it outperforms baseline developed specifically for tackling this task; importantly, inference requires just fraction of time compared the thinning- based baseline.
maintained and the limitation
Introduction
Continuous-tinn time event sequences are commonly found in real-world scenarios and applications such as financial transactions [3], communication in social network [34], and purchases e-Commerce systems [16]. This abundance of data for discrete events occuring at irregular intervals has lead to an increasing interest of the community in the last decade marked temporal point processes which are the standard way of modeling this kind of data.
Historically, Hawkes processes [14] and Poisson processes [7] have been extensively applied to various domains such as finance [13], seismology [15], and astronomy [1]. Despite their elegant mathematical framework and interpretability. the strong assumptions of the models reduce their flexibility and fail to capture the complex dynamics of real-world generating processes.
Advances in deep learning have allowed the incorporation of neural models like LSTMs [17] recurrent neural networks (RNN) into temporal point processes [5. 12, 24, 26, 29, 35, 38). As a result, these models are able learn more complex dependencies and attain superior performance than Hawkes/Poisson processes. Recently, the introduction of the (self-) attention mechanism [36] to modeling temporal point processes [41, 42, 44] has led new state-of-the-arr methods with extra flexibility.
Despite the advantages of these neural- based models, their dependence modeling the conditional intensity function creates limitations for both training and inference [35]. Training usually requires a Monte Carlo approximation of an integral that appears in the log- likelihood.. [29] proposed method to circumvent this approximation; however, the main shortcomings remained as discussed [35]. More importantly, inference is based the thinning algorithm [21, 22] which is computationally intensive and sensitive to the choice intensity function. To deal with these downsides, [35) parameterized the conditional distribution of the inter-event times by combining log- lor-normal mixture density network
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

with an RNN. The model's performance is comparable to that of the other intensity- based methods which use RNN/LSTM architecture but still inferior the Transformer-basee methods.
more recent work [30] has referred to the decomposition of the log-likelihood of a marked point process [6] to parameterize the distribution of marks given the time and history and the distribution of times given the history. This decomposition, as with [35], climinates the need for the thinning algorithm and additional approximations, while offering a rigorous, yet flexible framework for defining different distributions for occurrence times and marks. [30] used two different parametric models for each distribution, and their results for the time prediction task, despite the simplicity of their framework, were competitive or superior to neural- based baselines.
Inspired by the state-of of-tee----h--rr performance of the Transformer- based architectures and the com- putational efficiencyffeexiilitt of the intensity-free models, we develop a model for marked point processes that combines the advantages of these two methodologies. Our contributions are summa- rized below:
We propose novel model that is defined by two distributions: a distribution for the marks based on a Transformer architecture and a simple log-normal mixture model for the inter- event times which satisfies a simple Markov property.
Through extensive experimental study, we show the efficiency of our model in the next- event prediction task and the suitability of the intensity- free models for correctly predicting the next occurrence time over the methods relied on the thinning algorithm.
To the best of our knowledge, we are the first to experimentally show the limitations of the thinning algorithm on the predictive ability of the neural point processes.
We test our model the more challenging long- horizon prediction task and provide strong evidence that we can achieve better results in a fraction of time compared to models that have been specifically designed solve this task and, uncoincidentally. depend on the thinning algorithm.
2 Background
marked temporal point process (MTPP), observed in the interval (0, T), is a stochastic process
whose realizations are sequences of discrete events occurring at times 0 < t < < tN < T with corresponding event types (or marks) ky...., kN. where K E (1....,K] K]. The entire sequence is denoted by Hr = (V/,,k),..., (Ix, kw)]. The process is fully specified by the conditional intensity function (CIF) of the event of type l at time / conditioned on the event history Hi, {llg, kz) 1t < 4), (t) == >u(t HH,, 2 0,t>> > 4;; we use the asterisk * to denote the dependence on He. The CIF used to compute the infinitesimal probability of event & occurring at time i, Le. XE (t) IP (t++1 E [t,,+ + dt), ki+1 = k t++1 & (t,,t), He.). The log- likeliood of such an autoregressive multivariate point process is given by [14, 22]
N ((Hr)) = Σ AL)
K
- Σ A1( dt. Jcm
i=1
Modeling the intensity function by flexible model and then learning its parameters by maximizing
Eq. (I) has been the standard approach of many works [5, 12. 24, 26, 29, 35, 38, 41, 42, 44].
An equivalent way of deriving the log-likelihood in (1) without the use of > (t) is by following the decomposition of a multivariate distribution function in [6] (expression 2), expressed as
NN
((HT) = Σ flogp" (ks | t,) + log (t.)) + log (1 - -((T) Hew)),
where p" (A: (2) == p(k f,, H and (*) := == ft He.) are the conditional probability mass function (CPMF) of the event types and the conditional probability density function (CPDF) for the occurrence times, respectively. F(t H.,) = f (t) Vt > t, is the cumulative distribution function of (t). The last term in (2) is the logarithm of the survival function that expresses
'The notation of is slightly different compared to X but the definition remains consistent.

---

the probability that no event occurs in the interval (tv,T), The relation between (t) and the density/PMF is given by (t) ("(t)p"(kkt see Section 2.4 in [33].
1-F(1
We can represent the temporal part of the process terms of the inter-event times 7; := = 1 e R., to = O; the two representations are isomorphic and the relation between the conditional PDF of the inter-event time Ti until the next event and the conditional intensity function is given by
g" (Ti) == (nr H..) Ex= (g(t--1 + +1) exp
Ek= Ap(t + z)dz) (t,).
3 Decomposable Transformer Point Processes
To develop our proposed framework Decomposable Transformer Point Process (DTPP), adopt the decomposition in (2) and model (T) and p' t), separately. Despite the advantages of modeling the intensity function and the arguments in favor of this [9], we believe that modeling the probability density/mass function offers not only the same benefits as modeling the intensity function discussed in [35], but, more importantly, it allows us not to depend on the thinning algorithm
during inference. The technical details of each model are described in the next two sections.
3.1 Distribution of Marks
The conditional distribution of the event types is parameterized by continuous-time Trans- former architecture as the one described in [41]. More specifically, for any pair of events k). we evaluate an embedding (t) E IR based on the history He. Assuming an L-layer ar- chitecture, (t) is given by the concatenation of the embedding of each individual layer, i.e.
h (t) = h ht):
(0) (
(0) (t)]. (t): "()) The embedding of the base layer h (t) is independent of
time and learned by a simple weight vector for each mark, i.c. h2 (t) == h embedding of layer & {I,...,L}} for (t, k) is defined as
ERD The
(1) OK. (1,t,k)
h (1) = hg (1) + tanh
h
Σ
E CRDO
1+C
(HH,
where C > 0 is the normalization constant given by C Eenndeen Ok (tz, k:) and the unnor- malized attention weight is
e k (L,kk) exp
k. (T) (1) 0.
The operation of the non-linear activation function tanh element-wise and D = Σ/ D
The query, key, and value vectors 4k (L) (1), and vt ((), respectively, be computed by using the embedding of the previous layer and the corresponding weight matrices Q11), k(()) (D+(D((()- v(t) E RDC x(D+D(t-)) as follows,
de (t) = Q(0 (0) = x(Ox, (0) (t)= v(0)=
(5)
where X = 1 (1) E DD By 2(1) E R.D, we denote a temporal embedding of time defined as
cos (1/10 ((d11
if d odd,
(z(t)
(6)
sin (/100 d odd,
where d = O...., D 1. This encoding is the same as in [44] with small differences than the one used [41] where we found empirically the former to work slightly better than the latter. For a more detailed discussion regarding the architecture of the model and how it compares to previous Transformer-basee methods, see Appendix in [41]. Finally, we note that for extra model flexibility, multi-head self-attention can be casily obtained by the three equations in (5).
3

---

Having computed the top-layer embeddings hy) (t) for all A = 1,..., PMF p" (k |t as
,.....,, K, we model the conditional
exp Ww hx))))
p p'
Σ exp (w, hi()))
where wa are the learnable classifier weights. As is typical for these architectures, to avoid any data leakage from future events, we mask all future events (t,, kj) where t < tt and only use previous events for computing these embeddings.
3.2 Distribution of Inter-Event Times
For the modeling of the inter-event times, since they always take positive values, we choose a mixture of log-normal distributions whose parameters depend the value of the previously seen mark. Specifically, given that the previous occurred mark is k; the PDF2 of the next inter-event time r is defined as
(
(R) log pul
1
gT(T) - g( 1b) Σ
(k) T8m 22T
exp
-
(k)
8m
where {wm Jm=1 A are the mixture weights, {pm Jm=1 R are the mixture means, and (o9h) AM m== IR4 are the standard deviations, for any = 1,.... K. The log-normal mixture has several desirable features that justifies choice: (i) efficiently approximates distributions in low dimensions such as I-d distributions of inter-event times [23, 35] while satisfying a universal approximation property that provides theoretical guarantees regarding its approximation ability [8], (ii) closed-fome form moments are available and can be used for predicting the next time; for instance, the mean of the distribution is given as the weighted average of each of the log-normal means, i.e.
(8)
(k) LM
(R))2
(sm
E [T]
B
Σ
W
exp
Pm
(k)
(9)
Jm=1 can be done in fraction of time using fast off-the-shelf implementations based on the EM algorithm [10]. Finally, note that the dependence of the model only on the most recent mark implies Markov property since we do not need the entire history Hls to define our distribution.
(iii) learning the small number of parameters {wm , .4m
(k) (k) (b) LM
At first glance, this assumption might seem restrictive when it comes to capturing the complex dynamics of the process. Nevertheless, this assumption holds only for g" while p" is modeled by the flexible Transformer architecture that models the full history up the current time t. Hence, we do sacrifice any modeling power at all achieve efficiency. On the contrary, we can maintain both modeling power and computational efficiency due to the decomposition in (2) and the chosen models (8) and (7). Moreover, as our extensive experiments on the real-world data show, this assumption provides robust predictive model which is less prone to overfitting compared to more flexible neural net architectures since the Markov property can act as a strong regularizer.
3.3 Training and Prediction
The parameters [wm Am 8m m, of 9" (T) and the parameters (We, h Q K(O,, v(0 Fe of p" (le t) can be estimated by maximizing the log-likelihood in (2) using any stochastic gradient method. A crucial benefit of using the decomposition (2) is that it permits us to learn the above parameters separately follows:
(k) (k)
(00) mm, argmax (w pm
logg" (Tx) log (1 G(T) Hr))
(10)
[wg, (0)"
Q10 K(0' v(0 Je, = argmax
log p (ki tz);
(11)
To avoid notation cluttering, and k in lieu of and k;-1, respectielly to define the PDF

---

where G(T]
g" (r) This is major difference from previous work where the parameters the neural net parameterizing the conditional intensity function had to be learned all at once. By dividing the main objective function into two sub-objectives, since there is parameter- sharing between the two models, we can maximize the two objectives independently, and thus, having an easier optimization task than maximizing single set of parameters of given objective, such as (1).
The trained models g" and p* can now be used to predict either the time/type of the next event (next- event prediction) or the next P > 1 events (long-horizon prediction). For the rext-event-predictoon the predicted time f given the history He, is computed by using the mean of the appropriate mixture of log-normals while the corresponding predicted type of this event is evaluated based on He, and the true time 4441. i.e.
t + +
k = argmax p" (k t(41).
(12)
The above procedure is based on the minimum Bayes risk (MBR) principle [24] which aims to predict the time and type that minimizes the expected loss. This is an average L2 loss in the case of time prediction (deriving a root mean squared error) and an average 0-1 loss for the type prediction (deriving an error rate).
For the long-horizg longhhorzoon prediction task [11, 40], we need to predict sequence of events where, unlike the next-event prediction task, we do not have access to the true time when we predict the next event type. This could potentially Icad to a cascading error effect due to the autoregressive nature of the models designed for the less challenging task of next-event prediction. This is because after an error is made in the sequence of the predictions, it cannot be corrected, and thus the error accumulates and affects all subsequent predictions. We argue that this pathology can be alleviated by using model of times that provides accurate and robust predictions given the history. This assumption is verified experimentally in Section 5.2. To generate predicted sequence, we require the trained models g() and p() to sequentially predict events as in (12). Since we have no access to the true time 4,+1, we use as proxy the prediction i to predict 1 in turn. After the prediction of the new event, we append it to the history and then we repeat the same step given the updated history until we generate sequence of P events. The exact procedure is described in Algorithm 1 in the Appendix.
The main advantage of Algorithm I over other methods [40] that are based on the thinning algorithm is its computational efficiency. The algorithm is fully parallelizable, and it can produce single steps in parallel for a batch of event sequences. This is not possible for thinning-basee methods that require one to consider single sequences each time [40]. Consequently, our method is able to generate sequences orders of magnitude faster, which is verified by our experiments. Unlike other competitors [40] that are based on the thinning algorithm and therefore require random sampling, our algorithm fully deterministic; for comparison the thinning algorithm described in Algorithm 2 of the Appendix.
4 Related Work
The decomposition in (2) has been used in the past to provide both expressive and interpretable models 30J. For instance. [30] model the mark distribution with parametric model, inspired by the exponential intensity function of Hawkes process and the time distribution with a single log-normal; however, they use the mode instead of the mean of the distribution for predicting the time of the next event. They learn the parameters of their models separately, we describe in (10) and (11), but they use Variational Inference [4] to Iearn the parameters of p". They attain competitive results in terms of next-time prediction, but the model lacks the flexibility of a Transformer-base architecture, as our experiments show.
[35] is another work that takes advantage of the mixture of log-normal distributions to model the distribution of the inter-event times. The model based on an RNN architecture that produces a dred-dimensional embedding of the event history, which is used to generate the parameters of the mixture model, and the same embedding is employed to define the CPMF of the marks. In our case, we use a Transformer architecture to obtain this history embedding, which is utilized by the CPMF, exclusively. Finally, the proposed model in [35] assumes that the marks are conditionally independent of the time given the history, which is not the case for our framework, as is evident in (2).
5

---

Avg. Logellelllooo
-25
Aog LegeMolloood
Ang Log Hldiinod
-7.5
0.2-
10.0
0.1-
-125
THP
SauP
Astur
IPPp
DTW
THP
SAIP
1FTP
THH
SMP
Asil
(a) Mimic-II
(b) Amazon
(c) Taxi
1.25
2.0
Ang Log.iiliiiod
1.00-
Ang. Log: Hoclhooo
-22
0.75
-2.4
0.50-
0.25
-88
6.00
TRP
SAHP
UFTPP
Viorp
Emp
-3.0
THP
SAUP
A-MPP
IFTPP
11-0pp
DTTP
(d) Taobao
(e) Blackvvrfloo
Figure 1: Goodness-of-ff evaluation the five real-world datasets. We compare DTPP model
against five strong baselines. Results (larger is better) are accompanied by 95% bootstrap confidence intervals.
Finally, the CPMF of the marks for DTPP model shares the architecture the Attentive Neural Hawkes Process (A-NHP) [41]. Nevertheless, they use it to model the CIF while in our case model p".
5 Experiments
We considered two different tasks to assess the predictive performance of our proposed method: tifttess-offftttnex--eeeeeee prediction and long-horizon prediction. We compared method DTPP to several strong baselines over five real-world datasets and three synthetic ones. Description and summary statistics for all datasets used in this section are given in Appendix A. For the competing methods, we used their published implementations; more details are given in A.3. Experimental details not available this section can be found in Appendix A. Our framework was implemented with Py Torch [31] and scikit-learn [32]; the code is available at https: / //ggtttttbb///e com/ aresPanos/ dtpp.
5.1 Goodness-of-Fis Next-Event Prediction
We evaluated our DTPP model to determine how well generalizes and predicts the next event given the history the held-out dataset. For comparison, used five state-of-the baselines where the three of them model the CIF using Transformers, while the other two model the CPDF of inter event times and the CPMF of marks (see Section 4). The CIF-based baselines the Transformer Hawkes Process (THP) [44], the Self-Attentive Hawkes Process (SAHP) [42], and the Attentive Neural Hawkes Process (A-NHP) [41]. The CPDF-based ones the Intensity- -Free Temporal Point Process (IFTPP) [35], and the VI-Decoupled Point Process (VI-DPP) [30].
We fit the above six models diverse collection of five popular real-world datasets, each with varied characteristics: MIMIC-II [19], Amazon [28], Taxi [37]. Taobao [43], and StackOverlfow- V1 [20, 41]. Training details are given in Appendix A.2.
Goodness-of-Fit Figure shows the average logelikelihood for each model on the held-out data of the five real-word world datascts. Our DTPP model consistently outperforms the simple parametric

---

Table 1: Performance comparison between our model DTPP and various baselines in terms of next- event prediction. The root mean squared eтror (RMSE) measures the error of the predicted time of the next event, while the error rate (ERROR- -%) evaluates the error of the predicted mark given the true time. The results (lower is better) are accompanied by 95% bootstrap confidence intervals. 1,4 denote the CIF-hased methods, the CPDF-based methods that use single model, and the ones using seperate model, respectively.
AAZOON
TaXi
TAOBAO
STACKOVERFLOWW-Y
METHODS
RMSE
ERROR
RMSE
ERROR
RMSE
ERROR
RMSE
ERROR
THP!
0.62 =0.03 65.06 -1.04 0.37 #0.02 8.55 20.65 0.13
#0.02 41.43
20.78 1.32 40.03
53.86
70.76
SAHP
0.58 =0.01 62.58 =0.2 0.28 10.04 8.37 40.43 2.26
20.59 45.63
20.58
1.93 10.04
53.00
#0.32
A-NHP
0.42 =0.01 65.94 =0.31 0.29 10.02 7.67 20.44 1.44
10.53 43.96 10.55 1.18 10.01
52.17
10.32
IFTPPS
0.41 =0.05 64.08
-034 0.39 10.09 8.17 10.46 0.33
10.00 41.92
10.54
1.91 10.10
53.68
10.30
VI-DPP
0.38 00.00 65.49 0.34 0.11 10.02
9.49 40.42
0.07
#0.00 41.89 20.7
1.68 10.04
55.06
#0.32
DTPP
0.12 =0.00 59.06
=0.35 0.08 #0.01 7.12 40.40 0.05
=0.00
40.12
=0.50
1.07
40.03
50.41
=0.31
VI-DPP, indicating the flexibility of using the self-attention mechanism to model the CPDF. Except Mimic-II, DTPP achieves the highest or the second highest log-likelihood across the remaining datasets. Therefore, the separate parameterization of p" and 9" does not hurt performance compared to the models with a common set of learnable parameters. Finally, notice that DTPP outperforms all the CPDF-based methods average, while the two CPDF-based methods that employ deep learning architectures, i.e., DTPP and IFTPP, exhibit better performance than the CIF-based baselines. A plausible explanation is that the log-likelihood computation for the CIF-based baselines requires Monte Carlo integration, which could cause approximation errors; for the CPDF-based methods, this computation is exact. A-NHP is the clear winner among the ClF-b methods, as also shown in [41].
Next-Event Prediction. We evaluate the predictive capacity of all models by predicting each event (t,, k)) given its history He, i .,..., N on held-out data. Event time prediction is measured by root mean squared Error (RMSE) and event type prediction by error rate; Table summarizes the
results. DTPP outperforms all the baselines in both tasks. The wider performance gaps in RMSE between our model and the other baselines justify our choice of a inter-event distribution satisfying Markov property; this result also implies that we do not need long event histories capture the dynamics of these datasets. We also compare the average performance between CIF-based and CPDF-based (excluding VI-DPP) methods. We see that for the CIF-based baselines the average RMSE is 0.58 and the average error rate is 35.39% while for the CPDF-based ones we have 0.95 and 37. 1%,0% respectively. These results support our argument that the thinning algorithm tends to harm the time prediction accuracy; they also highlight the efficiency of using a separate model for the inter-event times. Additional results on Mimic-II can be found in Appendix B.I.
Synthetic datasets. To extra investigate the capabilities of our model in a more controlled manner, we created a dataset by generating sequences from a randomly initialized SAHP model. Although, each event has strong dependence on its history, Figure 2 shows that our model approximates the true log-likelihood as well as A-HHP. Moreover, DTPP's mixture model is more accurate than the thinning-basee A-NHP in time prediction. Moreover, we found that the only case that DTPP was significantly outperformed by A-NHP was on a synthetic dataset generated by a 1-d Hawkes process. Since no event types are present we only use a single mixture of log- log-normals which apparently the wrong model for this data. The results are illustrated in Figure 4 of the Appendix.
5.2 Long-Horizon Prediction
To test the performance of our model for this task, we followed the experimental setup of [40]. From the same work, we used the proposed HYPRO, which is the state-of-the-arr method for the long-horizon prediction task, to compare with DTPP. HYPRO is a globally normalized model that aims address cascading errors that occur in rego-segressive and locally normalized models, such as the models in Section 5.1. HYPRO and DTPP are based on the same Transformer architecture of
7

---

-1.00
True Log-Bkeldoood
Avg loe-likellbood
1.05
0.014
RMSE
1.10
0.012
- -1.15
0.010
A-NHP
DTPP
A-NHP
DTPP
Figure 2: Performance comparison between DTPP and A-NHP over the SAHP-S SAHP-Synthetic dataset.
A-NHP so the main difference is that HYPRO a CIF-based method, and thus, it requires the thinning algorithm to sample sequences. For our experiments, we use the distance-based regularization variant of HYPRO with a Multi-NCE loss as this method attains the best results in [40]. As DTPP and HYPRO share the same Transformer architecture, so we used the exact same hyperparameters for fair comparison. Note that even in this case, HYPRO has more than double number of parameters compared to DTPP since HYPRO requires extra Transformer model the energy function used for global normalization. More details on HYPRO training and hyperparameters can be found the Appendix A.
2.0
38
3.1
Avg OTD
RXSEE
Avg OTTO
RAISE
1.0
16
2,4
Avg
0.5
(a) Taxi
(b) Tanbao
(c) StackOverflow-VV
Figure Performance comparison over the three real-world datasets measured by RMSE* and average OTD (lower is better). The reported results for HYPRO are based on 16 weighted samples, ie. M 16 for Algorithm [40].
We used three of the previous real- woal-world datasets for evaluation because of their long sequences. These Taxi, Taobao, and Overflowr22 [20, 40]. For each dataset, goal is to predict the last 20 events a sequence, denoted by Hp. given the history; that is, 20 in Algorithm As is typical for the long-horizon prediction, the standard scores used for evaluating the model's performance the The optimal transport distance (OTD) [25] and the long-horizon RMSE (RMSE*) [40].
In Figure 3, we see that our DTPP method outperforms HYPRO across all datasets in terms of average OTD and RMSE. HYPRO achieves lower RMSE score only StackOverflow. These results provide corroborating evidence on argument that the thinning algorithm might negatively affect the performance of a neural point process even the case globally normalized models as HYPRO. It is also evident that locally normalized CPDF-based model such as DTPP is much more robust against the cascading error which ClF-based methods are vulnerable [41]. We believe that this robustness stems from the accurate predictions of the log-normal mixture model.
Apart from the predictive performance, we investigated the time required for the methods to generate all the predicted sequences of the held-out dataset. Since HYPRO's inference time is heavily relied on the thinning algorithm and hyperparameter that indicates the number of proposal sequences (denoted as M in [40]). we conducted an ablation study for varied number of proposals HYPRO against DTPP. For HYPRO's inference
investigate the inference time and performance the prediction time, we included the time required to generate the noise sequences so the energy function can be trained on. The inclusion of this time is justified by the importance the energy function has component of the framework, and it can be seen necessary pre-inference
time, apart from

---

Table 2: Performance comparison between model DTPP and HYPRO for the long-horizon
prediction task. For HYPRO, we use {2, 4, 8, 16, 32} weighted proposals (Algorithm 2 in [40]). We report the average optimal transport distance (avg OTD) and the time (in minutes) required for predicting all the long-horizon sequences of the held-out dataset (lower is better). "Params" denotes the number (x 109) of trainable parameters of each method. We include error bars based five runs.
TAXI
TAOBAO
STACKOVERFLOWW-
METHODS
PARAMS
AVG
OTD
TIME
AVG
OTD
TIME
AVG
OTD
TIME
HYPRO-2
20.35 10.24
44.81
10.01
39.73
10.37
43.53
10.03
39.84
10.12
46.32
10.01
HYPRO-4
19,86 10.18
47.31
#0.04
38.93
#0.22
46.57
#0.08
39.57
#0.17
48.84
#0.06
HYPRO-8
850
19.25 +0.30
52.61
+0.20
37.30
+0.48
53.10
#0.17
39.37
#0.30
50.22
#0.22
HYPRO-16 16
18.90
+0.34
62.36
#0.27
37.08
+0.39
65.63
=0.44
38.99
#0.16
64.82
=0.34
HYPRO-32
18.81
10.16
81.30
10.23
36.96
10.19
89.11
10.71
38.84
10.20
83.05
10.45
DTPP
400
15.00
10.30
0.01
10.00
28.83
10.26
0.17
10.01
36.75
10.40
0.03
10.00
SPEEDUP
8, 130x
524.2x
2768.3x
step. However, for completeness, we compute only the prediction time of HYPRO and report it in Table 6 of the Appendix.
The results are presented in Table 2 where we measure the performance using the average OTD; similar table for RMSE" is in Appendix B.3. We see that parallelizable framework takes advantage of modern GPU hardware and performs inference in a few seconds. Instead, the thinning algorithm constitutes HYPRO extremely slow and impractical for inference on large datasets. In some cases like the Taxi dataset, HYPRO needs 8, 130x more time than DTPP to perform inference. Moreover, DTPP attains better performance across all datasets even for larger number of proposals in HYPRO. These results verify our assumption about the robustness of the mixture model to predict accurately the next time; they also highlight the inaccurate predictions and computational burden of the thinning algorithm.
Discussion
We have presented DTPP, Transformer-base probabilistic model for continuous-time event se- quences. The model has been derived using the decomposability the likelihood of a MTPP in terms of its CPDF and CPMF. We have used a mixture of log- normnllalss and a Transformer architecture to model CPDF and CPMF, respectively. Our model satisfies some desirable properties compared to previous works that tried to model the CIF such closed-form computation of the log-likelihood and inference without resorting to the thinning algorithm. Extensive experiments on the standard task of next-event prediction showed that method outperformed all state-of-the-an autoregressive models, The results also reveal a more robust performance of the methods that do not require the thinning algorithm to generate event sequences over those they do. Finally, we have tested our model on the challenging task of long-horizu prediction of event sequences. Although our model has not been designed for this task, it outperformed the state-of-the-an baseline HYPRO which is also based on the thinning algorithm. This performance for DTPP was achieved in orders of magnitude faster than HYPRO.
Limitations and future work. The main limitation of the model stems from the modeling of p* using a deep learning architecture which is usually data-hungry and thus requires large amount of data to learn the model's parameters. For this reason, the model might be unsuitable for fata-scarce regimes since it could be prone to overfit. Regarding future work, the limitations of the thinning algorithm revealed by the experiments raise many interesting questions on how can we improve this pathology for the ClF-based methods so they match the performance of the CPDF-based ones since their representations are equivalent. Another interesting research direction would be the development of globally normalized model similar to HYPRO for CPDF-based models.

---

Acknowledgements
We would like thank Petros Dellaportas and Lina Gerontogianni for helpful discussions. This work was funded by Toyota Motor Europe.
References
[1] Gutti Jogesh Babu and Eric D) Feigelson. Spatial point processes in astronomy. Journal of Statistical Planning and Inference, 50(3):311-326, 1996.
[2] E. Bacry, M. Bompaire, S. Gaiffas, and S. Poulsen. tick: Python library for statistical learning, with particular emphasis on time-dependent modeling. ArXiv e-prints, July 2017.
[3] Emmanuel Bacry, Adrian luga, Matthieu Lasnier, and Charles-Albert Lehalle. Market impacts and the life cycle of investors orders. Market Microstructure and Liquidity, 1(02):1550009, 2015.
[4] David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 1 112(518):855 877, 2017.
[5] Alex Boyd, Robert Bamler, Stephan Mandt, and Padhraic Smyth. User-dependent neural sequence models for continuous-time event data. Advances in Neural Information Processing Systems, 33:21488-21499, 2020.
[6] David R Cox. Partial likelihood. Biometrika, (2(2):269-27.. 1975.
[7] Daryl J Daley and David Vere-Jones. Basic properties of the Poisson process. An introduction the theory of point processes: Volume I: Elementary theory and methods, pages 19-40, 2003.
[8] Anirban DasGupta. Asymptotic theory of statistics and probability, volume 180. Springer, 2008.
[9] Abir De, Utkarsh Upadhyay, and Manuel Gomez-Rodriguez. Temporal point processes. Techni- cal report, Technical report, Saarland University, 2019.
[10] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (methodological), 39(1):1-2,, 1977.
[11] Prathamesh Deshpande, Kamlesh Marathe, Abir De, and Sunita Sarawagi. Long horizon forecasting with temporal point processes. Proceedings of the 14th ACM International Conference Web Search and Data Mining, pages 571-579, 2021.
[12] Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 1555-1564, 2016.
[13] Joel Hasbrouck. Measuring the information content of stock trades. The Journal of Finance, 46(1):179 1991.
[14] Alan G Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90, 1971.
[15] Alan G Hawkes and David Oakes. A cluster process representation of a self-exciting process. Journal of Applied Probability, pages 493-503, 1974.
[16] Sergio Hernandez, Pedro Alvarez, Javier Fabra, and Joaquin Ezpeleta. Analysis of users' behavior in structured e-commerce websites. JEEE Access, 5:1 1949-99919988 2017.
[17] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):17335 1997.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1xXiv.1412.6980, 2014.
10

---

[19] Joon Lee. Daniel J Scott, Mauricio Villarroel, Gari D Clifford. Mohammed Saeed, and Roger G Mark. Open-access mimic-ii database for intensive care research. 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 8315-8318. IEEF, 2011.
[20] Jure Leskovec and Andrej Krevl. Snap datasets: Stanford large network dataset collection, 2014.
[21] PA W Lewis and Gerald S Shedler. Simulation of nonhomogeneous Poisson processes by thinning. Naval research logistics quarterly, 26(3):403-415 1979.
[22] Thomas Josef Liniger. Multivariate hawkes processes. PhD thesis, ETH Zurich, 2009.
[23] Geoffrey J McLachlan, Sharon Lee, and Suren I Rathnayake. Finite mixture models. Annual review of statistics and its application, 6:355-3777 1355--78,,00 2019.
[24] Hongyuan Mei and Jason M Eisner. The neural Hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, pages 6754 6764, 2017.
[25] Hongyuan Mei, Guanghui Qin, and Jason Fisner. Imputing missing events in continuous-ting event streams. In International Conference on Machine Learning, pages 4475-4485, PMLR, 2019.
[26] Hongyuan Mei, Guanghui Qin, Minjie Xu, and Jason Eisner. Neural datalog through time: Informed temporal modeling via logical specification. In International Conference on Machine Learning, pages 6808-6819, PMLR, 2020.
[27] Santhosh Narayanan, Ioannis Kosmidis, and Petros Dellaportas. Flexible marked spatio- temporal point processes with applications to event sequences from association football. Journal of the Royal Statistical Sociery Series C: Applied Statistics, page qlad085, 2023.
[28] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-grained aspects. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 188-197, 2019.
[29] Takahiro Omi, Naonori Ueda, and Kazuyuki Aihara. Fully neural network based model for general temporal point processes. arXiv preprint 1rXiv:1905.09660 2019.
[30] Aristeidis Panos, Ioannis Kosmidis, and Petros Dellaportas. Scalable marked point processes for exchangeable and non won-exchangeeble event sequences. In International Conference on Artificial Intelligence and Statistics, pages 236-252. PMLR, 2023.
[31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32:8026-8037, 2019.
[32] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[33] Jakob Gulddahl Rasmussen. Lecture notes: Temporal point processes and the conditional intensity function. arXiv preprint arXiv:/806.0022 2018.
[34] Diego Rybski, Sergey V Buldyrev, Shlomo Havlin, Fredrik Liljeros, and Hemán A Makse. Communication activity in a social network: relation between long-term correlations and inter-event clustering. Scientific reports, 2(1):560, 2012.
[35] Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. In International Conference Learning Representations, 2019.
11

---

[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
[37] Chris Whong. Foiling nyc's taxi trip data. https://chrisshonn.. cotpre//ooooooeeeaattttiiiiii nyc_taxi/, 2014.
[38] Shuai Xiao, Junchi Yan, Xiaokang Yang, Hongyuan Zha, and Stephen Chu. Modeling the intensity function point process via recurrent neural networks. In Proceedings of the AAAI conference artificial intelligence, 2017.
[39] Siqiao Xue. Xiaoming Shi, Zhixuan Chu, Yan Wang, Fan Zhou, Hongyan Hao, Caigao Jiang, Chen Pan, Yi Xu, James Y Zhang, et al. Easytpp: Towards open benchmarking the temporal point processes. arXiv preprint arXiv:2307.0809) 2023.
[40] Siqiao Xue, Xiaoming Shi, James Zhang, and Hongyuan Mei. Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences. Advances in Neural Information Processing Systems, 35:34641 -36500, 2022.
[41] Chenghao Yang. Hongyuan Mei, and Jason Eisner. Transformer embeddings of irregularly spaced events and their participants. In International Conference Learning Representations, 2022.
[42] Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive Hawkes process. In International Conference on Machine Learning, pages 11 1118-11199, PMLR, 2020.
[43] Han Zhu, Xiang Li, Pengye Zhang. Guozheng Li, Jie He, Han Li, and Kun Gai. Learning tree-based deep model for recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1079-1088, 2018.
[44] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In International Conference on Machine Learning, pages 11692-11702 PMLR, 2020.
12

---

A Experimental details
A.1 Dataset Details
Summary statisties and characteristics of the datasets used are given in Table A more detailed description is given below:
Hawkes Blawkes-Synnn -Synnthetic. This dataset contains synthetic event sequences from a univariate
Hawkes process sampled using Tick [2] whose conditional intensity function is defined by
X (t) = + Ei,e Σj=1 a exp(-8, (t- tz)). We use 1, a1 0.8, 81
1.0. This dataset has been used in [29].
Hawkes2-Synthetic. Same as Hawkesl -Synthetie where the parameters here are set as J = 2 0.2, a1 0.4, 31 = 1.0, 02 0.4, B2 20.
SAHP-Synthetic. We sample sequences from randomly initialized SAHP model using the thinning algorithm. The number of layers and the dimension of hidden states are 4 and 32, respectively. The same dataset has been used in [41].
MIMIC-II. The Multiparameter Intelligent Monitoring in Intensive Care (MIMIC-II) is a medical dataset of de-identified clinical visit records of intensive care unit patients for seven years. There are records of 650 patients/sequences where each one contains the time of the visit and the diagnosis of this visit; there are K=75 unique diseases. We try predict the time and the diagnosis of a patient.
This dataset includes time-stamped user product reviews behavior from January, 2008 to October, 2018. Each user has a sequence of review events with each event containing the timestamp and category of the reviewed product, with each category corresponding to an event type. As in [39], we use a subset of 5200 most active users with an average sequence length of 70 which is comprised of K 16 event types.
Amazon.
Taxi. This dataset records the times of taxi pick-up and drop-off events across the five boroughs of the New York city (Manhattan, Brooklyn, Queens, The Bronx, Staten Island). For each borough, we can have pick-up or drop-off event and thus, there are K 10 event types in total. As in [40], we pick randomly sampled subset of 2000 drivers where each driver has a sequence.
Taobao. The dataset comes from the 2018 Tianchi Big Data Competition. consists of time-stamped behavior records (c.g.. browsing, purchasing) of anonymous users on the online shopping platform Taobao from the 25th of November through the 3rd of December in 2017. The K=17 event types represent category group (e.g. men's clothing). The browsing sequences of the most active 2000 users are picked as event sequences. The time unit is hours and the average inter-arrival time 0.06.
The data comes from the well-known question-anse answering website StackOverflw where users are encouraged to answer questions they earn badges. There are 22 different types of badges. Each sequence corresponds to a user and each event gives the time and the type of budge user has been awarded. This dataset is the one processed and used in [12].
StackOverflow- V2. A truncated version of the original StackOverflow-VI. The time unit is 11 days and the average inter-arrival time is 0.95. The dataset has been used in [40]
A.2 Training Details
We use the Adam optimizer [18] with its default settings train all the models in Section 5. We use 200 epochs in total, a batch size of 8 sequences, and we apply early-stopping based on the log-likelihood of the held-out dev set.
Following [41], we use a common hyperparameter to define all dimensionalities of the query, key, and value vectors for the models with attention mechanisms, i.e. THP, SAHP, A-NHP, and DTPP-CPME. For these methods, we also need to specify the number of layers L.. We also denote by D the state space of the IFTPP model.
13

---

Table 3: Characteristics of the synthetic and real-world datasets.
DATASET
# OF EVENT TOKENS
SEQUENCE LENGTH
K
TRAIN
VAL
DBV
MIN
MEAN
MAX
HAWKES WEWKESI-SYNTIERR -SNNHHTTI
65536
16384
32768
64
64
64
HAWKES2-SYNTHETEE
65536
16384
32768
64
64
64
SAHP-SYNTHEET
71023
10242
20237
121
MIMIC-II
75
1930
222
237
2
33
AMAZON
16
288377
40995
84048
14
44
94
TAXI
10
51854
7404
14820
36
37
TAOBAO
75205
11737
28455
32
57
STACKOVERFLOW-Y
22
345116
38065
97233
41
72
736
STACKOVERFLOW-YY
22
90497
25762
26518
41
6.5
101
Table 4: Performance comparison between our model DTPP and various baselines in terms of next- event prediction on Mimic-II dataset. The root mean squared error (RMSE) measures the error the predicted time of the next event, while the error rate (ERROR-% evaluates the error of the predicted mark given the time. The results (lower is better) are accompanied by 95% bootstrap confidence intervals. 4 denote the CIF-based methods, the CPDF-base methods that use single model, and the ones using seperate model, respectively.
MIMIC-II
METHODS
RMSE
ERROR
THP
1.00 10.13 15. 12 10.99
SAHPt
1.49 #0.15 16.39
A-NHPI
1.00 #0.19 15.19 #3.16
IFTPP
0.74 #0.39 14.87 43.38
VI-DPP
0.95
20.40 16.96 16.30
DTPP
0.72 #0.39 14.49 15.86
The hyperparameters D and were fine-tuned for each combination of dataset and model. We grid- search the two parameters using the search spaces D E {4, 8, 16, 32, 64, 128] and L {1, 3, 5). We pick the set of values that achieve the highest log-likelihood on the dev set.
For IFTPP we use the same search for D before. Since IFTPP and DTPP (CPDF part) are based on a mixture of log-normals, we need define the number of components M. We fine tune over the space M E 21, 8, 16).
VI-DPP has only one hyperparameter that requires tuning. This is the number of cutt-off points Q [30]. As the original paper, we found that Q = works the best in all of our datasets.
For long-horizon prediction experiments, 128 and L 2 since this combination worked well on both
chose common size architecture for the HYPRO /
DTPP transformer by setting A-NHP and DTPP in all datasets in Section 5.1.
All experiments were carried out on the same Linux machine with dedicated reserved GPU used for acceleration.
A.3 Implementation Details
For A-NHP, we use the public Github repository https://github. com/yangalan1233 andp-andtt. andtt. For THP and SAHP, we also use the same repository where the corrected im- plementations of THP and SAHP are provided. For IFTPP, we use the public Github reposi- tory at https://githhtt com com/sshchurrff1-ttp while for VI-DPP we use the code at https: //github. waachuu..om//arreeeeeeee wommarrePanoss// latraanss/nneeepeaaaeeeeeee Processe........ Our code will be made pub- licly available on a Github repository after the review period.
14

---

B Extra Experimental Results
B.1 Results on Mimic-II
Table 4 presents the results for next-event prediction task on Mimic-II dataset. The results follow similar patterns in Table 1.
B.2 Results Synthetic datasets
Figure 4 shows the results for the two synthetic datasets generated by two different one-dimensional Hawkes processes. The DTPP's mixture of log-normals fails model correctly the processes since it lacks the flexibility of the neural network that A-NHP and IFTPP are based on. However, when comes to next-time prediction, DTPP performs on par with A-NHP, showing once more that the thinning algorithm decreases prediction accuracy. IFTPP is the clear winner for these two datasets.
B.3 Results Long-Horizon Prediction
We present additional results for the long-horizon prediction in Tables 5 and 6. Table reports the RMSE where the results show that DTPP outperforms HYPRO, regardless of its number of proposals. The only exception StackOverflow-v2, where HYPRO achieved a lower score but required at least 1,544 more running time than DTPP.
Finally, we report HYPRO's s time without considering the time required to generate noise sequences (Algorithm in [40]) from the trained auto-regressive model Pauto for training the energy function Eg; see discussion Section 5.2. Even in this case, our DTPP model orders magnitude faster than HYPRO.
Table 5: Performance comparison between our model DTPP and HYPRO for the long-horizon prediction task. For HYPRO, we use {2, 4, 8, 16, 32) weighted proposals (Algorithm 2 in [40]). We report the root mean squared error of the number of tokens for each type of event (RMSE*) and the time (in minutes) required to predict all the long-horizon sequences of the held-out dataset (lower is better). We include error bars for five runs.
TAXI
TAOBAO
STACKOVERFLOW--V
METHODS
PARAMS (K)
RMSE
TIME
RMSE"
TIME
RMSE
TIME
HYPRO-2
1.39 +0.03
44.81
#0.01
2.87 #0.03
43.53
#0.03
1.16 #0.01
46.32
=0.01
HYPRO-4
1.34 10.03
47.31
10.04
2.74
46.57
10.08
1.15 20.01
48.84
20.06
HYPRO-8
850
1.27 +0.02
$2.61
+0.20
10.03
53.10
10.17
1.15 40.02
54.14
-0.22
HYPRO-16
1.24 10.03
62.36
10.27
2.58 40.02
10.44 65.63
1.15 40.02
64.82
20.34
HYPRO-32
1.20 +0.03
81.30
+0.23
2.55 +0.02
89.11
#0.71
1.14 40.01
83.05
=0.45
DTPP
400
0.96 +0.03
0.01
#0.00
2.45 +0.04
0.17 =0.01
2.02 20.06
0.03 -0.00
SPEEDUP
8, 130x
524.2
2768.3x
15

---

-0.050
8.0
-0.70
8.5
Avg Log likelitood
Avg Log likelkkoood
-0.075
7.5
8.0
-0.100
RMSE
RMSE
-0.75
7.5
-0.125
7.0
-0.150
-0.80
6.5
6.5
-0.175
The leeidikkod
Ioe Lop
NHP ITPP DTPP
6.0
A NHP TFTPP
DTPP
N.P IFTPP
DEPP
6.0 KHP
TFTPP
DIPP
(a) Hawkes-1
(b) Hawkes-2
Figure 4: Goodness-of-it and next-time prediction comparison over the 1-d synthetic examples generated from a Hawkes process. The reported results are based on the test dataset. The black dotted line represents the true log-likelihood of the data (in nats).
Table 6: Time comparison (in minutes) between our model DTPP and HYPRO for the long-horizon prediction task. Unlike Tables 2 and here we only report the prediction time of HYPRO without including the time required to generate noise sequences (Algorithm in [40]) from the trained ruto-segressive model Pauto for training the energy function Eg We include emor bars based on five runs.
METHODS
# PARAMS (K)
TAXI
TAOBAO
STACKOVERFLLOW-Y2
HYPRO-2
2.41
3.13 10.03
2.48 10.01
HYPRO-4
4.91 20.04
6.17 #0.08
5.01 #0.06
HYPRO-8
850
10.21 +0.20
12.70 #0.17
10.30 +0.22
HYPRO-16
19.96 #0.27
25.23 #0.11
20.98 #0.31
HYPRO-32
38.90 10.23
10.71 48.70
39.21 10.44
DTPP
400
0.01 20.00
0.17 40.01
0.03 10.00
SPEEDUP
3, 890x
286.5x
1,307x
Algorithm Horizon Prediction for Decomposed Transformer Point Processes
Input: observed sequence Hr of N events over the interval (0, T) and the number of prediction steps P 1; trained inter-event model g and event types model p
Output: predicted sequence H. of events
I: function LHP(Hr, P.g" ,''p")
2: H.p + 0, H. < Hr
3: for 1 to P:
4:
Predict next-event time and event type
tN+p t + Eg BB (k 1){ H
5:
6:
kN+p argmaxp(( ty-p; H)
7:
Update predicted sequence and event history
8:
Ar +- Hp UJ (IN+ kN+P))
9:
H H U (UN44yyN44))
10:
end for
return Hp
12:
end function
16

---

Algorithm Thinning Algorithm
Input: observed sequence Hr of N events over the interval (0, and the number of prediction steps P > 1; intensity function A
Output: predicted sequence H.p of P events
1: function Thinning(Hr,, P, As.)
2: H.P + 0, H. t HT, - 0, it + T
3: for p
P:
4: Amax max oo) Ek=1 dx(1 {)
Compute
upper bound Amax
5:
repeat:
6:
T ~~ Exp(Amax)
7:
1+0++
8:
until S Ek=I de Ammx
b Accept proposed occurrence time with probability
Ek=1- (f) H))
9:
& ~ Cat(pI,...,, PK) where Pt= = Au(E Rt/ Σ k= (x H)
>> Sample
event type ks E [1,...,K) K} IN+p + kN+p & &
11:
Hp 5- Hp U kNtR), Al - A (NN+)
D Update predicted
sequence and event history
12:
end for
return Hp
14: end function
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See section 6
18

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
See Sections and A.3
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
See Sections and A.3
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: CooodeSbmssssoonl
/ //nips. /twpss.ccppub//////// https/cccppbllcguuueeeeeeoeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See Sections and A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [TODO]
Guidelines: See Section
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpriip///uuiss.....
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
This paper presents work whose goal is to advance the field Machine Learning. There many potential socictal consequences of our work, none which we feel must be specifically highlighted here.
Justification:
Guidelines: This work is concerned with modeling point processes and this
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
22

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Sections and A.1
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi seworsswihcodeee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
23

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "pKIlXb6zVn", "reviewer": "Reviewer_Txj7", "summary": "The paper introduces a Decomposable Transformer Point Process (DTPP), a novel framework for modeling marked point processes. It maintains the advantages of attention-based architectures while avoiding the computational intensity of the thinning algorithm. The model uses a mixture of log-normals for inter-event times and a Transformer architecture for the conditional probability mass function of event marks, achieving state-of-the-art performance in next-event prediction tasks and outperforming thinning-based methods in long-horizon prediction.", "questions": "See above.", "limitations": "The authors have adequately addressed the limitations", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Innovative Approach**: The paper proposes a new way to model marked point processes by decomposing the problem into manageable sub-problems, which is a creative advancement in the field.\n2. **Empirical Performance**: The DTPP model demonstrates improved performance over existing methods, particularly in next-event prediction and long-horizon forecasting, which is a significant contribution.", "weaknesses": "1. **Unclear motivation.** I am not an expert in this field, so I do not have a deep understanding of the field of neural point processes.  I was pretty confused when I tried to understand the necessity of decomposing the log-likelihood of a marked point.\n\n2. **The writing is a bit difficult to read.** The writing lacks clarity for readers outside the field. The overall quality needs improvement, as certain sections are challenging to comprehend due to complex sentence structures and unclear presentation of ideas. Specific examples include lines 60-63 and 128-136, where the convoluted language may hinder understanding for those unfamiliar with the subject matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We respond to your concerns below:\n\n1. Using the decomposition in Eq. (2) is equivalent to using the standard log-likelihood based on $λ_k^∗(t)$  in Eq. (1). For more details, see JG Rasmussen, 2018, \"Lecture notes: Temporal point processes and the conditional intensity function\". We chose the decomposition in (2) because it allows us to freely define different models for the times and marks. Therefore, we can have models with nice properties (eg mixture of log-normals) without depending on the computationally demanding thinning algorithm for generating samples. We discuss this in lines 89-94.\n\n2. We are unsure what is confusing in these lines. In lines 60-63, we just showcase the contributions of this paper. More details regarding the long-horizon prediction task can be found in Section 5.2. In lines 128-136, we introduce shortly the properties of the mixture of log-normal distributions and how this model can be used for modeling the distribution of inter-event times. The notation is quite standard but we encourage the reviewer to point specifically where the confusion comes from; we are happy to modify the text of the paper to increase readability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxdylovdJo", "reviewer": "Reviewer_pTAD", "summary": "This paper presents a novel framework for modeling marked temporal point processes (MTPPs) using Transformer-based architectures. The authors address the limitations of traditional methods that rely on computationally intensive thinning algorithms by proposing a decomposable approach that partly uses a Transformer. This approach models the conditional intensity function (CIF) using a Transformer architecture while separating the modeling of inter-event times and event types into two distinct components: the conditional probability density function (CPDF) for inter-event times and the conditional probability mass function (CPMF) for event types.", "questions": "1. Have you considered including more visualizations of the learned intensity functions, attention mechanisms, or specific case studies showing predicted sequences? Adding more visual representations of your model’s predictions and learned parameters would improve clarity and provide tangible insights into the model’s behavior.\n2. Discussion about potential theoretical extensions that could lead to the modifications to the Transformer architecture specifically suited for MTPPs. Probably some interesting inductive biases could be added here.\n3. Still, I do not see too much scientific novelty because literally Transformer was partly applied to a problem where it was not applied before.", "limitations": "Authors already discussed the limitations in the main paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Novel framework that uses a Transformer architecture for the first time to tackle this problem\n- Decomposing the likelihood into conditional probability density function (CPDF) and conditional probability mass function (CPMF) components\n- The proposed DTPP model is technically solid, with a well-structured approach to decomposing the MTPP likelihood and modeling the components using Transformers. The mathematical formulations are clearly presented.\n- The authors provide detailed implementation details and make their code available. This ensures that the results can be reproduced and verified by other researchers, which is important for the credibility and quality of the work.\n- The demonstrated improvements in predictive accuracy and computational efficiency over state-of-the-art methods highlight the significance of the proposed framework. The speedup achieved in long-horizon prediction tasks is particularly noteworthy.", "weaknesses": "- The primary contribution of the paper is the application of Transformer architecture to the problem of modeling marked temporal point processes. While this is a useful application, it does not introduce significant theoretical advancements or novel methodologies beyond leveraging existing models in a new context. To enhance the novelty, the authors could consider integrating more innovative elements or demonstrating new theoretical insights specifically tailored for MTPPs.\n- The paper could benefit from more detailed ablation studies to isolate the contributions of different components of the proposed framework. For example, assessing the impact of various hyperparameters, the influence of the Transformer architecture's depth and width, or the role of specific design choices in the decomposable framework would provide deeper insights into the model's functioning and robustness.\n- While the paper includes several figures and tables, additional visualizations could enhance clarity. For example, visualizing the learned intensity functions, the attention mechanisms within the Transformer, or case studies showing specific sequences predicted by the model could provide more tangible insights into the model's behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and your suggestions. We respond to your concerns below:\n\n1. Notice that we do not learn any intensity functions since our framework is based on the decomposition in Eq. (2). Given the black-box nature of the transformer-based architecture we refrained from including visualizations of the learned representations. This is a general problem for black-box models such as transformers or LSTM/RNN even though there has been some progress in the last few years [*] but in the context of computer vision tasks. To the best of our knowledge, previous works on MTPP using the transformer architecture do not provide such visualizations due to this reason. Only [44](Zuo, 2020) provides a visualization of attention patterns of different attention heads in different layers; however, we believe the results are more confusing rather than providing clarity since they have been arbitrarily chosen from one of the datasets without any insight regarding the configuration of the tranfsormer architecture.\n\n2. This is an interesting point and something that we aim to investigate in the future regarding the modification of the architecture. Nevertheless, tha currently used architecture is already tailored for MTPP since it shares components with [44] amd [41]. We are happy to include this as future direction in the paper. Thank you.\n\n3. We respectfully disagree with your assessment of lack of novelty. We indeed utilize ideas from previous works to build our final novel model but we fail to see why this is absence of novelty. For instance, [30] (Panos, 2023) used the same decomposition which is already known from [6] (Cox, 1975) and the functional form in [27] (Narayanan, 2023) to model the mark distribution. [35](Shchur, 2019) combined a mixture of log-normals with LSTM; all well-known ideas at that time. The continuous-time Transformer architecture for modeling point processes was first adopted by [44](Zuo, 2020) and [43] (Zhang, 2020) independently while [41] (Yang, 2022) later used the same transformer architecture with a modified way to model the intensity function. We also feel that the reviewer has overlooked our experimental evaluation which provides strong evidence of the state-of-the-art performance of our model over well-established baselines. Both the RMSE and ERROR metrics highlight the efficiency of combining a simple mixture of log-normals with a transformer architecture. We believe this is an important contribution and something that was not known to the community until now. The other contribution is the ability of our model to significantly outperform (in a fraction of time) the state-of-the-art HYPRO baseline on the long-horizon prediction task. This result showcases the importance of using a simple yet robust model for the inter-even times as the mixture of log-normals. We are also the first who investigate the limitations of thinning-based methods for the long-horizon prediction task. This result becomes more important if you consider that our method was never developed to deal with the more challenging long-horizon prediction task. We believe these results by themselves are novel enough and would be interesting for the community.\n\n[*] Chefer, et al. \"Transformer Interpretability Beyond Attention Visualization\", CVPR, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3gBt8AB1Je", "reviewer": "Reviewer_1AM8", "summary": "The work designs a novel transformer-based approach for modelling time series (e.g. predicting next event). The main novelty is the decomposition of the log-likelihood into a conditional probability mass and density functions. The former, implemented with a transformer, models the distribution over the event types; the latter models the event occurance with a log-normal mixture. The experimental results are compelling: the approach achieves convincing performance on next-even prediction (in terms of log-likelihood) and long-horizon prediction.", "questions": "* How does the size of the mixture model (M) affect the prediction and what was the methodology for choosing the optimal M?\n* The scale in Fig. 1 is wildely different. How does one explain the differences and what does it say about the data quality and/or task complexity?\n\nMinor remarks:\nl. 80 The integral’s upper bound coincides with the integrand variable.\nl. 150-155: Perhaps one could provide a comparison with previous work, what does this simplification of the problem translates to in practice (how much faster it is to train/optimize).", "limitations": "The limitations are discussed sufficiently in Sec. 6.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The writing and presentation are clear and precise. The technical introduction and the contribution have solid foundations.\n* The experiments are convincing.  I appreciate reporting the variance for transparency.\n* I also appreciate the details provided here and there without distracting from the main story (e.g. the hyperparameter values in the supp. material).", "weaknesses": "I do not see any major problems, but would encourage minor revisions towards reaching broader audience. Specifically:\n\n* The thinning algorithm plays an important role in motivating the approach and interpreting the results. The readers will appreciate a high-level recap of the algorithm, intead of having to look it up in the reference.\n* The justification for the decomposition (ll. 90-93) comes across a bit weak. Perhaps it could be improved by providing an analytical argument why it should offer the same benefits as the intensity function, and why depending on the thinning algorithm at inference time is a problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We respond to your concerns below:\n\n- Even though we do not present the full thinning algorithm in the main text due to space restrictions, we present the exact algorithm in the appendix on page 17. As we explain in lines 76-83, the expression of the two log-likelihoods in Eq. (1) and (2) are equivalent. Actually, there is a closed-form formula that relates the intensity function with the CPMF/CPDF; a derivation can be found in  Section 2.4 in  JG Rasmussen, 2018, \"Lecture notes: Temporal point processes and the conditional intensity function\"\n\n- We have included a table (attached pdf) with an ablation study on the influence of the number of mixture components M. The optimal M is chosen based on the log-likelihood of the held-out dev set as we explain in the Appendix (Section A.2).\n\n- One could potentially interpret the large (or low) log-lkl values as a proxy of the ability of the models to capture the complex dynamics of the event sequences. Larger values may indicate that the model provides a good approximation of the latent generative mechanism of the process however this might not be always true since it is heavily based on the quality of the training/test data and how representative is the sample at our disposal.\n\n- Thank you for spotting the typo in line 80. We will correct this in the revised version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We provide an ablation study on the influence of the number of mixture components M.

Meta-review: This paper addresses forecasting with marked point process models by integrating attention and sidestepping the intensive thinning algorithm currently required at inference. Experiments are conducted on next-event prediction and long-horizon prediction.

The reviews are all inclined to accept the paper, varying between borderline and weak accept.
 
Upon consideration of the paper, the reviews, the rebuttal, the authors-reviewers and AC-reviewers discussions, the AC summarizes these aspects:
 
(1) Novelty. The model is simple, but the contribution is non-trivial.

(2) The work is technically sound.

(3) The experimental evaluation is effective. As a summary of public and private discussion, the AC recommends exemplifying and discussing the predicted sequences and contextualizing the results with the qualitative properties of the datasets, given the significant variation of the likelihood scales.

The initial lamented weaknesses of clarity and novelty appear resolved and agreed upon by the end of the discussion period.
The AC has verified this by considering the paper.

Overall, the AC is happy to second the recommendation to accept this work!
Decision: Accept (poster)