Exploring Generative Neural Temporal Point Process
Authors: Haitao Lin
Lirong Wu
Guojiang Zhao
Liu Pai
Stan Z. Li
Venue: Accepted by TMLR ()

Abstract: Temporal point process (TPP) is commonly used to model the asynchronous event sequence featuring occurrence timestamps and revealed by probabilistic models conditioned on historical impacts.
While lots of previous works have focused on `goodness-of-fit' of TPP models by maximizing the likelihood, their predictive performance is unsatisfactory, which means the timestamps generated by models are far apart from true observations.
Recently, deep generative models such as denoising diffusion and score matching models have achieved great progress in image generating tasks by demonstrating their capability of generating samples of high quality.
However, there are no detailed and unified works exploring and studying the potential of generative models in the context of event prediction of TPP.
In this work, we try to fill the gap by designing a unified generative framework for neural temporal point process (GNTPP) model to explore their feasibility and effectiveness, and further improve models' predictive performance.  
Besides, in terms of measuring the historical impacts, we revise the attentive models which summarize influence from historical events with an adaptive reweighting term considering events' type relation and time intervals. 
Extensive experiments have been conducted to illustrate the improved predictive capability of GNTPP with a line of generative probabilistic decoders, and performance gain from the revised attention.  
To the best of our knowledge, this is the first work that adapts generative models in a complete unified framework and studies their effectiveness in the context of TPP.


Published in Transactions on Machine Learning Research (08/2022)
Exploring Generative Neural Temporal Point Process
Haitao Lin
linhailao Coaaaaaaao@uesstkkk.ede......
CAIRI, Westlake University
Zhefiang University
Lirong Wu
wwlirong@ruestlake.cm edu.c.
CAIRI, Westlake University
Zhejiang University
Guojiang Zhao
guojjane2Bendreu.cou.cm
CATRI, Westlake University Cornegie Mellon University
Pai Liu
palliu
School of Engineering, Westlake Unéversity
Stan Z. Li CAIRI, Westlake University
Stan. ZQ. Li@.westlake.eed..o.o
Reviewed on OpenReview: https: //tp://ppppe openreview. net/ forum? id= NdS
Abstract
Temporal point process (TPP) is commonly used to model the asynchronous event sequence featuring occurrence timestamps and revealed by probabilistie models conditioned histori- cal impacts. While lots of previous works have focused on 'goodness-ooffit ff-fit' of TPP models by maximizing the likelihood, their predictive performance is unsatisfactory, which means the timestamps generated by models are far apart from true observations. Recently, deep generative models such as denoising diffusion and score matching models have achioved great progress in image generating tasks by demonstrating their capability of generating samples of high quality. However, there are no complete and unified works exploring and studying the potential of generative models in the context of event occurence modeling for TPP. In this work, try to fill the gap by designing unified generative framework for neural temporal point process (GNTPP) model to explore their feasibility and effectiveness, and further improve models' predictive performance. Besides, in terms of mcasuring the historical impacts, we revise the attentive models which summarize influence from historical events with an adaptive reweighting term considering events' type relation and time intervals. Extensive experiments have been conducted illustrate the improved prodictive capability of GNTPP with line of generative probabilistic decoders, and performance gain from the revised attention. To the best of our knowledge, this is the first work that adapts generative models in complete unified framework and studies their effectiveness in the context of TPP. Our codebase including all the methods given in Section. 5.1.1 is open in https://gitta www/BIRD- TAARDD-TTDGNPPP We hope the code framework can facilitate future research in Neural TPPs.
1 Introduction
Various forms of human activity or natural phenomena can be represented as discrete events happening with irregular time intervals, such as electronic health records, purchases in e-commerce systems, and earthquakes

---

Published in Transactions on Machine Learning Research (08/2022)
with aftershocks. A natural choice for revealing the underlying mechanisms of the occurrence of events is temporal point processes (TPPs) (D.J. Daley, 2003 Isham & Westcott 1979:
Hawkes 1971p p, which describe
the probability distribution of time intervals and types of future events' occurrence by summarizing the impacts of historical observations.
Recently, with the rapid development in deep learning, TPP models also benefit. from great expressiveness of neural networks, from the first work proposed in Du et al. (0116). A neural TPP model can be divided into two parts history encoder and probabilistic decoder. Recently, great success has been achieved in modeling the TPPs thanks to fast developments in sequential models and generative models in deep learning (Shehur et al. 2021 Lin et al. 2021). The former concentrates on the competence of encoding and aggregating the past impacts of events on the next event's 's occurrence probability (Zhang et al. 2020a Zuo et al. 2020), which is called history encoder; The latter one aims to improve the flexibility and efficiency to approximate
the target distribution of occurrence time intervals conditioned on the historical encoding (Omi et al. 2019 Shchur et al. 2020a b). which is called probabilistic decoder.
Most of the previous works focus the gfoddnessoo-fitt of the proposed models, which can be quantified by negative log-likelihood (NLL). However, limited by this, to model the distribution in a general fashion, one needs to formulate the probabilistic decoder with certain families of functions whose likelihoods are tractable, such as the mixture of distributions whose support sets are positive real numbers (Shchur et al. 2020a Lin et al. 2021) or triangular maps as a generalization of autoregressive normalizing flows (Shchur et al, 2020b). Although some probabilistic decoders with intractable likelihoods still perform well in the evaluation of "goodness-of-fitt (Mei & Eisner 2017 Zhang et al.) 2020a] Zuo et al.) 2020), they rely on the stochastic numerical approximation to calculate the likelihood, which leads to unaffordable high computational cost despite their theoretically universal approxination ability (Soen et al., 2021). To sum up, both the requirements for a certain structure in the functional approximators, e.g. on the mixture of log- lor-normal distribution (Shchur et al.) 2020a), and the excessive emphasis on 'goodness-of-fit" as the optimization objective considcrably impose restrictions on the modcls' prodictive and extrapolation performance. This causes that the timestamp samples generated by them are far apart from the ground-truth observations which limits their application in real-world scenarios. Recent empirical studies show that these models' predictive performance is very unsatisfactory (Lin et al.] 2021), with extremely high error in the task of next arrival time prediction. As probabilistic models, good TPP model should not only demonstrate its goodness-offfitiing (lower NLI.),, but also have ability to generate next arrival time samples of high quality, as well as preserve the randomness and diversity of the generated samples.
Since the TPP models aim to approximate the target distribution of event occurrence conditioned on the historical influences, we can classify the TPP models into conditional probabilistic or generative models the field of deep learning, and lend the techniques in these fields to improve the predictive performance (Yan et al., 2018 Xiao et al.] 2017ah. In deep probabilistic models, the functional forms in energy- based models are usually less restrictive, and the optimization of the objective function unnormalized ligeliiel can be directly converted into a point estimation or regression problem, thus empowering the models to have an improved predictive ability. Recently, deep generative models including denoising diffusion models [Sohl-Dickse DicksDickttein et al.] 2015b Ho et al. 2020) and score matching models (Song et al. 2021b a Bao et al.) 2022) as an instance of energy- based deep generative model have attracted lots of scientific interests as it demonstrates great ability to generate image samples of high quality. Thanks to its less restrictive functional forms and revised unnormalized log-probability as the optimization objective, in other fields such as crystal material generation (Xie et al. 2021) and time series forecasting (Rasul et al. 2021), they are also employed for generative tasks and demonstrates great feasibility. Enlighted by this, we conjecture that the probabilistic decoders constructed by deep generative models in the context of TPP are likely to generate samples of time intervals that are close to ground truth observations, thus improving the predictive performance. Tn this way, we design a unified framework for generative neural temporal point process (GNTPP) by employing the deep generative models as the probabilistic decoder to approximate the target distribution of occurrence time. Besides, we revise the self-attentive history encoders (Zhang et al. 2020a) which measure the impacts of historical events with adaptive rewcighting terms considering events' type relation and time intervals.
In summary, the contributions of this paper are listed as follows:
2

---

Published in Transactions on Machine Learning Research (08/2022)
To the best of our knowledge, we are the first to establish complete framework of generative models to study their effectiveness for improving the predictive performance in TPPs, i.e. enabling TPP models to generate time sample of high quality.
In terms of history encoders, we revise the self-attentive encoders with adaptive reweighting terms, considering type relation and time intervals of historical observations, showing belter expressiveness.
We conduct extensive experiments on one complicated synthetic dataset and four real-world datasets, to explore the potential of GNTPP in the aspect of predictive performance and fitting ability. Besides, further studics give more analysis to ensure the effectiveness of the revised encoder.
2 Background
2.1
Temporal Point Process
2.1.1
Preliminaries
A realization of a TPP is sequence of event timestamps {tthiggN, where t, E Rt, and t, < t,+1 < T. In a marked TPP, it allocates a type m, (a.k.a. mark) to each event timestamps, where there are M types of events in total and m, E [M] with [M] = (1, MJ. A TPP usually described as a counting process, with the measure (t)) defined as the number of events occurring in the time interval (0,
We indicate with {(4,, hissen as observation of the process, and the history of certain timestamp is denoted H(t) m(t,, my),ty t}. In this way, the TPP can be characterized via its intensity function conditioned on H()), defined as
BI + 20) - -N()))
(1) = A((H(())) = lim AC-
(1)
At
which means the expected instantancous rate of happening the events given the history. Note that it is always a non-negative function of L. Given the conditional intensity, the probability density function of the occurrence timestamps reads
f AT(T)dT),
(²(t)= x²) exp( exp(-
(2)
The leading target of TPPs is to parameterize a model pt()), fit the distribution of the generated marked timestamps, i.c. q² (t), to inference probability density conditional intensity for further statistical prediction, such as next event arrival time prediction. Besides, in marked scenarios, parameterizing P7 (m) to predict the next event type is also an important task. More details on preliminaries are given in Appendix A. Usually, in the deep neural TPP models, the impacts of historical events H(t) on the distribution of time are summarized as a historical encoding h4-1, where - = arg max,en < t, and the parameters in p. and py (m) are determined by h;-1, for > 4, which reads
p² (t::((h;-1) po (f)h;-1); p² (m;;(h;,1) = (m
(3)
In summary, in deep neural TPPs, there are two key questions answer in modcling the process:
(1) How to measure the historical events' impacts on the next events' arrival time and type distribution. In other words, how to encode the historical events before time which is H(t) for 4-1 t into a vector h to parameterize po(t(h;-1) or pa (m)
(2) How to use a conditional probabilistic model p" (t, m; (h;-11)) po(t, m(h;-1) with respect time and type m, whose parameters are obtained by 0(h,11), approximate the true probability of events' time and types?
3

---

Published in Transactions on Machine Learning Research (08/2022)
2.1.2 History Encoders
For the task (1), it can be regarded as task sequence embedding, i.e. finding a mapping H which maps a sequence of historical event. time and types before I, i.e. H(I) {(1 ms))ssss-- where & 4-11, to a vector h IR called historical encoding. D is called 'embedding size' in this paper.
To increase expressiveness, the j-th event in the history set is firstly lifted into high-dimensional space, considering both temporal and type information, as
u(t;, m,) ej - (w((); m.
(4)
where w transforms one-dimension tj into high-dimension vector, which can be linear, trigonometric, and so on, Em is the embedding matrix for event types, and mj is the one-hot encoding of event type my- Commonly, to normalize the timestamps into a unifying scale, the event embeddings take the Tj = t, -t as the inputs, i.c. e = my) And then, another mapping will be used to map the sequence of embedding {er, B 1 into vector space of dimension D, by
h v(ee :
For example, units of recurrent neural networks (RNNs) including GRU and LSTM can all be used to map the sequence (Du et al. 2016 Omi et al. 2019 Shchur et al. 2020a), as
hg 0: h RNN(e, h4-1)
(6)
Therefore, the history encoder can be dismantled two parts,
h = u([u(TI, m1); u(T2, ;u(::- m;-1)l).
H(H(()) = v oo v
(7)
where u and v are the event encoder and sequence encoder respectively, and the composites both make up the history encoder.
The attention mechanisms (Vaswani et al. 2017 which have made great progress in language models prove to be superior history encoders for TPPs in the recent rescarch (Zhang et al. 2020a Zuo et al. 2020). In this paper, we follow their works in implementing self-attention mechanisms (Vaswani et al. 2017) but conduct revisions to the self-attentive encoders, leading to our revised attentive history encoders in our paper.
2.1.3 Probabilistic Decoders
For the lask (2), it is usually regarded setting up conditional probabilistic model p" (1, m((;;-10)))))) whose conditional information is contained by model's parameters 0(h,-1) obtained by historical encoding. Statistical inforence and prediction can be conducted according to the model, such as generating new sequences of events, or using expectation of time to predict the next arrival time. Besides, for interpretability, the relation among different types of events such as Granger causality (Xu al. 2016 Fichler et al. 2016) inferred by probabilistic models also arises research interests.
In the temporal domain, one choice is to directly formulate the conditional intensity function Xī(t) (Du et al, 2016), cumulative hazard function At(() = So (T(T)dr (0mi et al. 2019) or probability density function (t) [Shchur al., (220aa), and minimize the negative log- likeliool as the optimization objective. For example,
the loss of a single event's arrival time reads
4 log (o(ts)hu-1) + 1 Ae(t(ti-i)dd. Jt
However, as demonstrated in Equation. 2 and minimizing the negative likelihood requires the closed form of probability density function, which limits the flexibility of the models to approximate the true probabilistic distribution where the event data are generated. For example, one attempts to formulate At(t) will inevitably confront whether the integration of it (a.ka. cumulative hazard function) has closed forms, for
4

---

Published in Transactions on Machine Learning Research (08/2022)
Table 1: Description of exisiting neural TPP methods, following Lin et al. 20021
Methods
History Encoder
Probabilistic Decoder
Closed Likelihood
Flexible Sampling
RMTPP Du et al.
2016
RNN
Gompertz
LorNorm Shehur et al. 2020a
RNN
Log-normal
ERTPP Xiao et
2017b
RNN
Gaussian
Weib Lin et al.
2021
Transformer
Weibull
FNNInt al.
2019
Feod-forward Network
SAHP Zhang et sl.
2020a
Transformer
Exp-decayed + Nonlinear
THP Zuo et al.
2020
Transformer
Linear-decayed Nonlinear
X
WasGANTPP Xiao
2017a
RNN
GAN
manageable computation of the likelihood. Although this term can be approximated by numerical Monte Carlo integration methods, the deviation of the approximation from the analytical likelihood may occur due to insufficient sampling and high computational cost may be unaffordable. Another problem is that these likelihood-trrgteed models usually perform unsatisfactorily in next arrival time prediction (Lin et al. 2021 despite its good fitting capability in terms of negative log-likelihood
For these reasons and enlightened by effectiveness of adversarial and reinforcement learning Yaan et al, 2018 Arjovsky al., 2017 Upadhyay et al. 2018 Li et al.) 20188) in the context of TPPs, we conjecture that stale-of-the-art methodss and techniques in deep generalive models can be transferred to deep neural TPPs. Differing from the previous works focusing models' fitting ability in terms of higher likelihood, these models aim to promote models' prediction ability, i.c. to generate high-quality samples which are closer to ground truth observations. Inspired by great success achieved by recently proposed generative models Schl-Dicksterd et al. 2015a, Ho et al. 2020 Song et al., 2021b) which enjoy the advantages in generating image samples of good quality and have been extended to a line of fields (Xie et al. 2021 Rasul et al. 2021), we hope to deploy this new state-of the-art probabilistic model to TPPs, solve the dilemma of unsatisfactory predictive ability of neural TPPs well as further enhance models' flexibility.
2.2 A Brief Review
8.0
RMITP
5.6
5.0
ERTTP
WishMit
4.8
ENNNEr
leg[Second)
4.6
RMM
4.4
WiSGANTEP
4.0
3.6
4.0
3.0
3.8
25
3.6
7.6
7.8
8.0
0.2
8.4
0.6
Legheen
log(MEM)
Methods
(a) Complexity Comparison MOOC
(b) Predictive Performance Comparison MOOC
Figure 1: Intuitive explanation of our motivation: In (a), SAHP and THP cost more memory than others, and WASGANTPP is more time-consuming because of the adversarial training. In (b), the 'MAPE' in WASGANTPP as generative model is relatively smaller, in comparison to the classical TPP probabilistic dccoders. The detail of the experimental settings and results is given in Section 5 and Appendix B.4
We review recently-proposed neural TPP models, and give a brief disciption of them in Table 1 Most methods directly model the intensity function or probabilistic density function of the process, while only WasGANTPP
Xiao et al. 2017a) employed Wasscrstein GAN as the probabilistic decoder, whose lcarning target is an approximation to the Wasserstein distance between the empirical and model distributions, and allows flexible sample generation. The classical methods with closed-form likelihood usually show unsatisfactory performance,
5

---

Published in Transactions on Machine Learning Research (08/2022)
while SAHP, THP and WASGANTPP achieve improvements, shown in Figure. depends on numerical or Monte Carlo integration to approximate the likelihood because the probabilistic decoder has no closed form, leading to higher computational cost (shown in Figure I(a) Besides, FNNINT, SAHP and THP do not allow fiexible sampling, which limits the real-world application when one needs to draw out samples from the learned models. Figure | adn Table | give an inluitive remonstration of our motivation.
1(b) SAHP and THP
3 Methodology
3.1 Revised Attentive Encoder
Self-attention as the key module in Transformer (Vaswani et al. 2017 Dong et al. 2021 benefits from its fast parallel computing and capability of encoding long-term sequences in lots of fields. In attentive TPPs (Zhang ct al. 2020a), the events are embedded as vectors e, = [T) E m by positional encoding tochniques, where
w(Tr) [sin(wi] + w2T,)) cos(wij + w2rz)),
(9)
in which wij is positional term, and w2yz is time term. Or in Zuo al. (020)), it reads
w (T,)) [sin (ain(w1])) (w1T;): cos(w
(10)
After that, the historical encoding obtained by attention mechanisms can be written as
h; Σ ex)) Σ exp(o(e,,e.-)))
(11)
where o)::)) maps two embedding inlo a scalar called altention weight, egg. e(e (e;We) and & transforms e, into a series of D-dimensional vectors called values, e.g. e,, This calculation of Equation. can be regarded as summarizing all the previous events' influence, with different weights explo(e).e; 1)) Wjii- explo(e;.:=1))
softmax(ő(e; e;-1)).
Although the self-attentive history encoders are very expressive and flexible for both long-term and local dependencies, which prove to be effective in deep neural TPPs (Zuo et al.) 2020 Zhang et al. 2020a), we are motivatod by the following two problems raisod by event time intervals and types, and revise the classical attention mechanisms by multiplying two terms which consider time intervals and type relation respectively.
P.1. As shown in Equation. [1 in the scenarios where the positional encoding term is not used (Refer to Equation. (2) in Zuo et al. (2020)). if there are two events, with time intervals = b, - - and Tia = 1/2 112-1 which are equal and of the same type but i22 > bi the attention weights them will be totally equal because their event embeddings е₁ and ej, are the same. However, the impacts of the f2-th and ji-th event time
be not necessarily the same, ie. when the short-term events outweigh long-term ones, the impacts of t,, should be greater, since t,, ttj-
P.2. As discussed, the relations between event types are informative, which can provide interpretation of the fundamental mechanisms of the process. Self-attention can provide such relations, through averaging the attention weights of certain type of events to another (Zhang et al.) 2020a).. comparison, we hope that attention weights are just used for expressiveness, and the relations among events should be provided by other modules.
To solve the problem P.1., we revise the atlention weight by a lime-reweighhing term by exp {a(t - 4,),, where a is a learnable scaling parameter. This term will force the impacts short-term events to be greater (a < 0) or less (a 2 0) than ones of long-term events, when the two time intervals are the same. Although the position term in Zhang et al. (2020a) can also fix the problem, the exponential term can slightly improve the performance thanks to it further enhances models' expressivity (See Section 5.3

---

Published in Transactions on Machine Learning Research (08/2022)
To provide events' relations learned by the model well avoid flexibility of attention mechanisms discussed in P.2., we employ the type embedding E to calculate the cosine similarity of different types, and the inner product of two embedding vectors is used as another term to revise the attention weights (Zuo ct al. 2020
Zhang et al. 2021). In this way, the weight of event j in attention mechanisms can be written as
j = softmax((E,,, m) (ET 1) exp {a(4;-1 (a(()-aat))))eeeee t)))o(e,,
(12)
where E., m is normalized as a unit vector for all m E [M] as type embedding, and thus the inner product is
equivalent to cosine similarity. Note that in Eq. 12 when
m,-1) =0, the inftuence of m; to
m m,-1 will not be eliminated after the softmax (+); so in implementation, we map m,,,) ( to 109
to force the influence of type m, events to m;-1 be zero after softmax function if EE m)) ( m, = 0.
We deploy the revised attentive encoder into the Transformer, which are commonly used in Zhang al.
)020aa)
Zuo et al. (2020).
3.2 Generative Probabilistic Decoder
In the generative model, we directly model the time intervals instead of timestamps. For the observation [ty]ısi-, the next observed arrival time interval is T, = tt ft-1, while the next sampled arrival time is 4 = + 4-1 after the sample 9 is obtained.
3.2.1 Temporal Conditional Diffusion Denoising Probabilistic Model
Revised Attentive History Encoder
p(r(((h)))
P(r(9(ha))
*(r1((A))
Diffusion Probabilistic Decoder
x(-A(A :)]
Probabilistic Decoder
MM
MM
Weighted Summation
hg
h2
h
hg
W1
202
History Encoder
+
Time- Reweighting Attenlion&Tyer Product
e1
e2
e
slot (lual)
Event Embedding
E1
1
h4
4,
4..mg
4-1 1
4, 77
Figure2 2: The workflows of revised attentive history encoder with TCDDM as probabilistic decoder.
Temporal Diffusion Denoising Probabilistic Decoder. For notation simplicity, first introduce the temporal diffusion denoising decoder with no historical encoding as condition, and the i-th sample is denoted by 7 in that we are discussing occurrence of a single event. Denote the a single observation of event occurrence time interval by r = 70 ~ q(7"), where 70 E RR and 4(7") is unkown, and the probability density function by (7(7 which aims to approximate q(r9) and allows for casy sampling. Diffusion models (Sohl-Dickstein et al.) 2015a) are employed as latent variable models of the form po(r") == I po(=0:K) d where ....... are latent variables. The approximate posterior ((71:
17))
= IT 16
= NN /1 B B-T 13x))
(13)
is lixed to a Markov chain, which called the 'orwwrdd process. Bi,..., 3x E (0, are predefined parameters.
'Reverse process' is also Markov chain with learned Gaussian transitions starting with p(7*) NN 0,
po - = (( k), E7 kk)),
(14)
7

---

Published in Transactions on Machine Learning Research (08/2022)
The likelihood is not tractable, so the parameters 0 are learned to fit the data distribution by minimizing the negative log-likelihood via its variational bound Ho et. al. 2020
1
Aux ( +0) - -me(r A))l +C,
min Biin (- po - EEg 2
(15)
0
where is a constant which does not depend on 0, and i=*(T" (__(=",,)) == Vāu-18u (0 1-06
3R =
1-62
1 1-as-1 3g. The optimization objective in Equation. 15 is straightforward since it tries to use pg to predict
1-
Pk for every step k. To resemble learning process in multiple noise scales score matching (Song & Ermon 2019 2020,, it further reparameterizing Equation.
as
E,o,c
2 (1 as) ||e colvast" + vI age, kz),
(16)
since (²),)))) v + /1 -ue for em N(0,1)
Temporal Conditional Diffusion Denoising Probabilistic Decoder. We establish a temporal condi- tional diffusion denoising model (TCDDM) as the probabilistic decoder in GNTPP. In training, after h;-1 is obtained historical encoding, through a similar derivation the previous paragraph, we can obtain the temporal conditional variant of the objective of single event arrival time in Equation. (16) as
4 Eep,e [lle col colvā v1 āgc, h, 1)]
(17)
in which the technique of reweighting different noise term is employed. co neural network is conditioned the historical encodings h and the diffusion step k. Our formutaion of eg is foed-forward neural network, as
colVonT? + V1 o.c, h;-1,k)
(18)
(W (hhh h w
o + cos(E [g))) + ((2)) + b(3),
where w DxD, w Rx1 ((2) RDDD W E RIIDD 7( VanT! + VT die, E, is the learnable embedding matrix of step k and & is the one-hot encoding of k. In implementation, the residual block is used for fast and stable convergence.
In sampling, given the historical encoding h;-1, we first sample 2KK from the standard normal distribution ((0, 1), then take it and h4-1 as the imput of network Eq to get the approximated noise, and generally remove the noise with different scales recover the samples. This process very similar anncaled Langevin dynamics in score matching methods.
For inference, the prediction is based on Monte Carlo estimation. For example, when mean estimation is deployed to predict the next event arrival time interval after 4,-1, first sample a large amount of time interval from Pe(T h;-1) (e.g, 100 times), then use the average of sampled ( to estimate the mean of learned distribution, as the prediction value of next event arrival time interval, so the next arrival time is estimated as E [4]] s 1-11+ -5_,400 ((0)
Algorithm 1 Training for each timestamp t, > Algorithm Sampling f > f--1 via Langevin dynamics 4-1 in temporal point process in TCDDM
Input: noise *K ~ (0, and historical encoding h for h K to 1 do
1: Input:
Observation time interval Tr and historical encoding h
if k > then
repeat.
2 A( 1)
Initialize k ~ Uniform(1....,. K) and e
else
(/(0, 1)
== end if
4: Take gradient step on
4/-1
1 (f.
8e vT-an
colf!' b.,,h) /Eaz
Volk-- cola-o + vI - āpc, h1)1
end for
5:
until converged
Return: i :/ + +;-1
8

---

Published in Transactions on Machine Learning Research (08/2022)
He
Σg
9: 4x2/77, 14
Sampling
ft
h
Figure 3: Network structure of temporal conditional variational autoencoder the probabilistic decoder.
3.3
Temporal Conditional Variational AutoEncoder Probabilistic Model
We cstablish a temporal conditional variational autoencoder (TCVAE) as the probabilistic decoder (Kingma & Welling 2014 Pan et al.) 2020 which consists a variational encoder qę h,-1) as a conditional Gaussian distribution N (Mpe Ex) for approximating the prior standard Gaussian N/ I) and a variational decoder (T zi, 11-1) generate arrival time samples. The network structure is given in Figure 3 where the latent Gaussian variable E Rº. The training objective of single event's arrival time interval is the evidence lower bound, which can be written as
min DEL (z)(T h (.- I)) + Et [l] T:1
(19)
In sampling process, the encoder q h4-1) is abandoned. The decoder po (T|2, h;-1) transforms sample 2, which is generated from NN I) to the larget sample f conditioned on h;-1.
3.3.1 Temporal Conditional Generative Adversarial Network Probabilistic Model
Our temporal conditional generative adversarial network (TCGAN) decoder is mostly based on WASSERSTEIN
GAN in TPPs (Arjovsky et al. 2017 Xiao et al. 2017a). The probabilistic generator Po(T) 2, h;-1) is trained via adversarial process, which the other nelwork called discriminator d=(r(h,-1) trained to map the
samples scalar, for maximizing the Wasserstein distance between the distribution of generated samples i and the distribution of observed samples т₁. The final objective to optimize in TCGAN is
min max E, ~p((llii 1) ( de (2-1 (T, (1 1) de (f::h:;-))
(20)
8 T3
where the first term is to maximize the distance, and the second is to add a Lipschitz constraint as a regularization term proposed in original WASSERSTEIN GAN (Arjovsky et al. 2017) with v) as the loss weight. The formulation of the (o(= and (=(r//1-1) are similar the variational decoder in the TCVAE, both transforming the D-dimensional random variables into 1. After training, po( 2 h, can be used for sampling in the same process the variational decoder in TCVAF.
3.3.2 Temporal Conditional Continuous Normalizing Flow Probabilistic Model
As a classical generative model, normalizing flows (Papanakarios al. 2019) are constructed by a series of invertible equi-dimensional mapping. However, in TPPs, the input data sample is 1-dimensional time, and
9

---

Published in Transactions on Machine Learning Research (08/2022)
thus the flexibililty and powerful expressiveness of neural network is hard to harness. Therefore, here we choose to use temporal conditional continuous normalizing flows (TCCNF) (Mehrasa et al. 2020) which is based on Neural ODE (Chen et al. 2019 2021). Note that the & term in Neural ODE is here replaced by k, to avoid confusion. The TCCNF defines the distribution through the following dynamics system:
Fe(T(Ko))l = =<(o)) / kol-(()),
(21)
where TKka) .V( 1). So is implemented with the same structure of variational decoder in TCVAE. The invertibility of Fe (F(oo))) allows to not only conduct fast sampling, but also easily optimize the parameter set 0 by minimizing the negative log-likelihood on single time sample:
k1 Tr( (fo(T(k),k(..) ) x= ) θT(k)
min log(p((((k))) +
/
(22)
3.3.3
Temporal Conditional Noise Score Network Probabilistic Model
Finally, we establish the probabilistic decoder via temporal conditional noise score network (TCNSN) as score matching method, which aims to learn the gradient field of the target distribution (Song & Ermon
2019 2020). In specific, given sequence of noise levels } with noise distribution 9a. (f. h,,1), i.e.
N( Ti, h [,)), the training loss for a single arrival time on each noise level ax is as follows
2-11 100 (1o o V. log 9o, (F,lts, 1111)
(23)
where the se is the gradient of target distribution with the same formulation of variational decoders in TCVAE. According to Song & Ermon (2019), the weighted training objective can be written
of sø (F; Th hi-1) 1)
Ti 12: ok
min 0
(24)
Ok
where 74 ~ 4o (FilTe, h;-1).
In the sampling process, the Langevin dynamics (Welling & Teh 2011 is used, in which the sample is firstly drawn from a Gaussian distribution, then iteratively updated by
= -/- at 1; ox + 224z2
(25)
in different noise levels with different times, where ~ ~001))
3.4 Mark Modeling
When there exists more than one event lype, another predictive target is what type of event is most likely to happen, given the historical observations. The task regarded as a categorical classification. Based on the assumption that the mark and time distributions are conditionally independent given the historical
embedding (hhchur et al. (2021); Lin et al. (2021), we first. transform the historical encoding h;-1 to the discrete distribution's 's logit scores
k xhh,1)) = logit(
(26)
where lon E R.M, k RDD - R Then, soflmax function is used to transform logit scores into the categorical distribution, as
p(A; m)0(h; softmax((ogit(ff)))
(27)
where (fftmax((ogit(())))) means choose the m-th element after softmar's output. In training, a cross-entrops loss for categorical classification CE,- CE(p(m, (1((p(,,)h;1)) will be addod to the loss term, leading the final loss of single event to
I 4 + CF.
(28)
10

---

Published in Transactions on Machine Learning Research (08/2022)
4 Related Work
Deep Neural Temporal Point Process. From Du et al. 2016 which firstly employed RNNs as history encoders with a variant of Gompertz distribution as its probabilistic decoder. Following works, proposed to combine deep neural networks with TPPs, have achieved great progress (Lin et al. 2021 Shchur et al. 2021 h. For example, in terms of history encoder, a continuous time model (Mei & Eisner 2017 used recurrent units and introdduces a lemporal continuous memory cell in it. Recently, allention-based encoder (Zhang et al., 2020a Zuo et al. 2020) is established as new taate-of-the-art history encoder. In probabilistic decoders, Omi al. (0019) fit the cumulative harzard function with its derivative intensity function. Xiao et al. (2017b) and Shchur et al. (2020a) used the single Gaussian and the mixture of log-normal to approximate the target distribution respectively. In events dependency discovering, Mei et al. (2022) explicitly modeled dependencies between event types in the self-attention layer, and Zhang et al. [2020b] implicitly captured the underlying event interdependency by fitting a neural point process.
Probabilistic Generative TPP Models. A line works have boen proposed to deploy now progress in deep generative models to TPPs. For example, the reinforcement learning approaches which are similar to adversarial settings, used two networks with one generating samples and the other giving rewards are proposed sequentially (Upadhyay et al.) 2018 Li et al. 2018). And adversarial and discriminative learning (Yan et al, 2018 Xiao et al.) 2017a) have been proposed to further improve the predictive abilities of probabilistic decoders. Noise contrastive learning to maximize the difference of probabilistic distribution between random
noise and true samples also proved to be effective in learning TPPs (Guo et al. 2018
Mei et al. 2020
5 Experiments
5.1
Experimental Setup
5.1.1
Implementation Description
EDTPP
GNTPP
Conditional Intensity
History Encoder
Generative Model
Gompertz
(Revised) Attention
TCDOM
TCNSN
RNN
Webull
Gouesian
FNNint
TCONF
TOVAE
TOGAN
Figure 4: The hierarchical description of our experimental framework with modules integrated in GNTPP.
We first introduce our experimental framework for model comparison, as shown in Figure In the history encoder module, it includes: GRU, LSTM and TRANSFORMER with and without our revised attention. In the probabilistic decoder module, probabilistic models in EDTPP (Lin al., (021)) with closed likelihood including GAUSSIAN, GOMPERIZ, LOG-NORM, FRED-PORWARD NETWORK, and WEIBULL are implemented and integrated into our code. And the discussed neural generative models which is classified into our GNTPP including TCDDM, TCVAE, TCGAN, TCCNF and TCNSN are implemented.
5.1.2 Datasets
We use complex synthetic dataset which simulated by Hawkes process of five types of events with different impact functions (Appendix B.1.) and real-world datasets containing event data from various domains: MO0C (user interaction with online course system), Retweet (posts in social media), Stack Overflox
11

---

Published in Transactions on Machine Learning Research (08/2022)
(question-answering badges in website), Yelp (check-ins to restaurants). The data statistics are shown in Table 2 We clamp the maximum length of sequences to 1000. The dataset is split into 20% ratio for testing, 80% ratio for training with 20% in training set as validation set used for parameter tuning. All the time scale [0, Tmax is normalized into [o, 50] for mumerical stability, where max is the maximum of observed event occurrence time the training set. The detailed description given in Appendix B.1.
Table Dataset Statistios
Dataset
# of sequences
Mean length
Min length
Max length
of event type
MOOC
7047
493
Retweet
24000
108.75
50
264
3
Stack Overflow
6633
72.42
41
736
22
Yelp
300
717.15
424
2868
1
Synthetic
6000
580.36
380
835
5
5.1.3 Protocol
In the training process, pppee-prraeeters of every model are tuned in the range of learnnig rate': {1 x 10 x 10-1,1 x 10-"), 'embedding size' : {8, 16, 32}, 'layer number's {1, where 'embedding size' is the dimension of historical encoding, i.c. D. The hyper-paramcters are tuned on validation set. The maximum training epoch number is set as 100, and early stopping technique is used based on values of loss on validation set. The reported metrics the results of models trained with the lowest loss, except 'TCGAN" probabilistic decoder, whose parameters are choosen as the final epoch's 's results. The mean and standard deviation of cach metric is obtained according to experiments' results with different random seeds.
5.1.4 Metrics
To evaluate the predictive performance of each methods, deploy commonly used metric 'mean absolute percent error' (MAPE) for measuring the predictive performance of next arrival time (Zhang et al.) 2020a) and 'top-1 accuracy" (Topl_ACC) and 'top-3 accuracy' (Top3_ACC) to measure the predictive performance of next event. types Lin et al.] (2021). h. Note that there is only one event type in Yelp, so 'Top3_ACC* meaningful. However, the commonly used negative likelihood has no closed form in deep generative models. Therefore, we use another two metrics evaluate the "goodness-of-fitnee probability score' (CRPS), which is widely used in time series prediction (Rasul et al. 2021 Ben Taieb 2022 for mcasuring the compatibility of cumulative distribution function (CDF) F with an observation
ACC' is not
The first is 'continuous ranked
CRPS(F,() (F(()) - I(t 5 "))² dy. Regarding the model fitting next event arrival time's distribution (Jordan et al. 2018), we can calculate it on a single timestamp by using the empirical CDF as
CRPS(F,6) 1 5
C4.E 41
Be.M - l2,ele
(29)
252
where there are S samples (le, drawn from 1o( 4 is the ground truth observation. Equation. 29 reflects that CRPS can also evaluate models' the sampling quality as predictive performance in the first term. and the sampling diversity in the second term. Another metric "QQPlot-deviation (QQP-Dev) (Xiao et al, 2017a which can be calculated by first computing the empirical cumulative hazard function Aţ(t), and its distribution should be exponential with parameter 1. Thorefore, the deviation of the QQ plot of it v.s. Exp(1) is calculated, as metric 'QQP-Dev' Appendix B.2. gives details.
5.2 Performance Comparison
Here we choose methods whose probabilistic decoders are not generative models as baseline for performance comparison:
(1) RMTPP (Lin al. 2021 as the extension of RMTPP (Du al. 2016 whose probabilistic decoder
mixture of GOMPERIZ distribution.
12

---

Published in Transactions on Machine Learning Research (08/2022)
(2) LOGNORM (hhchur et al.
2020a.. which uses LOG-NORMAL distribution as its decoder.
(3) ERTPP (Lin et al. 2021 as the generalization of ERTPP (iao et al. 2017b), with a mixture of
GAUSSIAN as its decoder.
(4) FNNINT (Omi et al. 2019 which formulates the cumulative harzard function as a fully-connected FEED-FORWARD NETWORK, and its derivative w.r.t. time intensity.
(5) WEIBMIX MMarín et al. 2005 Lin et al. 2021 with WEIBULL mixture distribution as its decoder.
(6) SAHP (Zhag et al. 2020a), whose conditional intensity function is an exponential-decayed formula- tion, with a softplus as nonlinear activation function stacked in the final.
(7) THP Zuo et al. 2020 ,, whose intensity function reads X*(r) = softplus(az + W + +).
(8) DETER as a baseline for MAPE and ACC metrics, which is a lotally deterministic model with the probabilistic model replaced by a linear projection head whose weight and bias are all constraint to be positive, and trained by MSE loss as a regression task.
Table 3: Comparison of different methods' performance on the real-world datasets. Results in bold give the top-3 performance, where DETER is excluded. The comparison on NLL or ELBO is given in Appendix B.3.
Retvees
Mellook
MAPE(-]
CRPSC)
QUP Dev(4)
Topl_ ACC(t)
TTipl ADC()
MAPREE
CRPSSL
QQP (ev(4)
Rapl AOU(())
Top3 ACD
DErSS
17.6366x..75
0.300
1:000
RMTPP
67.2886463301
37. 3539
1967740002
0. 0.000
66.1180+-3347 6.110000011
1.7006a0.ess ress
0.6086 con
1.0000-00000
LOGNoRM
70.000G000000
36. 6712
1.6674
0.3002200333
0.7155 oca3
26.8065
0.4599..0000
1,7091 anc
1.00000
ERTPP
94.771100333
24.41332
1557]at.00s
0.704l2n ooes
71.5001 EAL0003
0 22282222
1.728tan.003
0d005S 2012
1:00000000000
WEISMIX
75.257041.1154
18. 1532
1977611.000
0.3400
0.6613
72. 22.501550000
0.37052
1.077
0.6088
1.0001000000
FNNIST
60.570554.4913
0.4203:0:335
0.7310
22.7680
1.0318
5368
SAHP
45.08/74664177
1.03364..0r
0.0527-0
1.0288 ress
0.0032-a roe
1:0000-0000
THP
41.057623.1192
1.0207...00
0,328720.3977
0.6531 Bone
16. 40044
1.02442:010004
0.565122 m000
1:00002000000
TCDDM
28.565944.3904
0.14
L10004c.00
0.4308-0:299
0.7408-0 m0s4
12.6058
0.2076-3-000
1.0127
0.6274 -a onn
1.0000-00000
TOVAK
9011
0,1465
4006944.000
0.8177400351
0.628220
12.2832
0.1868.5.000
1.044Ban.ces
0.5825
1.0000400000
TOGAN
24.418414.477 a.
1001224.001
0.4179
6.7270nn
25.00M1a11241
0.30841300044
0.6263
1.00091100000
TOCNI
26.319741.700
1.1636
108784..000
0.4297 L00005
13 13.06652110933
0.1625.2.002
2.008
0.5963 onee
1.0000120000
TCNSN
80.8511440077
133008
I 1.305ar.om
0.32922003113
63.399521.22
1. 11954:200
1,3291
0.5845 maz
1.000
Btack Ovezflow
Yelp
Methods
MAPE(1
CRPSIL)
QQP_ Dev(i)
Topl_ ACC
Topa ACC(T)
MAPECI
CRPS:L)
QQP
Dev(1)
Topl.
(OC(1)
Tap5_ ACC!T)
DETER
4.751840.934
0.5302400333
0.8827 0004
16.981 449
1.0000a 0000
TPP
7.0346211.400
6.25442
15017at.01a
0,53422300311
0.8055:n1
13.8 23.657
0.0057
10000
LOGNORM
13.9008x1.22:4
6.3577
15013
0.5335
0,8512
52. 12.16000am
0 20:40-
1.2840
1.000ss
ERTTP
17.200811.524
4.5747
152291r.a0s
0.5316110332
0.8526in
34.8052240003
1.26322010077
1.0000200000
WensMox
7.6280a1.2033
2.3028
1997881..000
0.502Faww0::
0,8654 oone
34,839 Ex0000
0 0.00002.0000
1.9770xxteee
100000000000
FNNINY
6. 6.1583en
1. 15725
0. 0.5338
0. 188332ssmm sene
16. 4022755
1,2570 core
1.0000s0 roo0
SAHP
5.521044:997
1.517744:000
0. 0.5235-40090
0 0.8278-0 erea
12 1474
1.0755 r4r4
1.0000+a con
5.63014nasaa
1.50884como
0:5310-0-331
14,4189400424
1.077540.0044
1.00004..000
TCDDM
4.904710305
18711
0.5371
0.8093.n1ca ocoe
10.84262xx1253
0.0570
1.1728
10000220
TOVAE
5.1397100995
0.51 oon2
0.84185 1032
9.8204:102933
0 103
=.1752
1000000:000 1000
TOGAN
5.08744403977
0.5458-00004
1.5178+r00s
0.53:0--0-544
0.8461 -o coee
12.047 me0
0 0.0608 -
1.0000+a 1oo0
TOCNT
6,302240041
0.4258
163194
0.5428 10438
0.8731.00000
23.4562
0.0575
1.0000s0.000
TCNSN
0317
0.8350
1600
0.5352:409122
0.8538
13.0613+22311
0.4274
a.5855
1.0000ss
The methods of (1) ~ (4) have closed-form expectation. Mean of FNNINT, SAHP and THP is obtained by numerical integration, and mean of GNTPP is obtained by Monte Carlo integration thanks to its advantages in flexible sampling. Note that it is possible to sample events from SAHP and THP using Ogata's thinning method (Ogata) 1981) since the intensity for both methods is monotonically decrcasing between events. Samples can also be drawn from FNNINT model using numerical root-finding (Shchur et al. 2020a), but these sampling methods designed especially for the three models have not yet been developed. Therefore, flexible sampling is not allowed for FNNINT, SAHP and THP (Table so we do not report their *CRPS: In all the generative methods, the trick of nog-orramlization used: The input samples are firstly normalizod by log M-eenn(logr) r) during training, and rescaled back by exp (log 7 VVr((oo7)) + Mean((ogT)) to make sure
Var(log T)
the sampled time intervals are posilive.
For fair comparison, we all use revised attentive encoder which is a variant of TRANSFORMER to obtain the historical encodings.
We conclude from the experimental results in Table that
13

---

Published in Transactions on Machine Learning Research (08/2022)
All these established generative methods show comparable effectiveness and feasibility in TPPs, except. TCNSN as score matching method. TCDDM, TCVAE and TCGAN usually show good performance in next arrival time prediction, compared with the diffusion decoder.
In spite of good performance, as a continuous model, TCCNF extremely time-consuming, whose time complexity is unaffordable as shown in Appendix B.4.
By using the numerical integration to obtain the estimated expectation of SAHP and THP, we find they can also reach comparable 'MAPE' to generalive decoders. However, the two models do not provide a flexible sampling methods, where the time interval samples cannot be flexibly drawn from the learned conditional distribution.
From 'CRPS' and 'QQP_ Dev' evaluating models" fitting abilities of arrival time, the generative decoders still outperforms others. For 'QQP_Deys SAHP and THP's show very competitive fitting ability thanks to its employing expressive formulation as the intensity function.
For the Synthetic dataset, results are given in Appendix B.3. In summary, the empirical results show that proposed generative neural temporal point process employs and demonstrates deep generative models' power in modeling the temporal point process.
Further Discussion on Generative Models. As TCDDM, TCCNF, and TCNSN can all be classified into score-based methods according to Song et al. 20211b, in which they are described as different stochastic differential equations, this raises a question to us: Why only TCNSN fails to model the temporal point process effectively? Following the former work (Song et al. 2021b; Song & Ermon 2019 2020), the continuous form of temproal conditional score-matching model is given by the stochastic differential equation (SDE)
dk)²(k)) dk
dr =
dw,
(30)
where w is a standard Wiener process. is a variance exploding process because the variance goes to infinity as k +00. In comparison, the forward process of temporal conditional diffusion model can be regarded variance perserving one, as
dr 2 /
2 B(k)rdk + VB(k)dw.
(31)
And temporal conditional continuous normalizing flow the deterministic process where dw 0, as
dr = So(T, k)dk;
(32)
where fo is a learnable neural network. Note that we omit the conditional notation the single event modeling for simplicity.
In the reverse process which is used for sampling (or denoising), these three models firstly sample tk NoO, 1). Tk is denoised by the learnable score function eg in TCDDM and TCNSN, invertible process in TCCNF, and To as the output of the final stage of the process is generated as the time interval sample. For the variance exploding property of TCNSN, in the reverse process as shown in Figure 5 the variance of the distribution will firstly become excessively large. As a result, later in small-variance scales, it cannot. recover the input signals and distributions attributing to the high variance of the early stage. In comparison, the variance in the sampling dynamics of TCDDM and TCCNF keeps stable, and the learned distributions are approaching the input gradually in the iteration of denoising process.
5.3 Advantages of Revised Encoders
In this part, we aim to conduct empirical study to prove the better expressivity our revised attentive encoder. We first fix the probabilistic decoder as diffusion decoder, and conduct experiments with different. history encoders, including our revised attentive (REV-ATT), self-attentive (ATT) and LSTM encoders demonstrate the advantages in expressiveness of the revised attention. Table shows the advantages of the
14

---

Published in Transactions on Machine Learning Research (08/2022)
k=0.9
k=0.8
k=0.7
k=0.6
k=0.5
k=0.4
k=0.3
k=0.2
k=0.1
k=0.0
em 1.00
a-0.89
--0.83
m00.7
mm0.72
em0.66
e-0.57
am0.53
m00.49
TCDOM
.-100
m-0.54
TCCNF
m1000
m=5.67
119.65 -0066
m21.02 2m.00 222.04
TONSN
20
20
20
20
Figure 5: The empirical distribution of the sample generating dynamics of TCDDM, TCCNF, and TCNSN. The visualization conducted in Stack Overflow dataset obtained by 5000 samples, with the iteration step set 100 in TCDDM and 1000 in TCNSN. We regard the time scale as [0, 0.9), and give the dynamics of the distribution change of the reverse sampling process at different discrete time points. As demonstrated, the variance of TCNSN is much larger than others, which prevents the model from recovering the distribution of the imput samples. Log-noramlization trick is used to force the intermediate samples to be positive while a is calculated with unnormalized latent variables for consistent order of numerical values.
Table 4: Comparison of different history encoders.
Good Anower
Gurs
Popular Question
Enosdes
MAPE
CRPS
QQP Dev
Topt. ACC
Top3 ACC
Famoux Quedtion
LSTM
0.1608a2000
0.4232a:.004
0.7279.4.0011
Mice Answer
ATT
0.1488450033
0.422645:001
0.7276-4.000
REV-ATT
29. a91
0.4308+2009)
0.7408-5.01
Nacromancer
Eusades
MАРE{))
CRPSSL
QQP (ev(I)
Topl ACD(T)
Top3. AOO(T)
Promoter
Yearing
3.0
LSTM
0:2079450001
L0022-8008
1.60834:.000
1.000-4.000
Reshal
ATT
10.3160-4
0.0033+1.000 0.6274-50025
Exightened
REV-ATT
15,0058
0.2076-3408
1.000-4.000
Populizz
Stack Ovwrflow
Fnosdes
MAPP(()
CRPS(1)
QQP Dev(1)
Top1 ACCIT)
Top3 ACC(1)
5.0381-4
0.533741-000
0.8626-4
Saltr Question
ATT
1.6683-0081
0.5826442000
118832-4.000
Booster
REV-ATT
0.4375
0.5371 93:344
0.8093-4/000
Publicist
Yelp
Eussdess
MAPE
CRPS
QQP
Dov
Top1 ACC
Top3 ACC
Figure 6: The relations similarity between event types of Stack Overflow which infer- enced by REV-ArT TCDDM. Rows are ar- ranged in the same order columns.
LSTM
10.908244000
1,179240082
1.000450000
ATT
10.01 ma
0.0571 40:00
1.02
0.000++:0000
RES-ATT
10.8
0.0570-30000
1.1728-6-8s -onss
1.00040amm
revision on two datasets, the revised attention outperforms others most metrics. Results on Synthetic dataset are shown in Appendix B.3.
The revised attentive encoder achieves overall improvements by small margin. The complete empirical study [Lin et al. 2021] has illustrate that the performance gain brought from history encoders is very small, and our revision can further brings tiny improvements.
Second, we give visualization shown in Figure on the events' relations similarity obtained by Em ET as discussed Section 3.1. It shows that the effects of some pairs of event types are relatively significant with high absolute value of event similarity, such as (Stellar Question, Great Answer) and (Great Question, Constituent). It indicates the statistical co-occurrence of the pairs of the event types in sequence.
15

---

Published in Transactions on Machine Learning Research (08/2022)
5.4 Hyperparameter Sensitivity Analysis
Several lyppe--araaeters affect the model performance, and in this part we try to explore their impacts. We conduct experiments on different 'embedding size', i.c. D and 'layer number!. The partial results of TCDDM are given in Figure and and details are shown in Appendix B.4.
(a) MAPE
(b) CRPS
(a) MAPE
(b) CRPS
Figure 7: Change of Performance with layer number of TCDDM on MOOC.
Figure 8: Change of Performance with embedding size of TCDDM MOOC.
The MAPE metric is more sensitive than CRPS with the change of the hyperparameter of the model. In MOOC dataset, the large 'layer number' and 'embedding size' is not beneficial to predictive performance.
6 Conclusion
A series of deep neural temporal point process (TPP) models called GNTPP, integrating deep generative models into the neural temporal point process and revising the attentive mechanisms to encode the history observation. GNTPP improves the predictive performance of TPPs, and demonstrates its good fitting ability. Besides, the feasibility and effectiveness of GNTPP have been proved by empirical studies. And experimental results show good expressiveness of our revised attentive encoders, with events' relation provided.
A complete framework with a series of methods are integrated into our code framework, and we hope the fair empirical study and easy-to-use code framework can make contributions advancing research progress in deep neural temporal point process in the future.
Acknowledgement
This work is supported part by National Natural Science Foundation of China, Geometric Deep Learning and Applications in Proteomics-Basee Cancer Diagnosis (No. U21A20427). We thank a lot all the reviewers who responsible, careful and professional in TMLR for their valuable and constructive comments.
References
Martin Arjovsky, Soumith Chintala, and Léon Botiou. Wasserstein gan, 2017.
E. Bacry, M. Bompaire, S. Gaïlfas, and S. Poulsen. tick: Python library for statistical learning, with particular emphasis time-dependent modeling. ArXiv e-prints, July 2017.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models, 2022. URL htttps://axiii wttp:o/////....oo..////gg/ http:s//arxiv.orr/aaa//001. 06503
Souhaib Ben Taieb. Learning quantile functions for temporal point processes with recurrent neural splines. In Gustan Camps Valls, Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statisties, volume 151 of Proceedings of Machine Learning Research, pp. 3219 3241. PMLR, 28 30 Mar 2022. URL https::/proceeeiingg mlr wttpps//115///ee-aaaeebeeo otg.:/ecenaaa....prreeese////aaa
html
Ricky Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud, Neural ordinary differential equations, 2019.
16

---

Published in Transactions on Machine Learning Research (08/2022)
Ricky T. Q Chen, Brandon Amos, and Maximilian Nickel. Neural spatio-temporal point processes, 2021.
D. Vere-Joncs D.J. Dalcy, An Introduction to the Theory of Point Processes, volume 1. Springer-Veriae Verllg New York, 2003.
Yihe Dong, Jean- -ean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, 2021.
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Komez-Rooriguue and Le Song, Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016.
Michael Eichler, Rainer Dahlhaus, and Johannes Dueck. Graphical modeling for multivariate hawkes processes with nonparametric link functions, 2016.
Ruocheng Guo, Jundong Tí, and Huan Liu. Initiator: Noise-contrastive estimation for marked temporal point process. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 2191-2197. International Joint Conferences Artificial Intelligence Organization, 2018. doi: 224963/jccai220 2011/303.83... URL https://doi..... http:///oo.org//////// www.r0ggg//... http:///0..9933jcaaa........ 2018//33
Alan G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1): 83 83-90, 1971. ISSN 00063444. URL. http::/www. www.:/ssttr.rrsstaabe/22333333
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Valerie Isham and Mark Westcott. self-correcting point process. Stochastic Processes and their Applications. 8(3):335 347, 1979. ISSN 0304-4149. doi: http:///0boorr//0000000000111110
URL https:
/ www.sciencedirect.com http:///ww...o.aoeccccec///rrtee ott.accch......./.ooooo.ooooo//a
Alexander Jordan, Fabian Krüger, and Sebastian Lerch. Evaluating probabilistic forecasts with scoringrules, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2014.
Shuang Li, Shuai Xiao, Shixiang Zhu, Nan Du, Yao Xie, and Le Song. Learning temporal point processes via reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol-
ume 31. Curran Associates, Inc., 2018. URL https://procceedings www.iigs..uur.r..o wew.rppp.ccppppe/2118///// Pace:o--pppppppppp
Haitao Lin, Cheng Tan, Lirong Wu, Zhangyang Gao, and Stan. Z. Li. An empirical study: Extensive deep temporal point process, 2021.
J. M. Marín, M. T. Rodrigucz-Bernal, and M. Wiper. Using weibull mixture distributions to model heterogeneous survival data. Commemications in Statistics Simulation and Computation, 34(3):673 684, 2005. doi: 10. L00000/AAA2000-0S00
URL https://doi http:////oooogg/..1000000 ort./.. 1ttg//00.0011/AA-20005555
Nazanin Mehrasa, Ruizhi Deng, Mohamed Osama Ahmed, Bo Chang, Jiawei He, Thibaut Durand, Mar- cus Brubaker, and Greg Mori. Point process flows, 2020. URL https://openrevier: http:///ppenrreeeeeeeer////rr//
EK1J2CEYPHI
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL https://proceedinggs nwwrins. 3455c3440aaabbe256444664aaaa Paper .ddf.
Hongyuan Mei, Tom Wan, and Jason Eisner. Noise-contrastive estimation for multivariate point processes, 2020.
17

---

Published in Transactions on Machine Learning Research (08/2022)
Hongyuan Mci, Chenghao Yang, and Jason Eisner. Transformer embeddings of irregularly spaced events
and their participants. In International Conference on Learning Representations, 2022. URL https http://rwiwe..ttttooorr//rdeerie
Y. Ogata. On lewis' simulation method for point processes. IEEE Transactions Information Theory, 27 Ttt.//TT...1111116111
(1):23-31, 31, 1981. doi:
Takahiro Omi, naonori ueda, and Kazuyuki Aihara. Fully neural network based model for general temporal point processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associales, Inc., 2019. URL https: p
Zhen Pan, Zhenya Huang, Defu Lian, and Enhong Chen. A variational point process model for social event sequences. Proceedings of the AAAT Conference on Artificial Intelligence, 34(01):173-188, Apr. 2020. doi: 10.1609/aaai.v3440) v3..0....3. URL https://ojs. aaai. httpa//wwwppp//.aaaaaaaaaaaa///a
George Papamakarios, Eric Nalismick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. 2019. doi: 10.48550/ 10.4855//RRXV.11112.. 1911V.1912 02762. URL
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariale probabilistic time series forecasting, 2021.
Oleksandr Shehur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. In International Conference Learning Representations, 2020a. URL httpps///peerrrreee...ottttoor Hd=HygOjhEYDH
Oleksandr Shchur, Nicholas Gao, Marin Bilos, and Stephan Günnemann. Fast and flexible temporal point processes with triangular maps, 2020b.
Oleksandr Shchur, Ali Caner Türkmen, Tim Jamuschowski, and Stephan Günnemann. Neural temporal point processos: A review, 2021.
Alexander Soen, Alexander Mathews, Daniel Grixti-Cheng, and Lexing Xie. Unipoint: Universally approxi- mating point processes intensities, 2021.
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256-2265, Lille, France, 2015a. PMLR. URL http:://proceedings..... mlr. www..r..prree//7////oooo---aaaaa html
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015b.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In
H. Wallach, H. Larochelle, A. Beygelzimer F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 11918 11930. Curran Associates, Inc., 2019. URL https? pdf
Yang Song and Stefano Ermon. Improved Techniques for Training Score-Based Generative Models. In
H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alchh-BBu,, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020. URL https: .pd]
Yang Song, Conor Durkan, Jain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models, 2021a. URL http://www......../.....////aaaa
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021b.
18

---

Published in Transactions on Machine Learning Research (08/2022)
Utkarsh Upadhyay, Abir De, and Manuel Gomez Rodriguez. Deep reinforcement learning of marked temporal point processes. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https::/prooeeedingg wwwriis...r......ee http://iinn...uiii....oppaaappaa httpp///.ccc/appe//////////00 .ddf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL http:
03762
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In JCML, 2011.
Shuai Xiao, Mehrdad Farajtabar, Xiaojing Ye, Junchi Yan, Le Song, and Hongyuan Zha. Wasserstein learning of deep generative point process models, 2017a.
Shuai Xiao, Junchi Yan, Stephen M. Chu, Xiaokang Yang, and Hongyuan Zha. Modeling the intensity function of point process via recurrent neural networks, 2017b.
Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation, 2021.
Hongteng Xu, Mehrdad Farajtabar, and Hongyuan Zha. Learning granger causality for hawkes processes. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1717 1726, New York, New York, USA, 20-22-400 Jun 2016. PMLR. URL attp:///rroceedings mlr wwwpppwws/sseeeppeeppe xuc16 html
Junchi Yan, Xin Liu, Liangliang Shi, Changsheng Li, and Hongyuan Zha. Improving maximum likelihood estimation of temporal point process via discrininative and adversarial learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 2948-2954. International Joint Conferences on Artificial Intelligence Organization, 2018. doi: 10.24963 10.24963/ijcai.2000 /ici:////aa..///000000 URL httpp:://doot orgg 10. 24963/ijcai.. 2018/409.
Qiang Zhang, Aldo Lipani, Omer Kirnap, and Emine Yilmaz. Self-attentive Hawkes process. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11183-11193. PMLR, 13-18 Jul 2020a. URL .ewpediiiinnmm.......mmmmm.mm. mlr, wttpa//v119//hhaagg html
Qiang Zhang, Aldo Lipani, and Emine Yilmaz. Learning neural point processes with latent graphs. In In Proceedings of the Web Conference 2021 (WWW 21), 2021. URL http::://o..org/00.11100.4 1145//44433/
3450135
Wei Zhang, Thomas Kobber Panum, Somesh Jha, Prasad Chalasani, and David Page. Cause: Learning granger causality from event sequences using attribution methods, 2020b. URL
org/abs/
2002.07906
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer Hawkes process. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11692-11702. PMLR, 13-18 Jul 2020. URI.
mlr. presss/119/zuo2000 html
A Preliminaries on Temporal Point Process
Temporal point process with markers. For a temporal point process {t.)21 as a real-valued stochastic process indexed on N+ such that T, S T almost surely (here T, representing the random variable), each random variable is generally viewed as the arrival timestamp of an event. When each timestamp is given a type marker, i.e. ((1, m2), the process is called marked temporal point process, also called multivariate
19

---

Published in Transactions on Machine Learning Research (08/2022)
point process as well.
Conditional intensity function and probability density function. As defined in Eq. 1 the probability density function and cumulative distribution function can be obtained through
X(t)
Et + dt) - (()))((()
PP(t; E + dt)
IP(4 E [4, di)l4 $ [4-1,, H(1;-1))
P(t, E [t,t+ + dt),t, t, $ [t-,1,t)
P(4 & 14-1, ()2())))
P(1, [1,4 + d() H((--1))
P(t: & ((-(tt))
p(t)H((t-)))
1 P(17((4-1))
(1)
1 P*(1))
In this way, the reverse relation can be given by
p=(1)= xxx))xxx( f (T())dr);
11 = 1===le exp(- f ((T)dr).
14-1
Example 1. (Poisson process) The (homogeneous) Poisson process is quite simply the point process where the conditional intensity function is independent of the past. For example, X (² X(t) = which is a constant.
Example 2. (Hawkes process) The conditional intensity function of which can be written as
(t) a Σ g(- tzims,B.). 2,<t
which measures all the impacts of all the historical events on the target timestamp t. The classical Hawkes process formulates the impact function g(t - bi B) = e)xp(((( - -t)) as the exponential function.
B Experiments
B.1 Sythetic Dataset Description
The synthetic datasets are generated by tick packages (Bacry al. 2017). using the Hawkes process generator. Four Hawkes kernels impact functions used with each process's 's intensity simulated according to Example 2, including
(t) 0.09 exp (0.4))
96 (t) 0.01 exp( -0.8t) + 0x03 0.05 e00(x000000
9e (t) 0.25 cos 30 exp(-0.11))
(t) (0.5 -t
The impact function (t) measuring impacts of type i type j is mnffrrmyy-randome chosen from above. A probability equalling to which we called cutting natio is set to force the impact to zero, thus lcading the Granger causality graph to be sparse. The cutting ratio is set as 0.2, and the total nunber of types set as 5.
https://github... con wwt.a///a/aaaatttaaiieeeetta
20

---

Published in Transactions on Machine Learning Research (08/2022)
B.2 Calculation of QQP_Dev
If the sequences come from the intensity function of point process X(t) then the integral A f411 X(T) (r)dr between consecutive events should be exponential distribution with parameter 1. Therefore, the QQ plot of A against exponential distribution with rate 1 should fall approximately along a 45-degree reference line. Therefore, we first use the model to sample series timestamps, and use them to estimate the empirical A Mean absolute deviation of the QQ plot of it v.8 Exp(1) from the line with slop 1 is 'QQP DEV.
B.3 Supplementary Results
We first give supplementary results of different methods on the Sythetic dataset shown by Table
Table 5: Comparison Sythetic dataset.
Synthetic
Methods
MAPE
CRPS
QQP Dov
Topl_ ACC
Top3_ACO ACC
E-RMTPP
22.8206+1.Gmm 0.1905+0.0010
1.7921 +0.0047
0.2497=0.000
0.6693-0.00x
LOGNORM
0.1916+0.0102 +4.620880.0000
1.7920++00044
0.2494+0.000
0.6687=0.008
E-ERTPP
54.6220.10.0m 000000000000000
10.0022
178811100006
0,2482.10.001 001011
0.6674_0.0012
WEIBMIX
26.591017:3426
0.106010.0029
1.977610.om 0.0000
0,247610.oo 00001
0.6668-0.008
FNNINT
4.522310.09r7
1.3344
0,2531
0.6724
SAHP
4.5198
1.1775++0.0002
0.2964+0.000 00.0003
0.7269-0.000)
THP
4.4958+0.138
1.1775++0.000
0.2474+0.00m
0.6667-0.0002
TCVAE
20.0000
0.061 10.0000000001
1.412442..0z2
0,2476
0.6670=0.000n
TOGAN
3.5009+0.112 1268
0.25 10010++..00 2880
1.4297+0.0222
0.19244+0.080 0.0884
0.5059-0.2438
TOCNF 4.7095++0.03m
0.0654+0.o0m
1.5787+0.0000
0.2465 0000
0.6669=0.00m
TONSN 33.9541 +0.9428
0.1109
1.5884++00000
0.2554 0001
0.6766=0.000
TCDDM
3.2323
0.050920.0mm
1.4061
0.2492=a.0022
0.6686=0.0019
We give the negative ELBO which is the upper bound of models' NLL of TCDDM, TCVAE, and TCNSN, and the exact NLL of other models except that TCGAN we give Wasserstein distance between the empirical and model distributions in the four real-workd dataset, as shown in Table
B.4 Complexity Comparison
We provide each methods mean training time for one epoch to figure out which methods are extremely time-consuming. shows all these methods are affordable in time complexity except CNSN, and CGAN also time-consuming. The test implemented on a single Nvidia- Nvidia-V100(325511 (350003)))) Tn all the test setting, batch size set as 16, embedding size is 32 and layer number For methods whose likelihood has no closod-form, Monte Carlo integration, where in each interval, the sample number is 100.
Table 6: Comparison on four real-world datasets on the NLL and ELBO.
Methods
MOOC
Retweet
Stack Overflow
Yelp
E-RMTPP
1.7504
- -1-9872
4.9031
-1.0832
LOGNORM
1.3635
-2.4197
4.8782
-1.2808
E- EREPTP
3.5791
-0.8876
5,0845
-0.9678
WEIBMIX
0.7950
2.5110
3.8717
-1.1125
FNNINT
-2.3024
-3.0044
1.8469
-1.2294
SAHP
2.3472
2.9955
1.8348
-1.6607
THP
0.1270
-1.7794
1.8591
-1.6349
TCDDM
$ 1.7609
S 0.7560
2.2450
0.0142
TCVAE
< 9.4754
< 7.5911
< 3.9727
S 6.2181
TCGAN
(0.0056)
(0.0001)
(0.0511)
(0.0560)
TCCNF
-2.8591
-3.0464
1.6881
-1,8791
TCNSN
S 2.1815
S 0.8340
< 2.4131
< 0.3517
21

---

Published in Transactions on Machine Learning Research (08/2022)
Table 7: Comparison of time complexity.
Time per Epoch
Methods
MODC
Retweet
Stack Overflow
Yelp
Synthetic
E-RMTPP
46"
2'08"
1'22"
12
1'26"
LOGNORM
39"
2'02"
1'14"
11"
1'27"
E-ERTPP
12"
2'14"
1'17"
11/
1'33"
WEIBMIX
49"
2'18"
1'27"
13/
1'22"
FNNINT
45"
2'08"
1'16"
14/
1'32"
SAHP
1'11"
2'42"
1'51"
21
2'14"
THP
57"
2'29"
1'30"
18"
2'02"
TCVAE
46"
2'43"
1'33"
11"
1'31"
TCGAN
2'30"
11'24"
3'77"
OOM
4'02"
TCCNF
5'42"
21'28"
7'06"
3'21"
5'47"
TONSN
33"
1'46"
42"
10"
56"
TCDDM
35"
1'39"
1'16"
13"
1'12"
Table Comparison of used memory.
Peak Memory in Training
Methods
MDOC
Retwest
Stack Overflow
Yelp
Synthetic
E- RMMPP
2174 MIB
1544 MiB
4288 MiB
22294 MiB
7074 MiB
LOGNORM
1976 MiB
1544 MiB
4096 MiB
22294 MiB
7078 MiB
E-ERTPP
1976 MIB
1544 MiB
4096 MiB
22294 MiB
7078 MiB
WEIRMIX
2078 MiB
1544 MiB
4290 MiB
22294 MiB
7080 MiB
FNNINT
1828 MiB
1438 MiB
3692 MiB
21366 MiB
10686 MiB
SAHP
7197 MiB
1606 MiB
4706 MiB
22730 MiB
15258 MiB
THP
6004 MiB
1496 MiB
4544 MiB
18384 MiB
7648 MiB
TCVAR
2286 MIB
1436 MiB
3198 MiB
27684 MiB
6854 MiB
TCGAN
3712 MiB
2164 MiB
4906 MiB
0OM
12318 MiB
TCCNF
2598 MIB
1486 MiB
3932 MiB
27836 MIB
4048 MiB
TCNSN
2926 MiB
1662 MiB
3884 MiB
22374 MiB
7984 MiB
TCDDM
3110 MiB
1542 MiB
3508 MiB
22764 MiB
8084 MiB
B.5 Hyperparameter Sensitivity Analysis
We give the CDDM's performance under different parameters on MDOC, Retweet and Stack Overflow, with layer number and embedding size set in range of 31 and (8, 16, 32) respectively. It shows that the large embeddiin size usually brings improvements, so recommend that it should be set as 32. And layer nunber should be set as 1 or 2.
2144
(a) MAPE
(b) CRPS
(c) QQP DEV
(d) Topl
Top3
Figure 9: Change of Performance with Layer Number on MOOC
22

---

Published in Transactions on Machine Learning Research (08/2022)
(a) MAPE
bb) CRPS
(c) QQP
DEV
(d) Topl ACC
(e) Top3 ACC
Figure 10: Change of Performance with Layer Number on Retweet
0.400
0.00
0.0%%
(a) MAPE
(b) CRPS
(c) QQP DEV
(d) Topl ACC
(e) Top3 ACC
Figure 11: Change of Performance with Layer Number on Stack Overflow
214
(a) MAPE
(b) CRPS
(c) QQP DEV QQV_DVV
(d) Sop3_ACC
Figure 12: Change of Performance with embedding size MOOC
23

---

Published in Transactions on Machine Learning Research (08/2022)
048
(a) MAPE
(b) CRPS
(c) QQP
DEV
(d) Topl ACC
(e) Top3 ACC
Figure 13: Change of Performance with embedding size on Retweet
3.80
C10-
MAPE
(b) CRPS
(c) QQP DEV
MECO
0.1-
(d) TCC
(e) Top3_ACC ACC
Figure 14: Change of Performance with embedding size StackOverflow
24


Reviews:
[{"review_id": "94C8x4eTq0", "reviewer": "Reviewer_C6d5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe paper is clearly written. All major modern generative models are considered as decoders. The experimental study is extensive. It should be a very good first paper to read for someone new to this field.\n\nWeaknesses: \n\nMy impression of this paper is that it seems like something between a survey paper and a technical contribution paper. The paper has its technical contributions, but it is somewhat limited, since this is not the first paper applying generative models to TPP modeling. On the other hand, it is also not as systematic and comprehensive as a full fledged survey paper. For example, when comparing different generative models as decoders, there are not many insights on the pros and cons of each model, other than the empirical evaluation.\n\nComing back to the technical contribution part, one claimed contribution is the augmentation of the self-attentive encoder. I think the two problems listed in section 3.1 are definitely important considerations when designing the model. But I would argue that they are not necessarily “problems”, they are more like different modeling assumptions or inductive biases. Incorporating more features or learnable parameters will improve the performance of the model in some cases, but probably not on all datasets.\n\nIn my opinion, the advantages of generative models over discriminative ones are that (1) there is a quantitative evaluation of the goodness-of-fit, for example, the log-likelihood or ELBO; (2) the probabilistic model can be used for sampling or posterior inference. In the experimental study, the evaluation metrics seems to be focusing on the predictive quality of the models, rather than the generative capabilities. I would suggest adding more model comparisons from this perspective.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors study several generative neural temporal point process models. For the history encoder part, the authors modify the self-attentive encoder by adding learnable time encoding and incorporating event type encoding. On the decoder part, various generative models are adopted, including diffusion models, VAE, GAN, normalizing flows, and score-based generative models. Finally, the authors conducted extensive experimental studies to evaluate the performance of different generative TPP models on several benchmark datasets.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "* Provide more insights into the choice of different probabilistic encoders, their respective pros and cons.\n* Consider alternative evaluation methods for generative models if applicable, quantitatively (e.g., comparing likelihood/ELBO) and qualitatively (e.g., latent variable analysis in https://arxiv.org/abs/1506.02216)\n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mZXAZpjIT3", "reviewer": "Reviewer_bbwf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper highlights an important problem with existing neural TPP models (\"good NLL does not imply good sample quality\"). This is highly relevant to the practical applications of TPPs in predictive tasks.\n- This work provides a thorough discussion of the different possible architectural choices in neural TPP models (both encoder, decoder and training procedures) and nicely connects to the previous works.\n\nWeaknesses:\n1. The code provided with the submission is incomplete. The `models` directory that should contain the implementations of all models is missing; and the link to Google Drive with the source code seems to be broken.\nBecause of this I couldn't resolve some of my concerns below by looking at the source code.\n\n2. The introduced \"relation between event types\" mechanism (property P.2 / L184 / Eq. 11) seems to be incorrect.\n    - According the Eq. 11 and Fig. 5, the weights $w_{j, i-1}$ can be negative since the event type embedding matrix $E_m$ is not guaranteed to be positive. Because of this, we cannot interpret the weights $w_{j, i-1}$ as attention coefficients anymore.\n    The denominator in Eq. 11 can even be equal to zero or negative, breaking the attention mechanism.\n    - Even if we restrict the mark embedding matrix $E_m$ to be positive, we still cannot interpret the matrix $W = E_m^T E_m^T \\in \\mathbb{R}^{M \\times M}$\n    in Eq. 11 as modeling interactions between marks / capturing Granger causality in the sense of https://arxiv.org/abs/1602.04511.\n\n        For the correct definition of Granger causality, we need the following property. For each past event $j$ and any mark $k \\in [M]$, it must hold that $W_{m_j, k} = 0 \\implies$ \"$p^*(t_i | m_i = k)$ is independent of event $(t_j, m_j)$\".\n        However, the definition provided in Eq. 11 uses the observed mark $m_{i-1}$ of the previous event number $i-1$, not the possible marks $k$ of the next event number $i$.\n        Therefore, the matrix $W$ doesn't carry the usual intuitive meaning of the Granger causality matrix for TPPs.\n\n3. Certain aspects of the experimental evaluation do not convincingly demonstrate the benefits of the proposed models.\n\n    - The improvements in predictive performance of the new encoder architecture (Table 4) are really minor.\n\n    - The learned similarity matrix (Figure 5) is quite dense, and it's not clear whether the larger values here indicate actual dependencies between event types or noise.\n    See also the point #2 above.\n    A quantitative evaluation compared to other methods for structure discovery (e.g., Hawkes process, [CAUSE](https://arxiv.org/abs/2002.07906)) would be more convincing.\n\n    - It's not clear whether the improved performance is based on the differences between the models or peculiarities of the training procedure (such as early stopping).\n\n        For example, all proposed model variants (TCDDM, TCVAE, TCGAN, TCCNF, TCNSN) as well as the baselines (RMTPP, LogNorm, WeibMix, FNNInt) use the exact same parametrization of the mark distribution $p^*(m_i)$ and are trained using the cross-entropy loss for marks.\n        However, the accuracy scores achieved by different models in Table 3 are quite different, which is very suprising.\n\n4. While the motivation for the proposed approach makes sense (\"low NLL doesn't imply good samples\"), it's not clear whether a much simpler baseline can't achieve the same or even better results.\nFor example, why don't we use a purely discriminative baseline that directly predicts the next inter-event time $\\hat{\\tau}$ and is trained by minimizing MAPE?\nIn my opinion, comparing to such a baseline is important if the focus of the paper is on inter-event time prediction.\n\n\nMinor comments:\n- Several points that need to be clarified in the paper:\n    - Does the LogNorm baseline use a single log-normal distribution or a mixture, as in the original reference?\n    - When computing mark prediction accuracy, do we consider $p^*(m_i)$, the marginal distribution of the next mark, or $p^*(m_i | t_i)$, the distribution of the next mark at time $t_i$?\n    The two quantities are identical for the proposed models where the marks are independent of the arrival time (Sec. 3.4), but are different for the SAHP and TAHP baselines.\n    - Do SAHP and THP models use the original encoders or the modified encoder (Eq. 11)?\n- Some missing citations:\n    - [Zhang et al., ICML 2020](https://arxiv.org/abs/2002.07906) also learn dependencies between event types in a marked neural TPP.\n    - [Mehrasa et al., 2020](https://openreview.net/forum?id=rklJ2CEYPH) also use continuous-time normalizing flows / VAEs to model the inter-event time distribution in TPPs.\n    - [Mei et al., ICLR 2022](https://openreview.net/forum?id=Rty5g9imm7H) also explicitly model dependencies between event types in the self-attention layer of a TPP (Eq. 8).\n    - [Ben Taieb, AISTATS 2022](https://proceedings.mlr.press/v151/ben-taieb22a.html) also uses CRPS to evaluate TPP predictions.\n- L88: Usually $N(t)$ is defined as the number of events in $(0, t]$, not $[0, t)$ (e.g., see in Daley & Vere-Jones).\n- L155: The objective function of WGAN doesn't provide a lower bound on the likelihood, but is an approximation to the Wasserstein distance between the empirical and model distributions\n- L369: It should be possible to sample events from SAHP and THP using Ogata's thinning method since the intensity for both methods is monotonically decreasing between events.\nIt's also possible to sample from the FNNInt model using numerical root-finding.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Temporal point processes (TPPs) are generative models for asynchronous event sequences occurring in continuous time.\nMost recent papers on neural TPPs evaluate goodness-of-fit of these models using negative log-likelihood (NLL).\nThe authors of this work point out that good NLL scores may not be correlated with good predictive performance of TPP models (i.e., good sample quality).\n\nThe main contribution of this work aims to improve the sample quality by modifying the probabilistic **decoders** used in neural TPP models.\nSpecifically, the authors propose several new decoders for neural TPPs based on other modern deep generative model architectures\n- diffusion model\n- variational autoencoder\n- generative adversarial network\n- continuous-time normalizing flows.\n\nThese new architectures combined with the respective training procedures lead to better time prediction compared to existing NTPP architectures.\n\nThe second contribution of this work is a modification to the self-attention **encoder** architecture used in some neural TPPs.\nThe new modified parametrization of the self-attention layer \n- replaces the positional embeddings with exponential smoothing and\n- explicitly models dependencies between different mark types (similar to a low-rank Granger causality matrix)\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The following points are critical for securing my recommendation for acceptance:\n\n1. Clarifying my concerns regarding the proposed attention mechanism and its connection to the Granger causality matrix (see point #2 above).\n2. A quantitative evaluation on the structure discovery task.\n3. Comparing to a purely discriminative baseline that directly predicts the next inter-event time $\\hat{\\tau}$ (see point #4 above).", "broader_impact_concerns": "There is no Broader Impact Statement provided, but I don't believe that there are any major ethical concerns that haven't been addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PJLz12ihhz", "reviewer": "Reviewer_o8BL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The idea is clear and new. \n2. The description of existing methods is clear in Table 1.\n3. Experiments are clear. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper revises the self-attentive encoders with adaptive reweighting for history encoders and adopts recently popular generative models, e.g. Conditional DDPM, VAE and others, to learn the temporal point processes. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. In Line204, what is q(t^0)? Known Standard Gaussian, unknown distribution, or empirical distribution? It is not clear.\n2. In Eq.12, I object to the assumption that q(t^k|t^{k-1}) is Gaussian. By definition, it is obvious that t^k> t^{k-1}, however in Gaussian assumption t^k \\in R. Similar situations exist for p(t^k) and Eq.13.\n3. For the original DDPM models, the interpolation between Gaussian noise and original data is one of the important reasons for good performance. However, similar data augmentation operations do not seem to be used.\n4. The full name of CDD should be specified. And the names of the methods should be related to TPP to distinguish the original models.\n5. The writing of contributions should be improved.\n\nMinor comments:\n1. Line79, exploring->explore\n2. Table 1, Discription-> Description; exsiting-> existing\n3. Line177, P.1. is not clear enough.\n4. Figure.1 -> Fig.1 or Figure 1. Similar situations exist for others.", "broader_impact_concerns": "na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aobwIhzmAd", "reviewer": "Reviewer_Y3mv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:** Application of generative models to Temporal Point processes is an interesting line of research and it is interesting to study if they provide a viable alternative to traditional conditional intensity based modelling. The authors study various families of generative models and the comparison would be appreciated by the community.\n \n**Weakness:** The authors make certain claims that are not backed up by evidence and certain claims that are false. The following are the claims that I have issues with:\n\n1. In the abstract and introduction, The authors claim previous works have focused on maximizing the likelihood of the next event which leads to poor predictive performance and that generative models can be used to improve performance. Again in section 2.1.3 the authors claim *“Differing from the previous works focusing on models’ fitting ability in terms of higher likelihood, these models aim to promote models’ prediction ability, i.e. to generate high-quality samples which are closer to ground truth observations”*.\nI do not quite understand what the authors are trying to imply. All the generative models that the authors have proposed either maximize a bound on the log-likelihood of the predicted event which surmounts to minimizing the KL divergence between the true and the predicted distribution (VAE/Diffusion/Score) or the Wasserstein distance (WGAN). The authors also do not convey why this is expected to improve the predictive performance. This brings me to my second point.\n\n2. If improving predictive performance is something the authors are interested in, an important baseline to be considered is adding this as an explicit penalty term along with the Negative log-likelihood loss as done in Transformers Hawkes Process (THP) (Zuo et. al). \n\n3. In section 2.2, the authors claim the Monte Carlo estimate used in THP and SAHP is biased. This is false, as the THP authors mention that the estimate is unbiased and I have verified it to the best of my knowledge.\n\n4. In section 2.3, to motivate the revised encoder, P1 mentions use time difference instead of actual event time to show a scenario where the embeddings are the same. This seems wrong as $j_1$ and $j_2$ are different, $\\omega(\\tau_{j_1})$ and $\\omega(\\tau_{j_2})$ are going to be different even if $\\tau_{j_1}$ and $\\tau_{j_2}$ are the same.\n\n5. In section 2.1.3, the authors make a comment about the computational cost of the monte carlo integral used in THP and SAHP. As the authors claim their method avoids this cost, It would be interesting to look at a comparison of FLOPs/ Wall-clock time for training/ inference as generative models come with their own overhead.\n\n6. The authors do not mention whether the baseline RMTPP uses the same time normalization they mention in section 4.1.2. Any scaling of the time of the events leads to a change in the negative log-likelihood by a constant factor and makes the comparison meaningless.\n\n**Other Concerns:**\n1. In section 2.3 the authors talk about transformers yet do not cite the original paper.\n\n2. The citation for Transformers Hawkes process which is heavily featured in the paper is incomplete.\n\n3. In section 2.6, The authors should mention their assumption that the mark and time distributions are conditionally independent given the historical embedding\n\n4. In section 4.2, the authors use an example from the tick library as their simulated example thus I think it is important to cite the work in the main paper.\n\n5. It would be nice to see the actual QQPlot to see how close it is to the $y=x$ line instead of just the deviation as done in SAHP.\n\n6. Section 4.2, line 1 is missing the word “decoder”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary of the contributions:** The authors propose a generative framework for neural temporal point process (GNTPP) to apply deep generative models in the context of Temporal Point Processes. The paper breaks down previous research into Temporal Point processes into two parts - historical encoders to generate context from observations and probabilistic decoders that generate a distribution over the next event time conditioned on this historical context. The authors then propose a modification to the attention mechanism of the historical encoder proposed by Zhang et. al and Zuo et. al to incorporate additional context from event types and study different conditional generative models (Diffusion/VAE/WGAN/ Score based/Flows) to learn the distribution of next event time conditioned on this historical context vector. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Requested changes:** Weakness 1, 2, 3, 4 need to be addressed by the authors and are critical for securing my recommendation. Weakness 5 and other concerns 5 would make the paper stronger.\n\n", "broader_impact_concerns": "**Broader impact concerns:** None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies neural temporal point process (TPP) models, and makes the following two contributions:

- It explores a range of modern generative models for the TPP decoder.
- It proposes a modification in the attention mechanism of certain TPP encoders.

The reviewers found both contributions interesting and sufficiently evaluated.

However, as part of the second contribution, the paper contains the following interpretability claim (lines 76--78):

"we revise the self-attentive encoders with adaptive reweighting terms [...] showing better interpretability"

The reviewers were not convinced that this claim was supported by evidence.

I'm willing to recommend the paper for acceptance as long as the authors revise their manuscript to *completely remove all claims* that their proposed modification leads to improved interpretability. At a minimum, I would expect to see the following changes:

- Lines 77--78: delete "showing better interpretability".
- Line 202: delete "To provide interpretation of the model".
- Lines 431--432: delete "prove the better [...] interpretability of our revised attentive encoder".
- Lines 454--455: delete "experimental results show good [...] interpretability of our revised attentive encoders"
Decision: Accept (poster)