Add and Thin: Diffusion for Temporal Point Processes
Authors: David Lüdke
Marin Biloš
Oleksandr Shchur
Marten Lienen
Stephan Günnemann
Venue: NeurIPS 2023 poster (2023)
Keywords: Point Processes
Diffusion
Temporal Data
Generative Model
Forecasting
Density Estimation
Denoising
TL;DR: We connect diffusion models with TPPs by introducing a novel model that naturally models point processes.

Abstract: Autoregressive neural networks within the temporal point process (TPP) framework have become the standard for modeling continuous-time event data. Even though these models can expressively capture event sequences in a one-step-ahead fashion, they are inherently limited for long-term forecasting applications due to the accumulation of errors caused by their sequential nature. To overcome these limitations, we derive ADD-THIN, a principled probabilistic denoising diffusion model for TPPs that operates on entire event sequences. Unlike existing diffusion approaches, ADD-THIN naturally handles data with discrete and continuous components. In experiments on synthetic and real-world datasets, our model matches the state-of-the-art TPP models in density estimation and strongly outperforms them in forecasting.

Add and Thin: Diffusion for Temporal Point Processes
David Lüdke!.2
Marin Biloš!. Oleksandr Shchur!: 4,4, Marten Lienen!.2 1.2 Stephan Günnemann
School of Computation, Information and Technology. Technical University of Munich, Germany Munich Data Science Institute, Technical University of Munich, Germany 3 Machine Learning Research, Morgan Stanley, United States Amazon Web Services, Germany
[d. luedke, m. bilos, o. shchur, m. lienen, s guennemann @@tum. de
Abstract
Autoregressive neural networks within the temporal point process (TPP) framework have become the standard for modeling continuous-tin event data. Even though these models can expressively capture event sequences in a one-step-ahead fashion, they are inherently limited for long-term forecasting applications due to the accumu- lation of еттоrs caused by their sequential nature. To overcome these limitations, we derive ADD-THIN, a principled probabilistic denoising diffusion model for TPPs that operates on entire event sequences. Unlike existing diffusion approaches, ADD- THIN naturally handles data with discrete and continuous components. In exper- iments on synthetic and real-world datasets, our model matches the state-of-the TPP models density estimation and strongly outperforms them in forecasting.
Introduction
Many machine learning applications involve the analysis of continuous- data, where the number of events and their times are random variables. This data type arises in various domains, including healthcare, neu- roscience, finance, social media, and seis- mology. Temporal point processes (TPPs) provide sound mathematical framework to model such event sequences, where the main problem is finding parametrization that can capture the seasonality and complex interac- tions (e.g.. excitation and inhibition) within point processes.
Noising Precess
Denoising Process
Amato
(
Ao 16 1:
(!!)
(88) Anne
Traditional TPP models [17, 20] employ simple parametric forms, limiting their flexibility to capture the intricacies of arbitrary TPPs. In recent years, various neural TPPs have been proposed (see [43] for an overview) that capture complex event interactions in an autoregressive manner,
Figure 1: Proposed noising and denoising process for ADD-THIN. (Left) Going from step 1 to step n, we add and remove some points at random. (Right) Given {(n) and (()) we know the intensity of points at step - 1. We approximate this intensity with our model, which enables sampling new sequences.
Code available https://wwwl cit, tum www.de//eeeeeeeeeee /am/d/mdmm/adda///a/aa/aa thin Work done while at the Technical University of Munich.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

often using recurrent neural networks (RNN) or transformer architectures. While autoregressive models are expressive and have shown good performance for one-step-ahead prediction, their suitability for forecasting remains limited due to the accumulation of errors in sequential sampling.
We propose to take a completely different approach: instead of autoregressive modeling, we apply a generative diffusion model, which iteratively refines entire event sequences from noise. Diffusion models [18, 45] have recently shown impressive results for different data domains, such as images [15, 18, 25], point clouds [31, 32], text [27] and time-series [1, 3, 24, 46]. But how can we apply diffusion models TPPs? We cannot straightforwardly apply existing Gaussian diffusion models to learn the mapping between TPPs due to the particular requirements that must be met, namely, the randomness the number of events and the strictly positive arrival times.
We present a novel diffusion-inspired model for TPPs that allows sampling entire event sequences once without relying on specific choice of parametric distribution. Instead, model learns the probabilistic mapping from complete noise, i.e., a homogeneous Poisson process (HPP), to data. More specifically, we learn a model to reverse our noising process of adding (superposition) and removing (thinning) points from the event sequence by matching the conditional inhomogencous denoising intensity An-1(t (()), ((*) as presented in Figure 1. Thereby, we achieve a natural way to generate sequences with varying numbers of events and expressively model arbitrary TPPs. In short, our contributions are as follows:
We connect diffusion models with TPPs by introducing novel model that naturally handles the discrete and continuous nature of point processes.
We propose a model that is flexible and permits parallel and closed-form sampling of entire event sequences, overcoming common limitations of autoregressive models.
We show that our model matches the performance of tate-of-the-art TPP models in density estimation and outperforms them in forecasting.
2 Background
2.1 Temporal point processes (TPPs)
Temporal point processes (TPPs) [8, 9] are stochastic processes that define probability distribution over event sequences whose number of points (events) K and their locations (arrival times) t; random. A realization of TPP can be represented
sequence of strictly increasing arrival times: t t= (41,... (k), 0 < 1 < ... < tx T. Viewing a TPP as a counting process, we can equivalently represent TPP realization by counting measure ((1) = Σx I(4 S t), for E [0, T]. The intensity characterizing TPP can be interpreted as the expected number of events per unit of time and defined
E[[ + A1) - N(L) H.
A(t| | H4) = lim AcL0
At
where Hi = (4: < <1 is the event history until time i, which acts as a filtration to the process.
TPPs have number of convenient theoretical properties, two of which will be central to our derivation of a noising process for TPPs later in the paper. The first property is superposition: If we combine events generated by TPPs with intensities X (t) and (t).), the resulting event sequence will again follow a TPP, but now with intensity A1 (t) + (z))). Conversely, randomly removing each event generated by TPP process with intensity A(t) with probability p is known as independent thinning. This is equivalent to sampling from a TPP with intensity (1 p))(t) [9].
Poisson process. A (inJhomoeeneous Poisson process the simplest class of TPPs, where the rate of event occurrence is independent of the history. Then the number of points on [0, T' follows a Poisson distribution with rate A(T) = _" X(t) dr. the context of our model, the Poisson process with constant intensity on [0. T), called homogeneous Poisson Process (HPP), represents the noise distribution. Even though Poisson processes can model seasonality, i.e., time-varying rates of event occurrence, they assume the events to be independent and do not capture the exciting or inhibiting behavior present the real world, e.g., a large earthquake increasing the likelihood of observing other carthquakes soon after.

---

Conditional intensity. Most TPP models leverage the conditional intensity function X(t He) or equivalently the conditional density p(t 74,) to overcome the independence points limitation of an inhomogeneous Poisson process. Historically, these intensity models were parameterized using hand-crafted functions [17, 20), whereas now, is more common to use neural networks for learning intensities from data [12, 33, 41]. While the conditional intensity provides a general framework to model TPPs, sampling from these models is inherently autoregressive.
2.2 Denoising diffusion probabilistic models
Diffusion models [18, 45] are a class of latent variable models that learn a generative model to reverse fixed probabilistic noising process Fig #1 - -...... N. which gradually adds noise to clean data To until no information remains, i.c., 2N >((xx)) For continuous data, the forward (noising) process is usually defined Then the Markov chain of the reverse process is captured by approximating the true posterior q To. x..)) with a model po (In-1 xx). Ultimately, sampling new realizations zy from the modeled data distribution po Po = [p(NN) I Po (T Fa) dzj VN is performed by starting with a sample from pure noise ØN p(#N) and gradually denoising it with the learned Z0.
fixed Markov chain g (zn-1) with Gaussian transitions,
model over N steps EN
NN-1
3 ADD-THIN
In the following, we derive a diffusion-like model for TPPs ADD-THIN. The two main components of this model are the forward process, which converts data to noise (noising), and the reverse process, which converts noise data (denoising). We want emphasize again that existing Gaussian diffusion models [18, 45] are not suited to model entire event sequences, given that the number of events is random and the arrival times are strietly positive. For this reason, we will derive a new noising and denoising process (Sections 3.1 & 3.2), present learnable parametrization and appropriate loss to approximate the posterior (Section 3.3) and introduce a sampling procedure (Sections 3.4 & 3.5).
3.1 Forward process Noising
Let (0)) = (4,,..., (x) denote an i.i.d. sample from a TPP (data process) with T = specified by the unknown (conditional) intensity Àg- We define a forward noising process as a sequence of TPPs that start with the true intensity Ao and converge to standard HPP, i.e., Ào + A + + AN:
(1) = andn (1)) + On Ahrr
( Thin
Add
where 1 >> a1 >> 02 > .... aN > 0 and λнРР denotes the constant intensity of an HPP. Equation 2 corresponds to a superposition of (i) process An-1 thinned with probability 1- an (removing old points), and (ii) an HPP with intensity (1 an (HPH (adding new points).
Property (Stationary intensity). For any starting intensity Ag, the intensity function AN given by Equation 2 converges towards AHPP. That is, the noised TPP will be an HPP with AHrr-
Proof. In Appendix B.1 we show that, given Xo and Equation 2, An given by:
An (t) = ā. Ao() + (1 ā² )HHP.
(3)
where a = II,; ay. Since II," a, + 0 as + oo, thus (1) + AHPP-
If all on are close to 1, each consecutive realization will be close to the one before because do not remove a lot of original points, nor do we add many new points. And if we have enough steps, we will almost surely converge to the HPP. Both of these properties will be very useful in training generative model that iteratively reverses the forward noising process.
But how can we sample points from X, if we do not have access to Ao? Since we know the event sequence (()) comes from a true process which specified with λo we can sample from a thinned process or Ao by thinning the points (()) independently with probability 1 ūk,, This shows that even though we cannot access Ag. we can sample from X, by simply thinning (!)) and adding new points from an HPP.
3

---

Posterior intensity: An-1 11 (00,
B (1)
Σ bee
6,(1)
tin
A
F
C (1
Oa-1 In (1(()
B
I-
C
B
D (1)(
an-1)(1 aa)Ma
D
E (1)
On An & (1)
1 En
G
th 1)
(1) + (0 (1) (0)
Figure 2: (Left) Illustration of all possible disjoint sets that can reach in our forward process going from (0)) to ((m) through fn-1) (Right) Posterior intensity describing the distribution of g(n-1) t(0), g(m), where each subset B-E can be generated by sampling from the intensity functions.
To recap, given clean sequence (0)), we obtain progressively noisier samples t(n) by both removing original points from (0) and adding new points at random locations. After N steps, we reach a sample corresponding to an containnn no information about the original data.
3.2 Reverse process - Denoising
To sample realizations t ~~ An starting from ( ~ \HPP, we need to learn to reverse the Markoy chain of the forward process, i.e., An -.... λg. or equivalently ((N) + ( {()),, which is an
(()), Conditioned
on ( the reverse process at step n is given by the posterior ((t(n-1) inhomogeneous Poisson process for the chosen forward process (Section 3. Therefore, the posterior can be represented by a history-independent intensity function A (())) (())).
As the forward process is defined by adding and thinning event sequences, the points in the random sequence (0=1) can be decomposed into disjoint sets of points based on whether they are also in {()) or t(m). We distinguish the following cases: points in t ( that were kept from 0 to (B), points in t(n-1), that were kept from 0 to n - but thinned at the n-th step (C), added points in ((n- that thinned in the n-th step (D) and added points in & {(a-1) that are kept in the n-th step (E). Since the sets B-E are disjoint, the posterior intensity is superposition of the intensities of each subsets of g(n-1); 1 (4 (()) ((m) (1) (O))( + (L) + (I)
To derive the intensity functions for cases B-E, we additionally define the following helper sets: A and F the points ((m) \ ((n-1) that have been added at step n. The full case distinction and derived intensities are further illustrated in Figure In the following paragraphs, we derive the intensity functions for cases B-E:
the points (0)) (11 that were thinned until
Case The set {() can be formally defined (()) n (()) since (t(0) ((m) \ ((n-1) 0 almost surely. This is because adding points at any of the locations t E (()) n t(n) carries zero measure at every noising step. Hence, given ()) t(m) the intensity can written as a sum of Dirae measures: (t) Ex, (((0)))))))) 6r, (t). Similar how the forward process generated (B) by preserving some points from ((),, sampling from the reverse process preserves points from t(m).
Case C: Given t (AUC) = (()) {(m), +(C) can be found by thinning and consists of points that were kept by step n 1 and removed at step n. Using the thinning of Equations 2 and 3, know the probability of point from ())) being in ((C) and ((AUC) is cn-1 (1- (²,) and 1 On, respectively. Since we already know (B)) we can consider the probability of finding point in {CC), given (AUC)
which is equal to 1- Consequently, (C) is given as a thinned sum of Dirac measures over ((AUC) (cf., Figure 2).
Case D: The set {(P) contains all points t é (t(0) U ((n) that were added until step n - 1 and thinned at step n. Again using Equations 2 and 3, we can see that these points were added with intensity (1 -.-1)))) )) and then removed with probability Os, at the next step. Equivalently, we can write down the intensity that governs this process as (D)(1) (1 an- 1n-1)(11 On AH)AHP, i.e. sample points from an HPP and thin them generate a sample (D)

---

t {()) ng(
{(n)
Temporal Embedding
Layer ID CouvNet
Classifier MLP
BBE
(0)
Aggr:
(0) ((1)
Gaussian
Temporal Eimbedding
Intensity MI.P
NIL
77 -
Mixture
(w: jt: d)
Figure 3: Architecture of our model predicting to from bn-
Case E: The set (R) can be found by thinning (RFF) t(() ( and contains the points that were added by step
and then kept at step The processes that generated & and (()) are two independent HPPs with intensities ( = (1 - ān_ )an=) and X =1 (.,a)))Ha
(t) derived in a similar way to A (t). (t). Since (5) and ( are independent HPPs and we know (EUF), the number of points in {( follows a Binomial distribution with
respectively, where A ( (t)
probability p = x Xiey (see Appendix B.2 for details). That means we can sample ( given ((m)
X(R)
( and (0) by simply thinning (EUFF) with probability 1 - p and express the intensity as a thinned sum of Dirac functions (cf., Figure
For sequences of the training set, where ((0) is known, we can compute these intensities for all samples *(n) and reverse the forward process. However, (0) is unknown when sampling new sequences. Therefore, similarly to the denoising diffusion approaches [18], in the next section, we show how to approximate the posterior intensity, given only t(") Further, in Section 3.4, we demonstrate how the trained neural network can be leveraged to sample new sequences Ag
3.3 Parametrization and training
In the previous section we have derived the intensity A q(t(n=1f (()), {()) for the reverse process, i.c. the intensity of points at step
(0)), ((m)) of the posterior given (())
and (())_ Now we want to approximate this posterior using a model pa (n-11=1) {((] n) to learn to
sample points ( given only t(m), As are only missing information about (()) we will learn
to model X (B) (t) and ((AU)) to approximate {() x (()) (cf.,, Figure 3) for each n and then have
access to the full posterior intensity from Section 3.2 reverse the noising process.
Sequence embedding. To condition our model on (n)) and we propose the following embeddings. We use sinusoidal embedding [47] to embed the diffusion time Further, we encode each arrival time I g(m) and inter-event time T; = 44 4;-1, with lg = 0, to produce temporal event embedding E Idd by applying a sinusoidal embedding [47]. Then we leverage three-layered ID-Convolutional Neural Network (CNN) with circular padding, dilation, and residual connections to compute context embedding C; R.d. Compared attention and RNN-based encoders, the CNN is computationally more efficient and scales better with the length of event sequences while allowing us to capture long-range dependencies between the events. Finally, global sequence embedding is generated by mean aggregation of the context embeddings.
Posterior approximation. The posterior defining (D) is independent (0) and f(m) and can be sampled from directly. The set & corresponds to those points that were kept from ((0) until the n-th step. Since all of these points are also included in (()) we specify a classifier ge(e,, C, c,,)) with an MLP that predicts which points from (n)) belong to (0), As (() is known during training, this a standard classification setting. We use the binary cross entropy (BCE) loss LBCE to train go- Note that classifying (B) from (()) simultancously predicts ((") (()) t (FUF), Therefore
subsequently attain (F) by thinning (RF.F) as explained in Section 3.2.
To sample points (C) we have to specify the intensity function ( (1) that will be thinned to attain ()) (ef., Section 3.2). As (L) is mixture of Dirae functions we use an unnormalized mixture of H weighted and truncated Gaussian density functions f on [0, to parameterize the
5

---

inhomogeneous intensity:
2 x (t) K Σ wjf (t:
where Wi = Softplus ([, 2]), Mj = Sigmoid(MMP, ([n, E)) and dj exp(-IMi.P- e))l) are parameterized by MLPs with two layers, hidden dimension of d and a ReLU activation. Note that the Gaussian function is the standard approximation of the Dirac delta function and can, in the limit - 0, perfectly approximate it. We include K, the number of points in (("), in the intensity to more directly model the number of events. Then X (AUC)) (l) is trained match samples t ((AUC) X(AUC) by minimizing the negative log-likelihood (NLL):
log A (L))))))) + ['"
LLLL log logg((( --
Σ (EKKAUD)
Ag ((AU))(I)
(5)
Thanks to the chosen parametrization, the integral term in LNLL can be efficiently computed in any deep-learning framework using the "erf" function, without relying on Monte Carlo approximation. We present an overview of our model architecture to predict ( from g(n) in Figure 3.
CNLL + LBCB. During training, we do not have to evaluate the true posterior or sample events from any of the posterior distributions. Instead, we can simply sample n and subsequently (()) and minimize L for g(n), Interestingly, in Appendix A, we show that L is equivalent to the Kullback-Leibler (KL) divergence between the approximate posterior (n ((m), n) and the true posterior q(t(n-1) ( ((m).. Ultimately, this shows that optimizing the evidence lower bound (ELBO) of the proposed model boils down to simply learning binary classification and fitting an inhomogeneous intensity.
Training objective. The full model trained to minimize
3.4 Sampling
To sample an event sequence from our model, we start by sampling ((N) from HPP with AhPP. Subsequently, for each n E [N,....1
Algorithm 1: Sampling
{(=N) AнPP;
(()) pre-
for E {N...., 1 do sample (BB)
dicted by classifying {(B) and sampling ((AUC) from X ((AUC)(t)
Σ go(ts e, C. m)o (t);
(t). Note that the Poisson distribution with intensity (T) = for X (1) dt param- eterizes the number of points in AUC. Therefore, {(AUC) can be sampled by first sampling the number of events and then sampling the event times from the nor- malized intensity X (AUC) (AAU))(T)) Given our predicted g0o we can sample f(n-1) from the posterior intensity defined in Section 3.2. By repeating this process, produce sequence of (mn))), Finally, we obtain a denoised sample (0) by pre- dicting it from f(1), We provide an overview of the sampling procedure as pseudo-code in Algorithm 1.
(
sample ((C) 1 1-a,
an-i-an
AB (AUC) (t n));
sample ((D) (1 an- ()
an AHPP:
sample iLB)
Σ
Or au dz, (I);
I an
LEBD) ((())
{(--1) + ((B) ( (((D) UE(E);
end sample ((AUC)
A
(11t,,1);; (t)+t)
sample ((B)
go(t: e; C, 1)8, (f): (.(())
Σ
t + ((B) UOPURR ((ALO);
return
3.5 Conditional sampling
The above-described process defines unconditional generative model for event sequences on an interval [o, T). For many (multi-step) forecasting applications, such as earthquake forecasting [10], we need to condition our samples previous event sequences and turn our model into conditional one that can generate future event sequences [H, H T] given the past observed events [0, H). To condition our generative model on a history, we apply simple GRU encoder to encode the history into a d-dimensional history embedding h, which subsequently conditions the classifier and intensity model by being added to the diffusion time embedding.

---

4 Related work
Autoregressive neural TPPs. Most neural TPPs model the intensity or density of each event con- ditional on history and consequently consist of two parts: history encoder and an intensity/density decoder, As history encoders, RNNs [12, 41] and attention- based set encoders [50, 52] have been proposed. Attention-basel encoders are postulated to better model long-range dependencies in the event sequences, but at the cost of more complex encoder structure [43]. To decode the intensity X4[H), density p(([H) or the cumulative hazard function from the history, uni-modal distributions [12], mixture-disttriuttonn [41]. a mixture of kernels [36, 44, 51], neural networks [37] and Gaussian diffusion [29] have been proposed. Another branch of neural TPPs models the event times conditional on a latent variable that follows a continuous- time evolution [S, 13, 16, 21], where, c.g., Hasan et al. [16] relate inter-event times of TPP the excursion of stochastic process. In general, most neural TPPs are trained by maximizing the log-liken oog-likelhhood, but other training approaches have been proposed [26, 29, 49]. We want highlight the difference of our model to two related works. TriTPP [42] learns a deterministic mapping between a latent HPP and a TPP using normalizing flows, which allows for parallel sampling. However, it models the conditional hazard function, which forces conditional dependency of later arrival times and can still produce error accumulation. Lin et al. [29] proposed autoregressive TPP model leveraging Gaussian diffusion approximate the conditional density. Besides being autoregressive, the model does not directly model the number of points in the TPP but instead is trained to maximize the ELBO of the next inter-event time.
Non-autoree neural TPPs. An alternative to the conditional (autoregressive) modeling of TPPs is to apply a latent variable model that learns to relate entire point processes to latent variables. The class of Cox processes [7, 11, 19] models point processes through a hierarchy of latent processes under the assumption that higher-level latent variables trigger lower-level realizations. ADD-THIN can be considered to be non-autoregressive latent variable model.
Denoising diffusion models. Recently, denoising diffusion models on continuous state spaces [18, 45] established the new state-of-the-an for many image applications [15, 18, 25]. Subsequently, diffusion models for other application domains such as point clouds [31, 32]. physical simulations
[23, 28, 30] and time-series [1, 3, 24, 46] emerged. While the majority of denoising diffusion models are based on Gaussian transition kernels in continuous state spaces proposed in [18, 45], a variety of diffusion models for discrete state spaces such as graphs [48], text [2, 27] and images [2, 6] have been presented. Here, we highlight the similarity of our work to the concurrent work of Chen and Zhou [6], who derived a diffusion process that models pixel values them to noise images. In contrast to the related work on continuous and discrete state space diffusion models, ADD-THIN constitutes a novel diffusion model defined on a state space that captures both the discrete and continuous components of point processes.
count distribution and thins
5 Experiments
We evaluate the proposed model in two settings: density estimation and forecasting. In density estimation, the goal is to learn an unconditional model for event sequences. As for forecasting, the objective is to accurately predict the entire future event sequence given the observed past events.
Data. ADD-THIN is evaluated on 7 real-world datasets proposed by Shchur al. [42] and 6 synthethic datasets from Omi et al. [37]. The synthetic datasets consist of Hawkes I and Hawkes2 [17], a self-correcting (SC) [20], inhomogeneous Poisson process (IPP) and a stationary and non- stationary renewal process (MRP, RP) [39, 11]. For the real-world datasets, we consider PUBG, Taxi. Twitter, and restaurant check-ins in Yelp1 and Yclp2. We split each dataset into train, validation, and test set containing 60%, 20%, and 20% of the event sequences. respectively. Further dataset details and statistics are reported in Appendix C.
Baselines. We apply the RNN- intensity-free TPP model from Shchur et al. [41]. Similar to Sharma et al. [40], we further combine the intensity-free model with an attention-basse encoder from Zuo et al. [[2]] as a Transformer baseline. Additionally, we compare our model to an autoregressive TPP model with continuous state Gaussian-diffusion [18] from Lin al. [29], which we abbreviate
as GD. Lastly, TriTPP [42] is used as a model that provides parallel but autoregressive sampling.
7

---

Table 1: MMD (4) between the TPP distribution of sampled sequences and hold-out test set (bold
best, underline second best).
The results with standard deviation are reported in Appendix E.
Hawkes
Hawkes2
SC
IPP
RP
MRP
PUBG
Reddit-C
Reddit-S
Taxi
Twitter
Yelp!
Yelp2
RNN
0.02
0.01
0.08
0.05
0.01
0.03
0.04
0.01
0.02
0.04
0.03
0.07
0.03
Transformer
0.03
0.04
0.19
0.10
0.02
0.19
0.06
0.05
0.09
0.09
0.08
0.12
0.14
0.06
0.06
0.13
0.08
0.05
0.14
0.11
0.03
0.03
0.10
0.15
0.12
0.10
TТTPP 0.03
0.04
0.23
0.04
0.02
0.05
0.06
0.09
0.12
0.07
0.04
0.06
0.06
ADD-THIN
0.02
0.02
0.19
0.03
0.02
0.10
0.03
0.01
0.02
0.04
0.08
0.04
Table 2: Wasserstein distance (1) between the distribution of the number of events of sampled
sequences and hold-out test set (bold best, underline second best). The results with standard deviation
reported in Appendix E.
Hawkes!
Huwkex2
SC
IPP
RP
MRP
PUBG
Redkit-C
Reddit-S
Taxi
Twitter
Yelp!
Yelp2
RNN
0.03
0.01
0.00
0.02
0.02
0.01
0.02
0.01
0.05
0.02
0.01
0.04
0.02
Transformer
0.06
0.04
0.06
0.07
0.04
0.11
0.04
0.08
0.11
0.13
0.05
0.11
0.21
GD
0.16
0.13
0.50
0.42
0.28
0.50
0.54
0.02
0.16
0.33
0.07
0.26
0.25
ThTPPP
0.03
0.03
0.01
0.01
0.02
0.03
0.03
0.09
0.09
0.01
0.03
0.04
ADD-TUIN
0.04
0.02
0.08
0.01
0.02
0.04
0.02
0.03
0.04
0.03
0.01
0.04
0.02
Training and model selection. We train each model by its proposed training loss using Adam [22]. For model, we set the number of diffusion steps to N 100, apply the cosine beta- schedule proposed in Glide [34], and set ÅнPP = for the noising process. We apply early stopping, hyperparameter tuning, and model selection the validation set for each model. Further hyperparameter and training details are reported in Appendix D.
5.1 Sampling -Denhin
Density estimation
A good TPP model should be flexible enough to fit event sequences from various processes. We evaluate the generative quality of the TPP models 13 synthetic and real-world datasets by drawing 4000 TPP sequences from each model and computing distance metrics between the samples and event sequences from a hold-out test set. In short, the goal of this experiment is show that our proposed model is a flexible TPP model that can generate event sequences that are 'close' to the samples from the data- geeiraaeeeeranngggg distribution.
Metrics. general, diffusion models cannot evaluate the exact likelihood. Instead, we evaluate the quality of samples by comparing the distributions of samples from each model and the test set with the maximum mean discrepancy measure (MMD) [14] as proposed by Shchur al. [42]. Furthermore, we compute the Wasserstein distance [38] between the distribution of sequence lengths of the sampled sequences and the test set. We report the results on the test set averaged over five runs with different seeds.
Results. Table presents the MMD results for all models and datasets. Among them, the RNN baseline demonstrates a strong performance across datasets and outperforms both Transformer and GD. Notably, ADD-THE THIN exhibits competitive results with the autoregressive baseline, surpassing or matching (+:0.01) on 11/13 datasets. Additionally, ADD-THI THIN consistently outperforms the Transformer and GD model on all datasets except SC and RP. Lastly, TriTPP performs well on most datasets but is outperformed or matched by our model on all but two datasets.
Table 2 shows the result for comparing the count distributions. Overall, the Wasserstein distance results align closely with the MMD results. However, the GD model is an exception, displaying considerably worse performance when focusing the count distribution. This outcome is expected since the training of the GD model only indirectly models the number of events by maximizing the ELBO of each diffusion step approximate the conditional density of the next event and not the likelihood of whole event sequences. Again, ADD- THIN shows very strong performance, further emphasizing its expressiveness.
In summary, these results demonstrate the flexibility of our model, which can effectively capture various complex TPP distributions and matches the state-of-the-att performance in density estimation.

---

Table Wasserstein distance between forecasted event sequence and ground truth reported for 50 random forecast windows on the test set (lower is better). The results with standard deviation are reported in Appendix E.
PUBG
Reddit-C
Reddit-S
Taxi
Twitter
Yelp1
Yelp2
Average Seq. Length
76.5
295.7
1129.0
98.4
14.9
30.5
55.2
RNN
6.15
35.22
39.23
4.14
2.04
1.28
2.21
Transformer
2.45
38.77
27.52
3.12
2.09
1.29
2.64
GD
5.44
44.72
64.25
4.32
2.16
1.52
4.25
ADD-THIN (Ours)
2.03
17.18
21.32
2.42
1.48
1.00
1.54
Table 4: Count MAPE x 100% between forecasted event sequences and ground truth reported for 50 random forecast windows on the test set (lower better). The results with standard deviation are reported in Appendix E.
PUBG
Reddit-C
Reddit-S
Taxi
Twitter
Yelp1
Yelp2
Average Seq. Length
76.5
295.7
1T29.0
98.4
14.9
30.5
55.2
RNN
1.72
5.47
0.68
0.54
0.95
0.59
0.72
Transformer
0.65
7.38
0.55
0.46
1.18
0.63
0.99
GD
1.66
10.49
1.33
0.71
1.43
0.78
1.65
ADD-THIN (Ours)
0.45
1.07
0.38
0.37
0.69
0.45
0.50
5.2 Conditional sampling -Forecasting
Forecasting event sequences from history is an important real-world application of TPP models. In this experiment, we evaluate the forecasting capability of our model on all real-world datasets. We evaluate each model's performance in forecasting the events in a forecasting window AT, by randomly drawing a starting point T, [AT, T - AT]. Then, the events in [0, T.] are considered forecasting time horizon.
history, and [T T. + AT]
In the experiment, we randomly sample 50 forecasting windows for each sequence from the test set, compute history embedding with each model's encoder, and then conditionally sample the forecast. from each model. Note that TriTPP does not allow for conditional sampling and is therefore not part of the forecasting experiment.
Metrics. To evaluate the forecasting experiment, will not compare distributions of TPPs but rather TPP instances. We measure the distance between two event sequences, i.c., forecast and ground truth data, by computing the distance between the count measures with the Wasserstein distance between two TPPs, as introduced by Xiao et al. [49]. Additionally, we report the mean absolute relative error (MAPE) between the predicted sequence length and ground truth sequence length the forecasting horizon. We report the results on the test set averaged over five runs with different seeds.
Results. Table presents the average Wasserstein distance between the predicted and ground truth forecast sequences. The results unequivocally demonstrate the superior performance of model by forecasting entire event sequences, surpassing all autoregressive baselines on all datasets. Notably, the disparity between ADD-THIN and the baselines is more pronounced for datasets with a higher number of events per sequence, indicating the accumulation of prediction errors the autoregressive models. Further, the transformer baseline achieves better forecasting results than the RNN baseline for some datasets with more events. This suggests that long-range attention can improve autoregressive forecasting and mitigate some error accumulation.
Event count (())
observed
40
20
forecast
0
2
Time t
Figure 4: 5%, 25%, 50%, 75%, and 95% quantile of forecasts generated by ADD-THIN for a Taxi event sequence (blue: history, black ground truth future).
Table reports the MAPE between the forecasted and ground truth sequence length. The MAPE results align consistently with the Wasserstein distance across all models and datasets. Figure

---

depicts the quantiles for 1000 forecasts generated by ADD-THI for one Taxi event sequence and highlights the predictive capacity of our model. Overall, our model outperforms TPP models in forecasting real-world event sequences.
6
Discussion
ADD-THIN vs. autoregressive TPP models. On a conceptual level, ADD-THIN presents a different trade-off compared to other TPP models: Instead of being autoregressive in event time, model gradually refines the entire event sequence in parallel at every diffusion step to produce a sample from the learned data distribution. Thereby, have found that our model is better suited for forecasting and modeling very long event sequences than autoregressive TPP models. Furthermore, the iterative refinement of the entire sequence allows us leverage simple and shared layers accurately model the long-range interaction between events and results in nearly constant sampling times across different sequence lengths (cf., Appendix E.3).
Limitations and future work. With ADD-THIN, we have derived a novel Luffusiin-inssiiee model for TPPs. Thereby, we focused modeling the arrival times of the events and did not model continuous and discrete marks, However, we see this as an exciting extension to our framework, which might incorporate Gaussian diffusion [18] for continuous marks and discrete diffusion [2] for discrete marks. Further, while generative diffusion is known to produce high-quality samples, it also can be expensive. Besides tuning the number of diffusion steps, future work could focus on alternative and faster sampling routines [35]. Ultimately, we hope that by having connected diffusion models with TPPs, we have opened a new direction modeling TPPs and broadened the field of diffusion- based models. Here, it would be especially interesting for us to see whether our framework could benefit other application domains in machine learning that involve sets of varying sizes, such as graph generation (molecules), point clouds, and spatial point processes.
7 Conclusion
By introducing ADD-THIN, we have connected the fields of diffuuson models and TPPs and derived a novel model that naturally handles the discrete and continuous nature of point processes. Our model permits parallel and closed-form sampling of entire event sequences, overcoming common limitations of autoregressive TPP models. In experimental evaluation, demonstrated the flexibility of ADD-THIN which can effectively capture complex TPP distributions and matches the state-of-the performance in density estimation. Additionally, in long-term forecasting task real-world data, our model distinctly outperforms the state-of-the-ard TPP models by predicting entire forecasting windows non-autoregressivel.
Broader impact
We see the proposed model as general framework to model continuous event data. As such, our method can be applied to many fields, where common application domains include traffic, social networks, and electronic health records. We do not find any use cases mentioned above raise ethical concerns; however, it essential to exercise caution when dealing with sensitive personal data.
Acknowledgements
This research was supported by the German Research Foundation, grant GU 1409/3-1.
10

---

References
[1] M. L. Alcaraz and N. Strodthoff. Diffusion-basse based time series imputation and forecasting with structured state space models. arXiv preprint arXiv:2208.09330 2022.
[2] J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 17981-17993, 2021.
[3] M. Biloš, K. Rasul, A. Schneider, Y. Nevmyvaka, and S. Günnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In International Conference on Machine Learning (ICML), 2023.
[4] T. Bosser and S. Taieb, On the predictive accuracy of neural temporal point process models for continuous- time event data. Transactions on Machine Learning Research, 2023.
[5] R. T. Chen, B. Amos, and M. Nickel. Neural spatio-temporal point processes. arXiv preprint arXiv:2011. 1.45533, 2020.
[6] Chen and M. Zhou. Learning to jump: Thinning and thickening latent counts for generative modeling. arXiv preprint arXiv:2305. 18375, 2023.
[7] D. R. Cox. Some statistical methods connected with series of events. Journal of the Royal Statistical Society: Series B (Methodological), 17(2):129-157) 1955.
[8] D. Daley and D. Vere-Jones. An Introduction to the Theory of Point Processes: Volume I: Elementary Theory and Methods. Probability and Its Applications. Springer New York, 2006.
[9] D. Daley and D. Vere-Jones. An introduction the theory of point processes: volume II: general theory and structure. Springer Science & Business Media, 2007.
[10] K. Cousiee-C O. Shchur, E. E. Brodsky, and S. Günnemann. Using deep learning for flexible and scalable earthquake forecasting. Geophysical Research Letters, 50(17) 2023.
[11] C. DR and COLL. The statistical analysis of dependencies in point processes. Stochastic Point Processes. Wiley: New York, pages 55-66, 1972.
[12] N. Du, H. Dai, R. Trivedi, U. Upadhyay, M. Gomez- and L. Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1555-1564 2016.
[13] J. Enguchard, D. Busbridge, A. Bozson, C. Woodcock, and N. Hammerla. Neural temporal point processes for modelling electronic health records. In Machine Learning for Health, pages 85-113. PMLR, 2020.
[14] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[15] X. Han, H. Zheng, and M. Zhou. Card: Classification and regression diffusion models, arXiv preprint arXiv:2206.07277 2022.
[16] A. Hasan, Y. Chen, Y. Ng, M. Abdelghani, A. Schneider, and V. Tarokh. Inference and sampling of point processes from diffusion excursions. In The 39th Conference on Uncertainty in Artificial Intelligence, 2023.
[17] A. G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90, 1971.
[18] Ho, Jain, and Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems (NeurIPS), 2020.
[19] C. Hong and C. Shelton. Deep neyman-scott processes. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics, volume 151, pages 3627-3646. PMLR, 2022.

---

[20] V. Isham and M. Westcott. A self-correcting point process. Stochastic processes and their applications, 8(3):335-347, 1979.
[21] J. Jia and A. R. Benson, Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
[22] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXivv1412.6980, 2014.
[23] G. Kohl, L.-W. Chen, and N. Thuerey. Turbulent flow simulation using autoregressive condi- tional diffusion models, 2023.
[24] M. Kollovich, A. F. Ansari, M. Dohlke-Schneider J. Zschiegner, H. Wang, and Y. Wang. Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting, arXiv preprint arXiv: 1494, 2023.
[25] H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li. and Y. Chen. Srdiff:: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-59, 2022.
[26] S. Li. S. Xiao, S. Zhu, N. Du, Y. Xie, and L. Song. Learning temporal point processes via reinforcement learning. Advances in neural information processing systems, 31, 2018.
[27] X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto. Diffusion-Im improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328- 4343, 2022.
[28] M. Lienen, D. Lüdke, J. Hansen-Palmun and S. Günnemann. From zero to turbulence: Genera- tive modeling for 3d flow simulation, 2023.
[29] H. Lin, L. Wu, G. Zhao, L. Pai, and S. Z. Li. Exploring generative neural temporal point process. Transactions Machine Learning Research, 2022.
[30] Lippe, B. Veeling, P. Perdikaris, R. E. Turner, and Brandstetter. Pde-refiner: Achieving accurate long rollouts with neural pde solvers. arXiv preprint arXiv:2308.05733 2023,
[31] S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[32] Z. Lyu, Z. Kong, X. Xu, L. Pan, and Lin. conditional point diffussion-refinemett paradigm for 3d point cloud completion. arXiv preprint arXiv:2112.03530 2021.
[33] H. Mei and J. M. Eisner. The neural hawkes process: A neurally molf-modulaiing multivariate point process. In Neural Information Processing Systems (NeurIPS), 2017.
[34] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.100 2021.
[35] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[36] M. Okawa, T. Iwata, T. Kurashima, Y. Tanaka, H. Toda, and N. Ueda. Deep mixture point processes: Spatio-temporal event prediction with rich contextual information. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 373-383, 2019.
[37] Omi, K. Aihara, et al. Fully neural network based model for general temporal point processes. Advances in neural information processing systems, 32, 2019.
[38] A. Ramdas, N. García Trillos, and M. Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47, 2017.
[39] B. Sevast' yanov. Renewal theory. Journal of Soviet Mathematics, 4(3):281-302, 1975.
12

---

[40] K. Sharma, Y. Zhang, E. Ferrara, and Y. Liu. Identifying coordinated accounts on social media through hidden influence and group behaviours. In Proceedings of the 27th ACM SICKDD Conference Knowledge Discovery & Data Mining, pages 1441-1451, 2021.
[41] O. Shchur, M. Biloš, and S. Günnemann. Intensity-free learning of temporal point processes. In International Conference on Learning Representations (ICLR), 2020.
[42] O. Shchur, N. Gao, M. Biloš, and S. Günnemann. Fast and flexible temporal point processes with triangular maps. In Advances Neural Information Processing Systems (NeurlPS), 2020.
[43] O. Shchur, A. C. Türkmen, T. Januschowski, and S. Günnemann. Neural temporal point processes: A review. arXiv preprint arXiv:2104.03522 2021.
[44] A. Soen, A. Mathews, D. Grixti-Cheng, and L. Xie. Unipoint: Universally approximating point processes intensities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9685-9694, 2021.
[45] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics In International Conference on Machine Learning (ICML), pages 2256-22655 2015.
[46] Y. Tashiro, J. Song, Y. Song, and S. Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34: 24804-24816, 2021.
[47] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] C. Vignac, I. Krawezuk, A. Siraudin, B. Wang, V. Cevher, and Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv: 14734, 2022.
[49] S. Xiao, M. Farajtabar, X. Ye, J. Yan, L. Song, and H. Zha. Wasserstein learning of deep generative point process models. Advances in neural information processing systems, 30, 2017.
[50] Q Zhang, A. Lipani, O. Kirnap, and E. Yilmaz. Self-attentive hawkes process. In International conference machine learning, pages 11 11183-11199 193. PMLR, M, 2020.
[51] W. Zhang, Panum, S. Jha, P. Chalasani, and D. Page. Cause: Learning granger causality from event sequences using attribution methods. In International Conference on Machine Learning, pages 11235-11244 PMLR, 2020.
[52] S. Zuo, H. Jiang, Z. Li, T. Zhao, and H. Zha. Transformer hawkes process. arXiv preprint 20Xiv:2002.09291, 2020.
13

---

A On the relationship between the applied loss and the ELBO
In the following section, we investigate the relationship between applied loss function and the ELBO to the unknown data distribution derived for diffusion models. The ELBO of diffusion models [18] is given follows:
LELBO - Esy DKL. (o(e(N) (10)) ll (ppe(l)))
logpo (t 1101)
Ex
Lo
(6)
N
DEL (g(t(n-1) (10),((n) po (t(aa11) ((11
Additionally, the Janossi density [8] of an event sequence on [0, T allows to represent each element of the ELBO in terms of our derived inhomogeneous (approximate) posterior intensities (Section 3.2 and Figure 2) as follows:
p(t) exp (-("xc0 A(t) A(tt)..
t.et CN: LN is constant as the intensity AHPP defining q(((N) ((0) and p(t(N) has no learnable parameters.
Co: We directly train our model to optimize this likelihood term described in Section 3.3.
Cu: The KI. divergence between two densities is defined as:
DKI(a || po) = E., [og((g(((=1)) (0),, (()))) - log(po (o {())))
(8)
where only the right-hant side relies on 0, letting us minimize the KL divergence by maximizing expectation over the likeliidl log(pe (+("-1) (())).
To add some additional context to the KL divergence in Cn and similar to the derivation of the posterior in Section 3.2, we will further distinguish three cases:
1. Case B & E: q(((m=1)
(()) {(a)) and (=(t(=-1) (()) defined by Bernuulli distribu-
tions over each element of (()), By definition the cross-entropy (q. p) of the distribution p relative to 9 is given by (g. p) (q) + DxL (g p), where H is the entropy and DEL the KI. divergence. We can see that minimizing the (binary) cross-entropy is equivalent minimizing the KL divergence, the entropy H(q) is constant for the data distribution.
2. Case C: Minimizing this KL. divergence by maximizing the E. [oo (t(((n)))))) ((()))]
is proposed in Section 3.2 learn Xg (AUC) (t). Note that q(((n-1) (0)), (()) sampled
from by independently thinning (0)) ((). Consequently, by minimizing the NLL of our intensity x (AUC)(()
(t) with regards to (0) t(m), we optimize the expectation in closed form.
3. Case D: Our parametrization uses the same intensity function for q(t(n-1) (()) {(n)) and
(n-1 t(m)), which does not rely on any learned parameters.
B Derivations
B.1 Direct forward sampling
Proof. We first repeat Equation 2:
(t) a 1nt) +11++111-11 Anprr
(2)
Then for the first step we can write:
(t a1\o(t) (1 -aa-))dpp
n=1
Il
aj
do
1 1-
aj
AHPP.
==1
14

---

Assuming Equation 3 holds for step 1:
1
n-1 II 02
(1)) -
II ==1
Ojj
Ao(1)
+
1
AHPP,
j=1
we can write for step n:
(t)
andn-1(t) (1 and)APP
n-1
On
II Ojj
do(t)= +
II 0i
AHPP + On AHPP
j=l
n
II aj
Ao
+
an
II
aj
АHPP + (1 AHPP
j=1
πI 0j Ao(() +
1
II
(n) + (1 on) AHPP:
which completes the proof by induction.
B.2 Conditional distribution of Poisson variables
Proposition. Given two independent random variables X1 Poisson(A1), X2 ~ Poisson(\2), X1 X1 + X2 = k is Binomial distributed, ie.. X1 X1 + X2 = k ~ Binomial(11; k, x
As
Proof. The Poisson distributed random variables X1 and X2 have the following joint probability mass function:
= x1, x - (A 17 A2 Tz
xn1!
x2!!
which further defines the joint probability mass function of P(X1 x1, X2 22, Y k) if t T2 = k. Additionally, it is well know that Y = X1 + X2 is Poisson random variable with
intensity A + X2 and therefore P(Y k) = (A1+ (Ag +* Then P(X1 3 Y k)) given by the following:
P(X1 = 11, X2 = L2, Y = k)
P(11 = I1 YY=
if + k.
(10)
P(Y =k)
1
A2 A32
ZTT oz.
if 01 + 12 = k;
(11)
(A)+ (Ar +2a)t
k!
k!
X 2
if r1 + 32 = k
(12)
zilzz! (A1 + X2)
k!
X
\ kー=z
(A1 6-11 xf + +22 = k,
(13)
211(k 21)! (A1
+ )2)
(A-211
k!
A1
(1-
if + x2 = k
211(k 21
(A1 + X2)
(A1 + A2)
(14)
where we have leveraged x2
k- - 1 and
Ag - [A++AaJ
d1 As we have shown P(X, σ
[A1+A2j
Y = k)) follows the Binomial distribution with p = [Ai+ta]
A
15

---

Table 5: Statistics for the synthetic datasets.
# Sequences
T
Average sequence length
lawkesl
1000
100
95.4
1.01 + 2.38
Hawkes2
1000
100
97.2
0.98 2.56
SC
1000
100
100.2
0.99 + 0.71
IPP
1000
100
100.3
0.99 - 2.22
RP
1000
100
109.2
0.83 2.76
MRP
1000
100
98.0
0.98 = 1.83
Table 6: Statistics for the real-world datasets.
# Sequences
T
Unit of time
Average sequence length
AT
PUBG
3001
30
minutes
76.5
0.41 + 0.56
5
Reddit-C
1356
24
hours
295.7
0.07 107
4
Reddit-S
1094
24
hours
1129.0
0.02 0.03
4
Taxi
182
24
hours
98.4
0.44
4
Twitter
2019
24
hours
14.9
1.26 2.80
4
Yelp1
319
24
hours
30.5
0.77 1.10
4
Yelp2
319
24
hours
55.2
0.43 0.96
C Datasets
Synthetic datasets. The six synthethic dataset were sampled by Shchur et al. [42] following the procedure in Section 4.1 of Omi et al. [37] and consist of 1000 sequences on the interval [0, 100).
Real-world datasets. The seven real-world datasets were proposed by Shehur al. [42] and consist of PUBG, Reddit-Comments, Saddit-Submisssons. Taxi, Twitter, Yelp1, and Yelp2. The event sequences of PUBG represent the death of players in game of Player Unknown's Battleground (PUBG). The event sequences of Reddit-Comments represent the comments on the askscience subreddit within 24 hours after opening the discussion in the period from 01.01.2018 until 31.12.2019, The event sequences of Reddit-Submissions represent the discussion submissions on the politics subreddit within day in the period from 01.01.2011 until 31. 31.12.20 The event sequences of Taxi represent taxi pick-ups in the south of Manhattan, New York. The event sequences of Twitter represent tweets by user 25073877. The event sequences of Yelpl represent check-ins for the McCarran International Airport recorded for 27 users in 2018. The event sequences of Yelp2 represent check-ins for all businesses in the city of Mississauga recorded for 27 users in 2018.
We report summary statistics on the datasets in Table and 6. Lately, the validity of some of the widely used real-world benchmark datasets was criticized [4]. In one-step-ahead prediction tasks with teacher forcing, very simple architectures achieved similar results to some the more advanced ones. However, this seems be more of problem of the task than the datasets. In our work, consider different tasks (density estimation and long-term forecasting) and metrics and have found significant empirical differences between the baselines these datasets.
D Experimental set-up
All models but the transformer baseline were trained Intel Xeon E5-2630 v4 @ 2.20 GHz CPU with 256GB RAM and an NVIDIA GeForce GTX 1080 Ti. Given its RAM requirement, the transformer baseline had be trained with batch size 32 NVIDIA A 100-PCIE-40GB for the Reddit-C and Reddit-S datasets.
Hyperparameter tuning. has been applied to all models. The hyperparameter tuning was done on the MMD loss between 1000 samples from the model and the validation set. We use a hidden dimension of 32 for models. Further, we have tuned the learning rate in (0.01, 0.001) for all models, the number of mixture components in {8, 16) for ADD-THIN, RNN and Transformer, the number of knots in (10, 20} for TriTPP and the number of attention layers in [2,3) for the transformer baseline. The values of all other baseline hyperparameters were set to the recommended
16

---

values given by the authors. Further, the GD baseline has been trained with batch size of 16. as recommended by the authors. For the forecasting task, we apply the optimal hyperparameters from the density estimation experiment.
Early-stopping Each model has been trained for up to 5000 epochs with early stopping on the MMD metric on the validation set for the density estimation task and on the Wasserstein distance metric on validation set for the forecasting task.
E
Additional results
E.1
Density estimation results with standard deviations
Table 7: Synthetic data: MMD (4)
between the TPP distribution of sampled sequences and hold-out
test set.
Hawkes1
Hawkes2
SC
IPP
RP
MRP
RNN
0.02++0.003
0.01=0.002
0.08++0.053
0.05++00009
0.01+00000
0.03++0.005
Transformer
0.03-10.011
0.0410.017
0.191100006
0.10-10.034
0.02:10.007
0.19:10.048
GD
0.06 004
0.06=0.002
0.13+00004
0.08=0.002
0.05=00002
0.14 008
TriTPP
0.03:10.002
0.04=10001
0.23:10.003
0.04110003
0.02:10002
0.05+10.004
ADD-THIE THIN (Ours)
0.02:10.004
0.02=0.002
0.1910.013
0.03=10.006
0.02+00001
0.10::0.030
Table 8: Real-world data: MMD (4) between the TPP distribution of sampled sequences and
hold-out test set.
PUBG
Reddit-C
Reddit-S
Taxi
Twitter
Yelp1
Yelp2
RNN
0.04-0.005
0.01++0.002
0.02+0.003
0.04+00.001
0.03+0.003
0.07++0.05
0.03++0.011
Transformer
0.06=0.014
0.05:10.025
0.0940.06
0.09:0.0114
0.08:10.22
0.12::0.026
0.14::0048
GD
0.11=0.023
0.03++0.001
0.03=0.001
0.1+0.002
0.15#0.011
0.12+0.01
0.1:00.001
TriTPP
0.06-0.001
0.09++0.002
0.12+0.003
0.07+00007
0.04+0.002
0.06++0.005
0.06+00000
ADD-THIN (Ours)
0.03=00015
0.01:10.005
0.02:10.001
0.04:10.006
0.041:0.06
0.08
0.04
Table Synthetic data: Wasserstein distance (4) between the distribution of the number of events of sampled sequences and hold-out test set.
Hawkesl
Hawkes2
SC
IPP
RP
MRP
RNN
0.03:4.0007
0.01=0.002
0.00:E0.003
0.02:40.006
0.02:10.002
0.01:00.004
Transformer
0.06:10.017
0.04-1001
0.06:10.008
0.07+10.33
0.04:10.005
0.11+:0.08
GD
0.16::0.016
0.13=0.012
0.550.025
0.42:10.009
0.28:10033
0,5+0.035
TrTTPP
0.03++0003
0.03=0.001
0.01+00.0
0.01+00001
0.02+00003
0.03++0.01
ADD- ADD-THIN (Ours)
0.04:10.009
0.02-0.006
0.0810.018
0.01-10.003
0.0210.001
0.04
Table 10: Real-world data: Wasserstein distance (4) between the distribution of the number of
events of sampled sequences and hold-out test set.
PUBG
Reddit-C
Reddit-S
Taxi
Twitter
Yelp!
Yelp2
RNN
0.02=0.004
0.01:20.004
0.05=0.013
0.02::0.002
0.01=:0.001
0.04220004
0.02:00.02
Transformer
0.04=0.013
0.08:10.088
0.11=00032
0.13+:0.73
0.05=0.021
0.11+0003
0.21:00077
GD
0.54-0.054
0.02++0.004
0.16+0.013
0.33++0.07
0.07+0.062
0.26++0.012
0.25+00007
TATPP
0.03=0.003
0.09-10.011
0.09=0.001
0.04+00001
0.01=0.001
0.03:10.006
0.04:10.022
ADD-THIN (Ours)
0.02=0.009
0.03:+0.007
0.04=0.002
0.03:40.007
0.01+0.004
0.04::0.066
0.02++0.006
17

---

E.2
Forecasting results with standard deviations
Table 11: Wasserstein distance (4) between forecasted event sequence and ground truth reported for
50 random forecast windows on the test set.
PUBG
Reddit-C
Reddit-S
Taxi
Twitter
Yelp1
Yelp2
Average Seq. Length
76.5
295.7
1129.0
98.4
14.9
30.5
55.2
RNN
6.13=2.53
35.22+44.22
39.23=2.06
4.14::0.5
2.04=0.08
1.20+0.03
2.21:0.06
Transformer
2.45=0.21
38.77-10.68
27.52-15.24
3.12=0.1
2.09=0.07
1.2990.11
2.64110.4
5.44002
44.72::1.77
64.25:4.45
4.32-0.3
2.16=0.23
1.2440.5
4.25-0.46
ADD-THIN (Ours)
2.03=0.01
17.18+1188
21.32=0.42
2.42+0.03
1.48=0.03
10000.02
1.54+0.04
Table 12: Count MAPE x 1000 (4) between forecasted event sequences and ground truth reported
for 50 random forecast windows on the test set.
PUBG
Reddit-C
Reddit-S
Taxi
Twitter
Yelp1
Yelp2
Average Seq. Length
76.5
295.7
1129.0
98.4
14.9
30.3
55.2
RNN
1.72:10.5
5.47:H0.22
0.68 0.07
0.54
0.95
0.59
0.72-0.06
Transformer
0.65
7.38.12.1
0.55
0.4610.04
1.1810.09
0.63.0.08
0.99
GD
1.66=0.06
10.49+0.42
1.33=0.12
0.71+00.55
1.43+0.2
166++0.2 0.45+0.02
0.78+00.1
ADD-THIN (Ours)
0.45
1,07+0019
0.38+0002
0.37++0.02
0.69++0.03
0.5++0.33
E.3
Sampling runtimes
We compare sampling runtimes on an NVIDIA GTX 1080 Ti across the different models in Fig- ADD-THIN maintains near-constant runtimes by refining the entire sequence in parallel. The autoregressive baselines RNN and Transformer show increasing runtimes, with longer sequences surpassing ADD-THIN'S runtime. TriTPP a highly optimized baseline computing the autoregressive interactions between event times in parallel by leveraging triangular maps, resulting in the fastest runtimes. Lastly, GD autoregressive in event time and gradually refines each event time over 100 diffusion steps, leading to the worst runtimes.
RNN
102
Trans
GD
THTTPP
ADD-THIN
Runtime (seconds)
101
109
10-1
10-2
102
10a
Average sequence length
Figure 5: Sampling runtime for a batch of 100 event sequences averaged over 100 runs. We report the trained model's sampling times for the real-world datasets with different sequence lengths (from left right: Twitter, Yelp Yelp 2, PUBG, Taxi, Reddit-C, Reddit-A).
18


Reviews:
[{"review_id": "vkrbl409dX", "reviewer": "Reviewer_1d9q", "summary": "The paper introduces a novel probabilistic diffusion framework for temporal point process. Its significance lies modeling a whole event sequence directly, overcoming common limitations of autoregressive models.", "questions": "A.\tPosterior approximation.  How good is the posterior approximation in Add and Thin? Are there any theoretical insights. \nB.\tNeural network. Could the authors provide some movitation of using 3-layered-d CNN?\nC.\tLoss. 1). Is the mixture of Gaussians eqn. (4) used to model the conditional intensity good for modeling an event time(or interevent time) t which is positive? Add and Thin is a latent variable model with latent variables t^{n-1}’s. Usually int the diffusion models reference 40 in the paper, latent variable models leverage variational inference to maximize the evidence lower bounds. But this model is not. Could the author shed some insights on this? \nD.\tCox process models. Do the authors have any experiment results with these baselines form reference [5,8,17]? \nE.\tExperiments. How many diffusion steps (n) are used? \n", "limitations": "The limit of the paper according to the authors is it is limited to TPP without marks. In addition, current sampling routines may not be efficient. TPP models can be applied applied to many fields, where common application domains  include traffic, social networks, and electronic health records.\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality:  This paper is very novel. It connects diffusion models with TPPs and model a whole event sequence , overcoming issues from autoregressive models.\n\nQuality: The formulation of the approach and method itself are sound by leveraging two properties of point process: thinning and superpositions. The discussion of the reverse process are reasonable by discussing 4 cases of transition from nth step to n-1 step.  The empirical results demonstrate the good performance of the proposed approach, esp. in forecasting tasks. \n\nClarity:  The paper is overall well-presented. Improvement can be made by better explaining A,B,C,D,E,F sets. \n\nSignificance:  The paper introduces a new way to model event sequences probabilistically  which is appealing and significant to the TPP community. \n", "weaknesses": "A.\tExperiments. The experimental evaluation on density estimation. The results of Add and Thin is only comparable to Intensity-free model by Shchur (RNN in the paper), if not worse.\nB.\tApplication and Limitation. One minor limitation is it only deals with TPP without marks. It would be interested in see something for Marked TPP.\nC.\tOverall presentation. It took me a while to understanding the notation of sets A-F. It would be nice to include such information in the caption of figure 2. \nD. No codes are given, I am not sure how reproducible the results are.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review, appreciation of our work, and suggestions to further improve our paper.\n> W.A Comparison to Shchur et al.\n\nWe want to point out that the intensity-free model by Shchur et al. is a very strong baseline, and both our model and this baseline achieve near-perfect metrics in the density estimation task on most datasets. Further, the proposed model is primarily motivated through the conditional sampling/forecasting task, where our model outperforms all baselines.\n> W.B Minor point: Marks\n\nWe agree that including marks would pose an interesting extension of our model. In this paper, we focused on deriving a sound diffusion process for TPPs to tackle the complex task of accurately modeling event sequences in time and leave the extension to marks for future work.\n> W.C Presentation of sets\n\nPlease refer to our response to all reviewers for an improved presentation of the sets and reverse process.\n> W.D Reproducibility of results\n\nWe will release a well-documented version of our code upon acceptance.\n> Q. Relation of the loss to the ELBO\n\nThank you for this important remark. Our model's objective function is indeed equivalent to the ELBO of a diffusion model, as shown below. We will add this discussion to the appendix and reference it in Sec. 3.3.\n\nThe ELBO of diffusion models [2] is derived as follows:\n\\begin{equation}\n\\begin{split}\n    \\mathcal{L}\\_{ELBO} = \\mathbb{E}\\_{q} \\Big[\\underbrace{D\\_{KL}\\left(q(\\mathbf{t}^{(N)}|\\mathbf{t}^{(0)})\\parallel p(\\mathbf{t}^{(N)})\\right)}\\_{\\mathcal{L}\\_{N}}+\\sum_{n=2}^{N} \\underbrace{D\\_{KL}\\left(q(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(0)},\\mathbf{t}^{(n)})\\parallel p_{\\theta}(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(n)})\\right)}\\_{\\mathcal{L}\\_{n}}-\\underbrace{\\log p_{\\theta}(\\mathbf{t}^{(0)}|\\mathbf{t}^{(1)})}\\_{\\mathcal{L}\\_{0}}\\Big].\n\\end{split}\n\\end{equation}\n\nThe Janossi density [3] of a TPP allows us to represent each element of the ELBO in terms of our derived inhomogeneous (approximate) posterior intensities (Sec. 3.2.) as follows:\n\n\\begin{equation}\n    p(\\mathbf{t}) =e^{-\\int_0^T \\lambda(t)dt} \\prod_{t_i\\in \\mathbf{t}} \\lambda(t_i).\n\\end{equation} \n\n$\\boldsymbol{\\mathcal{L}\\_{N}}$: $\\mathcal{L}\\_{N}$ is constant as $q(\\mathbf{t}^{(N)}|\\mathbf{t}^{(0)})$ and $p(\\mathbf{t}^{(N)})$ have no learnable parameters.\n\n$\\boldsymbol{\\mathcal{L}\\_{0}}$: We directly train our model to optimize this likelihood term as described in Sec. 3.3.\n\n$\\boldsymbol{\\mathcal{L}\\_{n}}$: The KL divergence between two densities is defined as:\n\\begin{align}\n    D\\_{KL}(q\\parallel p_{\\theta}) = \\mathbb{E}\\_{q}\\left[\\log (q(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(0)},\\mathbf{t}^{(n)}))-\\log (p_{\\theta}(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(n)}))\\right],\n\\end{align}\nwhere only the right-hand side relies on $\\theta$, letting us minimize the KL divergence by maximizing the expectation over $\\log(p_{\\theta}(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(n)}))$.\n\nTo add some additional context to the KL divergence in $\\mathcal{L}\\_{n}$ and similar to the derivation of the posterior, we will further distinguish three cases:\n\n- **Set E:** Our parametrization uses the same intensity function for $q(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(0)},\\mathbf{t}^{(n)})$ and $p_\\theta(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(n)})$, which does not rely on any learned parameters.\n- **Set B \\& D:** $q(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(0)}, \\mathbf{t}^{(n)})$ and $p_\\theta(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(n)})$ are defined by Bernoulli distributions over each element of $\\mathbf{t}^{(n)}$. The BCE minimizes the KL divergence between the two distributions as proposed in Sec. 3.3.\n- **Set C**: Minimizing this KL divergence by maximizing the $\\mathbb{E}\\_{q}\\left[\\log(p_{\\theta}(\\mathbf{t}^{(n-1)} |\\mathbf{t}^{(n)}))\\right]$ is proposed in Sec. 3.3 to learn $\\lambda_{\\theta}^{(A\\cup C)}(t)$. Note that $q(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(0)},\\mathbf{t}^{(n)})$ is sampled from by independently thinning $\\mathbf{t}^{(0)}\\setminus\\mathbf{t}^{(n)}$. Consequently, by minimizing the NLL of our intensity $\\lambda_{\\theta}^{(A\\cup C)}(t)$ with regards to $\\mathbf{t}^{(0)}\\setminus\\mathbf{t}^{(n)}$, we optimize the expectation in closed form.\n\n> Q.A Posterior approximation\n\nConsidering our ELBO answer, we directly minimize the KL divergence to match the true posterior. Here, the posterior intensity $\\lambda^{(A\\cup C)}$ consists of Dirac delta functions, which in the limit $\\sigma\\rightarrow0$ can be perfectly approximated by Gaussians [1]. Further, we use a universal function approximator (MLP) to parameterize the Bernoulli distribution in the classification.\n> Q.B Movitation CNN\n\nWe use a CNN with dilation and circular padding as it's computationally much more efficient than attention based-encoders and captures long-range dependencies better than RNN-based ones. We'll clarify this choice in Sec. 3.3 in the final version.\n> Q.C Gaussian mixture\n\nWe use truncated Gaussians, ensuring positive intensity only on [0, T].\n> Q.D Cox processes\n\nCox processes are traditional parametric TPP models, where finding an efficient posterior sampling via MCMC for the latent dimensions is one main line of research [17]. Yet, to the best of our knowledge, there doesn't exist a computationally efficient Cox processes implementation. For example, the training times reported for a two-layer model in [17] are about 1-2 orders of magnitude higher than any of our applied models, ultimately making a hyperparameter search and comparison across 13 benchmark datasets infeasible. Please let us know if there is any specific efficient implementation we can compare with.\n\n> Q.E $n$\n\nWe report this hyperparameter in Sec. 5 and use $n=100$ for all experiments.\n\n### References\n1. Ghatak, Ajoy, et al. \"The dirac delta function.\" Quantum Mechanics: Theory and Applications (2004): 3-18.\n2. Ho et al., [16] in the paper\n3. Daley and Vere-Jones, [6] in the paper"}, {"review_id": "kl6cEfUPIU", "reviewer": "Reviewer_1xQ2", "summary": "The paper proposes a probabilistic diffusion model for TPPs, ADD-THIN, that naturally handles the continuous and discrete nature of point processes and directly models whole event sequences. While autoregressive methods are expressive in modeling event sequences, ADD-THIN does not suffer from the accumulation of errors caused by sequential sampling for long-term forecasting applications. The proposed method matches the performance of state-of-the-art models in density estimation and outperform them for forecasting on both synthetic and real-world datasets.\n", "questions": "See Weaknesses.", "limitations": "The authors have a section on some limitations of their work, although a more in-depth discussion would have been appreciated. Specifically, they mention that sampling from their diffusion model can be quite expensive, but they do not provide computional trade-offs in terms of time/resources across baselines. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- To the best of my knowledge, this work represents the first attempt to combine diffusion models and Temporal Point Processes. The potential of generating complete sequences without relying on auto-regressive mechanisms presents a promising avenue for future research. It is worth noting that this paper shares similarities with the unpublished work available at: https://slideslive.com/embed/presentation/38922857.\n\n- The proposed method is compared to multiple baselines for both unconditional and conditional sampling (forecasting).\n", "weaknesses": "\n- The paper is poorly written and hard to read.\n\n- There are multiple statements that are either inaccurate or without references\n\t- \"The intensity of a TPP describes the expected number of points\"\n\t- Line 65-70, you either need to prove it or provide references that do so. \n\t- Line 72, \"the expected number of points on [0, T] follows a Poisson distribution with rate ...\" \n\t- \"Even though Poisson processes can model seasonality\"\t\n\t\t- How does a Poisson model seasonality?\n\t- \"We know that conditioned on ... is a random sequence of events with a density that can be completely described by its (unconditional) intensity function\"\n\t- \"Since the sets are disjoint, ...\"\n\t- Is there a difference between a counting process and measure? an intensity function and measure? You use these terms interchangeably.\n\n\n- Multiple TPP datasets that are used in the experiments have been shown to be inappropriate benchmarks for neural TPPs. See https://openreview.net/forum?id=3OSISBQPrM\n\n- Overall, the presentation of the paper could be greatly improved, especially Section 3. Specifically, it feels like adding more formal notations (e.g. for Section 3.2.) would allow to shorten the text, while helping the reader more easily grasp the idea behind the method. For instance, why defer to the Appendix the pseudo-code for the sampling procedure?  \n\nAdditionally, several design choices are not clearly motivated, while the experiments and evaluation of the proposed model falls a bit short of expectations. These concerns are summarized in the points below: \n\n1) Why include both $\\boldsymbol{e}i$ and $\\boldsymbol{c}_i$ in the parametrization of $g\\theta$? Are the $\\boldsymbol{c}_i$ constructed from neighboring events? Otherwise, it looks like there is redundancy between the information contained in the two embeddings. \n\n2) There is no real motivation behind the use of a positional encoding for $n$. Why do you need it?  \n\n3) What is the motivation for using an unormalized mixture of Gaussians to parametrize $\\lambda^{(A \\cup C)}$? Have you considered alternative parametrizations that can be found in the neural TPP literature, e.g. an unmarked version of RMTPP [1]? Additionally, more details regarding the architectures employed would be appreciated (e.g. what is $\\sigma$, and what is the model behind the MLP on page 5?). \n\n4) In the NLL objective of equation 5, is the objective solely computed on times from $A \\cup C$? If yes, this should be indicated explicitly in the objective.\n\n5) On page 6, it is not clear how you find that $\\Lambda^{(A \\cup C)}(T) = K \\sum_{i}^H w_i$. Given your definition of $\\lambda^{(A \\cup C)}(t)$, I would have expected this result to hold only if the integral was taken over $-\\infty, \\infty$. \n\n6) The paragraph on 'conditional sampling' requires more detailed explanations, as it is impossible to see from the text how you handle this scenario. Specifically, how is $\\boldsymbol{h}$ integrated into $g_{\\theta}$ and $\\lambda^{(A \\cup C)}(t)$? \n\n6) As $\\hat{t}^{(0)}$ is successively approximated at each time step $n$ during the reverse process, how can you ensure that the propagation of errors is mitigated and won't affect significantly the generated sequences after $N$ steps? \n\n7) Why only reporting the MAPE between the generated and truth sequences lengths? Isn't there a way to evaluate the quality of the generated samples within a sequence itself (e.g. using a measure of distance)? \n\n[1] Du, Nan and Dai, Hanjun and Trivedi, Rakshit and Upadhyay, Utkarsh and Gomez-Rodriguez, Manuel and Song, Le. (2016). Recurrent Marked Temporal Point Processes: Embedding Event History to Vector. SIGKDD. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the extensive review, feedback, and questions. In the following, we address the raised concerns and questions.\n> W.1,4 Readability and understandability (3, 3.2)\n\nConsidering the character limit, we refer to both our response to all reviewers, where we outline suggested enhancements for the reverse process and our answer to reviewer RwFj's comments regarding the method description. Further, we will use the additional page of the camera-ready version to move the pseudo-code of the sampling procedure from the App. to Sec. 3.4. Lastly, subsequent segments in this rebuttal are dedicated to address questions and misunderstandings in our method description.\n> W.2 Clarifications and references\n1. \"The intensity is defined as:\"; line 64 \"The intensity can be interpreted as the expected number of events per unit of time.\"\n2. Reference to [7]\n3. \"the number of points...\"\n4. \"...inhomogeneous Poisson processes...\" The time-varying intensity of an IPP can, e.g., capture the fact that more events happen at a certain day of the week, known as seasonality.\n5. Rewritten as part of the response to all reviewers. \n6. \"Since the sets B-E...\"\n7. As described in Sec. 2.1, a counting process (in our case TPP) can be described by its intensity measure or less technical function, while a realization can be represented by its counting measure. We will use \"intensity function\" throughout to make the text more consistent.\n\n> W.3 Benchmark datasets\n\nThank you for pointing out this relevant concurrent work, raising concerns in the evaluation of autoregressive TPP models with teacher forcing for some widely used TPP benchmark datasets. We were unable to incorporate its findings since the work was published 2 months after the NeurIPS 2023 submission deadline. In general, we agree that this is an important concern. However, likely these findings are not directly applicable to the setting considered in our work as we consider tasks, metrics, and model architectures that are all different and have found significant empirical differences between the baselines even on these datasets. We will investigate this issue in more detail and add a discussion to the revised version of the paper.\n> Design choices:\n\nWe clarify the motivation for the design choices in the following:\n1. In theory, $e_i$ can contain all information of $c_i$, but we have found that providing the event time explicitly gives better results.\n2. Positional encodings for n were first proposed in [16] to improve the model capacity. To our knowledge, all current diffusion models leverage positional encodings as $n$ strongly affects the posterior.\n3. In our diffusion model, all posterior intensities are inhomogeneous PP (c.f., PP section of the parametrization of RMTPP). The Gaussian distribution ($\\sigma\\rightarrow0$) is a standard approximation of the Dirac delta function in the posterior (see also answer to reviewer 1d9q). We explored other options, such as mixture of Beta and Weibull, but found their parametric form to restrict the modeling of Dirac delta functions. The MLPs have two layers, a hidden dimension of 32 and a ReLU activation (we add additional information to the App.). Lastly, $\\sigma$ refers to the sigmoid activation, which we will make explicit.\n4. The objective is only computed for $A\\cup C$ as indicated in Eq. 5. Note that the sum in Eq. 5 could equally be written as $\\sum_{t_i\\in\\mathbf{t}^{(A\\cup C)}}$.\n5. We employ re-normalized truncated Gaussians on the interval [0,T]. We will add this information to the paragraph.\n6. The intensity and classifier get the positional embedding of the diffusion time n as an input. The history embedding $\\mathbf{h}$ is simply added to this embedding. We will revise the last sentence to even better convey this information.\n7. When sampling from any diffusion model, a noisy sample is gradually refined to produce a sample from the learned data distribution. Hence, each subsequent diffusion step can correct for errors (or rather noise) of the previous ones. In our case, we refine the **entire event sequence** at every diffusion step to finally produce a data sample. This starkly contrasts TPP models that are autoregressive in the event time and do not adjust errors in earlier events. This difference becomes especially evident in our conditional sampling experiment, where the accumulation of error in event time restricts the forecast capability of autoregressive TPP models. Lastly, we would like to use this chance to refer to our response to reviewer 1d9q, where we elaborate on the relation of our loss and the ELBO and training our model by minimizing the KL divergence between our approximate and the true posterior.\n8. We think there has been a slight misunderstanding. We do not only report the MAPE for the sequence lengths but also report the Wasserstein distance between the generated and ground truth sequence. This Wasserstein distance measures the distance between the two counting measures representing each event sequence.\n\n> Computational complexity\n\nIn our response to all reviewers, we added a plot to the PDF that compares the sampling runtimes for the different models. We find that for longer sequences, the runtime of our unoptimized model is comparable to autoregressive models. This interesting finding can be explained on a conceptual level as our diffusion model presents a different trade-off: Instead of the autoregressive nature of common TPP models in event time, our model in parallel refines the entire sequence for a fixed number of N diffusion steps. Thereby, the sampling is almost constant across the different sequence lengths.\n> Similarity to unpublished work\n\nThank you for pointing out this work, which discusses the general idea of a hierarchical composition of simple intensity functions and defining invertible probabilistic transformation (destructors) of point processes. If you could provide us with a preprint or reference to this work, we would be happy to discuss and reference it."}, {"review_id": "v91IktLZoa", "reviewer": "Reviewer_RwFj", "summary": "The forward process adds points from homogeneous poisson process (HPP) into the sequence and removes points from original sequence ($\\mathbf{t}^{(0)}$). The goal is such that $\\mathbf{t}^{(n)}$ is HPP. The neural network is trained to approximate missing information about $\\mathbf{t}^{(0)}$, e.g. figure out which points from HPP came from $\\mathbf{t}^{(0)}$ and which points were removed at intermediate steps. The thinning and superposition theorems were used multiple times to derive intensities of various sets, where the sets categorize the points.", "questions": "1. Would like to introduce this recent other work that connects point processes and diffusions. in this case, the interarrival times of a point process is related to the length of a diffusion excursion. https://openreview.net/forum?id=KIlwyX7nCi\n\n2. What is the benefit of applying the proposed model compared to other methods of generating sequences?", "limitations": "1. Please address the potential limitations of using a neural network to approximate missing information about $\\mathbf{t}^{(0)}$.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The adding and thinning approach, training via classification and matching, to modeling and generating temporal point processes is original and interesting.", "weaknesses": "1. The writing and notation is extremely confusing. I read the paper multiple times.\n\n1a. e.g. please clearly specify the observation, is it a time sequence $x(t), t\\in[0,T]$ where $x(t) = 0$ for no event, $x(t) = 1$ for event, discretized in time. or is it a varying length sequence $\\mathbf{t}=(t_1,...,t_K), 0< t_1<...<t_K\\leq T$ with $K$ arrival times. also, section 3.3 writes about sequence embedding to get an input in $R^d$ but I am still confused.\n\n1b. e.g. what are the sets A and F? what are the differences between B and C, D and E? B seems to be the points preserved from $\\mathbf{t}^{(0)}$ all the way to HPP, C seems to be the points preserved from $\\mathbf{t}^{(0)}$ but removed in the next step. is there a clearer way of writing the material in section 3.2?\n\n1c. perhaps the background section can be shortened if we need more space to explain the method clearly.\n\n1d. would potentially increase the score if the model and method is clearly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback. In the following, we address the specific concerns and questions.\n\n> W.1a. Obervations\n\nGenerally a realization of a TPP can be represented as a sequence of strictly increasing arrival times: $\\mathbf{t} = (t_1, \\dots, t_K)$,  $0 < t_1 < \\dots < t_K \\leq T$. Equivalently, a realization can also be represented by its counting measure $N(t) = \\smash{\\sum_i^K \\mathbb{I}(t_i \\le t)}$, for $t \\in [0, T]$. In the implementation of our method we leverage the arrival times $\\mathbf{t}$, which we will make more explicit by adding the formal definition to the introduction of $\\mathbf{t}^{(0)}$ in line 101. Furthermore, we employ a temporal embedding followed by a sequence embedding with a CNN to attain embeddings of $\\mathbf{t}^{(n)}$ and subsequently parameterize the posterior.\n> W.1b. Unclear distinction between sets A,...,F\n\nWe have revised section 3.2 and present an improved explanation of the sets in the response to all reviewers to address this weakness.\n\n> W.1c. Shorten background section if more space is needed \n\nThank you for the suggestion. Given the more concise revision of section 3.2 and the additional page for the camera-ready version we will not need to cut down on the background section. Additionally, we will use the additional page to describe the method in more detail and explain the motivation behind the design choices.\n\n> W.1d. I would potentially increase the score if the model and method is clearly explained.\n\nWe hope that the revision of section 3.2 in the response to all reviewers presents a clearer explanation of the model and method. Further, we would like to add the following revisions and points to better explain the model and method:\n\n**Section 2.2**\n\n[Line 85-91]\n\nDiffusion models [16, 40] are a class of latent variable models that learn a generative model to reverse a fixed probabilistic noising process $x_{0} \\to x_{1} \\to \\dots \\to x_N$, which gradually adds noise to clean data $x_0$ until no information remains, i.e., $x_{N}\\sim p(x_N)$. For continuous data, the forward process is usually defined as a fixed Markov chain $q(x_{n} \\mid x_{n-1})$ with Gaussian transitions. Then the Markov chain of the reverse process is captured by approximating the true posterior $q(x_{n-1}|x_{n}, x_{0})$ with a model $p_{\\theta}(x_{n-1}|x_{n})$. Ultimately, sampling new realizations $x_0$ from the modeled data distribution $p_{\\theta}(x_{0}) = \\int p(x_{N}) \\prod^N_{n=1} p_{\\theta}(x_{n-1}| x_{n}) dx_{1},\\cdots,x_{N}$ is performed by starting with a sample from pure noise $x_{N}\\sim p(x_N)$ and gradually denoising it with the learned model over $N$ steps $x_{N} \\to x_{N-1} \\to \\dots \\to x_0$.\n\n**Section 3.3** \n\n[Line 173-177]\n\nIn the previous section we have derived the intensity $\\lambda_{n-1}(t | \\mathbf{t}^{(0)}, \\mathbf{t}^{(n)})$ of the posterior $q(\\mathbf{t}^{(n-1)}|\\mathbf{t}^{(0)}, \\mathbf{t}^{(n)})$ for the reverse process, i.e., the intensity of points at step $n-1$ given $\\mathbf{t}^{(0)}$ and $\\mathbf{t}^{(n)}$. Now we want to approximate this posterior using a model $p_{\\theta}(\\mathbf{t}^{(n-1)} | \\mathbf{t}^{(n)}, n)$ to learn to sample points $\\mathbf{t}^{(n-1)}$ given only $\\mathbf{t}^{(n)}$. As we are only missing information about $\\mathbf{t}^{(0)}$ we will learn to model $\\lambda^{(B)}(t)$ and $\\lambda^{(A\\cup C)}(t)$ to approximate $\\hat{\\mathbf{t}}^{(0)} \\approx \\mathbf{t}^{(0)}$ (see Figure 3.) for each $n$ and then have access to the full posterior intensity to reverse the noising process. \n\n[Line 178]\n\nTo condition our model on $\\mathbf{t}^{(n)}$ and $n$ we propose the following embeddings. [...]\n\n> Q1. Connection to work on diffusion excursion for TPPs.\n\nThank you for introducing us to this exciting concurrent work. In this paper, they model TPPs by relating the continuous latent state in a diffusion process to the event times by modeling the interevent times as the length of a diffusion excursion. Thereby, the proposed method is still autoregressive and more strongly related to TPP models, which model event times conditional on a continuous latent process (see, e.g. [4, 20, 11]). Nevertheless, the proposed connection between the continuous stochastic processes and TPPs is very interesting, and we will make sure to reference and discuss this work in our related work section.\n\n> Q2. Benefit of Add-Thin compared to other methods\n\nOur method is the first diffusion-based TPP model that generates entire event sequences, thereby overcoming some of the shortcomings of autoregressive models in forecasting. Additionally, by replacing the autoregressive parametrization in event time with an gradual refinement process of the entire event sequence for a fixed number of N diffusion steps, our model is better suited to model very long event sequences (c.f. sampling speed in response to all reviewers). Furthermore, the iterative refinement of the event sequence allows us to leverage simple and shared layers to model the long-range interaction between events, a task for which attention (computational complexity) and RNN-based encoders are known to struggle.\n\n> Please address the potential limitations of using a neural network to approximate missing information about $\\mathbf{t}^{(0)}$.\n\nOur model is trained to generate samples $\\mathbf{t} \\sim q(\\mathbf{t}^{(0)})$, by optimizing the ELBO of our probabilistic model (see discussion of the relationship of our loss and the ELBO in the response to reviewer 1d9q). In that context, the applied neural networks to approximate the posterior are universal function approximators and can in theory model the posterior distribution arbitrarily well. "}, {"review_id": "6wuDtAAYlJ", "reviewer": "Reviewer_6sim", "summary": "The submission derive a probabilistic diffusion model for TPPs. By proposing the Add-Thin framework, the proposed method can naturally handles the continuous and discrete nature of point processes and directly models the whole event sequences. Compared with the traditional autoregressive approaches for the temporal point process (TPP), the proposed framework is immune to the accumulation of errors caused by sequential sampling. ", "questions": "None. ", "limitations": "The authors discussed the limitations and potential negative societal impact of their work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The submission proposed a framework that learns the probabilistic mapping from complete noise, i.e., a homogeneous Poisson process (HPP), to data. More specifically, a model is learned to reverse the noising process of adding (superposition) and removing (thinning) points from the event sequence by matching the conditional inhomogeneous denoising intensity. This idea is very interesting. ", "weaknesses": "I do not find any obvious weaknesses. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and appreciation of our work.\n"}]

Author rebuttal:
We thank the reviewers for their valuable feedback and appreciation of the contribution and originality of our novel diffusion-based TPP model. We have attached a PDF with the model's sampling runtime and would further like to highlight parts of our answers to reviewer 1d9q (ELBO), reviewer 1xQ2 (Design choices), and reviewer RwFj (Model and method description). 

In the following, we re-introduce a high-level explanation of the reverse process and the treatment of sets A-F that we will include in Sec. 3.2 in the updated version of the paper. We want to emphasize that the following neither introduces new notations nor new findings and solely presents Sec. 3.2 to be more accessible to the reader.

**Reverse process and case distinction**

To sample realizations $\bf{t}\sim\lambda_0$ starting from $\mathbf{t}^{(N)}\sim\lambda_{HPP}$, we need to learn to reverse the Markov chain of the forward process. Conditioned on $\mathbf{t}^{(0)}$, the reverse process at step $n$ is given by the posterior $q(\mathbf{t}^{(n-1)}|\mathbf{t}^{(0)},\mathbf{t}^{(n)})$, which is an inhomogeneous Poisson process for the chosen forward process. Therefore, the posterior can be represented by a history-independent intensity function $\lambda_{n-1}(t|\mathbf{t}^{(0)},\mathbf{t}^{(n)})$. As the forward process is defined by adding and thinning event sequences, the posterior over the random sequence $\mathbf{t}^{(n-1)}$ can be decomposed into disjoint sets of points based on whether they are also in $\mathbf{t}^{(0)}$ or $\mathbf{t}^{(n)}$. We use this case distinction to derive the posterior intensity and distinguish the following cases: points in $\mathbf{t}^{(n-1)}$ that where kept from $0$ to $n$ (**B**), points in $\mathbf{t}^{(n-1)}$, that were kept from $0$ to $n-1$ but thinned at the $n$-th step (**C**), added points in $\mathbf{t}^{(n-1)}$ that are kept in the $n$-th step (**D**) and added points in $\mathbf{t}^{(n-1)}$ that are thinned in the $n$-th step (**E**). Since the sets **B-E** are disjoint, the posterior intensity is a superposition of the intensities of each subset of $\mathbf{t}^{(n-1)}$: $\lambda_{n-1}(t|\mathbf{t}^{(0)},\mathbf{t}^{(n)})=\lambda^{(B)}(t)+\lambda^{(C)}(t)+\lambda^{(D)}(t)+\lambda^{(E)}(t)$. 

To derive the intensity functions for cases B-E, we additionally define the following helper sets: **A** the points $\mathbf{t}^{(0)}\setminus\mathbf{t}^{(n-1)}$ that were thinned until $n-1$ and **F** the points $\mathbf{t}^{(n)}\setminus\mathbf{t}^{(n-1)}$ that have been added at step $n$. The full case distinction is further illustrated in Fig. 2. In the following paragraphs, we derive the intensity functions for cases B-E:

**B**: The set $\mathbf{t}^{(B)}$ can be formally defined as $\mathbf{t}^{(0)}\cap\mathbf{t}^{(n)}$ as $(\mathbf{t}^{(0)}\cap\mathbf{t}^{(n)})\setminus\mathbf{t}^{(n-1)}=\emptyset$, almost surely. This is because adding points at any of the locations $t\in\mathbf{t}^{(0)}\cap\mathbf{t}^{(n)}$ carries zero measure at every noising step. Hence, given $\mathbf{t}^{(0)}\cap\mathbf{t}^{(n)}$ the intensity can be written as a sum of Dirac measures: $\lambda^{(B)}(t)=\sum_{t_i\in(\mathbf{t}^{(0)}\cap\mathbf{t}^{(n)})}\delta_{t_i}(t)$.

**C**: Given $\mathbf{t}^{(A\cup C)}=\mathbf{t}^{(0)}\setminus\mathbf{t}^{(n)}$, $\mathbf{t}^{(C)}$ can be found by thinning and consists of points that were kept by step $n-1$ and removed at step $n$. Using the thinning of Eq. 2 \& 3, we know the probability of a point from $\mathbf{t}^{(0)}$ being in $\mathbf{t}^{(C)}$ and $\mathbf{t}^{(A\cup C)}$ is $\bar\alpha_{n-1}(1-\alpha_n)$ and $1-\bar\alpha_{n}$, respectively. Since we already know $\mathbf{t}^{(B)}$ we can consider the probability of finding a point in $\mathbf{t}^{(C)}$, given $\mathbf{t}^{(A\cup C)}$, which is equal to $\frac{\bar\alpha_{n-1}-\bar\alpha_{n}}{1-\bar\alpha_{n}}$.

**E**: The set $\mathbf{t}^{(E)}$ contains all points $t \notin (\mathbf{t}^{(0)}\cup \mathbf{t}^{(n)})$ that were added until step $n-1$ and thinned at step $n$. Again using Eq. 2 \& 3, we can see that these points were added with intensity $(1-\bar\alpha_{n-1})\lambda_{HPP}$ and then removed with probability $(1-\alpha_n)$ at the next step. Equivalently, we can write down the intensity that governs this process as $\lambda^{(E)}=(1-\bar\alpha_{n-1})(1-\alpha_n)\lambda_{HPP}$, i.e., sample points from an HPP and thin them to generate a sample $\mathbf{t}^{(E)}$.

**D**: The set $\mathbf{t}^{(D)}$ can be found by thinning $\mathbf{t}^{(D\cup F)}=\mathbf{t}^{(n)}\setminus\mathbf{t}^{(0)}$ and contains the points that were added by step $n-1$ and then kept at step $n$. The processes that generated $\mathbf{t}^{(D)}$ and $\mathbf{t}^{(F)}$ are two independent HPPs with intensities $\lambda^{(D)}=(1-\bar\alpha_{n-1})\alpha_n\lambda_{HPP}$ and $\lambda^{(F)}=(1-\alpha_{n})\lambda_{HPP}$, respectively, where $\lambda^{(D)}$ is derived similar to $\lambda^{(E)}$. Since $\mathbf{t}^{(D)}$ and $\mathbf{t}^{(F)}$ are independent HPPs and we know $\mathbf{t}^{(D\cup F)}$, the number of points in $\mathbf{t}^{(D)}$ follows a Binomial distribution with probability $p=\frac{\lambda^{(D)}}{\lambda^{(D)}+\lambda^{(F)}}$ (see A.2 for details). That means we can sample $\mathbf{t}^{(D)}$ given $\mathbf{t}^{(n)}$ and $\mathbf{t}^{(0)}$ by simply thinning $\mathbf{t}^{(D\cup F)}$ with probability $1-p$ and express the intensity as a thinned sum of Dirac measures (c.f., Fig. 2).

For sequences of the training set, where $\mathbf{t}^{(0)}$ is known, we can compute these intensities for all samples $\mathbf{t}^{(n)}$ and reverse the forward process. However, $\mathbf{t}^{(0)}$ is unknown when sampling new sequences. Therefore, similarly to the denoising diffusion approaches [16], in the next section, we show how to approximate the posterior intensity, given only $\mathbf{t}^{(n)}$. Further, in Sec. 3.4, we demonstrate how the trained neural network can be leveraged to sample new sequences $\mathbf{t}\sim\lambda_0$.

Meta-review: The paper presents a novel idea - a diffusion model for temporal point process. The original paper had clarity issues, but the authors responded well to reviewers' comments and now the scores seem to have moved towards acceptance. Still some concerns about clarity and presentation remain. I recommend accept as a poster based on the novelty.
Decision: Accept (poster)