Title: Add and Thin: Diffusion for Temporal Point Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Point Processes

Diffusion

Temporal Data

Generative Model

Forecasting

Density Estimation

Denoising
Abstract: Autoregressive neural networks within the temporal point process (TPP) framework have become the standard for modeling continuous-time event data. Even though these models can expressively capture event sequences in a one-step-ahead fashion, they are inherently limited for long-term forecasting applications due to the accumulation of errors caused by their sequential nature. To overcome these limitations, we derive ADD-THIN, a principled probabilistic denoising diffusion model for TPPs that operates on entire event sequences. Unlike existing diffusion approaches, ADD-THIN naturally handles data with discrete and continuous components. In experiments on synthetic and real-world datasets, our model matches the state-of-the-art TPP models in density estimation and strongly outperforms them in forecasting.

Add and Thin: Diffusion for Temporal Point Processes

Abstract

Autoregressive neural networks within the temporal point process (TPP) framework have become the standard for modeling continuous-time event data. Even though these models can expressively capture event sequences in a one-step-ahead fashion, they are inherently limited for long-term forecasting applications due to the accumulation of errors caused by their sequential nature. To overcome these limitations, we derive ADD-THIN, a principled probabilistic denoising diffusion model for TPPs that operates on entire event sequences. Unlike existing diffusion approaches, ADDTHIN naturally handles data with discrete and continuous components. In experiments on synthetic and real-world datasets, our model matches the state-of-the-art TPP models in density estimation and strongly outperforms them in forecasting.

1 Introduction

Many machine learning applications involve the analysis of continuous-time data, where the number of events and their times are random variables. This data type arises in various domains, including healthcare, neuroscience, finance, social media, and seismology. Temporal point processes (TPPs) provide a sound mathematical framework to model such event sequences, where the main problem is finding a parametrization that can capture the seasonality and complex interactions (e.g., excitation and inhibition) within point processes.

Traditional TPP models [17, 20] employ simple parametric forms, limiting their flexibility to capture the intricacies of arbitrary TPPs. In recent years, various neural TPPs have been proposed (see [43] for an overview) that capture complex event interactions in an autoregressive manner,

0

T

!

(")

~	$

$%%

!

(&)

~	$

'()(

!

(*+,)

0

T

!

(*)

$

*+,

(!|!

(*)

, !

(&)

)

t

(

n

)

t

(

n

-

1)

λ

n

-

1

(

t

|

t

(0)

,

t

(

n

)

t

(0)

∼

λ

Data

t

(

N

)

∼

λ

HPP

) =

Code is available at https://www.cs.cit.tum.de/daml/add-thin

λ

(

B

)

(

t

) +

λ

(

C

)

(

t

) +

λ

(

D

)

(

t

) +

λ

(

often using recurrent neural networks (RNN) or transformer architectures. While autoregressive models are expressive and have shown good performance for one-step-ahead prediction, their suitability for forecasting remains limited due to the accumulation of errors in sequential sampling.

We propose to take a completely different approach: instead of autoregressive modeling, we apply a generative diffusion model, which iteratively refines entire event sequences from noise. Diffusion models [18, 45] have recently shown impressive results for different data domains, such as images [15, 18, 25], point clouds [31, 32], text [27] and time-series [1, 3, 24, 46]. But how can we apply diffusion models to TPPs? We cannot straightforwardly apply existing Gaussian diffusion models to learn the mapping between TPPs due to the particular requirements that must be met, namely, the randomness in the number of events and the strictly positive arrival times.

We present a novel diffusion-inspired model for TPPs that allows sampling entire event sequences at once without relying on a specific choice of a parametric distribution. Instead, our model learns the probabilistic mapping from complete noise, i.e., a homogeneous Poisson process (HPP), to data. More specifically, we learn a model to reverse our noising process of adding (superposition) and removing (thinning) points from the event sequence by matching the conditional inhomogeneous denoising intensity λ n -1 ( t | t (0) , t ( n ) ) as presented in Figure 1. Thereby, we achieve a natural way to generate sequences with varying numbers of events and expressively model arbitrary TPPs. In short, our contributions are as follows:

We connect diffusion models with TPPs by introducing a novel model that naturally handles the discrete and continuous nature of point processes.

We propose a model that is flexible and permits parallel and closed-form sampling of entire event sequences, overcoming common limitations of autoregressive models.

We show that our model matches the performance of state-of-the-art TPP models in density estimation and outperforms them in forecasting.

2 Background

2.1 Temporal point processes (TPPs)

Temporal point processes (TPPs) [8, 9] are stochastic processes that define a probability distribution over event sequences whose number of points (events) K and their locations (arrival times) t i are random. A realization of a TPP can be represented as a sequence of strictly increasing arrival times: t = ( t 1 , . . . , t K ) , 0 < t 1 < · · · < t K ≤ T . Viewing a TPP as a counting process, we can equivalently represent a TPP realization by a counting measure N ( t ) = ∑ K i I ( t i ≤ t ) , for t ∈ [0 , T ] . The intensity characterizing a TPP can be interpreted as the expected number of events per unit of time and is defined as:

where H t = { t i : t i < t } is the event history until time t , which acts as a filtration to the process.

TPPs have a number of convenient theoretical properties, two of which will be central to our derivation of a noising process for TPPs later in the paper. The first property is superposition : If we combine events generated by TPPs with intensities λ 1 ( t ) and λ 2 ( t ) , the resulting event sequence will again follow a TPP, but now with intensity λ 1 ( t ) + λ 2 ( t ) . Conversely, randomly removing each event generated by a TPP process with intensity λ ( t ) with probability p is known as independent thinning . This is equivalent to sampling from a TPP with intensity (1 -p ) λ ( t ) [9].

Poisson process. A (in)homogeneous Poisson process is the simplest class of TPPs, where the rate of event occurrence is independent of the history. Then the number of points on [0 , T ] follows a Poisson distribution with rate Λ( T ) = ∫ T 0 λ ( t ) d t . In the context of our model, the Poisson process with a constant intensity on [0 , T ] , called homogeneous Poisson Process (HPP), represents the noise distribution. Even though Poisson processes can model seasonality, i.e., time-varying rates of event occurrence, they assume the events to be independent and do not capture the exciting or inhibiting behavior present in the real world, e.g., a large earthquake increasing the likelihood of observing other earthquakes soon after.

Conditional intensity. Most TPP models leverage the conditional intensity function λ ( t |H t ) or equivalently the conditional density p ( t |H t ) to overcome the independence of points limitation of an inhomogeneous Poisson process. Historically, these intensity models were parameterized using hand-crafted functions [17, 20], whereas now, it is more common to use neural networks for learning intensities from data [12, 33, 41]. While the conditional intensity provides a general framework to model TPPs, sampling from these models is inherently autoregressive.

2.2 Denoising diffusion probabilistic models

Diffusion models [18, 45] are a class of latent variable models that learn a generative model to reverse a fixed probabilistic noising process x 0 → x 1 →··· → x N , which gradually adds noise to clean data x 0 until no information remains, i.e., x N ∼ p ( x N ) . For continuous data, the forward (noising) process is usually defined as a fixed Markov chain q ( x n | x n -1 ) with Gaussian transitions. Then the Markov chain of the reverse process is captured by approximating the true posterior q ( x n -1 | x 0 , x n ) with a model p θ ( x n -1 | x n ) . Ultimately, sampling new realizations x 0 from the modeled data distribution p θ ( x 0 ) = ∫ p ( x N ) ∏ N n =1 p θ ( x n -1 | x n ) d x 1 . . . x N is performed by starting with a sample from pure noise x N ∼ p ( x N ) and gradually denoising it with the learned model over N steps x N → x N -1 →··· → x 0 .

3 ADD-THIN

In the following, we derive a diffusion-like model for TPPs-ADD-THIN. The two main components of this model are the forward process, which converts data to noise (noising), and the reverse process, which converts noise to data (denoising). We want to emphasize again that existing Gaussian diffusion models [18, 45] are not suited to model entire event sequences, given that the number of events is random and the arrival times are strictly positive. For this reason, we will derive a new noising and denoising process (Sections 3.1 & 3.2), present a learnable parametrization and appropriate loss to approximate the posterior (Section 3.3) and introduce a sampling procedure (Sections 3.4 & 3.5).

3.1 Forward process - Noising

Let t (0) = ( t 1 , . . . , t K ) denote an i.i.d. sample from a TPP (data process) with T = 1 specified by the unknown (conditional) intensity λ 0 . We define a forward noising process as a sequence of TPPs that start with the true intensity λ 0 and converge to a standard HPP, i.e., λ 0 → λ 1 →··· → λ N :

where 1 > α 1 > α 2 > · · · > α N > 0 and λ HPP denotes the constant intensity of an HPP. Equation 2 corresponds to a superposition of (i) a process λ n -1 thinned with probability 1 -α n (removing old points), and (ii) an HPP with intensity (1 -α n ) λ HPP (adding new points).

Property (Stationary intensity) . For any starting intensity λ 0 , the intensity function λ N given by Equation 2 converges towards λ HPP . That is, the noised TPP will be an HPP with λ HPP .

Proof. In Appendix B.1 we show that, given λ 0 and Equation 2, λ n is given by:

If all α n are close to 1 , each consecutive realization will be close to the one before because we do not remove a lot of original points, nor do we add many new points. And if we have enough steps, we will almost surely converge to the HPP. Both of these properties will be very useful in training a generative model that iteratively reverses the forward noising process.

But how can we sample points from λ n if we do not have access to λ 0 ? Since we know the event sequence t (0) comes from a true process which is specified with λ 0 , we can sample from a thinned process ¯ α n λ 0 ( t ) , by thinning the points t (0) independently with probability 1 -¯ α n . This shows that even though we cannot access λ 0 , we can sample from λ n by simply thinning t (0) and adding new points from an HPP.

D

Posterior intensity:

B

C

D

E

t

(

n

)

t

(0)

t

(

n

-

1)

λ

(

B

)

(

t

) =

∑

t

i

∈

t

(0)

∩

t

(

n

)

δ

t

i

(

t

)

λ

(

D

)

(

t

) = (1

-

¯

α

n

-

1

)(1

-

α

n

)

λ

HPP

(

t

)

λ

(

E

)

(

t

) =

∑

t

i

∈

t

(

n

)

\

t

(0)

(

α

n

-

¯

α

n

)

(1

-

¯

α

n

)

δ

t

i

(

t

)

λ

(

C

)

(

t

) =

∑

t

i

∈

t

(0)

\

t

(

n

)

¯

α

n

-

1

-

¯

α

n

1

-

¯

α

n

δ

t

i

(

t

)

λ

n

-

1

(

t

|

t

(0)

,

t

(

n

)

) =

λ

(

B

)

(

t

) +

λ

(

C

)

(

t

) +

λ

(

D

)

(

t

) +

λ

(

E

)

(

t

)

λ

n

-

1

(

t

|

t

(0)

,

t

(

n

)

) =

λ

(

B

)

(

t

) +

λ

(

C

)

(

t

) +

λ

(

λ

(

E

)

(

t

) =

∑

t

i

∈

t

(

n

)

\

t

(0)

α

n

-

¯

α

n

1

-

¯

α

n

δ

t

i

(

t

)

To recap, given a clean sequence t (0) , we obtain progressively noisier samples t ( n ) by both removing original points from t (0) and adding new points at random locations. After N steps, we reach a sample corresponding to an HPP-containing no information about the original data.

3.2 Reverse process - Denoising

To sample realizations t ∼ λ 0 starting from t ( N ) ∼ λ HPP , we need to learn to reverse the Markov chain of the forward process, i.e., λ N →··· → λ 0 , or equivalently t ( N ) →··· → t (0) . Conditioned on t (0) , the reverse process at step n is given by the posterior q ( t ( n -1) | t (0) , t ( n ) ) , which is an inhomogeneous Poisson process for the chosen forward process (Section 3.1). Therefore, the posterior can be represented by a history-independent intensity function λ n -1 ( t | t (0) , t ( n ) ) .

To derive the intensity functions for cases B -E , we additionally define the following helper sets: A the points t (0) \ t ( n -1) that were thinned until n -1 and F the points t ( n ) \ t ( n -1) that have been added at step n . The full case distinction and derived intensities are further illustrated in Figure 2. In the following paragraphs, we derive the intensity functions for cases B -E :

As the forward process is defined by adding and thinning event sequences, the points in the random sequence t ( n -1) can be decomposed into disjoint sets of points based on whether they are also in t (0) or t ( n ) . We distinguish the following cases: points in t ( n -1) that were kept from 0 to n ( B ), points in t ( n -1) , that were kept from 0 to n -1 but thinned at the n -th step ( C ), added points in t ( n -1) that are thinned in the n -th step ( D ) and added points in t ( n -1) that are kept in the n -th step ( E ). Since the sets B -E are disjoint, the posterior intensity is a superposition of the intensities of each subsets of t ( n -1) : λ n -1 ( t | t (0) , t ( n ) ) = λ ( B ) ( t ) + λ ( C ) ( t ) + λ ( D ) ( t ) + λ ( E ) ( t ) .

Case B: The set t ( B ) can be formally defined as t (0) ∩ t ( n ) since ( t (0) ∩ t ( n ) ) \ t ( n -1) = ∅ almost surely. This is because adding points at any of the locations t ∈ t (0) ∩ t ( n ) carries zero measure at every noising step. Hence, given t (0) ∩ t ( n ) the intensity can be written as a sum of Dirac measures: λ ( B ) ( t ) = ∑ t i ∈ ( t (0) ∩ t ( n ) ) δ t i ( t ) . Similar to how the forward process generated t ( B ) by preserving some points from t (0) , sampling from the reverse process preserves points from t ( n ) .

Case C: Given t ( A ∪ C ) = t (0) \ t ( n ) , t ( C ) can be found by thinning and consists of points that were kept by step n -1 and removed at step n . Using the thinning of Equations 2 and 3, we know the probability of a point from t (0) being in t ( C ) and t ( A ∪ C ) is ¯ α n -1 (1 -α n ) and 1 -¯ α n , respectively. Since we already know t ( B ) we can consider the probability of finding a point in t ( C ) , given t ( A ∪ C ) , which is equal to ¯ α n -1 -¯ α n 1 -¯ α n . Consequently, λ ( C ) ( t ) is given as a thinned sum of Dirac measures over t ( A ∪ C ) (cf., Figure 2).

Case D: The set t ( D ) contains all points t / ∈ ( t (0) ∪ t ( n ) ) that were added until step n -1 and thinned at step n . Again using Equations 2 and 3, we can see that these points were added with intensity (1 -¯ α n -1 ) λ HPP and then removed with probability α n at the next step. Equivalently, we can write down the intensity that governs this process as λ ( D ) ( t ) = (1 -¯ α n -1 )(1 -α n ) λ HPP , i.e., sample points from an HPP and thin them to generate a sample t ( D ) .

)

(

t

) +

λ

(

E

)

(

t

)

Temporal

Embedding

3 Layer 1D

ConvNet

Intensity

MLP

Classifier

MLP

Temporal

Embedding

Mixture

Mean Aggr.

("; 	%; 	&)

t

(

n

)

t

(0)

t

(0)

\

t

(

n

)

t

(0)

∩

t

(

n

)

L

BCE

L

NLL

n

Case E: The set t ( E ) can be found by thinning t ( E ∪ F ) = t ( n ) \ t (0) and contains the points that were added by step n -1 and then kept at step n . The processes that generated t ( E ) and t ( F ) are two independent HPPs with intensities λ ( E ) = (1 -¯ α n -1 ) α n λ HPP and λ ( F ) = (1 -α n ) λ HPP , respectively, where λ ( E ) ( t ) is derived in a similar way to λ ( D ) ( t ) . Since t ( E ) and t ( F ) are independent HPPs and we know t ( E ∪ F ) , the number of points in t ( E ) follows a Binomial distribution with probability p = λ ( E ) λ ( E ) + λ ( F ) (see Appendix B.2 for details). That means we can sample t ( E ) given t ( n ) and t (0) by simply thinning t ( E ∪ F ) with probability 1 -p and express the intensity as a thinned sum of Dirac functions (cf., Figure 2).

For sequences of the training set, where t (0) is known, we can compute these intensities for all samples t ( n ) and reverse the forward process. However, t (0) is unknown when sampling new sequences. Therefore, similarly to the denoising diffusion approaches [18], in the next section, we show how to approximate the posterior intensity, given only t ( n ) . Further, in Section 3.4, we demonstrate how the trained neural network can be leveraged to sample new sequences t ∼ λ 0 .

3.3 Parametrization and training

In the previous section we have derived the intensity λ n -1 ( t | t (0) , t ( n ) ) of the posterior q ( t ( n -1) | t (0) , t ( n ) ) for the reverse process, i.e., the intensity of points at step n -1 given t ( n ) and t (0) . Now we want to approximate this posterior using a model p θ ( t ( n -1) | t ( n ) , n ) to learn to sample points t ( n -1) given only t ( n ) . As we are only missing information about t (0) we will learn to model λ ( B ) ( t ) and λ ( A ∪ C ) ( t ) to approximate ˆ t (0) ≈ t (0) (cf., Figure 3) for each n and then have access to the full posterior intensity from Section 3.2 to reverse the noising process.

Sequence embedding. To condition our model on t ( n ) and n we propose the following embeddings. We use a sinusoidal embedding [47] to embed the diffusion time n . Further, we encode each arrival time t i ∈ t ( n ) and inter-event time τ i = t i -t i -1 , with t 0 = 0 , to produce a temporal event embedding e i ∈ R d by applying a sinusoidal embedding [47]. Then we leverage a three-layered 1D-Convolutional Neural Network (CNN) with circular padding, dilation, and residual connections to compute a context embedding c i ∈ R d . Compared to attention and RNN-based encoders, the CNN is computationally more efficient and scales better with the length of event sequences while allowing us to capture long-range dependencies between the events. Finally, a global sequence embedding c is generated by a mean aggregation of the context embeddings.

Posterior approximation. The posterior defining t ( D ) is independent of t (0) and t ( n ) and can be sampled from directly. The set t ( B ) corresponds to those points that were kept from t (0) until the n -th step. Since all of these points are also included in t ( n ) we specify a classifier g θ ( e i , c i , n ) with an MLP that predicts which points from t ( n ) belong to t (0) . As t (0) is known during training, this is a standard classification setting. We use the binary cross entropy (BCE) loss L BCE to train g θ . Note that classifying t ( B ) from t ( n ) simultaneously predicts t ( n ) \ t (0) = t ( E ∪ F ) . Therefore we can subsequently attain t ( E ) by thinning t ( E ∪ F ) as explained in Section 3.2.

To sample points t ( C ) we have to specify the intensity function λ ( A ∪ C ) ( t ) that will be thinned to attain t ( C ) (cf., Section 3.2). As λ ( A ∪ C ) ( t ) is a mixture of Dirac functions we use an unnormalized mixture of H weighted and truncated Gaussian density functions f on [0 , T ] to parameterize the

inhomogeneous intensity:

where w j = Softplus(MLP w ([ n, c ])) , µ j = Sigmoid(MLP µ ([ n, c ])) and σ j = exp( -| MLP σ ([ n, c ]) | ) are parameterized by MLPs with two layers, a hidden dimension of d and a ReLU activation. Note that the Gaussian function is the standard approximation of the Dirac delta function and can, in the limit σ → 0 , perfectly approximate it. We include K , the number of points in t ( n ) , in the intensity to more directly model the number of events. Then λ ( A ∪ C ) θ ( t ) is trained to match samples t ( A ∪ C ) ∼ λ ( A ∪ C ) by minimizing the negative log-likelihood (NLL):

Thanks to the chosen parametrization, the integral term in L NLL can be efficiently computed in any deep-learning framework using the 'erf' function, without relying on Monte Carlo approximation. We present an overview of our model architecture to predict t (0) from t ( n ) in Figure 3.

Training objective. The full model is trained to minimize L = L NLL + L BCE . During training, we do not have to evaluate the true posterior or sample events from any of the posterior distributions. Instead, we can simply sample n and subsequently t ( n ) and minimize L for t ( n ) . Interestingly, in Appendix A, we show that L is equivalent to the Kullback-Leibler (KL) divergence between the approximate posterior p θ ( t ( n -1) | t ( n ) , n ) and the true posterior q ( t ( n -1) | t (0) , t ( n ) ) . Ultimately, this shows that optimizing the evidence lower bound (ELBO) of the proposed model boils down to simply learning a binary classification and fitting an inhomogeneous intensity.

3.4 Sampling

To sample an event sequence from our model, we start by sampling t ( N ) from an HPP with λ HPP . Subsequently, for each n ∈ [ N, · · · , 1] , ˆ t (0) is predicted by classifying ˆ t ( B ) and sampling ˆ t ( A ∪ C ) from λ ( A ∪ C ) θ ( t ) . Note that the Poisson distribution with intensity Λ ( A ∪ C ) ( T ) = ∫ T 0 λ ( A ∪ C ) θ ( t ) d t parameterizes the number of points in A ∪ C . Therefore, ˆ t ( A ∪ C ) can be sampled by first sampling the number of events and then sampling the event times from the normalized intensity λ ( A ∪ C ) θ ( t ) / Λ ( A ∪ C ) ( T ) . Given our predicted ˆ t (0) we can sample ˆ t ( n -1) from the posterior intensity defined in Section 3.2. By repeating this process, we produce a sequence of t ( n -1) s. Finally, we obtain a denoised sample t (0) by preˆ (1)

Algorithm 1: Sampling

dicting it from t . Weprovide an overview of the sampling procedure as pseudo-code in Algorithm 1.

3.5 Conditional sampling

The above-described process defines an unconditional generative model for event sequences on an interval [0 , T ] . For many (multi-step) forecasting applications, such as earthquake forecasting [10], we need to condition our samples on previous event sequences and turn our model into a conditional one that can generate future event sequences in [ H,H + T ] given the past observed events in [0 , H ] . To condition our generative model on a history, we apply a simple GRU encoder to encode the history into a d -dimensional history embedding h , which subsequently conditions the classifier and intensity model by being added to the diffusion time embedding.

4 Related work

Autoregressive neural TPPs. Most neural TPPs model the intensity or density of each event conditional on a history and consequently consist of two parts: a history encoder and an intensity/density decoder. As history encoders, RNNs [12, 41] and attention-based set encoders [50, 52] have been proposed. Attention-based encoders are postulated to better model long-range dependencies in the event sequences, but at the cost of a more complex encoder structure [43]. To decode the intensity λ ( t |H ) , density p ( t |H ) or the cumulative hazard function from the history, uni-modal distributions [12], mixture-distributions [41], a mixture of kernels [36, 44, 51], neural networks [37] and Gaussian diffusion [29] have been proposed. Another branch of neural TPPs models the event times conditional on a latent variable that follows a continuous-time evolution [5, 13, 16, 21], where, e.g., Hasan et al. [16] relate inter-event times of a TPP to the excursion of a stochastic process. In general, most neural TPPs are trained by maximizing the log-likelihood, but other training approaches have been proposed [26, 29, 49]. We want to highlight the difference of our model to two related works. TriTPP [42] learns a deterministic mapping between a latent HPP and a TPP using normalizing flows, which allows for parallel sampling. However, it models the conditional hazard function, which forces a conditional dependency of later arrival times and can still produce error accumulation. Lin et al. [29] proposed an autoregressive TPP model leveraging Gaussian diffusion to approximate the conditional density. Besides being autoregressive, the model does not directly model the number of points in the TPP but instead is trained to maximize the ELBO of the next inter-event time.

Non-autoregressive neural TPPs. An alternative to the conditional (autoregressive) modeling of TPPs is to apply a latent variable model that learns to relate entire point processes to latent variables. The class of Cox processes [7, 11, 19] models point processes through a hierarchy of latent processes under the assumption that higher-level latent variables trigger lower-level realizations. ADD-THIN can be considered to be a non-autoregressive latent variable model.

Denoising diffusion models. Recently, denoising diffusion models on continuous state spaces [18, 45] established the new state-of-the-art for many image applications [15, 18, 25]. Subsequently, diffusion models for other application domains such as point clouds [31, 32], physical simulations [23, 28, 30] and time-series [1, 3, 24, 46] emerged. While the majority of denoising diffusion models are based on Gaussian transition kernels in continuous state spaces proposed in [18, 45], a variety of diffusion models for discrete state spaces such as graphs [48], text [2, 27] and images [2, 6] have been presented. Here, we highlight the similarity of our work to the concurrent work of Chen and Zhou [6], who derived a diffusion process that models pixel values as a count distribution and thins them to noise images. In contrast to the related work on continuous and discrete state space diffusion models, ADD-THIN constitutes a novel diffusion model defined on a state space that captures both the discrete and continuous components of point processes.

5 Experiments

We evaluate the proposed model in two settings: density estimation and forecasting. In density estimation, the goal is to learn an unconditional model for event sequences. As for forecasting, the objective is to accurately predict the entire future event sequence given the observed past events.

Data. ADD-THIN is evaluated on 7 real-world datasets proposed by Shchur et al. [42] and 6 synthethic datasets from Omi et al. [37]. The synthetic datasets consist of Hawkes1 and Hawkes2 [17], a self-correcting (SC) [20], inhomogeneous Poisson process (IPP) and a stationary and a nonstationary renewal process (MRP, RP) [39, 11]. For the real-world datasets, we consider PUBG, Reddit-Comments, Reddit-Submissions, Taxi, Twitter, and restaurant check-ins in Yelp1 and Yelp2. We split each dataset into train, validation, and test set containing 60%, 20%, and 20% of the event sequences, respectively. Further dataset details and statistics are reported in Appendix C.

Baselines. We apply the RNN -based intensity-free TPP model from Shchur et al. [41]. Similar to Sharma et al. [40], we further combine the intensity-free model with an attention-based encoder from Zuo et al. [52] as a Transformer baseline. Additionally, we compare our model to an autoregressive TPP model with a continuous state Gaussian-diffusion [18] from Lin et al. [29], which we abbreviate as GD . Lastly, TriTPP [42] is used as a model that provides parallel but autoregressive sampling.

Training and model selection. We train each model by its proposed training loss using Adam [22]. For our model, we set the number of diffusion steps to N = 100 , apply the cosine betaschedule proposed in Glide [34], and set λ HPP = 1 for the noising process. We apply early stopping, hyperparameter tuning, and model selection on the validation set for each model. Further hyperparameter and training details are reported in Appendix D.

5.1 Sampling - Density estimation

A good TPP model should be flexible enough to fit event sequences from various processes. We evaluate the generative quality of the TPP models on 13 synthetic and real-world datasets by drawing 4000 TPP sequences from each model and computing distance metrics between the samples and event sequences from a hold-out test set. In short, the goal of this experiment is to show that our proposed model is a flexible TPP model that can generate event sequences that are 'close' to the samples from the data-generating distribution.

Metrics. In general, diffusion models cannot evaluate the exact likelihood. Instead, we evaluate the quality of samples by comparing the distributions of samples from each model and the test set with the maximum mean discrepancy measure (MMD) [14] as proposed by Shchur et al. [42]. Furthermore, we compute the Wasserstein distance [38] between the distribution of sequence lengths of the sampled sequences and the test set. We report the results on the test set averaged over five runs with different seeds.

Results. Table 1 presents the MMD results for all models and datasets. Among them, the RNN baseline demonstrates a strong performance across all datasets and outperforms both Transformer and GD . Notably, ADD-THIN exhibits competitive results with the autoregressive baseline, surpassing or matching ( ± 0 . 01 ) it on 11/13 datasets. Additionally, ADD-THIN consistently outperforms the Transformer and GD model on all datasets except SC and RP. Lastly, TriTPP performs well on most datasets but is outperformed or matched by our model on all but two datasets.

Table 2 shows the result for comparing the count distributions. Overall, the Wasserstein distance results align closely with the MMD results. However, the GD model is an exception, displaying considerably worse performance when focusing on the count distribution. This outcome is expected since the training of the GD model only indirectly models the number of events by maximizing the ELBO of each diffusion step to approximate the conditional density of the next event and not the likelihood of whole event sequences. Again, ADD-THIN shows a very strong performance, further emphasizing its expressiveness.

In summary, these results demonstrate the flexibility of our model, which can effectively capture various complex TPP distributions and matches the state-of-the-art performance in density estimation.

5.2 Conditional sampling - Forecasting

Forecasting event sequences from history is an important real-world application of TPP models. In this experiment, we evaluate the forecasting capability of our model on all real-world datasets. We evaluate each model's performance in forecasting the events in a forecasting window ∆ T , by randomly drawing a starting point T s ∈ [∆ T, T -∆ T ] . Then, the events in [0 , T s ] are considered the history, and [ T s , T s +∆ T ] is the forecasting time horizon.

In the experiment, we randomly sample 50 forecasting windows for each sequence from the test set, compute history embedding with each model's encoder, and then conditionally sample the forecast from each model. Note that TriTPP does not allow for conditional sampling and is therefore not part of the forecasting experiment.

Metrics. To evaluate the forecasting experiment, we will not compare distributions of TPPs but rather TPP instances. We measure the distance between two event sequences, i.e., forecast and ground truth data, by computing the distance between the count measures with the Wasserstein distance between two TPPs, as introduced by Xiao et al. [49]. Additionally, we report the mean absolute relative error (MAPE) between the predicted sequence length and ground truth sequence length in the forecasting horizon. We report the results on the test set averaged over five runs with different seeds.

Results. Table 3 presents the average Wasserstein distance between the predicted and ground truth forecast sequences. The results unequivocally demonstrate the superior performance of our model by forecasting entire event sequences, surpassing all autoregressive baselines on all datasets. Notably, the disparity between ADD-THIN and the baselines is more pronounced for datasets with a higher number of events per sequence, indicating the accumulation of prediction errors in the autoregressive models. Further, the transformer baseline achieves better forecasting results than the RNN baseline for some datasets with more events. This suggests that long-range attention can improve autoregressive forecasting and mitigate some error accumulation.

0

2

4

6

8

Time

t

0

20

40

Event count

N

(

t

)

observed

forecast

Table 4 reports the MAPE between the forecasted and ground truth sequence length. The MAPE results align consistently with the Wasserstein distance across all models and datasets. Figure 4

depicts the quantiles for 1000 forecasts generated by ADD-THIN for one Taxi event sequence and highlights the predictive capacity of our model. Overall, our model outperforms state-of-the-art TPP models in forecasting real-world event sequences.

6 Discussion

ADD-THIN vs. autoregressive TPP models. On a conceptual level, ADD-THIN presents a different trade-off compared to other TPP models: Instead of being autoregressive in event time, our model gradually refines the entire event sequence in parallel at every diffusion step to produce a sample from the learned data distribution. Thereby, we have found that our model is better suited for forecasting and modeling very long event sequences than autoregressive TPP models. Furthermore, the iterative refinement of the entire sequence allows us to leverage simple and shared layers to accurately model the long-range interaction between events and results in nearly constant sampling times across different sequence lengths (cf., Appendix E.3).

Limitations and future work. With ADD-THIN, we have derived a novel diffusion-inspired model for TPPs. Thereby, we focused on modeling the arrival times of the events and did not model continuous and discrete marks. However, we see this as an exciting extension to our framework, which might incorporate Gaussian diffusion [18] for continuous marks and discrete diffusion [2] for discrete marks. Further, while generative diffusion is known to produce high-quality samples, it also can be expensive. Besides tuning the number of diffusion steps, future work could focus on alternative and faster sampling routines [35]. Ultimately, we hope that by having connected diffusion models with TPPs, we have opened a new direction to modeling TPPs and broadened the field of diffusion-based models. Here, it would be especially interesting for us to see whether our framework could benefit other application domains in machine learning that involve sets of varying sizes, such as graph generation (molecules), point clouds, and spatial point processes.

7 Conclusion

By introducing ADD-THIN, we have connected the fields of diffusion models and TPPs and derived a novel model that naturally handles the discrete and continuous nature of point processes. Our model permits parallel and closed-form sampling of entire event sequences, overcoming common limitations of autoregressive TPP models. In our experimental evaluation, we demonstrated the flexibility of ADD-THIN, which can effectively capture complex TPP distributions and matches the state-of-the-art performance in density estimation. Additionally, in a long-term forecasting task on real-world data, our model distinctly outperforms the state-of-the-art TPP models by predicting entire forecasting windows non-autoregressively.

Broader impact

We see the proposed model as a general framework to model continuous-time event data. As such, our method can be applied to many fields, where common application domains include traffic, social networks, and electronic health records. We do not find any use cases mentioned above raise ethical concerns; however, it is essential to exercise caution when dealing with sensitive personal data.

References

J. M. L. Alcaraz and N. Strodthoff. Diffusion-based time series imputation and forecasting with structured state space models. arXiv preprint arXiv:2208.09399 , 2022.

J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34: 17981-17993, 2021.

M. Biloš, K. Rasul, A. Schneider, Y. Nevmyvaka, and S. Günnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In International Conference on Machine Learning (ICML) , 2023.

T. Bosser and S. B. Taieb. On the predictive accuracy of neural temporal point process models for continuous-time event data. Transactions on Machine Learning Research , 2023.

R. T. Chen, B. Amos, and M. Nickel. Neural spatio-temporal point processes. arXiv preprint arXiv:2011.04583 , 2020.

T. Chen and M. Zhou. Learning to jump: Thinning and thickening latent counts for generative modeling. arXiv preprint arXiv:2305.18375 , 2023.

D. R. Cox. Some statistical methods connected with series of events. Journal of the Royal Statistical Society: Series B (Methodological) , 17(2):129-157, 1955.

D. Daley and D. Vere-Jones. An Introduction to the Theory of Point Processes: Volume I: Elementary Theory and Methods . Probability and Its Applications. Springer New York, 2006.

D. J. Daley and D. Vere-Jones. An introduction to the theory of point processes: volume II: general theory and structure . Springer Science & Business Media, 2007.

K. Dascher-Cousineau, O. Shchur, E. E. Brodsky, and S. Günnemann. Using deep learning for flexible and scalable earthquake forecasting. Geophysical Research Letters , 50(17), 2023.

C. DR and I. COLL. The statistical analysis of dependencies in point processes. Stochastic Point Processes. Wiley: New York , pages 55-66, 1972.

N. Du, H. Dai, R. Trivedi, U. Upadhyay, M. Gomez-Rodriguez, and L. Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1555-1564, 2016.

J. Enguehard, D. Busbridge, A. Bozson, C. Woodcock, and N. Hammerla. Neural temporal point processes for modelling electronic health records. In Machine Learning for Health , pages 85-113. PMLR, 2020.

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

X. Han, H. Zheng, and M. Zhou. Card: Classification and regression diffusion models. arXiv preprint arXiv:2206.07275 , 2022.

A. Hasan, Y. Chen, Y. Ng, M. Abdelghani, A. Schneider, and V. Tarokh. Inference and sampling of point processes from diffusion excursions. In The 39th Conference on Uncertainty in Artificial Intelligence , 2023.

A. G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika , 58(1):83-90, 1971.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Neural Information Processing Systems (NeurIPS) , 2020.

C. Hong and C. Shelton. Deep neyman-scott processes. In Proceedings of the 25th International Conference on Artificial Intelligence and Statistics , volume 151, pages 3627-3646. PMLR, 2022.

V. Isham and M. Westcott. A self-correcting point process. Stochastic processes and their applications , 8(3):335-347, 1979.

J. Jia and A. R. Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems , 32, 2019.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

G. Kohl, L.-W. Chen, and N. Thuerey. Turbulent flow simulation using autoregressive conditional diffusion models, 2023.

M. Kollovieh, A. F. Ansari, M. Bohlke-Schneider, J. Zschiegner, H. Wang, and Y. Wang. Predict, refine, synthesize: Self-guiding diffusion models for probabilistic time series forecasting. arXiv preprint arXiv:2307.11494 , 2023.

H. Li, Y. Yang, M. Chang, S. Chen, H. Feng, Z. Xu, Q. Li, and Y. Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing , 479:47-59, 2022.

S. Li, S. Xiao, S. Zhu, N. Du, Y. Xie, and L. Song. Learning temporal point processes via reinforcement learning. Advances in neural information processing systems , 31, 2018.

X. Li, J. Thickstun, I. Gulrajani, P. S. Liang, and T. B. Hashimoto. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems , 35:43284343, 2022.

M. Lienen, D. Lüdke, J. Hansen-Palmus, and S. Günnemann. From zero to turbulence: Generative modeling for 3d flow simulation, 2023.

H. Lin, L. Wu, G. Zhao, L. Pai, and S. Z. Li. Exploring generative neural temporal point process. Transactions on Machine Learning Research , 2022.

P. Lippe, B. S. Veeling, P. Perdikaris, R. E. Turner, and J. Brandstetter. Pde-refiner: Achieving accurate long rollouts with neural pde solvers. arXiv preprint arXiv:2308.05732 , 2023.

S. Luo and W. Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Z. Lyu, Z. Kong, X. Xu, L. Pan, and D. Lin. A conditional point diffusion-refinement paradigm for 3d point cloud completion. arXiv preprint arXiv:2112.03530 , 2021.

H. Mei and J. M. Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In Neural Information Processing Systems (NeurIPS) , 2017.

A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

M. Okawa, T. Iwata, T. Kurashima, Y. Tanaka, H. Toda, and N. Ueda. Deep mixture point processes: Spatio-temporal event prediction with rich contextual information. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 373-383, 2019.

T. Omi, K. Aihara, et al. Fully neural network based model for general temporal point processes. Advances in neural information processing systems , 32, 2019.

A. Ramdas, N. García Trillos, and M. Cuturi. On wasserstein two-sample testing and related families of nonparametric tests. Entropy , 19(2):47, 2017.

B. Sevast'yanov. Renewal theory. Journal of Soviet Mathematics , 4(3):281-302, 1975.

K. Sharma, Y. Zhang, E. Ferrara, and Y. Liu. Identifying coordinated accounts on social media through hidden influence and group behaviours. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1441-1451, 2021.

O. Shchur, M. Biloš, and S. Günnemann. Intensity-free learning of temporal point processes. In International Conference on Learning Representations (ICLR) , 2020.

O. Shchur, N. Gao, M. Biloš, and S. Günnemann. Fast and flexible temporal point processes with triangular maps. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

O. Shchur, A. C. Türkmen, T. Januschowski, and S. Günnemann. Neural temporal point processes: A review. arXiv preprint arXiv:2104.03528 , 2021.

A. Soen, A. Mathews, D. Grixti-Cheng, and L. Xie. Unipoint: Universally approximating point processes intensities. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 9685-9694, 2021.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML) , pages 2256-2265, 2015.

Y. Tashiro, J. Song, Y . Song, and S. Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems , 34: 24804-24816, 2021.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

C. Vignac, I. Krawczuk, A. Siraudin, B. Wang, V. Cevher, and P. Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734 , 2022.

S. Xiao, M. Farajtabar, X. Ye, J. Yan, L. Song, and H. Zha. Wasserstein learning of deep generative point process models. Advances in neural information processing systems , 30, 2017.

Q. Zhang, A. Lipani, O. Kirnap, and E. Yilmaz. Self-attentive hawkes process. In International conference on machine learning , pages 11183-11193. PMLR, 2020.

W. Zhang, T. Panum, S. Jha, P. Chalasani, and D. Page. Cause: Learning granger causality from event sequences using attribution methods. In International Conference on Machine Learning , pages 11235-11245. PMLR, 2020.

S. Zuo, H. Jiang, Z. Li, T. Zhao, and H. Zha. Transformer hawkes process. arXiv preprint arXiv:2002.09291 , 2020.

A On the relationship between the applied loss and the ELBO

In the following section, we investigate the relationship between our applied loss function and the ELBO to the unknown data distribution derived for diffusion models. The ELBO of diffusion models [18] is given as follows:

Additionally, the Janossi density [8] of an event sequence t on [0 , T ] allows us to represent each element of the ELBO in terms of our derived inhomogeneous (approximate) posterior intensities (Section 3.2 and Figure 2) as follows:

L N : L N is constant as the intensity λ HPP defining q ( t ( N ) | t (0) ) and p ( t ( N ) ) has no learnable parameters.

L 0 : We directly train our model to optimize this likelihood term as described in Section 3.3.

L n : The KL divergence between two densities is defined as:

where only the right-hand side relies on θ , letting us minimize the KL divergence by maximizing the expectation over the log-likelihood log( p θ ( t ( n -1) | t ( n ) )) .

To add some additional context to the KL divergence in L n and similar to the derivation of the posterior in Section 3.2, we will further distinguish three cases:

Case B & E: q ( t ( n -1) | t (0) , t ( n ) ) and p θ ( t ( n -1) | t ( n ) ) are defined by Bernoulli distributions over each element of t ( n ) . By definition the cross-entropy H ( q, p ) of the distribution p relative to q is given by H ( q, p ) = H ( q ) + D KL ( q ∥ p ) , where H ( q ) is the entropy and D KL the KL divergence. We can see that minimizing the (binary) cross-entropy is equivalent to minimizing the KL divergence, as the entropy H ( q ) is constant for the data distribution.

Case D: Our parametrization uses the same intensity function for q ( t ( n -1) | t (0) , t ( n ) ) and p θ ( t ( n -1) | t ( n ) ) , which does not rely on any learned parameters.

Case C: Minimizing this KL divergence by maximizing the E q [ log( p θ ( t ( n -1) | t ( n ) )) ] is proposed in Section 3.2 to learn λ ( A ∪ C ) θ ( t ) . Note that q ( t ( n -1) | t (0) , t ( n ) ) is sampled from by independently thinning t (0) \ t ( n ) . Consequently, by minimizing the NLL of our intensity λ ( A ∪ C ) θ ( t ) with regards to t (0) \ t ( n ) , we optimize the expectation in closed form.

B Derivations

B.1 Direct forward sampling

Proof. We first repeat Equation 2:

Then for the first step we can write:

Assuming Equation 3 holds for step n -1 :

we can write for step n :

which completes the proof by induction.

B.2 Conditional distribution of Poisson variables

Proposition. Given two independent random variables X 1 ∼ Poisson ( λ 1 ) , X 2 ∼ Poisson ( λ 2 ) , X 1 | X 1 + X 2 = k is Binomial distributed, i.e., X 1 | X 1 + X 2 = k ∼ Binomial ( x 1 ; k, λ 1 λ 1 + λ 1 ) .

Proof. The Poisson distributed random variables X 1 and X 2 have the following joint probability mass function:

which further defines the joint probability mass function of P ( X 1 = x 1 , X 2 = x 2 , Y = k ) if x 1 + x 2 = k . Additionally, it is well know that Y = X 1 + X 2 is a Poisson random variable with intensity λ 1 + λ 2 and therefore P ( Y = k ) = e -( λ 1 + λ 2 ) ( λ 1 + λ 2 ) k k ! . Then P ( X 1 = x 1 | Y = k ) is given by the following:

where we have leveraged x 2 = k -x 1 and λ 2 ( λ 1 + λ 2 ) = 1 -λ 1 ( λ 1 + λ 2 ) . As we have shown P ( X 1 = x 1 | Y = k ) follows the Binomial distribution with p = λ 1 ( λ 1 + λ 2 ) .

±

±

C Datasets

Synthetic datasets. The six synthethic dataset were sampled by Shchur et al. [42] following the procedure in Section 4.1 of Omi et al. [37] and consist of 1000 sequences on the interval [0 , 100] .

Real-world datasets. The seven real-world datasets were proposed by Shchur et al. [42] and consist of PUBG, Reddit-Comments, Reddit-Submissions, Taxi, Twitter, Yelp1, and Yelp2. The event sequences of PUBG represent the death of players in a game of Player Unknown's Battleground (PUBG). The event sequences of Reddit-Comments represent the comments on the askscience subreddit within 24 hours after opening the discussion in the period from 01.01.2018 until 31.12.2019. The event sequences of Reddit-Submissions represent the discussion submissions on the politics subreddit within a day in the period from 01.01.2017 until 31.12.2019. The event sequences of Taxi represent taxi pick-ups in the south of Manhattan, New York. The event sequences of Twitter represent tweets by user 25073877. The event sequences of Yelp1 represent check-ins for the McCarran International Airport recorded for 27 users in 2018. The event sequences of Yelp2 represent check-ins for all businesses in the city of Mississauga recorded for 27 users in 2018.

We report summary statistics on the datasets in Table 5 and 6. Lately, the validity of some of the widely used real-world benchmark datasets was criticized [4]. In one-step-ahead prediction tasks with teacher forcing, very simple architectures achieved similar results to some of the more advanced ones. However, this seems to be more of a problem of the task than the datasets. In our work, we consider different tasks (density estimation and long-term forecasting) and metrics and have found significant empirical differences between the baselines on these datasets.

D Experimental set-up

All models but the transformer baseline were trained on an Intel Xeon E5-2630 v4 @ 2.20 GHz CPU with 256GB RAM and an NVIDIA GeForce GTX 1080 Ti. Given its RAM requirement, the transformer baseline had to be trained with batch size 32 on an NVIDIA A100-PCIE-40GB for the Reddit-C and Reddit-S datasets.

Hyperparameter tuning. has been applied to all models. The hyperparameter tuning was done on the MMD loss between 1000 samples from the model and the validation set. We use a hidden dimension of 32 for all models. Further, we have tuned the learning rate in { 0 . 01 , 0 . 001 } for all models, the number of mixture components in { 8 , 16 } for ADD-THIN, RNN and Transformer , the number of knots in { 10 , 20 } for TriTPP and the number of attention layers in { 2 , 3 } for the transformer baseline. The values of all other baseline hyperparameters were set to the recommended

values given by the authors. Further, the GD baseline has been trained with a batch size of 16, as recommended by the authors. For the forecasting task, we apply the optimal hyperparameters from the density estimation experiment.

Early-stopping. Each model has been trained for up to 5000 epochs with early stopping on the MMDmetric on the validation set for the density estimation task and on the Wasserstein distance metric on the validation set for the forecasting task.

E Additional results

E.1 Density estimation results with standard deviations

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

±

E.2 Forecasting results with standard deviations

±

±

±

±

±

±

±

±

E.3 Sampling runtimes

We compare sampling runtimes on an NVIDIA GTX 1080 Ti across the different models in Figure 5. ADD-THIN maintains near-constant runtimes by refining the entire sequence in parallel. The autoregressive baselines RNN and Transformer show increasing runtimes, with longer sequences surpassing ADD-THIN's runtime. TriTPP is a highly optimized baseline computing the autoregressive interactions between event times in parallel by leveraging triangular maps, resulting in the fastest runtimes. Lastly, GD is autoregressive in event time and gradually refines each event time over 100 diffusion steps, leading to the worst runtimes.

10 2

10 3

Average sequence length

10

2

10

1

10 0

10 1

10 2

Runtime (seconds)

RNN

Trans

TriTPP

ADD-THIN

±

±

±

±

±

±

Peer reviews:
comments: SUMMARY:
The submission derive a probabilistic diffusion model for TPPs. By proposing the Add-Thin framework, the proposed method can naturally handles the continuous and discrete nature of point processes and directly models the whole event sequences. Compared with the traditional autoregressive approaches for the temporal point process (TPP), the proposed framework is immune to the accumulation of errors caused by sequential sampling.

STRENGTHS:
The submission proposed a framework that learns the probabilistic mapping from complete noise, i.e., a homogeneous Poisson process (HPP), to data. More specifically, a model is learned to reverse the noising process of adding (superposition) and removing (thinning) points from the event sequence by matching the conditional inhomogeneous denoising intensity. This idea is very interesting.

WEAKNESSES:
I do not find any obvious weaknesses.

QUESTIONS:
None.

LIMITATIONS:
The authors discussed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe forward process adds points from homogeneous poisson process (HPP) into the sequence and removes points from original sequence ($\\mathbf{t}^{(0)}$). The goal is such that $\\mathbf{t}^{(n)}$ is HPP. The neural network is trained to approximate missing information about $\\mathbf{t}^{(0)}$, e.g. figure out which points from HPP came from $\\mathbf{t}^{(0)}$ and which points were removed at intermediate steps. The thinning and superposition theorems were used multiple times to derive intensities of various sets, where the sets categorize the points.\n\nSTRENGTHS:\n1. The adding and thinning approach, training via classification and matching, to modeling and generating temporal point processes is original and interesting.\n\nWEAKNESSES:\n1. The writing and notation is extremely confusing. I read the paper multiple times.\n\n1a. e.g. please clearly specify the observation, is it a time sequence $x(t), t\\in[0,T]$ where $x(t) = 0$ for no event, $x(t) = 1$ for event, discretized in time. or is it a varying length sequence $\\mathbf{t}=(t_1,...,t_K), 0< t_1<...<t_K\\leq T$ with $K$ arrival times. also, section 3.3 writes about sequence embedding to get an input in $R^d$ but I am still confused.\n\n1b. e.g. what are the sets A and F? what are the differences between B and C, D and E? B seems to be the points preserved from $\\mathbf{t}^{(0)}$ all the way to HPP, C seems to be the points preserved from $\\mathbf{t}^{(0)}$ but removed in the next step. is there a clearer way of writing the material in section 3.2?\n\n1c. perhaps the background section can be shortened if we need more space to explain the method clearly.\n\n1d. would potentially increase the score if the model and method is clearly explained.\n\nQUESTIONS:\n1. Would like to introduce this recent other work that connects point processes and diffusions. in this case, the interarrival times of a point process is related to the length of a diffusion excursion. https://openreview.net/forum?id=KIlwyX7nCi\n\n2. What is the benefit of applying the proposed model compared to other methods of generating sequences?\n\nLIMITATIONS:\n1. Please address the potential limitations of using a neural network to approximate missing information about $\\mathbf{t}^{(0)}$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a probabilistic diffusion model for TPPs, ADD-THIN, that naturally handles the continuous and discrete nature of point processes and directly models whole event sequences. While autoregressive methods are expressive in modeling event sequences, ADD-THIN does not suffer from the accumulation of errors caused by sequential sampling for long-term forecasting applications. The proposed method matches the performance of state-of-the-art models in density estimation and outperform them for forecasting on both synthetic and real-world datasets.\n\nSTRENGTHS:\n- To the best of my knowledge, this work represents the first attempt to combine diffusion models and Temporal Point Processes. The potential of generating complete sequences without relying on auto-regressive mechanisms presents a promising avenue for future research. It is worth noting that this paper shares similarities with the unpublished work available at: https://slideslive.com/embed/presentation/38922857.\n\n- The proposed method is compared to multiple baselines for both unconditional and conditional sampling (forecasting).\n\nWEAKNESSES:\n- The paper is poorly written and hard to read.\n\n- There are multiple statements that are either inaccurate or without references\n\t- \"The intensity of a TPP describes the expected number of points\"\n\t- Line 65-70, you either need to prove it or provide references that do so. \n\t- Line 72, \"the expected number of points on [0, T] follows a Poisson distribution with rate ...\" \n\t- \"Even though Poisson processes can model seasonality\"\t\n\t\t- How does a Poisson model seasonality?\n\t- \"We know that conditioned on ... is a random sequence of events with a density that can be completely described by its (unconditional) intensity function\"\n\t- \"Since the sets are disjoint, ...\"\n\t- Is there a difference between a counting process and measure? an intensity function and measure? You use these terms interchangeably.\n\n\n- Multiple TPP datasets that are used in the experiments have been shown to be inappropriate benchmarks for neural TPPs. See https://openreview.net/forum?id=3OSISBQPrM\n\n- Overall, the presentation of the paper could be greatly improved, especially Section 3. Specifically, it feels like adding more formal notations (e.g. for Section 3.2.) would allow to shorten the text, while helping the reader more easily grasp the idea behind the method. For instance, why defer to the Appendix the pseudo-code for the sampling procedure?  \n\nAdditionally, several design choices are not clearly motivated, while the experiments and evaluation of the proposed model falls a bit short of expectations. These concerns are summarized in the points below: \n\n1) Why include both $\\boldsymbol{e}i$ and $\\boldsymbol{c}_i$ in the parametrization of $g\\theta$? Are the $\\boldsymbol{c}_i$ constructed from neighboring events? Otherwise, it looks like there is redundancy between the information contained in the two embeddings. \n\n2) There is no real motivation behind the use of a positional encoding for $n$. Why do you need it?  \n\n3) What is the motivation for using an unormalized mixture of Gaussians to parametrize $\\lambda^{(A \\cup C)}$? Have you considered alternative parametrizations that can be found in the neural TPP literature, e.g. an unmarked version of RMTPP [1]? Additionally, more details regarding the architectures employed would be appreciated (e.g. what is $\\sigma$, and what is the model behind the MLP on page 5?). \n\n4) In the NLL objective of equation 5, is the objective solely computed on times from $A \\cup C$? If yes, this should be indicated explicitly in the objective.\n\n5) On page 6, it is not clear how you find that $\\Lambda^{(A \\cup C)}(T) = K \\sum_{i}^H w_i$. Given your definition of $\\lambda^{(A \\cup C)}(t)$, I would have expected this result to hold only if the integral was taken over $-\\infty, \\infty$. \n\n6) The paragraph on 'conditional sampling' requires more detailed explanations, as it is impossible to see from the text how you handle this scenario. Specifically, how is $\\boldsymbol{h}$ integrated into $g_{\\theta}$ and $\\lambda^{(A \\cup C)}(t)$? \n\n6) As $\\hat{t}^{(0)}$ is successively approximated at each time step $n$ during the reverse process, how can you ensure that the propagation of errors is mitigated and won't affect significantly the generated sequences after $N$ steps? \n\n7) Why only reporting the MAPE between the generated and truth sequences lengths? Isn't there a way to evaluate the quality of the generated samples within a sequence itself (e.g. using a measure of distance)? \n\n[1] Du, Nan and Dai, Hanjun and Trivedi, Rakshit and Upadhyay, Utkarsh and Gomez-Rodriguez, Manuel and Song, Le. (2016). Recurrent Marked Temporal Point Processes: Embedding Event History to Vector. SIGKDD.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nThe authors have a section on some limitations of their work, although a more in-depth discussion would have been appreciated. Specifically, they mention that sampling from their diffusion model can be quite expensive, but they do not provide computional trade-offs in terms of time/resources across baselines.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel probabilistic diffusion framework for temporal point process. Its significance lies modeling a whole event sequence directly, overcoming common limitations of autoregressive models.\n\nSTRENGTHS:\nOriginality:  This paper is very novel. It connects diffusion models with TPPs and model a whole event sequence , overcoming issues from autoregressive models.\n\nQuality: The formulation of the approach and method itself are sound by leveraging two properties of point process: thinning and superpositions. The discussion of the reverse process are reasonable by discussing 4 cases of transition from nth step to n-1 step.  The empirical results demonstrate the good performance of the proposed approach, esp. in forecasting tasks. \n\nClarity:  The paper is overall well-presented. Improvement can be made by better explaining A,B,C,D,E,F sets. \n\nSignificance:  The paper introduces a new way to model event sequences probabilistically  which is appealing and significant to the TPP community.\n\nWEAKNESSES:\nA.\tExperiments. The experimental evaluation on density estimation. The results of Add and Thin is only comparable to Intensity-free model by Shchur (RNN in the paper), if not worse.\nB.\tApplication and Limitation. One minor limitation is it only deals with TPP without marks. It would be interested in see something for Marked TPP.\nC.\tOverall presentation. It took me a while to understanding the notation of sets A-F. It would be nice to include such information in the caption of figure 2. \nD. No codes are given, I am not sure how reproducible the results are.\n\nQUESTIONS:\nA.\tPosterior approximation.  How good is the posterior approximation in Add and Thin? Are there any theoretical insights. \nB.\tNeural network. Could the authors provide some movitation of using 3-layered-d CNN?\nC.\tLoss. 1). Is the mixture of Gaussians eqn. (4) used to model the conditional intensity good for modeling an event time(or interevent time) t which is positive? Add and Thin is a latent variable model with latent variables t^{n-1}’s. Usually int the diffusion models reference 40 in the paper, latent variable models leverage variational inference to maximize the evidence lower bounds. But this model is not. Could the author shed some insights on this? \nD.\tCox process models. Do the authors have any experiment results with these baselines form reference [5,8,17]? \nE.\tExperiments. How many diffusion steps (n) are used?\n\nLIMITATIONS:\nThe limit of the paper according to the authors is it is limited to TPP without marks. In addition, current sampling routines may not be efficient. TPP models can be applied applied to many fields, where common application domains  include traffic, social networks, and electronic health records.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-17"}