SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To sample an event sequence from our model, we start by sampling  $\pmb{t}^{(N)}$  from an HPP with  $\lambda_{\mathrm{HPP}}$ . Subsequently, for each  $n \in [N,\dots ,1]$ ,  $\hat{\pmb{t}}^{(0)}$  is predicted by classifying  $\hat{\pmb{t}}^{(B)}$  and sampling  $\hat{\pmb{t}}^{(A\cup C)}$  from  $\lambda_{\pmb{\theta}}^{(A\cup C)}(t)$ . Note that the Poisson distribution with intensity  $\Lambda^{(A\cup C)}(T) = \int_0^T\lambda_\theta^{(A\cup C)}(t)\mathrm{d}t$  parameterizes the number of points in  $A \cup C$ . Therefore,  $\hat{\pmb{t}}^{(A\cup C)}$  can be sampled by first sampling the number of events and then sampling the event times from the normalized intensity  $\lambda_{\pmb{\theta}}^{(A\cup C)}(t) / \Lambda^{(A\cup C)}(T)$ . Given our predicted  $\hat{\pmb{t}}^{(0)}$  we can sample  $\hat{\pmb{t}}^{(n - 1)}$  from the posterior intensity defined in Section 3.2. By repeating this process, we produce a sequence of  $t^{(n - 1)}s$ . Finally, we obtain a denoised sample  $t^{(0)}$  by pre
Algorithm 1: Sampling
$\pmb{t}^{(n = N)}\sim \lambda_{\mathrm{HPP}}$
for  $n\in \{N,\ldots ,1\}$  do
$$
\text {s a m p l e} \hat {\boldsymbol {t}} ^ {(B)} \sim \sum_ {t _ {i} \in \boldsymbol {t} ^ {(n)}} g _ {\theta} \left(t _ {i} \mid \boldsymbol {e} _ {i}, \boldsymbol {c} _ {i}, n\right) \delta_ {t _ {i}} (t);
$$
$$
\text {s a m p l e} \hat {\boldsymbol {t}} ^ {(C)} \sim \frac {\bar {\alpha} _ {n - 1} - \bar {\alpha} _ {n}}{1 - \bar {\alpha} _ {n}} \lambda_ {\theta} ^ {(A \cup C)} (t \mid \boldsymbol {t} ^ {(n)}, n);
$$
$$
\text {s a m p l e} \hat {\boldsymbol {t}} ^ {(D)} \sim (1 - \bar {\alpha} _ {n - 1}) (1 - \alpha_ {n}) \lambda_ {\mathrm {H P P}};
$$
$$
\text{sample}\hat{\boldsymbol{t}}^{(E)}\sim \sum_{t_{i}\in \boldsymbol{t}^{(n)}\setminus \hat{\boldsymbol{t}}^{(B)}}\frac{\alpha_{n} - \bar{\alpha}_{n}}{1 - \bar{\alpha}_{n}}\delta_{t_{i}}(t);
$$
$$
\boldsymbol {t} ^ {(n - 1)} \leftarrow \hat {\boldsymbol {t}} ^ {(B)} \cup \hat {\boldsymbol {t}} ^ {(C)} \cup \hat {\boldsymbol {t}} ^ {(D)} \cup \hat {\boldsymbol {t}} ^ {(E)};
$$
end sample  $\hat{\pmb{t}}^{(A\cup C)}\sim \lambda_{\theta}^{(A\cup C)}(t\mid \pmb{t}^{(1)},1);$
$$
\text {s a m p l e} \hat {\boldsymbol {t}} ^ {(B)} \sim \sum_ {t _ {i} \in \boldsymbol {t} ^ {(1)}} g _ {\theta} \left(t _ {i} \mid \boldsymbol {e} _ {i}, \boldsymbol {c} _ {i}, 1\right) \delta_ {t _ {i}} (t);
$$
$$
\boldsymbol {t} \leftarrow \hat {\boldsymbol {t}} ^ {(B)} \cup \hat {\boldsymbol {t}} ^ {(A \cup C)};
$$
return  $t$
dicting it from  $\hat{t}^{(1)}$ . We provide an overview of the sampling procedure as pseudo-code in Algorithm 1.
3.5 Conditional sampling
The above-described process defines an unconditional generative model for event sequences on an interval  $[0, T]$ . For many (multi-step) forecasting applications, such as earthquake forecasting [10], we need to condition our samples on previous event sequences and turn our model into a conditional one that can generate future event sequences in  $[H, H + T]$  given the past observed events in  $[0, H]$ . To condition our generative model on a history, we apply a simple GRU encoder to encode the history into a  $d$ -dimensional history embedding  $h$ , which subsequently conditions the classifier and intensity model by being added to the diffusion time embedding.
Autoregressive neural TPPs. Most neural TPPs model the intensity or density of each event conditional on a history and consequently consist of two parts: a history encoder and an intensity/density decoder. As history encoders, RNNs [12, 41] and attention-based set encoders [50, 52] have been proposed. Attention-based encoders are postulated to better model long-range dependencies in the event sequences, but at the cost of a more complex encoder structure [43]. To decode the intensity  $\lambda(t|\mathcal{H})$ , density  $p(t|\mathcal{H})$  or the cumulative hazard function from the history, uni-modal distributions [12], mixture-distributions [41], a mixture of kernels [36, 44, 51], neural networks [37] and Gaussian diffusion [29] have been proposed. Another branch of neural TPPs models the event times conditional on a latent variable that follows a continuous-time evolution [5, 13, 16, 21], where, e.g., Hasan et al. [16] relate inter-event times of a TPP to the excursion of a stochastic process. In general, most neural TPPs are trained by maximizing the log-likelihood, but other training approaches have been proposed [26, 29, 49]. We want to highlight the difference of our model to two related works. TriTPP [42] learns a deterministic mapping between a latent HPP and a TPP using normalizing flows, which allows for parallel sampling. However, it models the conditional hazard function, which forces a conditional dependency of later arrival times and can still produce error accumulation. Lin et al. [29] proposed an autoregressive TPP model leveraging Gaussian diffusion to approximate the conditional density. Besides being autoregressive, the model does not directly model the number of points in the TPP but instead is trained to maximize the ELBO of the next inter-event time.
Non-autoregressive neural TPPs. An alternative to the conditional (autoregressive) modeling of TPPs is to apply a latent variable model that learns to relate entire point processes to latent variables. The class of Cox processes [7, 11, 19] models point processes through a hierarchy of latent processes under the assumption that higher-level latent variables trigger lower-level realizations. ADD-THIN can be considered to be a non-autoregressive latent variable model.
Denoising diffusion models. Recently, denoising diffusion models on continuous state spaces [18, 45] established the new state-of-the-art for many image applications [15, 18, 25]. Subsequently, diffusion models for other application domains such as point clouds [31, 32], physical simulations [23, 28, 30] and time-series [1, 3, 24, 46] emerged. While the majority of denoising diffusion models are based on Gaussian transition kernels in continuous state spaces proposed in [18, 45], a variety of diffusion models for discrete state spaces such as graphs [48], text [2, 27] and images [2, 6] have been presented. Here, we highlight the similarity of our work to the concurrent work of Chen and Zhou [6], who derived a diffusion process that models pixel values as a count distribution and thins them to noise images. In contrast to the related work on continuous and discrete state space diffusion models, ADD-THIN constitutes a novel diffusion model defined on a state space that captures both the discrete and continuous components of point processes.
We evaluate the proposed model in two settings: density estimation and forecasting. In density estimation, the goal is to learn an unconditional model for event sequences. As for forecasting, the objective is to accurately predict the entire future event sequence given the observed past events.
Data. ADD-THIN is evaluated on 7 real-world datasets proposed by Shchur et al. [42] and 6 synthetic datasets from Omi et al. [37]. The synthetic datasets consist of Hawkes1 and Hawkes2 [17], a self-correcting (SC) [20], inhomogeneous Poisson process (IPP) and a stationary and a nonstationary renewal process (MRP, RP) [39, 11]. For the real-world datasets, we consider PUBG, Reddit-Comments, Reddit-Submissions, Taxi, Twitter, and restaurant check-ins in Yelp1 and Yelp2. We split each dataset into train, validation, and test set containing  $60\%$ ,  $20\%$ , and  $20\%$  of the event sequences, respectively. Further dataset details and statistics are reported in Appendix C.
Baselines. We apply the RNN-based intensity-free TPP model from Shchur et al. [41]. Similar to Sharma et al. [40], we further combine the intensity-free model with an attention-based encoder from Zuo et al. [52] as a Transformer baseline. Additionally, we compare our model to an autoregressive TPP model with a continuous state Gaussian-diffusion [18] from Lin et al. [29], which we abbreviate as  $GD$ . Lastly, TriTPP [42] is used as a model that provides parallel but autoregressive sampling.
Table 1: MMD  $(\downarrow)$  between the TPP distribution of sampled sequences and hold-out test set (bold best, underline second best). The results with standard deviation are reported in Appendix E.
Hawkes1	Hawkes2	SC	IPP	RP	MRP	PUBG	Reddit-C	Reddit-S	Taxi	Twitter	Yelp1	Yelp2
RNN	0.02	0.01	0.08	0.05	0.01	0.03	0.04	0.01	0.02	0.04	0.03	0.07	0.03
Transformer	0.03	0.04	0.19	0.10	0.02	0.19	0.06	0.05	0.09	0.09	0.08	0.12	0.14
GD	0.06	0.06	0.13	0.08	0.05	0.14	0.11	0.03	0.03	0.10	0.15	0.12	0.10
TriTPP	0.03	0.04	0.23	0.04	0.02	0.05	0.06	0.09	0.12	0.07	0.04	0.06	0.06
ADD-THIN	0.02	0.02	0.19	0.03	0.02	0.10	0.03	0.01	0.02	0.04	0.04	0.08	0.04
Table 2: Wasserstein distance (↓) between the distribution of the number of events of sampled sequences and hold-out test set (bold best, underline second best). The results with standard deviation are reported in Appendix E.
Hawkes1	Hawkes2	SC	IPP	RP	MRP	PUBG	Reddit-C	Reddit-S	Taxi	Twitter	Yelp1	Yelp2
RNN	0.03	0.01	0.00	0.02	0.02	0.01	0.02	0.01	0.05	0.02	0.01	0.04	0.02
Transformer	0.06	0.04	0.06	0.07	0.04	0.11	0.04	0.08	0.11	0.13	0.05	0.11	0.21
GD	0.16	0.13	0.50	0.42	0.28	0.50	0.54	0.02	0.16	0.33	0.07	0.26	0.25
TriTPP	0.03	0.03	0.01	0.01	0.02	0.03	0.03	0.09	0.09	0.04	0.01	0.03	0.04
ADD-THIN	0.04	0.02	0.08	0.01	0.02	0.04	0.02	0.03	0.04	0.03	0.01	0.04	0.02
Training and model selection. We train each model by its proposed training loss using Adam [22]. For our model, we set the number of diffusion steps to  $N = 100$ , apply the cosine betaschedule proposed in Glide [34], and set  $\lambda_{\mathrm{HPP}} = 1$  for the noising process. We apply early stopping, hyperparameter tuning, and model selection on the validation set for each model. Further hyperparameter and training details are reported in Appendix D.
5.1 Sampling - Density estimation
A good TPP model should be flexible enough to fit event sequences from various processes. We evaluate the generative quality of the TPP models on 13 synthetic and real-world datasets by drawing 4000 TPP sequences from each model and computing distance metrics between the samples and event sequences from a hold-out test set. In short, the goal of this experiment is to show that our proposed model is a flexible TPP model that can generate event sequences that are 'close' to the samples from the data-generating distribution.
Metrics. In general, diffusion models cannot evaluate the exact likelihood. Instead, we evaluate the quality of samples by comparing the distributions of samples from each model and the test set with the maximum mean discrepancy measure (MMD) [14] as proposed by Shchur et al. [42]. Furthermore, we compute the Wasserstein distance [38] between the distribution of sequence lengths of the sampled sequences and the test set. We report the results on the test set averaged over five runs with different seeds.
Results. Table 1 presents the MMD results for all models and datasets. Among them, the RNN baseline demonstrates a strong performance across all datasets and outperforms both Transformer and  $GD$ . Notably, ADD-THIN exhibits competitive results with the autoregressive baseline, surpassing or matching  $(\pm 0.01)$  it on 11/13 datasets. Additionally, ADD-THIN consistently outperforms the Transformer and  $GD$  model on all datasets except SC and RP. Lastly, TriTPP performs well on most datasets but is outperformed or matched by our model on all but two datasets.
Table 2 shows the result for comparing the count distributions. Overall, the Wasserstein distance results align closely with the MMD results. However, the  $GD$  model is an exception, displaying considerably worse performance when focusing on the count distribution. This outcome is expected since the training of the  $GD$  model only indirectly models the number of events by maximizing the ELBO of each diffusion step to approximate the conditional density of the next event and not the likelihood of whole event sequences. Again, ADD-THIN shows a very strong performance, further emphasizing its expressiveness.
In summary, these results demonstrate the flexibility of our model, which can effectively capture various complex TPP distributions and matches the state-of-the-art performance in density estimation.
Table 3: Wasserstein distance between forecasted event sequence and ground truth reported for 50 random forecast windows on the test set (lower is better). The results with standard deviation are reported in Appendix E.
PUBG	Reddit-C	Reddit-S	Taxi	Twitter	Yelp1	Yelp2
Average Seq. Length	76.5	295.7	1129.0	98.4	14.9	30.5	55.2
RNN	6.15	35.22	39.23	4.14	2.04	1.28	2.21
Transformer	2.45	38.77	27.52	3.12	2.09	1.29	2.64
GD	5.44	44.72	64.25	4.32	2.16	1.52	4.25
ADD-THIN (Ours)	2.03	17.18	21.32	2.42	1.48	1.00	1.54
Table 4: Count MAPE  $\times 100\%$  between forecasted event sequences and ground truth reported for 50 random forecast windows on the test set (lower is better). The results with standard deviation are reported in Appendix E.
PUBG	Reddit-C	Reddit-S	Taxi	Twitter	Yelp1	Yelp2
Average Seq. Length	76.5	295.7	1129.0	98.4	14.9	30.5	55.2
RNN	1.72	5.47	0.68	0.54	0.95	0.59	0.72
Transformer	0.65	7.38	0.55	0.46	1.18	0.63	0.99
GD	1.66	10.49	1.33	0.71	1.43	0.78	1.65
ADD-THIN (Ours)	0.45	1.07	0.38	0.37	0.69	0.45	0.50
5.2 Conditional sampling - Forecasting
Forecasting event sequences from history is an important real-world application of TPP models. In this experiment, we evaluate the forecasting capability of our model on all real-world datasets. We evaluate each model's performance in forecasting the events in a forecasting window  $\Delta T$ , by randomly drawing a starting point  $T_{s} \in [\Delta T, T - \Delta T]$ . Then, the events in  $[0, T_{s}]$  are considered the history, and  $[T_{s}, T_{s} + \Delta T]$  is the forecasting time horizon.
In the experiment, we randomly sample 50 forecasting windows for each sequence from the test set, compute history embedding with each model's encoder, and then conditionally sample the forecast from each model. Note that TriTPP does not allow for conditional sampling and is therefore not part of the forecasting experiment.
Metrics. To evaluate the forecasting experiment, we will not compare distributions of TPPs but rather TPP instances. We measure the distance between two event sequences, i.e., forecast and ground truth data, by computing the distance between the count measures with the Wasserstein distance between two TPPs, as introduced by Xiao et al. [49]. Additionally, we report the mean absolute relative error (MAPE) between the predicted sequence length and ground truth sequence length in the forecasting horizon. We report the results on the test set averaged over five runs with different seeds.
Results. Table 3 presents the average Wasserstein distance between the predicted and ground truth forecast sequences. The results unequivocally demonstrate the superior performance of our model by forecasting entire event sequences, surpassing all autoregressive baselines on all datasets. Notably, the disparity between ADD-THIN and the baselines is more pronounced for datasets with a higher number of events per sequence, indicating the accumulation of prediction errors in the autoregressive models. Further, the transformer baseline achieves better forecasting results than the RNN baseline for some datasets with more events. This suggests that long-range attention can improve autoregressive forecasting and mitigate some error accumulation.
Figure 4:  $5\%$ ,  $25\%$ ,  $50\%$ ,  $75\%$ , and  $95\%$  quantile of forecasts generated by ADD-THIN for a Taxi event sequence (blue: history, black ground truth future).
Table 4 reports the MAPE between the forecasted and ground truth sequence length. The MAPE results align consistently with the Wasserstein distance across all models and datasets. Figure 4
depicts the quantiles for 1000 forecasts generated by ADD-THIN for one Taxi event sequence and highlights the predictive capacity of our model. Overall, our model outperforms state-of-the-art TPP models in forecasting real-world event sequences.
ADD-THIN vs. autoregressive TPP models. On a conceptual level, ADD-THIN presents a different trade-off compared to other TPP models: Instead of being autoregressive in event time, our model gradually refines the entire event sequence in parallel at every diffusion step to produce a sample from the learned data distribution. Thereby, we have found that our model is better suited for forecasting and modeling very long event sequences than autoregressive TPP models. Furthermore, the iterative refinement of the entire sequence allows us to leverage simple and shared layers to accurately model the long-range interaction between events and results in nearly constant sampling times across different sequence lengths (cf., Appendix E.3).
Limitations and future work. With ADD-THIN, we have derived a novel diffusion-inspired model for TPPs. Thereby, we focused on modeling the arrival times of the events and did not model continuous and discrete marks. However, we see this as an exciting extension to our framework, which might incorporate Gaussian diffusion [18] for continuous marks and discrete diffusion [2] for discrete marks. Further, while generative diffusion is known to produce high-quality samples, it also can be expensive. Besides tuning the number of diffusion steps, future work could focus on alternative and faster sampling routines [35]. Ultimately, we hope that by having connected diffusion models with TPPs, we have opened a new direction to modeling TPPs and broadened the field of diffusion-based models. Here, it would be especially interesting for us to see whether our framework could benefit other application domains in machine learning that involve sets of varying sizes, such as graph generation (molecules), point clouds, and spatial point processes.
By introducing ADD-THIN, we have connected the fields of diffusion models and TPPs and derived a novel model that naturally handles the discrete and continuous nature of point processes. Our model permits parallel and closed-form sampling of entire event sequences, overcoming common limitations of autoregressive TPP models. In our experimental evaluation, we demonstrated the flexibility of ADD-THIN, which can effectively capture complex TPP distributions and matches the state-of-the-art performance in density estimation. Additionally, in a long-term forecasting task on real-world data, our model distinctly outperforms the state-of-the-art TPP models by predicting entire forecasting windows non-autoregressively.
Broader impact
We see the proposed model as a general framework to model continuous-time event data. As such, our method can be applied to many fields, where common application domains include traffic, social networks, and electronic health records. We do not find any use cases mentioned above raise ethical concerns; however, it is essential to exercise caution when dealing with sensitive personal data.
This research was supported by the German Research Foundation, grant GU 1409/3-1.
[40] K. Sharma, Y. Zhang, E. Ferrara, and Y. Liu. Identifying coordinated accounts on social media through hidden influence and group behaviours. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1441–1451, 2021.
[41] O. Shchur, M. Biloš, and S. Gunnemann. Intensity-free learning of temporal point processes. In International Conference on Learning Representations (ICLR), 2020.
[42] O. Shchur, N. Gao, M. Biloš, and S. Gunnemann. Fast and flexible temporal point processes with triangular maps. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[43] O. Shchur, A. C. Turkmen, T. Januschowski, and S. Gunnemann. Neural temporal point processes: A review. arXiv preprint arXiv:2104.03528, 2021.
[44] A. Soen, A. Mathews, D. Grixti-Cheng, and L. Xie. Unipoint: Universally approximating point processes intensities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9685–9694, 2021.
[45] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML), pages 2256-2265, 2015.
[46] Y. Tashiro, J. Song, Y. Song, and S. Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34: 24804-24816, 2021.
[47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[48] C. Vignac, I. Krawczuk, A. Siraudin, B. Wang, V. Cevher, and P. Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734, 2022.
[49] S. Xiao, M. Farajtabar, X. Ye, J. Yan, L. Song, and H. Zha. Wasserstein learning of deep generative point process models. Advances in neural information processing systems, 30, 2017.
[50] Q. Zhang, A. Lipani, O. Kirnap, and E. Yilmaz. Self-attentive hawkes process. In International conference on machine learning, pages 11183–11193. PMLR, 2020.
[51] W. Zhang, T. Panum, S. Jha, P. Chalasani, and D. Page. Cause: Learning granger causality from event sequences using attribution methods. In International Conference on Machine Learning, pages 11235-11245. PMLR, 2020.
[52] S. Zuo, H. Jiang, Z. Li, T. Zhao, and H. Zha. Transformer hawkes process. arXiv preprint arXiv:2002.09291, 2020.
A On the relationship between the applied loss and the ELBO
In the following section, we investigate the relationship between our applied loss function and the ELBO to the unknown data distribution derived for diffusion models. The ELBO of diffusion models [18] is given as follows:
$$
\begin{array}{l} \mathcal {L} _ {E L B O} = \mathbb {E} _ {q} \left[ \underbrace {D _ {K L} \left(q (\boldsymbol {t} ^ {(N)} \mid \boldsymbol {t} ^ {(0)}) \| p (\boldsymbol {t} ^ {(N)})\right)} _ {\mathcal {L} _ {N}} - \underbrace {\log p _ {\theta} (\boldsymbol {t} ^ {(0)} \mid \boldsymbol {t} ^ {(1)})} _ {\mathcal {L} _ {0}} + \right. \\ \sum_ {n = 2} ^ {N} \underbrace {D _ {K L} \left(q \left(\boldsymbol {t} ^ {(n - 1)} \mid \boldsymbol {t} ^ {(0)} , \boldsymbol {t} ^ {(n)}\right) \| p _ {\theta} \left(\boldsymbol {t} ^ {(n - 1)} \mid \boldsymbol {t} ^ {(n)}\right)\right)} _ {\mathcal {L} _ {n}} ]. \tag {6} \\ \end{array}
$$
Additionally, the Janossi density [8] of an event sequence  $t$  on  $[0,T]$  allows us to represent each element of the ELBO in terms of our derived inhomogeneous (approximate) posterior intensities (Section 3.2 and Figure 2) as follows:
$$
p (\boldsymbol {t}) = \exp \left(- \int_ {0} ^ {T} \lambda (t) \mathrm {d} t\right) \prod_ {t _ {i} \in \boldsymbol {t}} \lambda \left(t _ {i}\right). \tag {7}
$$
$\mathcal{L}_N$ :  $\mathcal{L}_N$  is constant as the intensity  $\lambda_{\mathrm{HPP}}$  defining  $q(\pmb{t}^{(N)} \mid \pmb{t}^{(0)})$  and  $p(\pmb{t}^{(N)})$  has no learnable parameters.
$\mathcal{L}_0$ : We directly train our model to optimize this likelihood term as described in Section 3.3.
$\mathcal{L}_n$ : The KL divergence between two densities is defined as:
$$
D _ {K L} (q \| p _ {\theta}) = \mathbb {E} _ {q} \left[ \log \left(q \left(\boldsymbol {t} ^ {(n - 1)} \mid \boldsymbol {t} ^ {(0)}, \boldsymbol {t} ^ {(n)}\right)\right) - \log \left(p _ {\theta} \left(\boldsymbol {t} ^ {(n - 1)} \mid \boldsymbol {t} ^ {(n)}\right)\right) \right], \tag {8}
$$
where only the right-hand side relies on  $\theta$ , letting us minimize the KL divergence by maximizing the expectation over the log-likelihood  $\log(p_{\theta}(\pmb{t}^{(n-1)} \mid \pmb{t}^{(n)}))$ .
To add some additional context to the KL divergence in  $\mathcal{L}_n$  and similar to the derivation of the posterior in Section 3.2, we will further distinguish three cases: