EventFlow: Forecasting Continuous-Time Event Data with Flow Matching

Authors: Gavin Kerrigan, Kai Nelson, Padhraic Smyth
Primary area: generative models
Keywords: temporal point processes, generative models, event sequences

Abstract:
Continuous-time event sequences, in which events occur at irregular intervals, are ubiquitous across a wide range of industrial and scientific domains. The contemporary modeling paradigm is to treat such data as realizations of a temporal point process, and in machine learning it is common to model temporal point processes in an autoregressive fashion using a neural network. While autoregressive models are successful in predicting the time of a single subsequent event, their performance can be unsatisfactory in forecasting longer horizons due to cascading errors. We propose $\texttt{EventFlow}$, a non-autoregressive generative model for temporal point processes. Our model builds on the flow matching framework in order to directly learn joint distributions over event times, side-stepping the autoregressive process. $\texttt{EventFlow}$ is likelihood-free, easy to implement and sample from, and either matches or surpasses the performance of state-of-the-art models in both unconditional and conditional generation tasks on a set of standard benchmarks.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730148389270

thread_type:
official_review

<thread_object>
creator: reviewer_u5wj

<content>
summary:
The authors created a non-autoregressive model framework for continuous-time event sequences by using flow-matching. This allows them to skip the autoregressive process and also account for irregular time intervals. They achieve good results on a synthetic dataset and real world datasets compared to other baselines.

strengths:
Paper is clear and is quite original with the framework of flow-matching; math is clear and has good structure.

weaknesses:
I would've liked to see some more comparison against non-autoregressive type models for generation, such that it clarifies why the authors choose to use flow matching in particular. Comparatively, MMD wise the metric always measures with respect to the distribution, so with respect to this metric joint distribution models instead of autoregressive ones would have the advantage here.
I would like to see some more discussion on the synthetic datasets with IFTPP; why are the performances on the synthetic plot better as compared to the real world data?

questions:
How does the length affect the corresponding accuracy? Is there an error vs sequence position plot?
Just a note that the appendix should be in the supplementary material and not the main portion :)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730390157766

thread_type:
official_review

<thread_object>
creator: reviewer_u733

<content>
summary:
The paper proposes EventFlow, a flow-matching model for temporal point processes. EventFlow is evaluated on conditional and unconditional generative tasks. In contrast to previous diffusion-based approaches, balanced couplings are used, and the number of events does not change during generation. Therefore, the inference process consists of multiple steps: (1) EventFlow samples (unconditional) or predicts (conditional) the number of events. (2) A prior sequence $\gamma_0$ is sampled from a mixed-binomial TPP. (3) The sequence $\gamma_1$ is computed by solving the ODE with the vector field $v_\theta$.

In an experimental evaluation, EventFlow shows promising results and consistently outperforms multiple baselines.

strengths:
- EventFlow models the event times of sequences via a CFM. The resulting model is more straightforward, more elegant, and effective than previous diffusion-based approaches.
- The evaluation includes a reasonable number of experiments, including unconditional and conditional comparisons on multiple datasets. 
- The results show strong improvements compared to diffusion-based baselines.
- The paper is easy to follow, and the methodology is clearly described.
- The baselines are reasonably tuned.

weaknesses:
- The biggest weakness, in my opinion, results from the use of balanced couplings. Modeling the event count distribution $p_\theta(n\mid \mathcal{H})$ with a deterministic classifier limits the distributional forecast, which is not the case for AddAndThin. I assume this would be more noticeable in a likelihood-based evaluation.
- The paper does not discuss whether a likelihood evaluation is possible. As EventFlow is based on a CNF, evaluating the likelihoods of single sequences should be possible. Including an NLL-based comparison, as done in other works (Shchur et al., 2019), would strengthen the evaluation.
- Certain hyperparameters are missing. How many neural function evaluations are used to solve the ODE? An ablation would help compare the performance with previous diffusion-based approaches.
- A runtime comparison with AddAndThin needs to be included. Especially as EventFlow uses an attention-based model.

Minors:
- page 4 footnote: i.e. -> i.e., 
- Shchur et al. (2019) is cited twice (arxiv and ICLR)

questions:
- Is computing the likelihood for single sequences possible?
- Are the prior sequences ordered? I.e., can the paths of events cross when interpolating between $\gamma_0$ and $\gamma_1$?
- Are the quantitative results of the classifier $p_\theta(n\mid \mathcal{H})$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730408052357

thread_type:
official_review

<thread_object>
creator: reviewer_9ccg

<content>
summary:
This paper introduces EventFlow, a novel generative model for temporal point processes (TPPs). Unlike existing autoregressive models that predict events one at a time and suffer from compounding errors, EventFlow directly learns joint distributions over event times, enabling more accurate multi-step forecasting. The model is likelihood-free, making it easier to implement and sample from than existing methods. EventFlow's performance is evaluated on various synthetic and real-world datasets, consistently outperforming existing methods in both unconditional generation and multi-step forecasting tasks (forecasting in a given window) .

strengths:
Originality: this is a novel generative model for temporal point processes that bypass the autoregressive paradigm; potential the second model in this category ( the first being diffusion point process by ludke et al.) 

Quality: the paper is very technical and technicality comes from defining appropriate probability measures; the authors reaches to an important conclusion – proposition 1, based on which they proposed eventflow. The proposed method eventflow including interpolant construction and training, and sampling seems very solid (although I did check the full details.) The paper also examines the two important cases conditional forecasting and unconditional with respect to history and conducts experiments which demonstrate the effectiveness. 

Clarity: it is well presented for the most part. 

Significance: I think this is an interest line of research as it deviates from traditional autoregressive models for TPP and I believe it is worthy of investigation by our community.

weaknesses:
My main concern/weakness is evaluation. Especially for forecasting tasks where a separate model is learned for the event count distribution pφ(n | H). The authors treat the problem as classification. I assume learning is from the training data, where in some target window we have {1,…,N} events as target. The learned pφ(n | H) is then used to sample a count n for a specific instance for further inference task. For the experiments, while the authors use MMD as a metric for sequence distance, I don’t know how the forecasted number of events compared with the ground truth. Similarly I am not entirely sure how the forecasting times compared with the ground truth. Some visual aids will be very helpful.

questions:
Can the authors clarify at what flow time do the authors use as results reported for your experiments, since s is discretized.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730676678550

thread_type:
official_review

<thread_object>
creator: reviewer_wudp

<content>
summary:
The work proposes a generative approach of modelling (unmarked) temporal point process based on diffusion models. The work decompose the distributions of temporal pointe process into the joint distribution of the number of events in a TPP realization and the joint distribution of event time conditioned on the number of events via a diffusion model in the rectified flow fashion. The proposed approach can be conditioned on history information for generate future events. Evaluation results on both unconditional generation results on commonly used TPP datasets and conditional future events forecasting shows superior performance of the model over baselines.

strengths:
The work has the following strengths:
* This work extends diffusion models to the domain of temporal point process with a clear motivation of modelling joint distributions of events. This motivation positions the work as a novel approach toward the challenging task of future events forecasting in a fixed time horizon instead of single next event prediction. Experiment results also show clear advantages of the proposed work in multi-events forecasting in a fixed time window over existing works.
* The decomposition of TPP distribution into the distribution of the number of events and the joint distributions of event times is also a novel approach toward TPP modelling.

weaknesses:
The work has the following weakness:
* The presentation of the work has room of improvements:
    * It is not clear what’s the purpose of introducing balanced coupling of TPPs in Section 4.1. The interpolation between two event sequences can be well defined without introducing balanced coupling and the number of events is modelled separately. In other words, the presentations of the methods does not necessarily rely on the introduction of balanced coupling and the theoretical results of the iff condition for the balanced coupling set to be non-empty. It would be helpful to add some intuitive motivation for this section in the work.
    * In the current presentation of approach, the original contribution (Section 4.1, 4.2, separate modelling of event count and event time) and existing work (Sections 4.3, 4.4, mostly rectified flows[1]) are intertwined in one section. I would suggest the authors to consider either clearly separating their original contributions from the methods in existing works or more clearly stating their contributions in terms of methodology instead of using generic statements of contributions like a generative model for TPP.
* The work constrains the type of TPP it models to unmarked temporal point process without event category labels. The practical values of the work is limited. 
* It is arguable that MSE/RMSE for next event prediction is one of the most important evaluation results for TPP models due to its wide existence in many existing TPP works [2, 3, 4], practical applicability, and easiness to compare between models. As the work is capable of conditional generation, not including this evaluation task results is disappointing.

References

[1] Liu, Xingchao, Chengyue Gong, and Qiang Liu. "Flow straight and fast: Learning to generate and transfer data with rectified flow." arXiv preprint arXiv:2209.03003 (2022).

[2] Yang, Chenghao, Hongyuan Mei, and Jason Eisner. "Transformer embeddings of irregularly spaced events and their participants." arXiv preprint arXiv:2201.00044 (2021).

[3] Shchur, Oleksandr, Marin Biloš, and Stephan Günnemann. "Intensity-free learning of temporal point processes." arXiv preprint arXiv:1909.12127 (2019).

[4] Xue, Siqiao, et al. "Easytpp: Towards open benchmarking the temporal point processes." arXiv preprint arXiv:2307.08097 (2023).

questions:
1. Do the models for conditional and unconditional generation share parameters?
2. The EventFlow model does not guarantee the generated sequences of event times preserves the original order of $\gamma_0$ but simply relies on learning such prior from training data. Does the work anpply any post-processing to generated results to deal with this potential problem? If not, do the authors encounter situations where the generated event sequence times are not in an increasing order?
3. The author claims the approach is likelihood-free. Is the re-ordering of sampled $\gamma_0$ part of the reasons that make the model incapable of evaluating likelihood of event sequences? Is it possible to define the distribution which $\gamma_0$ is sampled from in a fashion such that the $t_s$ are naturally in an ascending order?
4. Can the approach be extended to model marked TPP with event category labels in trivial ways like using the the hidden states of the denoising neural networks $\v_\theta$ to predict event categories?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732664199916

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to All Reviewers

comment:
We thank the reviewers for their detailed, constructive reviews and valuable suggestions for improving our work. We have updated our submission to incorporate these suggestions, and changes to our manuscript are highlighted as blue text.

## Forecasting Results

While preparing our response, a careful review of our results resulted in us discovering a discrepancy in our experimental analysis which affects the forecasting results originally presented in Table 1. In particular, the sequence distance (Equation 9) depends on a choice of time $T$, corresponding to the maximum allowed event time. Previous work (Lüdke 2023) and our baselines implemented this using the value of $T$ corresponding to the maximum allowed event time across all sequences in a dataset. However, when forecasting, sequences are not generated until this time $T$, but rather only on some subwindow $[T_0, T_E]$ with $T_{E} = T_0 + \Delta T$ being some given ending time. Our original results in Table 1 for our method were computed using $T_E$, resulting in overly optimistic results.

We believe that the version of the distance metric using $T_E$ is more suitable for evaluating forecasts, as the version using $T$ implicitly depends on the location of the forecasting window $[T_S, T_E]$ within the support $[0, T]$. That is, if a given forecast is simply translated from the forecast window $[T_S, T_E]$ to a new window $[T_S’, T_E’]$, the distance using $T$ will change. However, the distance using $T_E$ (respectively $T_E’$) will remain invariant.

We have recomputed Table 1 using $T_E$ for all methods. Overall, our method still obtains the best mean performance on 5/7 of the datasets, and obtains the second best performance on the remaining 2/7 datasets (PUBG, Reddit-C) with competitive scores. While the numerical results of this experiment have changed, our key conclusions do not, as these results still indicate our model obtains state-of-the-art forecasting performance.

## Event Count Evaluation

Reviewers 9ccG and u733 suggest isolating and evaluating the performance of the learned event count distribution, i.e., $p_\phi(n \mid H)$. To that end, we include in Table 6 (Appendix C) a table of the mean absolute relative error (MARE) between the true and forecasted sequence lengths. This metric serves to evaluate only the predicted number of events in a forecast.

Here, we find that Add-and-Thin has strong performance (mean rank: 1.3), whereas our method (mean rank: 3), diffusion (mean rank: 3.1) perform comparably, while IFTPP (mean rank: 3.6) and NHP lag slightly behind (mean rank: 4)

While our method has room for improvement, we emphasize that even though our approach to learning $p(n \mid H)$ is quite simple it still achieves competitive results. Designing better techniques for predicting the event counts is an exciting direction for future work, and would likely lead to even stronger forecasting results. We further emphasize that our model shows strong forecasting performance (Table 1), which simultaneously measures both the event counts and their times, and this is the primary relevant metric for the problem we address in this paper.

We also perform an ablation study (see Table 1) where we set the number of events $n$ to be the true number of events in a given forecast. This serves to isolate only the event time prediction component of our model, and goes to show that future work improving the predictions for $n$ can lead to significant forecasting gains.

## NFE Ablation

Reviewer u733 mentions the number of function evaluations (NFEs) used by our model at generation time (i.e., in Equation 9). We originally used 25 NFEs. Here, we perform an additional ablation on the choice of NFEs. To do so, we took our models used in the forecasting experiment and reduced the NFEs to 10 or 1 at generation time. The resulting forecast errors are reported in Table 1. We find that 10 NFEs is sufficient to obtain results on-par with those we obtained previously with 25 NFEs. Interestingly, with only a single NFE, we observe only a small drop in forecasting performance, with our model still obtaining the best forecasting performance on 4/7 datasets and second-best performance on the remaining 3/7 datasets. We emphasize that Add-and-Thin uses 100 NFEs at generation time, and the diffusion model uses 1000 NFEs per generated event. The remaining baselines (NHP, IFTPP) require one NFE per generated event. Thus, our method is able to simultaneously obtain strong forecasting performance while only requiring a small number of NFEs.

The level of performance obtained by our model with a small number of NFEs is enabled by our carefully designed interpolant construction (Equation 3). As these interpolants are linear, the resulting paths are typically straight and easy to integrate (i.e., requiring only a small number of steps in the differential equation solver).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces EventFlow, a non-autoregressive generative model for continuous-time event sequences based on flow matching. EventFlow models the joint distribution of event times, avoiding the cascading errors of autoregressive methods. It demonstrates strong performance on unconditional and conditional forecasting tasks across several datasets. The paper proposes a new application of flow matching to temporal point processes (TPPs).