HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?

## Abstract

Abstract Accurately forecasting multiple future events within a given time horizon is crucial for finance, retail, social networks, and healthcare applications. Event timing and labels are typically modeled using Marked Temporal Point Processes (MTPP), with evaluations often focused on next-event prediction quality. While some studies have extended evaluations to a fixed number of future events, we demonstrate that this approach leads to inaccuracies in handling false positives and false negatives. To address these issues, we propose a novel evaluation method inspired by object detection techniques from computer vision. Specifically, we introduce Temporal mean Average Precision (T-mAP), a temporal variant of mAP, which overcomes the limitations of existing long-horizon evaluation metrics. Our extensive experiments demonstrate that models with strong next-event prediction accuracy can yield poor long-horizon forecasts and vice versa, indicating that specialized methods are needed for each task. To support further research, we release HoTPP 1 1 1 https://github.com/ivan-chai/hotpp-benchmark , the first benchmark designed explicitly for evaluating long-horizon MTPP predictions. HoTPP includes large-scale datasets with up to 43 million events and provides optimized procedures for both autoregressive and parallel inference, paving the way for future advancements in the field.

## 1 Introduction

The world is full of events. Internet activity, e-commerce transactions, retail operations, clinical visits, and numerous other aspects of our lives generate vast amounts of data in the form of timestamps and related information. In the era of AI, it is crucial to develop methods capable of handling these complex data streams. We refer to this type of data as Event Sequences (ESs). Event sequences differ fundamentally from other data types. Unlike tabular data (Wang & Sun, ), ESs include timestamps and possess an inherent order. In contrast to time series data (Lim & Zohren, ), ESs are characterized by irregular time intervals and structured descriptions of each event. These differences necessitate the development of specialized models and evaluation practices.

Sequence modeling is the primary task in Event Sequences (ESs) domain. In its simplest form, each event is defined by its type and occurrence time, a framework commonly known as Marked Temporal Point Processes (MTPP) (Rizoiu et al., ). Some studies extend MTPP to incorporate additional data fields and model complex dependencies between them (McDermott et al., ). However, the majority of MTPP approaches, as well as their evaluation pipelines, primarily focus on predicting the next event based on historical data.

In practice, a common question arises: what events will occur, and when, within a specific time horizon? Forecasting multiple future events presents unique challenges that differ from traditional next-event prediction tasks. For instance, autoregressive event sequence prediction involves applying the model to its own potentially erroneous predictions. However, the challenges of autoregressive prediction in the context of MTPP have not been thoroughly explored. Another difficulty lies in evaluation. Methods like Dynamic Time Warping (DTW) are typically unsuitable for event sequences due to strict ordering constraints (Su & Hua, ). Some works have applied Optimal Transport Distance (OTD), a variant of the Wasserstein distance, to compare sequences of predefined lengths. Still, the limitations of this metric have not been previously considered.

In this work, we provide the first in-depth analysis of models and metrics for long-horizon event forecasting, establishing a rigorous evaluation framework and a baseline for MTPP studies. Our key contributions are as follows:

- 1.
We demonstrate that widely used evaluation methods for MTPPs often overlook critical aspects of model performance. We demonstrate that simple rule-based baselines can sometimes outperform popular deep learning methods when evaluated using OTD.
- 2.
We introduce Temporal mean Average Precision (T-mAP), a novel evaluation metric inspired by best practices in computer vision. T-mAP evaluates variable length sequences within a specified time horizon, as illustrated in Figure 1. Unlike previous approaches, T-mAP accurately accounts for false positives and false negatives while being invariant to linear calibration. Additionally, we address a theoretical gap in computer vision by proving the correctness of the T-mAP computation algorithm.
- 3.
Using our established methodology, we demonstrate that high next-event prediction accuracy does not necessarily translate into high-quality long-horizon forecasts; in many cases, our experiments show the opposite. This highlights the necessity of developing specialized models for the long-horizon prediction task.
- 4.
We release HoTPP, a new open-source benchmark designed to facilitate long-horizon event sequence prediction research. HoTPP brings together datasets and methods from various domains, including financial transactions, social networks, healthcare, and recommender systems, greatly expanding the diversity and scale of data compared to prior benchmarks. Additionally, we offer an efficient inference algorithm necessary for large-scale evaluations.

Figure: Figure 1: Temporal mAP (T-mAP) evaluation pipeline. Unlike previous methods, T-mAP evaluates sequences of variable lengths within the prediction horizon. It further improves performance assessment by analyzing label distributions rather than relying on fixed predictions.
Refer to caption: x1.png

## 2 Related Work

Marked Temporal Point Processes.
A Marked Temporal Point Process (MTPP) is a stochastic process consisting of a sequence of pairs $(t_{1},l_{1}),(t_{2},l_{2}),\dots$, where $t_{1}<t_{2}<\dots$ represent event times and $l_{i}\in\{1,\dots,L\}$ denote event type labels (Rizoiu et al., ). Common approaches to MTPP modeling focus on predicting the next event. A basic solution involves independently predicting the event time and type. A more advanced approach splits the original sequence into subsequences, one for each event type, and independently models each subsequence’s timing. Depending on the time-step distribution, the process is called Poisson or Hawkes.

Over the last decade, the focus has shifted toward increasing model flexibility by applying neural architectures. Several works have employed classical RNNs (Du et al., ; Xiao et al., ; Omi et al., ) and transformers (Zuo et al., ; Zhang et al., ), while others have proposed architectures with continuous time (Mei & Eisner, ; Rubanova et al., ; Kuleshov et al., ). In this work, we evaluate MTPP models with both discrete and continuous time architectures. Unlike previous benchmarks, we also assess simple rule-based predictors and popular methods from related fields, including GPT-like prediction models for event sequences (McDermott et al., ; Padhi et al., ) and Next-K models from time series analysis (Lim & Zohren, ). For more details on MTPP modeling, refer to Appendix A.

MTPP Evaluation. Previous benchmarks have primarily focused on medical data or traditional MTPP datasets. EventStream-GPT (McDermott et al., ) and TemporAI (Saveliev & van der Schaar, ) consider only medical data and do not implement methods from the MTPP field, despite their applicability. Early MTPP benchmarks, such as Tick (Bacry et al., ) and PyHawkes(^2^22[https://github.com/slinderman/pyhawkes](https://github.com/slinderman/pyhawkes)), implement classical machine learning approaches but exclude modern neural networks. While PoPPy (Xu, ) and EasyTPP (Xue et al., ) include neural methods, they do not consider rule-based and Next-K approaches. Furthermore, PoPPy does not evaluate long-horizon predictions at all. EasyTPP addresses this limitation to some extent by reporting the OTD metric, though it does not provide the corresponding evaluation code.

Previous work also highlighted the limitations of Dynamic Time Warping (DTW) for event sequence evaluation (Su & Hua, ), showing that DTW’s strict ordering constraints are impractical and should be avoided. This work proposed an alternative, the Order-preserving Wasserstein Distance (OPW). In our study, we address the limitations of OTD, a variant of the Wasserstein Distance, and propose a new metric called T-mAP. Unlike OPW, T-mAP operates on timestamps rather than event indices and involves two hyperparameters, compared to three in OPW.

## 3 Limitations of the Next-Event and OTD Metrics

Figure: Figure 2: Comparison of OTD and T-mAP metrics. In example 1.a prefix evaluation of three events using OTD results in an incorrect, false negative. T-mAP addresses this issue by comparing all events within the time horizon, as shown in example 1.b. Additionally, OTD evaluates only the label with the maximum probability. From the OTD perspective, cases 2.a and 2.b have the same quality. In contrast, T-mAP evaluates the entire distribution, yielding a low score of 0.33 in case 2.a and a high score of 1 in case 2.b.
Refer to caption: x2.png

MTPPs are typically evaluated based on the accuracy of next-event predictions, with time and type predictions assessed independently. The quality of type predictions is measured by the error rate, while time prediction error is evaluated using either Mean Absolute Error (MAE) or Root Mean Squared Error (RMSE). However, these metrics do not account for the model’s ability to predict multiple future events. For example, in autoregressive models, the outputs are fed back as inputs for subsequent steps, which can lead to cumulative prediction errors. As a result, long-horizon evaluation metrics are necessary.

Recent studies have advanced the measurement of horizon prediction quality by employing Optimal Transport Distance (OTD) (Mei et al., ). This metric is analogous to the edit distance between the predicted event sequence and the ground truth. Suppose there is a predicted sequence $S_{p}=\{(t^{p}_{i},l^{p}_{i})\}_{i=1}^{n_{p}},0<t_{1}^{p}<t_{2}^{p}<\dots$, and a ground truth sequence $S_{gt}=\{(t^{gt}_{j},l^{gt}_{j})\}_{j=1}^{n_{gt}},0<t_{1}^{gt}<t_{2}^{gt}<\dots$. These two sequences form a bipartite graph $\mathcal{G}(S_{p},S_{gt})$. The prediction is connected to the ground truth event if their types are equal. Denote $M(S_{p},S_{gt})$ a set of all possible matches in the graph $\mathcal{G}(S_{p},S_{gt})$. OTD finds the minimum cost among all possible matchings:

$$ $\mathrm{OTD}(S_{p},S_{gt})=\min\limits_{m\in M(S_{p},S_{gt})}\left[\sum\limits _{(i,j)\in m}|t^{p}_{i}-t^{gt}_{j}|+C_{del}U_{p}(m)+C_{ins}U_{gt}(m)\right],$ (1) $$

where $U_{p}(m)$ is the number of unmatched predictions in matching $m$, $U_{gt}(m)$ is the number of unmatched ground truth events, $C_{ins}$ is an insertion cost, and $C_{del}$ is a deletion cost. It is common to take $C_{ins}=C_{del}$ (Mei et al., ). It has been proven that OTD is a valid metric, as it is symmetric, equals zero for identical sequences, and satisfies the triangle inequality.

OTD is computed between fixed-size prefixes, but this approach limits metric flexibility in assessing models with imprecise time step predictions, as illustrated in Fig. 2.1. If the model predicts events too frequently, the first $K$ predicted events will correspond to the early part of the ground truth sequence, resulting in false negatives. This outcome is inaccurate because allowing the model to predict more events would alter the evaluation. Conversely, if the time step is too large, the first $K$ predicted events will extend far beyond the horizon of the first $K$ target events, leading to a significant number of false positives. This is undesirable as it prevents the inclusion of additional ground truth events in the evaluation. Therefore, evaluating a dynamic number of events is essential to properly align with the ground truth’s time horizon.

MTPP evaluation metrics can be categorized into two groups: those considering event indices and those assessing time and label prediction quality independently of actual positions. Next-event metrics rely on ordering, even when events share identical timestamps, making the order meaningless. While Optimal Transport Distance (OTD) itself is invariant to order, the extraction of sequences for comparison is influenced by event indices. For example, when OTD compares prefixes of length $K$, the algorithm must place $K$ correctly predicted events at the beginning of a sequence to minimize the OTD score. As a result, OTD, like next-event metrics, can depend on event ordering even when the order cannot be uniquely determined. This dependency is non-trivial and difficult to measure or control accurately.

Another limitation of the OTD metric is its inability to evaluate the full predicted distribution of labels, as shown in Figure 2.2. OTD considers only the labels with the highest probability, ignoring the complete distribution. This makes it dependent on model calibration and limits the ability to assess performance across a broader range of event types, such as long-tail predictions. However, models typically predict probabilities for all classes, allowing for a more comprehensive assessment. Therefore, we aim to develop an evaluation metric that accurately captures performance across common and rare classes.

## 4 Temporal mAP: Prediction as Detection

In this section, we introduce a novel metric, Temporal mAP (T-mAP), which analyses all errors within a predefined time horizon, explicitly controls ordering, and is invariant to linear calibration. T-mAP is inspired by object detection methods from computer vision (Everingham et al., ), as illustrated in Fig. 1. The goal of object detection is to localize objects within an image and identify their types. In event sequences, we tackle a similar problem but consider the time dimension instead of horizontal and vertical axes. Unlike object detection, where objects have spatial size, each event in an MTTP is a point without a duration. Therefore, we replace the intersection-over-union (IoU) similarity between bounding boxes with the absolute time difference.

T-mAP incorporates concepts from OTD but addresses its limitations, as outlined in the previous section. Firstly, T-mAP evaluates label probabilities instead of relying on final predictions. Secondly, T-mAP restricts the prediction horizon rather than the number of events. These adjustments result in significant differences in formulation and computation, detailed below.

### 4.1 Definition

T-mAP is parameterized by the horizon length $T$ and the maximum allowed time error $\delta$. T-mAP compares predicted and ground truth sequences within the interval $T$ from the last observed event. Consider a simplified scenario with a single event type $l$. Assume the model predicts timestamps and presence scores (logits or probabilities) for several future events: $S^{l}_{p}=\{(t^{p}_{i},s^{p}_{i})\},1\leq i\leq n_{p}$. The corresponding ground truth sequence is $S^{l}_{gt}=\{t^{gt}_{j}\},1\leq j\leq n_{gt}$. For simplicity, assume the sequences $S^{l}_{p}$ and $S^{l}_{gt}$ are filtered to include only events within the horizon $T$.

For any threshold value $h$, we can select a subset of the predicted sequence $S^{l}_{p}$ with scores exceeding the threshold:

$$ $S^{l}_{>}(h)=\{t^{p}_{i}:\exists(t^{p}_{i},s^{p}_{i})\in S^{l}_{p},s^{p}_{i}>h\}.$ (2) $$

By definition, a predicted event $i$ can be matched with a ground truth event $j$ iff $|t^{p}_{i}-t^{gt}_{j}|\leq\delta$, meaning the time difference between the predicted and ground truth events is less than or equal to $\delta$. T-mAP identifies the matching that maximizes precision and recall, i.e., matching with maximum cover $c$. The precision of this matching is calculated as $c/|S^{l}_{>}(h)|$ and the recall as $c/|S^{l}_{gt}|$. For any threshold $h$, we can count true positives (TP), false positives (FP), and false negatives (FN) across all predicted and ground truth sequences. Note that there are no true negatives, as the model cannot explicitly predict the absence of an event. Similar to binary classification, we can define a precision-recall curve by varying the threshold $h$ and then estimate the Average Precision (AP), the area under the precision-recall curve. Finally, T-mAP is defined as the average AP across all classes.

### 4.2 Computation

Finding the optimal matching independently for each threshold value $h$ is impractical; thus, we need a more efficient method to evaluate T-mAP. This section shows how to optimize T-mAP computation using an assignment problem solver, like the Jonker-Volgenant algorithm (Jonker & Volgenant, ). The resulting complexity of T-mAP computation is $\mathcal{O}(BN^{3})$, where $B$ is the number of evaluated sequences and $N=\max(n_{p},n_{gt})$ is the number of events within the horizon.

Figure: Figure 3: T-mAP computation pipeline.
Refer to caption: x3.png

For each pair of sequences $S^{l}_{p}$ and $S^{l}_{gt}$ we define a weighted bipartite graph $\mathcal{G}(S^{l}_{p},S^{l}_{gt})$ with $|S^{l}_{p}|$ vertices in the first and $|S^{l}_{gt}|$ vertices in the second part. For each pair of prediction $i$ and ground truth event $j$ with $|t^{p}_{i}-t^{gt}_{j}|\leq\delta$ we add an edge with weight $-s^{p}_{i}$, equal to negative logit of the target class, as shown in Fig. 3.a. Jonker-Volgenant algorithm finds the matching with the maximum number of edges in the graph, such that the resulting matching minimizes the total cost of selected edges, as shown in Fig. 3.b. We call this matching optimal matching and denote a set of all optimal matching possibilities as $M(\mathcal{G})$. For any threshold $h$, there is a subgraph $\mathcal{G}_{h}(S^{l}_{>}(h),S^{l}_{gt})$ with events whose scores are greater than $h$. The following theorem holds:

###### Theorem 4.1 .

For any threshold $h$ there exists an optimal matching in the graph $\mathcal{G}_{h}$, that is a subset of an optimal matching in the full graph $\mathcal{G}$:

$$ $\forall h\forall m\in M(\mathcal{G})\exists m_{h}\in M(\mathcal{G}_{h}):m_{h} \subset m.$ (3) $$

According to this theorem, we can compute the matching for the prediction $S^{l}_{gt}$ and subsequently reuse it for all thresholds $h$ and subsequences $S^{l}_{>}(h)$ to construct a precision-recall curve for the entire dataset, as shown in Fig. 3.c. The proof of the theorem, the study of calibration dependency, and the complete algorithm for T-mAP evaluation are provided in Appendix B.

###### Theorem 4.1 .

### 4.3 T-mAP Hyperparameters

T-mAP has two hyperparameters: the maximum allowed time delta $\delta$ and evaluation horizon $H$. We set $\delta$ twice the cost of the OTD because when the model predicts an incorrect label, OTD removes the prediction and adds the ground truth event with the total cost equal to $2C$. The horizon $H$ must be larger than $\delta$ to evaluate timestamp quality adequately. Therefore, depending on the dataset, we select $H$ to be approximately 3-7 times larger than $\delta$, ensuring the horizon captures an average of 6-15 events. The empirical study of T-mAP hyperparameters is presented in Figure 4. As shown, the chosen $\delta$ parameter is positioned to the right of the initial slope of the parameter-quality curve, ensuring an optimal balance between time granularity and task difficulty.

Figure: Figure 4: T-mAP dependency on the $\delta$ parameter. The dashed line indicates the selected value. Results for the StackOverflow and Amazon datasets are provided in the Appendix E.
Refer to caption: x4.png

## 5 HoTPP Benchmark

The HoTPP benchmark integrates data preprocessing, training, and evaluation in a single toolbox. Unlike previous benchmarks, HoTPP introduces the novel T-mAP metric for long-horizon prediction. HoTPP differs from prior MTPP benchmarks by including simple rule-based baselines and next-k models, simultaneously predicting multiple future events. The HoTPP benchmark is designed with a focus on simplicity, extensibility, evaluation stability, reproducibility, and computational efficiency.

Simplicity and Extensibility. The benchmark code is organized into a clear structure, separating the core library, dataset-specific scripts, and configuration files. New methods can be integrated at various levels, including the configuration file, model architecture, loss function, metric, and training module. Core components are reusable through the Hydra configuration library (Yadan, ). For example, HoTPP can apply the HYPRO rescoring method to backbone models such as IFTPP, RMTPP, NHP, and ODE. Another example is our Next-K implementation, which can be applied to IFTTP or RMTPP.

Evaluation Stability. In the MTPP domain, many datasets contain only a few thousand sequences for training and evaluation. For example, the StackOverflow test set includes 401 sequences, Retweet contains 1956, and Amazon has 1851 sequences for testing. Previous long-horizon evaluation pipelines (Xue et al., ; ) made predictions only at the end of each sequence, resulting in a limited number of predictions and reduced evaluation stability. To address this limitation, we evaluate long-horizon predictions at multiple intermediate points.

Reproducibility. The HoTPP benchmark ensures reproducibility at multiple levels. First, we use the PytorchLightning library (Falcon & The PyTorch Lightning team, ) for training with a specified random seed and report multi-seed evaluation results. Second, data preprocessing is carefully designed to ensure datasets are constructed reproducibly. Finally, we specify the environment in a Dockerfile.

Computational Efficiency. Some methods are particularly slow, especially during autoregressive inference on large datasets. Straightforward evaluation can take several hours on a single Nvidia V100 GPU. To accelerate computation, we optimized the training and inference pipelines in two ways. First, we implemented an efficient RNN that reuses computations during parallel autoregressive inference from multiple starting positions. Second, we developed highly optimized versions of the thinning algorithm used for sampling in NHP and continuous-time neural architectures (NHP and ODE), achieving up to a 4x performance improvement compared to the official implementations. These optimizations allowed us to conduct the first large-scale evaluations of algorithms such as NHP, AttNHP, ODE, and HYPRO on datasets like Transactions and MIMIC-IV. Additionally, we provide the first open-source CUDA implementation of the batched linear assignment solver, significantly enhancing the applicability of the proposed T-mAP metric.

A detailed description of the benchmark can be found in Appendix C, with further details on HoTPP’s computational performance in Appendix C.5.

Table: Table 1: Evaluation results. The best result is shown in bold. The mean and standard deviation of each metric computed during five runs with different random seeds are reported.

## 6 Experiments

We conduct experiments on five datasets of varying sizes and origins: the Transactions dataset (AI-Academy, ), the MIMIC-IV medical dataset (Johnson et al., ), two social networks datasets, Retweet (Zhao et al., ) and StackOverflow (Jure, ), and the Amazon reviews dataset (Jianmo, ).
Dataset statistics and evaluation parameters are provided in Appendix C.4.

We evaluate representative modeling methods from different groups:

- 1.
Rule-based baselines. MostPopular Generates a constant output with the most popular label from the prefix and the average time step. The Last N baseline outputs N previous events with adjusted timestamps.
- 2.
Intensity-free approaches. We implement IFTPP method, which combines mean absolute error (MAE) of the time step prediction with cross-entropy categorical loss for labels (Shchur et al., ; Padhi et al., ; McDermott et al., ).
- 3.
Intensity-based approaches. We implement RMTPP (Du et al., ) as an example of the TPP approach with a traditional RNN. We add NHP (Mei & Eisner, ), based on a continuous time LSTM architecture. We also evaluate an AttNHP approach that utilizes a continuous time transformer model (Yang et al., ) and ODE (Rubanova et al., ).
- 4.
Next-K approaches. We adapt IFTPP and RMTPP to predict multiple future events directly, implementing IFTPP-K and RMTPP-K, respectively. These approaches originate in time series analysis and have not been previously applied in the MTPP domain.
- 5.
Reranking. We implement HYPRO (Xue et al., ), which generates multiple hypotheses and selects the best sequence using a contrastive approach.

Additional details on these methods are provided in Appendix A, with training specifics outlined in Appendix D. Hyperparameters for both the methods and metrics are presented in Appendix E. The main evaluation results are shown in Table 1. In the following sections, we will discuss these results from various perspectives and offer additional analysis of the methods’ behavior.

### 6.1 Metrics Comparison

The evaluation results in Table 1 show that the OTD metric can yield low values even for simple rule-based baselines. For instance, the Last baseline achieves the lowest OTD values on the Retweet and Amazon datasets. In contrast, the T-mAP metric more clearly distinguishes between rule-based baselines and deep learning models. This difference arises from the ability to measure model confidence. OTD evaluates predicted labels alone, while T-mAP measures average precision by adjusting logits thresholds. Unlike rule-based baselines, deep models provide confidence scores, whereas baselines output only labels, resulting in low T-mAP scores for simple baselines.

The HYPRO and NHP methods maximize T-mAP, while the lowest OTD values are achieved by statistical baselines, IFTPP, and RMTPP. The difference between the two metrics lies in balancing the importance of event order versus time prediction quality. OTD evaluates the first 5-10 events, and to minimize OTD, a model must correctly identify the events that constitute the beginning of the sequence. IFTPP and RMTPP jointly predict the next event’s time and label, leading to better ordering. In contrast, methods like NHP, AttNHP, and ODE predict time steps independently for each event type. When the differences between timestamps are small, these methods can result in random ordering, negatively affecting the OTD score. T-mAP, on the other hand, is less sensitive to event order as long as the time predictions are accurate.

The HYPRO method, designed to differentiate between ground truth and predicted sequences using a discriminator model, achieves high T-mAP scores. This highlights T-mAP’s ability to reward models that generate realistic sequences. We also observed that methods with high T-mAP scores often have low next-event prediction accuracy. For instance, on the Retweet, Amazon, and MIMIC-IV datasets, high long-horizon performance is often accompanied by low next-event prediction scores. This is particularly true for the HYPRO method, whose loss function is heavily focused on long-horizon quality. The contrast on the MIMIC-IV dataset can be attributed to the large number of events with identical timestamps, making precise event ordering more difficult and impacting both next-event and OTD scores.

### 6.2 Intensity-Based vs Intensity-Free

Previous works mostly compared either intensity-based neural methods (Du et al., ; Xue et al., ) or intensity-free approaches (Padhi et al., ; McDermott et al., ). The only exception is IFTPP (Shchur et al., ), which did not compare with NHP and included only test set likelihood as an evaluation metric. This raises the question: which approach is superior? We compare the intensity-free IFTPP method with intensity-based RMTPP, NHP, and ODE approaches. Table 1 shows that the intensity-free IFTPP method excels in the next-event MAE prediction, as it optimizes this metric. In other scenarios, the results vary between datasets. Therefore, we conclude that there is no preferred solution, and both approaches warrant attention in future research.

### 6.3 Autoregression vs Direct Horizon Prediction

We compared the IFTPP and RMTPP methods with their Next-K counterparts. Our results indicate that autoregressive models generally perform better regarding next-event prediction quality, while Next-K approaches improve T-mAP scores on all datasets except Transactions. The OTD-based rankings vary across datasets, with no single approach consistently outperforming the others. Despite these findings, little effort has been made to adapt Next-K models to MTPPs, suggesting future research in this area.

We also observed that the Next-K approaches exhibit some interesting properties compared to autoregression. For instance, the entropy of label distributions in autoregression models decreases with increasing generation steps, as shown in Figure 5. The likely reason for this behavior is the dependency of the model’s future output on its past errors. Conversely, Next-K models demonstrate increasing entropy, which is expected as future events become harder to predict. This suggests that Next-K models have the potential to predict better the distributions of future labels, a factor that should be considered in future research.

Figure: Figure 5: Entropy of label distributions as a function of the position in the generated sequence. Results for StackOverflow and Amazon datasets are provided in Appendix F.
Refer to caption: x5.png

### 6.4 The Maximum Sequence Length

We observed that long-horizon prediction quality largely depends on the maximum number of allowed predictions in autoregression and Next-K models. As shown in Fig. 6, the optimal number of predicted events is usually less than the maximum number of events on the horizon. For example, in the Transactions dataset, the optimal length ranges from 3 to 5, depending on the method. This indicates that the quality of confidence estimation degrades for events further in the future, therefore it is beneficial to limit the number of predictions manually.

Figure: Figure 6: T-mAP dependency on the maximum length of the predicted sequence. Results for the StackOverflow and Amazon datasets are detailed in Appendix G.
Refer to caption: x6.png

Therefore, we conclude that greater attention should be given to probability calibration between generation steps. This finding also highlights progress in long-horizon prediction tasks, particularly determining the maximum horizon that can be accurately modeled. T-mAP is the only metric capable of evaluating the optimal predicted sequence length, while the OTD metric remains unaffected by sequence length as it compares fixed-size prefixes.

## 7 Limitations and Future Work

The proposed benchmark has several shortcomings. The training process for some methods is relatively slow. For example, a continuous time LSTM from the NHP method lacks an effective open-source GPU implementation, which could be developed in the future. Autoregression can also be optimized using specialized GPU implementations, which would significantly impact HYPRO training. Additionally, the list of implemented backbone architectures and losses can be extended, for example, by incorporating hybrid (Deshpande et al., ) and diffusion models (Zhou et al., ).

Our benchmark encourages future research to develop improved techniques for predicting future events and establish simple baselines for better measuring progress in the field. For example, Next-K models, which predict multiple future events without autoregression, show promising results in long-horizon prediction tasks. We suggest further exploration of these models. We also highlight the importance of label distribution estimation and emphasize the need to improve confidence estimation and calibration.

The T-mAP metric can be applied in domains beyond event sequences, like action recognition (Su & Hua, ). T-mAP can potentially offer better timestamp evaluation in these domains than evaluating indices with OPW. T-mAP can also provide a more natural hyperparameter selection regarding the modeling horizon and maximum allowed time error. Our theoretical justification of T-mAP can potentially be adapted to the mAP estimation algorithms in computer vision.

## 8 Conclusion

In this paper, we propose the HoTPP benchmark to assess the quality of long-horizon events forecasting. Our extensive evaluations using established datasets and various predictive models reveal a critical insight: high accuracy in next-event prediction does not necessarily correlate with superior performance in horizon prediction. This finding underscores the need for benchmarks like HoTPP, which emphasize the importance of long-horizon accuracy and robustness. By shifting the focus from short-term to long-term predictive capabilities, HoTPP aims to drive the development of more sophisticated and reliable event sequence prediction models. This, in turn, has the potential to significantly enhance the practical applications of sequential event prediction in various domains, fostering innovation and improving decision-making processes across a wide range of industries.

## Appendix A Background on Marked Temporal Point Processes Modeling

Intensity-based approaches.
Modeling the probability density function (PDF) $f^{*}(t_{i})=f(t_{i}|t_{1},\dots,t_{i-1})$ for the next event time is typically challenging, as it requires the additional constraint that its integral equals one. Instead, the non-negative intensity function $\lambda(t_{i})\geq 0$ is usually modeled. The following equation gives the relationship between the PDF and the intensity function:

$$ $f^{*}(t_{i})=\lambda(t_{i})\exp\left(-\int_{t_{i-1}}^{t_{i}}\lambda(s)ds\right).$ (4) $$

The derivation is provided in (Rizoiu et al., ). Different TPPs are characterized by their intensity function $\lambda(t_{i})$. In Poisson and non-homogeneous Poisson processes, the intensity function is independent of previous events, meaning event occurrences depend solely on external factors. In contrast, self-exciting processes are characterized by previous events increasing the intensity of future events. A notable example of a self-exciting process is the Hawkes process, in which each event linearly affects the future intensity:

$$ $\lambda(t_{i})=\lambda_{0}(t_{i})+\sum\limits_{k=1}^{i-1}\phi(t_{i}-t_{k}),$ (5) $$

where $\lambda_{0}(t_{i})\geq 0$ is the base intensity function, independent of previous events, and $\phi(x)\geq 0$ is the so-called memory kernel function. Given the intensity function $\lambda(t_{i})$, predictions are typically made by sampling or expectation estimation. Sampling is usually performed using the thinning algorithm, a specific rejection sampling approach. For details on the implementation of sampling, please refer to (Rizoiu et al., ).

Recent research has focused on modeling complex intensity functions. Various neural network architectures have been adapted to address this problem. These approaches differ in the type of neural network used and the model of the intensity function. Neural architectures range from simple RNNs and Transformers to specially designed continuous-time models like NHP (Mei & Eisner, ) and Neural ODEs (Rubanova et al., ). The intensity function between events can be modeled as a sum of intensities from previous events, as in Hawkes processes, or by directly predicting the inter-event intensity given the context embedding.

Intensity-free modeling. Some methods evaluate the next event time distribution without using intensity functions. For example, intensity-free (Shchur et al., ) represents the distribution as a mixture of Gaussians or through normalizing flows. Instead of predicting the distribution directly, other approaches solve a regression problem using MAE or RMSE loss. However, it can be shown that both MAE and RMSE losses are closely related to distribution prediction. Specifically, RMSE is analogous to log-likelihood optimization with a Normal distribution, and MAE is similar to the log-likelihood of a Laplace distribution (Bishop, ). In our experiments, we evaluated a model trained with MAE loss as an example of an intensity-free method.

Rescoring with HYPRO. HYPRO (Xue et al., ) is an extension applicable to any sequence prediction method capable of sampling. HYPRO takes a pretrained generative model and trains an additional scoring module to select the best sequence from a sample. It is trained with a contrastive loss to distinguish between the generated sequence and the ground truth. HYPRO generates multiple sequences with a background model during inference and selects the best one by maximizing the estimated score. Although HYPRO is intended to improve quality compared to simple sampling, it is unclear whether HYPRO outperforms expectation-based prediction. In our work, we apply HYPRO to the outputs of the RMTPP model.

Next-K models. While not commonly used in the MTPP field, Next-K approaches are popular in time series modeling (Lim & Zohren, ). These methods predict multiple future events at once, avoiding the need for autoregressive inference. The advantages of Next-K approaches include fast inference and stability, as predictions do not depend on potential errors from previous steps, unlike autoregression. The main limitation is a fixed prediction horizon, as the model cannot predict sequences of arbitrary length. However, applying a modified autoregressive approach can potentially address this limitation.

In our work, we implement a Next-K variant of the IFTPP model, which is straightforward. Given K predictions and K ground truth events, we compute MAE and cross-entropy losses between corresponding pairs of events from both sequences. The Next-K variant of the RMTPP approach is more complex, as it violates Hawkes assumptions: the $i$-th prediction doesn’t depend on predictions $1\dots i-1$. Consequently, RMTPP-K, unlike RMTPP, cannot model dependencies between the predicted K events. Despite this, RMTPP-K still performs strongly, particularly on the StackOverflow dataset.

## Appendix B T-mAP Computation and Proofs

Definitions and scope. In this section, we consider weighted bipartite graphs. The first part consists of predictions, and the second consists of ground truth events. We assume that all edges connected to the same prediction have the same weight. A matching is a set of edges $(a_{i},b_{i})$ between predictions and ground truth events. A matching is termed optimal if it (1) has the maximum size among all possible matching and (2) has the minimum total weight among all matches of that size. We denote all optimal matchings in the graph $\mathcal{G}$ as $M(\mathcal{G})$.

###### Lemma A .

Consider a graph $\mathcal{G}^{\prime}$, constructed from a graph $\mathcal{G}$ by adding one ground truth vertex with corresponding weighted edges. Then any optimal matching $m^{\prime}\in M(\mathcal{G}^{\prime})$ will either have a size greater than the sizes of matchings in $M(\mathcal{G})$ or have a total weight equal to that of matchings in $M(\mathcal{G})$.

###### Proof.

Figure: Figure 7: Illustration of the iterative process in Lemma A.
Refer to caption: x7.png

If the size of $m^{\prime}$ is greater than the size of matchings in $M(\mathcal{G})$ then the lemma holds. The optimal matching in the extended graph $\mathcal{G}^{\prime}$ cannot have a size smaller than the optimal matching in $\mathcal{G}$. Now, consider the case when both matchings have the same size. We will show that $m^{\prime}$ has a total weight equal to that of the matchings in $M(\mathcal{G})$.

Denote $b$ the new ground truth event in graph $\mathcal{G}^{\prime}$. If $m^{\prime}$ does not include vertex $b$, then $m^{\prime}$ is also optimal in $\mathcal{G}$, and the lemma holds. If $m^{\prime}$ includes vertex $b$, then we can walk through the graph with the following process:

- 1.
Start from the new vertex $b$.
- 2.
If we are at a ground truth event $b_{i}$ and there is an edge $(a_{i},b_{i})$ in matching $m^{\prime}$, move to the vertex $a_{i}$.
- 3.
If we are at a prediction $a_{i}$ and there is an edge $(a_{i},b_{i})$ in matching $m$, move to the vertex $b_{i}$.

Example processes are illustrated in Fig. 7. The vertices in each process do not repeat; otherwise, there would be two edges in either $m$ or $m^{\prime}$ with the same vertex, which contradicts the definition of a matching.

If the process finishes at a predicted event, then $m^{\prime}$ has a size greater than the size of $m$, and the lemma holds. If the process finishes at a ground truth event, we can replace a part of matching $m^{\prime}$ traced with a part of matching $m$. The resulting matching $\hat{m}$ will have a size equal to both $m$ and $m^{\prime}$ and will have a total weight equal to both matchings. This proves the lemma.
∎

###### Theorem 4.1 .

For any threshold $h$, there exists an optimal matching in the graph $\mathcal{G}_{h}$ such that it is a subset of an optimal matching in the full graph $\mathcal{G}$:

$$ $\forall h\forall m\in M(\mathcal{G})\exists m_{h}\in M(\mathcal{G}_{h}):m_{h} \subset m.$ (6) $$

###### Proof.

If the threshold is lower than any score $s^{p}_{i}$, then $\mathcal{G}_{h}=\mathcal{G}$, and $m_{h}=m$ satisfies the theorem. Otherwise, some predictions are filtered by the threshold and $\mathcal{G}_{h}\subset\mathcal{G}$.

Without the loss of generality, assume that the threshold is low enough to filter out only one prediction. Otherwise, we can construct a chain of thresholds $h_{1}<h_{2}<\dots<h$, with each subsequent threshold filtering an additional vertex, and apply the theorem iteratively.

Denote $i$ the only vertex present in $\mathcal{G}$ and filtered from $\mathcal{G}_{h}$. If there is no edge containing vertex $i$ in the matching $m(\mathcal{G})$, then $m(\mathcal{G})$ is also the optimal matching in $\mathcal{G}_{h}$, and theorem holds.

Consider the case when matching $m(\mathcal{G})$ contains an edge $(i,j)$. Let $\hat{m}_{h}=m(\mathcal{G})\setminus\{(i,j)\}$, i.e., the matching without edge $(i,j)$. If it is optimal for the graph $\mathcal{G}_{h}$, then it satisfies the theorem. Otherwise, there is an optimal matching $m_{h}$ with either (a) more vertices or (b) a smaller total weight than $\hat{m}_{h}$.

In (a), matching $m_{h}$ has a size greater than the size of $\hat{m}_{h}$. It follows, that $m_{h}$ has the maximum size in the full graph $\mathcal{G}$. If the total weight of $m_{h}$ is larger or equal to the optimal weight in $\mathcal{G}$, then it can not be less than the total weight of $\hat{m}_{h}$, which contradicts our assumption. At the same time, the total weight of $m_{h}$ can not be less than the optimal weight in the full graph $\mathcal{G}$. It follows that case (a) leads to a contradiction.

Consider the case (b) when the size of $m_{h}$ equals $\hat{m}_{h}$. If $m_{h}$ does not include vertex $j$, then both $m_{h}$ and $\hat{m}_{h}$ are optimal. Otherwise, we can remove vertex $j$ from $\mathcal{G}_{h}$ and construct a new optimal matching $m^{\prime}_{h}$ without $j$ by using Lemma A, and again, both matchings $m^{\prime}_{h}$ and $\hat{m}_{h}$ are optimal. This concludes the proof.
∎

Algorithm. Using the theorem, we can introduce an effective algorithm for T-mAP computation. T-mAP is defined on a batch of predictions $\{\mathcal{S}_{p}^{i}\}_{i=1}^{n}$ and ground truth sequences $\{\mathcal{S}_{gt}^{i}\}_{i=1}^{n}$. Let $\mathcal{S}^{i,l}_{gt}$ denote a subsequence of the ground truth sequence $\mathcal{S}^{i}_{gt}$ containing all events with label $l$. By definition, multiclass T-mAP is the average of the average precision (AP) values for each label $l$:

$$ $\mathrm{T\text{-}mAP}(\{\mathcal{S}_{p}^{i}\},\{\mathcal{S}_{gt}^{i}\})=\frac{ 1}{L}\sum\limits_{l=1}^{L}\mathrm{AP}(\{\mathcal{S}_{p}^{i}\},\{\mathcal{S}_{ gt}^{i,l}\}).$ (7) $$

Consider AP computation for a particular label $l$. AP is computed as the area under the precision-recall curve:

$$ $\mathrm{AP}=\sum\limits_{i}\left(\mathrm{Rec}_{i}-\mathrm{Rec}_{i-1}\right) \mathrm{Prec}_{i},$ (8) $$

where $\mathrm{Rec}_{i}$ is the $i$-th recall value in a sorted sequence and $\mathrm{Prec}$ is the corresponding precision. Iteration is done among all distinct recall values ($\mathrm{Rec}_{0}=0$).

We have several correct and incorrect predictions for each threshold $h$ on the predicted label probability. These quantities define precision and recall values along with the total number of ground truth events. A prediction is correct if it has an assigned ground truth event within the required time interval $|t^{p}_{i}-t_{j}^{gt}|\leq\delta$. Note that each target can be assigned to at most one prediction. Therefore, we define a matching, i.e., the correspondence between predictions and ground truth events. The theorem states that the maximum size matching for each threshold $h$ can be constructed as a subset of an optimal matching between full sequences $\{\mathcal{S}_{p}^{i}\}_{i=1}^{n}$ and $\mathcal{S}^{i,l}_{gt}$, where optimal matching is a solution of the assignment problem with a bipartite graph, defined in Section 4.2.

As the matching, without loss of generality, can be considered constant for different thresholds, we can split all predictions into two parts: those that were assigned a ground truth and those that were unmatched. The matched set constitutes potential true or false positives, depending on the threshold. The unmatched set is always considered a false positive. Similarly, unmatched ground truth events are always considered false negatives. The resulting algorithm for AP computation for each label $l$ involves the following steps:

- 1.
Compute the optimal matching between predictions and ground truth events.
- 2.
Collect (a) scores of matched predictions, (b) scores of unmatched predictions, and (c) the number of unmatched ground truth events.
- 3.
Assign a positive label to matched predictions and a negative label to unmatched ones.
- 4.
Evaluate maximum recall as the fraction of matched ground truth events.
- 5.
Find the area under the precision-recall curve for the constructed binary classification problem from item 3 and multiply it by the maximum recall value.

We have thus defined all necessary steps for T-mAP evaluation. Its complexity is $\mathcal{O}(LBN^{3})$, where $L$ is the number of classes, $B$ is the number of sequence pairs, and $N$ is the maximum length of predicted and ground truth sequences.

Calibration dependency. Calibration influences the weights assigned to the edges of the graph $\mathcal{G}$. T-mAP computation involves two key steps for each class label: matching and AP estimation. While average precision (AP) is invariant to monotonic transformations of predicted class logits, the matching step is only invariant to linear transformations. Specifically, the optimal matching seeks to minimize the total weight in the following form:

$$ $C=\operatorname*{arg\,min}\limits_{m\in M(\mathcal{G})}\sum\limits_{(i,j)\in m }(-s^{p}_{i}).$ (9) $$

A linear transformation of logits with a positive scaling factor will adjust the total weight accordingly, but the optimal matching will remain unchanged. Since the matching is performed independently for each class, we conclude that T-mAP is invariant to linear calibration.

###### Lemma A .

###### Proof.

###### Theorem 4.1 .

###### Proof.

## Appendix C HoTPP Benchmark Details

### C.1 Architecture

HoTPP incorporates best practices from extensible and reproducible ML pipelines. It leverages PyTorch Lightning (Falcon & The PyTorch Lightning team, ) as the core training library, ensuring reproducibility and portability across various computing environments. Additionally, HoTPP utilizes the Hydra configuration library (Yadan, ) to enhance extensibility. The overall architecture is illustrated in Figure 8. HoTPP supports both discrete-time and continuous-time models, as well as RNN and Transformer architectures. Implementing a new method requires only the addition of essential components, while the rest can be configured through Hydra and configuration files.

Figure: Figure 8: The architecture of the HoTPP library.
Refer to caption: x8.png

### C.2 Metrics

MTPP models are typically evaluated based on the accuracy of next-event predictions. Common metrics include mean absolute error (MAE) or root mean square error (RMSE) for time shifts, and accuracy or error rate for label predictions. Some studies also assess test set likelihood as predicted by the model, but we do not include this measure as it is intractable for IFTPP and Next-K models. Previous works have advanced long-horizon evaluation using optimal transport distance (OTD) (Mei et al., ; Xue et al., ). In addition to these metrics, we evaluate the novel T-mAP metric, which addresses the shortcomings of previous metrics, as discussed in Section 3.

### C.3 Backbones

We use three types of architectures in our experiments. The first is the GRU network (Cho et al., ), which is one of the top-performing neural architectures for event sequences (Babaev et al., ). We also implement the continuous-time LSTM (CT-LSTM) from the NHP method (Mei & Eisner, ). Since CT-LSTM requires a specialized loss function and increases training time, we use it exclusively with the NHP method, preferring GRU in other cases. An additional advantage of GRU is that its output is equal to its hidden state, simplifying the estimation of intermediate hidden states for autoregressive inference starting from the middle of a sequence. Lastly, we implement the AttNHP continuous-time transformer model (Yang et al., ).

### C.4 Datasets

For the first time, we combine domains such as financial transactions, social networks, and medical records into a single evaluation benchmark. Specifically, we provide evaluation results on a transactional dataset (Babaev et al., ), the MIMIC-IV medical dataset (Johnson et al., ), and social network datasets (Retweet (Zhao et al., ), StackOverflow (Jure, ), and Amazon (Jianmo, )). These datasets represent diverse underlying processes: social network data is influenced by cascades (Zhao et al., ), medical records exhibit repetitive patterns, and transactional data reflects daily activities, combining regularity with significant uncertainty.

The dataset statistics are presented in Table 2. The Transactions dataset has the longest average sequence length and the largest number of classes, while MIMIC-IV contains the highest number of sequences. Retweet is of medium size, whereas Amazon and StackOverflow can be considered small datasets.

Table: Table 2: Datasets statistics

Transactions(^3^33[https://huggingface.co/datasets/dllllb/age-group-prediction](https://huggingface.co/datasets/dllllb/age-group-prediction)), Retweet(^4^44[https://huggingface.co/datasets/easytpp/retweet](https://huggingface.co/datasets/easytpp/retweet)), Amazon(^5^55[https://huggingface.co/datasets/easytpp/amazon](https://huggingface.co/datasets/easytpp/amazon)), and StackOverflow(^6^66[https://huggingface.co/datasets/easytpp/stackoverflow](https://huggingface.co/datasets/easytpp/stackoverflow)) datasets were obtained from the HuggingFace repository. Transactions data was released in competition and come with a free license(^7^77[https://www.kaggle.com/competitions/clients-age-group/data](https://www.kaggle.com/competitions/clients-age-group/data)). Retweet, Amazon, and StackOverflow come with an Apache 2.0 license. MIMIC-IV is subject to PhysioNet Credentialed Health Data License 1.5.0, which requires ethical use of this dataset. Because of a complex data structure, we implement a custom preprocessing pipeline for the MIMIC-IV dataset.

Notes on MIMIC-IV preprocessing

MIMIC-IV is a publicly available electronic health record database that includes patient diagnoses, lab measurements, procedures, and treatments. We leverage the data preprocessing pipeline from EventStreamGPT to construct intermediate representations in EFGPT format. This process yields three key entities: a subjects data frame containing time-independent patient records, an events data frame listing event types occurring to subjects at specific timestamps, and a measurements data frame with time-dependent measurement values linked to the events data frame.

We combine events and measurements to create sequences of events for each subject. The classification labels are generally divided into two categories: diagnoses, represented by ICD codes, and event types, such as admissions, procedures, and measurements. A key challenge is that relying solely on diagnoses results in sequences that are too short, while using only event types leads to highly repetitive sequences dominated by periodic events, such as treatment start and finish.

However, using both diagnoses and event types together introduces another issue: diagnoses are sparsely distributed within a constant stream of repeated procedures, leading to imbalanced class distributions and poor performance. To mitigate this, we filter the data by removing duplicate events between diagnoses, allowing us to retain useful treatment data while preserving class balance.

The final labels are created by converting ICD-9 and ICD-10 codes to ICD-10 chapters using General Equivalence Mapping (GEM) and adding event types as additional classes. This conversion is necessary because the number of ICD codes is too large to use them directly as classes. Additionally, we address the issue of multiple diagnoses occurring in a single event by sorting timestamps for reproducibility.

### C.5 Performance improvements

The HoTPP benchmark provides highly optimized training and inference procedures for efficient evaluation on datasets containing up to tens of millions of events. First, we implement parallel RNN inference, which reuses computations when inference is initiated from multiple starting points within a sequence. Additionally, we optimize the code and apply PyTorch Script to the CT-LSTM model from NHP and the ODE model. CT-LSTM from EasyTPP and the ODE model from the original repository serve as baselines. The timing results are presented in Table 3. Experiments were conducted using a synthetic batch with a size of 64, a sequence length of 100, and an embedding dimension of 64. Evaluation was performed on a single Nvidia RTX 4060 GPU. The results show that HoTPP is 17 times faster at RNN autoregressive inference compared to simple prefix extension. HoTPP also accelerates CT-LSTM and ODE by 4 to 8 times. These optimizations significantly extend the applicability of the implemented methods to larger-scale datasets(^8^88Experiments are documented in the “notebooks” folder of the HoTPP repository). Additionally, HoTPP offers a GPU implementation of the Hungarian algorithm, which also finds applications in computer vision.

**Table 3: HoTPP computation speed improvements in terms of seconds per batch.**
| Implementation | Study |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| RNN autoreg.<br>inference | CT-LSTM<br>inference | CT-LSTM<br>train | ODE<br>inference | ODE<br>train |  |
| Baseline | 2.44 | 0.0158 | 0.0519 | 0.08 | 0.162 |
| HoTPP | 0.14 | 0.0025 | 0.0276 | 0.01 | 0.048 |

## Appendix D Training details

We trained each model using the Adam optimizer with a learning rate of 0.001 and a scheduler that reduces the learning rate by 20% after every 5 epochs. Gradient clipping was applied, with a maximum L2-norm set to 1.

We performed computations on NVIDIA V100 and A100 GPUs, with some smaller experiments conducted on an Nvidia RTX 4060. Each method was trained on a single GPU. The training time varied depending on the dataset and method, ranging from 5 minutes for RMTPP on the StackOverflow dataset to 40 minutes for the same method on the Transactions dataset, and up to 15 hours for NHP on Transactions. Multi-seed evaluations took approximately five times longer to complete.

## Appendix E Hyperparameters

We analyzed the behavior of OTD and T-mAP metrics depending on their parameters. The results are shown in Figure 9. The OTD cost has little effect on the ranking of methods but significantly influenced the metric scale. The choice of the $\delta$ parameter in T-mAP can, in some cases, affect method rankings, but as long as $\delta$ is not set too small or too large, the metric demonstrates stable rankings. We set the $\delta$ parameter to approximately 10-30% of the horizon duration.

Evaluation hyperparameters are presented in Table 4. Dataset-specific training parameters are listed in Table 5.

Table: Table 4: Evaluation hyperparameters.

Table: Table 5: Training hyperparameters.

Figure: Figure 9: The dependency of metric values on the metric parameter.
Refer to caption: x9.png

## Appendix F Labels Entropy Degradation

For simplicity, some datasets were omitted in the main part of the paper. In Figure 10, we show the dependency of label distribution entropy on step size for all datasets.

Figure: Figure 10: Entropy of label distributions as a function of the position in the generated sequence.
Refer to caption: x14.png

## Appendix G The Optimal Sequence Length

For simplicity, some datasets were omitted from the main part of the paper. Figure 11 illustrates the relationship between label distribution entropy and step size across all datasets.

Figure: Figure 11: T-mAP dependency on the maximum length of the predicted sequence.
Refer to caption: x15.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Could the authors provide more detailed reasoning on why T-mAP is the most suitable metric for long-horizon MTPP evaluation? A comparison with other metrics, such as OPW, would help clarify the unique advantages of T-mAP for certain datasets or applications. Are there specific scenarios where T-mAP particularly excels?

2. How does T-mAP adapt to different domains represented in the HoTPP benchmark, such as healthcare vs. social media? It would be helpful if the authors could provide additional domain-specific analysis or clarify if they observed any notable trends in metric performance across different datasets.

4. Given the computational intensity of some methods (e.g., continuous-time LSTM), what optimizations, if any, do the authors recommend for users with limited hardware resources? Would simplifying certain models or using hybrid methods maintain benchmark validity while improving accessibility?

4. Next-K models are briefly discussed, but can the authors elaborate on alternative structures or settings within this family of models? Exploring how these models perform differently across long-horizon tasks could provide insights into their benefits or limitations and would help clarify if more complex Next-K structures could outperform standard autoregressive models.
soundness: 3
strengths: 1. Innovative Evaluation Metric: The paper introduces Temporal mean Average Precision (T-mAP), a novel metric inspired by object detection, which addresses limitations in existing evaluation methods for long-horizon forecasting by accurately handling false positives and negatives, offering a refined measurement of model performance in Marked Temporal Point Processes (MTPP).
2. Practical Benchmark: The HoTPP benchmark developed in this work includes large-scale, diverse datasets and optimized inference procedures, establishing a standardized framework that supports both autoregressive and parallel inference, greatly enhancing research accessibility and reproducibility in long-horizon event forecasting.
3. Comprehensive Empirical Analysis: The paper rigorously evaluates various models, including rule-based and advanced neural methods, across multiple datasets, providing robust empirical evidence that reveals critical insights into the performance trade-offs of next-event vs. long-horizon forecasting.
4. Clear and Structured Presentation: The paper clearly articulates the challenges in long-horizon event prediction, explaining the proposed methodology and its advantages with illustrative figures and well-organized tables, making complex concepts acc
summary: This paper introduces HoTPP (Horizon Temporal Point Process Benchmark), a benchmark for evaluating long-horizon event sequence prediction models. Its main contributions include: proposing a novel evaluation metric called Temporal mean Average Precision (T-mAP), inspired by object detection metrics in computer vision, which properly handles variable-length sequences within a prediction horizon and accounts for false positives and false negatives; demonstrating through comprehensive analysis that models with high next-event prediction accuracy don't necessarily perform well at long-horizon forecasting, suggesting the need for specialized approaches for each task; developing the HoTPP benchmark, which includes large-scale datasets from diverse domains (finance, healthcare, social networks) with up to 43 million events, implementation of various modeling approaches (including rule-based baselines, intensity-free methods, intensity-based methods, and Next-K prediction models), optimized procedures for both autoregressive and parallel inference, and theoretical proof of T-mAP computation correctness; and revealing through extensive experiments across multiple datasets the trade-offs between next-event and long-horizon prediction performance, benefits of Next-K approaches for long-horizon predictions, importance of proper sequence length selection, and analysis of label distribution entropy degradation in autoregressive predictions.
weaknesses: 1. Limited Justification of Metric Selection: While the T-mAP metric is an innovative contribution, the paper could strengthen its justification for why T-mAP is superior to other established metrics in specific application scenarios. Including more detailed comparisons with alternative metrics, such as Order-preserving Wasserstein Distance (OPW), could provide further evidence of T-mAP's effectiveness, especially in complex event sequences.
2. Lack of Fine-Grained Analysis for Different Domains: The datasets span diverse fields like healthcare and social media. However, the paper does not profoundly explore how T-mAP performs within these domains. Analyzing domain-specific challenges or model performance variations across fields could add depth and highlight the metric’s adaptability, further demonstrating HoTPP’s real-world applicability.
3. Computational Constraints on Benchmark Implementation: Some models, like the continuous-time LSTM, require extensive computational resources, limiting their practical applicability. The paper could improve by suggesting or including optimizations, such as more efficient GPU implementations or leveraging hybrid models, making the benchmark more accessible to researchers with limited resources.
4. Limited Exploration of Next-K Models: Although the paper discusses Next-K models and their potential in improving long-horizon forecasting, there is little exploration of variations within this model family. Providing examples or implementing alternative Next-K structures could substantiate the claims regarding their advantages, offering actionable insights for researchers interested in non-autoregressive alternatives.
5. Lack of Qualitative Error Analysis: The paper could benefit from qualitative error analysis to clarify why some models underperform on long-horizon metrics. Visual examples or error case studies might offer valuable insights into prediction failures, guiding future model improvements by highlighting common error patterns in long-horizon event forecasting.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Deep methods do not necessarily outperform rule-based methods, how to illustrate the superiority of the proposed metric by comparing them?
2. What is the main message of section 4.3? It seems to only show that the metric increases monotonically with delta.
3. Can authors discuss which scenarios are most suitable for different types of metrics? Like how does the proposed metric handle sparse or highly irregular data compared to traditional ones?
soundness: 2
strengths: 1. The paper is well-organized, with clear sections delineating the introduction, related work, methodology, experiments, and conclusions.
2. Figures and tables are used effectively to illustrate concepts and present experimental results.
3. The significance of forecasting multiple future events is substantial.
summary: The paper introduces a new metric, Temporal mean Average Precision (T-mAP), which is designed for evaluating long-horizon predictions in marked temporal point processes (MTPP). This offers a new perspective on evaluating forecasting accuracy beyond traditional metrics like OTD. It also introduce HoTPP, a new benchmark for long-horizon forecasting, which provides large datasets and optimized procedures for both autoregressive and parallel inference.
weaknesses: 1. The motivation in section 3, while intended to address deficiencies in existing methods such as OTD, appears weakly articulated. The example provided in Figure 2 lacks clarity, particularly why OTD cannot correctly match the last triangle with the ground truth. Furthermore, the assertion that OTD is computed between fixed-size prefixes is confusing since n_p and n_gt vary, which contradicts the description in line 178 to 185.
2. Some claims are not clear. For example, in line 230 T-mAP identifies the matching that maximizes both precision and recall simultaneously. Typically, these metrics are in a trade-off relationship. How T-mAP manages to maximize both?
3. The experimental section (Section 6.1) lacks specific references to the methods and datasets discussed, which makes it difficult to follow and verify the stated findings. Direct references to specific methods and datasets should be included to enhance clarity.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: - How T-mAP is better than the existing metric? 

- More analysis on long-horizon event forecasting are needed. 

- Why the baselines are not new?
soundness: 2
strengths: - This paper proposes a new metric, namely T-mAP, to better evaluate the performance of long horizon events forecasting. 

- This paper includes large-scale datasets with up to 43 million events. 

- This paper release HoTPP open-source benchmark to facilitate future research.
summary: This paper benchmarks the task of long horizon events forecasting. The main contribution of this paper is to propose a new metric, namely T-mAP. Moreover, this paper also includes large-scale datasets with up to 43 million events. Although with these contributions, this paper is not well-written and not easy to understand.
weaknesses: - This paper is not well-written and hard to follow. The main contribution of this paper is to propose T-mAP to evaluate models on the task of long horizon events forecasting. However, why T-mAP is better than the existing metric, i.e., OTD, is not clearly introduced. 

- The experiments are not extensive. In the title of this paper is about long-horizon event forecasting, but the experiments covers equally on the topic of the next-item forecasting. 

- The number of compared baselines is not many, and the baselines are not proposed recently. It seems that the topic of event forecasting is not very hot, so the motivation that we need a benchmark is very strong.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 2
questions: See Weaknesses.
soundness: 2
strengths: The author's motivation is clearly expressed, the approach sounds interesting, the problem of modeling long sequences of events is well addressed, and the large dataset developed has the potential to advance the field.
summary: The authors introduce Temporal mean Average Precision (T-mAP), a temporal variant of mAP that overcomes the limitations of existing long-term evaluation metrics. They also release HoTPP, the first benchmark specifically designed to evaluate long-term MTPP predictions, which includes a large-scale dataset of up to 43 million events.
weaknesses: 1. The paper is difficult to understand. What is the ‘structured descriptions of each event’, what is the challenges of autoregressive prediction in line 48. What are the challenges of autoregressive prediction in line 052? How are subsequences handled for each event type in line 99. This increases the difficulty for readers to understand, and it is recommended that the author explain these terms in more detail.

2. The motivation is vague. Why do we need long-term event prediction? In fact, we can execute Next-event task multiple times to get similar results.

3. How does the proposed metrics perform in a long-tail prediction scenario?

4. Does the proposed metric take into account the time error of event occurrence? If the time interval corresponding to the event is far away, can the method cope with this situation?

Meta-review: The reviewers find the paper difficult to understand and raise many general questions that cannot be resolved in a rebuttal as most of them  require substantial changes to the original submission and in sum a major revision. For example, it does not become obvious why the solved problem is challenging and hence, why it is important? Are there use cases that would exemplify the need for long-term predictions? When the basic understanding of the problem setting is not given, it is almost impossible to understand the contribution. Reviewers also highlighted many other issues in their reviews, eg. regarding the handling of subsequences, empirical evaluation, etc. In sum, at the current stage, the paper is not yet ready for publication at ICLR.

Decision: Reject