HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?

Authors: Ivan Karpukhin, Foma Shipilov, Andrey Savchenko
Primary area: datasets and benchmarks
Keywords: Event Sequences, Marked Temporal Point Processes, Long Horizon Forecasting, Evaluation Metric, Benchmark

Abstract:
Accurately forecasting multiple future events within a given time horizon is crucial for applications in finance, retail, social networks, and healthcare. Event timing and labels are typically modeled using Marked Temporal Point Processes (MTPP), with evaluations often focused on next-event prediction quality. While some studies have extended evaluations to a fixed number of future events, we demonstrate that this approach leads to inaccuracies in handling false positives and false negatives. To address these issues, we propose a novel evaluation method inspired by object detection techniques from computer vision. Specifically, we introduce Temporal mean Average Precision (T-mAP), a temporal variant of mAP, which overcomes the limitations of existing long-horizon evaluation metrics. Our extensive experiments demonstrate that models with strong next-event prediction accuracy can yield poor long-horizon forecasts, and vice versa, indicating that specialized methods are needed for each task. To support further research, we release HoTPP, the first benchmark specifically designed for evaluating long-horizon MTPP predictions. HoTPP includes large-scale datasets with up to 43 million events and provides optimized procedures for both autoregressive and parallel inference, paving the way for future advancements in the field.

Decision: reject
Reviewer scores: 5, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730185879761

thread_type:
official_review

<thread_object>
creator: reviewer_z31z

<content>
summary:
The authors introduce Temporal mean Average Precision (T-mAP), a temporal variant of mAP that overcomes the limitations of existing long-term evaluation metrics. They also release HoTPP, the first benchmark specifically designed to evaluate long-term MTPP predictions, which includes a large-scale dataset of up to 43 million events.

strengths:
The author's motivation is clearly expressed, the approach sounds interesting, the problem of modeling long sequences of events is well addressed, and the large dataset developed has the potential to advance the field.

weaknesses:
1. The paper is difficult to understand. What is the ‘structured descriptions of each event’, what is the challenges of autoregressive prediction in line 48. What are the challenges of autoregressive prediction in line 052? How are subsequences handled for each event type in line 99. This increases the difficulty for readers to understand, and it is recommended that the author explain these terms in more detail.

2. The motivation is vague. Why do we need long-term event prediction? In fact, we can execute Next-event task multiple times to get similar results.

3. How does the proposed metrics perform in a long-tail prediction scenario?

4. Does the proposed metric take into account the time error of event occurrence? If the time interval corresponding to the event is far away, can the method cope with this situation?

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730462662055

thread_type:
official_review

<thread_object>
creator: reviewer_nkzj

<content>
summary:
This paper benchmarks the task of long horizon events forecasting. The main contribution of this paper is to propose a new metric, namely T-mAP. Moreover, this paper also includes large-scale datasets with up to 43 million events. Although with these contributions, this paper is not well-written and not easy to understand.

strengths:
- This paper proposes a new metric, namely T-mAP, to better evaluate the performance of long horizon events forecasting. 

- This paper includes large-scale datasets with up to 43 million events. 

- This paper release HoTPP open-source benchmark to facilitate future research.

weaknesses:
- This paper is not well-written and hard to follow. The main contribution of this paper is to propose T-mAP to evaluate models on the task of long horizon events forecasting. However, why T-mAP is better than the existing metric, i.e., OTD, is not clearly introduced. 

- The experiments are not extensive. In the title of this paper is about long-horizon event forecasting, but the experiments covers equally on the topic of the next-item forecasting. 

- The number of compared baselines is not many, and the baselines are not proposed recently. It seems that the topic of event forecasting is not very hot, so the motivation that we need a benchmark is very strong.

questions:
- How T-mAP is better than the existing metric? 

- More analysis on long-horizon event forecasting are needed. 

- Why the baselines are not new?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730599768364

thread_type:
official_review

<thread_object>
creator: reviewer_adkc

<content>
summary:
The paper introduces a new metric, Temporal mean Average Precision (T-mAP), which is designed for evaluating long-horizon predictions in marked temporal point processes (MTPP). This offers a new perspective on evaluating forecasting accuracy beyond traditional metrics like OTD. It also introduce HoTPP, a new benchmark for long-horizon forecasting, which provides large datasets and optimized procedures for both autoregressive and parallel inference.

strengths:
1. The paper is well-organized, with clear sections delineating the introduction, related work, methodology, experiments, and conclusions.
2. Figures and tables are used effectively to illustrate concepts and present experimental results.
3. The significance of forecasting multiple future events is substantial.

weaknesses:
1. The motivation in section 3, while intended to address deficiencies in existing methods such as OTD, appears weakly articulated. The example provided in Figure 2 lacks clarity, particularly why OTD cannot correctly match the last triangle with the ground truth. Furthermore, the assertion that OTD is computed between fixed-size prefixes is confusing since n_p and n_gt vary, which contradicts the description in line 178 to 185.
2. Some claims are not clear. For example, in line 230 T-mAP identifies the matching that maximizes both precision and recall simultaneously. Typically, these metrics are in a trade-off relationship. How T-mAP manages to maximize both?
3. The experimental section (Section 6.1) lacks specific references to the methods and datasets discussed, which makes it difficult to follow and verify the stated findings. Direct references to specific methods and datasets should be included to enhance clarity.

questions:
1. Deep methods do not necessarily outperform rule-based methods, how to illustrate the superiority of the proposed metric by comparing them?
2. What is the main message of section 4.3? It seems to only show that the metric increases monotonically with delta.
3. Can authors discuss which scenarios are most suitable for different types of metrics? Like how does the proposed metric handle sparse or highly irregular data compared to traditional ones?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707058840

thread_type:
official_review

<thread_object>
creator: reviewer_x3ut

<content>
summary:
This paper introduces HoTPP (Horizon Temporal Point Process Benchmark), a benchmark for evaluating long-horizon event sequence prediction models. Its main contributions include: proposing a novel evaluation metric called Temporal mean Average Precision (T-mAP), inspired by object detection metrics in computer vision, which properly handles variable-length sequences within a prediction horizon and accounts for false positives and false negatives; demonstrating through comprehensive analysis that models with high next-event prediction accuracy don't necessarily perform well at long-horizon forecasting, suggesting the need for specialized approaches for each task; developing the HoTPP benchmark, which includes large-scale datasets from diverse domains (finance, healthcare, social networks) with up to 43 million events, implementation of various modeling approaches (including rule-based baselines, intensity-free methods, intensity-based methods, and Next-K prediction models), optimized procedures for both autoregressive and parallel inference, and theoretical proof of T-mAP computation correctness; and revealing through extensive experiments across multiple datasets the trade-offs between next-event and long-horizon prediction performance, benefits of Next-K approaches for long-horizon predictions, importance of proper sequence length selection, and analysis of label distribution entropy degradation in autoregressive predictions.

strengths:
1. Innovative Evaluation Metric: The paper introduces Temporal mean Average Precision (T-mAP), a novel metric inspired by object detection, which addresses limitations in existing evaluation methods for long-horizon forecasting by accurately handling false positives and negatives, offering a refined measurement of model performance in Marked Temporal Point Processes (MTPP).
2. Practical Benchmark: The HoTPP benchmark developed in this work includes large-scale, diverse datasets and optimized inference procedures, establishing a standardized framework that supports both autoregressive and parallel inference, greatly enhancing research accessibility and reproducibility in long-horizon event forecasting.
3. Comprehensive Empirical Analysis: The paper rigorously evaluates various models, including rule-based and advanced neural methods, across multiple datasets, providing robust empirical evidence that reveals critical insights into the performance trade-offs of next-event vs. long-horizon forecasting.
4. Clear and Structured Presentation: The paper clearly articulates the challenges in long-horizon event prediction, explaining the proposed methodology and its advantages with illustrative figures and well-organized tables, making complex concepts acc

weaknesses:
1. Limited Justification of Metric Selection: While the T-mAP metric is an innovative contribution, the paper could strengthen its justification for why T-mAP is superior to other established metrics in specific application scenarios. Including more detailed comparisons with alternative metrics, such as Order-preserving Wasserstein Distance (OPW), could provide further evidence of T-mAP's effectiveness, especially in complex event sequences.
2. Lack of Fine-Grained Analysis for Different Domains: The datasets span diverse fields like healthcare and social media. However, the paper does not profoundly explore how T-mAP performs within these domains. Analyzing domain-specific challenges or model performance variations across fields could add depth and highlight the metric’s adaptability, further demonstrating HoTPP’s real-world applicability.
3. Computational Constraints on Benchmark Implementation: Some models, like the continuous-time LSTM, require extensive computational resources, limiting their practical applicability. The paper could improve by suggesting or including optimizations, such as more efficient GPU implementations or leveraging hybrid models, making the benchmark more accessible to researchers with limited resources.
4. Limited Exploration of Next-K Models: Although the paper discusses Next-K models and their potential in improving long-horizon forecasting, there is little exploration of variations within this model family. Providing examples or implementing alternative Next-K structures could substantiate the claims regarding their advantages, offering actionable insights for researchers interested in non-autoregressive alternatives.
5. Lack of Qualitative Error Analysis: The paper could benefit from qualitative error analysis to clarify why some models underperform on long-horizon metrics. Visual examples or error case studies might offer valuable insights into prediction failures, guiding future model improvements by highlighting common error patterns in long-horizon event forecasting.

questions:
1. Could the authors provide more detailed reasoning on why T-mAP is the most suitable metric for long-horizon MTPP evaluation? A comparison with other metrics, such as OPW, would help clarify the unique advantages of T-mAP for certain datasets or applications. Are there specific scenarios where T-mAP particularly excels?

2. How does T-mAP adapt to different domains represented in the HoTPP benchmark, such as healthcare vs. social media? It would be helpful if the authors could provide additional domain-specific analysis or clarify if they observed any notable trends in metric performance across different datasets.

4. Given the computational intensity of some methods (e.g., continuous-time LSTM), what optimizations, if any, do the authors recommend for users with limited hardware resources? Would simplifying certain models or using hybrid methods maintain benchmark validity while improving accessibility?

4. Next-K models are briefly discussed, but can the authors elaborate on alternative structures or settings within this family of models? Exploring how these models perform differently across long-horizon tasks could provide insights into their benefits or limitations and would help clarify if more complex Next-K structures could outperform standard autoregressive models.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732628525286

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for spending time and providing detailed feedback. We recognize that the main concerns focus on the proposed T-mAP metric. We address these concerns with additional empirical results and clarifications. We also want to emphasize other contributions of our work, which we think were not properly recognized.

**Theoretical Contributions:**

1. We identify and address the limitations of the OTD and next-event metrics in the MTPP domain, namely incorrect accounting for false positives and false negatives, and sensitivity to calibration (Section 3). The latter is important limitation, as calibration is omitted in previous works [1, 2, 3].
2. We introduce the T-mAP metric, which resolves these limitations (Section 4).
3. We prove matching consistency among decision thresholds in the T-mAP computation algorithm, a type of proof that is notably absent in computer vision [4]. Additionally, we demonstrate the invariance of T-mAP to linear calibration (Appendix B).

**Technical Contributions:**

1. We implemented parallel inference starting from multiple points within the input sequence, which is crucial for long-horizon evaluation. In Appendix C.5 we demonstrate, that our parallel autoregressive RMTPP inference on StackOverflow is 12 times faster than multiple inference calls with different prefix sizes.
2. We optimized continuous-time models using PyTorch JIT, achieving substantial speed improvements. As shown in Appendix C.5, our CT-LSTM is twice as fast as EasyTPP, and our ODE-RNN is 4 times faster than the original implementation \[3\].
3. We provided a GPU-accelerated linear sum assignment solver, that attracted the attention of computer vision researchers. In CV, this algorithm is typically executed on the CPU. Our GPU implementation is about 16 times faster than CPU in our experiments.

**Empirical Contributions:**

1. We for the first time apply advanced methods like NHP and ODE-RNN [3] to large datasets such as MIMIC-IV and Transactions. The latter is 30 times larger than previously studied datasets, like Retweet. Large datasets present new challenges in computational efficiency, which we addressed above. Dataset statistics are described in Appendix C.4.
2. We evaluate Next-K approaches for MTPP for the first time, showing that they are competitive with autoregressive methods while offering higher computational efficiency. For example, on the StackOverflow dataset, sequence generation with IFTPP-K is four times faster compared to IFTPP.

**T-mAP motivation**

We provide a theoretical foundation for T-mAP in Section 3, demonstrating that it resolves issues in accounting for false positives and false negatives inherent in next-event and OTD metrics. Additionally, we show that, unlike label error rate and OTD, T-mAP is invariant to linear calibration—a crucial property given that most prior works do not incorporate calibration. Our experiments further reveal that next-event and OTD metrics fail to capture model confidence effectively, often favoring simplistic copy-and-paste baselines. These metrics also struggle to reflect the long-horizon prediction improvements achieved by HYPRO. Furthermore, we added Appendix H, which illustrates T-mAP's ability to evaluate highly irregular event sequences, and Appendix I, which highlights its importance in assessing long-tail prediction. Together, these findings underscore our central claim that T-mAP is a more robust and sensitive metric for evaluating long-horizon predictions compared to existing alternatives.

**Domain and benchmark importance**

Interest in event sequence modeling continues to grow [1], yet the field remains underexplored, largely due to the sensitive nature of the data involved. Event sequences are common in domains such as finance (e.g., banking), retail, and healthcare, where strict data privacy requirements often necessitate extensive anonymization. This limits the availability of datasets for research and hinders progress in the field. Despite these challenges, modeling event sequences is essential for these industries. Accurate models enable critical applications, including strategic planning, personalized communication, and advanced analytics. These capabilities provide significant benefits, such as financial gains in finance and retail or improved diagnostic accuracy in healthcare. To address these needs, our benchmark sets a new standard by emphasizing speed, scalability, reproducibility, and a diverse range of evaluation tasks, being the first benchmark for long-horizon prediction in the field.

[1] Xue S. et al. “EasyTPP: Towards Open Benchmarking Temporal Point Processes”, ICLR 2024

[2] Xue S. et al. “Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences”, NeurIPS 2022

[3] Rubanova Y. et al. “Latent ordinary differential equations for irregularly-sampled time series” NeurIPS 2019

[4] Lin T. Y. et al. “Microsoft CoCo: Common objects in context”, ECCV 2014
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers find the paper difficult to understand and raise many general questions that cannot be resolved in a rebuttal as most of them  require substantial changes to the original submission and in sum a major revision. For example, it does not become obvious why the solved problem is challenging and hence, why it is important? Are there use cases that would exemplify the need for long-term predictions? When the basic understanding of the problem setting is not given, it is almost impossible to understand the contribution. Reviewers also highlighted many other issues in their reviews, eg. regarding the handling of subsequences, empirical evaluation, etc. In sum, at the current stage, the paper is not yet ready for publication at ICLR.