Rare-Mark-Aware Next Event Prediction In Marked Event Streams

Authors: Sishun Liu, KE DENG, Yongli Ren, Yan Wang, Xiuzhen Zhang
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Marked Temporal Point Process

Abstract:
In marked event streams, Marked Temporal Point Process (MTPP) is central to predicting when and what mark the next event will occur based on the history. In various real-world applications, the mark distribution is significantly imbalanced, i.e., some marks are frequent, and others are rare. We unveil that such imbalance can cause the rare mark missing issue when predicting the next event – frequent marks are dominant, and rare marks often have no chance. However, rare marks can be essential in some applications (e.g., the occurrence of a 7-magnitude earthquake), and missing such rare marks in the next event prediction is risky. To address this issue, we tackle a novel Rare-mark-aware Next Event Prediction problem (RM-NEP), answering two questions for each mark m: “what is the probability that the mark of the next event is m?, and if m, when will the next event happen?”. Solving RM-NEP gives rare marks equal opportunity as frequent marks in the next event prediction. This guarantees that rare marks are always included in the predicted results. Moreover, RM-NEP allows arbitrary number of rare marks samples for time prediction without interference from frequent marks, ensuring the time prediction is accurate. To solve RM-NEP effectively, we first unify the improper integration of two different functions into one and then develop a novel Integral-free Neural Marked Temporal Point Process (IFNMTPP) to approximate the target integral directly. Extensive experiments on real-world and synthetic datasets demonstrate the superior performance of our solution for RM-NEP against various baselines.

Decision: reject
Reviewer scores: 3, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1730452987522

thread_type:
official_review

<thread_object>
creator: reviewer_odxb

<content>
summary:
This paper focuses on utilizing Marked Temporal Point Process (MTPP) models to address the Next-event Prediction (NEP) problem. It highlights a primary challenge of NEP: the imbalanced distribution of mark types. To address this, the paper introduces a new problem, Rare-mark-aware Next Event Prediction (RM-NEP), which is designed to ensure that rare marks consistently appear in prediction results. The paper also presents a novel IFNMTPP model to resolve issues related to inadequate integration over infinite time intervals when estimating the probability of marks and their timing in RM-NEP.

strengths:
1. The proposed RM-NEP problem offers fresh insights into the NEP challenge and the field of MTPP, presenting a potentially effective solution for addressing the issue of imbalanced mark types.
2. The paper is well-written, with a clear and fluent presentation of the NEP problem, its challenges, and the proposed solution.
3. The IFNMTPP model is straightforward in its design, with empirical studies demonstrating its superior efficiency.

weaknesses:
1. Although the primary focus of the paper is on accurately predicting rare mark types, Table 3 suggests that IFNMTPP does not show significant superiority in mark prediction performance. Instead, its strengths appear more pronounced in time prediction and efficiency. The paper could benefit from more detailed experimental analysis regarding the accuracy of predicting rare mark types.
2. The illustration depicting the architecture of IFNMTPP could be refined to provide a clearer demonstration of its design.

questions:
1. (Related to W1) Could the authors elaborate on how their experimental results empirically demonstrate the effectiveness of the proposed RM-NEP problem and IFNMTPP model in addressing the issue of missing rare marks in NEP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730473695758

thread_type:
official_review

<thread_object>
creator: reviewer_o69x

<content>
summary:
This paper investigates how to reduce the problem of rare mark missing when event prediction is imbalanced, thereby reducing the risk of missing key events. The paper provides a detailed description of the proposed IFNMTPP method and conducts comparative experiments on multiple datasets. Results show the performance of IFNMTPP.

strengths:
S1: The paper studies the RM-NEP problem, unifies abnormal integrals, and proposes IFNMTPP to ensure that the prediction results of rare marks are not missed when the marks are imbalanced.
S2. The paper is well-articulated, offering a clear explanation of the concepts and methodologies employed.
S3. Extensive experiments on real-world and synthetic datasets demonstrate the effectiveness of the proposed method.

weaknesses:
W1. The purpose of this article is to improve the prediction accuracy of rare events. According to the experimental results of macro-F1 in Table 3, there is a slight improvement in the prediction accuracy of rare marks. In addition, earthquakes are unlikely to be accurately predicted through event prediction. Both the accuracy of frequent marks and rare marks before and after improvement are very low. Does this study have practical application value?
W2. Figure 2 is not very clear. It is recommended to refine it. The symbols inside are not consistent with the description in the text, such as v, s, and f.
W3. Incorrect punctuation is used in line 20 and line 78.

questions:
Q1: BookOrder's mark type [1] account for over 40%. Does this meet the definition of the rare mark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730618832451

thread_type:
official_review

<thread_object>
creator: reviewer_zlck

<content>
summary:
This paper makes a substantial contribution to the field of MTPPs by addressing the rare mark missing issue and providing a computationally efficient solution through IFNMTPP. The work is theoretically robust, empirically validated, and has practical significance in domains where rare events play a critical role.

strengths:
1.The paper provides a thorough theoretical foundation for the RM-NEP problem, including detailed derivations of the probability distributions and the integral-free approximation.

2.This paper proposes a novel approach, IFNMTPP, which avoids the computational burden of traditional numerical integration methods (e.g., Monte Carlo integration) by directly approximating the integral using a neural network. This is a computationally efficient solution that enables the model to handle large-scale datasets.

3.The authors conduct extensive experiments on various datasets, showing that their approach consistently outperforms existing baselines. The empirical results are strong and demonstrate the practical utility of the proposed method.

weaknesses:
W1: If I understand correctly, RM-NEP assumes that rare marks can be predicted accurately by decoupling time and mark prediction. Does this assumption hold across different types of datasets, especially when the marks exhibit temporal correlations?

W2: The IFNMTPP model approximates improper integrals using a "monotonically decreasing neural network." However, the paper does not provide sufficient details about how this approximation is performed, nor does it explain the intuition behind why a monotonically decreasing function is appropriate.

questions:
Q1: How interpretable are the results of RM-NEP, particularly for rare marks? Does the neural network-based approximation provide any insight into why a rare mark might be predicted? 

Q2: The paper focuses on marked temporal point processes where marks are categorical. How well does the proposed method generalize to cases where the marks are continuous.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678959198

thread_type:
official_review

<thread_object>
creator: reviewer_i8f9

<content>
summary:
This paper solves a problem in marked event prediction when the distribution of marks is significantly imbalanced i.e., some marks are frequent, and others are rare. The paper introduces a problem namely Rare-mark-aware Next Event Prediction (RM-NEP) and solves the problem to answer two questions: “what is the probability that the mark of the next event is m? and if m, when will the next event happen?”. Solving RM-NEP gives rare marks equal opportunity as frequent marks in the next event prediction. This guarantees that rare marks are always included in the predicted results. To solve RM-NEP effectively, the authors first unify the improper integration of two different functions into one and then develop a novel Integral-free Neural Marked Temporal Point Process (IFNMTPP) to approximate the target integral directly.

strengths:
1.	The problem is interesting.
2.	The Figures are intuitive.

weaknesses:
1.	The main difference between the problem of the paper and the existing problem is not clear. For example, what are the differences between RM-NEP and rare event forecasting? 
2.	The motivation of RM-NEP is not convincing. (i) If a mark is rare (i.e., it occurs very few times in the history). Then, it can be dominated by frequent marks in the prediction. This phenomenon is completely normal.  (ii) If a mark is rare and important compared to other marks, why don’t we only consider that mark as a single variable so that there is no imbalance anymore?
3.	The paper is not self-contained. For example, how the existing studies solve NEP is not clear. The authors only list a large number of papers in the Related Work section. Similarly, how the existing studies model MTPP is not clear. The authors only list a large number of papers in the Introduction section. A summarization and comparison are needed to provide a better understanding.
4.	Some words are hard to understand. For example, RMTPP is not defined.
5.	Some notations are not defined. For example, what is $\tau$?
6.	Intuitively, can we solve the problem by undersampling dominating marks?
7.	I cannot understand lines 297-299. If t=t_l then the integration equals 0.
8.	The main idea of using integral-free comes from FullyNN by using IEM. Basically, the authors adapt it to marked events, which is straightforward. 
9.	The authors do not prove why using IEM can achieve the integral-free solution.
10.	There is no ablation study. For example, what is the performance of TFNMTPP with different imbance ratios?

questions:
1.	If a mark is rare and important compared to other marks, why don’t we only consider that mark as a single variable such that there is no imbalancing anymore.
2.	What is the performance of TFNMTPP with different imbance ratios.
3.	Intuitively, can we solve the problem by undersampling dominating marks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533977994

thread_type:
official_comment

<thread_object>
creator: area_chair_bkt8

<content>
title:
Acknowledge the author responses

comment:
Dear Reviewers,

Thank you very much for your effort. As the discussion period is coming to an end, please acknowledge the author responses and adjust the rating if necessary.

Sincerely,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732693873978

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response

comment:
We appreciate all four thorough reviews of our paper. Based on these reviews, we made the following changes to our paper (all line numbers and section numbers correspond to the revised version):

- As a response to the comment of reviewer i8f9, o69X, and oDxB. We updated Figure 2 in section 3.2 "Integral-Free Neural Marked Temporal Point Process (IFNMTPP)" by including more structural details of IFNMTPP.
- To address the reviewer's concern about the marginal performance improvement of our method in mark prediction, we rewrite section 4.2 to provide more analyses of the results in Table 3.
- As a response to weakness 3 of reviewer i8f9, we replaced citations in the Introduction section with a review paper by Shchur et al. We also changed the Preliminary section from section 3 to section 2 and placed the Related Work section after the Experiment section for better readability.
- As a response to weakness 4 of reviewer i8f9, we add details for RMTPP (Recurrent Marked Temporal Point Process). Specifically, we added the full name of RMTPP when we first mentioned it in line 339 and introduced it in the Related Work section, starting from line 496.
- As a response to weakness 5 of reviewer i8f9, we explain the meaning of $\tau$ at line 118 by adding "where $\tau$ means time."
- As a response to weakness 7 of reviewer i8f9, we polished the paragraph under Equation (8) from line 257 to clarify that we are discussing $\Gamma^*(m, t)$, not $F^*(m, t)$.
- As a response to weakness 2 of reviewer ZLCK, we explain why $\Gamma^*(m, t)$ is a monotonically decreasing function at line 258 by adding "$\Gamma^*(m, t)$ is monotonically decreasing as its derivative $-p^*(m, t)$ is always smaller than 0." We also modified the paragraph starting from line 288 to explain how IFNMTPP approximates $\Gamma^*(m, t)$ and why a monotonically decreasing function is appropriate for IFNMTPP.
- As a response to weakness 3 of reviewer o69X, we have removed the comma after the question mark in "...what is the probability that the mark of the next event is $m$? and if $m$, when will the next event happen?..." at line 20, line 78, line 195, and line 535.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732797808328

thread_type:
official_comment

<thread_object>
creator: area_chair_bkt8

<content>
title:
Discussion needed

comment:
Dear Reviewers,

As you are aware, the discussion period has been extended until December 2. Therefore, I strongly urge you to participate in the discussion as soon as possible if you have not yet had the opportunity to read the authors' response and engage in a discussion with them. Thank you very much.

Sincerely,
Area Chair
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a rare event forecasting problem, RM-NEP, in the context of the marked temporal point process (MTPP). The reviewers agreed that the paper is well-written and the problem is very interesting.  However, they also raised several concerns.  Most commonly, the performance improvement is not very impressive.  Regarding the novelty, the reviewers pointed out that RM-NEP is not significantly different from the MTPP.  Although the authors did not agree with this point, I think that the reviewers made a reasonable point.  This paper is indeed a borderline paper.  However, in my batch, there are quite sufficient papers which are strongly supported by the reviewers.  Thus, I would like to recommend a reject.  If there is room, the senior AC can change my recommendation.