Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose an approach to separate exogenous and endogenous events generated by marked temporal point processes (MTPPs). In this context, an event consists of a timestamp and a mark, which is assumed to be a category or class. Many MTPPs are self-exciting or self-inhibiting so that new events are generated due to the influence of previous events--these are endogenous events. Other events are considered to be exogenous events. Given a set of events, the authors proposed approach attempts to select a subset of exogenous events using a stochastic greedy approach. Along the way, they establish some important theoretical properties regarding monotonicity and submodularity of MTPPs that allows them to derive an approximation guarantee for their algorithm. They also demonstrate reasonable empirical results on simulated and real data sets.
main_review: **Originality: High**  
While the problem of separating exogenous and endogenous events in temporal point processes (TPPs) or MTPPs is not new, the proposed approach completely changes the way we think about exogenous and endogenous events in MTPPs. The typical approach, as the authors mention, is to assume that the conditional intensity function $\lambda(t)$ is the sum of exogenous and endogenous components, which are then treated as a single MTPP. This sum is then treated as the starting point to separate exogenous and endogenous.

The authors instead propose to think about exogenous and endogenous as separate MTPPs whose events are interleaved so that they look like they are generated from a single MTPP. They demonstrate the relationship between their approach and the usual sum approach in Appendix B.1. They exploit their new modeling approach to better classify exogenous and endogenous events.

The discussion of related work is fairly comprehensive. One missing reference is Li and Zha (2013), who use a mixture of linear Hawkes processes to classify events as being from different classes.

Reference:
Li, L., & Zha, H. (2013). Dyadic event attribution in social networks with mixtures of hawkes processes. Proceedings of the 22nd ACM international conference on Conference on information & knowledge management - CIKM '13, 1667-1672. doi:10.1145/2505515.2505609

**Quality: Moderate**  
The quality of the theoretical results is higher than the experimental results, in my opinion (although I did not carefully check proofs). Theorems 3-6 enable the authors to derive the approximation guarantee for their algorithm (Theorem 7), and as the authors note, it may be of independent interest.

The experimental results presented suggest that the proposed TPP-Select algorithm works, but does not establish superiority of TPP-Select compared to baselines at the proposed task: separating exogenous and endogenous events. The authors opt to present only qualitative simulation results while deferring quantitative results to Appendix E. I think this is a mistake. Furthermore, the authors don't compare error rates of different approaches with respect to classification of exogenous/endogenous; instead, they only evaluate prediction error rates.

I would suggest 2 additional experiments:
- On your current synthetic data, measure also classification error for TPP-Select and compare it to the EM and k-means baselines, which also attempt to identify the exogenous event set.
- On your real data experiments, assume that all events are endogenous and then inject (interleave) some fake events generated according to your exogenous model. Then evaluate classification error with respect to these fake events.

**Clarity: Moderate**  
The quality of the presentation is decent, with lots of minor issues as I list below. More importantly, I had to read through almost the entire paper before I understood the distinction between exogenous and endogenous that the authors are making. I would suggest showing a linear Hawkes process as an illustrative example in Section 2.2 to clearly show that $\mu$ corresponds to the exogenous component.

The choice of experiment results to include in the main paper and supplement could also be improved. I think Figure 1 and 2 should go to the supplement while something similar to Figure 3 should go into the main paper (see discussion above for improvements to the simulation experiments).

Minor issues:
- Definition of $\lambda(t)$ in (1) is a bit sloppy--it should be the limit of the stated quantity as $dt \rightarrow 0$.
- In equation (9), the subscript $\lambda$ on $\theta$ and $\kappa$ should be replaced with the black dot that is being used elsewhere.
- Line 211: MTTPs -> MTPPs
- Line 295: missing end ) in the first exp function

**Significance: High**  
This paper presents a novel way of thinking about TPPs. Given the large amount of research being done on TPPs recently (particularly non-linear and neural TPPs), I think this has the potential to be very impactful.

limitations_and_societal_impact: Both are discussed in the supplement. The discussion of limitations is somewhat brief and should be expanded upon. Another limitation that I noticed is that the proposed algorithm is a hard decision algorithm--unlike the EM algorithm baseline used in this paper or the variational inference approach in Li and Zha (2013), which output a probability that a given event is exogenous.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies a particular kind of temporal point process, with a focus on trying to identify exogenous events. The problem is formulated as a set function maximization, some theoretical results about monotonicity and alpha-sub-modularity are presented, a stochastic greedy algorithm (based on [38]) is proposed, and some experiments are conducted.
main_review: This paper makes some good theoretical contributions to the literature; it could have been (and could yet become) a stronger paper than its current form. I am left with the sense that there are innovative ideas here, but the results do not quite match up. I am open to upgrading my score after the rebuttal period, particularly if the authors can either convince me about some of their experiment-related choices, and/or make revisions. Let me begin with my major concerns about the paper.

Experiments:

I was enjoying the paper and was looking forward to giving it a strong accept, until I reached the experiments section, which made me change my mind instantly. I’m baffled by the authors’ choices. The paper is essentially about identifying exogeneous events, so I don’t understand why Section 5 delves into predicting next times and next labels. If they want to target these tasks, they are not considering the appropriate baseline temporal point process models. 

My conjecture is that the authors felt they needed to use real data, and since real data does not usually come with labels about exogeneous / endogenous, they decided to tackle the prediction task. In the process, they have left out relevant baselines for this task. I believe they were better off focusing on synthetic data and showing that their method is able to identify exogeneous events better than relevant baseline approaches. I am not convinced their model is better than the state-of-the-art for prediction (it probably isn’t, which is fine).

I have some clarification questions about the experiments:

1. What exactly are the baselines in 5.1 doing? If I understand the task correctly, it is to predict the next time/label given the history of events. So, at any event occurrence in the test set, each model predicts the next time and event label. Is that correct? Are all of the baseline models generative models, i.e. can they generate new data? And why do they need to identify the exogenous events if the task is merely prediction? I would really like to understand what the authors were thinking here. It seems this entire experiment should be done with synthetic data, where the task is to identify exogeneous vs. endogenous events.
2. Why did the authors not compare with more recent models such as neural Hawkes, etc. for Table 2?

Literature:

For a paper with so many references, I’m surprised the authors have left out a lot of relevant literature. First, they have left out literature on outliers in temporal point processes, which is very closely related to the submission. I would be interested to hear how the authors think their work is different, and why they did not consider some of these as suitable baselines. I list a few papers below about outliers and also missing data (which I think should also be mentioned):

Event Outlier Detection in Continuous Time: 
http://proceedings.mlr.press/v139/liu21g/liu21g.pdf

Detecting Anomalous Event Sequences with Temporal Point Processes:
https://arxiv.org/abs/2106.04465

Imputing missing events in continuous-time event streams: 
http://proceedings.mlr.press/v97/mei19a/mei19a.pdf

Learning Temporal Point Processes with Intermittent Observations:
http://proceedings.mlr.press/v130/gupta21a.html

The authors mention multivariate point processes right at the end, but do not refer to the vast amount of literature on graphical models for temporal point processes. Some of this work is a generalization of the model considered in the paper. Here are some references:

Universal Models of Multivariate Temporal Point Processes: 
http://proceedings.mlr.press/v51/gunawardana16.html

Proximal Graphical Event Models:
https://papers.nips.cc/paper/2018/file/f1ababf130ee6a25f12da7478af8f1ac-Paper.pdf

Also, I recommend citing the following paper, since it models exogenous events that affect state variables in a continuous-time Bayes net. Event-Driven Continuous Time Bayesian Networks:
https://ojs.aaai.org//index.php/AAAI/article/view/5725


Other comments/suggestions:

The term “externalities” is mentioned in the Abstract, with no explanation.

The description about the state-of-the-art in line 36 is inaccurate. Multivariate models do not consider these with “equal likelihood”. Please fix this line.

I recommend title case for section headings.

It may be worthwhile explaining the distribution choices in equation (4).

I found page 5 dense and hard to read. The work itself seems like a good contribution but its hard to follow. I suggest breaking it up further and adding more examples that clarify implications of some of the results. More discussion is needed here. Importantly, I think it’s very important to add more caveats about the specific MTPPs that are in scope for some of the results.

Equation (11) looks wrong for both parts. Lower and upper bounds are identical on the left, and t is used instead of y on the right.

Is k assumed known in algorithm 1? How is it determined in practice? Later on, it seems it has a major effect on results.

Can the authors say more about the subset sizes in step 3 of Algorithm 1? I understand that any subset is equally likely to be chosen. It seems that if small subsets are chosen, the algorithm is indeed efficient but performance could be poor. Did they try experiments where they controlled the subset size?

Figure 1 is not as easy to read as the authors perhaps intend. I could not spot the great results that the authors claim. I see both greener and redder as my eyes scan to the right. Perhaps a table with classification numbers would help here?

There is a mistake in lines 315-317: 7 datasets are implied here but only 5 were used.

There are several typos/grammatical errors. Here are some: line 120 (allows), 171 (variety), 184-186 (entire sentence needs editing), 332 (hyper-parameters), 335 (on in), etc.
limitations_and_societal_impact: The authors should say much more about scope of their work; please see comments in my main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for identifying hidden labels indicating whether each event has been triggered by exogenous or endogenous influence. The parameter estimation problem of temporal point processes with event selection procedures is reformulated as a set function optimization problem. To solve this problem, the authors propose an approximate algorithm. The effectiveness of the proposed model is demonstrated using synthetic and real-world datasets.
main_review: The major contribution of this paper is to bridge between the point process estimation problem and the submodularity and to develop a feasible optimization algorithm. This is technically solid. Thus, my opinion tends to accept. But there are some questions/suggestions as follows.

(1) In this paper, the exogenous events are regarded as those that are triggered by background intensities. They are defined as constants generated from probability distributions, as in Eq. (4). In a practical situation, one would like to model them as time-dependent functions such as periodic functions. Does the proposal allow such situations?

(2) Another option to efficiently infer triggering labels is the use of Gibbs sampler for latent variables (i.e., labels). Such methods are used for analyzing social and advertising effects, as in [R1, R2, R3]. They can also discern the events that are triggered by background intensity. These studies might be related to those that are cited in "Modeling external effects" of Section 1.3. It would be great to add a discussion about the differences and advantages of the proposal against these prior works. 

[R1] T. Iwata, A. Shah, and Z. Ghahramani. Discovering Latent Influence in Online Social Activities via Shared Cascade Poisson Processes, KDD, 2013.

[R2] Scott W. Linderman, et al., Discovering Latent Network Structure in Point Process Data, ICML, 2014.

[R3] Y. Tanaka, et al., Inferring Latent Triggers of Purchases with Consideration of Social Effects and Media Advertisements, WSDM, 2016.
limitations_and_societal_impact: Please see comments (1) and (2) in my main review.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper aims at identifying the set of exogenous events from a set of unlabeled events for learning marked temporal point process. This paper shows solving the exogenous event set selection problem, in conjunction with parameter estimation problem, can be achieved by maximizing a monotone and alpha-submodular set function and then proposes TPP-SELECT framework. The experiments on both synthetic and real data show the proposed method performs better than baselines.
main_review: 1. Overall, this paper is well organized and explains the framework clearly.
2. This paper targets at a very important and interesting problem, i.e., identifying the set of exogenous events without labels for marked temporal point processes. And it’s novel that applying submodular set function method to this problem, although it seems to be a relatively straightforward application.
3. The theoretical analysis is quite thorough.

Update:
Thank the authors for their comments. Most of my concerns are addressed. I will keep my score.
limitations_and_societal_impact: 1. The current method doesn’t consider the triggering patterns among endogenous events, while there always exists a branch process among them. I wonder to know if this work can be extended to deal with such structure.
2. In Table 1, most of the mark prediction errors are larger than 0.5. Does there exist sample imbalance in those data set? Maybe ROC or AOC is a more suitable metric? I also suggest the authors test their method on data with different ratios of exogenous events to endogenous events.
3. There are some errors in Table 2: (1) Base MTPP achieves better results on Election data for time prediction error and on Club data for mark prediction error; (2) Base MTPP gets the same results with TPP-SELECT on BookOrder data for time prediction error. The authors should be more careful for the experimental results.
4. From Table 2, it seems base MTPP achieves comparable results with TPP-SELECT. However, I think base MTPP is much more efficient than TPP-SELECT. I suggest the authors provide further comparisons on convergence rate and run-time.
review_title: null

Meta-review: The submission provides useful theory and experimental support for methods to label Marked Temporal Point Processes. There was extensive engagement between the authors and reviewers, and the reviewers came away well satisfied by the extended experiments. We ask that the authors please update the paper to incorporate as much of the useful additional exposition as possible, updating the main paper where possible or including supplemental material where appropriate. 
Decision: Accept (Poster)