Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose an approach for estimating counterfactuals in temporal point processes.

In the proposed approach, a temporal point process is modeled using the Gumbel-Max structural causal model under the assumptions of monotonicity and binary random variables. The error terms in this model are all independent of each other, which implies Markovianity, i.e., no unmeasured confounders). Under those assumption, counterfactuals are always identifiable. 

The authors' contribution is a sampling algorithm based on the Lewis' thinning algorithm and the  superposition theorem. Lewis’ thinning algorithm is a popular method to sample from the a temporal point process. They authors use the superposition theorem to sample plausible sequences of events that the thinning process would have rejected if it had accepted the observed sequence of real events. The proposed sampling algorithm was evaluated using both synthetic and real epidemiological data.

strengths_and_weaknesses: The paper addresses the important and relevant problem of identifying and estimating counterfactuals in temporal point processes. 

However some assumptions are not clearly stated in the paper. The assumption that the $U_i$ variables are jointly independent noise is equivalent to the assumption that the system is Markovian (i.e., absence of unmeasured confounders between all pairs of variables). Moreover, the proposed model makes the assumption that $U_i$ is an additive noise. 

Those assumptions are not part of the general definition of an SCM (as in Pearl, 2009), but are being used in the proposed sampling algorithm. They must be clearly stated as additional assumptions of the method and a their plausibility should be discussed.  Those are strong assumptions, most likely violated in many practical setting, so it is a critical limitation of the method. Further, the empirical study should address the robustness of the method under violation of those assumptions. 

Without a clear statement of the underlying assumptions and a more detailed empirical study assessing robustness against violations of the assumptions, it is hard to judge the soundness and relevance of the proposed method.
questions: No questions.
limitations: In line 103, SCMs are defined with $U_i$ as jointly independent noises, which implies a very particular case of SCMs where there is no unmeasured confounders (i.e., a Markovian system). This is not the general definition of an SCM (as in Pearl, 2009) and should be clarified as an additional (and strong) assumption of the work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors proposed a simple but solid method to answer counterfactual questions for temporal point processes.
In particular, the authors focused on the counterfactual question that whether a historical event will happen if the corresponding intensity is changed. 
For the inhomogeneous Poisson process, the authors demonstrated that via sampling events with a rejection-related intensity (i.e., lambda_max - lambda_accept) and superposing the events to observed ones, the counterfactual question can be answered simply based on a thinning process.
Moreover, leveraging the fact that Hawkes process can be implemented as the superposition of various inhomogeneous Poisson process, the proposed thinning process can be extended to answer the counterfactual question of linear Hawkes process — just applying the method to each branch of the Hawkes process.
A simulation experiment on epidemiological data demonstrates the rationality of the proposed method.
strengths_and_weaknesses: Strength:
(1) The paper is well-written and the idea is easy to follow.
(2) The idea is simple and theoretically supported, and the method is easy to implement. Additionally, Proposition 1 fits our intuition. 
(3) The application of epidemiological data is important and interesting.

Weaknesses:
(1) If my understanding is correct, the current application scenarios are limited to inhomogenous Poisson process and linear Hawkes process. It would be nice if the authors can discuss how to extend the proposed method to more complicated TPPs, e.g., neural Hawkes, or nonlinear Hawkes.
questions: No more questions.
limitations: (1) The method is focusing on inhomogeneous Poisson process and linear Hawks process. Whether it can be applied to other TPPs is unknown and needs to be discussed in depth.

(2) Besides epidemiological data, it would be nice if the authors can discuss more potential applications, e.g., recommendation systems, and voting data.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes an approach to address counter-factual queries in certain types of temporal point processes, such as what would have happened if the conditional intensity was different in the past. This is a query that has not been well studied in TPPs, due to several technically challenging issues. The proposed work essentially relies on leveraging the thinning algorithm together with a Gumbel-Max structural causal model. There is also a sampling algorithm that uses this causal model of thinning together with the superposition theorem to simulate counterfactual realizations of TPPs. A limited application is studied in the main text for experiments.
strengths_and_weaknesses: Strengths:

I think this paper has several new ideas for how to handle counterfactual queries in TPPs. It brings together previous ideas related to TPPs (such as thinning and superposition) and structural causal models (such as a recent Gumbel-Max version) but I think it does so in an innovative way. There are many non-trivial aspects in my view, such as how to handle the fact that “rejected” events from the thinning algorithm are not seen in the observational data. Thus I believe the work is sufficiently novel and significant.

Although there are a few details in some sections which I could not completely follow, I’m generally favorable towards the paper. I also however have some concerns and many suggestions, which I outline below.

Weaknesses (with questions and suggestions):

Firstly, the title of the paper is over-encompassing. There are several assumptions made, and as far as I understand, the approach does not necessarily work for all TPPs. A specific title that is more descriptive would be appropriate. I highly encourage the authors to propose and use a different title. For me, one of the major weaknesses of the work is the lack of detail around the limitations and assumptions (even beyond the title). The authors need to be clearer about the scope of the work and the implications of the assumptions.

A fundamental question for the authors: why do counterfactuals enable for better targeted interventions? Shouldn’t it be sufficient to have an interventional model if one is only interested in interventions? I found some of the motivation for the work on counterfactuals to be unfounded. Counterfactuals are clearly important for some sort of analysis, but I would like to better understand from the authors how it helps them in their problem setting.

I find some of the positioning of the related work to be unclear and even incorrect. For instance, Didelez (2008) is clearly related to Granger causality for multivariate TPPs, yet it is categorized in a separate bucket – I do not understand what “predicting quantities related to an interventional distribution” means. Is this a reference to average treatment effects? Could the authors please clarify? I recommend the authors refer to other work on Granger causal models as there is a lot of literature on graphical representations of TPPs. The ones that are cited are more neural network inspired than founded on graphical model foundations. Also, the authors should do a better job of mentioning why some of the prior work (such as by Roysland) is not directly relevant.

I found the experimentation section to be on the weaker side. I am surprised that synthetic experiments were placed in the Appendix, given that the paper is about counterfactual analysis. To me this is a sign that the scope of the work is quite narrow. This is not a major problem for me, but I think this should be made more explicit.
questions: I have posed some questions previously but here are a few more questions and comments:

Could the authors please clarify that the proposed ideas work for univariate TPPs? What classes of TPPs are covered by the approach?

Why is [4] cited on line 21?

The authors make an incorrect distinction in the second paragraph of Section 1. How does it matter whether a machine makes a decision or a human? And how does it matter if the stakes are “high” or “low”? A decision is a decision, and I do not understand how counterfactuals are needed for “high-stake” decisions but not “low-stake” decisions. This entire paragraph needs to be rewritten, in my view. And the authors need to clearly explain the value of having a model that enables counterfactual queries.

There are several concepts introduced on page 2 that would not be easily understood by reader not familiar with TPPs.

I’m surprised that Pearl’s work and related work is not cited at all in Section 1. It seems to be left for much later.

g_i should probably be f_i on line 118.

Monotonicity is not clearly explained on page 3.

The Gumbel-Max trick mentioned in footnote 4 needs an explanation.

In lines 173-177, should “unlikely” be “impossible”, based on the probabilities in Proposition 1?

In my opinion, the application studied is unnecessarily involved, for what it ultimately illustrates. It is far from a real application and more like a case study that is mildly real-inspired. I understand of course that there is a general challenge around demonstrating counterfactual analysis, but I wonder if a variety of purely synthetic data experiments in the main text would have been more suitable. 
limitations: The authors need to write a lot more clearly about the many limitations of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a causal model of thinning for temporal point processes. Given a set of observed events for a point process one can use the superposition theorem to sample plausible sequences of rejected events. Given the observed events and the rejected events, a SCM can be used to sample counterfactual events. These are realisations of the point process we would observe under alternative intensity functions. A Gumbel-Max SCM is used to ensure identifiability of the counterfactual distributions. 
strengths_and_weaknesses: The paper is well written and easy to follow. This is one of the few papers I have read combining point processes with causality which makes it novel and interesting despite being quite simple. The paper would benefit from clarifying the points given below: 
	
- Section 3. I found the last part of this section (from line 153), which I believe to be the most important from a causality point of view, quite confusing. Given that the authors use the language of structural causal models to frame the problem, it would be useful to add a causal graph representing the data generating mechanism. This is important as it determines how the equation under line 157 is computed and what type of adjustment the do-calculus requires. For instance do(.) is dropped ‘because U and $\lambda$ are independent in the counterfactual SCM'. This is not evident from your discussion and it could be clarified using causal graphs. In addition, a short sudo algorithm describing how to sample from the distribution after line 157 would clarify the paper.
- Line 155. How you estimate p(U) is very important in this paper as it also justifies the choice of the Gumble max SCM. This should be explained in the preliminaries or in the appendix.
- Section 4. Depending on the value of $\lambda_{max}$ the number of rejected events consistent with $\lambda_m$ changes. How would that change the counterfactual realization that you would generate from Alg 1? Would that change the distribution of $U_i$ and therefore the distribution at line 7 of the algorithm? 
- In Alg 1 the function ACC(.) takes $\lambda_{max}$ as input, where is that used? I don't think you need it once you have $H_{max}$
- Could you clarify why we need to restrict the SCM to be the Gumbel max. Is this to ensure that given the data from observational distribution we are sure that we can identify the counterfactual distribution correctly?
- What is preventing you from working with points in a higher dimensional space? What about spatial point processes for example?

*Experimental section:*
- I liked the application the authors look at in this paper as it highlights the importance of the problem addressed. However, assessing the behavior of the proposed methodology with real-world data is difficult. I would prefer to see at least one experiment for the synthetic settings in the main paper. 
- Given that you assume the intensity to be known, the only approximations you are introducing are the estimation of the distribution of the noise variables and the Monte-Carlo approximation of the counterfactual thinning probability. Given these approximations, I wonder if one could assess how close the generated counterfactual realizations are to the ground truth. In a synthetic setting you can obtain counterfactual realizations from a known SCM and compare them with those you generate.

*Minor:*
Line 150 what do you mean by $t_i \sim \lambda_{max}$? I think in thinning the total number of points are sampled from a Poisson and then the points are uniformly distributed?

questions: See section above.
limitations: Yes, I cannot identify potential negative societal impact for this work.
review_title: null

Meta-review: metareview: This paper proposed a simple yet sensible method to answer counterfactual questions for temporal point processes. Specifically, the authors focus on the counterfactual question of whether a historical event would have happened if the corresponding intensity had been changed. Reviewers agree that the idea is clearly presented and theoretically plausible, with interesting and important epidemiological applications. Please also pay attention to the reviewers' concerns about the paper title and the presentation of some specific parts of the paper.
confidence: Less certain
Decision: Accept