Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper describes a method for the causal analysis of long streams of
event data that can be modeled as multivariate point processes (MPPs).
The paper defines the average treatment effect (ATE) between pairs of
events in MPPs that occur within a window, and derives how to estimate
the ATE as the expected difference of average outcome rates.  When
covariates differ or assumptions are violated, this estimate may be
biased, but adjusting for the propensity score yields an unbiased
estimator, which the paper does through inverse probability of treatment
weighting (IPTW) (including a stabilized version).

Experiments on data from known, synthetic MPPs (proximal graphical event
models and Hawkes processes) show that the proposed IPTW estimation
method more accurately recovers the ATE than baseline conditional
intensity scores.  Similarly, experiments on a diabetes dataset show
that stable IPTW recovers more true causal pairs at 2 out of 3 levels of
recall compared to the conditional intensity scores.  These results
extend to using the cumulative rather than the conditional intensity
rate.

main_review: Overall
-------

Overall, I think this is an excellent paper.  In particular, it balances
well the presentation of high level concepts and motivations while
providing rigorous, supporting details.  The framework of causal
inference in multivariate point processes (MPPs) it develops could play
a central role in establishing a comprehensive methodology for causal
inference with event sequence data.


Originality
-----------

While all of the pieces of this paper have been well established, this
paper combines them in a new way by deriving the standard causal
inference measure (ATE) -- and how to estimate it well -- for a class of
models (MPPs) that are very relevant to practical applications.  The
paper situates its contributions clearly in comparison to survival
analysis, hazard models, and Granger causality.  From the meticulous
nature of the paper, I expect related work is adequately cited, but I
don't know the literature in this area thoroughly enough to assuredly
make that judgment.


Quality
-------

The submission is very meticulously written and is technically sound as
far as I have investigated.  (I did not check the supplement nor the
proofs therein.)  Statements are well-cited, and experimental evidence
supports the claims of effectiveness.  I don't have any complaints about
the methodology or the completeness.  The authors do not include an
explicit evaluation of strengths and weaknesses, but they have defined
their scope well enough to allow those to be inferred.

My main suggestion is to use figures to illustrate the results rather
than just listing them in tables.  Yes, I know it takes a lot more work,
but the end result is so much easier for our feeble human brains to
interpret.  (If you think the specific numbers are important, then put
them in the supplement, perhaps as a data file.)  This alone brings my
evaluation down one point.

I also suggest including proof sketches in the main body of the paper as
a general rule of thumb (if it is accepted you get an extra page), but
the theoretical results are as expected, so I'm less concerned compared
to unexpected / interesting results.


Clarity
-------

This paper is one of the clearest I have ever reviewed.  Bravo!  (I have
no suggestions.)


Significance
------------

I think other researchers are quite likely to build on this work in
helping to establish a framework of causality for MPPs.  Since much of
the observational data we collect is event-based, I believe causality
with MPPs is quite important, and this paper could be a foundational
contribution to the area.


Details
-------

These are just some things I noticed; they are not comprehensive.

78-79 It would be nice to have a citation here, if only for the
new-comers.  (Where is it shown that ...?)

218 Is the square brackets and pipes notation, '[...|...]', meant to be
expectation?  That's what the text implies, but there's no 'E'.  Please
fix or clarify.

411 "hawkes" -> "Hawkes"

421 "2010" -> "2020"

limitations_and_societal_impact: The authors didn't really address the limitations or potential negative
impacts of their work.  (However, many limitations are implicitly
addressed in that one can deduct them from the content, but it's always
nice to have important things explicitly addressed.)  The only thing I
can quickly think of other than general, routine cautions about causal
models and confounding (which I don't miss here), is some discussion of
the window size and its consequences.  (But some experiments showed no
effect.)  I agree with the authors (in the checklist) that there is
unlikely to be societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a model for estimating the average causal effect in point processes where the estimand of interest is the effect of an event occurring within a window on the rate parameter of a point process. The authors use the formulation commonly used for inverse propensity scores of generalized treatments (e.g., from Robins, Imai). The authors describe the properties of the propensity score as a balancing weight by leveraging known results and advocate for the use of stabilization of inverse propensity scores to reduce variance as in Robins(2000). The weights are then used within an outcome model which is a point process model. Empirical results show promising performance compared to a range of alternatives. 

main_review: I think this is an interesting and important problem area, and the authors provide what I feel to be a reasonable approach. While the paper is on the lighter side in terms of theoretical contributions, I think the approach is novel and provides a nice addition to the literature. 

My largest concerns are as follows:
- The authors could do more to tie the proposed approach to the larger literature. While the point process application is novel (to my eyes), there is a large literature on propensity scores and balancing weights for general treatments in the statistics literature which should be referenced in order to provide proper context.

- It's unclear to me how much is being sacrificed by considering the quantity "event happens any number of times within the window" over "number of times an event is observed within the window". It would seem to me that these are different estimands, however it is not clear when we would prefer this "event happens at all" estimand. Can the authors provide intuition on this point?

- It would be nice to have consistency / convergence results for the proposed estimator.

- It's unclear how one would get confidence intervals from the proposed approach for the estimates. 

- The authors offer a heuristic for choosing the window size for the estimate. Intuitively, it would seem that smaller window sizes should result in greater precision. Can the authors provide intuition for the observed behavior which appears either flat or bell shaped?

- It would be much more convincing if the authors ran experiments on any of the motivating domain examples. As-is it is difficult to reason over the true efficacy of the proposed approach on the synthetic domains. 

Some smaller points:
- The authors consistently conflate propensity scores with balancing weights. Propensity scores are balancing weights under correct specification but the converse is not necessarily true. Given the growing literature on balancing weights this distinction is important to adhered to for the benefit of the reader.

- The citation to Zubizaretta (2015) on line 298 is incorrect. Zubizarreta’s paper concerns the construction of balancing weights, not stabilized propensity scores. I believe the citation for stabilized IPTW should be Robins (2000).
limitations_and_societal_impact: Any method for causal inference can be mis-applied an have negative societal impacts, however I don't think there is anything specific to this work that is unaddressed and should be acknowledged.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper provides tools to estimate average treatment effect (ATE) between event pairs. The authors first define the quantity for multivariate point processes together with the necessary concepts from Neyman–Rubin causality, then they derive a propensity score in this setting. Finally, they give a method for estimating the ATE and evaluate the method on synthetic and real world datasets.

main_review: The work precisely define causal assumptions of the method: ignorability and overlap. The method do not require causal sufficiency, unobserved confounders are allowed as long as the covariates form a proper adjustment set for their effect. However it will ignore indirect effects mediated by these covariate variables.

What does it mean that $\mathcal{K}$ is $y$'s future occurrences at $t$? Later we see that $Y_i$ is actually equal the instantaneous expected number of occurrences  OR the cumulative expected number of occurrences of the outcome. At this point the entity $\mathcal{K}$ is somewhat mysterious.

Line 131 (typo): y is the effect/outcome not a treatment

Line 180: Ignorability is misinterpreted here. Ignorability means that the counterfactual outcomes are independent from the actual outcome given the covariates. It is actually telling that the current value of Z in the real world have no extra information to determine the counterfactual outcomes given the knowledge of the covariates. It is closely related to the back-door criterion, see Pearl: Causality, 2009, Chapter 3; 11.

Line 198: What do you mean by "the coarsest of such a function is the propensity score" ? 

Line 204: missing : given observing covariates $\mathbf{x}_t^w$



limitations_and_societal_impact: The work precisely define causal assumptions defining the limits of applicability.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a causal inference framework for a multivariate point process model. The key idea is to form a series of binary processes Z_t and X_t using a moving window [t-w,t] to indicate whether processes Z and X have at least one event in the window [t-w,t].  An average treatment event (ATE) is defined and propensity scores are proposed for unbiased estimation of ATE. The authors use synthetic experiments and real-world datasets to demonstrate how the proposed method work compared to baseline approaches.
main_review: The paper is fairly well written and the concepts proposed are clear and new. I would say that the work is quite original in terms of conceptual development.
limitations_and_societal_impact: 1. Line 154, I have some doubts on the assumption (3) "the specific times of z’s occurrences do not further affect y’s rate at time t.". This is at least not the case for Hawke's process. I wonder how many existing point process models satisfy this assumption. Can you give some examples?

2. Line 240, if the dimension of X_t^w is high, wouldn't the propensity estimates be very unstable and very high dimensional?

3. In Corollary 3. only an unbiased estimator for the ATE is given. But to do causal inference, one would need some distributional results for ATE. Can this be done? Otherwise, I do not see much usage of the proposed method in real applications.
review_title: null

Meta-review: Causal inference in point processes is a novel and open area, and the reviewers agreed that this is a useful contribution both methodologically and from an applied perspective. There are a number of changes that you should definitely make to improve the manuscript.
Decision: Accept (Poster)