Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes an estimation strategy for treatment effects on long term outcomes in a difficult setting with sequentially correlated treatments. They consider a certain case where unobserved values of the treatment show up in a small "new" dataset without long term outcomes recorded but where there is access to an "old" dataset with old treatments that include long term outcomes. Assuming certain nuisance functions (conditional expectations of various parts of the graphical model) can be estimated consistently with [insert favorite ML method here], they use Double ML to derive the asymptotic normality of their treatment effect estimators, as well as double robustness with respect to two of the nuisance functions.
main_review: Nice work! In general the mathematical ingredients and how they piece together are clear but the writing is pretty unclear. Warning: I am fairly new to DML (a few months of reading) and have only seen it in simpler non-sequential settings eg basic ATE. However, all of my questions are not about the final application of DML / orthogonality itself but the setup leading up to it.

Some comments/questions

- how can A be px1 when the treatment vector is K dim ? 

- This caused a lot of confusion for me when reading this paper, even though I could appreciate a lot of the piece-by-piece without knowing the answer to this question: could you describe the precise usage of the word "novel" in general in this work? The pharmaceutical example is estimation of causal effect of a new drug, but wouldn't this correspond to adding an extra treatment variable T,k+1 to the conditioning set of models like Y|T,S for a given timestep? That is Y|T1,...Tk,S vs Y|Y1,...,Tk+1,S?  And the models on the observational set are not estimated on such an expanded conditioning set? Or is it that you are estimating outcomes Ybar under some new treatment *value* T* in R^k such that T* not observed in observational data? Can you try to clarify here what precisely is meant by novel with an example? What assumptions are necessary on the nature of the new treatment?

- line 69: small off by one error in the M term sum, kappa should start at 1 instead of 0 as done later in line 165

- lines 94-96 since I had trouble understanding this description in words how correlated treatments cause bias but I understood the mathematical example starting at line 109. Maybe you can mention at 94-96 that this is demonstrated below?

- lines 135 (under eq 5): it would be useful to point out that gamma(I+B)S,t shows up earlier as the first term in eq2: g0(S,t). Likewise that eq 6 (line 137) recovers two of the terms in line 121 just above eq 1. Help the reader realize exactly where they saw these terms before.

- (important) After eq 6, a nice summary sentence, enumerated list, or formal Algorithm environment would be good to make sure people understand the algorithm. If I understand correctly it is  (Step one) First predict Y,t+1  | T,t+1, St as a linear function of T,T+1 and S,t that takes the specific form in (4) (Step 2) define adjusted outcome  Ybar,adj,t = Yt + Y,t+1 - alpha,t+1,Tt+1 which depends on the first step estimate through alpha (Step 3) Then predict this adjusted outcome from Tt,S,t-1 (Step 4) Finally the coef. in front of T,t (in a correctly specified model) is the causal effect theta0. Is it right? 

- lines 142-144, what do you mean by "both" approaches, do you mean (one) surrogate index without dynamic adjustment, and (two) dynamic adjustment? 

- line 164 to eq 7: you used E_o and E_e before defining them in 171

- sections 3.1 vs 3.2, DML / asymptotic normality: You mention in both sections that because of the orthogonal representation that normality results hold. For 3.1 (unadjusted) you say it in words, and for 3.2 (adjusted) you prove it (theorem 3.3). Does it trivially hold for section 3.1 (unadjusted) because of the proof in 3.2? Or just because it's the standard case already covered by DML?

- what about a small fully-synthetic experiment with a correlated earlier and later treatment where you are estimating the effect of the earlier treatment without adjustment and where you plot estimation bias vs strength of correlation (through either of the two routes: kappa or lambda*S)

- is there no public dataset that fits this problem setup that you can also include in addition to the semi-synthetic experiment?

- This paper is definitely missing a conclusion.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper develops a method for estimating cumulative effects of treatments when available data is limited to historical data of old treatments and short-term data of new treatments. A key challenge in this setting is that treatment assignments over time are correlated with each other, making it difficult to isolate the effects of individual treatments on the cumulative, long-term outcomes. This paper expands on previous methodology for this problem which make use of "surrogate indices" that mediate the effects of treatment on outcomes at each time step. Specifically, the authors estimate the isolated long-term reward of a treatment in the absence of future treatments. The methodology allows for continuous treatments and has double robustness properties (with respect to the surrogate index and surrogate score models). Semi-synthetic experiments show that the proposed dynamic adjustment approach lowers estimation error.
main_review: The topic of this paper is is very interesting and seems quite important: estimating long term effects of treatments from primarily historical data is important evidence for making early-stage decisions about new treatment policies. The authors state that their method combines (and extends) the surrogate index approach of Athey et al. (2020), double machine learning (Chernozhukov et al., 2018a), and dynamic effect adjustment (Lewis & Syrgkanis, 2020). I must admit that, of these techniques, I am only familiar with double machine learning. However, to the best of my understanding of the paper, the authors elegantly combine and extend these approaches to create a holistic solution to the long-term effect estimation problem.

As someone unfamiliar with the surrogate index approach of Athey et al. (2020), at times I had trouble following the methodology. For example, I do not believe the authors provide examples of surrogate variables, except to say that a single time period's outcome or "other measures... of a customer's trajectory" could be used as proxies. I think the investment firm example from the introduction could be fleshed out a little more to make clear what the proxies would be in that scenario.

Some of the organization in Sections 2 and 3 was confusing to me. In Section 2 the authors begin by describing the estimation strategy of Athey et al. (2020), then informally describe the assumptions required for identification before describing violations of these assumptions that would induce bias in the described estimation strategy. I think this section could be streamlined to more concisely state the key information that will be needed for problem setup and to understand the method. The target estimand and the assumptions required should be formally stated up front in Section 2. Further, the causal path assumption (that at each time step surrogates entirely mediate the effects of treatments on outcomes) is never formally stated. The graph in Fig 1 is very helpful for visualizing this, but it should be accompanied by the formal statements. Given this setup, the authors could then introduce the unadjusted surrogate index and clearly define the bias (right now the bias is shown in equation 3 but the authors do not draw much attention to this). The beginning of Section 3 and Section 3.3 were written in a way where it was much easier to follow. I think the authors should use this style of previewing what will be shown in each subsection in other parts of the paper as well.

The results in this paper are interesting and important. I think the paper could be greatly improved by cleanly and concisely introducing  the formal problem setup and previewing section content to help guide the reader.


UPDATE:
Thanks to the reviewers for their response. I remain positive about this work and will be maintaining my score. I definitely think the suggested improvements to clarity will help this work be appreciated by a broader audience (that includes myself!). I also think it should be possible for the authors to flesh out the hypothetical example(s) without revealing proprietary information---surely the proposed method is useful beyond its particular application at your organization! I genuinely believe these changes would take this paper from "good" to "great."
limitations_and_societal_impact: The authors should at least make an explicit statement somewhere in the paper regarding limitations or that they do not foresee negative impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper considers the problem of estimating a long-term treatment effect. It considers the setting where there are two datasets: an observational dataset, which contains the long-term outcome, the treatment and the surrogate variable, and an experimental dataset, which contains only the treatment and the surrogate variable. The authors generalize the surrogate index method proposed by Athey et al. (2020), by considering dynamic treatment policies. Their new proposed method is a combination of the surrogate index approach, the double machine learning approach and the dynamic treatment effect estimation approach. 
main_review: Originality: The question considered is new and very interesting. The proposed method is a novel combination of the three approaches: the surrogate index approach, the double machine learning approach and the dynamic treatment effect estimation approach. My concern is that, in terms of methodology innovations, there is not too much new outside of the three existing approaches. 

Quality: As far as I checked, all claims are well supported. The theories and experimental results are explained very clearly. 

Clarity: The paper is well written and I enjoyed reading it. Section 2.1 is particularly helpful in demonstrating the main idea. For the numerical experiments, I think it would be helpful to explain the five methods in a clearer way. Currently it is a bit confusing what the purple bar does. 

Significance: The problem considered by the authors are important. I agree that treatment policies are often dynamic in practice. The solution to the problem is nice and strong; it has good theoretical properties and performs good empirically. Again, as mentioned above, my concern is that I don’t see too much new outside of the three existing methods. 
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of estimating the long-term effect of some new treatment in the case where the treatment assignment is decided adaptively. Athey et al.(2020) proposed the long-term effect estimation for the stable treatment assignment case. This paper applied the method proposed in Lewis & Sygkanis(2020) to adjust the approach of Athey et al.(2020) for the case where the treatment assignment is decided adaptively.
main_review: Estimating the long-term effect is an important problem in the application of causal inference and econometrics. However, I am quite not sure that this research is well motivated. Do we really need to know the long-term effect of a particular arm while using some dynamic treatment regime? I appreciate it if you could explain and add some examples.

For the experimentation, you should report your result of the synthetic data experiment reported in the appendix with more detailed information about the data generating process.

Some minor issues

- line 121: what is "noise" in the equation?
- line 121: the definition of I is ambiguous here
- section 3.2: the actual estimation step should be explained here. At least, you should refer to the appendix here.
- line 280: "for this excercise," ?
- line 282: the definition of "total" is somewhat ambiguous. For me, the definition in the appendix helped.
limitations_and_societal_impact: They adequately addressed the limitation in their paper.
review_title: null

Meta-review: Thanks to the authors for this engaging and submission.  The reviewers were all quite positive about this work, and I'm happy to recommend acceptance.

One shared concern among the reviewers is clarity of the submission --- as reviewer ZEEt notes, sections 2 and 3 are a bit confusing in their presentation, and should be streamlined in the final version.  And as reviewer SfcW discussed, the incorporation of a new synthetic experiment will help understand the relationship between estimation bias and strength of correlation, even if placed in the appendix and referred to in the main text.
Decision: Accept (Poster)