Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper combines the $g$-estimation approach to identifying dynamic treatment effects in structural nested mean models with orthogonal ML to obtain asymptotically normal estimates of the treatment effects. The main benefits of using orthogonal ML to estimate the treatment effects is it allows high-dimensional state spaces and is statistically efficient (with cross-fitting).
main_review: The most interesting part of this paper is the identification strategy (Lem 1 and Thm 2). Although both seem to follow from straightforward manipulations of (1), I suggest the authors elaborate on the identification strategy. In particular, I suggest the authors include the proofs of Lem 1 and Thm 2 in the main paper. This better motivates the "peeling" method. In light of the page limit, I suggest the authors defer either the simulation results or one of the extensions (sec 6 or 7) to appendices.

Another way that the paper can be improved is by demonstrating the applicability of the proposed method on real data. The linear MDP imposes strong restrictions on the data generating process, thereby limiting the applicability of the method. I suggest the authors show that the method works on real data to assuage concerns regarding the proposed method's applicability.
limitations_and_societal_impact: The authors discussed limitations of their method. There are not direct potential negative societal impacts of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a DML (double machine learning) based estimation methodology of dynamic treatment effects. Writing the conditional moment restriction on the "adjusted" outcome allows the authors to bring the methodology of orthogonal score to dynamic treatment effects. Using this insight, on a partially linear model, the authors propose a progressive (called peeling) algorithm where  the "adjusted" outcome can be regressed on the treatment and covariates to estimate the coefficient of the treatment. The authors then show that under slower than root-n estimation rates of the nuisance functions, the theta parameters still enjoy root-n consistent estimates. The authors then discuss using a single sequence of observations to estimate the effect of an intervention (dynamic) on a discounted outcome. The method is then shown to generalize to non-linear sequential models (SNMMs) which a much more general class than partially linear models.
main_review: The paper is mostly clearly written despite the inevitable denseness of equations when dealing with effect estimation in sequential data. With the generalization of the method to nested mean models, a lot of my concerns about generality are alleviated but it is unclear how much the knowledge of the representation plays a role in estimation. For example, it seems like while the method does not restrict the class of representation phi(X), it cannot be learned as  part of the process. Could the authors comment on this?
limitations_and_societal_impact: The paper solves the propose problem well enough but it must be noted that structural nested mean models do not contain all possible models. But I believe this is not a significant limitation.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Contributions of this paper are the following: 
- Extending Double/Debiased Machine Learning (DML) theories from the static treatment regimes to dynamic treatment regimes for considering a semi-parametric Markovian model with the flexible high-dimensional state, under the partial linear model settings and its generalization to Structural Nested Mean Model (Section 7). 
- In extending, the authors proposed to employ (minimal) parametric assumptions to avoid the ill-posed problem and deal with high-dimensional covariates.
main_review: This paper focuses on extending DML theories to the dynamic treatment regimes by employing semi-parametric assumptions to avoid problems that could occur in nonparametric settings. This paper's contributions are significant given that partial linear setting and dynamic treatment regimes have important practical implications. 

Here are comments for helping readers understand the results and some questions that readers might be curious about. 
- 1. In Eq. (1), What is 'p' in p(Tt-1, Xt)? I couldn't find its definition. 
- 2. The model in Eq.(1) looks somewhat restricted because it confines the relationship between variables as linear. It would be better if practical motivations or examples that could be covered by Eq. (1) are provided. 
- 3. Sec. 7 look significant because these address my concern in comment 2 (above). It would be helpful if results in Sec. 7 are formalized as Lemmas or Theorems. 
- 4. I'd like to see if practical experiments corroborate with the proposed theories, given that the settings discussed in this paper are simple and highly practical (partial linear model with dynamic treatment regimes)
- 5. Is it nontrivial to extend results in Thm. 4 for showing doubly robustness (robustness against model misspecification)? 

I'd like to suggest highlighting results in Section 7 or even consider reorganizing to show them first. Personally, even if I liked strong theories and highly practical settings of the work at first glance, I was concerned that settings in Sec. 2 looks restricted. But, my concerns were addressed by results in Section 7. Having said that, it'd be more attractive if results in Section 7 come earlier. 
limitations_and_societal_impact: Some of limitations are commented in the Main review. 
- I'd like to propose to explain the possibility of practical experiments (or possible scenarios) corroborating with the proposed theories. 
- What are the parametric assumptions in SNMM? Could they be explained more explicitly? 
- Can Thm. 4 be more extended to cover doubly robustness results?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose an extension of the double/debiased machine learning framework to estimate the dynamic effects of treatments and apply it to a concrete **linear Markovian high-dimensional state space model** and to general structural nested mean models. The method is based on a sequential regression peeling process, which we show can be equivalently interpreted as a Neyman orthogonal moment estimator. The authors prove both finite sample and asymptotic bounds for the estimation method. The experiments are super-minimal and do not provide convincing evidence about the algorithm's empirical performance.
main_review: ## Strengths
1. Algorithm 1 is rather simple, which increases its potential impact.
2. The theoretical analyses seem correct.
3. the idea of having a Z-estimator using the moment conditions to get the desirable theoretical rates is interesting.

## Weaknesses
1. The experiments are not convincing. The authors should report the uncertainty intervals for the `dyn-direct`   algorithm too.
2. The results only hold for the MDP in Eq. (1), which is slightly less restrictive than the linear model.
3. The authors should pick a less generic title. There are other double robust dynamic treatment effect estimation algorithms.
4. The notation is rather messy. For example, the authors use $d$ for the dimension of the treatment (line 84) and dynamic policy (line 232). 
5. The assumptions are rather spread out during the text, especially in Section 7.

## Further Comments
1. Please enable line numbers for the Algorithm.
2. The equation at line 115 needs more explanation. Where did the shocks $\delta_m$ come from?
limitations_and_societal_impact: 1. The algorithm only applies to a certain MDP. In the SNMMs, they have the classical causality assumptions such as *sequential conditional exogeneity condition*. The authors need to acknowledge these limitations and prevent a false sense of over-confidence in their results when these assumptions are violated.
2. Otherwise, the problem solved in this paper is well-studied and can potentially have a positive societal impact in reducing biases in our estimations.
review_title: null

Meta-review: There is a consensus among reviewers that the submission provides a solid
contribution for the challenging problem of estimating dynamic treatment
effects, i.e., estimating at time t+m the the causal effect of a treatment
given at time t. Although based on known DML/Neyman orthogonal scores concepts,
the methodology leverages a novel peeling argument which may have further applications in sequential causal inference problems.
Finally no major issues with the theory were raised in the reviews.
For all these reasons, I recommend to accept the paper.

The authors are encouraged to include in the camera-ready version of the paper
a semi-synthetic experiment that use characteristics from real world datasets,
and to implement the improvements that emerged during the discussion (e.g.,
avoiding d for the policy, expanding on the relationship with [8, 9, 39, 12] in
the related works section)
I also encourage the authors to discuss briefly
the practicability of verifying the various rate conditions appearing
in the assumptions, e.g., when is it possible to observe from the data
whether these rate conditions are satisfied for a given dataset.
Decision: Accept (Poster)