Double/Debiased Machine Learning for Dynamic Treatment Effects
Authors: Greg Lewis
Vasilis Syrgkanis
Venue: NeurIPS 2021 Poster (2021)
Keywords: dynamic treatment regime
high-dimensional
treatment effects
double machine learning

Abstract: We consider the estimation of treatment effects in settings when multiple treatments are assigned over time and treatments can have a causal effect on future outcomes. We propose an extension of the double/debiased machine learning framework to estimate the dynamic effects of treatments and apply it to a concrete linear Markovian high-dimensional state space model and to general structural nested mean models. Our method allows the use of arbitrary machine learning methods to control for the high dimensional state, subject to a mean square error guarantee, while still allowing parametric estimation and construction of confidence intervals for the dynamic treatment effect parameters of interest.  Our method is based on a sequential regression peeling process, which we show can be equivalently interpreted as a Neyman orthogonal moment estimator. This allows us to show root-n asymptotic normality of the estimated causal effects.

Double/Debiased Machine Learning for Dynamic Treatment Effects
Greg Lewis Microsoft Research g ewi fmmmmmiiii..mmeii crosoft.com com
Vasilis Syrgkanis Microsoft Research vasyêmi www.ott.ccooooooooooooooooooo.o. com
Abstract
We consider the estimation of treatment effects in settings when multiple treatments are assigned over time and treatments can have a causal effect on future outcomes. We propose extension of the double/debiased machine learning framework to estimate the dynamic effects treatments and apply it to concrete linear Markovian high-dimenssonal state space model and to general structural nested mean models. Our method allows the use of arbitrary machine learning methods to control for the high dimensional state, subject to mcan square error guarantee, while still allowing parametric estimation and construction of confidence intervals for the dynamic treatment effect parameters of interest. Our method is based on a sequential regression peeling process, which we show can be equivalently interpreted Neyman orthogonal moment estimator. This allows to show root-n asymptotic normality of the estimated causal effects.
Introduction
Improving outcomes often requires multiple treatments: patients may need course of drugs to manage disease, soil may need multiple additives improve fertility, companies may need multiple marketing efforts to close the sale. To make data-driven decisions, policy-makers need precise estimates of what will happen when new policy pursued. Because of its importance, this topic has been studied by many communities and under multiple regimes and formulations; examples include the field of reinforcement learning [34], longitudinal analysis in biostatisties [2], the dynamic treatment regime in causal inference and adaptive clinical trials [19].
This paper offers new method for estimating and making inferences about the counterfactual effect of new treatment policy. Our method is designed to work with observational data, in an environment with multiple treatments (either discrete or continuous), and a high-dimensional state. Valid causal inference is necessary to correctly attribute changes in outcomes to the different treatments applied. But it is more challenging than in a static context, since there are multiple causal pathways from treatments to subsequent outcomes (e.g. directly, or by changing future states, or by affecting intermediate outcomes, or by influencing future treatments).
Our work bridges many distinct literatures. The first is the econometrics literature on semi remi-parametric inference [22, 33, 21, 1,7, 6) 6]. We extend this literature, which has typically focused on static treatment regimes, consider semi-parametrie Markovian model with flexible high-dimensional state and more broadly dynamic treatment regimes. We propose an estimation algorithm that estimates the dynamic effects of interest, from observational data, at parametric root-n rates. We prove asymptotic normality of the estimates, even when the state is high-dimensional and machine learning algorithms are used to control for indirect effects of the state, as opposed effects stemming from the treatments. Our formulation can be viewed as a dynamic version of Robinson's 's classic partial linear model in remi-parametric inference [33], where the controls evolve over time and are affected by prior
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

treatments. Our estimation algorithm builds upon and generalizes the recent work of [6. 8] to estimate not only contemporaneous effects, but also dynamic effects over time. In particular, we propose a sequential residualization approach, where the effects at every period are estimated in a Neyman orthogonal manner and then peeled-off from the outcome, so as to define a new "calibrated" outcome, which will be used to estimate the effect of the treatment in the previous period.
In doing this, we build on results in the semi-paramettre inference literature in bio-statistics [3, 37, 36] on estimating causal effects in structural nested models (see [39, 29, 16, 4] for recent overviews). In particular, our identification strategy for dynamic treatment effects is variant of the well-studied g- formula and g-estimation for structural nested mean models (SNMMs) [27, 30, 28, 32, 31, 29, 20, 38]. The works cited above have provided doubly robust estimators for this setting. One practical challenge faced by these estimation approaches is that the nuisance functions required are hard to estimate when there are many treatments or treatments are continuous.
Our approach addresses these challenges by providing Neyman orthogonal g-estimation algorithm for linear structural nested mean models [28, 29], that allows for both continuous and discrete treatments in cach time period. We use a two stage algorithm, where in the first stage a scquence of regression and classification models are fitted and the second stage a simple linear system of equations is solved. This approach also allows for an easy sample-splin shappe--plitinn/roosss---s fitting approach, which allows the use of arbitrary machine learning approaches in the first stage.
While Neyman orthogonality is a weaker condition than double robustness, it is sufficient to achieve robustness bias introduced by using machine learning to estimate the nuisance parameters. This approach thus permits machine learning for the nuisance model estimation, which is practically important in nigh-dimensional state space/control variable settings. We view this as the main contribution of this paper to the biostatistics literature. Other contributions are 1) we provide formal guarantees for the case of a single time-series dataset, as opposed to cross-scctional 2) we extend to linear structural nested models with heterogeneous coefficients, thus allowing for infinite dimensional target functions. Using the recent framework of orthogonal statistical learning [12), these can also be estimated via machine learning techniques. The resulting algorithm is an extension of the popular in practice RLearner algorithm [24] for heterogeneous treatment effects, but in the dynamic treatment regime.
series; and
Our work also closely related to the work on doubly robust estimation in longitudinal data analysis and in offline policy evaluation in reinforcement learning [23, 35, 26, 18, 17] from the causal machine learning community. However, one crucial point of departure from all of these works is that we formulate minimal parametric assumptions that allow us to avoid non-paraectric rates or high- variance estimation. Typical fully non- non-paraeetric approaches settings, requires the repeated estimation of inverse propensity weights at each step, which leads to a dependence on quantities that can be very ill-posed in practice, such as the product of the ratios of states under the observational and the target policy. Moreover, these approaches are typically restricted to discrete states and actions. Our goal is to capture settings where the treatment can also be continuous (investment level, price discount level, drug dosage), and the state contains many continuous variables and is potentially high dimensional. Inverse propensity weighting approaches, though more robust in terms of the assumptions that they make the model, can be quite prohibitive in these settings even in modcrately large data sets.
off-policy evaluation in dynamic
Our work is also somewhat related to the online debiasing literature [11, 10, 42, 14), since we perform inference from adaptively collected data. But our work is different in two main ways. First, the inferential target in this paper is more general, since that literature restricts itself to inference on contemporaneous effects (i.e. effect of treatment on same period outcome) or on policy values in the absence of dynamic endogenous state (i.e. assuming no dynamic effects through Markovian state variable). Further, our results extend to the more general structural nested mean model setting. The second important difference is that, in our main setting, we assume multiple independent small chains of samples (cach from a separate treated unit). This sidesteps one of the key technical difficulties in the literature above, which is performing inference from single chain of serially correlated and adaptively collected samples. With multiple short independent chains, the main technical difficulty tackled by these works, namely the potential non-convergence of the covariance matrices that are used in the effect estimate, vanishes. In that respect, that literature is closer to our results on single adaptive chain (see Section 6). However, for our results in this setting, we make homoskedastic noise assumption in the treatment choice policy (i.e. that the noise term in the treatment policy enters additively and is drawn d.),,,, which allows us to easily show convergence of the associated
2

---

covariance matrices of our Neyman orthogonal estimator and hence again sidestep a key technical difficulty. It is an interesting avenue for future research if this i.i.d. additive noise assumption can be dropped from our results on single adaptive chain by utilizing ideas from this literature (e.g. adaptive re-weighted variants of our Neyman orthogonal estimator; as for instance described in [11]).
2 Model and Preliminaries
We state our main results in stylized partially linear state space Markov decision process and in Section 7 we show that our results generalize to more complex Markovian models, known in the biostatistics and causal inference literature Structural Nested Mcan Models (SNMMs). For simplicity of exposition we focus on this simplified version, as it captures all the complexities needed to highlight main contributions. We consider sequence {X,, Te, Y, mh where X, E R is the state at time t, T3& E R is the action treatment at time and Y, E IR is observed outcome of interest at time t. We assume that these variables are related via a linear Markovian process:
Vt E (I,...,m,) m)
X A T; 1 + B Xt + 7t
T, = p X.) +
(I)
Y, = 6T. + N' 6t
where N/: Se and are exogenous mean-zero random shocks, independent of all contemporaneous and lagged treatments and states, that for simplicity we assume are each drawn i.i.d. across time with To = Xo O. The function p could be any propensity function.
Our goal is to estimate the effect of a change in the treatment policy on the final outcome Y,.. More concretely, suppose that were to make an intervention and set each of the treatments to some sequence of values: {r -m J. then what would be the expected difference in the final outcome Yn as compared to some baseline policy? ? For simplicity and without loss of generality, we will consider the baseline policy to be setting all treatments to zero. We will denote this expected difference as: (TT
Tm)- Tm)) if denote with R(T) E [Ym do = -1...... Tm Tm then V(r) R(r) R(0). In
Equivalently, we can express the quantity we are interested in do-calculus:
Appendix A, we will also analyze the estimation of the effect of adaptive counterfactual treatment
policies, where the treatment at each step can be function of the state.
3 Identification via Dynamic Effects
Our first observation is that we can decompose the quantity (T) into the estimation of the dynamic trealment effects: if we were make an intervention and increase the treatment at period m by I unit, then what is the change 0₂ in the outcome Yn. for E (0. ...,,.,,;; assuming that we set all subsequent treatments to zero (or equivalently to some constant value). This quantity the effect in the final outcome, that does not go through the changes in the subsequent treatments, due to our observational Markovian treatment policy, but only the part of the effect that goes through changes in the state space Xt, that is not part of our decision process. This effect can also be expressed in terms of the constants in our Markov process as:
YK E (1...., m 1} 0x = Bx-1 A
The counterfactual value function V Rdm - R, can be expressed in terms of the dynamic treatment effects as: (TTI,..., Tm) x mmmm-
Lemma 1.
Thus to estimate the function V. it suffices to estimate the dynamic treatment effects: Og. On- We first start by showing that the parameters 0u...., Om identifiable from the observational data. Identification is not immediately obvious. Write the final outcome as a linear function of all the treatments and the initial state: Ym = Σx= TmTm-xx + p2óm + Cn- The shock 6m contains components that are heavily correlated with the treatments, since the shocks at period t affect the state at period + 1, which in turn affect the observed treatment at period t ++1 1. In other words, if simultancous treatment problem, where the final outcome the outcome, then we essentially have problem of unmcasured confounding (implicitly because we ignored the intermediate states).
we view the problem
However, we show that these dynamic effects can be identified from the observational data via a sequential peeling process, which as we show in the next section, leads to an estimation strategy that achieves parametric rates. This is one of the main contributions of the paper.

---

Theorem 2. The dynamic treatment effects satisfy the following set of conditional moment restrictions: 4q E {0,....71) m
BI B|ym mmmm- 0gTm-q 1L X Tm-q: Xmn m:
where: Ym.m-q= = m'm - Ex 0m Moreover, we let Tm-y:= == Tm-q- E[I'my Xm-gl and we have that the covariance matrix J = E.Tm- - m- gl is invertible, then these conditional moment restrictions uniquely identify 4q.
Proof. For any q E (0,...,m. 1}. by repeatedly unrolling the state
Xm, 9 1 times we have that:
9-1
Ym= = 86 + Σ n n'BB-1 ATm + pp
Σ
1/m + 6m
q
wi m
V m + pl B X +
Tmm-x 1o--1mm-xx
Cm
1/m T m-g
Σ
WTT, μt 8 X
Σ
1 1xB Mm-R + cm
g+1
k=0
Thus by re-arranging the equation, we have:
Y Ym, m-
V Vm-gTm- + p B X
1 1== 1mm cm
m
k=0
Since for all 2 m g, Tl: are subsequent random shocks to the variables T'm Xm- we have that they are mean-zero conditional Tm-q. xm Thus:
E Y "mm-Tmm- Ba Xm-g Tm-
X Xm-
Thus we have concluded that for any E {l,.... m):
E[ym.t - TiTT, - ul Bmn 9 X, Te, X₂] =0
which concludes the proof of the first part of the theorem.
For the second part, consider any q and let = m q. Note that by taking an expectation of the above condition for with respect to T, conditional on Xe, get that it implies
B| E[mm,m-y Xm-g] = V' Fin-TTm-e Tm-q - μt B Xm-q Tm-g. X Xm-gl
Dynamic DML Estimation
We now address the estimation problem. We assume that we are given access to n i.i.d. samples from the Markovian process, i.e. we are given n independent time-series, and we denote sample i, with {X; Ti, Y Our goal is to develop an estimator of the function V or equivalently of the parameter vector = (0o.... 0m). We will consider the case of high-dimensional state space, Le. p > n,, but low dimensional treatment space and low dimensional number of periods m, i.e. d, m < n is constant independent of n. We want to estimate the parameters 0 at Vri-rates and in way that our estimator is asymptotically normal, so that we can construct asymptotically valid confidence intervals around our dynamic treatment effects and estimate of the function V. The latter is a non-trivial task due to the figh-dimensionalityy would be statistically impossible if we were to take the direct route of estimating the whole Markov process (i.e. the high-dimensional quantities A, B, p):: if these quantities have a number of non-zero coefficients that grows with any polynomial rate, then known results sparse linear regression, preclude their estimation at root-n rates (see e.g. [40]). However, we are not really interested in these low-level parameters of the dynamic process, but solely on the low dimensional parameter vector 0. We will treat this problem as semi-parametti inference problem and develop a Neyman orthogonal estimator for the parameter vector [22, 33, 1. 6].
the state space. For instance, the latter
4

---

In particular, we consider sequential version of the double machine learning algorithm proposed in [6]. the case of single time-period, i.e. m 0, then [6], recommends the following estimator for lo: using half of your data, fit model (o) of E|Yo Xo). i.e. that predicts the outcome Yo from the controls Xo and a model (X)) for E|To Xol. Then estimate lg on the other half of the data, based on the estimating equation:
m(0: po: go) = E (Yo 0o Tb) To) =
where Yo YY= -xo and To To Po (Xo) are the residual outcome and treatment.
We propose a sequential version this process that we call Dynamic DML for sequential dou- ble/debiased machine learning. Intuitively our algorithm proceeds as follows:
1. We can construct estimate &g of do in robust (Neyman orthogonal) manner, by applying the approach of [6] on the final step the process, i.c. on time step Tmi this will estimate all the contemporaneous effects of the treatments,
2. Subsequently we can remove the effect of the observed final step treatment from the observed
final step outcome, i.e. by re-defining the random variable Y: Y m Y 0g T doing this have removed any effects on Yū caused by the final treatment T
3. We can then estimate the one-step dynamic effect 01, by performing the residuallon-residuall estimation approach with target outcome the "calibrated" outcome Y: m treatment T
and controls X Theorem 2 tells us that the required conditional exogeneity moment required to apply the residualization process valid for these random variables. We continue in a similar manner, by removing the estimated effect of T from Y m,m-1 and repeating the above process.
We provide a formal statement of the Dynamic DML process in Algorithm 1, which also describes more formally the sample splitting and cross-fitting approach that we follow order to estimate the nuisance models and required for calculating the estimated residuals.
Algorithm Dynamic DML
Randomly split the samples in S, for each & E (O..... m) do
Regress Ym Xm-r using S to learn estimate 9. of model (x) E[ym Xm x] and calculate residuals Y m,m-x Y (x; other half; vice versa 5 learn model and evaluate on S.
for each (0,...... . x) do
Regress Tm-r on Xm-x using S to learn estimate pT,* of model (r, = E(Tm-r X Xm-x x = z| on the first half and calculate residuals T m- = T Th PT,M (X m-)x) on other half, and vice versa.
end for end for
Using all the data S S for 0 m do Regress Y
Y m-m
Eter ETex,,T mmTmm-m- T m iec find solution 6, to:
Vi Tr mmmmmm-- a.m-N
T mm-mm 8.m-
end for
Estimation Rates and Normality
Our main theorems are show that subject the first stage models of the conditional expectations achieving a small (but relatively slow) estimation error, then the recovered parameters are root-n- consistent and asymptotically normal. Our asymptotic normality proof relies on showing that one can
5

---

re-interpret our Dynamic DML estimator as a Z-estimator' based on set of moments that satisfy the property Neyman orthogonality"..
Let p, q denote the vector all nuisance functions and p", q"" their true values. Moreover, denote the vector of dynamic effect parameters, with 0o its true value. We provide both finite sample rates mean squared error rates and asymptotic normality of our estimates (proofs in Appendix B.3 and B.4). Morcover, in the appendix discuss how the required guarantees for the nuisance functions can be easily satisficd using estimation algorithms such as the Lasso and under sparsity conditions.
Theorem 3 (Finite Sample). Suppose that [E[G< t 2X1 and that each coordinate h of each nuisance model (4x, Pr,nTT, satisfy that w.p. 1 8:
1 h- 22 Exx - h(X))²) S Gn.6
where expectation is with respect to the corresponding input of each model. Moreover, suppose that:"
Σ
S Cm
Then w.p. 26:
10, 0,,ll12 < 50 (aCm ( (Vog(ddm00) en, 1) [c(m]
max
{
m+1
where C.
max
If each coordinate h. of each nuisance model
23 23- (mm
satisfies: E ll| - h" [|
S cns then:
(Vlostdmm)
2h 1)
E
max [m)
0
01, SO
d
n
Theorem (Asymptotic Normality). Suppose that the nuisance models {ęx. Pr.N satisfy that:
VR,7 T S K 1/ - 22, IlpT,x Pr,xll2 S o(n" -1/4)
Moreover, suppose that all random variables are bounded and that Elçe Gtl = AI. Then:
VTi 0n) + V)
where V -M- and M is
block lower triangular matrix consisting of blocks of
size d x d, whose block entries are of the form:
VT S k. MR.7
Xm-x)) Sm-xl
and E is m block diagonal matrix whose diagonal block entries take the form: En Elea- 6m-1Sm
Concrete Rates for Lasso Nuisance Estimates. Suppose that the observational policy p is also linear, i.c. p(X)= AX. Then all the models 9. and Pr,. are high-dimensional linear functions of their input arguments, i.e. (x) = ø'x and Pr,x.j (x) = Tr T. If these linear functions satisfy a sparsity constraint then under standard regularity assumptions
guarantee if regression to estimate each of these functions that w.p. 8, the estimation error of all nuisance
the Lasso
models is O S (og(p/6)
where s is an upper bound on the number of non-zero coefficients.
One sufficient regularity condition is that the expected co- variance matrix of every period's 's state has full rank, i.e. E[X,X;) 2 XT. Thus the requirements of the main theorems of this section would be satisfied as long as the sparsity grows as s = o(n1/4). These conditions are for instance satisfied if coordinates of the high-dimensional state have any effect on the final outcome (i.e. are outcome-relevant) and coordinates the high-dimensional state enter the observational policy.
A Z-estimator is solution to empirical analogue of vector of moment equations Elm(W; 0,v) 0.
where W are all random variables, m is vector valued function and 0 E IR is a vector parameter of interest and E V is potentially infinite dimensional nuisance parameter. The true parameters satisfy (W; fo vo]]
2 A vector of moments satisfies Neyman orthogonality if Vv E V: Blm Oo, t t(v vo)) l--== 0
Where by absolute value we denote coordinate

---

6 Dependent Single Time-Series Samples
Thus far we have assumed that we are working with n independent time series, each of duration m. Though this is applicable to many settings where we have panel data with many units over time, in some other settings is unrcasonable to assume that we have many units over time, but rather that we have the same unit over long period. In this casc, would want to do asymptotics as the number of periods grows. Our goal is still to estimate the dynamic treatment effects (i.c. the effect of a treatment at period É on an outcome in period + K, for {0,.... .0,.....e m.) for some fixed look-ahead horizon
These quantities can allow us to evaluate the effect of counterfactual treatment policies on the discounted sum of the outcomes, i.c. Ei=g YY, for 7 1. We can write the counterfactual value function for any non-adaptive policy as: V(T) Σi= rL Engs 0-- Assuming outcomes are bounded, the effect Eysi 0g on any period t can be at most some constant. Thus taking m to be roughly log,, (n), suffices to achieve good approximation of the effect function V (Tr, since the rewards vanish after that many periods, i.e. we let: (T) = E' 7" Egei AT then observe that: (7) V(+)ll < O(+").. Thus after m = log1/~(() (n),, we have that the approximation error is smaller than 1/vrn Thus it suffices to learn the dynamic treatment effect parameters for a small number of steps. To account for this logarithmic growth, we will make the dependence on m explicit in our theorems below.
For any m, we will estimate these parameters by splitting the time-series into sequential B n blocks of size m. Then we will treat each of these blocks roughly as independent observations and we denote the resulting estimate as Ó. The main challenge in our proofs is dealing with the fact that these blocks are not independent but serially correlated. However, we can still apply techniques, such as martingale Bernstein concentration inequalities and martingale Central Limit Theorems to achieve the desired estimation rates.
The other important change that we need to make is in the way that we fit our nuisance estimates. To avoid using future samples to train models that will be used prior samples (which would ruin the martingale structure), we instead propose a progressive nuisance estimation fitting approach, where at every period, all prior blocks are used to train the nuisance models and then they are evaluated on the next block. We present the normality theorem and defer the finite sample MSE result to the appendix.
Theorem 5. Let Fo denote the filtration up until block b. Suppose that B[GG | FF) 2 2XI and that each coordinate h of each nuisance model (9,,Pr,x Tr satisfies: Vb 2 B/2
B ht 11,,21 (1/4) S ((1
Moreover, suppose that: | IE T Fo
S em and that m. satisfies that mCm
ol O), where C. ;= max { 1, (2x-0(m den.s)
m++1
Then:
V 1BE-1/2M(( - 0m) + N(O,T)
where M, E are the same as in Theorem 4.
Generalization to Structural Nested Mean Models (SNMMs)
We now generalize our results by drawing connection between algorithm and geestimation of structural nested models in biostatistics [27] and show how our DynamicDML algorithm can be applied almost verbatim with small change of variables and estimate dynamic effects in much more general setting. Consider an arbitrary time-series process (X,, Tr Tm 3
with X, E X, and T, E Tr. Let Y denote some final outcome of interest, For any time t, let X, {X1.... X.) and T, = {Ti ........T T.}. denote the sequence of the variables up until time t and similarly, let X, = (X,, (X,,.... X,,) and T, (T.,.. .,, Tm ). We will also denote with 3,, F,, Ie, corresponding realizations of the latter random sequences. Let π (MI,.... 7m) denote any dynamic policy, such that for each t, n, maps a history 2,, F into a next period action T. For any such dynamic policy,
let (*) denote the counterfactual outcome under policy d. For any static policy 7 E x Te. we will overload notation and let (7) denote the counterfactual outcome under this static treatment policy.

---

Moreover, for any two policies (static or dynamic) we will be denoting with (7t, xr+1), that follows x' up until time and then continues with policy т. We 0 E T, denote a baseline policy value, which could be appropriately instantiated based on the context.
1, the policy
We assume that the data generating process satisfics the sequential conditional exogeneity condition:
(y T E T 11 T. X;
Identification of mean counterfactual outcomes [[} for a target policy of interest я can be expressed in terms of the following conditional expectation functions:
Yy (#,, Fa) E [y(te (f-1110)))) T. TI, X, = -1
which corresponds to the mean change in outcome if we go to all units which received treatment To up until time and had observed state history I. and we remove their last treatment, while we subsequently always continue with the larget policy प. These functions are known as the blip functions [4, 29). Theorem 3.1 of [29] shows that via a telescoping sum argument and invoking the sequential randomization condition that:
(f X Ti Fel = Y Ey X
(,, =
where Px TT) = : (X,, TY+E Pe (x, T.).. Importantly, the conditioning set contains the observed t periods treat- ment. Intuitively, each term Pj. removes from the outcome the blip effect of the observed action T and adds the blip effect of the target action क (X,,, I2- Thus for any parameterization of the blip functions //e (āx, Fri Wx),, if we let (+))) == Y Σj= Pj (X,, T2), then the true parameter vector must satisfy the moment restrictions:
and hence also: E[
Vt E (m]. Vf: (v) (f, - EIf T.) T(,)- = 0
If the blip functions admit linear parametric form:
(T Fei te) W.d(Te, Fe)
for some known feature vector o, satisfying (fTx, (Tr- 0)) 0, then we can consider the subset of the moment restrictions of the form:
Vt [m] (b) (o(X,, TT) ((o T.) X Ti-]]]] -0
Moreover, we can also subtract from H, (ı+), its conditional expectation B(H,(1) | X,, F.-1] while maintaining the moment condition:
44 E [m]: (1)) - (b))(v) I-)) (d(R,, E(o(X., T.) T. = =0
This is the doubly robust moment condition proposed by [28, 29], where it is shown that an estimator of tb based on this moment is correct if either the estimate of E (v) X., Ti-1 or the estimate of ([ ()) X2, 1 is correct. However, estimating E[Z,, scamingly knowledge of v. which is cumbersome and can be im-practical (c.g. by constructing poor preliminary estimates of v [16] (see Technical Point 21.5) note that for binary treatment and when the target policy is the all-zero policy, then E H (b) X, T.j can be estimated by regressing the outcome of the population that received zero subsequent treatment on the history. However, such a population can be quite small in practice and can have severe co-variate imbalances compared to the overall population. Moreover, this approach only applies to the case of a binary treatment and a static target policy.
T..). itself requires
Here, we can achieve a Neyman orthogonal moment for identifying vi which is sufficient for robustness to biases stemming from machine lcarning models used to train the nuisance components, while avoiding the cumbersome part of estimating the nuisance E H[ (w) X,, T,-1]. Morcover, our approach leads strongly convex loss for the parameters each step of the recursive process, which is beneficial for finite sample guarantees and subsequently for generalizing it to linear models with parameter heterogeneity with respect exogenous covariates.
In particular, let Q3 = o(8,, TTy) o(X,,
T (X,, T)-1)) and for any & S J. let T3.- =
Q, - -[Q, X. T--1] and Ym,t := Y B[Y X., T. Then, note that:
He(w) B[H,(v) | Xr, Tr-nl = Ym.. Σ Wf,T.
1=1

---

Moreover, note that since the second term in Q2 only depends on the conditioning set X,, T1, then Ti = ex,,, T:) El Xx, Te- 1]. Thus we conclude that the true parameters must be satisfying the moment restrictions:
At C [m]
[ EEj=--
KTT TfTe,
Tt. =0
These are exactly of the same form the moment restrictions considered in the definition of the Dynamic DML algorithm and its analysis. Thus the results we presented so far directly extend to the estimation of structural nested mean models, with linear parameterization of the blip functions, simply by using the different definition of the residual variables Y, and Tr, and letting 1ht 0m- in the definition of Algorithm and in Theorems 3 and 4. Moreover, note that the nuisance models that are trained in the first stage have a larger conditioning set which includes all past history of states and treatments X,, Ti-1- If we made further restrictions that there was a "funnel statc" S₁ at cach period that summarizes the history and such that any dependence of the future the past going through that funnel state, then conditioning only on that state would have been sufficient. This is what we essentially did in the linear Markovian model. Moreover, the linear Markovian model with a static policy is a special case where the blip functions take the simple form: >/ (2,,, tc) t and are target policy independent.
Thus our Dynamic DML algorithm extends to the estimation of the structural parameters in structural nested mean model for any target dynamic policy d and any user defined baseline policy 0 (referred to as double regime structural nested mean model). It allows for the estimation of the nuisance functions with arbitrary machine learning algorithms, subject to a relatively slow meaa-squared-error condition and reduces estimation to simple regression and classification oracles in the first stage, with only a simple linear system of equations in the second phase, which can also be solved in linear time in recursive manner.
Dynamic effect heterogeneity Finally, we note that our moment condition that identifies each parameter 2/2 is the derivative of a square loss and can be written as the solution to the following
2 fixing the solution square loss minimization problem: ming, E - - - vTT....
b for any j > f, from previous iterations. This allows us to generalize the Dynamic DML to the case where we allow non-parametric heterogeneity in the parameters th,, with respect to an exogenous fixed covariate vector of cach sample, denoted as Xo. i.c. (T,, F.) = (/( F.). for a known feature map ó and unknown heterogencous parameters t). Thus gesstimation approach to SNMMs to allow for infinite dimensional parameters of the blip functions, as long as the input to these infinite dimensional parameters is fixed and not changing endogenously by the treatments (e.g. fixed characteristics of a unit). This can be achieved by simply minimizing recursively the square loss:
essentially generalize the
E , - (X.) 27
min E vx(-)
over arbitrary function spaces (-). Using techniques from the recently introduced orthogonal statistical learning framework [12], we show in the appendix that this estimation method provides mean-squared-error guarantees on the recovered heterogeneous parameters vtt that are robust errors in the nuisance functions. This heterogeneous extension can also be viewed as an analogue of the RLeamer meta-learner algorithm [24], generalized the dynamic treatment regime setting.
8 Experimental Results
We consider data drawn from the DGP presented in Equation (1), with linear observational policy: p(T,11, X,) = CTT.-1 + D X,, with Xo, To = and (ts Ge. 7) standard normal r.v.'. (recall that d is the number of treatments and p the number of state variables). We consider the instance where: A.j- = 5,, for all i E [p]. E [d], B = 5 lp. C = 2 la. Dj:, 1 2] = 4, Di: p] = O, u[1 : 2] = 88 We consider settings where the effect is constant, i.e. Og E Rd or heterogeneous, where (x(x) Og + (go, X[S]} for some known low dimensional subset S of the states. We compare the dynamic DML to several benchmarks. The results are presented in Figures 1, comparing the estimates of the dynamic DML algorithm number of other alternatives on a single instance.

---

They fall into two categories. In the "static" set of approaches, each of the contemporaneous and lag effects estimated one at a time, either by direct regression or (static) DML. So for example, to estimate the one period lag effect 0,, we would regress Y, on T; 1, with controls X. We consider direct regression with no controls ("no-ctrls"), direct and DML with controls from the inital period (i.e X,-1, "init-ctrls" and "init-ctrls-dmll) and direct and DML with controls from the same period as the outcome (i.e X,, "fin-ctrls" and fiin-ctrls-mml" As an alternative to all of these, we try a "direct" dynamic approach, where initially estimate 0g using lasso regression of Y, with all the controls and past treatments, and return the coefficient on Te. and then "peel" off the estimated effect in the main text before running another Lasso regression of Y₁ InT, T get the first lag effect etc. So this approach incorporates the peeling effect but doesn't 't do any orthogonalization. The point estimate for 0o. 02 and 0g depicted in the three panels of Figure 1 and the error bars correspond to the constructed confidence interval. For all three, the dynamic DML is relatively close the truth and the confidence interval contains the truth. The remaining approaches are not, although for the contemporancous effect the approaches with final period controls have similar performance it really in the lagged effects that the differences become most apparent. Subsequently we run multiple experiments to evaluate the performance of DynamicDML. In each setting, we run 1000 Monte Carlo experiments, where each experiment draws then estimated the effects and lag effects based on our Dynamic DML algorithm. Figure 2 considers the case of two treatments, and shows that the algorithm performs well in terms of giving reasonable coverage guarantees for nominal 95% coverage, actual coverage varies from 91 % to 94.5%. The right panel shows that the average estimates are close to the truth. In Figure in Appendix E we repeat the experiments with N 2000, and actual coverage is now tightly in the range 94% to 95%, and the average estimates remain relatively unbiased. Finally, in the appendix we explore scenario with heterogencous effects and find qualitatively similar performance.
500 samples from the above DGP and
133
L.0
355
325
Figure 1: Comparison of DynamicDML (with confidence intervals) with benchmarks on a single
instance. n 400, n4 2, nz 100, s 10, ((2) .5. ((i) .5. (m) .5, C'
0.93
0.92
16
1,
4
(a) Coverage
(b) Point estimates
Figure 2: n 500, n, = 2, n. 450, 2, (()) = 1, 1G) 5, (n.)
10

---

References
[1] Chunrong Ai and Xiaohong Chen. Efficient estimation of models with conditional moment restrictions containing unknown functions. Econometrica, 11(6))1795-1843, 2003.
[2] Heejung Bang and James M. Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 6144)6622 2005.
[3] Peter Bickel, Chris AJ Klaassen, Peter Bickel, Ya' acov Ritov, J Klaassen, Jon A Wellner, and YA' Acov Ritov. Efficient and adaptive estimation for semiparametric models, volume 4. Johns Hopkins University Press Baltimore, 1993.
[4] Bibhas Chakraborty and Erica E. M. Moodie. G-computation: Parametric Estimation of Optimal DTRs, pages 101-112. Springer New York, New York, NY, 2013.
[5] V. Chernozhukov, J. C. Escanciano, H. Ichimura, W. K. Newey, and J. M. Robins. Locally Robust Semiparametrie Estimation. ArXiv e-prints, July 2016.
[6] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duño, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-668, 2018.
[7] Victor Chernozhukov, Juan Carlos Escanciano, Hidehiko Ichimura, Whitney K. Newey, and James M. Robins. Locally Robust Semiparametric Estimation. arXiv e-prints, page 1rXiv:1608.0000. July 2016.
[8] Victor Chernozhukov, Matt Goldman, Vira Semenova, and Matt Taddy. Orthogonal machine learning for demand estimation: High dimensional causal inference in dynamic panels. arXiv preprint arXiv:
2017.
[9] Victor Chernozhukov, Denis Nekipelov, Vira Semenova, and Vasilis Syrgkanis. Plug-in regu- larized estimation of high-dimensional parameters in nonlinear semiparametric models. arXiv preprint arXiv: 04823, 2018
[10] Yash Deshpande, Adel Javanmard, and Mohammad Mehrabi. Online debiasing for adaptively collected -eh-dimensional data with applications to time series analysis. arXiv preprint arXiv:1911.01044 2019.
[11] Yash Deshpande, Lester Mackey, Vasilis Syrgkanis, and Matt Taddy. Accurate inference for adaptive linear models. In International Conference on Machine Learning, pages 1194-1203. PMLR, 2018.
[12] Dylan Foster and Vasilis Syrgkanis. Orthogonal statistical learning. 1rXiv=190..09036 2019.
arXiv preprint
[13] David A. Freedman. On tail probabilities for martingales. Ann. Probab., 3(1):100 118, 02 1975.
[14] Vitor Hadad, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. Proceedings of the National Academy of Sciences, 118(15), 2021.
[15] P. Hall and C.C. Heyde. 3 the central limit theorem. In P. Hall and C.C. Heyde, editors, Martingale Limit Theory and its Application, Probability and Mathematical Statisties: A Series of Monographs and Textbooks, pages 51 - 96. Academic Press, 1980.
[16] Miguel A Hernán and James Robins. Causal inference: What if, 2010.
[17] Nathan Kallus and Masatoshi Uchara. Double reinforcement learning for efficient off-policy evaluation in markov decision processes, 2019.
[18] Nathan Kallus and Masatoshi Uchara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning, 2019.
11

---

[19] Huitan Lei, Inbal Nahum-Shani, K Lynch, David Oslin, and Susan A Murphy. A" smart" design for building individualized treatment sequences. Annual review of clinical psychology, 8:21-48, 2012.
[20] Judith J Lok and Victor DeGruttoa. Impact of time to start treatment following infection with application to initiating haart in hiv-positive patients. Biometrics, 68(3):745-754, 2012.
[21] Whitney K Newey. Semiparametric efficiency bounds. Journal of applied econometrics, 5(2):99-335, 1990.
[22] Jerzy Neyman. <(a) tests and their use. Sankhya, pages 1-21, 1979.
[23] Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning when-to-trees policies, 2019.
[24] Xinkun Nie and Stefan Wager. Quasi-oracle estimation of heterogeneous treatment effects. arXiv preprint
2017.
[25] Thomas Peel, Sandrine Anthoine, and Liva Ralaivola. Empirical bernstein inequality for martingales: Application to online learning. 2013.
(26) Maya Petersen, Joshua Schwab, Susan Gruber, Nello Blaser, Michael Schomaker, and Mark van der Laan. Targeted maximum likelihood estimation for dynamic and static longitudinal marginal structural working models. Journal of causal inference, 2(2):147-185, 2014.
[27] James Robins. A new approach to causal inference in mortality studies with sustained exposure prriod-application control of the healthy worker survivor effect. Mathematical modelling. 11::12):1333111 1512, 1986.
[28] James M Robins. Correcting for non-compliance in randomized trials using structural nested mean models. Communications in Statistics -tatrstissTTheory and methods, 23(8):2379-2412, 1994.
[29] James M. Robins. Optimal Structural Nested Models for Optimal Sequential Decisions, pages 189 326. Springer New York, New York, NY, 2004.
[30] James M Robins, Donald Blevins, Grant Ritter, and Michael Wulfsohn. G-estimation of the effect of prophylaxis therapy for pneumocystis carinii pneumonia on the survival of aids patients. Epidemiology, pages 319-336, 1992.
[31] James M Robins, Miguel Angel Hernan, and Babette Brumback. Marginal structural models and causal inference in epidemiology. 2000.
[32] James M Robins and Ya' fa'acov Ritov. Toward curse of dimensionality appropriate (coda) asymptotic theory for carimparaeetric models. Statistics in medicine, 16(3):285-319, 1997.
[33] Peter M Robinson. Root-n-consistent semiparametric regression. Econometrica: Journal of the Econometric Society, pages 931-954, 1988.
[34] Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.
[35] Philip S. Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for rein- forcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML' 16, pages 2139-2148. JMLR.org org, 2016.
[36] Anastasios Tsiatis. Semiparametric theory and missing data. Springer Science & Business Media, 2007.
[37] Mark Van der Laan, MJ Laan, and James M Robins. Unified methods for censored longitudinal data and causality. Springer Science & Business Media, 2003.
[38] Stijn Vansteelandt, Marshall Joffe, et al. Structural nested models and g-estimation: the partially realized promise. Statistical Science, 29(4):707-731 2014.
[39] Stijn Vansteelandt and Arvid Sjolander. Revisiting g-estimation of the effect of time-varying exposure subject to varying confounding, Epidemiologic Methods, 5(1):37 56, 2016.
12

---

[40] JM Wainwright. High-dimensional statistics: A non-asymptotic viewpoint. preparation. University of California, Berkeley, 2015.
[41] Martin J. Wainwright. High-Dimensional Statistics: A Aon-ssypiioiic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
[42] Kelly W. Zhang, Lucas Janson, and Susan A. Murphy. Inference for batched bandits. CoRR, abs/2002.032177, 2020.
13


Reviews:
[{"review_id": "xARZZhOXXCw", "reviewer": "Reviewer_EKsS", "summary": "This paper combines the $g$-estimation approach to identifying dynamic treatment effects in structural nested mean models with orthogonal ML to obtain asymptotically normal estimates of the treatment effects. The main benefits of using orthogonal ML to estimate the treatment effects is it allows high-dimensional state spaces and is statistically efficient (with cross-fitting).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The most interesting part of this paper is the identification strategy (Lem 1 and Thm 2). Although both seem to follow from straightforward manipulations of (1), I suggest the authors elaborate on the identification strategy. In particular, I suggest the authors include the proofs of Lem 1 and Thm 2 in the main paper. This better motivates the \"peeling\" method. In light of the page limit, I suggest the authors defer either the simulation results or one of the extensions (sec 6 or 7) to appendices.\n\nAnother way that the paper can be improved is by demonstrating the applicability of the proposed method on real data. The linear MDP imposes strong restrictions on the data generating process, thereby limiting the applicability of the method. I suggest the authors show that the method works on real data to assuage concerns regarding the proposed method's applicability.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed limitations of their method. There are not direct potential negative societal impacts of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "_y7msaeKkpR", "reviewer": "Reviewer_bwP4", "summary": "The authors propose an extension of the double/debiased machine learning framework to estimate the dynamic effects of treatments and apply it to a concrete **linear Markovian high-dimensional state space model** and to general structural nested mean models. The method is based on a sequential regression peeling process, which we show can be equivalently interpreted as a Neyman orthogonal moment estimator. The authors prove both finite sample and asymptotic bounds for the estimation method. The experiments are super-minimal and do not provide convincing evidence about the algorithm's empirical performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. Algorithm 1 is rather simple, which increases its potential impact.\n2. The theoretical analyses seem correct.\n3. the idea of having a Z-estimator using the moment conditions to get the desirable theoretical rates is interesting.\n\n## Weaknesses\n1. The experiments are not convincing. The authors should report the uncertainty intervals for the `dyn-direct`   algorithm too.\n2. The results only hold for the MDP in Eq. (1), which is slightly less restrictive than the linear model.\n3. The authors should pick a less generic title. There are other double robust dynamic treatment effect estimation algorithms.\n4. The notation is rather messy. For example, the authors use $d$ for the dimension of the treatment (line 84) and dynamic policy (line 232). \n5. The assumptions are rather spread out during the text, especially in Section 7.\n\n## Further Comments\n1. Please enable line numbers for the Algorithm.\n2. The equation at line 115 needs more explanation. Where did the shocks $\\delta_m$ come from?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The algorithm only applies to a certain MDP. In the SNMMs, they have the classical causality assumptions such as *sequential conditional exogeneity condition*. The authors need to acknowledge these limitations and prevent a false sense of over-confidence in their results when these assumptions are violated.\n2. Otherwise, the problem solved in this paper is well-studied and can potentially have a positive societal impact in reducing biases in our estimations.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "Zf9RRo_qNW0", "reviewer": "Reviewer_FZDr", "summary": "The paper proposes a DML (double machine learning) based estimation methodology of dynamic treatment effects. Writing the conditional moment restriction on the \"adjusted\" outcome allows the authors to bring the methodology of orthogonal score to dynamic treatment effects. Using this insight, on a partially linear model, the authors propose a progressive (called peeling) algorithm where  the \"adjusted\" outcome can be regressed on the treatment and covariates to estimate the coefficient of the treatment. The authors then show that under slower than root-n estimation rates of the nuisance functions, the theta parameters still enjoy root-n consistent estimates. The authors then discuss using a single sequence of observations to estimate the effect of an intervention (dynamic) on a discounted outcome. The method is then shown to generalize to non-linear sequential models (SNMMs) which a much more general class than partially linear models.", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is mostly clearly written despite the inevitable denseness of equations when dealing with effect estimation in sequential data. With the generalization of the method to nested mean models, a lot of my concerns about generality are alleviated but it is unclear how much the knowledge of the representation plays a role in estimation. For example, it seems like while the method does not restrict the class of representation phi(X), it cannot be learned as  part of the process. Could the authors comment on this?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper solves the propose problem well enough but it must be noted that structural nested mean models do not contain all possible models. But I believe this is not a significant limitation.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "4hOtthPgEf9", "reviewer": "Reviewer_U835", "summary": "Contributions of this paper are the following: \n- Extending Double/Debiased Machine Learning (DML) theories from the static treatment regimes to dynamic treatment regimes for considering a semi-parametric Markovian model with the flexible high-dimensional state, under the partial linear model settings and its generalization to Structural Nested Mean Model (Section 7). \n- In extending, the authors proposed to employ (minimal) parametric assumptions to avoid the ill-posed problem and deal with high-dimensional covariates.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on extending DML theories to the dynamic treatment regimes by employing semi-parametric assumptions to avoid problems that could occur in nonparametric settings. This paper's contributions are significant given that partial linear setting and dynamic treatment regimes have important practical implications. \n\nHere are comments for helping readers understand the results and some questions that readers might be curious about. \n- 1. In Eq. (1), What is 'p' in p(Tt-1, Xt)? I couldn't find its definition. \n- 2. The model in Eq.(1) looks somewhat restricted because it confines the relationship between variables as linear. It would be better if practical motivations or examples that could be covered by Eq. (1) are provided. \n- 3. Sec. 7 look significant because these address my concern in comment 2 (above). It would be helpful if results in Sec. 7 are formalized as Lemmas or Theorems. \n- 4. I'd like to see if practical experiments corroborate with the proposed theories, given that the settings discussed in this paper are simple and highly practical (partial linear model with dynamic treatment regimes)\n- 5. Is it nontrivial to extend results in Thm. 4 for showing doubly robustness (robustness against model misspecification)? \n\nI'd like to suggest highlighting results in Section 7 or even consider reorganizing to show them first. Personally, even if I liked strong theories and highly practical settings of the work at first glance, I was concerned that settings in Sec. 2 looks restricted. But, my concerns were addressed by results in Section 7. Having said that, it'd be more attractive if results in Section 7 come earlier. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some of limitations are commented in the Main review. \n- I'd like to propose to explain the possibility of practical experiments (or possible scenarios) corroborating with the proposed theories. \n- What are the parametric assumptions in SNMM? Could they be explained more explicitly? \n- Can Thm. 4 be more extended to cover doubly robustness results? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a consensus among reviewers that the submission provides a solid
contribution for the challenging problem of estimating dynamic treatment
effects, i.e., estimating at time t+m the the causal effect of a treatment
given at time t. Although based on known DML/Neyman orthogonal scores concepts,
the methodology leverages a novel peeling argument which may have further applications in sequential causal inference problems.
Finally no major issues with the theory were raised in the reviews.
For all these reasons, I recommend to accept the paper.

The authors are encouraged to include in the camera-ready version of the paper
a semi-synthetic experiment that use characteristics from real world datasets,
and to implement the improvements that emerged during the discussion (e.g.,
avoiding d for the policy, expanding on the relationship with [8, 9, 39, 12] in
the related works section)
I also encourage the authors to discuss briefly
the practicability of verifying the various rate conditions appearing
in the assumptions, e.g., when is it possible to observe from the data
whether these rate conditions are satisfied for a given dataset.
Decision: Accept (Poster)