Estimating the Long-Term Effects of Novel Treatments
Authors: Keith Battocchi
Eleanor Dillon
Maggie Hei
Greg Lewis
Miruna Oprescu
Vasilis Syrgkanis
Venue: NeurIPS 2021 Poster (2021)
Keywords: long-term effects
dynamic treatment effects
surrogates
high-dimensional
double machine learning

Abstract: Policy makers often need to estimate the long-term effects of novel treatments, while only having historical data of older treatment options. We propose a surrogate-based approach using a long-term dataset where only past treatments were administered and a short-term dataset where novel treatments have been administered. Our approach generalizes previous surrogate-style methods, allowing for continuous treatments and serially-correlated treatment policies while maintaining consistency and root-n asymptotically normal estimates under a Markovian assumption on the data and the observational policy. Using a semi-synthetic dataset on customer incentives from a major corporation, we evaluate the performance of our method and discuss solutions to practical challenges when deploying our methodology.

Estimating the Long-Term Effects of Novel Treatments
Keith Battocchi'
Eleanor W. Dillon Microsoft Research
Maggie Hei Microsoft Research
Greg Lewis Microsoft Research
Microsoft Research
Miruna Oprescu Microsoft Research
Vasilis Syrgkanis
Microsoft Research
Abstract
Policy makers often need to estimate the long-term effects of newly-developed treatments, while only having historical data of older treatment options. We propose surrogate-based approach using a long-term dataset where only past treatments were administered and a short-term dataset where novel treatments have been ad- ministered. Our approach generalizes previous surrogate-styl methods, allowing for continuous treatments and serially-correlated treatment policies while main- taining consistency and root-n asymptotically normal estimates under Markovian assumption on the data and the observational policy. Using semi-synthetic dataset on customer incentives from major corporation, we evaluate the performance of our method and discuss solutions to practical challenges when deploying our methodology.
Introduction
Businesses invent new ways of interacting with their customers. Marketing departments devise new campaigns. Pharmaceutical companies roll out trials of new drugs. In many cases, decision makers expect these novel treatments to affect outcomes like sales or health over several years, but they are eager begin evaluating these innovations within months of deployment.
We propose an estimation methodology that leverages historical data, collected under earlier treatment regimes, to derive early estimates of the long-term effects of novel, newly-deployed treatments. [Athey (et al. (2020) develop surrogate index as one solution this problem. This method relies on an assumption that causal effects on long-term outcomes are channeled through set of observable short-term proxies. This assumption allows one to use the historical long-term data set to learn mapping from short-term signals projected long-term reward and subsequently estimate the causal effect of novel treatments on this index of surrogates.
In practice, treatment policies are often dynamic: treatments are assigned repeatedly, and their assignments depend on past treatments and short-term outcomes. Unfortunately, this pattern can easily break the assumptions needed for the surrogate index to work. For example, suppose that firm offers multiple investments auto-correlated, i.e. if customer receives an investment this month, then they will receive an investment with high probability in one of the subsequent months. These future investments can substantially increase the long-term outcome of interest, and this increase will be attributed to the short-term proxies. The surrogate index thus formed will tend to over predict long-run run outcomes, and when it is used to measure the treatment effect of some treatment in the short-run data, the estimated treatment effect will be bigger (in absolute magnitude) than the truth.
particular customer in the historical data and these investments
"Correspondence to: Vasilis Syrgkanis <vasy microsoft.com..
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The main methodological innovation of this paper is to use the dynamic treatment effect analysis Lewis Syrgkanis (2020) on the historical data in order to create an unbiased, dynamically adjusted surrogate index. Our new index takes the interpretation of the projected long-term reward in the absence of any future treatments. This model produces unbiased causal effect estimates of the long-term effects of the novel treatments, even with auto-correlated treatments.
second contribution of the paper is generalize the final causal analysis step allow for multiple continuous treatments rather than a single binary treatment, as in Athey et al. (2020). Our new estimator for this expanded surrogate approach allows for the construction of valid confidence intervals, even when using flexible machine learning models at both stages of the estimation to deal with high-dimensional data. In short, we show how one can combine three recently developed techniques, i) the surrogate index approach of Athey et al. (2020), ii) the double machine learning approach of @CChrnozhukov et al. (2018) 8a) and iti) the dynamic treatment effect estimation approach Lewis & Syrgkanis (2020), in single data analysis pipeline to estimate treatment effects in the presence of dynamic treatment policies.
Our work lies in the broader field of estimating causal effects with machine learning and Neyman
orthogonality (Neyman 1979 Robinson 1988; Ai & Chen 2003 Chernozhukov al., 2016
Chernozhukoy et al., 201 8b). Moreover, it relates to the work on machine learning estimation of
treatment effects in the dynamic treatment regime (Nie et al., 2019 Thomas & Brunskill, 2016
Petersen et al. [014 Kallus & Uchara 2019b a Lewis & Syrgkanis| |000]]00000000 Bodory et al.; |Sinhh et al.
2020) and on structural nested models in biostatistics (Robins) [ Robins al.] 1992] Robins
1994 Robins & Ritov| 1997. Robins et al. (2000] Lok & DeGruttola 2012 Vansteelandt et al. 2014
Vansteelandt Sjolander) 2016). Finally, relates to the surrogacy literature in causal inference (Prentice 1989; |Begg & Leung 2000; Frangakis & Rubin 2002] Freedman et al., 1992). In the
following sections we build on insights in these works to design a new, streamlined method that
resolves several difficult aspects of a common applied problem.
2 Problem and Methodology
We illustrate the difficulties of our problem and how we our method resolves them with a running example of a firm making investments in its customers in order to increase subsequent purchases. In this section we describe the problem set up and data requirements and use a simple two-period model to build intuition for how our approach estimates long-term causal effects in this environment and why earlier approaches fail. the following section we provide formal proofs our methodology in the general, multi-period case.
2.1 Problem statement
A firm has a number of distinct treatments T2.. Tk it offer to customers, such dis- counts or access to special services. At each period (say, a month), a vector of treatments We also observe vector of p
Ti. = (Tr,, Tit,k) R is applied to each customer characteristics Xi, = (X....,...., Xi..p)) E IRP, some of which are constant within customer (e.g. industry) and some of which vary over time and eustomer (e.g. last month's revenue). We observe the outcome of interest Yr, E R (e.g. monthly revenue).
Treatments may affect outcomes over many future periods, and this horizon could vary by treatment. We are therefore interested in identifying the average effect of each treatment at some period f, on the cumulative outcome in the subsequent M periods, i.e. Yr,, = End YL.+,R Suppose direct inference impossible because we have not yet observed the subsequent M periods following the introduction of new treatment. Instead, we have access to vector of d short-term proxies/surrogates, S., (S.......... at.ra) that act as leading indicators for our target outcome. In this example, we would like surrogates that capture customer's trajectory such as the next few months of revenue, intensity of product usage, new contract commitments, and participation in company events.
2.2 Dynamic Adjusted Surrogate Index
The key innovation of all surrogate-bases solutions is to use the proxy variables to bridge two incomplete data sets to estimate Y,,- A long-term observational data set need only include customer characteristics X1,t surrogates S1,41, realized M M-period outcomes Y... A second short-term,

---

experimental data set, includes customer features, X1,, surrogates, St., and treatments T2, This experimental data is restricted to the period where all treatments of interest have been introduced. but only requires few months of leading surrogates rather than M periods of leading outcomes. We propose the following estimation strategy:
Use the observational data set to estimate the expected long-term outcome net of any post- period f treatments using the recursive methodology of Lewis & Syrgkanis (2020). This adjusted outcome, y: is the expected cumulative outcome starting in period / under the assumption that customer receives no further treatments over the next M periods.
2. Still using the observational data set, construct model that predicts this adjusted outcome from Si, and Xiç:
90 ad (s,,, == =| Xi,l-
(surr. model)
Any machine learning estimation algorithm can be used to construct g.
3. Finally, calculate the adjusted surrogate index using the surrogates in the experimental data set:
Ir, = g(Sa,t, Xit)-
(predicted surrogate index)
This predicted index becomes the outcome in the final causal model.
This approach will be consistent the treatment effects under three assumptions, which we illustrate as causal graph in Figure
1. Conditional Independence: There are no paths from current treatments to future outcomes except through current future surrogates. Formally, Ta 1 Yia (x,, St.. for all t.
2. Stability: Relationships between current and future surrogates and between surrogates and outcomes, marked red in the figure, are constant over time and across samples.
3. Unconfoundedness: There are no variables beyond our observed that directly affect both the treatment assignment and outcome.
Note that the first assumption allows relationships between current and future treatments and between current treatments and future surrogates, in contrast to the canonical surrogate model. The stability assumption does not extend to the relationships between treatments and surrogates, which allows for the introduction of novel treatments in the experimental data not found in the observational data. The model able to account for steady growth natural way: as the surrogates grow, the predicted outcomes grow too.
investments
T--1
T
T
Controls/ surrogates
S
St
S
nessured canfouncers
patts surrogaton
outcomes
Y Y
Y2
YEs
Figure 1: Causal graph representation of the main assumptions of our causal analysis.
2.3 A two-period example
For conciseness, we now collapse X,, and S so that the surrogates each period and the controls in the next period are all denoted by St,-1- In other words, all customer observable characteristics

---

in the next period are candidate surrogates and become controls for the period after next. This is without loss of generality. For the remainder of this section only, we further simplify by assuming a linear Markovian model of how the random variables evolve over time and focusing on a single scalar investment (k = 1). With these simplifications, the evolution of the random variables can be described in three equations:
S = AT + BS + E,,t
(evolution of controls)
Yu = 7/S,,t Ct.
(outcome equation)
(treatment policy) where A is p X 1 matrix, B is p p matrix, are p-dimensional vectors and is a scalar. The terms Er, Ss.e, Ti, are exogenous mean-zero independent noise terms. For conciseness we drop the customer index in the remaining equations.
Ti,+1 = *T., + +. +
Target effect estimand of treatment T, on Y, can be derived from the structural model equations:
With two periods, the long term outcome Y,, = Yi, + Vit+i. The effect
Y, = = + 7 S + noise = > A'Al++1 + B + L)AT; + (BB + 1)BS;-1 + noise Thus we see that the effect of T7 on + + Y,, assuming future treatments are held constant, is: 8 (B I)A.
Bias from a standard surrogate index Athey et al. (2020) propose training a surrogate index using the realized cumulative outcome Y, on S,, essentially our proposed step 2 using raw instead of adjusted outcomes. Following that approach, we would estimate:
(S,) E|Y S 7' + B)S, + + AE S
Subsequently, if we estimate the causal effect of Ti on 90 (S,), then this effect would be:
E e+1 S.
0. = 0o + + 7
(3)
B
The standard estimate contains a bias stemming from the fact that investment today can lead to higher investment tomorrow. As illustrated in equation above, this bias can stem from two channels, both of which are plausible in many use cases. In our example, customer that is treated once may build stronger relationship with the firm and receive further treatments (i.e. K 2 the treatment policy). Alternatively, if the firm targets treatments to fast-growing customers, any treatment that successfully drives surrogate metrics higher will make the customer eligible for further treatment (A' 2 0).
Dynamic adjustment With only two periods, our first step simply requires estimating the single- period causal effect of Tet1 on Y441, controlling for S.. This conditional expectation is equal to:
E T441, = 7'AT4++ + BBB
(4)
We can now create an adjusted long-term outcome, Y = Y, Y our estimate of > from this first step.
where 0: is
The dynamically adjusted surrogate index builds this adjustment into equation (2) above:
Do (S.) EE(Y, S.) E|Y411 - AT4T (s.]
=7'S. ''S. + TE (S, S411
(5)
7' BBSs
This new index captures the projected M-period outcomes as if the customer was offered no future treatments.
When we estimate the effect of the treatment based on this adjusted surrogate index we recover:
B[ge adi (S,) Tr, St-1l = / + B)E[S, Te, St-1l
= -/ B)AT, + 7'(T B)B
With the dynamic adjustment, the coefficient in front of T, that we recover the true causal effect O 7 A. Lewis & Syrgkanis (2020) show how this adjustment approach can be extended to many periods, via a recursive peeling process, and also to high-dimensional surrogates and controls via dynamic double machine learning approach.

---

Estimating causal effects of new treatments With this adjusted surrogate index in hand, we can also estimate the long-term effect of any other treatment that was introduced more recently. this example, the stationarity assumption of the proposed surrogate approaches requires that B, which governs how the surrogate evolves, and 7. which governs how surrogates translate per-period outcomes, do not change between the observational and experimental data-sets. These two parameters govern how surrogates today relate to future outcomes the absence of any treatment.
Under such a condition, if we introduce a new treatment TTt", which has a different effect A on the surrogates (and hence on the long-term outcome), then the effect of this treatment on the long-term outcome is 6% = -/ + B) "e This 68th is exactly the outcome of estimating the adi causal effect of T 9o (S++1) controlling for St.
3 Formal Guarantees
In this section we present our main results. In addition to expanding to M periods, we introduce two substantive innovations over the basic strategy presented in the two period example above. First, we develop a generalization of the doubly robust estimation method in Athey et al. (2020) to the case of multiple continuous treatments. Second, we relax the linearity assumptions and make use of orthogonal machine learning techniques Chernozhukov CechniquesCheemozuukov et al. (2018a) to allow for rich set of potential confounders and valid analytic confidence intervals.
the structural model
We now assume that we observe the full M-period time series, (So, S1, Yi, T2, S2, ......; TM, SM, YM),, for each sample in the observational popula- tion, denoted as o. In contrast, we only observe (So. T1, S1) ) for each sample from the experimental population, denoted
Notation Throughout, we will denote with E., [-] the expectation conditional on the experimental population and [.[[] the expectation conditional the observational population. Moreover, for any vector-valued function f that takes as input a random variable Z, we denote with:
| v E. [Z]
and analogously llflzze. We denote with E., the empirical expectation over all the samples, i.e. for any random variable Z, E,, == 말 Σ. Z,, and with E.,, and Eo,n the empirical expectation over the experimental and observational samples correspondingly, i.e. Er, [Z] = ne Eise Ze and
E.,.. [Z] = no Eieo Z4.
Our goal is to isolate the causal effect of treatments in period 1, Tı, the long term outcome P1 = -t=1 Y, Formally, we fix the future treatments that each sample receives and we change
the treatment Ti from some value to to some other value t1, the treatment effect is:
T(1,, to) [BE do = t1), Te....... TM]
do = to), T2,..... Tmrl].
(8)
We present theoretical results in two steps. For exposition, begin by assuming treatments happen only at period 1. This is the setting analyzed in Anthhhyy et al. (2020), albeit only for the case of single binary treatment Ti []] We then show how this approach can be modified to incorporate dynamic treatment policy in the observational and experimental sample.
3.1 Double/Debiased Correction without Dynamic Adjustment
To begin, we assume that ..,...; TM 0. Throughout, we assume partially linear relationship between the treatment and the long-term outcome:
E, Y 7ı, So] = 0 (o), + by(
(PLR)
Athey al. 2020) also allowed for estimation of average treatment effects, in the case when there arbitrary treatment effect heterogeneity. In this work, we assume treatment effects are constant. generalization to the case of arbitrary treatment effect heterogeneity is feasible, but would require the estimation of conditional covariance matrices, which would make the estimation algorithm more brittle and the exposition much more complex.
5

---

for some known feature map o(:,)), +), but arbitrary function bu(-).
Formally, the invariance of the surrogate-outcomme relationship requires that that the mean-relationship between the surrogates S1 and the long-term outcome does not change between the observational and the experimental sample:
(S1) == [ S1] E. Y S1].
(IR)
Under the PLR assumption and the causal graph governing our data, we have that:
T(41, (u) 0g E(o((4,, So) c(lo, So)]-
We establish three consistent estimators for Oo that follow the same intuition as Athey et al. (2020).
adapted consider linear effects of continuous treatments rather than single binary treatment.
Theorem 3.1 (Identification). Denote the residual surrogate index and the residual treatment with:
(o1) (S1) Ee( So)]
(10)
Ti == o(T,, So) - E, So) So]
(11)
Then under assumptions (PIR) and (IR):
On [7, T Fe T (Si))
(surrogate index rep.)
Pr(e St) Pr(o)
8o
- E.
E, [T] (1 So)
(surrogate score rep.)
Pr(o S1) Pr(c)
Pr(e [1, S6) Pr(o)
0n E. [TT
[E, (T) (Si)) + E.
Pr(o S1, Sb) Pr(e) E.[Th Si, So] - (00()))
(orthogonal rep.)
The core estimation challenge that the surrogate approach resolves is that the treatments and outcome of interest are not observed in single dataset. Intuitively, the first surrogate index representation approaches this challenge by using realized treatments from the experimental sample and, in place of realized outcomes, substitutes the expected outcome conditional on the surrogates, go (S), which can be identified from the observational sample and then constructed in the experimental sample.
The second surrogate score representation reverses this substitution. The second term pairs an expectation of the treatment conditional on surrogates, [,, Si, So). with the realized outcomes from the observational sample. This representation requires an added ratio of probabilities of appearing in each sample, Pr(e) and Pr(o), to adjust for sampling variation across the two datasets.
The third orthogonal representation blends the first two representations and satisfies Neyman orthogonality, which allows the construction of confidence intervals and double robustness.
The parameter identified in the equations in Theorem 3.1 is interpretable even if the partially linear assumption is violated. In this case, the equations are identifying the best linear projection of the variation in the long-tem outcome that is not explained by the initial state. We formulate the estimation of tn based on the third orthogonal representation as a Z-estimator based on a vector of moment equations that depends vector of nuisance functions fo, i.e.:
m(Bo: fo) == EZ d, fo))] =0 and such that it satisfies the Neyman orthogonality condition:
(12)
Dimo(os fo). sol 0t m0 fo + +I [o))
1=
Subsequently, this will allow invoke the results in Chernozhukov et al. (2018)), to derive an asymptotic normal estimator, even when high-dimensional, regularized approaches are used to estimate the nuisance functions fo-
Theorem 3.2 (Othogonal Moment Formulation). Let f = (g, q, P. h), denote a set of nuisance functions and define:
(S1, = Pr(e S1)
E. [T] Si,
Pr(o S1, 8o)
Po (SS) = [oT),, So) [%]
ho (So) == Eg(g(S)) So]

---

Then 8, is the solution to the moment equation:
m(fo; fo) = me (f0: fo) + m.(00s fo) =
me(#:f) = [(c) (o(()) - h(Sb) - -p(S)) (o(T1,
p((o))
mo(0o; = E [1{) ((I, So) - g(S)))
Moreover, the moment m satisfies the Neyman orthogonality property with respect to f. Furthermore, satisfies a stronger double robustness property with respect to g and i.e. if we denote with O the solution to m(0; 0: Q: po. h) = 0, then:
Ee [T (o- (e-6))ll=
Pr(o) Pr(e) 1990
9112.0 11qu
$112.0
We estimate On by method of moments, solving the empirical analogues of these moment conditions, after estimating the nuisance functions f. We show that this estimator is asymptotically normal under relatively mild rate conditions the estimation of the nuisances in Appendix Theorem A.2 (Theorem 3.3 below, establishes the same result for more general case).
3.2 Double/Debiased Correction with Dynamic Adjustment
We consider the case of potentially serially correlated treatments in all periods. To achieve consistent estimates of the effects period treatments in this setting we need to again assume that all the relationships linear, albeit potentially high-dimensional:
S: = AT; + BS:- +
Y, = CS: + 1)
(13)
T++1 = DT,-1 + GS,-1 + St
where et, nt Se are i.i.d. random shocks. In this case, the main identification result in Lewis & Syrgkanis (2020), shows that:
[t,,, 1o) Σ ( to) = = (t, to)
(14)
where 0, :- >M O1.r and 01,, are interpreted as the dynamic effect of treatment at period 1 on the outcome at period 6, controlling for all future treatments. This quantity is equivalent to the effect assuming that all future treatments are zero, since the effects of each period treatment under this linear model are linearly separable. This property simplifies both the estimation and the interpretation of the effect quantity many causal quantities of potential interest collapse in this setting to the same object.
The identification results in Lewis & Syrgkanis (2200) also imply that we can identify the quantity 01, by estimating the effect Ti on the multi-period dynamically adjusted target outcome:
Y,
T.T.T 2=2
Te = 0 Ti
So + /41
(15)
for some exogenous mean-zero random shock variable μ1. This is the setup that we analyzed in the previous section, albeit with Y, replaced by yrlli and with Φ(T, S.) = T, The PLR assumption with respect to this dynamically adjusted target immediately holds by the equation above.
Furthermore, we need to assume that the invariance of the surrogate-outcome assumption holds, albeit now for the dynamically adjusted long-tenm term outcome:
adi (Si) : [Y," S.] = E. (Y S]]
(dynIR)
This assumption is more permissive in practice than the standard invariance assumption as we no longer require that the dynamic treatment policy in the observational data be the same in the
7

---

experimental data, but only that the adjusted outcomes retain the same relationship with the surrogates. For estimation, we can apply the same analysis as in the previous section. The differences are that we now have this new target long-term outcome and 2) we must additionally account for the variance of the Og terms that are being estimated by the dynamic adjustment algorithm. We articulate the full algorithm including these additions in Appendix [ We show below that the method of moments estimator with plug in nuisance estimates is asymptotically normal:
Theorem 3.3 (Surrogates with Dynamic Adjustment: Estimation and Asymptotic Normality), Let:
co co(20 = 1(e} (St) - T
(Si)
Ti +1fo}
(Y1Td)
90 (S1)
go (S So)
(o.c(2:0,50)
(2:0,.))
V2 M: 1r, = 1(0} ( E Tn. - - TTat
F
k=T+1
A2 M: w. (Z; so) = W2:00
Vr (,0,0,,o)). fo)l
where we use the short-hand notation Ye,r = Y. B|X, ST-1] and Y
and Pe. = To [T I ST-1]- Let he,r = B S--1] and PL,- (S,-)) E(T, S.-1]. Let f = {g. 9. p. h, {hr,s, Pr. Prrssssss denote the set of all muisance functions and let fo denote their true value. Consider the estimator based on the empirical version of the orthogonal moment, with plug-in in nuisance estimates f trained separate sample, i.e. 0 solves the empirical moment equation:
EEE 2/ 0,
En [2'o (Z; O2
En [e (Z;
m, f): =0
m. (0: f) = (Z:0,)) =
(16)
En [W (Z:0,,
If IIf roll2,0 op(n 1/4), for r {(hr,e, Pr, Pr, and if l| holle 0/(n)
|| polle = (p(n-1/4) and 1190 م112,0 l190 - 12,00 o Op(n (p(n-1/2) then:
Vn 00 -0-00) 00) t NO, Jo (2ie + Do,1 + 2o,2)) )
where Je, Ee, 1, E defined in the appendix.
4 Semi-Synthetic Experimental Evaluation
To evaluate the performance of our proposed estimation strategy we construct a semi-synthetic dataset that retains qualitative characteristics of data real-world incentive investments in customers at major corporation, while preserving confidentiality. The semi-synthetic data set preserves several common patterns that require thoughtful attention. The treatments, in this case incentive investments, are lumpy: in most periods most customers get no investments. Proxies, which include single period values of the outcome of interest, are highly auto-correlated over time. Treatments also corre-ooreelted, and correlated with past values of proxies. The data also include time-invariant controls that affect proxies and treatments in all periods.
To build the data we estimate a series of moments from a real-world dataset: a full covariance matrix of all proxies, treatments, and controls in one period and a series of linear prediction models (lassoCV) of each proxy and treatment on a set lags of each treatment, lags of each proxy, and time-invariant controls. Using these values, we draw key characteristics of each family of parameters. Finally, we use these new parameters to simulate proxies, treatments, and controls by drawing set of initial values from the covariance matrix and forward simulating to match intertemporal relationships from the transformed prediction models. Finally, we arbitrarily select proxy be the outcome of interest and construct the cumulative sum of this selected outcome over four or eight periods. For further details on the data generation process, see the appendix.
parameters from distributions matching the
The true treatment effects in the synthetic data are known functions of parameters from the linear prediction models. We then compare these true causal effeett to estimated effects using variety of
8

---

approaches, gradually incorporating our proposed innovations. Seuure the distribution of the lg estimation error (I foll2) for each approach over 100 simulated data sets. The top row plots the estimation error when estimating the effect on four periods outcomes, increasing the sample size of each simulation from left to right, while the bottom row shows the same for the effect on eight periods of outcome.
114020, nn 1.pet/yped 1.00)=320
IP2000, FLDGIMAM 7 7,00
1=5000 TUPUIAbb (t 11:001200
1-10200 FLSuNooumL, 1L00p=100
ail Acriogete
ail Auropiid
201 Aurpatt
Oyw Frea
Preet
turopeet
of pumpee
tree
Hl sumy pl
surogeeee
bola
ai sumple de
Now trett
cuttryet
lij budl
summy
Oier treat
wy
Now
Figure 2: Experimental performance for M 4 periods and M 8 periods.
Because we construct a single, long synthetic data set for this experiment is possible to estimate the treatment effects on realized long-term outcomes directly, unlike the typical use case for a surrogate approach. We begin by estimating the effect of each treatment at time t directly on the realized total outcomes, rather than the surrogate index, without dynamic adjustment. The blue, "total" bars in each panel of Figure Q display substantial bias from this method, which does nothing to control for future treatments. Because current treatments are positively correlated with future treatments in the semi-synthetic data, this first approach consistently overestimates treatment effects.
We then estimate the same set of treatment effects using the surrogate approach described in Section 3.1 still without dynamic adjustment. The distribution of estimation errors from this approach is represented in the orange "surrogate" bars. The estimated treatment effects are still substantially larger than the true effects on average, but the surrogate model exhibits slightly less bias than the direct "total" approach. Intuitively, because the surrogate approach is only capturing the relationship between treatment and outcome that passes through the surrogates it picks up less of the bias resulting from future correlated treatments than the direct approach.
The third set of green "adj. total" bars plot errors when estimating treatment effects on adjusted realized outcomes using the method of Lewis & Syrgkanis (2020), the preferred approach when all treatments of interest and realized long-run outcomes are available in a single data set. This method removes the effects of future treatments from the long-run outcome in a first step and, as expected, exhibits significantly less bias than the first two methods, particularly for reasonably large samples.
The final two bars illustrate the success of the adjusted surrogate approach described in Section 3.2 As illustrated by the red "adj. surrogate" bars, our proposed approach is highly accurate in predicting long-term effects with a performance comparable to that of having access to the raw long-term outcome itself. The final purple "new treat." bars show that the approach works equally well when considering the effect of novel treatment that appears only in the experimental sample and was not part of the dynamic adjustment. Overall, this methodology overcomes a common data limitation when considering long-term effects of novel treatments and expands the surrogate approach to consider a common, and previously problematic, pattern of serially correlated treatments.
Acknowledgments and Disclosure of Funding
The authors have no sources of additional funding to disclose.

---

References
Ai, C. and Chen, X. Efficient estimation models with conditional moment restrictions containing unknown functions. Econometrica, (1)6611795-184 2003.
Athey, S., Chetty, R., Imbens, G., and Kang, H. Estimating treatment effects using multiple surrogates: The role of the surrogate score and the surrogate index, 2020.
Begg, C. B. and Leung, D. H. On the use of surrogate end points randomized trials. Journal of the Royal Statistical Society: Series (Statistics in Sociery), 163(():15-22) 2000.
Bodory, H., Huber, M., and Lafférs, L.. Evaluating (weighted) dynamic treatment effects by double machine learning.
Chernozhukov, V., Escanciano, J. C., Ichimura, H., Newey, W. K., and Robins, J. M. Locally Robust Saki:11608.0003333 July 2016.
Semiparametric Estimation. arXiv e-prints,
Chernozhukov, V., Chetverikov, D., Demirer, M., Dufto, E., Hansen, C., Newey, W., and Robins, Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68, 01 2018a. ISSN 1368-4221. doi: 10.111 101111/ecj..120990 URL https
12097
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 21(1):C1-C68, 2018b.
Frangakis, C. E. and Rubin, D. B. Principal stratification in causal inference. Biometrics, 58(1): 21 21-29, 2002.
Freedman, L. S., Graubard, B. I., and Schatzkin, A. Statistical validation of intermediate endpoints for chronic diseases. Statistics in medicine, 11(2):167-778, 1992.
Kallus, N. and Uehara, M. Double reinforcement learning for efficient off-policy evaluation in markov decision processes, 2019a.
Kallus, N. and Uehara, M. Efficiently breaking the curse of horizon in off- evaluation with double reinforcement learning, 2019b.
Lewis, G. and Syrgkanis, V. Double/debiased machine learning for dynamic treatment effects, 2020.
Lok, J. J. and DeGruttola, V. Impact of time to start treatment following infection with application to initiating haart in hiv-positive patients. Biometrics, 68(3):745-754, 2012.
Neyman, J. c(a) tests and their use. Sankhya, pp. 1-21, 1979.
Nic, X., Brunskill, E., and Wager, S. Learning when-to-tre treat policies, 2019.
Petersen, M., Schwab, J., Gruber, S., Blaser, N., Schomaker, M., and van der Laan, M. Targeted maximum likelihood estimation for dynamic and static longitudinal marginal structural working models. Journal of causal inference, 2(2): 2014.
Prentice, R. L. Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in medicine, 8(4):431-440) 1989.
Robins, J. A new approach to causal inference in mortality studies with sustained exposure period- application control of the healthy worker survivor effect. Mathematical modelling, 7(9-12): 1512-1512, 1986.
Robins, M. Correcting for non-compliance in randomized trials using structural nested mean models. Communications in Statistics-Thest Theory and methods, 23(8):2379-22 2212222,, 1994.
Robins, J. M. and Ritov, Y. Toward a curse dimensionality appropriate (coda) asymptotic theory for semi semi-parametrie models. Statistics in medicine, 16(3):285-319 1997.
10

---

Robins, J. M., Blevins, D., Ritter, G., and Wulfsohn, M. G-estimation of the effect of prophylaxis therapy for pneumocystis carinii pneumonia on the survival of aids patients. Epidemiology, pp- 319-336, 1992.
Robins, J. M., Hernan, M. A., and Brumback, B. Marginal structural models and causal inference in epidemiology, 2000.
Robinson, P. M. Root-n-consistent semiparametric regression. Econometrica: Journal of the Econometric Society, pp. 931-954, 1988.
Singh, R., Xu, L., and Gretton, A. Kernel methods for policy evaluation: Treatment effects, mediation analysis, and off-policy planning. arXiv preprint TrXiv:2010.04855, 2020.
Thomas, P. S. and Brunskill, E. Data-efficient off-po policy evaluation for reinforcement learning. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML'16, pp. 2139-2148. JMLR.org. 2016.
Vansteclandt, S. and Sjolander, A. Revisiting g-estimation of the effect of time- mime-ngrying exposure subject to time-varying confounding. Epidemiologic Methods, 5(1):37 - 56, 2016. doi: https: 10tp:/0000.111/000000000000 5/1/article-p37. xml
URLhttps: http://wwwwwerr www.wwgrurter.....mmmrmrrrrmo wwwpe/eerr.o.vvieeeeeeeee waanm/view/owwoe www.a/aaaa//aaas//////aa em/
Vansteelandt, S., Joffe, M., et al. Structural nested models and g-estimation: the partially realized promise. Statistical Science, 29(4)7707-731 2014.


Reviews:
[{"review_id": "_wD9-GC69pY", "reviewer": "Reviewer_SfcW", "summary": "This paper proposes an estimation strategy for treatment effects on long term outcomes in a difficult setting with sequentially correlated treatments. They consider a certain case where unobserved values of the treatment show up in a small \"new\" dataset without long term outcomes recorded but where there is access to an \"old\" dataset with old treatments that include long term outcomes. Assuming certain nuisance functions (conditional expectations of various parts of the graphical model) can be estimated consistently with [insert favorite ML method here], they use Double ML to derive the asymptotic normality of their treatment effect estimators, as well as double robustness with respect to two of the nuisance functions.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Nice work! In general the mathematical ingredients and how they piece together are clear but the writing is pretty unclear. Warning: I am fairly new to DML (a few months of reading) and have only seen it in simpler non-sequential settings eg basic ATE. However, all of my questions are not about the final application of DML / orthogonality itself but the setup leading up to it.\n\nSome comments/questions\n\n- how can A be px1 when the treatment vector is K dim ? \n\n- This caused a lot of confusion for me when reading this paper, even though I could appreciate a lot of the piece-by-piece without knowing the answer to this question: could you describe the precise usage of the word \"novel\" in general in this work? The pharmaceutical example is estimation of causal effect of a new drug, but wouldn't this correspond to adding an extra treatment variable T,k+1 to the conditioning set of models like Y|T,S for a given timestep? That is Y|T1,...Tk,S vs Y|Y1,...,Tk+1,S?  And the models on the observational set are not estimated on such an expanded conditioning set? Or is it that you are estimating outcomes Ybar under some new treatment *value* T* in R^k such that T* not observed in observational data? Can you try to clarify here what precisely is meant by novel with an example? What assumptions are necessary on the nature of the new treatment?\n\n- line 69: small off by one error in the M term sum, kappa should start at 1 instead of 0 as done later in line 165\n\n- lines 94-96 since I had trouble understanding this description in words how correlated treatments cause bias but I understood the mathematical example starting at line 109. Maybe you can mention at 94-96 that this is demonstrated below?\n\n- lines 135 (under eq 5): it would be useful to point out that gamma(I+B)S,t shows up earlier as the first term in eq2: g0(S,t). Likewise that eq 6 (line 137) recovers two of the terms in line 121 just above eq 1. Help the reader realize exactly where they saw these terms before.\n\n- (important) After eq 6, a nice summary sentence, enumerated list, or formal Algorithm environment would be good to make sure people understand the algorithm. If I understand correctly it is  (Step one) First predict Y,t+1  | T,t+1, St as a linear function of T,T+1 and S,t that takes the specific form in (4) (Step 2) define adjusted outcome  Ybar,adj,t = Yt + Y,t+1 - alpha,t+1,Tt+1 which depends on the first step estimate through alpha (Step 3) Then predict this adjusted outcome from Tt,S,t-1 (Step 4) Finally the coef. in front of T,t (in a correctly specified model) is the causal effect theta0. Is it right? \n\n- lines 142-144, what do you mean by \"both\" approaches, do you mean (one) surrogate index without dynamic adjustment, and (two) dynamic adjustment? \n\n- line 164 to eq 7: you used E_o and E_e before defining them in 171\n\n- sections 3.1 vs 3.2, DML / asymptotic normality: You mention in both sections that because of the orthogonal representation that normality results hold. For 3.1 (unadjusted) you say it in words, and for 3.2 (adjusted) you prove it (theorem 3.3). Does it trivially hold for section 3.1 (unadjusted) because of the proof in 3.2? Or just because it's the standard case already covered by DML?\n\n- what about a small fully-synthetic experiment with a correlated earlier and later treatment where you are estimating the effect of the earlier treatment without adjustment and where you plot estimation bias vs strength of correlation (through either of the two routes: kappa or lambda*S)\n\n- is there no public dataset that fits this problem setup that you can also include in addition to the semi-synthetic experiment?\n\n- This paper is definitely missing a conclusion.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "Ik6dJUXTuZ", "reviewer": "Reviewer_ZEEt", "summary": "This paper develops a method for estimating cumulative effects of treatments when available data is limited to historical data of old treatments and short-term data of new treatments. A key challenge in this setting is that treatment assignments over time are correlated with each other, making it difficult to isolate the effects of individual treatments on the cumulative, long-term outcomes. This paper expands on previous methodology for this problem which make use of \"surrogate indices\" that mediate the effects of treatment on outcomes at each time step. Specifically, the authors estimate the isolated long-term reward of a treatment in the absence of future treatments. The methodology allows for continuous treatments and has double robustness properties (with respect to the surrogate index and surrogate score models). Semi-synthetic experiments show that the proposed dynamic adjustment approach lowers estimation error.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of this paper is is very interesting and seems quite important: estimating long term effects of treatments from primarily historical data is important evidence for making early-stage decisions about new treatment policies. The authors state that their method combines (and extends) the surrogate index approach of Athey et al. (2020), double machine learning (Chernozhukov et al., 2018a), and dynamic effect adjustment (Lewis & Syrgkanis, 2020). I must admit that, of these techniques, I am only familiar with double machine learning. However, to the best of my understanding of the paper, the authors elegantly combine and extend these approaches to create a holistic solution to the long-term effect estimation problem.\n\nAs someone unfamiliar with the surrogate index approach of Athey et al. (2020), at times I had trouble following the methodology. For example, I do not believe the authors provide examples of surrogate variables, except to say that a single time period's outcome or \"other measures... of a customer's trajectory\" could be used as proxies. I think the investment firm example from the introduction could be fleshed out a little more to make clear what the proxies would be in that scenario.\n\nSome of the organization in Sections 2 and 3 was confusing to me. In Section 2 the authors begin by describing the estimation strategy of Athey et al. (2020), then informally describe the assumptions required for identification before describing violations of these assumptions that would induce bias in the described estimation strategy. I think this section could be streamlined to more concisely state the key information that will be needed for problem setup and to understand the method. The target estimand and the assumptions required should be formally stated up front in Section 2. Further, the causal path assumption (that at each time step surrogates entirely mediate the effects of treatments on outcomes) is never formally stated. The graph in Fig 1 is very helpful for visualizing this, but it should be accompanied by the formal statements. Given this setup, the authors could then introduce the unadjusted surrogate index and clearly define the bias (right now the bias is shown in equation 3 but the authors do not draw much attention to this). The beginning of Section 3 and Section 3.3 were written in a way where it was much easier to follow. I think the authors should use this style of previewing what will be shown in each subsection in other parts of the paper as well.\n\nThe results in this paper are interesting and important. I think the paper could be greatly improved by cleanly and concisely introducing  the formal problem setup and previewing section content to help guide the reader.\n\n\nUPDATE:\nThanks to the reviewers for their response. I remain positive about this work and will be maintaining my score. I definitely think the suggested improvements to clarity will help this work be appreciated by a broader audience (that includes myself!). I also think it should be possible for the authors to flesh out the hypothetical example(s) without revealing proprietary information---surely the proposed method is useful beyond its particular application at your organization! I genuinely believe these changes would take this paper from \"good\" to \"great.\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should at least make an explicit statement somewhere in the paper regarding limitations or that they do not foresee negative impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "APkshgvLeEm", "reviewer": "Reviewer_JAeM", "summary": "The paper considers the problem of estimating a long-term treatment effect. It considers the setting where there are two datasets: an observational dataset, which contains the long-term outcome, the treatment and the surrogate variable, and an experimental dataset, which contains only the treatment and the surrogate variable. The authors generalize the surrogate index method proposed by Athey et al. (2020), by considering dynamic treatment policies. Their new proposed method is a combination of the surrogate index approach, the double machine learning approach and the dynamic treatment effect estimation approach. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The question considered is new and very interesting. The proposed method is a novel combination of the three approaches: the surrogate index approach, the double machine learning approach and the dynamic treatment effect estimation approach. My concern is that, in terms of methodology innovations, there is not too much new outside of the three existing approaches. \n\nQuality: As far as I checked, all claims are well supported. The theories and experimental results are explained very clearly. \n\nClarity: The paper is well written and I enjoyed reading it. Section 2.1 is particularly helpful in demonstrating the main idea. For the numerical experiments, I think it would be helpful to explain the five methods in a clearer way. Currently it is a bit confusing what the purple bar does. \n\nSignificance: The problem considered by the authors are important. I agree that treatment policies are often dynamic in practice. The solution to the problem is nice and strong; it has good theoretical properties and performs good empirically. Again, as mentioned above, my concern is that I don’t see too much new outside of the three existing methods. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "8nhbCMcRD4H", "reviewer": "Reviewer_5T1d", "summary": "This paper tackles the problem of estimating the long-term effect of some new treatment in the case where the treatment assignment is decided adaptively. Athey et al.(2020) proposed the long-term effect estimation for the stable treatment assignment case. This paper applied the method proposed in Lewis & Sygkanis(2020) to adjust the approach of Athey et al.(2020) for the case where the treatment assignment is decided adaptively.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Estimating the long-term effect is an important problem in the application of causal inference and econometrics. However, I am quite not sure that this research is well motivated. Do we really need to know the long-term effect of a particular arm while using some dynamic treatment regime? I appreciate it if you could explain and add some examples.\n\nFor the experimentation, you should report your result of the synthetic data experiment reported in the appendix with more detailed information about the data generating process.\n\nSome minor issues\n\n- line 121: what is \"noise\" in the equation?\n- line 121: the definition of I is ambiguous here\n- section 3.2: the actual estimation step should be explained here. At least, you should refer to the appendix here.\n- line 280: \"for this excercise,\" ?\n- line 282: the definition of \"total\" is somewhat ambiguous. For me, the definition in the appendix helped.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They adequately addressed the limitation in their paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks to the authors for this engaging and submission.  The reviewers were all quite positive about this work, and I'm happy to recommend acceptance.

One shared concern among the reviewers is clarity of the submission --- as reviewer ZEEt notes, sections 2 and 3 are a bit confusing in their presentation, and should be streamlined in the final version.  And as reviewer SfcW discussed, the incorporation of a new synthetic experiment will help understand the relationship between estimation bias and strength of correlation, even if placed in the appendix and referred to in the main text.
Decision: Accept (Poster)