Estimating the Long-Term Effects of Novel Treatments

1 Introduction . Businesses invent new ways of interacting with their customers . Marketing departments devise new campaigns . Pharmaceutical companies roll out trials of new drugs . In many cases , decision makers expect these novel treatments to affect outcomes like sales or health over several years , but they are eager to begin evaluating these innovations within months of deployment . We propose an estimation methodology that leverages historical data , collected under earlier treatment regimes , to derive early estimates of the long-term effects of novel , newly-deployed treatments . Athey et al . ( 2020 ) develop a surrogate index as one solution to this problem . This method relies on an assumption that causal effects on long-term outcomes are channeled through a set of observable short-term proxies . This assumption allows one to use the historical long-term data set to learn a mapping from short-term signals to a projected long-term reward and subsequently estimate the causal effect of novel treatments on this index of surrogates . In practice , treatment policies are often dynamic : treatments are assigned repeatedly , and their assignments depend on past treatments and short-term outcomes . Unfortunately , this pattern can easily break the assumptions needed for the surrogate index to work . For example , suppose that a firm offers multiple investments to a particular customer in the historical data and these investments are auto-correlated , i.e . if a customer receives an investment this month , then they will receive an investment with high probability in one of the subsequent months . These future investments can substantially increase the long-term outcome of interest , and this increase will be attributed to the short-term proxies . The surrogate index thus formed will tend to over-predict long-run outcomes , and so when it is used to measure the treatment effect of some new treatment in the short-run data , the estimated treatment effect will be bigger ( in absolute magnitude ) than the truth . ∗Correspondence to : Vasilis Syrgkanis < vasy @ microsoft.com > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The main methodological innovation of this paper is to use the dynamic treatment effect analysis of Lewis & Syrgkanis ( 2020 ) on the historical data in order to create an unbiased , dynamically adjusted surrogate index . Our new index takes the interpretation of the projected long-term reward in the absence of any future treatments . This model produces unbiased causal effect estimates of the long-term effects of the novel treatments , even with auto-correlated treatments . A second contribution of the paper is to generalize the final causal analysis step to allow for multiple continuous treatments rather than a single binary treatment , as in Athey et al . ( 2020 ) . Our new estimator for this expanded surrogate approach allows for the construction of valid confidence intervals , even when using flexible machine learning models at both stages of the estimation to deal with high-dimensional data . In short , we show how one can combine three recently developed techniques , i ) the surrogate index approach of Athey et al . ( 2020 ) , ii ) the double machine learning approach of Chernozhukov et al . ( 2018a ) and iii ) the dynamic treatment effect estimation approach of Lewis & Syrgkanis ( 2020 ) , in a single data analysis pipeline to estimate treatment effects in the presence of dynamic treatment policies . Our work lies in the broader field of estimating causal effects with machine learning and Neyman orthogonality ( Neyman , 1979 ; Robinson , 1988 ; Ai & Chen , 2003 ; Chernozhukov et al. , 2016 ; Chernozhukov et al. , 2018b ) . Moreover , it relates to the work on machine learning estimation of treatment effects in the dynamic treatment regime ( Nie et al. , 2019 ; Thomas & Brunskill , 2016 ; Petersen et al. , 2014 ; Kallus & Uehara , 2019b , a ; Lewis & Syrgkanis , 2020 ; Bodory et al . ; Singh et al. , 2020 ) and on structural nested models in biostatistics ( Robins , 1986 ; Robins et al. , 1992 ; Robins , 1994 ; Robins & Ritov , 1997 ; Robins et al. , 2000 ; Lok & DeGruttola , 2012 ; Vansteelandt et al. , 2014 ; Vansteelandt & Sjolander , 2016 ) . Finally , it relates to the surrogacy literature in causal inference ( Prentice , 1989 ; Begg & Leung , 2000 ; Frangakis & Rubin , 2002 ; Freedman et al. , 1992 ) . In the following sections we build on insights in these works to design a new , streamlined method that resolves several difficult aspects of a common applied problem . 2 Problem and Methodology . We illustrate the difficulties of our problem and how we our method resolves them with a running example of a firm making investments in its customers in order to increase subsequent purchases . In this section we describe the problem set up and data requirements and use a simple two-period model to build intuition for how our approach estimates long-term causal effects in this environment and why earlier approaches fail . In the following section we provide formal proofs of our methodology in the general , multi-period case . 2.1 Problem statement . A firm has a number of distinct treatments T1 , T2 . . . Tk it can offer to customers , such as discounts or access to special services . At each period t ( say , a month ) , a vector of treatments Ti , t = ( Ti , t,1 , . . . , Ti , t , k ) ∈ Rk , is applied to each customer i . We also observe a vector of p characteristics Xi , t = ( Xi , t,1 , . . . , Xi , t , p ) ∈ Rp , some of which are constant within customer ( e.g . industry ) and some of which vary over time and customer ( e.g . last month ’ s revenue ) . We observe the outcome of interest Yi , t ∈ R ( e.g . monthly revenue ) . Treatments may affect outcomes over many future periods , and this horizon could vary by treatment . We are therefore interested in identifying the average effect of each treatment at some period t , on the cumulative outcome in the subsequent M periods , i.e . Ȳi , t = ∑M κ=1 Yi , t+κ . Suppose direct inference is impossible because we have not yet observed the subsequent M periods following the introduction of a new treatment . Instead , we have access to a vector of d short-term proxies/surrogates , Si , t = ( Si , t,1 , . . . , Si , t , d ) that act as leading indicators for our target outcome . In this example , we would like surrogates that capture a customer ’ s trajectory such as the next few months of revenue , intensity of product usage , new contract commitments , and participation in company events . 2.2 Dynamic Adjusted Surrogate Index . The key innovation of all surrogate-based solutions is to use the proxy variables to bridge two incomplete data sets to estimate Ȳi , t . A long-term observational data set need only include customer characteristics Xi , t , surrogates Si , t+1 , realized M -period outcomes Ȳi , t . A second short-term , experimental data set , includes customer features , Xi , t , surrogates , Si , t , and treatments Ti , t . This experimental data set is restricted to the period where all treatments of interest have been introduced , but only requires a few months of leading surrogates rather than M periods of leading outcomes . We propose the following estimation strategy : 1 . Use the observational data set to estimate the expected long-term outcome net of any postperiod t treatments using the recursive methodology of Lewis & Syrgkanis ( 2020 ) . This adjusted outcome , Ȳ adjit , is the expected cumulative outcome starting in period t under the assumption that customer i receives no further treatments over the next M periods . 2 . Still using the observational data set , construct a model that predicts this adjusted outcome from Si , t and Xi , t : gadj0 ( Si , t , Xi , t ) : = E [ Ȳ adj i , t | Si , t , Xi , t ] . ( surr . model ) Any machine learning estimation algorithm can be used to construct ĝ . 3 . Finally , calculate the adjusted surrogate index using the surrogates in the experimental data set : Ii , t : = ĝ ( Si , t , Xi , t ) . ( predicted surrogate index ) This predicted index becomes the outcome in the final causal model . This approach will be consistent for the treatment effects under three assumptions , which we illustrate as a causal graph in Figure 1 . 1 . Conditional Independence : There are no paths from current treatments to future outcomes except through current or future surrogates . Formally , Tit ⊥ Yis|Xit , Sit .. Sis for all s ≥ t. 2 . Stability : Relationships between current and future surrogates and between surrogates and outcomes , marked in red in the figure , are constant over time and across samples . 3 . Unconfoundedness : There are no variables beyond our observed set that directly affect both the treatment assignment and outcome . Note that the first assumption allows relationships between current and future treatments and between current treatments and future surrogates , in contrast to the canonical surrogate model . The stability assumption does not extend to the relationships between treatments and surrogates , which allows for the introduction of novel treatments in the experimental data not found in the observational data . The model is able to account for steady growth in a natural way : as the surrogates grow , the predicted outcomes grow too . 2.3 A two-period example . For conciseness , we now collapseXi , t and Si , t−1 so that the surrogates at each period and the controls in the next period are all denoted by Si , t−1 . In other words , all customer observable characteristics in the next period are candidate surrogates and become controls for the period after next . This is without loss of generality . For the remainder of this section only , we further simplify by assuming a linear Markovian model of how the random variables evolve over time and focusing on a single scalar investment ( k = 1 ) . With these simplifications , the evolution of the random variables can be described in three equations : Si , t = ATi , t +BSi , t−1 + i , t ( evolution of controls ) Yi , t = γ ′Si , t + ζi , t ( outcome equation ) Ti , t+1 = κTi , t + λ ′Si , t + ηi , t ( treatment policy ) where A is an p× 1 matrix , B is a p× p matrix , γ , λ are p-dimensional vectors and κ is a scalar . The terms i , t , ζi , t , ηi , t are exogenous mean-zero independent noise terms . For conciseness we drop the customer index i in the remaining equations . Target effect estimand With two periods , the long term outcome Ȳi , t = Yi , t + Yi , t+1 . The effect of treatment Tt on Ȳt can be derived from the structural model equations : Ȳt = γ ′St + γ ′St+1 + noise = γ′ATt+1 + γ′ ( B + I ) ATt + γ′ ( B + I ) BSt−1 + noise2 Thus we see that the effect of Tt on Yt+1 + Yt , assuming future treatments are held constant , is : θ0 : = γ ′ ( B + I ) A . ( 1 ) Bias from a standard surrogate index Athey et al . ( 2020 ) propose training a surrogate index using the realized cumulative outcome Ȳt on St , essentially our proposed step 2 using raw instead of adjusted outcomes . Following that approach , we would estimate : g0 ( St ) = E [ Ȳt | St ] = γ′ ( I +B ) St + γ′AE [ Tt+1 | St ] ( 2 ) Subsequently , if we estimate the causal effect of Tt on g0 ( St ) , then this effect would be : θ∗ = θ0 + γ ′A E [ E [ Tt+1 | St ] Tt ] E [ T 2t ] ( 3 ) The standard estimate contains a bias stemming from the fact that investment today can lead to higher investment tomorrow . As illustrated in equation above , this bias can stem from two channels , both of which are plausible in many use cases . In our example , a customer that is treated once may build a stronger relationship with the firm and receive further treatments ( i.e . κ ≥ 0 in the treatment policy ) . Alternatively , if the firm targets treatments to fast-growing customers , any treatment that successfully drives surrogate metrics higher will make the customer eligible for further treatment ( λ′A ≥ 0 ) . Dynamic adjustment With only two periods , our first step simply requires estimating the singleperiod causal effect of Tt+1 on Yt+1 , controlling for St . This conditional expectation is equal to : E [ Yt+1 | Tt+1 , St ] = γ′ATt+1 + γ′BSt . ( 4 ) We can now create an adjusted long-term outcome , Ȳ adjt : = Yt + Yt+1 − α′t+1Tt+1 , where αt+1 is our estimate of γ′A from this first step . The dynamically adjusted surrogate index builds this adjustment into equation ( 2 ) above : gadj0 ( St ) = E [ Yt | St ] + E [ Yt+1 − γ′ATt+1 | St ] = γ′St + E [ γ′BSt | St+1 ] = γ′ ( I +B ) St ( 5 ) This new index captures the projected M -period outcomes as if the customer was offered no future treatments . When we estimate the effect of the treatment based on this adjusted surrogate index we recover : E [ gadj0 ( St ) | Tt , St−1 ] = γ′ ( I +B ) E [ St | Tt , St−1 ] = γ′ ( I +B ) ATt + γ ′ ( I +B ) BSt−1 ( 6 ) With the dynamic adjustment , the coefficient in front of Tt that we recover is the true causal effect θ0 = γ ′ ( I +B ) A. Lewis & Syrgkanis ( 2020 ) show how this adjustment approach can be extended to many periods , via a recursive peeling process , and also to high-dimensional surrogates and controls via a dynamic double machine learning approach . Estimating causal effects of new treatments With this adjusted surrogate index in hand , we can also estimate the long-term effect of any other treatment that was introduced more recently . In this example , the stationarity assumption of the proposed surrogate approaches requires that B , which governs how the surrogate evolves , and γ , which governs how surrogates translate to per-period outcomes , do not change between the observational and experimental data-sets . These two parameters govern how surrogates today relate to future outcomes in the absence of any treatment . Under such a condition , if we introduce a new treatment T newt , which has a different effect A new on the surrogates ( and hence on the long-term outcome ) , then the effect of this treatment on the long-term outcome is θnew0 = γ ′ ( I + B ) Anew . This θnew0 is exactly the outcome of estimating the causal effect of T newt on g adj 0 ( St+1 ) controlling for St .

Summary: This paper proposes an estimation strategy for treatment effects on long term outcomes in a difficult setting with sequentially correlated treatments. They consider a certain case where unobserved values of the treatment show up in a small "new" dataset without long term outcomes recorded but where there is access to an "old" dataset with old treatments that include long term outcomes. Assuming certain nuisance functions (conditional expectations of various parts of the graphical model) can be estimated consistently with [insert favorite ML method here], they use Double ML to derive the asymptotic normality of their treatment effect estimators, as well as double robustness with respect to two of the nuisance functions.