Estimating the Long-Term Effects of Novel Treatments

1 Introduction . Businesses invent new ways of interacting with their customers . Marketing departments devise new campaigns . Pharmaceutical companies roll out trials of new drugs . In many cases , decision makers expect these novel treatments to affect outcomes like sales or health over several years , but they are eager to begin evaluating these innovations within months of deployment . We propose an estimation methodology that leverages historical data , collected under earlier treatment regimes , to derive early estimates of the long-term effects of novel , newly-deployed treatments . Athey et al . ( 2020 ) develop a surrogate index as one solution to this problem . This method relies on an assumption that causal effects on long-term outcomes are channeled through a set of observable short-term proxies . This assumption allows one to use the historical long-term data set to learn a mapping from short-term signals to a projected long-term reward and subsequently estimate the causal effect of novel treatments on this index of surrogates . In practice , treatment policies are often dynamic : treatments are assigned repeatedly , and their assignments depend on past treatments and short-term outcomes . Unfortunately , this pattern can easily break the assumptions needed for the surrogate index to work . For example , suppose that a firm offers multiple investments to a particular customer in the historical data and these investments are auto-correlated , i.e . if a customer receives an investment this month , then they will receive an investment with high probability in one of the subsequent months . These future investments can substantially increase the long-term outcome of interest , and this increase will be attributed to the short-term proxies . The surrogate index thus formed will tend to over-predict long-run outcomes , and so when it is used to measure the treatment effect of some new treatment in the short-run data , the estimated treatment effect will be bigger ( in absolute magnitude ) than the truth . ∗Correspondence to : Vasilis Syrgkanis < vasy @ microsoft.com > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The main methodological innovation of this paper is to use the dynamic treatment effect analysis of Lewis & Syrgkanis ( 2020 ) on the historical data in order to create an unbiased , dynamically adjusted surrogate index . Our new index takes the interpretation of the projected long-term reward in the absence of any future treatments . This model produces unbiased causal effect estimates of the long-term effects of the novel treatments , even with auto-correlated treatments . A second contribution of the paper is to generalize the final causal analysis step to allow for multiple continuous treatments rather than a single binary treatment , as in Athey et al . ( 2020 ) . Our new estimator for this expanded surrogate approach allows for the construction of valid confidence intervals , even when using flexible machine learning models at both stages of the estimation to deal with high-dimensional data . In short , we show how one can combine three recently developed techniques , i ) the surrogate index approach of Athey et al . ( 2020 ) , ii ) the double machine learning approach of Chernozhukov et al . ( 2018a ) and iii ) the dynamic treatment effect estimation approach of Lewis & Syrgkanis ( 2020 ) , in a single data analysis pipeline to estimate treatment effects in the presence of dynamic treatment policies . Our work lies in the broader field of estimating causal effects with machine learning and Neyman orthogonality ( Neyman , 1979 ; Robinson , 1988 ; Ai & Chen , 2003 ; Chernozhukov et al. , 2016 ; Chernozhukov et al. , 2018b ) . Moreover , it relates to the work on machine learning estimation of treatment effects in the dynamic treatment regime ( Nie et al. , 2019 ; Thomas & Brunskill , 2016 ; Petersen et al. , 2014 ; Kallus & Uehara , 2019b , a ; Lewis & Syrgkanis , 2020 ; Bodory et al . ; Singh et al. , 2020 ) and on structural nested models in biostatistics ( Robins , 1986 ; Robins et al. , 1992 ; Robins , 1994 ; Robins & Ritov , 1997 ; Robins et al. , 2000 ; Lok & DeGruttola , 2012 ; Vansteelandt et al. , 2014 ; Vansteelandt & Sjolander , 2016 ) . Finally , it relates to the surrogacy literature in causal inference ( Prentice , 1989 ; Begg & Leung , 2000 ; Frangakis & Rubin , 2002 ; Freedman et al. , 1992 ) . In the following sections we build on insights in these works to design a new , streamlined method that resolves several difficult aspects of a common applied problem . 2 Problem and Methodology . We illustrate the difficulties of our problem and how we our method resolves them with a running example of a firm making investments in its customers in order to increase subsequent purchases . In this section we describe the problem set up and data requirements and use a simple two-period model to build intuition for how our approach estimates long-term causal effects in this environment and why earlier approaches fail . In the following section we provide formal proofs of our methodology in the general , multi-period case . 2.1 Problem statement . A firm has a number of distinct treatments T1 , T2 . . . Tk it can offer to customers , such as discounts or access to special services . At each period t ( say , a month ) , a vector of treatments Ti , t = ( Ti , t,1 , . . . , Ti , t , k ) ∈ Rk , is applied to each customer i . We also observe a vector of p characteristics Xi , t = ( Xi , t,1 , . . . , Xi , t , p ) ∈ Rp , some of which are constant within customer ( e.g . industry ) and some of which vary over time and customer ( e.g . last month ’ s revenue ) . We observe the outcome of interest Yi , t ∈ R ( e.g . monthly revenue ) . Treatments may affect outcomes over many future periods , and this horizon could vary by treatment . We are therefore interested in identifying the average effect of each treatment at some period t , on the cumulative outcome in the subsequent M periods , i.e . Ȳi , t = ∑M κ=1 Yi , t+κ . Suppose direct inference is impossible because we have not yet observed the subsequent M periods following the introduction of a new treatment . Instead , we have access to a vector of d short-term proxies/surrogates , Si , t = ( Si , t,1 , . . . , Si , t , d ) that act as leading indicators for our target outcome . In this example , we would like surrogates that capture a customer ’ s trajectory such as the next few months of revenue , intensity of product usage , new contract commitments , and participation in company events . 2.2 Dynamic Adjusted Surrogate Index . The key innovation of all surrogate-based solutions is to use the proxy variables to bridge two incomplete data sets to estimate Ȳi , t . A long-term observational data set need only include customer characteristics Xi , t , surrogates Si , t+1 , realized M -period outcomes Ȳi , t . A second short-term , experimental data set , includes customer features , Xi , t , surrogates , Si , t , and treatments Ti , t . This experimental data set is restricted to the period where all treatments of interest have been introduced , but only requires a few months of leading surrogates rather than M periods of leading outcomes . We propose the following estimation strategy : 1 . Use the observational data set to estimate the expected long-term outcome net of any postperiod t treatments using the recursive methodology of Lewis & Syrgkanis ( 2020 ) . This adjusted outcome , Ȳ adjit , is the expected cumulative outcome starting in period t under the assumption that customer i receives no further treatments over the next M periods . 2 . Still using the observational data set , construct a model that predicts this adjusted outcome from Si , t and Xi , t : gadj0 ( Si , t , Xi , t ) : = E [ Ȳ adj i , t | Si , t , Xi , t ] . ( surr . model ) Any machine learning estimation algorithm can be used to construct ĝ . 3 . Finally , calculate the adjusted surrogate index using the surrogates in the experimental data set : Ii , t : = ĝ ( Si , t , Xi , t ) . ( predicted surrogate index ) This predicted index becomes the outcome in the final causal model . This approach will be consistent for the treatment effects under three assumptions , which we illustrate as a causal graph in Figure 1 . 1 . Conditional Independence : There are no paths from current treatments to future outcomes except through current or future surrogates . Formally , Tit ⊥ Yis|Xit , Sit .. Sis for all s ≥ t. 2 . Stability : Relationships between current and future surrogates and between surrogates and outcomes , marked in red in the figure , are constant over time and across samples . 3 . Unconfoundedness : There are no variables beyond our observed set that directly affect both the treatment assignment and outcome . Note that the first assumption allows relationships between current and future treatments and between current treatments and future surrogates , in contrast to the canonical surrogate model . The stability assumption does not extend to the relationships between treatments and surrogates , which allows for the introduction of novel treatments in the experimental data not found in the observational data . The model is able to account for steady growth in a natural way : as the surrogates grow , the predicted outcomes grow too . 2.3 A two-period example . For conciseness , we now collapseXi , t and Si , t−1 so that the surrogates at each period and the controls in the next period are all denoted by Si , t−1 . In other words , all customer observable characteristics in the next period are candidate surrogates and become controls for the period after next . This is without loss of generality . For the remainder of this section only , we further simplify by assuming a linear Markovian model of how the random variables evolve over time and focusing on a single scalar investment ( k = 1 ) . With these simplifications , the evolution of the random variables can be described in three equations : Si , t = ATi , t +BSi , t−1 + i , t ( evolution of controls ) Yi , t = γ ′Si , t + ζi , t ( outcome equation ) Ti , t+1 = κTi , t + λ ′Si , t + ηi , t ( treatment policy ) where A is an p× 1 matrix , B is a p× p matrix , γ , λ are p-dimensional vectors and κ is a scalar . The terms i , t , ζi , t , ηi , t are exogenous mean-zero independent noise terms . For conciseness we drop the customer index i in the remaining equations . Target effect estimand With two periods , the long term outcome Ȳi , t = Yi , t + Yi , t+1 . The effect of treatment Tt on Ȳt can be derived from the structural model equations : Ȳt = γ ′St + γ ′St+1 + noise = γ′ATt+1 + γ′ ( B + I ) ATt + γ′ ( B + I ) BSt−1 + noise2 Thus we see that the effect of Tt on Yt+1 + Yt , assuming future treatments are held constant , is : θ0 : = γ ′ ( B + I ) A . ( 1 ) Bias from a standard surrogate index Athey et al . ( 2020 ) propose training a surrogate index using the realized cumulative outcome Ȳt on St , essentially our proposed step 2 using raw instead of adjusted outcomes . Following that approach , we would estimate : g0 ( St ) = E [ Ȳt | St ] = γ′ ( I +B ) St + γ′AE [ Tt+1 | St ] ( 2 ) Subsequently , if we estimate the causal effect of Tt on g0 ( St ) , then this effect would be : θ∗ = θ0 + γ ′A E [ E [ Tt+1 | St ] Tt ] E [ T 2t ] ( 3 ) The standard estimate contains a bias stemming from the fact that investment today can lead to higher investment tomorrow . As illustrated in equation above , this bias can stem from two channels , both of which are plausible in many use cases . In our example , a customer that is treated once may build a stronger relationship with the firm and receive further treatments ( i.e . κ ≥ 0 in the treatment policy ) . Alternatively , if the firm targets treatments to fast-growing customers , any treatment that successfully drives surrogate metrics higher will make the customer eligible for further treatment ( λ′A ≥ 0 ) . Dynamic adjustment With only two periods , our first step simply requires estimating the singleperiod causal effect of Tt+1 on Yt+1 , controlling for St . This conditional expectation is equal to : E [ Yt+1 | Tt+1 , St ] = γ′ATt+1 + γ′BSt . ( 4 ) We can now create an adjusted long-term outcome , Ȳ adjt : = Yt + Yt+1 − α′t+1Tt+1 , where αt+1 is our estimate of γ′A from this first step . The dynamically adjusted surrogate index builds this adjustment into equation ( 2 ) above : gadj0 ( St ) = E [ Yt | St ] + E [ Yt+1 − γ′ATt+1 | St ] = γ′St + E [ γ′BSt | St+1 ] = γ′ ( I +B ) St ( 5 ) This new index captures the projected M -period outcomes as if the customer was offered no future treatments . When we estimate the effect of the treatment based on this adjusted surrogate index we recover : E [ gadj0 ( St ) | Tt , St−1 ] = γ′ ( I +B ) E [ St | Tt , St−1 ] = γ′ ( I +B ) ATt + γ ′ ( I +B ) BSt−1 ( 6 ) With the dynamic adjustment , the coefficient in front of Tt that we recover is the true causal effect θ0 = γ ′ ( I +B ) A. Lewis & Syrgkanis ( 2020 ) show how this adjustment approach can be extended to many periods , via a recursive peeling process , and also to high-dimensional surrogates and controls via a dynamic double machine learning approach . Estimating causal effects of new treatments With this adjusted surrogate index in hand , we can also estimate the long-term effect of any other treatment that was introduced more recently . In this example , the stationarity assumption of the proposed surrogate approaches requires that B , which governs how the surrogate evolves , and γ , which governs how surrogates translate to per-period outcomes , do not change between the observational and experimental data-sets . These two parameters govern how surrogates today relate to future outcomes in the absence of any treatment . Under such a condition , if we introduce a new treatment T newt , which has a different effect A new on the surrogates ( and hence on the long-term outcome ) , then the effect of this treatment on the long-term outcome is θnew0 = γ ′ ( I + B ) Anew . This θnew0 is exactly the outcome of estimating the causal effect of T newt on g adj 0 ( St+1 ) controlling for St .

Summary: This paper tackles the problem of estimating the long-term effect of some new treatment in the case where the treatment assignment is decided adaptively. Athey et al.(2020) proposed the long-term effect estimation for the stable treatment assignment case. This paper applied the method proposed in Lewis & Sygkanis(2020) to adjust the approach of Athey et al.(2020) for the case where the treatment assignment is decided adaptively.