Estimating counterfactual treatment outcomes over time through adversarially balanced representations

1 INTRODUCTION . As clinical decision-makers are often faced with the problem of choosing between treatment alternatives for patients , reliably estimating their effects is paramount . While clinical trials represent the gold standard for causal inference , they are expensive , have a few patients and narrow inclusion criteria ( Booth & Tannock , 2014 ) . Leveraging the increasingly available observational data about patients , such as electronic health records , represents a more viable alternative for estimating treatment effects . A large number of methods have been proposed for performing causal inference using observational data in the static setting ( Johansson et al. , 2016 ; Shalit et al. , 2017 ; Alaa & van der Schaar , 2017 ; Li & Fu , 2017 ; Yoon et al. , 2018 ; Alaa & van der Schaar , 2018 ; Yao et al. , 2018 ) and only a few methods address the longitudinal setting ( Xu et al. , 2016 ; Roy et al. , 2016 ; Soleimani et al. , 2017 ; Schulam & Saria , 2017 ; Lim et al. , 2018 ) . However , estimating the effects of treatments over time poses unique opportunities such as understanding how diseases evolve under different treatment plans , how individual patients respond to medication over time , but also which are optimal timings for assigning treatments , thus providing new tools to improve clinical decision support systems . The biggest challenge when estimating the effects of time-dependent treatments from observational data involves correctly handling the time-dependent confounders : patient covariates that are affected by past treatments which then influence future treatments and outcomes ( Platt et al. , 2009 ) . For instance , consider that treatment A is given when a certain patient covariate ( e.g . white blood cell count ) has been outside of normal range values for several consecutive timesteps . Suppose also that this patient covariate was itself affected by the past administration of treatment B . If these patients are more likely to die , without adjusting for the time-dependent confounding ( e.g . the changes in the white blood cell count over time ) , we will incorrectly conclude that treatment A is harmful to patients . Moreover , estimating the effect of a different sequence of treatments on the patient outcome would require not only adjusting for the bias at the current step ( in treatment A ) , but also for the bias introduced by the previous application of treatment B . Existing methods for causal inference in the static setting can not be applied in this longitudinal setting since they are designed to handle the cross-sectional set-up , where the treatment and outcome depend only on a static value of the patient covariates . If we consider again the above example , these methods would not be able to model how the changes in patient covariates over time affect the assignment of treatments and they would also not be able to estimate the effect of a sequence of treatments on the patient outcome ( e.g . sequential application of treatment A followed by treatment B ) . Different models that can handle these temporal dependencies in the observational data and varying-length patient histories are needed for estimating treatment effects over time . Time-dependent confounders are present in observational data because doctors follow policies : the history of the patients ’ covariates and the patients ’ response to past treatments are used to decide future treatments ( Mansournia et al. , 2012 ) . The direct use of supervised learning methods will be biased by the treatment policies present in the observational data and will not be able to correctly estimate counterfactuals for different treatment assignment policies . Standard methods for adjusting for time-varying confounding and estimating the effects of timevarying exposures are based on ideas from epidemiology . The most widely used among these are Marginal Structural Models ( MSMs ) ( Robins et al. , 2000 ; Mansournia et al. , 2012 ) which use the inverse probability of treatment weighting ( IPTW ) to adjust for the time-dependent confounding bias . Through IPTW , MSMs create a pseudo-population where the probability of treatment does not depend on the time-varying confounders . However , MSMs are not robust to model misspecification in computing the IPTWs . MSMs can also give high-variance estimates due to extreme weights ; computing the IPTW involves dividing by probability of assigning a treatment conditional on patient history which can be numerically unstable if the probability is small . We introduce the Counterfactual Recurrent Network ( CRN ) , a novel sequence-to-sequence architecture for estimating treatment effects over time . CRN leverages recent advances in representation learning ( Bengio et al. , 2012 ) and domain adversarial training ( Ganin et al. , 2016 ) to overcome the problems of existing methods for causal inference over time . Our main contributions are as follows . Treatment invariant representations over time . CRN constructs treatment invariant representations at each timestep in order to break the association between patient history and treatment assignment and thus removes the bias from time-dependent confounders . For this , CRN uses domain adversarial training ( Ganin et al. , 2016 ; Li et al. , 2018 ; Sebag et al. , 2019 ) to trade-off between build- ing this balancing representation and predicting patient outcomes . We show that these representations remove the bias from time-varying confounders and can be reliably used for estimating counterfactual outcomes . This represents the first work that introduces ideas from domain adaptation to the area of estimating treatment effects over time . In addition , by building balancing representations , we propose a novel way of removing the bias introduced by time-varying confounders . Counterfactual estimation of future outcomes . To estimate counterfactual outcomes for treatment plans ( and not just single treatments ) , we integrate the domain adversarial training procedure as part of a sequence-to-sequence architecture . CRN consists of an encoder network which builds treatment invariant representations of the patient history that are used to initialize the decoder . The decoder network estimates outcomes under an intended sequence of future treatments , while also updating the balanced representation . By performing counterfactual estimation of future treatment outcomes , CRN can be used to answer critical medical questions such as deciding when to give treatments to patients , when to start and stop treatment regimes , and also how to select from multiple treatments over time . We illustrate in Figure 1 the applicability of our method in choosing optimal cancer treatments . In our experiments , we evaluate CRN in a realistic set-up using a model of tumour growth ( Geng et al. , 2017 ) . We show that CRN achieves better performance in predicting counterfactual outcomes , but also in choosing the right treatment and timing of treatment than current state-of-the-art methods . 2 RELATED WORK . We focus on methods for estimating treatment effects over time and for building balancing representations for causal inference . A more in-depth review of related work is in Appendix A . Treatment effects over time . Standard methods for estimating the effects of time-varying exposures were first developed in the epidemiology literature and include the g-computation formula , Structural Nested Models and Marginal Structural Models ( MSMs ) ( Robins , 1986 ; 1994 ; Robins et al. , 2000 ; Robins & Hernán , 2008 ) . Originally , these methods have used predictors performing logistic/linear regression which makes them unsuitable for handling complex time-dependencies ( Hernán et al. , 2001 ; Mansournia et al. , 2012 ; Mortimer et al. , 2005 ) . To address these limitations , methods that use Bayesian non-parametrics or recurrent neural networks as part of these frameworks have been proposed . ( Xu et al. , 2016 ; Roy et al. , 2016 ; Lim et al. , 2018 ) . To begin with , Xu et al . ( 2016 ) use Gaussian processes to model discrete patient outcomes as a generalized mixed-effects model and uses the g-computation method to handle time-varying confounders . Soleimani et al . ( 2017 ) extend the approach in Xu et al . ( 2016 ) to the continuous time-setting and model treatment responses using linear time-invariant dynamical systems . Roy et al . ( 2016 ) use Dirichlet and Gaussian processes to model the observational data and estimate the IPTW in Marginal Structural Models . Schulam & Saria ( 2017 ) build upon work from Lok et al . ( 2008 ) ; Arjas & Parner ( 2004 ) and use marked point processes and Gaussian processes to learn causal effects in continuous-time data . These Bayesian non-parametric methods make strong assumptions about model structure and consequently can not handle well heterogeneous treatment effects arising from baseline variables ( Soleimani et al. , 2017 ; Schulam & Saria , 2017 ) and multiple treatment outcomes ( Xu et al. , 2016 ; Schulam & Saria , 2017 ) . The work most related to ours is the one of Lim et al . ( 2018 ) which improves on the standard MSMs by using recurrent neural networks to estimate the inverse probability of treatment weights ( IPTWs ) . Lim et al . ( 2018 ) introduces Recurrent Marginal Structural Networks ( RMSNs ) which also use a sequence-to-sequence deep learning architecture to forecast treatment responses in a similar fashion to our model . However , RMSNs require training additional RNNs to estimate the propensity weights and does not overcome the fundamental problems with IPTWs , such as the high-variance of the weights . Conversely , CRN takes advantage of the recent advances in machine learning , in particular , representation learning to propose a novel way of handling time-varying confounders . Balancing representations for treatment effect estimation . Balancing the distribution of control and treated groups has been used for counterfactual estimation in the static setting . The methods proposed in the static setting for balancing representations are based on using discrepancy measures in the representation space between treated and untreated patients , which do not generalize to multiple treatments ( Johansson et al. , 2016 ; Shalit et al. , 2017 ; Li & Fu , 2017 ; Yao et al. , 2018 ) . Moreover , due to the sequential assignment of treatments in the longitudinal setting , and due to the change of patient covariates over time according to previous treatments , the methods for the static setting are not directly applicable to the time-varying setting ( Hernán et al. , 2000 ; Mansournia et al. , 2012 ) . 3 PROBLEM FORMULATION . Consider an observational datasetD = { { x ( i ) t , a ( i ) t , y ( i ) t+1 } T ( i ) t=1 ∪ { v ( i ) } } N i=1 consisting of information about N independent patients . For each patient ( i ) , we observe time-dependent covariates X ( i ) t ∈ Xt , treatment received A ( i ) t ∈ { A1 , . . . AK } = A and outcomes Y ( i ) t+1 ∈ Yt+1 for T ( i ) discrete timesteps . The patient can also have baseline covariates V ( i ) ∈ V such as gender and genetic information . Note that the outcome Y ( i ) t+1 will be part of the observed covariates X ( i ) t+1 . For simplicity , the patient superscript ( i ) will be omitted unless explicitly needed . We adopt the potential outcomes framework proposed by ( Neyman , 1923 ; Rubin , 1978 ) and extended by ( Robins & Hernán , 2008 ) to account for time-varying treatments . Let Y [ ā ] be the potential outcomes , either factual or counterfactual , for each possible course of treatment ā . Let H̄t = ( X̄t , Āt−1 , V ) represent the history of the patient covariates X̄t = ( X1 , . . . , Xt ) , treatment assignments Āt = ( A1 , . . . , At ) and static features V. We want to estimate : E ( Yt+τ [ ā ( t , t+ τ − 1 ) ] |H̄t ) , ( 1 ) where ā ( t , t+ τ − 1 ) = [ at , . . .at+τ−1 ] represents a possible sequence of treatments from timestep t just until before the potential outcome Yt+τ is observed . We make the standard assumptions ( Robins et al. , 2000 ; Lim et al. , 2018 ) needed to identify the treatment effects : consistency , positivity and no hidden confounders ( sequential strong ignorability ) . See Appendix B for more more details .

Summary: The paper introduces Counterfactual Recurrent Network (CRN) that is able to estimate the effects of various treatments from longitudinal data. The claim is that the model can decide (i) treatment plan; (ii) optimal time of treatment; and (iii) when to stop treatment. The proposed method attempts to learn time-invariant representations that are not predictive of the next treatment by borrowing ideas from Ganin, et al. (2016)’s work on for domain adversarial training. In fact, this paper is an extension of (Atan et al., 2018) to be applicable for longitudinal data.