Deconfounding Reinforcement Learning in Observational Settings

1 INTRODUCTION . In recent years , reinforcement learning ( RL ) has made great progress , spawning a large number of successful applications especially in terms of games ( Silver et al. , 2016 ; Mnih et al. , 2013 ; OpenAI , 2018 ) . Within this background , much attention has been devoted to the development of RL algorithms with the goal of improving treatment policies in healthcare ( Gottesman et al. , 2018 ) . In fact , various RL algorithms have been proposed to infer better decision-making strategies for mechanical ventilation ( Prasad et al. , 2017 ) , sepsis management ( Raghu et al. , 2017a ; b ) , and treatment of schizophrenia ( Shortreed et al. , 2011 ) . In healthcare , a common practice is to focus on the observational setting , because ones do not wish to experiment with patients ’ lives without evidence that the proposed treatment strategy is better than the current practice ( Gottesman et al. , 2018 ) 1 . As pointed out in ( Raghu et al. , 2017a ) , even if in the observational setting , RL also has advantages over other machine learning algorithms especially in two situations : when the ground truth of a “ good ” treatment strategy is unclear in medical literature ( Marik , 2015 ) , and when training examples do not represent optimal behavior . On the other hand , although causal inference ( Pearl , 2009 ) has been extensively explored and used in healthcare and medicine ( Liu et al . ; Soleimani et al. , 2017 ; Schulam & Saria , 2017 ; Alaa et al. , 2017 ; Alaa & van der Schaar , 2018 ; Atan et al. , 2016 ) , the efficient approach to dealing with time-varying data is still unclear ( Peters et al. , 2017 ; Hernn & Robins , 2018 ) . On the basis of the discussion above , in this paper we attempt to combine advantages on both sides to cope with an important family of RL problems in the observational setting , that is , learning good policies solely from the historical data produced by real environments with confounding bias . This type of problems are inevitable in the future RL research with the burgeoning development in healthcare and medicine . To the best of our knowledge , however , little work has been done in the direction . Confounding is a causal concept that is described in the language of causality instead of probability and statistics ( Pearl , 2009 ) . Confounding bias occurs when a variable influences both who is selected for the treatment and the outcome of the experiment ( Pearl & Mackenzie , 2018 ) , which naturally corresponds to the action and the reward in RL , respectively . As a matter of fact , confounders have been extensively studied in epidemiology , sociology , and economics . Take for example the widespread kidney stones in which the size of the kidney stone is a confounding factor affecting both the treatment and the recovery ( Peters et al. , 2017 ; Pearl , 2009 ) , whether deconfounding the size of the kidney stone or not entirely determines how to choose a more effective treatment . Similarly , in RL , if unobserved potential confounders exist , they would affect both ac- 1Another straightforward example is that , in economics , considering the risky cost in terms of time and money , it is not practical to study the optimal strategy by actually buying and selling stocks in the market . tions and rewards when an agent interacts with environments and eventually influence the policy to be optimized . It is widely acknowledged that one should draw a causal graph before one can achieve any causal conclusion ( Pearl , 2009 ; Pearl & Mackenzie , 2018 ) . Throughout the paper , we assume that , given causal assumptions , we first estimate a model from the observational data we collected from real environments or simulators , and then optimize a policy on the basis of the learned model . As mentioned previously , this assumption is quite useful in real-world RL applications ( e.g. , healthcare , medicine , economics , etc ) , because in most circumstances , except the data we observed , we neither know anything about real environments nor are allowed to do anything in real environments probably for the sake of ethics , laws or cost . In order to adjust for confounders , we present a general formulation for addressing this class of RL problems , namely deconfounding reinforcement learning ( DRL ) . More specifically , given several common confounding assumptions , under some conditions for identification we first estimate a latent-variable model from observational data in which we simultaneously discover the latent confounders and infer how they affect action and reward , then deconfound the confounders using the causal language developed by ( Pearl , 2009 ) , and finally optimize the policy based on the deconfounding model we calculated . On the basis of the proposed formulation , we extend one popular RL algorithm , the Actor-Critic method , to its corresponding deconfounding variant , which is straightforward to be applied to other RL algorithms . Due to lack of datasets in this respect , we revise the classic control toolkit in OpenAI Gym ( Brockman et al. , 2016 ) , making it a benchmark for comparison of DRL algorithms . In addition , we also devise a confounding version of the MNIST dataset ( LeCun et al. , 1998 ) to verify the performance of our causal model . Finally , we conduct extensive experiments to demonstrate the superiority of the proposed formulation in confounded environments , in comparison to traditional RL algorithms . To sum up , our contributions in this paper are as follows : 1 . We propose a general formulation to address a family of RL problems in confounded envi- ronments , namely deconfounding reinforcement learning ( DRL ) ; 2 . We present the deconfounding variant of Actor-Critic methods , which is obvious to be applied to other RL methods ; 3 . We develop a benchmark for DRL by revising the toolkit for classic control in OpenAI Gym ( Brockman et al. , 2016 ) and by devising a confounding version of the MNIST dataset ( LeCun et al. , 1998 ) ; 4 . We perform a comprehensive comparison of DRL algorithms with their vanilla versions , showing that the proposed approach has an advantage in confounded environments . 5 . To the best of our knowledge , this is the first attempt to build a bridge between confounding and the full RL problem . This is one of few research papers aiming at understanding the connections between causal inference and the full RL . 2 BACKGROUND . In this section , we briefly review confounding in causal inference . We recommend Pearls excellent monograph for further reading ( Pearl , 2009 ; Pearl & Mackenzie , 2018 ) . 2.1 SIMPSON ’ S PARADOX . Let us begin with one of the most famous paradoxes in statistics : Simpson ’ s Paradox . Consider the previously mentioned kidney stones , a classic example of Simpson ’ s paradox ( Peters et al. , 2017 ) . We collect electronic patient records to investigate the effectiveness of two treatments against kidney stones , where although the overall probability of recovery is higher for patients who took treatment b , treatment a performs better than treatment b on both patients with small kidney stones and with large kidney stones . More precisely , we have p ( R = 1|T = b ) > p ( R = 1|T = a ) ; but p ( R = 1|T = b , Z = 0 ) < p ( R = 1|T = a , Z = 0 ) , p ( R = 1|T = b , Z = 1 ) < p ( R = 1|T = a , Z = 1 ) ; ( 1 ) where Z is the size of the stone , T the treatment , and R the recovery ( all binary ) . How do we cope with this inversion of conclusion ? Which treatment do you prefer if you had kidney stones ? Does treatment b cause recovery ? The answers to these questions depend on the causal relationship between treatment , recovery , and the size of the kidney stone . 2.2 CONFOUNDING . An intuitive explanation for this kidney stone example of Simpson ’ s paradox is that larger stones are more severe than small stones and are much more likely to be treated with treatment a , resulting in that treatment a looks worse than treatment b . Therefore , it is straightforward to assume that the true underlying causal diagram of the kidney stone example is shown in Figure 1 ( a ) , where confounding occurs because the size of kidney stones influences both treatment and recovery . Here , the size of kidney stones is called confounder . The term “ confounding ” originally meant “ mixing ” in English , which describes that the true causal effect T → R is “ mixed ” with the spurious correlation between T and R induced by the fork T ← Z → R ( Pearl & Mackenzie , 2018 ) . In other words , we will not be able to disentangle the true effect of T on R from the spurious effect if we do not have data on Z. Conversely , if we have measurements of Z , it is easy to deconfound the true and spurious effects by adjusting for Z that averages the effect of T on R in each subgroup of Z ( i.e. , different size groups in the case of kidney stones ) . 2.3 DO-OPERATOR AND ADJUSTMENT CRITERION . From the viewpoint of causal inference , we can also use the language of intervention , namely dooperator , to formulate confounding . In fact , in the example of kidney stones , what we are interested in is how these two treatments compare when we force all patients to take treatment a or b , rather than which treatment has a higher recovery rate given only the observational patient records . Mathematically , we focus on the true effect p ( R = 1|do ( T = a ) ) ( i.e. , intervention distribution where patients are forced to take treatment a ) instead of the spurious effect p ( R = 1|T = a ) ( i.e. , observational distribution where patients are observed to take treatment a ) . Therefore , as described previously , confounding can be naturally formulated by the discrepancy between p ( R|T ) and p ( R|do ( T ) ) . Generally speaking , do-operator can be executed in two common ways : by Randomized Controlled Trials ( RCTs ) ( Fisher , 1935 ) and by adjustment formulas ( i.e. , Back-door criterion and Front-door criterion ) ( Pearl , 2009 ) . RCTs is the so-called golden standard but rather limited due to many impractical factors ( e.g. , safety , laws , ethics , physically infeasibility , etc. ) . Back-door and Front-door criterions require a known causal diagram in which causal assumptions are provided in advance . According to the Back-door criterion , in the kidney stone example , we can immediately attain p ( R = 1|do ( T = a ) ) = ∑1 z=0 p ( R = 1|T = a , Z = z ) p ( Z = z ) . ( 2 ) In fact , apart from the two adjustment formulas , we are provided with a syntactic method of deriving claims about interventions , namely do-calculus , developed by Pearl ( Pearl , 2009 ) . The do-calculus consists of three rules which can be repeatedly applied to simplify the expression for an interventional distribution . It offers a powerful tool to convert intervention probabilities to observation probabilities so that we can estimate the former from observational data alone .

Summary: The paper addresses an important and often overlooked issue in off-policy reinforcement learning - the possibility of confounding between the agent's actions and the rewards. This is a subject which has been exhaustively explored in the causal inference literature, and the authors are very correct in suggesting that it should be incorporated into the world of reinforcement learning.  Specifically they propose a generative model with a global latent confounder that is inferred using a variational autoencoder architecture.