Title: DECONFOUNDING REINFORCEMENT LEARNING IN OBSERVATIONAL SETTINGS

Abstract: In this paper, we propose a general formulation to cope with a family of reinforcement learning tasks in observational settings, that is, learning good policies solely from the historical data produced by real environments with confounders (i.e., the factors affecting both actions and rewards). Based on the proposed approach, we extend one representative of reinforcement learning algorithms: the Actor-Critic method, to its deconfounding variant, which is also straightforward to be applied to other algorithms. In addition, due to lack of datasets in this direction, a benchmark is developed for deconfounding reinforcement learning algorithms by revising OpenAI Gym and MNIST. We demonstrate that the proposed algorithms are superior to traditional reinforcement learning algorithms in confounded environments. To the best of our knowledge, this is the first time that confounders are taken into consideration for addressing full reinforcement learning problems.

1. INTRODUCTION:

In recent years, reinforcement learning (RL) has made great progress, spawning a large number of successful applications especially in terms of games (Silver et al., 2016; Mnih et al., 2013; Ope-nAI, 2018) . Within this background, much attention has been devoted to the development of RL algorithms with the goal of improving treatment policies in healthcare (Gottesman et al., 2018) . In fact, various RL algorithms have been proposed to infer better decision-making strategies for mechanical ventilation (Prasad et al., 2017) , sepsis management (Raghu et al., 2017a; b) , and treatment of schizophrenia (Shortreed et al., 2011) . In healthcare, a common practice is to focus on the observational setting, because ones do not wish to experiment with patients' lives without evidence that the proposed treatment strategy is better than the current practice (Gottesman et al., 2018) 1 . As pointed out in (Raghu et al., 2017a) , even if in the observational setting, RL also has advantages over other machine learning algorithms especially in two situations: when the ground truth of a "good" treatment strategy is unclear in medical literature (Marik, 2015) , and when training examples do not represent optimal behavior. On the other hand, although causal inference (Pearl, 2009) has been extensively explored and used in healthcare and medicine (Liu et al.; Soleimani et al., 2017; Schulam & Saria, 2017; Alaa et al., 2017; Alaa & van der Schaar, 2018; Atan et al., 2016) , the efficient approach to dealing with time-varying data is still unclear (Peters et al., 2017; Hernn & Robins, 2018) . On the basis of the discussion above, in this paper we attempt to combine advantages on both sides to cope with an important family of RL problems in the observational setting, that is, learning good policies solely from the historical data produced by real environments with confounding bias.

This type of problems are inevitable in the future RL research with the burgeoning development in healthcare and medicine. To the best of our knowledge, however, little work has been done in the direction. Confounding is a causal concept that is described in the language of causality instead of probability and statistics (Pearl, 2009) . Confounding bias occurs when a variable influences both who is selected for the treatment and the outcome of the experiment (Pearl & Mackenzie, 2018) , which naturally corresponds to the action and the reward in RL, respectively. As a matter of fact, confounders have been extensively studied in epidemiology, sociology, and economics. Take for example the widespread kidney stones in which the size of the kidney stone is a confounding factor affecting both the treatment and the recovery (Peters et al., 2017; Pearl, 2009) , whether deconfounding the size of the kidney stone or not entirely determines how to choose a more effective treatment. Similarly, in RL, if unobserved potential confounders exist, they would affect both ac-tions and rewards when an agent interacts with environments and eventually influence the policy to be optimized.

It is widely acknowledged that one should draw a causal graph before one can achieve any causal conclusion (Pearl, 2009; Pearl & Mackenzie, 2018) . Throughout the paper, we assume that, given causal assumptions, we first estimate a model from the observational data we collected from real environments or simulators, and then optimize a policy on the basis of the learned model. As mentioned previously, this assumption is quite useful in real-world RL applications (e.g., healthcare, medicine, economics, etc), because in most circumstances, except the data we observed, we neither know anything about real environments nor are allowed to do anything in real environments probably for the sake of ethics, laws or cost.

In order to adjust for confounders, we present a general formulation for addressing this class of RL problems, namely deconfounding reinforcement learning (DRL). More specifically, given several common confounding assumptions, under some conditions for identification we first estimate a latent-variable model from observational data in which we simultaneously discover the latent confounders and infer how they affect action and reward, then deconfound the confounders using the causal language developed by (Pearl, 2009) , and finally optimize the policy based on the deconfounding model we calculated. On the basis of the proposed formulation, we extend one popular RL algorithm, the Actor-Critic method, to its corresponding deconfounding variant, which is straightforward to be applied to other RL algorithms. Due to lack of datasets in this respect, we revise the classic control toolkit in OpenAI Gym (Brockman et al., 2016) , making it a benchmark for comparison of DRL algorithms. In addition, we also devise a confounding version of the MNIST dataset (LeCun et al., 1998) to verify the performance of our causal model. Finally, we conduct extensive experiments to demonstrate the superiority of the proposed formulation in confounded environments, in comparison to traditional RL algorithms.

To sum up, our contributions in this paper are as follows:

1. We propose a general formulation to address a family of RL problems in confounded environments, namely deconfounding reinforcement learning (DRL); 2. We present the deconfounding variant of Actor-Critic methods, which is obvious to be applied to other RL methods; 3. We develop a benchmark for DRL by revising the toolkit for classic control in OpenAI Gym (Brockman et al., 2016) and by devising a confounding version of the MNIST dataset (LeCun et al., 1998) ; 4. We perform a comprehensive comparison of DRL algorithms with their vanilla versions,

showing that the proposed approach has an advantage in confounded environments. 5. To the best of our knowledge, this is the first attempt to build a bridge between confounding and the full RL problem. This is one of few research papers aiming at understanding the connections between causal inference and the full RL.

2. BACKGROUND:

In this section, we briefly review confounding in causal inference. We recommend Pearls excellent monograph for further reading (Pearl, 2009; Pearl & Mackenzie, 2018) .

2.1. SIMPSON'S PARADOX:

Let us begin with one of the most famous paradoxes in statistics: Simpson's Paradox. Consider the previously mentioned kidney stones, a classic example of Simpson's paradox (Peters et al., 2017) .

We collect electronic patient records to investigate the effectiveness of two treatments against kidney stones, where although the overall probability of recovery is higher for patients who took treatment b, treatment a performs better than treatment b on both patients with small kidney stones and with large kidney stones. More precisely, we have

EQUATION

)

where Z is the size of the stone, T the treatment, and R the recovery (all binary). How do we cope with this inversion of conclusion? Which treatment do you prefer if you had kidney stones? Does treatment b cause recovery? The answers to these questions depend on the causal relationship between treatment, recovery, and the size of the kidney stone.

2.2 CONFOUNDING An intuitive explanation for this kidney stone example of Simpson's paradox is that larger stones are more severe than small stones and are much more likely to be treated with treatment a, resulting in that treatment a looks worse than treatment b. Therefore, it is straightforward to assume that the true underlying causal diagram of the kidney stone example is shown in Figure 1 (a), where confounding occurs because the size of kidney stones influences both treatment and recovery. Here, the size of kidney stones is called confounder. The term "confounding" originally meant "mixing" in English, which describes that the true causal effect T → R is "mixed" with the spurious correlation between T and R induced by the fork T ← Z → R (Pearl & Mackenzie, 2018) . In other words, we will not be able to disentangle the true effect of T on R from the spurious effect if we do not have data on Z.

Conversely, if we have measurements of Z, it is easy to deconfound the true and spurious effects by adjusting for Z that averages the effect of T on R in each subgroup of Z (i.e., different size groups in the case of kidney stones).

2.3. DO-OPERATOR AND ADJUSTMENT CRITERION:

From the viewpoint of causal inference, we can also use the language of intervention, namely dooperator, to formulate confounding. In fact, in the example of kidney stones, what we are interested in is how these two treatments compare when we force all patients to take treatment a or b, rather than which treatment has a higher recovery rate given only the observational patient records. Mathematically, we focus on the true effect p(R = 1|do(T = a)) (i.e., intervention distribution where patients are forced to take treatment a) instead of the spurious effect p(R = 1|T = a) (i.e., observational distribution where patients are observed to take treatment a). Therefore, as described previously, confounding can be naturally formulated by the discrepancy between p(R|T ) and p(R|do(T )).

Generally speaking, do-operator can be executed in two common ways: by Randomized Controlled Trials (RCTs) (Fisher, 1935) and by adjustment formulas (i.e., Back-door criterion and Front-door criterion) (Pearl, 2009) . RCTs is the so-called golden standard but rather limited due to many impractical factors (e.g., safety, laws, ethics, physically infeasibility, etc.). Back-door and Front-door criterions require a known causal diagram in which causal assumptions are provided in advance.

According to the Back-door criterion, in the kidney stone example, we can immediately attain

p(R = 1|do(T = a)) = ∑ 1 z=0 p(R = 1|T = a, Z = z)p(Z = z).

(2)

In fact, apart from the two adjustment formulas, we are provided with a syntactic method of deriving claims about interventions, namely do-calculus, developed by Pearl (Pearl, 2009) . The do-calculus consists of three rules which can be repeatedly applied to simplify the expression for an interventional distribution. It offers a powerful tool to convert intervention probabilities to observation probabilities so that we can estimate the former from observational data alone.

2.4. PROXY VARIABLES FOR CONFOUNDING:

If confounders can be measured, then they can be adjusted for through the methods we discussed in Section 2.3. However, in most cases where confounders are hidden or unmeasured, without further assumptions, it is impossible to estimate the effect of the intervention on the outcome. A common practice is to leverage observed proxy variables of the confounders (Angrist & Pischke, 2008; Maddala & Lahiri, 1992; Montgomery et al., 2000) , which is one of the promising directions of exploiting big-data to conduct causal inference in the presence of multiple proxy variables for unmeasured confounders (Louizos et al., 2017) . However, using proxy variables to correctly recover causal effects should meet strict mathematical assumptions (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Wooldridge, 2009) . Unfortunately, in practice, we do not know whether or not the complicated data from the real world meet those assumptions. Hence, following the same spirit from (Louizos et al., 2017) , with multiple proxy variables available, under relaxed assumptions we estimate a latent-variable model in which we simultaneously discover the latent confounders and infer how they affect treatment and outcome.

3. DECONFOUNDING REINFORCEMENT LEARNING:

In this section, we will formally introduce deconfounding reinforcement learning (DRL). Generally speaking, DRL consists of two steps: learning a deconfounding model shown in Figure 1 (b) and optimizing a policy based on the learned deconfounding model. The main idea in step 1 is to simultaneously discover the hidden confounder and infer the causal effect through an estimation of a latent-variable model. More specifically, we first discuss the time-independent confounding assumption in Section 3.1, and then, based on the assumption, formalize the deconfounding model in Section 3.2. Section 3.3 talks about the problem of identification in our model, which is a central issue in causal inference. After that, we present the details about how to learn the model via variational inference in Section 3.4.

Step 2 is provided in Section 3.5 where we describe how to design the deconfounding actor-critic method. It is straightforward to apply this to other RL algorithms in the same manner.

3.1 CAUSAL ASSUMPTIONS Without loss of generality, as shown in Figure 1 (b)2 , we assume there exists a common confounder in the sequential model, which is time-independent for each individual or for each procedure. This assumption is so general that it would apply to various RL tasks across domains. For example, in personalized medicine or precision medicine, socio-economic status can affect both the medication strategy a patient has access to, and the patients general health (Louizos et al., 2017) . Therefore socio-economic status acts as confounder between the medication and health outcomes, in which case socio-economic status is time-independent for each patient during the course of treatment. In agriculture, soil fertility may serve as one of confounders affecting both the application of fertilizer and the yield of each plot (Pearl & Mackenzie, 2018) . In this circumstance, soil fertility is stable and thought of as a time-independent factor within a period of time (e.g., several months, the growth circle of crops, etc.). In the example of stock markets, apart from socio-economic status as mentioned above, government policy may also act as one of confounders, all of which can be seen time-independent during a reasonable period of time.

3.2. THE MODEL:

Given the causal assumption, we first fit a generative model to a sequence of observational data: observations, actions, and rewards, where actions and rewards are confounded by one or several unknown factors. Formally, Let ⃗ x = (x 1 , . . . , x T ), ⃗ a = (a 1 , . . . , a T -1 ), ⃗ r = (r 2 , . . . , r T +1 ), ⃗ z = (z 1 , . . . , z T ) be the sequence of observations, actions, rewards, and corresponding latent states, respectively. The confounder is denoted by u, and it is worth noting that here u may stand for more than one confounder in which multiple confounders are seen as a whole represented by u. We assume that

x t ∈ R Dx , a t ∈ R Da , r t ∈ R Dr , z t ∈ R Dz , and u ∈ R Du , where D z ≪ D x .

The generative model for DRL is then given by:

EQUATION

Note that we parametrize each probability distribution as a Gaussian with its mean and variance modeled by nonlinear functions f k and each f k is parametrized by a neural network with its own parameters θ k for k = 1, . . . , 8. Note that Equation ( 4) is not necessary in our model, but it is potentially useful acting as a prior policy because it is learned from the observational data containing, for example, the real treatment strategies by doctors.

3.3. IDENTIFICATION OF CAUSAL EFFECT:

In our deconfounding model shown in Figure 1 (b), the key difference from traditional RL lies in the reward function3 . To be more precise, assuming that an agent standing at state z t performs an action a t = a, unlike the traditional reward p(r t |z t , a t = a), our deconfounding version based on do-operator as depicted in Section 2.3 is given by

EQUATION

EQUATION

)

where Equation 8is by the rules of do-calculus applied to the causal graph in Figure 1 (b) (Pearl, 2009) . In fact, we can utilize the back-door criterion to directly obtain Equation ( 8). Furthermore, through Equation ( 8) we proved that p(r t |z t , do(a t = a)) can be identified from the joint distribution p(u, ⃗ z, ⃗ x, ⃗ a, ⃗ r). Then, motivated by the idea that one is supposed to use the knowledge inferred from the joint distribution between proxy variables and confounders to adjust for the hidden confounders (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Wooldridge, 2009) , in this paper we also assume that the joint distribution p(u, ⃗ z, ⃗ x, ⃗ a, ⃗ r) can be approximately recovered solely from the observations (⃗ x, ⃗ a, ⃗ r). The details about identification can be found in Appendix A.

3.4. LEARNING:

Since the nonlinear functions parametrized by neural networks make inference intractable, we will learn the parameters of the model θ k by employing variational inference along with an inference model, a neural network which approximates the intractable posterior (Rezende et al., 2014; Kingma & Welling, 2013; Krishnan et al., 2015) . More specifically, using the variational principle, we posit an approximate posterior distribution q ϕ (z|x) to obtain the following lower bound on the marginal likelihood:

EQUATION

)

where the inequality is by Jensen's inequality and ϕ is the parameter of the inference model q(z|x).

Note that in this general case x stands for observational variables and z for latent variables.

3.4.1. VARIATIONAL LOWER BOUND:

Directly applying the lower bound in Inequality (9) to our model, we obtain

EQUATION

)

Using the Markov property of our model, the full distribution can be factorized in the following way:

p θ (⃗ x, ⃗ a, ⃗ r, ⃗ z, u) = p(u)p(z 1 ) [ ∏ T t=1 p(x t |z t , u)p(a t |z t , u)p(r t+1 |z t , a t , u) ] [ ∏ T t=2 p(z t |z t-1 , a t-1 )

] .

(11)

In addition, for simplicity's sake, we also have the factorization assumption for the posterior approximation:

EQUATION

)

Combining Equation ( 10), ( 11) and ( 12) yields:

EQUATION

)

where we omit subscripts θ and ϕ, and a more detailed derivation can be found in Appendix B.

Obviously, Equation ( 13) is differentiable with respect to the parameters of the model θ and ϕ.

Using the reparametrization trick (Kingma & Welling, 2013), we can directly apply backpropagation to update the parameters.

3.4.2. INFERENCE MODEL:

From the factorization form in Equation ( 12), we can see that there are two types of inference models: q(u|⃗ x, ⃗ a, ⃗ r) and q(⃗ z|⃗ x, ⃗ a, ⃗ r). Similar to the generative model in Section 3.2, we also parametrize both of them as Gaussian:

EQUATION

)

In fact, as shown in Equation ( 12), q(⃗ z|⃗ x, ⃗ a, ⃗ r) can be further factorized as the product of q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) for t = 1, . . . , T . Taking a closer look at this term, based on the Markov property of our model, we have

z t |= x 1 , . . . , x t-1 , a 1 , . . . , a t-2 , r 2 , . . . , r t |z t-1

, and then the term can be simplified as follows,

EQUATION

Equation ( 16) tells us that z t depends on z t-1 and all the current and future observed data (⃗ x, ⃗ a, ⃗ r). Meanwhile, the conditional independence above means that z t-1 contains all the historical data. Therefore, it is natural to calculate z t based on the whole sequence of data, which is exactly what recurrent neural networks (RNNs) do. Inspired by (Krishnan et al., 2015; 2017) , we similarly choose a bi-directional LSTM (Zaremba & Sutskever, 2014) to parameterize f 11 and f 12 in Equation ( 15). Considering Equation ( 14) has the same structure as Equation ( 15), f 9 and f 10 are parameterized by a bi-directional LSTM as well. More details about the architecture can be found in Appendix D.

Note that for the task of sample predictions, that is, at any time step t, given a new x t , we require to know a t and r t+1 before inferring the distribution over z t . Hence, we need to introduce two auxiliary distributions, denoted by red and blue dashed lines in Figure 1 (b), to help conduct counterfactual reasoning (i.e., sample prediction on unseen x t ). To be more precise, we have

EQUATION

where f 13 , f 14 , f 15 , and f 16 are also parameterized by neural networks. To estimate the parameters, we will add these two extra terms in the variational lower bound (Equation ( 13)):

EQUATION

)

3.5 DECONFOUNDING RL ALGORITHMS Now we have all the building blocks for DRL algorithms. Once our model is learned from the observational data, it can be directly used as a dynamic environment like those in OpenAI Gym (Brockman et al., 2016) . We can exploit the learned model to generate rollouts for policy optimization. In practice, Equation ( 8) is approximated using the Monte Carlo method as follows:

EQUATION

)

where N is the number of samples from the prior p(u). In the presence of observational data, a better estimate could be given on the basis of the samples drawn from the approximate posterior q(u|⃗ x, ⃗ a, ⃗ r) which we compute through the inference network presented in Section 3.4.2.

On the basis of our deconfounding reward function, it is straightforward to extend traditional RL algorithms to their corresponding deconfounding version. In this paper, we select one representative of them: the Actor-Critic method (Sutton et al., 1998) , but it is straightforward to be applied to other algorithms.

Deconfounding Actor-Critic Methods The actor-critic method is a policy-based method directly parameterizing the policy π(a|z; θ), which aims to reduce the variance of the estimate of the policy gradient by subtracting a learned function of the state b(z), known as a baseline, from the return.

The learned value function V (z; ϕ) is commonly used as the baseline. Taking into consideration that the return is the estimate of Q(z, a; ϕ Q ) and b(z) is the estimate of V (z; ϕ V ), the gradient of the actor-critic loss function at step time t is given by

EQUATION

)

where 

Q(z t , a t ; ϕ Q )-V (z t ; ϕ V ) used

4. EXPERIMENTAL RESULTS:

It is widely acknowledged that evaluating approaches dealing with confounding is always challenging due to lack of groundtruth and benchmark datasets. Besides, little work has been done before in DRL, which renders evaluating such algorithms in this respect much harder. Therefore, we first develop benchmark datasets for evaluation of our algorithms, primarily by revising the MNIST dataset (LeCun et al., 1998) and two environments in OpenAI Gym (Brockman et al., 2016) : CartPole and Pendulum. We then evaluate our model as well as compare two proposed deconfounding algorithms to their corresponding vanilla versions on the benchmark datasets we developed.

4.1. IMPLEMENTATION DETAILS:

We used Tensorflow (Abadi et al., 2016) for the implementation of our model and DRL algorithms.

Optimization was done with Adam (Kingma & Ba, 2014). Unless stated otherwise, the setting of all the hyperparameters and architectures of the neural networks we adopted in this paper can be found in Appendix J.

To verify how good the learned model is, we performed two types of tasks: reconstruction and counterfactual reasoning. The reconstructions were performed by feeding the input sequence into the learned inference network, and then sampling from the resulting posterior distribution according to Equation (15), and finally feeding those samples into the generative network described in Equation (3). The counterfactual reasoning, that is, predicting x t+1 given x t we have not seen in the training set and a model trained in the training set, were executed through four steps: 1) Given an x t unseen in the training set, we estimate a t and r t+1 based on Equation ( 17) and ( 18); 2) Once we have x t , a t , and r t+1 , it is easy to estimate z t from Equation (15); 3) Using the estimated z t and a t , we can directly compute z t+1 from Equation ( 6); (4) The final step is to reconstruct x t+1 from z t+1 and u according to Equation (3). Repeating the four steps, we can counterfactually reason out a sequence of data.

To evaluate the confounder u, there are also two scenarios. The easy one is that, given a sequence of observational data (⃗ x, ⃗ a, ⃗ r), it is obvious to estimate u from Equation ( 14). The more challenging one is to calculate u given only x t at any time step. Following the same steps used in the task of counterfactual reasoning, we first compute a t and r t+1 based on Equation ( 17) and ( 18), and then estimate u through Equation ( 14).

4.2 CONFOUNDING DATASETS Due to the space limit, we only introduce the Confounding Pendulum dataset in this section. Confounding CartPole and Confounding MNIST are devised in the same way and can be found in Appendix H.1 and Appendix H.2, respectively.

Confounding Pendulum Motivated by (Krishnan et al., 2015) where the authors synthesized a dataset mimicking the healthcare data under harsh conditions (e.g., noisy laboratory measurements, surgeries and drugs affected by patient age and gender, etc.), as well as considering our focus on RL tasks, we revised the original Pendulum in OpenAI Gym (Brockman et al., 2016) to develop a confounding Pendulum dataset in which a binary confounder is introduced. More specifically, we select 100 different screen images of Pendulum to create a synthetic dataset where actions are joint effort with the value of between -2 and 24 . First, 20% bit-flip noise are added to each image , and then, based on a random policy but confounded by a binary factor (will be described soon), actions are performed on each noisy image for five time steps, which produces a large number of 5-step sequences of noisy images. To each generated sequence, exactly one sequence of three consecutive squares (2 × 2 in pixel) is superimposed with the top-left corner of the images in a random starting location. The squares within the sequences are intended to be analogous to seasonal flu or other ailments that a patient could exhibit that are independent of the actions and which last several time steps (Krishnan et al., 2015) . We aim to show that our model can learn long-range patterns, which plays an important role in medical applications. We treat such generated sequences of images as the observations ⃗ x. The training/validation/test set respectively comprises 140000/28000/28000 sequences of length five.

Now we are arriving at the key stage: how to define a reasonable relationship between the confounder, action, and reward. For simplicity of notation, we denote the confounder by u, action by a, and reward by r. u is a binary variable mimicking the socio-economic status (i.e., the rich and the poor) . The range of actions a ∈ [-2, 2] is grouped into two categories T 1 (i.e., 1 ≤ |T 1 | ≤ 2) and T 2 (i.e., 0 ≤ |T 2 | ≤ 1) representing different treatments (i.e., T 1 is more valid and T 2 ). How to choose the treatment depends on u. The reward is defined as follows,

EQUATION

where r o is the original reward in Pendulum, which is a function of a5 , and r c is the extra reward caused by both u and a, where r c comes from a two Gaussian mixture

EQUATION

with σ and µ rc fixed and π rc determined by both u and a (see more in Appendix H.3). Obviously, in the definition above, r is a function of a and u. It is worth noting that u has an influence on x t through a t-1 and z t , meaning that x t contains some piece of information about u and therefore can be viewed as its proxy variable. In our case, for the sake of generality and practice, we assume the influence between the confounder, action and reward is stochastic. Take the case of kidney stones for example, even though treatment a is more valid than treatment b, there are still some patients choosing treatment b in each category, but with different probabilities. All the details can be found in Appendix H.3, where a straightforward analogy is provided as well.

4.3. PERFORMANCE ANALYSIS OF THE DECONFOUNDING MODEL:

In this section, to demonstrate the validity of our deconfounding model, denoted by M decon , we compare with the original model (i.e., the model similar to that shown in Figure 1 (b) but without the confounder u), denoted by M orin . We train M decon by optimizing Equation ( 19) but train M orin by a little different loss function excluding the confounder u whose full derivation can be found in Appendix C. Both models are separately trained in a batch manner on the training set (i.e., 140K sequences of length five of images) of the confounding dataset. Afterwards, following the steps depicted in Section 4.1, we use each trained model to perform the reconstruction task on the training set, and both reconstruction and counterfactual reasoning tasks on the testing set (i.e., 28K sequences of length five of images). multiple latent states resulting in more blurry samples. Likewise, we can attain the same conclusion from the samples respectively produced by M orin and M decon on the confounding MNIST dataset, as shown in Figure 7 of Appendix I.

In addition, looking closely at the squares on the generated digit samples (colored in yellow box), it is obvious to observe that M orin will generate non-consecutive squares in the task of counterfactual reasoning, which does not really make sense because only consecutive patterns appear in the training set. In contrast, this case does not take place on the samples from M decon , showing that our deconfounding model is able to cope with long-range patterns.

It is worth noting that, as shown in Figure 8 of Appendix I, by visualizing the 2-dimensional confounder u, we can discern that although the prior distribution of the confounder is assumed to be a unit Gaussian distribution, the model still can learn two obvious clusters from the data because it is originally a binary variable. It demonstrates that our model has an advantage in learning confounders even if the assumed prior over them were not that accurate.

4.4. COMPARISON OF RL ALGORITHMS:

In this section, we will evaluate the proposed deconfounding actor-critic (AC) method by comparing with its vanilla version on the confounding Pendulum dataset. In the vanilla AC method, given a learned M orin , we optimize the policy by calculating the gradient presented in Equation ( 21) on the basis of the trajectories/rollouts generated through M orin . Equation ( 21) involves two functions: V (z t ; ϕ V ) and π(a t |z t ; θ), whose parameters can be found in Appendix J. It is worth noting that, in this vanilla case, each reward r t+1 is produced from the conditional distribution p(r t+1 |z t , a t ). In contrast, the proposed deconfounding AC method is built on M decon . Although the same gradient method (Equation ( 21)) is utilized to optimize the policy, we base the deconfounding AC approach on the different trajectories/rollouts generated by M decon in which each reward r t+1 relies on the interventional distribution p(r t+1 |z t , do(a t )) computed using Equation (20).

In the training phase, for both vanilla AC and deconfounding AC, we run a respective experiment over 1500 episodes with 200 time steps each. In order to reduce non-stationarity and to decorrelate updates, the generated data is stored in an experience replay memory and then randomly sampled in a batch manner (Mnih et al., 2013; Riedmiller, 2005; Schulman et al., 2015; Van Hasselt et al., 2016) . In each episode, we summarize all the rewards and further average the sums over a window of 100 episodes to obtain a more smooth curve. As shown in Figure 3 (a), obviously our deconfounding AC algorithm performs significantly better than the vanilla AC algorithm in the confounded environment.

In the testing phase, we first randomly select 100 samples from the testing set, each starting a new episode, and then use the learned policies to perform reasoning over 200 time steps as we did during the training time. From the resulting 100 episodes, we plot the total reward for each, shown in Figure 3 (b), and compute the percentage of the optimal action T 1 in each episode, presented in Figure 3(c) .

It is worth noting that Figure 3 (c) tells us that in each episode our deconfounding AC almost chooses the optimal action at each time step, whilst the vanilla AC makes a wrong decision for more than half time. 2018) discussed some issues of evaluating RL algorithms in observational health setting. However, all the work mentioned above did not take into account confounders in their models. Louizos et al. (2017) attempted to learn individual-level causal effects from observational data using variational auto-encoder to estimate the unknown confounder given a causal graph in then nontemporal setting. Paxton et al. (2013) developed predictive models based on electronic medical records without using causal inference. Saria et al. (2010) proposed a nonparametric Bayesian method to analyze clinical temporal data. Soleimani et al. (2017) represented the treatment response curves using linear time-invariant dynamical systems which provides a flexible approach to modeling response over time. Although the latter two work modeled the sequential data, they both do not exploit RL or causal inference. Bareinboim et al. (2015) considered the problem of bandits with unobserved confounders, which is one quite simple RL setting without state transitions. Sen et al. (2016) and Ramoly et al. (2017) further studied contextual bandits with latent confounders. Forney et al. (2017) circumvented some problems caused by unobserved confounders in Multi-Armed Bandit by counterfactual-based decision-making. Zhang & Bareinboim (2017) leveraged causal inference to tackle the problem of transferring knowledge across bandit agents. However, all these methods are based on the bandit problem, a simplified version of RL, instead of the full RL problem.

In fact, as far as we are concerned, this is the first attempt to build a bridge between confounding and the full RL problem, and this is also one of few research papers aiming at understanding the connections between causal inference and full RL.

6. CONCLUSION AND FUTURE WORK:

To address the confounding issue in RL, we introduced a general formulation, namely deconfounding reinforcement learning. On the basis of the proposed formulation, we presented deconfounding variants of actor-critic methods and showed their superior performance on confounding datasets that we created by revising OpenAI Gym and MNIST. In the future, we will collaborate with hospitals and apply our approach to real-world medical datasets. We also hope that our work will stimulate further investigation of connections between causal inference and RL.

Appendices:



A IDENTIFICATION OF CAUSAL EFFECT:

As pointed out in (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Wooldridge, 2009) , it is impossible to recover the joint distribution from the observational data if the hidden confounder has no connection to the observed variables. Fortunately, there still exist a large number of possible circumstances, and we refer readers to (Louizos et al., 2017; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Allman et al., 2009) for more details.

Here, we only focus on one possible case presented in Figure 1 of (Louizos et al., 2017) (see Appendix F.1), because the result can be directly used to identify the causal effect in our model. More precisely, in our model, at each time step, the 4-tuple (z t , x t , a t , r t+1 ) is exactly the same as that case in Figure 1 of (Louizos et al., 2017) , and therefore p(z t , x t , a t , r t+1 ) can be approximately recovered only from the observations (x t , a t , r t+1 ). Likewise, the joint distribution over the other 4-tuple (u, x t , a t , r t+1 ) can be recovered from (x t , a t , r t+1 ) in the same manner (see Appendix G).

Applying this rule repeatedly to the sequential data, we are finally able to approximately recover p(u, ⃗ z, ⃗ x, ⃗ a, ⃗ r) solely from the observations (⃗ x, ⃗ a, ⃗ r).

We recover the joint distribution using Variational Auto-Encoders (VAEs) (Kingma & Welling, 2013), which is a powerful tool to recover a very large class of latent-variable models as a minimizer of an optimization problem (Tran et al., 2015; Louizos et al., 2017) . Generally speaking, however, VAEs currently have little theory available to guarantee that the true model can be identified through learning (Louizos et al., 2017) . The main reason behind is that, in such latent variable models, confounders and causal parameters, both of which are unknown and might influence each other, will inevitably result in the case that different causal parameters could generate the same observable data (D'Amour, June 9, 2018). Fortunately, identification is possible when the latent confounder and their relationship to the outcome are themselves identified. For instance, in the presence of proxy variables of the latent confounder, as described in the previous paragraph, both our model and the one in (Louizos et al., 2017) are identified under some conditions outlined in (Louizos et al., 2017; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Allman et al., 2009) . In addition, even if only considering our model, because of the existence of many proxy variables of the confounders (e.g., each of z t and u has at least T proxy variables {x t } t=1,...,T , where z t can be viewed as hidden states in HMMs (Allman et al., 2009) ), identification can be also justified if the latent confounders z t and u are categorical (Allman et al., 2009) . In practice, however, we are not able to know the exact nature of the confounders, e.g., what kinds of distributions they follow, if categorical how many categories they have, etc. Hence, we take advantage of VAEs, which allows us to make substantially weaker assumptions about the data generating process and the structure of the hidden confounders (Louizos et al., 2017) . Nevertheless, we empirically show that our approach is more beneficial to learning a better policy in the existence of confounders.

Note that, in our model we have to differentiate the two types of confounders: the time-independent confounder u and time-dependent confounders {z t } t=1,...,T , each playing a respective role in the model. The former, as a global confounder, will affect the whole course of treatment, and therefore should be adjusted for. In the example of kidney stones, the existence of the confounder (i.e., the size of stones) will lead to a wrong treatment if not adjusting for it. In contrast, the time-varying confounders {z t } act as states in RL, which, in principle, should not be adjusted for, because the goal in RL is to learn a good policy in which any action is indeed supposed to be conditional on a specific state. On the other hand, in terms of rewards, what an agent expects at each time step is exactly the immediate reward when taking a specific action at a specific state, without the need of adjusting for states. It is worth noting that the case with time-varying confounders {z t } can be thought of to meet a pseudo or weak causal sufficiency assumption under which the causal effects of actions on rewards will not be influenced by states at each time step (Zhang et al., 2017; 2015) . This key difference motivates us to only adjust for the time-independent confounder u in this paper.

In addition, as shown in Figure 1 (b), in our case where a policy depending only on the confounder is applied to generating the data (Section 4.2), the arrow from z t to a t is not necessary so that z t can be viewed as not a confounder of a t and r t+1 . This also provides another reason why we do not need to adjust for z t .

B VARIATIONAL LOWER BOUND FOR M DECON log p θ (⃗ x, ⃗ a, ⃗ r) -KL (q(u|⃗ x, ⃗ a, ⃗ r)||p(u)) -KL (q(z 1 |⃗ x, ⃗ a, ⃗ r)||p(z 1 ))

= log ∫ u ∫ ⃗ z p θ (⃗ x, ⃗ a, ⃗ r, ⃗ z, u)d⃗ zdu ≥ ∫ u ∫ ⃗ z q ϕ (⃗ z, u|⃗ x, ⃗ a, ⃗ r) log p θ (⃗ x, ⃗ a, ⃗ r, ⃗ z, u) q ϕ (⃗ z, u|⃗ x, ⃗ a, ⃗ r) d⃗ zdu = ∫ u ∫ ⃗ z q ψ (u|⃗ x, ⃗ a, ⃗ r)q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) log p θ (⃗ x, ⃗ a, ⃗ r, ⃗ z, u) q ψ (u|⃗ x, ⃗ a, ⃗ r)q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) d⃗ zdu (factorization assumption) = ∫ u ∫ ⃗ z q ψ (u|⃗ x, ⃗ a, ⃗ r)q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) log p(u)p(z 1 ) [ ∏ T t=1 p(x t |z t , u)p(a t |z t , u)p(r t+1 |z t , a t , u) ] [ ∏ T t=2 p(z t |z t-1 , a t-1 ) ] q ψ (u|⃗ x, ⃗ a, ⃗ r)q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) d⃗ zdu = ∫ u ∫ ⃗ z q(u|⃗ x, ⃗ a, ⃗ r)q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log p(u)p(z 1 ) ∏ T t=1 p(x t |z t , u)p(a t |z t , u)p(r t+1 |z t , a t , u) ∏ T t=2 p(z t |z t-1 , a t-1 ) q(u|⃗ x, ⃗ a, ⃗ r)q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) d⃗ zdu = T ∑ t=1 ∫ u ∫ z1 • • • ∫ z T q(u|⃗ x, ⃗ a, ⃗ r)q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗

+ ∫ u ∫ z1 • • • ∫ z T q(u|⃗ x, ⃗ a, ⃗ r)q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log p(u) q(u|⃗ x, ⃗ a, ⃗ r) d⃗ zdu + ∫ u ∫ z1 • • • ∫ z T q(u|⃗ x, ⃗ a, ⃗ r)q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log p(z 1 ) q(z 1 |⃗ x, ⃗ a, ⃗ r) d⃗ zdu + T ∑ t=2 ∫ u ∫ z1 • • • ∫ z T q(u|⃗ x, ⃗ a, ⃗ r)q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log p(z t |z t-1 , a t-1 ) q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) d⃗ zdu = T ∑ t=1 ∫ u ∫ zt q(u|⃗ x, ⃗ a, ⃗ r)q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) log (p(x t |z t , u)p(a t |z t , u)p(r t+1 |z t , a t , u)) dz t du + ∫ u q(u|⃗ x, ⃗ a, ⃗ r) log p(u) q(u|⃗ x, ⃗ a, ⃗ r) du + ∫ z1 q(z 1 |⃗ x, ⃗ a, ⃗ r) log p(z 1 ) q(z 1 |⃗ x, ⃗ a, ⃗ r) dz 1 + T ∑ t=2 ∫ zt-1 ∫ zt q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) log p(z t |z t-1 , a t-1 ) q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) dz t dz t-1 = T ∑ t=1 E zt∼q(zt|zt-1,⃗ x,⃗ a,⃗

- T ∑ t=2 E zt-1∼q(zt-1|zt-2,⃗ x,⃗ a,⃗ r) [KL (q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r)||p(z t |z t-1 , a t-1 ))] . C VARIATIONAL LOWER BOUND FOR M ORIN log p θ (⃗ x, ⃗ a, ⃗ r) = log ∫ ⃗ z p θ (⃗ x, ⃗ a, ⃗ r, ⃗ z)d⃗ z ≥ ∫ ⃗ z q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) log p θ (⃗ x, ⃗ a, ⃗ r, ⃗ z) q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) d⃗ z = ∫ ⃗ z q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) log p(z 1 ) [ ∏ T t=1 p(x t |z t )p(a t |z t )p(r t+1 |z t , a t ) ] [ ∏ T t=2 p(z t |z t-1 , a t-1 ) ] q ϕ (⃗ z|⃗ x, ⃗ a, ⃗ r) d⃗ z = ∫ ⃗ z q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log p(z 1 ) ∏ T t=1 p(x t |z t )p(a t |z t )p(r t+1 |z t , a t ) ∏ T t=2 p(z t |z t-1 , a t-1 ) q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) d⃗ z = T ∑ t=1 ∫ z1 • • • ∫ z T q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log (p(x t |z t )p(a t |z t )p(r t+1 |z t , a t )) d⃗ z + ∫ z1 • • • ∫ z T q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log p(z 1 ) q(z 1 |⃗ x, ⃗ a, ⃗ r) d⃗ z + T ∑ t=2 ∫ z1 • • • ∫ z T q(z 1 |⃗ x, ⃗ a, ⃗ r) • • • q(z T |z T -1 , ⃗ x, ⃗ a, ⃗ r) log p(z t |z t-1 , a t-1 ) q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) d⃗ z = T ∑ t=1 ∫ zt q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) log (p(x t |z t )p(a t |z t )p(r t+1 |z t , a t )) dz t + ∫ z1 q(z 1 |⃗ x, ⃗ a, ⃗ r) log p(z 1 ) q(z 1 |⃗ x, ⃗ a, ⃗ r) dz 1 + T ∑ t=2 ∫ zt-1 ∫ zt q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) log p(z t |z t-1 , a t-1 ) q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r) dz t dz t-1 = T ∑ t=1 E zt∼q(zt|zt-1,⃗ x,⃗ a,⃗ r) [log p(x t |z t ) + log p(a t |z t ) + log p(r t+1 |z t , a t )] -KL (q(z 1 |⃗ x, ⃗ a, ⃗ r)||p(z 1 )) - T ∑ t=2 E zt-1∼q(zt-1|zt-2,⃗ x,⃗ a,⃗ r) [KL (q(z t |z t-1 , ⃗ x, ⃗ a, ⃗ r)||p(z t |z t-1 , a t-1 ))] .

D BI-DIRECTIONAL LSTM:

In our inference model, we use a similar architecture of bi-directional LSTM to that in (Krishnan et al., 2017) . Apart from different inputs, the main difference is to introduce a t-1 to computing the combined hidden feature as shown in the following formula.

⃗ 0 ẑ1 ẑ2 ẑ3 function (µ 1 , Σ 1 ) (µ 2 , Σ 2 ) (µ 3 , Σ 3 ) Backward LSTM h right 1 h right 2 h right 3 Forward LSTM h lef t 1 h lef t 2 h lef t 3 x 1 , a 1 , r 2 x 2 , a 2 , r 3 x 3 , a 3 , r 4 h combined = 1 4 ( tanh(W z • z t-1 + b z ) + tanh(W a • a t-1 + b a ) + h left t + h right t ) µ t = W µ • h combined + b µ σ 2 t = softplus (W σ 2 • h combined + b σ 2 )

E TRAINING DETAILS:

As mentioned in the main body, DRL consists of two steps: learning a deconfounding model and optimizing a policy based on the learned deconfounding model. To be more specific, in step 1, given the observational data (⃗ x, ⃗ a, ⃗ r), we learn the deconfounding model by optimizing the variational lower bound as presented in Equation ( 19). Once the deconfounding model is learned, we know the state transition function p(z t |z t-1 , a t-1 ) and can also calculate the deconfounding reward function p(r t |z t , do(a t )) according to Equation (20). In step 2, we treat the learned deconfounding model as a RL environment like CartPole in OpenAI Gym, and directly exploit it to generate trajectories/rollouts through the state transition function and the deconfounding reward function. On the basis of the generated trajectories/rollouts, we can train the policy network using Equation ( 21).

F EXAMPLE OF KIDNEY STONES F.1 EXAMPLE OF A PROXY VARIABLE:

For convenience, we directly move Figure 1 of (Louizos et al., 2017) here, as shown in Figure 4 .

F.2 INTUITION:

Take for example the case of kidney stones, in the existence of the confounder (i.e., the size of stones), one prefers treatment b when considering the overall probability of recovery, whilst one chooses with more probabilities treatment a when investigating the recovery rate in each size. The correct answer is treatment a in this example. The do-operator in Equation (2) of Section 2.3 provides a formal approach to seeking the correct solution to such confounded problems, which can not be addressed within the traditional RL framework, because the traditional framework only involves the conditional probability (e.g., the overall probability of recovery) instead of the interventional probability as do-operator does. 

G ANALYSIS OF THE DECONFOUNDING MODEL:

At each time step t, from the deconfounding model presented in Figure 1 (b), we can extract two 4tuple components, centering at z t (Figure 5 ) and u (Figure 6 ), respectively. It is apparent to observe that both Figure 5 and Figure 6 share the exact same structure with Figure 4 .

x The original reward term r o is defined as the minus degree between the upright position and the position the digit rotates to. For example, if the digit rotates to the position of 3 o'clock or 9 o'clock, then both rewards are -90.

H.3 DETAILS ABOUT CONFOUNDING SIMULATIONS:

The probabilities between the confounder, action and reward are presented as follows, p(u = 0) = 0.8, p(u = 1) = 0.2; (31)

Note that, in our model r c is a function of u, a t , and z t . We omit the notation of the state in the conditional probability of r c because z t is fixed at each time step, which does not affect the probability. To help readers understand this setting, we can make a straightforward analogy. More specifically, the confounder u stands for the socio-economic status where u = 0 represents the poor and u = 1 the rich. Treatment T 1 is more expensive but more valid than Treatment T 2 . R 1 means a healthier feedback/recovery than R 2 . Equation ( 24) says that the poor are much more than the rich (it is indeed in reality). Equation ( 25) and Equation ( 26) tell us the fact that the poor tend to choose the cheaper but less valid treatment T 2 whilst the rich prefer the more expensive but more valid treatment T 1 , which also makes sense in the real world. Finally, Equations ( 27)-( 30) reveal that T 1 is a better treatment than T 2 within both poor and rich population.

I ADDITIONAL EXPERIMENTAL RESULTS:

As shown in Figure 7 and Figure 8 .

J EXPERIMENTAL SETTINGS:

As shown in Table 1 and Table 2 . 

:

Another straightforward example is that, in economics, considering the risky cost in terms of time and money, it is not practical to study the optimal strategy by actually buying and selling stocks in the market.

Note that, in our case where a policy depending only on the confounder is applied to generating the data (Section 4.2), the arrow from zt to at is not necessary when learning the model so that zt can be viewed as not a confounder of at and rt+1. This also provides another reason why we do not need to adjust for zt. However, in some other cases such as when applied to medical data, the strategies of treatment from physicians definitely contain valuable information about zt and at, and therefore the arrow between them is necessary when learning the model.

An intuition can be found in Appendix F.2.

More details can be found at https://github.com/openai/gym/wiki/Pendulum-v0.

In fact, here ro is a function of both a and a state, and we mention only a to emphasize that the confounder affects the action.

Figures and Tables:

Figure 1: (a) Causal diagram for kidney stones. (b) The model for deconfounding reinforcement learning. Solid nodes denote observed variables and open nodes represent unobserved variables. Red and blue arrows emphasize the observed variables affected by u and by z t , respectively. The causal effects of interest are colored in green.

Figure2presents a comparison of M decon and M orin , in terms of reconstruction and counterfactual reasoning on the confounding Pendulum dataset. The second row is based on M decon (Figure1(b)), whilst the top row comes from M orin . It is evident that the results generated by the deconfounding model is superior to those produced by the model not taking into account the confounder. To be more specific, as shown in the zoom of samples on the bottom row, M orin generates more blurry images than M decon , because, without modelling the confounder u, M orin is forced to average over its

Figure 2: Reconstruction and counterfactual reasoning on the confounding Pendulum dataset. Top row: results from the model without the confounder u (M orin ); Second row: results from the model with the confounder u (M decon ). The last two rows are the zoom of samples selected in the same positions from the top row (dashed boxes for M orin ) and the second row (solid boxes for M decon ), respectively.

Figure 3: Comparison of vanilla (Orin AC) and deconfounding (Decon AC) Actor-Critic methods on the confounding Pendulum dataset. (a) total reward over 1500 episodes in the training phase; (b) total reward over 100 episodes in the testing phase; (c) Probability of optimal action over 100 episodes, corresponding to (b). 5 RELATED WORK Krishnan et al. (2015; 2017) used deep neural networks to model nonlinear state space models and leveraged a structured variational approximation parameterized by recurrent neural networks to mimic the posterior distribution. Levine (2018) reformulated RL and control problems to probabilistic inference, which allows us to bear a large pool of approximate inference methods, and flexibly extend the model. Raghu et al. (2017a;b) exploited continuous state-space models and deep RL to deduce treatment policies for septic patients from observational data. Gottesman et al. (2018) discussed some issues of evaluating RL algorithms in observational health setting. However, all the work mentioned above did not take into account confounders in their models.

(24) p(a = T 1 |u = 0) = 0.24, p(a = T 2 |u = 0) = 0.76; (25) p(a = T 1 |u = 1) = 0.77, p(a = T 2 |u = 1) = 0.23; (26) p(r c = R 1 |a = T 1 , u = 0) = 0.93, p(r c = R 2 |a = T 1 , u = 0) = 0.07; (27) p(r c = R 1 |a = T 2 , u = 0) = 0.87, p(r c = R 2 |a = T 2 , u = 0) = 0.13; (28) p(r c = R 1 |a = T 1 , u = 1) = 0.73, p(r c = R 2 |a = T 1 , u = 1) = 0.27; (29) p(r c = R 1 |a = T 2 , u = 1) = 0.69, p(r c = R 2 |a = T 2 , u = 1) = 0.31;(30) µ rc=R1 = -1, µ rc=R2 = -200, σ = 2.

Figure 7: Reconstruction and counterfactual reasoning on the confounding MNIST dataset. Top row: results from the model without the confounder u (M orin ); Bottom row: results from the model with the confounder u (M decon ). The sequences boxed in yellow have non-consecutive squares.

to be seen as an estimate of the advantage of action a t in state z t . In practice, Q(z t , a t ; ϕ Q ) is usually replaced with one-step return, that is, r t+1 + V (z t+1 ; ϕ V ). Importantly, in deconfounding actor-critic methods, we use r t+1 ∼ p(r t+1 |z t , do(a t )) (Equation (20)), as opposed to r t+1 ∼ p(r t+1 |z t , a t ) in vanilla actor-critic methods. The details about training can be found in Appendix E.

Table content: <table/>

r) log (p(x t |z t , u)p(a t |z t , u)p(r t+1 |z t , a t , u)) d⃗ zdu

Table content: <table/>

Example of a proxy variable. t is a treatment, e.g., medication; y is an outcome, e.g., mortality; Z is an unobserved confounder, e.g., socio-economic status; and X is noisy views on the hidden confounder Z, say income in the last year and place of residence.

Table content: <table><tr><td>y</td><td/></tr><tr><td>X</td><td>t</td></tr><tr><td>Z</td><td/></tr><tr><td>Figure 4:</td><td/></tr></table>

We follow the same setting to develop a confounding MNIST dataset, but the definitions of action and the original reward term are different. Similar to the Healing MNIST dataset(Krishnan et al., 2015), rotations of digit images are encoded as the actions ⃗ a (-45 ≤ a ≤ 45) which, according to the binary confounder u, is divided into two categories (22.5 ≤ |a| ≤ 45) and (0 ≤ |a| < 22.5).

Table content: <table><tr><td>H CONFOUNDING DATASETS</td></tr><tr><td>H.1 CONFOUNDING CARTPOLE</td></tr><tr><td>The confounding CartPole dataset can be implemented in the exact same manner as Confounding</td></tr><tr><td>Pendulum, except for the action which is originally binary and can be naturally divided into two</td></tr><tr><td>categories.</td></tr><tr><td>H.2 CONFOUNDING MNIST</td></tr><tr><td>x t</td></tr><tr><td>z t</td></tr><tr><td>a t</td></tr><tr><td>r t+1</td></tr></table>

Hyperparameters for deconfounding reinforcement learning. Note that, we set the dimension u to 1 in Section 4.4 for simplicity.

Table content: <table><tr><td>Hyperparameter</td><td>Value</td></tr><tr><td>Deconfounding Model</td><td/></tr><tr><td>learning rate</td><td>0.0001</td></tr><tr><td>dimension of z t</td><td>50</td></tr><tr><td>dimension of x t</td><td>784</td></tr><tr><td>dimension of a t</td><td>1</td></tr><tr><td>dimension of r t</td><td>1</td></tr><tr><td>dimension of u</td><td>2</td></tr><tr><td>dimension of LSTM unit</td><td>100</td></tr><tr><td>batch size</td><td>128</td></tr><tr><td>number of steps</td><td>5</td></tr><tr><td>number of epoch</td><td>400</td></tr><tr><td>Deconfounding AC</td><td/></tr><tr><td>number of episodes</td><td>1500</td></tr><tr><td>number of time steps in each episode</td><td>200</td></tr><tr><td>capacity of the replay memory</td><td>100,000</td></tr><tr><td>batch size</td><td>128</td></tr><tr><td>sample size of u</td><td>200</td></tr></table>

Architectures for deconfounding reinforcement learning. Here FC LSTM n,t for a LSTM layer rolling out for t steps with latent size of n, {•} for the parallel operators, and [•]for the sequential operators. In default, FC k and Conv k,n are followed by a softplus activation layer and a batch-norm layer, which are omitted here for simplicity. Note that, in our setting, the

Table content: <table><tr><td>units, Conv k,n for a convolution layer with</td></tr></table>

Peer review (iclr2019):
 The paper addresses an important and often overlooked issue in off-policy reinforcement learning - the possibility of confounding between the agent*s actions and the rewards. This is a subject which has been exhaustively explored in the causal inference literature, and the authors are very correct in suggesting that it should be incorporated into the world of reinforcement learning. Specifically they propose a generative model with a global latent confounder that is inferred using a variational autoencoder architecture. The paper is generally well-written, though some points could be made clearer in my opinion, as detailed below. The experiments are constructed by introducing confounding into existing datasets; performance seems to be good, but I am not entirely sure whether the given architecture is necessary, see comments below. High-level comments: (1) Classic RL deals with confounders all the time. The state is a confounder between the action and the reward. The issue of confounding becomes less trivial when one is performing off-policy RL when the original policy is *unknown*. This is exactly the case that the authors mention when they cite the recent work by Gottesman et al. (2018) who deal with using RL to learn from the actions of physicians in a hospital. While I am sure the authors are aware of these distinctions, I think the paper would be better if this is spelled out very explicitly. This includes explaining why this issue doesn*t come up in classic RL. (2) Assuming the case above - off-policy RL with unknown confounders - one would usually assume "no unmeasured confounding", i.e. that the observed actions are an unknown but learnable function of the observed states. That is basically the scenario of most off-policy RL. (3) However, the authors strive to go one step beyond the case (2), to a situation where there is an *unmeasured* confounder affecting both observed actions and rewards. If nothing is known about this unmeasured confounder, then it is generally impossible to learn effective policies, as the causal effects of actions are not identifiable from the observed data. In this paper, the authors make an implicit assumption that while the confounder is unmeasured, it can still be inferred from the data. This is an intermediate step between "no unmeasured confounding" and "complete unmeasured confounding". This is related to work on using proxy variables e.g. Kuroki & Pearl (2014) and even more closely related to the work cited by Louizos et al. (2017). Again, I think the paper would be much improved if all this is addressed explicitly. (4) An important consequence of point (3) above is that in fact adding the single global latent-confounder U is not, in itself, very important from a causal perspective. The sequence of variables Z_1... Z_T are already latent confounders that are assumed to be inferrable from data. It is true that the addition of the global U might change the statistical and optimization properties of the model. This leads to a very important conclusion: the authors should test their model with and without U. I think this specific ablation experiment is crucial. In many cases I am sure that the assumption of a global latent confounder is a good one and is especially useful in the VAE case where it will make optimization more stable. However, in principle, all of U*s roles could be taken within the sequence of Z*s, and I am curious to see in practice how big of an effect it has. (5) I wish to add that even if the U variable turns out to not add much empirically, this work is still valid since the sequence of Z*s can themselves be considered inferred latent confounders. Specific comments: (1) 2.3: there are more than 2 ways of computing the do-operator. RCTs and backdoor are the best known approaches, but not the only ones, e.g. there is frontdoor adjustment. (2) I think the paper would be easier to follow if there was one concrete example used throughout. This will make it easier to understand and possibly verify/criticize the assumptions of the generative model. (3) Related to "higher-level point (4)" above, in eqs. 17 & 18 note that Z_t is unknown, same as U. Both are inferred. This also leads to the question which Z_t is actually used in practice? Is it the mean, or is it also sampled from the approximate posterior q? (4) Below eq. 19, it would be very useful for the readers if you could explain exactly when would there be a difference between the two versions p(r_{t+1}|z_t,a_t) and p(r_{t+1}|z_t, do(a_t=a)) (5) In the description of all the experiments I was missing a crucial point: how does the introduced confounder affect the reward? Is it only through the different actions? The way it is currently explained, it seems like the added variable introduces lack of *overlap*, but not strictly confounding. (6) The description of the experiment in 4.3 could be more detailed. What exactly was the training and test? What RL method was used? What did the baseline optimize for? I would like to see an ablation experiment where U is not included in the model. (7) In 4.5, what is the "vanilla" method? And as mentioned above, I would like to see an ablation experiment where U is not included in the model.