Year: 2018

Abstract:
In this paper, we propose a general formulation to cope with a family of reinforcement learning tasks in observational settings, that is, learning good policies solely from the historical data produced by real environments with confounders (i.e., the factors affecting both actions and rewards). Based on the proposed approach, we extend one representative of reinforcement learning algorithms: the ActorCritic method, to its deconfounding variant, which is also straightforward to be applied to other algorithms. In addition, due to lack of datasets in this direction, a benchmark is developed for deconfounding reinforcement learning algorithms by revising OpenAI Gym and MNIST. We demonstrate that the proposed algorithms are superior to traditional reinforcement learning algorithms in confounded environments. To the best of our knowledge, this is the first time that confounders are taken into consideration for addressing full reinforcement learning problems.

heading: 1 INTRODUCTION
text: In recent years, reinforcement learning (RL) has made great progress, spawning a large number of successful applications especially in terms of games (Silver et al., 2016; Mnih et al., 2013; OpenAI, 2018). Within this background, much attention has been devoted to the development of RL algorithms with the goal of improving treatment policies in healthcare (Gottesman et al., 2018). In fact, various RL algorithms have been proposed to infer better decision-making strategies for mechanical ventilation (Prasad et al., 2017), sepsis management (Raghu et al., 2017a;b), and treatment of schizophrenia (Shortreed et al., 2011). In healthcare, a common practice is to focus on the observational setting, because ones do not wish to experiment with patients’ lives without evidence that the proposed treatment strategy is better than the current practice (Gottesman et al., 2018) 1. As pointed out in (Raghu et al., 2017a), even if in the observational setting, RL also has advantages over other machine learning algorithms especially in two situations: when the ground truth of a “good” treatment strategy is unclear in medical literature (Marik, 2015), and when training examples do not represent optimal behavior. On the other hand, although causal inference (Pearl, 2009) has been extensively explored and used in healthcare and medicine (Liu et al.; Soleimani et al., 2017; Schulam & Saria, 2017; Alaa et al., 2017; Alaa & van der Schaar, 2018; Atan et al., 2016), the efficient approach to dealing with time-varying data is still unclear (Peters et al., 2017; Hernn & Robins, 2018). On the basis of the discussion above, in this paper we attempt to combine advantages on both sides to cope with an important family of RL problems in the observational setting, that is, learning good policies solely from the historical data produced by real environments with confounding bias.
This type of problems are inevitable in the future RL research with the burgeoning development in healthcare and medicine. To the best of our knowledge, however, little work has been done in the direction. Confounding is a causal concept that is described in the language of causality instead of probability and statistics (Pearl, 2009). Confounding bias occurs when a variable influences both who is selected for the treatment and the outcome of the experiment (Pearl & Mackenzie, 2018), which naturally corresponds to the action and the reward in RL, respectively. As a matter of fact, confounders have been extensively studied in epidemiology, sociology, and economics. Take for example the widespread kidney stones in which the size of the kidney stone is a confounding factor affecting both the treatment and the recovery (Peters et al., 2017; Pearl, 2009), whether deconfounding the size of the kidney stone or not entirely determines how to choose a more effective treatment. Similarly, in RL, if unobserved potential confounders exist, they would affect both ac-
1Another straightforward example is that, in economics, considering the risky cost in terms of time and money, it is not practical to study the optimal strategy by actually buying and selling stocks in the market.
tions and rewards when an agent interacts with environments and eventually influence the policy to be optimized.
It is widely acknowledged that one should draw a causal graph before one can achieve any causal conclusion (Pearl, 2009; Pearl & Mackenzie, 2018). Throughout the paper, we assume that, given causal assumptions, we first estimate a model from the observational data we collected from real environments or simulators, and then optimize a policy on the basis of the learned model. As mentioned previously, this assumption is quite useful in real-world RL applications (e.g., healthcare, medicine, economics, etc), because in most circumstances, except the data we observed, we neither know anything about real environments nor are allowed to do anything in real environments probably for the sake of ethics, laws or cost.
In order to adjust for confounders, we present a general formulation for addressing this class of RL problems, namely deconfounding reinforcement learning (DRL). More specifically, given several common confounding assumptions, under some conditions for identification we first estimate a latent-variable model from observational data in which we simultaneously discover the latent confounders and infer how they affect action and reward, then deconfound the confounders using the causal language developed by (Pearl, 2009), and finally optimize the policy based on the deconfounding model we calculated. On the basis of the proposed formulation, we extend one popular RL algorithm, the Actor-Critic method, to its corresponding deconfounding variant, which is straightforward to be applied to other RL algorithms. Due to lack of datasets in this respect, we revise the classic control toolkit in OpenAI Gym (Brockman et al., 2016), making it a benchmark for comparison of DRL algorithms. In addition, we also devise a confounding version of the MNIST dataset (LeCun et al., 1998) to verify the performance of our causal model. Finally, we conduct extensive experiments to demonstrate the superiority of the proposed formulation in confounded environments, in comparison to traditional RL algorithms.
To sum up, our contributions in this paper are as follows: 1. We propose a general formulation to address a family of RL problems in confounded envi-
ronments, namely deconfounding reinforcement learning (DRL); 2. We present the deconfounding variant of Actor-Critic methods, which is obvious to be
applied to other RL methods; 3. We develop a benchmark for DRL by revising the toolkit for classic control in OpenAI
Gym (Brockman et al., 2016) and by devising a confounding version of the MNIST dataset (LeCun et al., 1998);
4. We perform a comprehensive comparison of DRL algorithms with their vanilla versions, showing that the proposed approach has an advantage in confounded environments.
5. To the best of our knowledge, this is the first attempt to build a bridge between confounding and the full RL problem. This is one of few research papers aiming at understanding the connections between causal inference and the full RL.

heading: 2 BACKGROUND
text: In this section, we briefly review confounding in causal inference. We recommend Pearls excellent monograph for further reading (Pearl, 2009; Pearl & Mackenzie, 2018).

heading: 2.1 SIMPSON’S PARADOX
text: Let us begin with one of the most famous paradoxes in statistics: Simpson’s Paradox. Consider the previously mentioned kidney stones, a classic example of Simpson’s paradox (Peters et al., 2017). We collect electronic patient records to investigate the effectiveness of two treatments against kidney stones, where although the overall probability of recovery is higher for patients who took treatment b, treatment a performs better than treatment b on both patients with small kidney stones and with large kidney stones. More precisely, we have
p(R = 1|T = b) > p(R = 1|T = a); but p(R = 1|T = b, Z = 0) < p(R = 1|T = a, Z = 0), p(R = 1|T = b, Z = 1) < p(R = 1|T = a, Z = 1); (1)
where Z is the size of the stone, T the treatment, and R the recovery (all binary). How do we cope with this inversion of conclusion? Which treatment do you prefer if you had kidney stones?
Does treatment b cause recovery? The answers to these questions depend on the causal relationship between treatment, recovery, and the size of the kidney stone.

heading: 2.2 CONFOUNDING
text: An intuitive explanation for this kidney stone example of Simpson’s paradox is that larger stones are more severe than small stones and are much more likely to be treated with treatment a, resulting in that treatment a looks worse than treatment b. Therefore, it is straightforward to assume that the true underlying causal diagram of the kidney stone example is shown in Figure 1(a), where confounding occurs because the size of kidney stones influences both treatment and recovery. Here, the size of kidney stones is called confounder. The term “confounding” originally meant “mixing” in English, which describes that the true causal effect T → R is “mixed” with the spurious correlation between T and R induced by the fork T ← Z → R (Pearl & Mackenzie, 2018). In other words, we will not be able to disentangle the true effect of T on R from the spurious effect if we do not have data on Z. Conversely, if we have measurements of Z, it is easy to deconfound the true and spurious effects by adjusting for Z that averages the effect of T on R in each subgroup of Z (i.e., different size groups in the case of kidney stones).

heading: 2.3 DO-OPERATOR AND ADJUSTMENT CRITERION
text: From the viewpoint of causal inference, we can also use the language of intervention, namely dooperator, to formulate confounding. In fact, in the example of kidney stones, what we are interested in is how these two treatments compare when we force all patients to take treatment a or b, rather than which treatment has a higher recovery rate given only the observational patient records. Mathematically, we focus on the true effect p(R = 1|do(T = a)) (i.e., intervention distribution where patients are forced to take treatment a) instead of the spurious effect p(R = 1|T = a) (i.e., observational distribution where patients are observed to take treatment a). Therefore, as described previously, confounding can be naturally formulated by the discrepancy between p(R|T ) and p(R|do(T )). Generally speaking, do-operator can be executed in two common ways: by Randomized Controlled Trials (RCTs) (Fisher, 1935) and by adjustment formulas (i.e., Back-door criterion and Front-door criterion) (Pearl, 2009). RCTs is the so-called golden standard but rather limited due to many impractical factors (e.g., safety, laws, ethics, physically infeasibility, etc.). Back-door and Front-door criterions require a known causal diagram in which causal assumptions are provided in advance. According to the Back-door criterion, in the kidney stone example, we can immediately attain
p(R = 1|do(T = a)) = ∑1
z=0 p(R = 1|T = a, Z = z)p(Z = z). (2)
In fact, apart from the two adjustment formulas, we are provided with a syntactic method of deriving claims about interventions, namely do-calculus, developed by Pearl (Pearl, 2009). The do-calculus consists of three rules which can be repeatedly applied to simplify the expression for an interventional distribution. It offers a powerful tool to convert intervention probabilities to observation probabilities so that we can estimate the former from observational data alone.

heading: 2.4 PROXY VARIABLES FOR CONFOUNDING
text: If confounders can be measured, then they can be adjusted for through the methods we discussed in Section 2.3. However, in most cases where confounders are hidden or unmeasured, without further assumptions, it is impossible to estimate the effect of the intervention on the outcome. A common practice is to leverage observed proxy variables of the confounders (Angrist & Pischke, 2008; Maddala & Lahiri, 1992; Montgomery et al., 2000), which is one of the promising directions of exploiting big-data to conduct causal inference in the presence of multiple proxy variables for unmeasured confounders (Louizos et al., 2017). However, using proxy variables to correctly recover causal effects should meet strict mathematical assumptions (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Wooldridge, 2009). Unfortunately, in practice, we do not know whether or not the complicated data from the real world meet those assumptions. Hence, following the same spirit from (Louizos et al., 2017), with multiple proxy variables available, under relaxed assumptions we estimate a latent-variable model in which we simultaneously discover the latent confounders and infer how they affect treatment and outcome.

heading: 3 DECONFOUNDING REINFORCEMENT LEARNING
text: In this section, we will formally introduce deconfounding reinforcement learning (DRL). Generally speaking, DRL consists of two steps: learning a deconfounding model shown in Figure 1(b) and optimizing a policy based on the learned deconfounding model. The main idea in step 1 is to simultaneously discover the hidden confounder and infer the causal effect through an estimation of
a latent-variable model. More specifically, we first discuss the time-independent confounding assumption in Section 3.1, and then, based on the assumption, formalize the deconfounding model in Section 3.2. Section 3.3 talks about the problem of identification in our model, which is a central issue in causal inference. After that, we present the details about how to learn the model via variational inference in Section 3.4. Step 2 is provided in Section 3.5 where we describe how to design the deconfounding actor-critic method. It is straightforward to apply this to other RL algorithms in the same manner.

heading: 3.1 CAUSAL ASSUMPTIONS
text: Without loss of generality, as shown in Figure 1(b) 2, we assume there exists a common confounder in the sequential model, which is time-independent for each individual or for each procedure. This assumption is so general that it would apply to various RL tasks across domains. For example, in personalized medicine or precision medicine, socio-economic status can affect both the medication strategy a patient has access to, and the patients general health (Louizos et al., 2017). Therefore socio-economic status acts as confounder between the medication and health outcomes, in which case socio-economic status is time-independent for each patient during the course of treatment. In agriculture, soil fertility may serve as one of confounders affecting both the application of fertilizer and the yield of each plot (Pearl & Mackenzie, 2018). In this circumstance, soil fertility is stable and thought of as a time-independent factor within a period of time (e.g., several months, the growth circle of crops, etc.). In the example of stock markets, apart from socio-economic status as mentioned above, government policy may also act as one of confounders, all of which can be seen time-independent during a reasonable period of time.

heading: 3.2 THE MODEL
text: Given the causal assumption, we first fit a generative model to a sequence of observational data: observations, actions, and rewards, where actions and rewards are confounded by one or several unknown factors. Formally, Let x⃗ = (x1, . . . , xT ), a⃗ = (a1, . . . , aT−1), r⃗ = (r2, . . . , rT+1), z⃗ = (z1, . . . , zT ) be the sequence of observations, actions, rewards, and corresponding latent states, respectively. The confounder is denoted by u, and it is worth noting that here u may stand for more than one confounder in which multiple confounders are seen as a whole represented by u. We assume that xt ∈ RDx , at ∈ RDa , rt ∈ RDr , zt ∈ RDz , and u ∈ RDu , where Dz ≪ Dx. The generative model for DRL is then given by:
p(zt) = ∏Dz
j=1 N (ztj |0, 1); p(u) = ∏Du j=1 N (uj |0, 1);
p(xt|zt, u) = N ( xt|µ̂xt , σ̂xt 2 ) ; µ̂xt = f1(zt, u) σ̂ x t 2 = f2(zt, u); (3)
p(at|zt, u) = N ( at|µ̂at , σ̂at 2 ) ; µ̂at = f3(zt, u) σ̂ a t 2 = f4(zt, u); (4)
2Note that, in our case where a policy depending only on the confounder is applied to generating the data (Section 4.2), the arrow from zt to at is not necessary when learning the model so that zt can be viewed as not a confounder of at and rt+1. This also provides another reason why we do not need to adjust for zt. However, in some other cases such as when applied to medical data, the strategies of treatment from physicians definitely contain valuable information about zt and at, and therefore the arrow between them is necessary when learning the model.
p(rt+1|zt, at, u) = N ( rt+1|µ̂rt , σ̂rt 2 ) ; µ̂rt = f5(zt, at, u) σ̂ r t 2 = f6(zt, at, u); (5)
p(zt|zt−1, at−1) = N ( zt|µ̂zt , σ̂zt 2 ) ; µ̂zt = f7(zt−1, at−1) σ̂ z t 2 = f8(zt−1, at−1). (6)
Note that we parametrize each probability distribution as a Gaussian with its mean and variance modeled by nonlinear functions fk and each fk is parametrized by a neural network with its own parameters θk for k = 1, . . . , 8. Note that Equation (4) is not necessary in our model, but it is potentially useful acting as a prior policy because it is learned from the observational data containing, for example, the real treatment strategies by doctors.

heading: 3.3 IDENTIFICATION OF CAUSAL EFFECT
text: In our deconfounding model shown in Figure 1(b), the key difference from traditional RL lies in the reward function 3. To be more precise, assuming that an agent standing at state zt performs an action at = a, unlike the traditional reward p(rt|zt, at = a), our deconfounding version based on do-operator as depicted in Section 2.3 is given by
p(rt|zt, do(at = a)) = ∫ u p(rt|zt, do(at = a), u)p(u|zt, do(at = a))du, (7)
= ∫ u p(rt|zt, at = a, u)p(u)du, (8)
where Equation 8 is by the rules of do-calculus applied to the causal graph in Figure 1(b) (Pearl, 2009). In fact, we can utilize the back-door criterion to directly obtain Equation (8). Furthermore, through Equation (8) we proved that p(rt|zt, do(at = a)) can be identified from the joint distribution p(u, z⃗, x⃗, a⃗, r⃗). Then, motivated by the idea that one is supposed to use the knowledge inferred from the joint distribution between proxy variables and confounders to adjust for the hidden confounders (Louizos et al., 2017; Edwards et al., 2015; Kuroki & Pearl, 2014; Miao et al., 2018; Pearl, 2012; Wooldridge, 2009), in this paper we also assume that the joint distribution p(u, z⃗, x⃗, a⃗, r⃗) can be approximately recovered solely from the observations (x⃗, a⃗, r⃗). The details about identification can be found in Appendix A.

heading: 3.4 LEARNING
text: Since the nonlinear functions parametrized by neural networks make inference intractable, we will learn the parameters of the model θk by employing variational inference along with an inference model, a neural network which approximates the intractable posterior (Rezende et al., 2014; Kingma & Welling, 2013; Krishnan et al., 2015). More specifically, using the variational principle, we posit an approximate posterior distribution qϕ(z|x) to obtain the following lower bound on the marginal likelihood:
log pθ(x) ≥ E qϕ(z|x) [log pθ(x|z)]− KL (qϕ(z|x)||pθ(z)) , (9)
where the inequality is by Jensen’s inequality and ϕ is the parameter of the inference model q(z|x). Note that in this general case x stands for observational variables and z for latent variables.

heading: 3.4.1 VARIATIONAL LOWER BOUND
text: Directly applying the lower bound in Inequality (9) to our model, we obtain
log pθ(x⃗, a⃗, r⃗) ≥ E qϕ(z⃗,u|x⃗,⃗a,r⃗) [log pθ(x⃗, a⃗, r⃗|z⃗, u)]− KL (qϕ(z⃗, u|x⃗, a⃗, r⃗)||pθ(z⃗, u))
= L(x⃗, a⃗, r⃗; θ, ϕ). (10)
Using the Markov property of our model, the full distribution can be factorized in the following way:
pθ(x⃗, a⃗, r⃗, z⃗, u) = p(u)p(z1) [∏T t=1 p(xt|zt, u)p(at|zt, u)p(rt+1|zt, at, u) ] [∏T t=2 p(zt|zt−1, at−1) ] .
(11)
In addition, for simplicity’s sake, we also have the factorization assumption for the posterior approximation:
qϕ(z⃗, u|x⃗, a⃗, r⃗) = q(u|x⃗, a⃗, r⃗)q(z1|x⃗, a⃗, r⃗) ∏T
t=2 q(zt|zt−1, x⃗, a⃗, r⃗). (12)
3An intuition can be found in Appendix F.2.
Combining Equation (10), (11) and (12) yields: log pθ(x⃗, a⃗, r⃗) ≥L(x⃗, a⃗, r⃗; θ, ϕ)
= ∑T
t=1 E
zt∼q(zt|zt−1,x⃗,⃗a,r⃗) u∼q(u|x⃗,⃗a,r⃗)
[log p(xt|zt, u) + log p(at|zt, u) + log p(rt+1|zt, at, u)]
− KL (q(u|x⃗, a⃗, r⃗)||p(u))− KL (q(z1|x⃗, a⃗, r⃗)||p(z1)) − ∑T
t=2 E
zt−1∼ q(zt−1|zt−2,x⃗,⃗a,r⃗)
[KL (q(zt|zt−1, x⃗, a⃗, r⃗)||p(zt|zt−1, at−1))] , (13)
where we omit subscripts θ and ϕ, and a more detailed derivation can be found in Appendix B. Obviously, Equation (13) is differentiable with respect to the parameters of the model θ and ϕ. Using the reparametrization trick (Kingma & Welling, 2013), we can directly apply backpropagation to update the parameters.

heading: 3.4.2 INFERENCE MODEL
text: From the factorization form in Equation (12), we can see that there are two types of inference models: q(u|x⃗, a⃗, r⃗) and q(z⃗|x⃗, a⃗, r⃗). Similar to the generative model in Section 3.2, we also parametrize both of them as Gaussian:
q(u|x⃗, a⃗, r⃗) = N ( u|µ̂ut , σ̂ut 2 ) ; µ̂ut = f9(x⃗, a⃗, r⃗) σ̂ u t 2 = f10(x⃗, a⃗, r⃗); (14)
q(z⃗|x⃗, a⃗, r⃗) = N ( z⃗|µ̂zt , σ̂ z t 2 ) ; µ̂zt = f11(x⃗, a⃗, r⃗) σ̂ z t 2 = f12(x⃗, a⃗, r⃗). (15)
In fact, as shown in Equation (12), q(z⃗|x⃗, a⃗, r⃗) can be further factorized as the product of q(zt|zt−1, x⃗, a⃗, r⃗) for t = 1, . . . , T . Taking a closer look at this term, based on the Markov property of our model, we have zt |= x1, . . . , xt−1, a1, . . . , at−2, r2, . . . , rt|zt−1, and then the term can be simplified as follows,
q(zt|zt−1, x⃗, a⃗, r⃗) = q(zt| zt−1, at−1︸ ︷︷ ︸ past , xt, at, rt+1︸ ︷︷ ︸ current , xt+1, at+1, rt+2, . . . , xT , aT , rT+1︸ ︷︷ ︸ future ). (16)
Equation (16) tells us that zt depends on zt−1 and all the current and future observed data (x⃗, a⃗, r⃗). Meanwhile, the conditional independence above means that zt−1 contains all the historical data. Therefore, it is natural to calculate zt based on the whole sequence of data, which is exactly what recurrent neural networks (RNNs) do. Inspired by (Krishnan et al., 2015; 2017), we similarly choose a bi-directional LSTM (Zaremba & Sutskever, 2014) to parameterize f11 and f12 in Equation (15). Considering Equation (14) has the same structure as Equation (15), f9 and f10 are parameterized by a bi-directional LSTM as well. More details about the architecture can be found in Appendix D.
Note that for the task of sample predictions, that is, at any time step t, given a new xt, we require to know at and rt+1 before inferring the distribution over zt. Hence, we need to introduce two auxiliary distributions, denoted by red and blue dashed lines in Figure 1(b), to help conduct counterfactual reasoning (i.e., sample prediction on unseen xt). To be more precise, we have
q(at|xt) = N ( µ = µ̂at , σ 2 = σ̂at 2 ) µ̂at = f13(xt) σ̂ a t 2 = f14(xt); (17)
q(rt+1|xt, at) = N ( µ = µ̂rt , σ 2 = σ̂rt 2 ) µ̂rt = f15(xt, at) σ̂ r t 2 = f16(xt, at), (18)
where f13, f14, f15, and f16 are also parameterized by neural networks. To estimate the parameters, we will add these two extra terms in the variational lower bound (Equation (13)):
LDRL = L+ ∑T
t=1 (log q(at|xt) + log q(rt+1|xt, at)) . (19)

heading: 3.5 DECONFOUNDING RL ALGORITHMS
text: Now we have all the building blocks for DRL algorithms. Once our model is learned from the observational data, it can be directly used as a dynamic environment like those in OpenAI Gym (Brockman et al., 2016). We can exploit the learned model to generate rollouts for policy optimization. In practice, Equation (8) is approximated using the Monte Carlo method as follows:
p(rt|zt = z, do(at = a)) = 1
N ∑N i=1 p(rt|zt = z, at = a, ui) ui ∼ p(u), (20)
where N is the number of samples from the prior p(u). In the presence of observational data, a better estimate could be given on the basis of the samples drawn from the approximate posterior q(u|x⃗, a⃗, r⃗) which we compute through the inference network presented in Section 3.4.2. On the basis of our deconfounding reward function, it is straightforward to extend traditional RL algorithms to their corresponding deconfounding version. In this paper, we select one representative of them: the Actor-Critic method (Sutton et al., 1998), but it is straightforward to be applied to other algorithms.
Deconfounding Actor-Critic Methods The actor-critic method is a policy-based method directly parameterizing the policy π(a|z; θ), which aims to reduce the variance of the estimate of the policy gradient by subtracting a learned function of the state b(z), known as a baseline, from the return. The learned value function V (z;ϕ) is commonly used as the baseline. Taking into consideration that the return is the estimate of Q(z, a;ϕQ) and b(z) is the estimate of V (z;ϕV ), the gradient of the actor-critic loss function at step time t is given by
∇J(θ) = Eπ [(Q(zt, at;ϕQ)− V (zt;ϕV ))∇θ lnπ(at|zt; θ)] , (21)
where Q(zt, at;ϕQ)−V (zt;ϕV ) used to be seen as an estimate of the advantage of action at in state zt. In practice, Q(zt, at;ϕQ) is usually replaced with one-step return, that is, rt+1 + V (zt+1;ϕV ). Importantly, in deconfounding actor-critic methods, we use rt+1 ∼ p(rt+1|zt, do(at)) (Equation (20)), as opposed to rt+1 ∼ p(rt+1|zt, at) in vanilla actor-critic methods. The details about training can be found in Appendix E.

heading: 4 EXPERIMENTAL RESULTS
text: It is widely acknowledged that evaluating approaches dealing with confounding is always challenging due to lack of groundtruth and benchmark datasets. Besides, little work has been done before in DRL, which renders evaluating such algorithms in this respect much harder. Therefore, we first develop benchmark datasets for evaluation of our algorithms, primarily by revising the MNIST dataset (LeCun et al., 1998) and two environments in OpenAI Gym (Brockman et al., 2016): CartPole and Pendulum. We then evaluate our model as well as compare two proposed deconfounding algorithms to their corresponding vanilla versions on the benchmark datasets we developed.

heading: 4.1 IMPLEMENTATION DETAILS
text: We used Tensorflow (Abadi et al., 2016) for the implementation of our model and DRL algorithms. Optimization was done with Adam (Kingma & Ba, 2014). Unless stated otherwise, the setting of all the hyperparameters and architectures of the neural networks we adopted in this paper can be found in Appendix J.
To verify how good the learned model is, we performed two types of tasks: reconstruction and counterfactual reasoning. The reconstructions were performed by feeding the input sequence into the learned inference network, and then sampling from the resulting posterior distribution according to Equation (15), and finally feeding those samples into the generative network described in Equation (3). The counterfactual reasoning, that is, predicting xt+1 given xt we have not seen in the training set and a model trained in the training set, were executed through four steps: 1) Given an xt unseen in the training set, we estimate at and rt+1 based on Equation (17) and (18); 2) Once we have xt, at, and rt+1, it is easy to estimate zt from Equation (15); 3) Using the estimated zt and at, we can directly compute zt+1 from Equation (6); (4) The final step is to reconstruct xt+1 from zt+1 and u according to Equation (3). Repeating the four steps, we can counterfactually reason out a sequence of data.
To evaluate the confounder u, there are also two scenarios. The easy one is that, given a sequence of observational data (x⃗, a⃗, r⃗), it is obvious to estimate u from Equation (14). The more challenging one is to calculate u given only xt at any time step. Following the same steps used in the task of counterfactual reasoning, we first compute at and rt+1 based on Equation (17) and (18), and then estimate u through Equation (14).

heading: 4.2 CONFOUNDING DATASETS
text: Due to the space limit, we only introduce the Confounding Pendulum dataset in this section. Confounding CartPole and Confounding MNIST are devised in the same way and can be found in Appendix H.1 and Appendix H.2, respectively.
Confounding Pendulum Motivated by (Krishnan et al., 2015) where the authors synthesized a dataset mimicking the healthcare data under harsh conditions (e.g., noisy laboratory measurements, surgeries and drugs affected by patient age and gender, etc.), as well as considering our focus on RL tasks, we revised the original Pendulum in OpenAI Gym (Brockman et al., 2016) to develop a confounding Pendulum dataset in which a binary confounder is introduced. More specifically, we select 100 different screen images of Pendulum to create a synthetic dataset where actions are joint effort with the value of between −2 and 2 4. First, 20% bit-flip noise are added to each image , and then, based on a random policy but confounded by a binary factor (will be described soon), actions are performed on each noisy image for five time steps, which produces a large number of 5-step sequences of noisy images. To each generated sequence, exactly one sequence of three consecutive squares (2× 2 in pixel) is superimposed with the top-left corner of the images in a random starting location. The squares within the sequences are intended to be analogous to seasonal flu or other ailments that a patient could exhibit that are independent of the actions and which last several time steps (Krishnan et al., 2015). We aim to show that our model can learn long-range patterns, which plays an important role in medical applications. We treat such generated sequences of images as the observations x⃗. The training/validation/test set respectively comprises 140000/28000/28000 sequences of length five.
Now we are arriving at the key stage: how to define a reasonable relationship between the confounder, action, and reward. For simplicity of notation, we denote the confounder by u, action by a, and reward by r. u is a binary variable mimicking the socio-economic status (i.e., the rich and the poor) . The range of actions a ∈ [−2, 2] is grouped into two categories T1 (i.e., 1 ≤ |T1| ≤ 2) and T2 (i.e., 0 ≤ |T2| ≤ 1) representing different treatments (i.e., T1 is more valid and T2). How to choose the treatment depends on u. The reward is defined as follows,
r = ro + rc, (22)
where ro is the original reward in Pendulum, which is a function of a 5, and rc is the extra reward caused by both u and a, where rc comes from a two Gaussian mixture
rc ∼ ∑
rc={R1,R2} πrcN (µrc , σ2), ∑ rc={R1,R2} πrc = 1, πrc=R1 , πrc=R2 ≥ 0, (23)
with σ and µrc fixed and πrc determined by both u and a (see more in Appendix H.3). Obviously, in the definition above, r is a function of a and u. It is worth noting that u has an influence on xt through at−1 and zt, meaning that xt contains some piece of information about u and therefore can be viewed as its proxy variable. In our case, for the sake of generality and practice, we assume the influence between the confounder, action and reward is stochastic. Take the case of kidney stones for example, even though treatment a is more valid than treatment b, there are still some patients choosing treatment b in each category, but with different probabilities. All the details can be found in Appendix H.3, where a straightforward analogy is provided as well.

heading: 4.3 PERFORMANCE ANALYSIS OF THE DECONFOUNDING MODEL
text: In this section, to demonstrate the validity of our deconfounding model, denoted by Mdecon, we compare with the original model (i.e., the model similar to that shown in Figure 1(b) but without the confounder u), denoted by Morin. We train Mdecon by optimizing Equation (19) but train Morin by a little different loss function excluding the confounder u whose full derivation can be found in Appendix C. Both models are separately trained in a batch manner on the training set (i.e., 140K sequences of length five of images) of the confounding dataset. Afterwards, following the steps depicted in Section 4.1, we use each trained model to perform the reconstruction task on the training set, and both reconstruction and counterfactual reasoning tasks on the testing set (i.e., 28K sequences of length five of images).
Figure 2 presents a comparison of Mdecon and Morin, in terms of reconstruction and counterfactual reasoning on the confounding Pendulum dataset. The second row is based on Mdecon (Figure 1(b)), whilst the top row comes from Morin. It is evident that the results generated by the deconfounding model is superior to those produced by the model not taking into account the confounder. To be more specific, as shown in the zoom of samples on the bottom row, Morin generates more blurry images than Mdecon, because, without modelling the confounder u, Morin is forced to average over its
4More details can be found at https://github.com/openai/gym/wiki/Pendulum-v0. 5In fact, here ro is a function of both a and a state, and we mention only a to emphasize that the confounder
affects the action.
multiple latent states resulting in more blurry samples. Likewise, we can attain the same conclusion from the samples respectively produced by Morin and Mdecon on the confounding MNIST dataset, as shown in Figure 7 of Appendix I.
In addition, looking closely at the squares on the generated digit samples (colored in yellow box), it is obvious to observe that Morin will generate non-consecutive squares in the task of counterfactual reasoning, which does not really make sense because only consecutive patterns appear in the training set. In contrast, this case does not take place on the samples from Mdecon, showing that our deconfounding model is able to cope with long-range patterns.
It is worth noting that, as shown in Figure 8 of Appendix I, by visualizing the 2-dimensional confounder u, we can discern that although the prior distribution of the confounder is assumed to be a unit Gaussian distribution, the model still can learn two obvious clusters from the data because it is originally a binary variable. It demonstrates that our model has an advantage in learning confounders even if the assumed prior over them were not that accurate.

heading: 4.4 COMPARISON OF RL ALGORITHMS
text: In this section, we will evaluate the proposed deconfounding actor-critic (AC) method by comparing with its vanilla version on the confounding Pendulum dataset. In the vanilla AC method, given a learned Morin, we optimize the policy by calculating the gradient presented in Equation (21) on the basis of the trajectories/rollouts generated through Morin. Equation (21) involves two functions: V (zt;ϕV ) and π(at|zt; θ), whose parameters can be found in Appendix J. It is worth noting that, in this vanilla case, each reward rt+1 is produced from the conditional distribution p(rt+1|zt, at). In contrast, the proposed deconfounding AC method is built on Mdecon. Although the same gradient method (Equation (21)) is utilized to optimize the policy, we base the deconfounding AC approach on the different trajectories/rollouts generated by Mdecon in which each reward rt+1 relies on the interventional distribution p(rt+1|zt, do(at)) computed using Equation (20). In the training phase, for both vanilla AC and deconfounding AC, we run a respective experiment over 1500 episodes with 200 time steps each. In order to reduce non-stationarity and to decorrelate updates, the generated data is stored in an experience replay memory and then randomly sampled in a batch manner (Mnih et al., 2013; Riedmiller, 2005; Schulman et al., 2015; Van Hasselt et al., 2016). In each episode, we summarize all the rewards and further average the sums over a window of 100 episodes to obtain a more smooth curve. As shown in Figure 3(a), obviously our deconfounding AC algorithm performs significantly better than the vanilla AC algorithm in the confounded environment.
In the testing phase, we first randomly select 100 samples from the testing set, each starting a new episode, and then use the learned policies to perform reasoning over 200 time steps as we did during the training time. From the resulting 100 episodes, we plot the total reward for each, shown in Figure 3(b), and compute the percentage of the optimal action T1 in each episode, presented in Figure 3(c). It is worth noting that Figure 3(c) tells us that in each episode our deconfounding AC almost chooses the optimal action at each time step, whilst the vanilla AC makes a wrong decision for more than half time.

heading: 5 RELATED WORK
text: Krishnan et al. (2015; 2017) used deep neural networks to model nonlinear state space models and leveraged a structured variational approximation parameterized by recurrent neural networks to mimic the posterior distribution. Levine (2018) reformulated RL and control problems to probabilistic inference, which allows us to bear a large pool of approximate inference methods, and flexibly extend the model. Raghu et al. (2017a;b) exploited continuous state-space models and deep RL to deduce treatment policies for septic patients from observational data. Gottesman et al. (2018) discussed some issues of evaluating RL algorithms in observational health setting. However, all the work mentioned above did not take into account confounders in their models.
Louizos et al. (2017) attempted to learn individual-level causal effects from observational data using variational auto-encoder to estimate the unknown confounder given a causal graph in then nontemporal setting. Paxton et al. (2013) developed predictive models based on electronic medical records without using causal inference. Saria et al. (2010) proposed a nonparametric Bayesian method to analyze clinical temporal data. Soleimani et al. (2017) represented the treatment response curves using linear time-invariant dynamical systems which provides a flexible approach to modeling response over time. Although the latter two work modeled the sequential data, they both do not exploit RL or causal inference.
Bareinboim et al. (2015) considered the problem of bandits with unobserved confounders, which is one quite simple RL setting without state transitions. Sen et al. (2016) and Ramoly et al. (2017) further studied contextual bandits with latent confounders. Forney et al. (2017) circumvented some problems caused by unobserved confounders in Multi-Armed Bandit by counterfactual-based decision-making. Zhang & Bareinboim (2017) leveraged causal inference to tackle the problem of transferring knowledge across bandit agents. However, all these methods are based on the bandit problem, a simplified version of RL, instead of the full RL problem.
In fact, as far as we are concerned, this is the first attempt to build a bridge between confounding and the full RL problem, and this is also one of few research papers aiming at understanding the connections between causal inference and full RL.

heading: 6 CONCLUSION AND FUTURE WORK
text: To address the confounding issue in RL, we introduced a general formulation, namely deconfounding reinforcement learning. On the basis of the proposed formulation, we presented deconfounding variants of actor-critic methods and showed their superior performance on confounding datasets that we created by revising OpenAI Gym and MNIST. In the future, we will collaborate with hospitals and apply our approach to real-world medical datasets. We also hope that our work will stimulate further investigation of connections between causal inference and RL.

heading: D BI-DIRECTIONAL LSTM
text: In our inference model, we use a similar architecture of bi-directional LSTM to that in (Krishnan et al., 2017). Apart from different inputs, the main difference is to introduce at−1 to computing the
combined hidden feature as shown in the following formula.
0⃗ ẑ1 ẑ2 ẑ3
function (µ1,Σ1) (µ2,Σ2) (µ3,Σ3)
Backward LSTM hright1 h right 2 h right 3
Forward LSTM hleft1 h left 2 h left 3
x1, a1, r2 x2, a2, r3 x3, a3, r4
hcombined = 1
4
( tanh(Wz · zt−1 + bz) + tanh(Wa · at−1 + ba) + hleftt + h right t ) µt = Wµ · hcombined + bµ σ2t = softplus (Wσ2 · hcombined + bσ2)

heading: E TRAINING DETAILS
text: As mentioned in the main body, DRL consists of two steps: learning a deconfounding model and optimizing a policy based on the learned deconfounding model. To be more specific, in step 1, given the observational data (x⃗, a⃗, r⃗), we learn the deconfounding model by optimizing the variational lower bound as presented in Equation (19). Once the deconfounding model is learned, we know the state transition function p(zt|zt−1, at−1) and can also calculate the deconfounding reward function p(rt|zt, do(at)) according to Equation (20). In step 2, we treat the learned deconfounding model as a RL environment like CartPole in OpenAI Gym, and directly exploit it to generate trajectories/rollouts through the state transition function and the deconfounding reward function. On the basis of the generated trajectories/rollouts, we can train the policy network using Equation (21).

heading: F EXAMPLE OF KIDNEY STONES
text: F.1 EXAMPLE OF A PROXY VARIABLE
For convenience, we directly move Figure 1 of (Louizos et al., 2017) here, as shown in Figure 4.
F.2 INTUITION
Take for example the case of kidney stones, in the existence of the confounder (i.e., the size of stones), one prefers treatment b when considering the overall probability of recovery, whilst one chooses with more probabilities treatment a when investigating the recovery rate in each size. The correct answer is treatment a in this example. The do-operator in Equation (2) of Section 2.3 provides a formal approach to seeking the correct solution to such confounded problems, which can not be addressed within the traditional RL framework, because the traditional framework only involves the conditional probability (e.g., the overall probability of recovery) instead of the interventional probability as do-operator does.

heading: G ANALYSIS OF THE DECONFOUNDING MODEL
text: At each time step t, from the deconfounding model presented in Figure 1(b), we can extract two 4- tuple components, centering at zt (Figure 5) and u (Figure 6), respectively. It is apparent to observe that both Figure 5 and Figure 6 share the exact same structure with Figure 4.

heading: H CONFOUNDING DATASETS
text: H.1 CONFOUNDING CARTPOLE
The confounding CartPole dataset can be implemented in the exact same manner as Confounding Pendulum, except for the action which is originally binary and can be naturally divided into two categories.
H.2 CONFOUNDING MNIST
We follow the same setting to develop a confounding MNIST dataset, but the definitions of action and the original reward term are different. Similar to the Healing MNIST dataset (Krishnan et al., 2015), rotations of digit images are encoded as the actions a⃗ (−45 ≤ a ≤ 45) which, according to the binary confounder u, is divided into two categories (22.5 ≤ |a| ≤ 45) and (0 ≤ |a| < 22.5). The original reward term ro is defined as the minus degree between the upright position and the position the digit rotates to. For example, if the digit rotates to the position of 3 o’clock or 9 o’clock, then both rewards are −90.
H.3 DETAILS ABOUT CONFOUNDING SIMULATIONS
The probabilities between the confounder, action and reward are presented as follows,
p(u = 0) = 0.8, p(u = 1) = 0.2; (24) p(a = T1|u = 0) = 0.24, p(a = T2|u = 0) = 0.76; (25) p(a = T1|u = 1) = 0.77, p(a = T2|u = 1) = 0.23; (26) p(rc = R1|a = T1, u = 0) = 0.93, p(rc = R2|a = T1, u = 0) = 0.07; (27) p(rc = R1|a = T2, u = 0) = 0.87, p(rc = R2|a = T2, u = 0) = 0.13; (28) p(rc = R1|a = T1, u = 1) = 0.73, p(rc = R2|a = T1, u = 1) = 0.27; (29) p(rc = R1|a = T2, u = 1) = 0.69, p(rc = R2|a = T2, u = 1) = 0.31; (30) µrc=R1 = −1, µrc=R2 = −200, σ = 2. (31)
Note that, in our model rc is a function of u, at, and zt. We omit the notation of the state in the conditional probability of rc because zt is fixed at each time step, which does not affect the probability. To help readers understand this setting, we can make a straightforward analogy. More specifically, the confounder u stands for the socio-economic status where u = 0 represents the poor and u = 1 the rich. Treatment T1 is more expensive but more valid than Treatment T2. R1 means a healthier feedback/recovery than R2. Equation (24) says that the poor are much more than the rich (it is indeed in reality). Equation (25) and Equation (26) tell us the fact that the poor tend to choose the cheaper but less valid treatment T2 whilst the rich prefer the more expensive but more valid treatment T1, which also makes sense in the real world. Finally, Equations (27)-(30) reveal that T1 is a better treatment than T2 within both poor and rich population.

heading: I ADDITIONAL EXPERIMENTAL RESULTS
text: As shown in Figure 7 and Figure 8.

heading: J EXPERIMENTAL SETTINGS
text: As shown in Table 1 and Table 2.
Fu nc
tio n
A rc
hi te
ct ur e D ec on fo un di ng M od el f 1 ,f 2 FC 5 1 2 → C on v 7 ,3 2 → C on v 1 4 ,1 6 → C on
v 2 8 ,1 →
FC 7 8 4 → {s
ig m
oi d,
so ft
pl us }
f 3 ,f
4 {F
C 1 0 0 ,F
C 1 0 0 } →
FC 2 0 0 →
FC 2 0 0 →
FC 1 → {t a n h ,s
of tp
lu s}
f 5 ,f
6 {F
C 1 0 0 ,F
C 1 0 0 ,F
C 1 0 0 } →
FC 1 0 0 →
FC 1 0 0 →
FC 1 → {s
ig m
oi d,
so ft
pl us }
f 7 ,f
8 {F
C 1 0 0 ,F
C 1 0 0 } →
FC 1 0 0 →
FC 1 0 0 →
FC 5 0 → {N
on e,
so ft
pl us }
f 9 ,f
1 0
{[ C
on v 5 ,1 6 →
C on
v 5 ,3 2 →
C on
v 5 ,3 2 →
FC 1 0 0 ],
FC 1 0 0 ,F
C 1 0 0 } →
FC 1 0 0 →
FC 1 0 0 →
L ST
M 1 0 0 ,5 →
FC 1 → {N
on e,
so ft
pl us }
f 1 1 ,f
1 2
{[ C
on v 5 ,1 6 →
C on
v 5 ,3 2 →
C on
v 5 ,3 2 →
FC 1 0 0 ],
FC 1 0 0 ,F
C 1 0 0 } →
FC 1 0 0 →
FC 1 0 0 →
L ST
M 1 0 0 ,5 →
FC 5 0 → {N
on e,
so ft
pl us }
f 1 3 ,f
1 4
C on
v 5 ,1 6 →
C on
v 5 ,3 2 →
C on
v 5 ,3 2 →
FC 1 → {t a n h ,s
of tp
lu s}
f 1 5 ,f
1 6
{[ C
on v 5 ,1 6 →
C on
v 5 ,3 2 →
C on
v 5 ,3 2 →
FC 1 0 0 ],
FC 1 0 0 } →
FC 1 0 0 →
FC 1 → {s
ig m
oi d,
so ft
pl us }
D ec
on fo
un di
ng A C V (z t ;ϕ V ) FC 3 0 0 → FC 3 0 0 → {N
on e,
so ft
pl us }
π (a t |z t ;θ )
FC 3 0 0 →
FC 3 0 0 → {t a n h ,s
of tp
lu s}
Ta bl
e 2:
A rc
hi te
ct ur
es fo
r de
co nf
ou nd
in g
re in
fo rc
em en
tl ea
rn in
g. H
er e
FC k
st an
ds fo
r a
fu lly
-c on
ne ct
ed la
ye r
w ith
k un
its ,C
on v k ,n
fo r
a co
nv ol
ut io
n la
ye r
w ith
n fil
te rs
of si
ze k ×
k ,L
ST M n ,t
fo ra
L ST
M la
ye rr
ol lin
g ou
tf or
t st
ep s
w ith
la te
nt si
ze of
n ,{ ·}
fo rt
he pa
ra lle
lo pe
ra to
rs ,a
nd [·]
fo rt
he se
qu en
tia lo
pe ra
to rs
.I n
de fa
ul t,
FC k
an d
C on
v k ,n
ar e
fo llo
w ed
by a
so ft
pl us
ac tiv
at io
n la
ye r
an d
a ba
tc h-
no rm
la ye
r, w
hi ch
ar e
om itt
ed he
re fo
r si
m pl
ic ity
. N
ot e
th at
,i n
ou r
se tti
ng ,t he tw o fu nc tio ns in ea ch pa ir {f 2 i− 1 ,f 2 i} i= 1 ,. .. ,8 sh ar e th e sa m e pa ra m et er s ex ce pt fo rt he la st la ye r.

Review summaries:
The paper addresses an important and often overlooked issue in off-policy reinforcement learning - the possibility of confounding between the agent's actions and the rewards. This is a subject which has been exhaustively explored in the causal inference literature, and the authors are very correct in suggesting that it should be incorporated into the world of reinforcement learning.  Specifically they propose a generative model with a global latent confounder that is inferred using a variational autoencoder architecture.