Causal Reinforcement Learning using Observational and Interventional Data

1 INTRODUCTION . As human beings , a key ingredient in our learning process is experimentation : we perform actions in our environment and we measure their outcomes . Another ingredient , maybe less understood , is observation : we observe the behaviour of others acting and evolving in the environment , be it people , animals , or even plants . It is well-known that observation alone is not sufficient to infer how our environment works , or more precisely to predict the outcome of our actions , especially when the behaviours we observe depend on hidden information1 . And yet a whole field of science , astronomy , heavily relies on the observation of celestial bodies in the sky , on which experimentation is virtually impossible . So which role exactly does observation play during learning ? And in particular , how do we combine observation and experimentation ? In the context of reinforcement learning ( RL ) , a related question is the combination of offline data , resulting from observations , with online data resulting from experimentation , in order to improve the performance of a learning agent . In the Markov Decision Process ( MDP ) setting , where the agent observes the entire state of the environment , the answer is straightforward and practical solutions exist , leading to the fastly growing field of offline reinforcement learning [ 17 ; 18 ] where large databases of demonstrations can be efficiently leveraged . In the more general Partially-Observable MDP ( POMDP ) setting however , the question turns out to be much more challenging . A typical example is in the context of medicine , where offline data is collected from physicians who may rely on information absent from their patient ’ s medical records , such as their wealthiness or their lifestyle . Suppose that wealthy patients in general get prescribed specific treatments by their physicians , because they can 1Simply put , correlation does not imply causation . Or , citing Pearl [ 24 ] , “ behind every causal conclusion there must lie some causal assumption that is not testable in observational studies ” . afford it , while being less at risk to develop severe conditions regardless of their treatment , because they can also afford a healthier lifestyle . This creates a spurious correlation called confounding , and will cause a naive recommender system to wrongly infer that a treatment has positive health effects . A second example is in the context of autonomous driving , where offline data is collected from human drivers who have a wider field of vision than the camera on which the robot driver relies . Suppose human drivers push the brakes when they see a person waiting to cross the street , and only when the person walks in front of the car it enters the camera ’ s field of vision . Then , again , a naive robot might wrongly infer from its observations that whenever brakes are pushed , a person appears in front of the car . Suppose now that the robot ’ s objective is to avoid collisions with pedestrians , they it might get regrettably reluctant to push the brakes . Of course , in both those situations , the learning agent will eventually infer the right causal effects of its actions if it collects enough online data from its own interactions . However , in both those situations also , performing many interventions for the sole purpose of seeing what happens is not really realistic , while collecting offline data by observing the behaviour of human agents is much more affordable . In this paper we study the question of combining offline and online data under the Partially-Observable Markov Decision Process ( POMDP ) setting , by importing tools and ideas from the well-established field of causality [ 23 ] into the model-based RL framework . Our contribution is three-fold : 1 . We formalize model-based RL as a causal inference problem using the framework of docalculus [ 25 ] , which allows us to reason formally about online and offline scenarios in an intuitive manner ( Section 3 ) . 2 . We present a generic method for combining offline and online data in model-based RL ( Section 4 ) , with a formal proof of correctness even when the offline policy relies on privileged hidden information ( confounding variable ) , and a proof of efficiency in the asymptotic case ( with respect to using online data only ) . 3 . We propose a practical implementation of our method , and illustrate its effectiveness on three experiments with synthetic toy problems ( Section 6 ) . While our proposed method can be formulated outside of the framework of do-calculus , in this paper we hope to demonstrate that it offers a principled and intuitive tool to reason about model-based RL . By relating common concepts from RL and causality , we wish that our contribution will ultimately help to bridge the gap between the two communities . 2 BACKGROUND . 2.1 NOTATION . In this paper , upper-case letters in italics denote random variables ( e.g . X , Y ) , while their lower-case counterpart denote their value ( e.g . x , y ) and their calligraphic counterpart their domain ( e.g. , x ∈ X ) . We consider only discrete random variables . To keep our notation uncluttered , with a slight abuse of notations and use p ( x ) to denote sometimes the event probability p ( X = x ) , and sometimes the whole probability distribution of X , which should be clear from the context . In the context of sequential models we also distinguish random variables with a temporal index t , which might be fixed ( e.g. , o0 , o1 ) , or undefined ( e.g. , p ( st+1|st , at ) denotes at the same time the distributions p ( s1|s0 , a0 ) and p ( s2|s1 , a1 ) ) . We also adopt a compact notation for sequences of contiguous variables ( e.g. , s0→T = ( s0 , . . . , sT ) ∈ ST+1 ) , and for summations over sets ( ∑ x∈X ⇐⇒ ∑X x ) . We assume the reader is familiar with the concepts of conditional independence ( X ⊥⊥ Y | Z ) and probabilistic graphical models based on directed acyclic graphs ( DAGs ) , which can be found in most introductory textbooks , e.g . Pearl [ 22 ] ; Studeny [ 29 ] ; Koller and Friedman [ 15 ] . In the following we will use do-calculus to derive formal solutions to model-based RL in various POMDP settings . We refer the reader to Pearl [ 25 ] for a thorough introduction , and give a description of rules R1 , R2 and R3 used in our derivations in the appendix ( Section A.1 ) . 2.2 PARTIALLY-OBSERVABLE MARKOV DECISION PROCESS . We consider Partially-Observable Markov Decision Processes ( POMDPs ) of the form M = ( S , O , A , pinit , pobs , ptrans , r ) , with hidden states s ∈ S , observations o ∈ O , actions a ∈ A , ini- tial state distribution pinit ( s0 ) , state transition distribution ptrans ( st+1|st , at ) , observation distribution pobs ( ot|st ) , and reward2 function r : O → R. For simplicity we assume episodic tasks with finite horizon H . We further denote a complete trajectory τ = ( o0 , a0 , . . . , oH ) , and for convenience we introduce the concept of a history at time t , ht = ( o0 , a0 , . . . , ot ) . A common control scenario for POMDPs is when actions are decided based on all the available information from the past . We call this the standard POMDP setting . The control mechanism can be represented as a stochastic policy π ( at|ht ) , which together with the POMDP dynamics pinit , pobs and ptrans defines a probability distribution over trajectories τ , pstd ( τ ) = S|τ|+1∑ s0→|τ| pinit ( s0 ) pobs ( o0|s0 ) |τ |−1∏ t=0 π ( at|ht ) ptrans ( st+1|st , at ) pobs ( ot+1|st+1 ) . This whole data-generation mechanism can be represented visually as a DAG , represented in Figure 1 . A key characteristic in this setting is that At ⊥⊥ St | Ht is always true , that is , every action is independent of the current state given the history . 2.3 MODEL-BASED RL . Assuming the objective is the long-term reward , the POMDP control problem formulates as : π ? = argmax π E τ∼pstd |τ |∑ t=0 r ( ot ) . ( 1 ) Model-based RL relies on the estimation of the POMDP transition model pstd ( ot+1|ht , at ) to solve ( 1 ) , which decomposes into two sub-problems : 1. learning : given a dataset D , estimate a transition model q̂ ( ot+1|ht , at ) ≈ pstd ( ot+1|ht , at ) ; 2. planning : given a history ht and a transition model q̂ , decide on an optimal action at . As we will see shortly , the transition model q̂ seeked by model-based RL is inherently causal [ 9 ] . In this work we consider only the first problem above , that is , learning the ( causal ) POMDP transition model from data . 3 MODEL-BASED RL AS CAUSAL INFERENCE . Decision problems , such as those arising in POMDPs , can naturally be formulated in terms of causal queries where actions directly translate into do statements . For example , given past information about the POMDP process , what will be the causal effect of an action ( intervention ) on future rewards ? 3 3.1 THE INTERVENTIONAL REGIME . In the interventional regime , we assume a datasetDint of episodes τ collected in the standard POMDP setting from an arbitrary decision policy π ( at|ht ) , Dint ∼ pinit , ptrans , pobs , π . Let us now adopt a causal perspective and reason in terms of interventions in the causal system , depicted in Figure 1 . Consider that we want to control the system , that is , replace π with π ? , in order to maximize a long-term outcome . Then , evaluating the effect of each action on the system is a causal inference problem . In order to decide on the best first action a0 given h0 = ( o0 ) , one must evaluate a series of causal queries in the form pstd ( o1|o0 , do ( a0 ) ) , then pstd ( o2|o0 , do ( a0 ) , o1 , do ( a1 ) ) , and so on , and finally using those causal distributions for planning by solving a Bellman equation . Conveniently , in the interventional regime , applying rule R2 of do-calculus on the causal DAG results 2Without loss of generality we consider the reward to be part of the observation ot to simplify our notation . 3A guiding example accompanying this section can be found in the appendix ( Section A.3 ) . pinit St ptrans St+1 ptrans pobs Ot pobs Ot+1 pobs At πprv At−1 πprv Figure 2 : Privileged POMDP setting . in those queries being trivially identifiable from pstd ( τ ) . In fact , those queries exactly boil down to the standard POMDP transition model that model-based RL seeks to estimate , pstd ( ot+1|o0→t , do ( a0→t ) ) = pstd ( ot+1|ht , at ) . ( 2 ) As such , model-based RL can be naturally reinterpreted in terms of causal inference . Also , a convenient property in this regime is that pstd ( ot+1|ht , at ) does not depend on the control policy π that was used to build the dataset Dint . The only requirement , in order to estimate transition probabilities for every ht , at combination , is that π has a non-zero chance to explore every action , that is , π ( at|ht ) > 0 , ∀at , ht . Then , an unbiased estimate of the standard POMDP transition model can be obtained simply via log-likelihood maximization : q̂ = argmax q∈Q Dint∑ τ |τ |−1∑ t=0 log q ( ot+1|ht , at ) . ( 3 ) In some situations it is very reasonable to assume an interventional regime , for example when it is known to hold by construction . This is the case with online RL data , as the learning agent itself explicitly controls the data-collection policy π ( at|ht ) . But it can also be the case with offline RL data , if one knows that the data-collection policy did not use any additional information besides the information available to the learning agent , ht . In Atari video games for example , it is hard to imagine a human player using any kind of privileged information related to the machine ’ s internal state st other than the video and audio outputs from the game .

Summary: This paper considers the model-based reinforcement learning (RL) problem by combining the offline and online data. The online data, i.e., the interventional data, is generated from the standard partially-observable Markov decision process (POMDP), while the offline data, i.e., the observational data, is generated from the privileged POMDP where the offline learner had the access to the state information (i.e., the unobserved confounder) to make an action. The authors proposed an augmented learning procedure to safely combine these two separate different sources and learn a more efficient policy. Their method is shown both theoretically and empirically better than not using offline data. My main concerns lie in their framework and assumptions, novelty compared with existing literature, and comparison studies.