Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

1 INTRODUCTION . Imagine that a month ago Alice had two job offers from companies a1 and a2 . She decided to join a1 because of the larger salary , in spite of an awkward feeling during the job interview . Since then she learned a lot about a1 and recently received information about a2 from a friend , prodding her now to imagine what would have happened had she joined a2 . Re-evaluating her decision in hindsight in this way , she concludes that she made a regrettable decision . She could and should have known that a2 was a better choice , had she only interpreted the cues during the interview correctly ... This example tries to illustrate the everyday human capacity to reason about alternate , counterfactual outcomes of past experience with the goal of “ mining worlds that could have been ” ( Pearl & Mackenzie , 2018 ) . Social psychologists theorize that such cognitive processes are beneficial for improving future decision making ( Roese , 1997 ) . In this paper we aim to leverage possible advantages of counterfactual reasoning for learning decision making in the reinforcement learning ( RL ) framework . In spite of recent success , learning policies with standard , model-free RL algorithms can be notoriously data inefficient . This issue can in principle be addressed by learning policies on data synthesized from a model . However , a mismatch between the model and the true environment , often unavoidable in practice , can cause this approach to fail ( Talvitie , 2014 ) , resulting in policies that do not generalize to the real environment ( Jiang et al. , 2015 ) . Motivated by the introductory example , we propose the Counterfactually-Guided Policy Search ( CF-GPS ) algorithm : Instead of relying on data synthesized from scratch by a model , policies are trained on model predictions of alternate outcomes of past experience from the true environment under counterfactual actions , i.e . actions that had not actually been taken , while everything else remaining the same ( Pearl , 2009 ) . At the heart of CF-GPS are structural causal models ( SCMs ) which model the environment with two ingredients ( Wright , 1920 ) : 1 ) Independent random variables , called scenarios here , summarize all aspects of the environment that can not be influenced by the agent , e.g . the properties of the companies in Alice ’ s job search example . 2 ) Deterministic transition functions ( also called causal mechanisms ) take these scenarios , together with the agent ’ s actions , as input and produce the predicted outcome . The central idea of CF-GPS is that , instead of running an agent on scenarios sampled de novo from a model , scenarios are inferred in hindsight from given off-policy data , and then evaluate and improve the agent on these specific scenarios using given or learned causal mechanisms ( Balke & Pearl , 1994 ) . CF-GPS can be regarded as a meta-algorithm that extends a given model-base RL algorithm by “ grounding ” or “ anchoring ” model-based predictions in inferred scenarios . As a result , this approach explicitly allows to trade-off historical data for model bias . We show empirically in a conceptually simple setting , where unknown initial states are inferred in hindsight and re-used to evalute to counterfactual actions , that this can mitigate model mismatch . CF-GPS differs substantially from standard off-policy RL algorithms based on Importance Sampling ( IS ) , where historical data is re-weighted with respect to the importance weights to evaluate or learn new policies ( Precup , 2000 ) . In contrast , CF-GPS explicitly reasons counterfactually about given off-policy data . Our main contributions are : 1 . We formulate model-based RL in POMDPs in terms of structural causal models , thereby connecting concepts from reinforcement learning and causal inference . 2 . We provide the first results , to the best of our knowledge , showing that counterfactual reasoning in structural causal models on off-policy data can facilitate solving non-trivial RL tasks . 3 . We show that two previously proposed classes of RL algorithms , namely Guided Policy Search ( Levine & Koltun , 2013 ) and Stochastic Value Gradient methods ( Heess et al. , 2015 ) , can be interpreted as counterfactual methods , opening up possible generalizations . The paper is structured as follows . We first give a self-contained , high-level recapitulation of structural causal models and counterfactual inference , as these are less widely known in the RL and generative model communities . In particular we show how to model POMDPs with SCMs . Based on this exposition , we first consider the task of policy evaluation and discuss how we can leverage counterfactual inference in SCMs to improve over standard model-based methods . We then generalize this approach to the policy search setting resulting in the CF-GPS algorithm . We close by highlighting connections to previously proposed algorithms and by discussing assumptions and limitations of the proposed method . 2 PRELIMINARIES . We denote random variables ( RVs ) with capital letters , e.g . X , and particular values with lower caps , e.g . x . For a distribution P over a vector-valued random variable X , we denote the marginal over Y ⊂ X by PY ( and density pY ) ; however we often omit the subscript if it is clear from the context , e.g . as in Y ∼ P . We assume the episodic , Partially Observable Markov Decision Process ( POMDP ) setting with states St , actions At and observations Ot , for t = 1 , . . . , T . For ease of notation , we assume that Ot includes the reward Rt . The undiscounted return is denoted by G = ∑T t=1Rt . We consider stochastic policies π ( at|ht ) over actions conditioned on observation histories Ht = ( O1 , A1 , . . . , At−1 , Ot ) . We denote the resulting distribution over trajectories T = ( S1 , O1 , A1 , . . . , AT−1 , ST , OT ) induced by running π in the environment with T ∼ Pπ and the corresponding density by pπ ( τ ) . 2.1 STRUCTURAL CAUSAL MODELS . Definition 1 ( Structural causal model ) . A structural causal model ( SCM ) M over X = ( X1 , . . . , XN ) is given by a DAG G over nodes X , independent noise RVs U = ( U1 , . . . , UN ) with distributions PUi and functions f1 , . . . , fN such that Xi = fi ( pai , Ui ) , where pai ⊂ X are the parents of Xi in G. An SCM entails a distribution P with density p over ( X , U ) . We also refer to U as scenarios and to fi as causal mechanisms . We give a ( broad ) definition of an intervention in an SCM . This also includes what is known as stochastic interventions or mechanism changes ( Korb et al. , 2004 ) which generalize atomic interventions ( Pearl , 2009 ) . Definition 2 ( Intervention in SCM ) . An intervention I in an SCMM consists of replacing some of the original fi ( pai , Ui ) with other functions f I i ( pa I i , Ui ) where pa I i are the parents in a new DAG GI . We denote the resulting SCM withMdo ( I ) with distribution P do ( I ) and density pdo ( I ) . SCM representation of POMDPs We can represent any given POMDP ( under a policy π ) by an SCMM over trajectories T in the following way . We express all conditional distributions , e.g . the transition kernel PSt+1|St , At , as deterministic functions with independent noise variables U , such as St+1 = fst ( St , At , Ust ) . This is always possible using auto-regressive uniformization , see Lemma 2 in the appendix . The DAG G of the resulting SCM is shown in fig . 1 . This procedure is closely related to the ‘ reparameterization trick ’ for models with location-scale distributions ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . We denote the distribution over T entailed by the SCM with Pπ and its density by pπ to highlight the role of π ; note the difference to the true environment distribution Pπ with density pπ . Running a different policy µ instead of π in the environment can be expressed as an intervention I ( π → µ ) consisting of replacing At = fπ ( Ht , Uat ) by At = fµ ( Ht , Uat ) . We denote the resulting model distribution over trajectories with P do ( I ( π→µ ) ) = Pµ ( analogously pµ ) . Intuition Here , we illustrate the main advantage of SCMs using the example of Alice ’ s job choice from the introduction . We model it as contextual bandit with feedback shown in fig . 1 . Alice has some initial knowledge given by the context Uc that is available to her before taking action A of joining company A = a1 or A = a2 . We model Alice ’ s decision as A = fπ ( Uc , Ua ) , where Ua captures potential indeterminacy in Alice ’ s decision making . The outcome O = fo ( A , Uc , Uo ) also depends on the scenario Uo , capturing all relevant , unobserved and highly complex properties of the two companies such as working conditions etc . Given this model , we can reason about alternate outcomes fo ( a1 , uc , uo ) and fo ( a2 , uc , uo ) for same the scenario uo . This is not possible if we only model the outcome on the level of the conditional distribution PO|A , Uc . 2.2 COUNTERFACTUAL INFERENCE IN SCMS . For an SCM over X , we define a counterfactual query as a triple ( x̂o , I , Xq ) of observations x̂o of some variablesXo ⊂ X , an intervention I and query variablesXq ⊂ X . The semantics of the query are that , having observed x̂o , we want to infer what Xq would have been had we done intervention I , while ‘ keeping everything else the same ’ . Counterfactual inference ( CFI ) in SCMs answers the query in the following way ( Balke & Pearl , 1994 ) : 1 . Infer the unobserved noise source U conditioned on the observations x̂o , i.e . compute p ( U |x̂o ) and replace the prior p ( U ) with p ( U |x̂o ) . Denote the resulting SCM byMx̂o . 2 . Perform intervention I on Mx̂o . This yields M do ( I ) x̂o , which entails the counterfactual distribution pdo ( I ) |x̂o ( x ) . Return the marginal pdo ( I ) |x̂o ( xq ) . Note that our definition explicitly allows for partial observations Xo ⊂ X in accordance with Pearl ( 2009 ) . A sampled-based version , denoted as CFI , is presented in Algorithm 1 . An interesting property of the counterfactual distribution pdo ( I ) |x̂o is that marginalizing it over observations x̂o yields an unbiased estimator of the density of Xq under intervention I . Lemma 1 ( CFI for simulation ) . Let observations x̂o ∼ p come from a SCMM with density p. Then the counterfactual density pdo ( I ) |x̂o is an unbiased estimator of pdo ( I ) , i.e . Ex̂o∼p [ pdo ( I ) |x̂o ( x ) ] = pdo ( I ) ( x ) The proof is straightforward and outlined in the Appendix A . This lemma and the marginal independence of the Ui leads to the following corollary ; the proof is given in the appendix . Corollary 1 ( Mixed counterfactual and prior simulation from an SCM ) . Assume we have observations x̂o ∼ p. We can simulate from M , under any intervention I , i.e . obtain unbiased samples from Mdo ( I ) , by first sampling values uCF for an arbitrary subset UCF ⊂ U from the posterior p ( uCF|x̂o ) and the remaining UPrior : = U\UCF from the prior p ( uPrior ) , and then computing X with noise u = uCF ∪ uPrior . The corollary essentially states that we can sample from the modelMI , by sampling some of the Ui from the prior , and inferring the rest from data x̂o ( as long as the latter was also sampled fromM ) . We will make use of this later for running a POMDP model on scenariosUst inferred from data while randomizing the action noise Uat . We note that the noise variables UCF from the posterior PUCF|x̂o are not independent anymore . Nevertheless , SCMs with non-independent noise distributions arising from counterfactual inference , denoted here by Mx̂o , are commonly considered in the literature ( Peters et al. , 2017 ) . Intuition Returning to Alice ’ s job example from the introduction , we give some intuition for counterfactual inference in SCMs . Given the concrete outcome ô , under observed context ûc and having joined company â = a1 , Alice can try to infer the underlying scenario uo ∼ p ( uo|a1 , ûc , ô ) that she experiences ; this includes factors such as work conditions etc . She can then reason counterfactually about the outcome had she joined the other company , which is given by fo ( a2 , ûc , uo ) . This can in principle enable her to make better decisions in the future in similar scenarios by changing her policy fπ ( A , Uc , Ua ) such that the action with the preferred outcome becomes more likely under ûc , uo . In particular she can do so without having to use her ( likely imperfect ) prior model over possible companies p ( Uo ) . She can use the counterfactual predictions discussed above instead to learn from her experience . We use this insight for counterfactual policy evaluation and search below .

Summary: Proposes Counterfactual Guided Policy Search (CF-GPS), which uses counterfactual inference from sampled trajectories to improve an approximate simulator that is used for policy evaluation. Counterfactual inference is formalized with structural causal models of the POMDP. The method is evaluated in partially-observed Sokoban problems. The dynamics model is assumed known, and a learned model maps observation histories to a conditional distribution on the starting state. CF-GPS outperforms model-based policy search and a "GPS-like" algorithm in these domains. GPS in MDPs is shown to be a particular case of CF-GPS, and a connection is also suggested between stochastic value gradient and CF-GPS.