Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search

1 INTRODUCTION . Imagine that a month ago Alice had two job offers from companies a1 and a2 . She decided to join a1 because of the larger salary , in spite of an awkward feeling during the job interview . Since then she learned a lot about a1 and recently received information about a2 from a friend , prodding her now to imagine what would have happened had she joined a2 . Re-evaluating her decision in hindsight in this way , she concludes that she made a regrettable decision . She could and should have known that a2 was a better choice , had she only interpreted the cues during the interview correctly ... This example tries to illustrate the everyday human capacity to reason about alternate , counterfactual outcomes of past experience with the goal of “ mining worlds that could have been ” ( Pearl & Mackenzie , 2018 ) . Social psychologists theorize that such cognitive processes are beneficial for improving future decision making ( Roese , 1997 ) . In this paper we aim to leverage possible advantages of counterfactual reasoning for learning decision making in the reinforcement learning ( RL ) framework . In spite of recent success , learning policies with standard , model-free RL algorithms can be notoriously data inefficient . This issue can in principle be addressed by learning policies on data synthesized from a model . However , a mismatch between the model and the true environment , often unavoidable in practice , can cause this approach to fail ( Talvitie , 2014 ) , resulting in policies that do not generalize to the real environment ( Jiang et al. , 2015 ) . Motivated by the introductory example , we propose the Counterfactually-Guided Policy Search ( CF-GPS ) algorithm : Instead of relying on data synthesized from scratch by a model , policies are trained on model predictions of alternate outcomes of past experience from the true environment under counterfactual actions , i.e . actions that had not actually been taken , while everything else remaining the same ( Pearl , 2009 ) . At the heart of CF-GPS are structural causal models ( SCMs ) which model the environment with two ingredients ( Wright , 1920 ) : 1 ) Independent random variables , called scenarios here , summarize all aspects of the environment that can not be influenced by the agent , e.g . the properties of the companies in Alice ’ s job search example . 2 ) Deterministic transition functions ( also called causal mechanisms ) take these scenarios , together with the agent ’ s actions , as input and produce the predicted outcome . The central idea of CF-GPS is that , instead of running an agent on scenarios sampled de novo from a model , scenarios are inferred in hindsight from given off-policy data , and then evaluate and improve the agent on these specific scenarios using given or learned causal mechanisms ( Balke & Pearl , 1994 ) . CF-GPS can be regarded as a meta-algorithm that extends a given model-base RL algorithm by “ grounding ” or “ anchoring ” model-based predictions in inferred scenarios . As a result , this approach explicitly allows to trade-off historical data for model bias . We show empirically in a conceptually simple setting , where unknown initial states are inferred in hindsight and re-used to evalute to counterfactual actions , that this can mitigate model mismatch . CF-GPS differs substantially from standard off-policy RL algorithms based on Importance Sampling ( IS ) , where historical data is re-weighted with respect to the importance weights to evaluate or learn new policies ( Precup , 2000 ) . In contrast , CF-GPS explicitly reasons counterfactually about given off-policy data . Our main contributions are : 1 . We formulate model-based RL in POMDPs in terms of structural causal models , thereby connecting concepts from reinforcement learning and causal inference . 2 . We provide the first results , to the best of our knowledge , showing that counterfactual reasoning in structural causal models on off-policy data can facilitate solving non-trivial RL tasks . 3 . We show that two previously proposed classes of RL algorithms , namely Guided Policy Search ( Levine & Koltun , 2013 ) and Stochastic Value Gradient methods ( Heess et al. , 2015 ) , can be interpreted as counterfactual methods , opening up possible generalizations . The paper is structured as follows . We first give a self-contained , high-level recapitulation of structural causal models and counterfactual inference , as these are less widely known in the RL and generative model communities . In particular we show how to model POMDPs with SCMs . Based on this exposition , we first consider the task of policy evaluation and discuss how we can leverage counterfactual inference in SCMs to improve over standard model-based methods . We then generalize this approach to the policy search setting resulting in the CF-GPS algorithm . We close by highlighting connections to previously proposed algorithms and by discussing assumptions and limitations of the proposed method . 2 PRELIMINARIES . We denote random variables ( RVs ) with capital letters , e.g . X , and particular values with lower caps , e.g . x . For a distribution P over a vector-valued random variable X , we denote the marginal over Y ⊂ X by PY ( and density pY ) ; however we often omit the subscript if it is clear from the context , e.g . as in Y ∼ P . We assume the episodic , Partially Observable Markov Decision Process ( POMDP ) setting with states St , actions At and observations Ot , for t = 1 , . . . , T . For ease of notation , we assume that Ot includes the reward Rt . The undiscounted return is denoted by G = ∑T t=1Rt . We consider stochastic policies π ( at|ht ) over actions conditioned on observation histories Ht = ( O1 , A1 , . . . , At−1 , Ot ) . We denote the resulting distribution over trajectories T = ( S1 , O1 , A1 , . . . , AT−1 , ST , OT ) induced by running π in the environment with T ∼ Pπ and the corresponding density by pπ ( τ ) . 2.1 STRUCTURAL CAUSAL MODELS . Definition 1 ( Structural causal model ) . A structural causal model ( SCM ) M over X = ( X1 , . . . , XN ) is given by a DAG G over nodes X , independent noise RVs U = ( U1 , . . . , UN ) with distributions PUi and functions f1 , . . . , fN such that Xi = fi ( pai , Ui ) , where pai ⊂ X are the parents of Xi in G. An SCM entails a distribution P with density p over ( X , U ) . We also refer to U as scenarios and to fi as causal mechanisms . We give a ( broad ) definition of an intervention in an SCM . This also includes what is known as stochastic interventions or mechanism changes ( Korb et al. , 2004 ) which generalize atomic interventions ( Pearl , 2009 ) . Definition 2 ( Intervention in SCM ) . An intervention I in an SCMM consists of replacing some of the original fi ( pai , Ui ) with other functions f I i ( pa I i , Ui ) where pa I i are the parents in a new DAG GI . We denote the resulting SCM withMdo ( I ) with distribution P do ( I ) and density pdo ( I ) . SCM representation of POMDPs We can represent any given POMDP ( under a policy π ) by an SCMM over trajectories T in the following way . We express all conditional distributions , e.g . the transition kernel PSt+1|St , At , as deterministic functions with independent noise variables U , such as St+1 = fst ( St , At , Ust ) . This is always possible using auto-regressive uniformization , see Lemma 2 in the appendix . The DAG G of the resulting SCM is shown in fig . 1 . This procedure is closely related to the ‘ reparameterization trick ’ for models with location-scale distributions ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ) . We denote the distribution over T entailed by the SCM with Pπ and its density by pπ to highlight the role of π ; note the difference to the true environment distribution Pπ with density pπ . Running a different policy µ instead of π in the environment can be expressed as an intervention I ( π → µ ) consisting of replacing At = fπ ( Ht , Uat ) by At = fµ ( Ht , Uat ) . We denote the resulting model distribution over trajectories with P do ( I ( π→µ ) ) = Pµ ( analogously pµ ) . Intuition Here , we illustrate the main advantage of SCMs using the example of Alice ’ s job choice from the introduction . We model it as contextual bandit with feedback shown in fig . 1 . Alice has some initial knowledge given by the context Uc that is available to her before taking action A of joining company A = a1 or A = a2 . We model Alice ’ s decision as A = fπ ( Uc , Ua ) , where Ua captures potential indeterminacy in Alice ’ s decision making . The outcome O = fo ( A , Uc , Uo ) also depends on the scenario Uo , capturing all relevant , unobserved and highly complex properties of the two companies such as working conditions etc . Given this model , we can reason about alternate outcomes fo ( a1 , uc , uo ) and fo ( a2 , uc , uo ) for same the scenario uo . This is not possible if we only model the outcome on the level of the conditional distribution PO|A , Uc . 2.2 COUNTERFACTUAL INFERENCE IN SCMS . For an SCM over X , we define a counterfactual query as a triple ( x̂o , I , Xq ) of observations x̂o of some variablesXo ⊂ X , an intervention I and query variablesXq ⊂ X . The semantics of the query are that , having observed x̂o , we want to infer what Xq would have been had we done intervention I , while ‘ keeping everything else the same ’ . Counterfactual inference ( CFI ) in SCMs answers the query in the following way ( Balke & Pearl , 1994 ) : 1 . Infer the unobserved noise source U conditioned on the observations x̂o , i.e . compute p ( U |x̂o ) and replace the prior p ( U ) with p ( U |x̂o ) . Denote the resulting SCM byMx̂o . 2 . Perform intervention I on Mx̂o . This yields M do ( I ) x̂o , which entails the counterfactual distribution pdo ( I ) |x̂o ( x ) . Return the marginal pdo ( I ) |x̂o ( xq ) . Note that our definition explicitly allows for partial observations Xo ⊂ X in accordance with Pearl ( 2009 ) . A sampled-based version , denoted as CFI , is presented in Algorithm 1 . An interesting property of the counterfactual distribution pdo ( I ) |x̂o is that marginalizing it over observations x̂o yields an unbiased estimator of the density of Xq under intervention I . Lemma 1 ( CFI for simulation ) . Let observations x̂o ∼ p come from a SCMM with density p. Then the counterfactual density pdo ( I ) |x̂o is an unbiased estimator of pdo ( I ) , i.e . Ex̂o∼p [ pdo ( I ) |x̂o ( x ) ] = pdo ( I ) ( x ) The proof is straightforward and outlined in the Appendix A . This lemma and the marginal independence of the Ui leads to the following corollary ; the proof is given in the appendix . Corollary 1 ( Mixed counterfactual and prior simulation from an SCM ) . Assume we have observations x̂o ∼ p. We can simulate from M , under any intervention I , i.e . obtain unbiased samples from Mdo ( I ) , by first sampling values uCF for an arbitrary subset UCF ⊂ U from the posterior p ( uCF|x̂o ) and the remaining UPrior : = U\UCF from the prior p ( uPrior ) , and then computing X with noise u = uCF ∪ uPrior . The corollary essentially states that we can sample from the modelMI , by sampling some of the Ui from the prior , and inferring the rest from data x̂o ( as long as the latter was also sampled fromM ) . We will make use of this later for running a POMDP model on scenariosUst inferred from data while randomizing the action noise Uat . We note that the noise variables UCF from the posterior PUCF|x̂o are not independent anymore . Nevertheless , SCMs with non-independent noise distributions arising from counterfactual inference , denoted here by Mx̂o , are commonly considered in the literature ( Peters et al. , 2017 ) . Intuition Returning to Alice ’ s job example from the introduction , we give some intuition for counterfactual inference in SCMs . Given the concrete outcome ô , under observed context ûc and having joined company â = a1 , Alice can try to infer the underlying scenario uo ∼ p ( uo|a1 , ûc , ô ) that she experiences ; this includes factors such as work conditions etc . She can then reason counterfactually about the outcome had she joined the other company , which is given by fo ( a2 , ûc , uo ) . This can in principle enable her to make better decisions in the future in similar scenarios by changing her policy fπ ( A , Uc , Ua ) such that the action with the preferred outcome becomes more likely under ûc , uo . In particular she can do so without having to use her ( likely imperfect ) prior model over possible companies p ( Uo ) . She can use the counterfactual predictions discussed above instead to learn from her experience . We use this insight for counterfactual policy evaluation and search below .

Summary: This paper proposes a policy evaluation and search method assisted by a counterfactual model, in contrast previous work using vanilla (non-causal) models. With “no model mismatch” assumption the policy evaluation estimator is unbiased. Empirically, the paper compares Guided Policy Search with counterfactual model (CF-GPS) with vanilla GPS, model based RL algorithm and show benefit in terms of (empirical) sample complexity.