Model-Free Counterfactual Credit Assignment

1 INTRODUCTION . Reinforcement learning ( RL ) agents act in their environments and learn to achieve desirable outcomes by maximizing a reward signal . A key difficulty is the problem of credit assignment ( Minsky , 1961 ) , i.e . to understand the relation between actions and outcomes and to determine to what extent an outcome was caused by external , uncontrollable factors , i.e . to determine the share of ‘ skill ’ and ‘ luck ’ . One possible solution to this problem is for the agent to build a model of the environment , and use it to obtain a more fine-grained understanding of the effects of an action . While this topic has recently generated a lot of interest ( Ha & Schmidhuber , 2018 ; Hamrick , 2019 ; Kaiser et al. , 2019 ; Schrittwieser et al. , 2019 ) , it remains difficult to model complex , partially observed environments . In contrast , model-free reinforcement learning algorithms such as policy gradient methods ( Williams , 1992 ; Sutton et al. , 2000 ) perform simple time-based credit assignment , where events and rewards happening after an action are credited to that action , post hoc ergo propter hoc . While unbiased in expectation , this coarse-grained credit assignment typically has high variance , and the agent will require a large amount of experience to learn the correct relation between actions and rewards . Another issue of model-free methods is that counterfactual reasoning , i.e . reasoning about what would have happened had different actions been taken with everything else remaining the same , is not possible . Given a trajectory , model-free methods can in fact only learn about the actions that were actually taken to produce the data , and this limits the ability of the agent to learn quickly . As environments grow in complexity due to partial observability , scale , long time horizons , and large number of agents , actions taken by the agent will only affect a vanishing part of the outcome , making it increasingly difficult to learn from classical reinforcement learning algorithms . We need better credit assignment techniques . In this paper , we investigate a new method of credit assignment for model-free reinforcement learning which we call Counterfactual Credit Assignment ( CCA ) , that leverages hindsight information to implicitly perform counterfactual evaluation - an estimate of the return for actions other than the ones which were chosen . These counterfactual returns can be used to form unbiased and lower variance estimates of the policy gradient by building future-conditional baselines . Unlike classical Q functions , which also provide an estimate of the return for all actions but do so by averaging over all possible futures , our methods provide trajectory-specific counterfactual estimates , i.e . an estimate of the return for different actions , but keeping as many of the external factors constant between the return and its counterfactual estimate . Our method is inspired by ideas from causality theory , but does not require learning a model of the environment . Our main contributions are : a ) proposing a set of environments which further our understanding of when difficult credit assignment leads to poor policy learning ; b ) introducing new model-free policy gradient algorithms , with sufficient conditions for unbiasedness and guarantees for lower variance . In the appendix , we further c ) present a collection of model-based policy gradient algorithms extending previous work on counterfactual policy search ; d ) connect the literature about causality theory , in particular notions of treatment effects , to concepts from the reinforcement learning literature . 2 COUNTERFACTUAL CREDIT ASSIGNMENT . 2.1 NOTATION . We use capital letters for random variables and lowercase for the value they take . Consider a generic MDP ( X , A , p , r , γ ) . Given a current state x ∈ X and assuming an agent takes action a ∈ A , the agent receives reward r ( x , a ) and transitions to a state y ∼ p ( ·|x , a ) . The state ( resp . action , reward ) of the agent at step t is denoted Xt ( resp . At , Rt ) . The initial state of the agent X0 is a fixed x0 . The agent acts according to a policy π , i.e . action At is sampled from the policy πθ ( ·|Xt ) where θ are the parameters of the policy , and aims to optimize the expected discounted return E [ G ] = E [ ∑ t γ tRt ] . The return Gt from step t is Gt = ∑ t′≥t γ t′−tRt′ . Finally , we define the score function sθ ( πθ , a , x ) = ∇θ log πθ ( a|x ) ; the score function at time t is denoted St = ∇θ log πθ ( At|Xt ) . In the case of a partially observed environment , we assume the agent receives an observation Et at every time step , and simply define Xt to be the set of all previous observations , actions and rewards Xt = ( O≤t ) , with Ot = ( Et , At−1 , Rt−1 ) .1 P ( X ) will denote the probability distribution of a random variable X . 2.2 POLICY GRADIENT ALGORITHMS . We begin by recalling two forms of policy gradient algorithms and the credit assignment assumptions they make . The first is the REINFORCE algorithm introduced by Williams ( 1992 ) , which we will also call the single-action policy gradient estimator : Proposition 1 ( single action estimator ) . The gradient of E [ G ] is given by ∇θE [ G ] = E [ ∑ t≥0 γ t St ( Gt − V ( Xt ) ) ] , where V ( Xt ) = E [ Gt|Xt ] . The appeal of this estimator lies in its simplicity and generality : to evaluate it , the only requirement is the ability to simulate trajectories , and compute both the score function and the return . Let us note two credit assignment features of the estimator . First , the score function St is multiplied not by the whole return G , but by the return from time t. Intuitively , action At can only affect states and rewards coming after time t , and it is therefore pointless to credit action At with past rewards . Second , removing the value function V ( Xt ) from the return Gt does not bias the estimator and typically reduces variance . This estimator updates the policy through the score term ; note however the learning signal only updates the policy πθ ( a|Xt ) at the value taken by action At = a ( other values are only updated through normalization ) . The policy gradient theorem from ( Sutton et al. , 2000 ) , which we will also call all-action policy gradient , shows it is possible to provide learning signal to all actions , given we have access to a Q-function Qπ ( x , a ) = E [ Gt|Xt = x , At = a ] , which we will call a critic in the following . Proposition 2 ( All-action policy gradient estimator ) . The gradient of E [ G ] is given by ∇θE [ G ] = E [ ∑ t γ t ∑ a∇θπθ ( a|Xt ) Qπθ ( Xt , a ) ] . A particularity of the all-actions policy gradient estimator is that the term at time t for updating the policy ∇πθ ( a|Xt ) Qπθ ( Xt , a ) depends only on past information ; this is in contrast with the score function estimates above which depend on the return , a function of the entire trajectory . Proofs can be found in appendix D.1 . 2.3 INTUITIVE EXAMPLE ON HINDSIGHT REASONING AND SKILL VERSUS LUCK . Imagine a scenario in which Alice just moved to a new city , is learning to play soccer , and goes to the local soccer field to play a friendly game with a group of other kids she has never met . As the game goes on , Alice does not seem to play at her best and makes some mistakes . It turns out however her partner Megan is a strong player , and eventually scores the goal that makes the game a victory . What should Alice learn from this game ? 1Previous actions and rewards are provided as part of the observation as it is generally beneficial to do so in partially observable Markov decision processes . When using the single-action policy gradient estimate , the outcome of the game being a victory , and assuming a ±1 reward scheme , all her actions are made more likely ; this is in spite of the fact that during this particular game she may not have played well and that the victory is actually due to her strong teammate . From an RL point of view , her actions are wrongly credited for the victory and positively reinforced as a result ; effectively , Alice was lucky rather than skillful . Regular baselines do not mitigate this issue , as Alice did not a priori know the skill of Megan , resulting in a guess she had a 50 % chance of winning the game and corresponding baseline of 0 . This could be fixed by understanding that Megan ’ s strong play were not a consequence of Alice ’ s play , that her skill was a priori unknown but known in hindsight , and that it is therefore valid to retroactively include her skill level in the baseline . A hindsight baseline , conditioned on Megan ’ s estimated skill level , would therefore be closer to 1 , driving the advantage ( and corresponding learning signal ) close to 0 . As pointed out by Buesing et al . ( 2019 ) , situations in which hindsight information is helpful in understanding a trajectory are frequent . In that work , the authors adopt a model-based framework , where hindsight information is used to ground counterfactual trajectories ( i.e . trajectories under different actions , but same randomness ) . Our proposed approach follows a similar intuition , but is model-free : we attempt to measure—instead of model— information known in hindsight to compute a future-conditional baseline , with the constraint that the captured information must not have been caused by the agent . 2.4 FUTURE-CONDITIONAL POLICY GRADIENT ESTIMATOR ( FC-PG ) . Intuitively , our approach for assigning proper credit to action At is as follows : via learning statistics Φt we capture relevant information from the rest of the trajectory , e.g . including observations Ot′ at times t′ greater than t. We then learn value functions which are conditioned on the additional hindsight information contained in Φt . In general , these future-conditional values and critics would be biased for use in a policy gradient algorithm ; we therefore need to correct their impact on the policy gradient through an importance correction term . Theorem 1 ( Future single-action policy gradient estimator ) . Let Φt be an arbitrary random variable . The following is an unbiased estimator of the gradient of E [ G ] : ∇θE [ G ] = E [ ∑ t γt St ( Gt − πθ ( At|Xt ) Pπθ ( At|Xt , Φt ) V ( Xt , Φt ) ) ] ( 1 ) where V ( Xt , Φt ) = E [ Gt|Xt , Φt ] is the future Φ-conditional value function2 , and Pπθ ( At|Xt , Φt ) is the posterior probability of action At given ( Xt , Φt ) , for trajectories generated by policy πθ . Theorem 2 ( Future all-action policy gradient estimator ) . The following is an unbiased estimator of the gradient of E [ G ] : ∇θE [ G ] = E [ ∑ t γt ∑ a ∇θ log πθ ( a|Xt ) Pπθ ( a|Xt , Φt ) Qπθ ( Xt , Φt , a ) ] ( 2 ) where Qπ ( Xt , Φt , a ) = E [ Gt|Xt , Φt , At = a ] is the future-conditional Q function ( critic ) . Furthermore , we have Qπθ ( Xt , a ) = E [ Qπθ ( Xt , Φt , a ) Pπ ( a|Xt , Φt ) π ( a|Xt ) ] . Proofs can be found in appendix D.2 . These estimators bear similarity to ( and indeed , generalize ) the Hindsight Credit Assignment estimator ( Harutyunyan et al. , 2019 ) , see the literature review and appendix C for a discussion of the connections .

Summary: The paper explores a new approach to credit assignment that complements existing work. It focuses on model-free approaches to credit assignment using hindsight information. In contrast to some prior work on this topic, e.g., (Harutyunyan et al. 2019), the paper does not rely explicitly on hand-crafted information, but instead learns to extract useful hindsight information. The contributions of the paper are two-fold. First, the paper introduces two new policy gradient estimators, FC-PG and CCA-PG, and it proves that the novel gradient estimators are unbiased. Second, it provides experimental evidence that the novel estimators are beneficial compared to some prior work (in particular (Harutyunyan et al. 2019)).