Hindsight policy gradients

1 INTRODUCTION . In a traditional reinforcement learning setting , an agent interacts with an environment in a sequence of episodes , observing states and acting according to a policy that ideally maximizes expected cumulative reward . If an agent is required to pursue different goals across episodes , its goal-conditional policy may be represented by a probability distribution over actions for every combination of state and goal . This distinction between states and goals is particularly useful when the probability of a state transition given an action is independent of the goal pursued by the agent . Learning such goal-conditional behavior has received significant attention in machine learning and robotics , especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent ( Schmidhuber & Huber , 1990 ; Da Silva et al. , 2012 ; Kupcsik et al. , 2013 ; Deisenroth et al. , 2014 ; Schaul et al. , 2015 ; Zhu et al. , 2017 ; Kober et al. , 2012 ; Ghosh et al. , 2018 ; Mankowitz et al. , 2018 ; Pathak et al. , 2018 ) . Consequently , developing goal-based curricula to facilitate learning has also attracted considerable interest ( Fabisch & Metzen , 2014 ; Florensa et al. , 2017 ; Sukhbaatar et al. , 2018 ; Srivastava et al. , 2013 ) . In hierarchical reinforcement learning , goal-conditional policies may enable agents to plan using subgoals , which abstracts the details involved in lower-level decisions ( Oh et al. , 2017 ; Vezhnevets et al. , 2017 ; Kulkarni et al. , 2016 ; Levy et al. , 2017 ) . In a typical sparse-reward environment , an agent receives a non-zero reward only upon reaching a goal state . Besides being natural , this task formulation avoids the potentially difficult problem of reward shaping , which often biases the learning process towards suboptimal behavior ( Ng et al. , 1999 ) . Unfortunately , sparse-reward environments remain particularly challenging for traditional reinforcement learning algorithms ( Andrychowicz et al. , 2017 ; Florensa et al. , 2017 ) . For example , consider an agent tasked with traveling between cities . In a sparse-reward formulation , if reaching a desired destination by chance is unlikely , a learning agent will rarely obtain reward signals . At ∗Work performed while at IDSIA . the same time , it seems natural to expect that an agent will learn how to reach the cities it visited regardless of its desired destinations . In this context , the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended is called hindsight . This capacity was recently introduced by Andrychowicz et al . ( 2017 ) to off-policy reinforcement learning algorithms that rely on experience replay ( Lin , 1992 ) . In earlier work , Karkus et al . ( 2016 ) introduced hindsight to policy search based on Bayesian optimization ( Metzen et al. , 2015 ) . In this paper , we demonstrate how hindsight can be introduced to policy gradient methods ( Williams , 1986 ; 1992 ; Sutton et al. , 1999a ) , generalizing this idea to a successful class of reinforcement learning algorithms ( Peters & Schaal , 2008 ; Duan et al. , 2016 ) . In contrast to previous work on hindsight , our approach relies on importance sampling ( Bishop , 2013 ) . In reinforcement learning , importance sampling has been traditionally employed in order to efficiently reuse information obtained by earlier policies during learning ( Precup et al. , 2000 ; Peshkin & Shelton , 2002 ; Jie & Abbeel , 2010 ; Thomas et al. , 2015 ; Munos et al. , 2016 ) . In comparison , our approach attempts to efficiently learn about different goals using information obtained by the current policy for a specific goal . This approach leads to multiple formulations of a hindsight policy gradient that relate to well-known policy gradient results . In comparison to conventional ( goal-conditional ) policy gradient estimators , our proposed estimators lead to remarkable sample efficiency on a diverse selection of sparse-reward environments . 2 PRELIMINARIES . We denote random variables by upper case letters and assignments to these variables by corresponding lower case letters . We let Val ( X ) denote the set of valid assignments to a random variable X . We also omit the subscript that typically relates a probability function to random variables when there is no risk of ambiguity . For instance , we may use p ( x ) to denote pX ( x ) and p ( y ) to denote pY ( y ) . Consider an agent that interacts with its environment in a sequence of episodes , each of which lasts for exactly T time steps . The agent receives a goal g ∈ Val ( G ) at the beginning of each episode . At every time step t , the agent observes a state st ∈ Val ( St ) , receives a reward r ( st , g ) ∈ R , and chooses an action at ∈ Val ( At ) . For simplicity of notation , suppose that Val ( G ) , Val ( St ) , and Val ( At ) are finite for every t. In our setting , a goal-conditional policy defines a probability distribution over actions for every combination of state and goal . The same policy is used to make decisions at every time step . Let τ = s1 , a1 , s2 , a2 , . . . , sT−1 , aT−1 , sT denote a trajectory . We assume that the probability p ( τ | g , θ ) of trajectory τ given goal g and a policy parameterized by θ ∈ Val ( Θ ) is given by p ( τ | g , θ ) = p ( s1 ) T−1∏ t=1 p ( at | st , g , θ ) p ( st+1 | st , at ) . ( 1 ) In contrast to a Markov decision process , this formulation allows the probability of a state transition given an action to change across time steps within an episode . More importantly , it implicitly states that the probability of a state transition given an action is independent of the goal pursued by the agent , which we denote by St+1 ⊥ G | St , At . For every τ , g , and θ , we also assume that p ( τ | g , θ ) is non-zero and differentiable with respect to θ . Assuming that G ⊥ Θ , the expected return η ( θ ) of a policy parameterized by θ is given by η ( θ ) = E [ T∑ t=1 r ( St , G ) | θ ] = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T∑ t=1 r ( st , g ) . ( 2 ) The action-value function is given by Qθt ( s , a , g ) = E [ ∑T t′=t+1 r ( St′ , g ) | St = s , At = a , g , θ ] , the value function by V θt ( s , g ) = E [ Qθt ( s , At , g ) | St = s , g , θ ] , and the advantage function by Aθt ( s , a , g ) = Q θ t ( s , a , g ) − V θt ( s , g ) . 3 GOAL-CONDITIONAL POLICY GRADIENTS . This section presents results for goal-conditional policies that are analogous to well-known results for conventional policies ( Peters & Schaal , 2008 ) . They establish the foundation for the results presented in the next section . The corresponding proofs are included in Appendix A for completeness . The objective of policy gradient methods is finding policy parameters that achieve maximum expected return . When combined with Monte Carlo techniques ( Bishop , 2013 ) , the following result allows pursuing this objective using gradient-based optimization . Theorem 3.1 ( Goal-conditional policy gradient ) . The gradient ∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) T∑ t′=t+1 r ( st′ , g ) . ( 3 ) The following result allows employing a baseline to reduce the variance of the gradient estimator . Theorem 3.2 ( Goal-conditional policy gradient , baseline formulation ) . For every t , θ , and associated real-valued ( baseline ) function bθt , the gradient ∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) [ [ T∑ t′=t+1 r ( st′ , g ) ] − bθt ( st , g ) ] . ( 4 ) Appendix A.7 presents the constant baselines that minimize the ( elementwise ) variance of the corresponding estimator . However , such baselines are usually impractical to compute ( or estimate ) , and the variance of the estimator may be reduced further by a baseline function that depends on state and goal . Although generally suboptimal , it is typical to let the baseline function bθt approximate the value function V θt ( Greensmith et al. , 2004 ) . Lastly , actor-critic methods may rely on the following result for goal-conditional policies . Theorem 3.3 ( Goal-conditional policy gradient , advantage formulation ) . The gradient∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) Aθt ( st , at , g ) . ( 5 ) 4 HINDSIGHT POLICY GRADIENTS . This section presents the novel ideas that introduce hindsight to policy gradient methods . The corresponding proofs can be found in Appendix B . Suppose that the reward r ( s , g ) is known for every combination of state s and goal g , as in previous work on hindsight ( Andrychowicz et al. , 2017 ; Karkus et al. , 2016 ) . In that case , it is possible to evaluate a trajectory obtained while trying to achieve an original goal g′ for an alternative goal g. Using importance sampling , this information can be exploited using the following central result . Theorem 4.1 ( Every-decision hindsight policy gradient ) . For an arbitrary ( original ) goal g′ , the gradient∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) T∑ t′=t+1 [ T−1∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) ] r ( st′ , g ) . ( 6 ) In the formulation presented above , every reward is multiplied by the ratio between the likelihood of the corresponding trajectory under an alternative goal and the likelihood under the original goal ( see Eq . 1 ) . Intuitively , every reward should instead be multiplied by a likelihood ratio that only considers the corresponding trajectory up to the previous action . This intuition underlies the following important result , named after an analogous result for action-value functions by Precup et al . ( 2000 ) . Theorem 4.2 ( Per-decision hindsight policy gradient ) . For an arbitrary ( original ) goal g′ , the gradient∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) T∑ t′=t+1 t′−1∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) r ( st′ , g ) . ( 7 ) The following lemma allows introducing baselines to hindsight policy gradients ( see App . B.4 ) . Lemma 4.1 . For every g′ , t , θ , and associated real-valued ( baseline ) function bθt , ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) [ t∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) ] bθt ( st , g ) = 0 . ( 8 ) Appendix B.7 presents the constant baselines that minimize the ( elementwise ) variance of the corresponding gradient estimator . By analogy with the conventional practice , we suggest letting the baseline function bθt approximate the value function V θ t instead . Importantly , the choice of likelihood ratio in Lemma 4.1 is far from unique . However , besides leading to straightforward estimation , it also underlies the advantage formulation presented below . Theorem 4.3 ( Hindsight policy gradient , advantage formulation ) . For an arbitrary ( original ) goal g′ , the gradient ∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) [ t∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) ] Aθt ( st , at , g ) . ( 9 ) Fortunately , the following result allows approximating the advantage under a goal using a state transition collected while pursuing another goal ( see App . D.4 ) . Theorem 4.4 . For every t and θ , the advantage function Aθt is given by Aθt ( s , a , g ) = E [ r ( St+1 , g ) + V θ t+1 ( St+1 , g ) − V θt ( s , g ) | St = s , At = a ] . ( 10 )

Summary: This paper extends the work of Hindsight Experience Replay to (goal-conditioned) policy gradient methods. Hindsight, which allows one to learn policies conditioned on some goal g, from off-policy experience generated by following goal g’, is cast in the framework of importance sampling. The authors show how one can simply rewrite the goal-conditioned policy gradient by first sampling a trajectory, conditioned on some goal $g’$ and then computing the closed form gradient in expectation over all goals. This gradient is unbiased if the rewards are off-policy corrected along the generated trajectories. While this naive formulation is found to be unstable , the authors propose a simple normalized importance sampling formulation which appears to work well in practice. To further reduce variance and computational costs, the authors also propose goal subsampling mechanisms, which sample goals which are likely along the generated trajectories. The method is evaluated on the same bit-flipping environment as [1], and a variety of discrete environments (grid worlds, Ms. Pac-Man, simulated robot arm) where the method appears highly effective. Unfortunately for reasons which remain unclear, hindsight policy gradients with value baselines appear unstable.