Hindsight policy gradients

1 INTRODUCTION . In a traditional reinforcement learning setting , an agent interacts with an environment in a sequence of episodes , observing states and acting according to a policy that ideally maximizes expected cumulative reward . If an agent is required to pursue different goals across episodes , its goal-conditional policy may be represented by a probability distribution over actions for every combination of state and goal . This distinction between states and goals is particularly useful when the probability of a state transition given an action is independent of the goal pursued by the agent . Learning such goal-conditional behavior has received significant attention in machine learning and robotics , especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent ( Schmidhuber & Huber , 1990 ; Da Silva et al. , 2012 ; Kupcsik et al. , 2013 ; Deisenroth et al. , 2014 ; Schaul et al. , 2015 ; Zhu et al. , 2017 ; Kober et al. , 2012 ; Ghosh et al. , 2018 ; Mankowitz et al. , 2018 ; Pathak et al. , 2018 ) . Consequently , developing goal-based curricula to facilitate learning has also attracted considerable interest ( Fabisch & Metzen , 2014 ; Florensa et al. , 2017 ; Sukhbaatar et al. , 2018 ; Srivastava et al. , 2013 ) . In hierarchical reinforcement learning , goal-conditional policies may enable agents to plan using subgoals , which abstracts the details involved in lower-level decisions ( Oh et al. , 2017 ; Vezhnevets et al. , 2017 ; Kulkarni et al. , 2016 ; Levy et al. , 2017 ) . In a typical sparse-reward environment , an agent receives a non-zero reward only upon reaching a goal state . Besides being natural , this task formulation avoids the potentially difficult problem of reward shaping , which often biases the learning process towards suboptimal behavior ( Ng et al. , 1999 ) . Unfortunately , sparse-reward environments remain particularly challenging for traditional reinforcement learning algorithms ( Andrychowicz et al. , 2017 ; Florensa et al. , 2017 ) . For example , consider an agent tasked with traveling between cities . In a sparse-reward formulation , if reaching a desired destination by chance is unlikely , a learning agent will rarely obtain reward signals . At ∗Work performed while at IDSIA . the same time , it seems natural to expect that an agent will learn how to reach the cities it visited regardless of its desired destinations . In this context , the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended is called hindsight . This capacity was recently introduced by Andrychowicz et al . ( 2017 ) to off-policy reinforcement learning algorithms that rely on experience replay ( Lin , 1992 ) . In earlier work , Karkus et al . ( 2016 ) introduced hindsight to policy search based on Bayesian optimization ( Metzen et al. , 2015 ) . In this paper , we demonstrate how hindsight can be introduced to policy gradient methods ( Williams , 1986 ; 1992 ; Sutton et al. , 1999a ) , generalizing this idea to a successful class of reinforcement learning algorithms ( Peters & Schaal , 2008 ; Duan et al. , 2016 ) . In contrast to previous work on hindsight , our approach relies on importance sampling ( Bishop , 2013 ) . In reinforcement learning , importance sampling has been traditionally employed in order to efficiently reuse information obtained by earlier policies during learning ( Precup et al. , 2000 ; Peshkin & Shelton , 2002 ; Jie & Abbeel , 2010 ; Thomas et al. , 2015 ; Munos et al. , 2016 ) . In comparison , our approach attempts to efficiently learn about different goals using information obtained by the current policy for a specific goal . This approach leads to multiple formulations of a hindsight policy gradient that relate to well-known policy gradient results . In comparison to conventional ( goal-conditional ) policy gradient estimators , our proposed estimators lead to remarkable sample efficiency on a diverse selection of sparse-reward environments . 2 PRELIMINARIES . We denote random variables by upper case letters and assignments to these variables by corresponding lower case letters . We let Val ( X ) denote the set of valid assignments to a random variable X . We also omit the subscript that typically relates a probability function to random variables when there is no risk of ambiguity . For instance , we may use p ( x ) to denote pX ( x ) and p ( y ) to denote pY ( y ) . Consider an agent that interacts with its environment in a sequence of episodes , each of which lasts for exactly T time steps . The agent receives a goal g ∈ Val ( G ) at the beginning of each episode . At every time step t , the agent observes a state st ∈ Val ( St ) , receives a reward r ( st , g ) ∈ R , and chooses an action at ∈ Val ( At ) . For simplicity of notation , suppose that Val ( G ) , Val ( St ) , and Val ( At ) are finite for every t. In our setting , a goal-conditional policy defines a probability distribution over actions for every combination of state and goal . The same policy is used to make decisions at every time step . Let τ = s1 , a1 , s2 , a2 , . . . , sT−1 , aT−1 , sT denote a trajectory . We assume that the probability p ( τ | g , θ ) of trajectory τ given goal g and a policy parameterized by θ ∈ Val ( Θ ) is given by p ( τ | g , θ ) = p ( s1 ) T−1∏ t=1 p ( at | st , g , θ ) p ( st+1 | st , at ) . ( 1 ) In contrast to a Markov decision process , this formulation allows the probability of a state transition given an action to change across time steps within an episode . More importantly , it implicitly states that the probability of a state transition given an action is independent of the goal pursued by the agent , which we denote by St+1 ⊥ G | St , At . For every τ , g , and θ , we also assume that p ( τ | g , θ ) is non-zero and differentiable with respect to θ . Assuming that G ⊥ Θ , the expected return η ( θ ) of a policy parameterized by θ is given by η ( θ ) = E [ T∑ t=1 r ( St , G ) | θ ] = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T∑ t=1 r ( st , g ) . ( 2 ) The action-value function is given by Qθt ( s , a , g ) = E [ ∑T t′=t+1 r ( St′ , g ) | St = s , At = a , g , θ ] , the value function by V θt ( s , g ) = E [ Qθt ( s , At , g ) | St = s , g , θ ] , and the advantage function by Aθt ( s , a , g ) = Q θ t ( s , a , g ) − V θt ( s , g ) . 3 GOAL-CONDITIONAL POLICY GRADIENTS . This section presents results for goal-conditional policies that are analogous to well-known results for conventional policies ( Peters & Schaal , 2008 ) . They establish the foundation for the results presented in the next section . The corresponding proofs are included in Appendix A for completeness . The objective of policy gradient methods is finding policy parameters that achieve maximum expected return . When combined with Monte Carlo techniques ( Bishop , 2013 ) , the following result allows pursuing this objective using gradient-based optimization . Theorem 3.1 ( Goal-conditional policy gradient ) . The gradient ∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) T∑ t′=t+1 r ( st′ , g ) . ( 3 ) The following result allows employing a baseline to reduce the variance of the gradient estimator . Theorem 3.2 ( Goal-conditional policy gradient , baseline formulation ) . For every t , θ , and associated real-valued ( baseline ) function bθt , the gradient ∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) [ [ T∑ t′=t+1 r ( st′ , g ) ] − bθt ( st , g ) ] . ( 4 ) Appendix A.7 presents the constant baselines that minimize the ( elementwise ) variance of the corresponding estimator . However , such baselines are usually impractical to compute ( or estimate ) , and the variance of the estimator may be reduced further by a baseline function that depends on state and goal . Although generally suboptimal , it is typical to let the baseline function bθt approximate the value function V θt ( Greensmith et al. , 2004 ) . Lastly , actor-critic methods may rely on the following result for goal-conditional policies . Theorem 3.3 ( Goal-conditional policy gradient , advantage formulation ) . The gradient∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ g p ( g ) ∑ τ p ( τ | g , θ ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) Aθt ( st , at , g ) . ( 5 ) 4 HINDSIGHT POLICY GRADIENTS . This section presents the novel ideas that introduce hindsight to policy gradient methods . The corresponding proofs can be found in Appendix B . Suppose that the reward r ( s , g ) is known for every combination of state s and goal g , as in previous work on hindsight ( Andrychowicz et al. , 2017 ; Karkus et al. , 2016 ) . In that case , it is possible to evaluate a trajectory obtained while trying to achieve an original goal g′ for an alternative goal g. Using importance sampling , this information can be exploited using the following central result . Theorem 4.1 ( Every-decision hindsight policy gradient ) . For an arbitrary ( original ) goal g′ , the gradient∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) T∑ t′=t+1 [ T−1∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) ] r ( st′ , g ) . ( 6 ) In the formulation presented above , every reward is multiplied by the ratio between the likelihood of the corresponding trajectory under an alternative goal and the likelihood under the original goal ( see Eq . 1 ) . Intuitively , every reward should instead be multiplied by a likelihood ratio that only considers the corresponding trajectory up to the previous action . This intuition underlies the following important result , named after an analogous result for action-value functions by Precup et al . ( 2000 ) . Theorem 4.2 ( Per-decision hindsight policy gradient ) . For an arbitrary ( original ) goal g′ , the gradient∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) T∑ t′=t+1 t′−1∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) r ( st′ , g ) . ( 7 ) The following lemma allows introducing baselines to hindsight policy gradients ( see App . B.4 ) . Lemma 4.1 . For every g′ , t , θ , and associated real-valued ( baseline ) function bθt , ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) [ t∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) ] bθt ( st , g ) = 0 . ( 8 ) Appendix B.7 presents the constant baselines that minimize the ( elementwise ) variance of the corresponding gradient estimator . By analogy with the conventional practice , we suggest letting the baseline function bθt approximate the value function V θ t instead . Importantly , the choice of likelihood ratio in Lemma 4.1 is far from unique . However , besides leading to straightforward estimation , it also underlies the advantage formulation presented below . Theorem 4.3 ( Hindsight policy gradient , advantage formulation ) . For an arbitrary ( original ) goal g′ , the gradient ∇η ( θ ) of the expected return with respect to θ is given by ∇η ( θ ) = ∑ τ p ( τ | g′ , θ ) ∑ g p ( g ) T−1∑ t=1 ∇ log p ( at | st , g , θ ) [ t∏ k=1 p ( ak | sk , g , θ ) p ( ak | sk , g′ , θ ) ] Aθt ( st , at , g ) . ( 9 ) Fortunately , the following result allows approximating the advantage under a goal using a state transition collected while pursuing another goal ( see App . D.4 ) . Theorem 4.4 . For every t and θ , the advantage function Aθt is given by Aθt ( s , a , g ) = E [ r ( St+1 , g ) + V θ t+1 ( St+1 , g ) − V θt ( s , g ) | St = s , At = a ] . ( 10 )

Summary: Following recent work on Hindsight Experience Replay (Andrychowicz et al. 2017), the authors extend the idea to policy gradient methods. They formally describe the goal-conditioned policy gradient setup and derive the extensions of the classical policy gradient estimators. Their key insight to deriving a computationally efficient estimator is that for many situations, only a small number of goals will be "active" in a single trajectory. Then, they conduct extensive experiments on a range of problems and show that their approach leads to improvements in sample efficiency for goal-conditioned tasks.