Hindsight Trust Region Policy Optimization

1 INTRODUCTION . Reinforcement Learning has been a heuristic approach confronting a great many real-world problems from playing complex strategic games ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Justesen et al. , 2019 ) to the precise control of robots ( Levine et al. , 2016 ; Mahler & Goldberg , 2017 ; Quillen et al. , 2018 ) , in which policy gradient methods play very important roles ( Sutton et al. , 2000 ; Deisenroth et al. , 2013 ) . Among them , the ones based on trust region including Trust Region Policy Optimization ( Schulman et al. , 2015a ) and Proximal Policy Optimization ( Schulman et al. , 2017 ) have achieved stable and effective performances on several benchmark tasks . Later on , they have been verified in a variety of applications including skill learning ( Nagabandi et al. , 2018 ) , multi-agent control ( Gupta et al. , 2017 ) , imitation learning ( Ho et al. , 2016 ) , and have been investigated further to be combined with more advanced techniques ( Nachum et al. , 2017 ; Houthooft et al. , 2016 ; Heess et al. , 2017 ) . One unresolved core issue in reinforcement learning is efficiently training the agent in sparse reward environments , in which the agent is given a distinctively high feedback only upon reaching the desired final goal state . On one hand , generalizing reinforcement learning methods to sparse reward scenarios obviates designing delicate reward mechanism , which is known as reward shaping ( Ng et al. , 1999 ) ; on the other hand , receiving rewards only when precisely reaching the final goal states also guarantees that the agent can focus on the intended task itself without any deviation . Despite the extensive use of policy gradient methods , they tend to be vulnerable when dealing with sparse reward scenarios . Admittedly , policy gradient may work in simple and sufficiently rewarding environments through massive random exploration . However , since it relies heavily on the expected return , the chances in complex and sparsely rewarding scenarios become rather slim , which often makes it unfeasible to converge to a policy by exploring randomly . Recently , several works have been devoted to solving the problem of sparse reward , mainly applying either hierarchical reinforcement learning ( Kulkarni et al. , 2016 ; Vezhnevets et al. , 2017 ; Le et al. , 2018 ; Marino et al. , 2019 ) or a hindsight methodology , including Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) , Hindsight Policy Gradient ( Rauber et al. , 2019 ) and their extensions ( Fang et al. , 2019 ; Levy et al. , 2019 ) . The idea of Hindsight Experience Replay ( HER ) is to regard the ending states obtained through the interaction under current policy as alternative goals , and therefore generate more effective training data comparing to that with only real goals . Such augmentation overcomes the defects of random exploration and allows the agent to progressively move towards intended goals . It is proven to be promising when dealing with sparse reward reinforcement learning problems . For Hindsight Policy Gradient ( HPG ) , it introduces hindsight to policy gradient approach and improves sample efficiency in sparse reward environments . Yet , its learning curve for policy update still oscillates considerably . Because it inherits the intrinsic high variance of policy gradient methods which has been widely studied in Schulman et al . ( 2015b ) , Gu et al . ( 2016 ) and Wu et al . ( 2018 ) . Furthermore , introducing hindsight to policy gradient methods would lead to greater variance ( Rauber et al. , 2019 ) . Consequently , such exacerbation would cause obstructive instability during the optimization process . To design an advanced and efficient on-policy reinforcement learning algorithm with hindsight experience , the main problem is the contradiction between on-policy data needed by the training process and the severely off-policy hindsight experience we can get . Moreover , for TRPO , one of the most significant property is the approximated monotonic converging process . Therefore , how these advantages can be preserved when the agent is trained with hindsight data also remains unsolved . In this paper , we propose a methodology called Hindsight Trust Region Policy Optimization ( HTRPO ) . Starting from TRPO , a hindsight form of policy optimization problem within trust region is theoretically derived , which can be approximately solved with the Monte Carlo estimator using severely off-policy hindsight experience data . HTRPO extends the effective and monotonically iterative policy optimization procedure within trust region to accommodate sparse reward environments . In HTRPO , both the objective function and the expectation of KL divergence between policies are estimated using generated hindsight data instead of on-policy data . To overcome the high variance and instability in KL divergence estimation , another f -divergence is applied to approximate KL divergence , and both theoretically and practically , it is proved to be more efficient and stable . We demonstrate that on several benchmark tasks , HTRPO can significantly improve the performance and sample efficiency in sparse reward scenarios while maintains the learning stability . From the experiments , we illustrate that HTRPO can be neatly applied to not only simple discrete tasks but continuous environments as well . Besides , it is verified that HTRPO can be generalized to different hyperparameter settings with little impact on performance level . 2 PRELIMINARIES . Reinforcement Learning Formulation and Notation . Consider the standard infinite-horizon reinforcement learning formulation which can be defined by tuple ( S , A , π , ρ0 , r , γ ) . S represents the set of states and A denotes the set of actions . π : S → P ( A ) is a policy that represents an agent ’ s behavior by mapping states to a probability distribution over actions . ρ0 denotes the distribution of the initial state s0 . Reward function r : S → R defines the reward obtained from the environment and γ ∈ ( 0 , 1 ) is a discount factor . In this paper , the policy is a differentiable function regarding parameter θ . We follow the standard formalism of state-action value function Q ( s , a ) , state value function V ( s ) and advantage function A ( s , a ) in Sutton & Barto ( 2018 ) . We also adopt the definition of γ-discounted state visitation distribution as ρθ ( s ) = ( 1 − γ ) ∑∞ t=0 γ tP ( st = s ) ( Ho et al. , 2016 ) , in which the coefficient 1−γ is added to keep the integration of ρθ ( s ) as 1 . Correspondingly , γ-discounted state-action visitation distribution ( Ho et al. , 2016 ) , also known as occupancy measure ( Ho & Ermon , 2016 ) , is defined as ρθ ( s , a ) = ρθ ( s ) × πθ ( a|s ) , in which πθ ( a|s ) stands for the policy under parameter θ . Trust Region Policy Optimization ( TRPO ) . Schulman et al . ( 2015a ) proposes an iterative trust region method that effectively optimizes policy by maximizing the per-iteration policy improvement . The optimization problem proposed in TRPO can be formalized as follows : max θ LTRPO ( θ ) ( 1 ) s.t . E s∼ρθ̃ ( s ) [ DKL ( πθ̃ ( a|s ) ||πθ ( a|s ) ) ] ≤ ( 2 ) in which ρθ̃ ( s ) = ∑∞ t=0 γ tP ( st = s ) . θ denotes the parameter of the new policy while θ̃ is that of the old one . Trajectory is represented by τ = s1 , a1 , s2 , a2 , .... The objective function LTRPO ( θ ) can be given out in the form of expeted return : LTRPO ( θ ) = E s , a∼ρθ̃ ( s , a ) [ πθ ( a|s ) πθ̃ ( a|s ) Aθ̃ ( s , a ) ] ( 3 ) Hindsight Policy Gradient ( HPG ) . After generalizing the concept of hindsight , Rauber et al . ( 2019 ) combines the idea with policy gradient methods . Though goal-conditioned reinforcement learning has been explored for a long time and actively investigated in recent works ( Peters & Schaal , 2008 ; Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Nachum et al. , 2018 ; Held et al. , 2018 ; Nair et al. , 2018 ; Veeriah et al. , 2018 ) , HPG firstly extends the idea of hindsight to goal-conditioned policy gradient and shows that the policy gradient can be computed in expectation over all goals . The goal-conditioned policy gradient is derived as follows : ∇θη ( θ ) = E g [ E τ∼pθ ( τ |g ) [ T−1∑ t=1 ∇θ log πθ ( at | st , g ) Aθ ( st , at , g ) ] ] ( 4 ) Then , by applying hindsight formulation , it rewrites goal-conditioned policy gradient with trajectories conditioned on some other goal g′ using importance sampling ( Bishop , 2016 ) to improve sample efficiency in sparse-reward scenarios . In this paper , we propose an approach that introduces the idea of hindsight to TRPO , called Hindsight Trust Region Policy Optimization ( HTRPO ) , aiming to further improve policy performance and sample efficiency for reinforcement learning with sparse rewards . In Section 3 and Section 4 , we demonstrate how to redesign the objective function and the constraints starting from TRPO respectively . 3 EXPECTED RETURN AND POLICY GRADIENTS OF HTRPO . In order to apply hindsight methodology , this section presents the main steps for the derivation of HTRPO objective function . Starting from the original optimization problem in TRPO , the objective function can be written in the following variant form : Lθ̃ ( θ ) = E τ∼pθ̃ ( τ ) [ ∞∑ t=0 γt πθ ( at|st ) πθ̃ ( at|st ) Aθ̃ ( st , at ) ] ( 5 ) The derivation process of this variant form is shown explicitly in Appendix A.1 and in Schulman et al . ( 2015a ) . Given the expression above , we consider the goal-conditioned objective function of TRPO as a premise for hindsight formulation . Similar to equation 4 , Lθ̃ ( θ ) can be correspondingly given out in the following form : Lθ̃ ( θ ) = E g [ E τ∼pθ̃ ( τ |g ) [ ∞∑ t=0 γt πθ ( at|st , g ) πθ̃ ( at|st , g ) Aθ̃ ( st , at , g ) ] ] ( 6 ) For the record , though it seems that equation 6 makes it possible for off-policy learning , it can be used as the objective only when policy πθ is close to the old policy πθ̃ , i.e . within the trust region . Using severely off-policy data like hindsight experience will make the learning process diverge . Therefore , importance sampling need to be integrated to correct the difference of the trajectory distribution caused by changing the goal . Based on the goal-conditioned form of the objective function , the following theorem gives out the hindsight objective function conditioned on some goal g′ with the distribution correction derived from importance sampling . Theorem 3.1 ( HTRPO Objective Function ) . For the original goal g and an alternative goal g′ , the object function of HTRPO Lθ̃ ( θ ) is given by : Lθ̃ ( θ ) = E g′ [ E τ∼pθ ( τ |g ) [ ∞∑ t=0 t∏ k=1 πθ̃ ( ak|sk , g′ ) πθ̃ ( ak|sk , g ) γt πθ ( at|st , g′ ) πθ̃ ( at|st , g′ ) Aθ̃ ( st , at , g ′ ) ] ] , ( 7 ) in which , τ = s1 , a1 , s2 , a2 , ... , st , at . Appendix A.2 presents an explicit proof on how the hindsight-form objective function derives from equation 6 . It will be solved under a KL divergence expectation constraint , which will be discussed in detail in Section 4 . Intuitively , equation 7 provides a way to compute the expected return in terms of the advantage with new-goal-conditioned hindsight experiences which are generated from interactions directed by old goals . Naturally , Theorem 3.2 gives out the gradient of HTRPO objective function that will be applied to solve the optimization problem . Detailed steps of computing the gradient is presented in Appendix A.3 . Theorem 3.2 ( Gradient of HTRPO Objective Function ) . For the original goal g and an alternative goal g′ , the gradient∇θLθ̃ ( θ ) of HTRPO object function with respect to θ is given by the following expression : ∇θLθ̃ ( θ ) = E g′ [ E τ∼pθ ( τ |g ) [ ∞∑ t=0 t∏ k=1 πθ̃ ( ak|sk , g′ ) πθ̃ ( ak|sk , g ) γt ∇θπθ ( at|st , g′ ) πθ̃ ( at|st , g′ ) Aθ̃ ( st , at , g ′ ) ] ] , ( 8 ) in which τ = s1 , a1 , s2 , a2 , ... , st , at .

Summary: The paper builds on top of prior work in hindsight policy gradients (Rauber et.al.) and trust region policy optimization (Schulman et.al.), proposing a hindsight trust region policy optimization. Conceptually this direction makes a lot of sense, since hindsight is in general shown to be useful when training goal conditioned policies. The formulation generally  appears to be sound and is a straightforward extension of the importance sampling techniques from Rabuer et.al. to the TRPO setting. Experimental results show that the proposed changes bring significant improvements over baselines on sparse reward settings.