Hindsight Trust Region Policy Optimization

1 INTRODUCTION . Reinforcement Learning has been a heuristic approach confronting a great many real-world problems from playing complex strategic games ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Justesen et al. , 2019 ) to the precise control of robots ( Levine et al. , 2016 ; Mahler & Goldberg , 2017 ; Quillen et al. , 2018 ) , in which policy gradient methods play very important roles ( Sutton et al. , 2000 ; Deisenroth et al. , 2013 ) . Among them , the ones based on trust region including Trust Region Policy Optimization ( Schulman et al. , 2015a ) and Proximal Policy Optimization ( Schulman et al. , 2017 ) have achieved stable and effective performances on several benchmark tasks . Later on , they have been verified in a variety of applications including skill learning ( Nagabandi et al. , 2018 ) , multi-agent control ( Gupta et al. , 2017 ) , imitation learning ( Ho et al. , 2016 ) , and have been investigated further to be combined with more advanced techniques ( Nachum et al. , 2017 ; Houthooft et al. , 2016 ; Heess et al. , 2017 ) . One unresolved core issue in reinforcement learning is efficiently training the agent in sparse reward environments , in which the agent is given a distinctively high feedback only upon reaching the desired final goal state . On one hand , generalizing reinforcement learning methods to sparse reward scenarios obviates designing delicate reward mechanism , which is known as reward shaping ( Ng et al. , 1999 ) ; on the other hand , receiving rewards only when precisely reaching the final goal states also guarantees that the agent can focus on the intended task itself without any deviation . Despite the extensive use of policy gradient methods , they tend to be vulnerable when dealing with sparse reward scenarios . Admittedly , policy gradient may work in simple and sufficiently rewarding environments through massive random exploration . However , since it relies heavily on the expected return , the chances in complex and sparsely rewarding scenarios become rather slim , which often makes it unfeasible to converge to a policy by exploring randomly . Recently , several works have been devoted to solving the problem of sparse reward , mainly applying either hierarchical reinforcement learning ( Kulkarni et al. , 2016 ; Vezhnevets et al. , 2017 ; Le et al. , 2018 ; Marino et al. , 2019 ) or a hindsight methodology , including Hindsight Experience Replay ( Andrychowicz et al. , 2017 ) , Hindsight Policy Gradient ( Rauber et al. , 2019 ) and their extensions ( Fang et al. , 2019 ; Levy et al. , 2019 ) . The idea of Hindsight Experience Replay ( HER ) is to regard the ending states obtained through the interaction under current policy as alternative goals , and therefore generate more effective training data comparing to that with only real goals . Such augmentation overcomes the defects of random exploration and allows the agent to progressively move towards intended goals . It is proven to be promising when dealing with sparse reward reinforcement learning problems . For Hindsight Policy Gradient ( HPG ) , it introduces hindsight to policy gradient approach and improves sample efficiency in sparse reward environments . Yet , its learning curve for policy update still oscillates considerably . Because it inherits the intrinsic high variance of policy gradient methods which has been widely studied in Schulman et al . ( 2015b ) , Gu et al . ( 2016 ) and Wu et al . ( 2018 ) . Furthermore , introducing hindsight to policy gradient methods would lead to greater variance ( Rauber et al. , 2019 ) . Consequently , such exacerbation would cause obstructive instability during the optimization process . To design an advanced and efficient on-policy reinforcement learning algorithm with hindsight experience , the main problem is the contradiction between on-policy data needed by the training process and the severely off-policy hindsight experience we can get . Moreover , for TRPO , one of the most significant property is the approximated monotonic converging process . Therefore , how these advantages can be preserved when the agent is trained with hindsight data also remains unsolved . In this paper , we propose a methodology called Hindsight Trust Region Policy Optimization ( HTRPO ) . Starting from TRPO , a hindsight form of policy optimization problem within trust region is theoretically derived , which can be approximately solved with the Monte Carlo estimator using severely off-policy hindsight experience data . HTRPO extends the effective and monotonically iterative policy optimization procedure within trust region to accommodate sparse reward environments . In HTRPO , both the objective function and the expectation of KL divergence between policies are estimated using generated hindsight data instead of on-policy data . To overcome the high variance and instability in KL divergence estimation , another f -divergence is applied to approximate KL divergence , and both theoretically and practically , it is proved to be more efficient and stable . We demonstrate that on several benchmark tasks , HTRPO can significantly improve the performance and sample efficiency in sparse reward scenarios while maintains the learning stability . From the experiments , we illustrate that HTRPO can be neatly applied to not only simple discrete tasks but continuous environments as well . Besides , it is verified that HTRPO can be generalized to different hyperparameter settings with little impact on performance level . 2 PRELIMINARIES . Reinforcement Learning Formulation and Notation . Consider the standard infinite-horizon reinforcement learning formulation which can be defined by tuple ( S , A , π , ρ0 , r , γ ) . S represents the set of states and A denotes the set of actions . π : S → P ( A ) is a policy that represents an agent ’ s behavior by mapping states to a probability distribution over actions . ρ0 denotes the distribution of the initial state s0 . Reward function r : S → R defines the reward obtained from the environment and γ ∈ ( 0 , 1 ) is a discount factor . In this paper , the policy is a differentiable function regarding parameter θ . We follow the standard formalism of state-action value function Q ( s , a ) , state value function V ( s ) and advantage function A ( s , a ) in Sutton & Barto ( 2018 ) . We also adopt the definition of γ-discounted state visitation distribution as ρθ ( s ) = ( 1 − γ ) ∑∞ t=0 γ tP ( st = s ) ( Ho et al. , 2016 ) , in which the coefficient 1−γ is added to keep the integration of ρθ ( s ) as 1 . Correspondingly , γ-discounted state-action visitation distribution ( Ho et al. , 2016 ) , also known as occupancy measure ( Ho & Ermon , 2016 ) , is defined as ρθ ( s , a ) = ρθ ( s ) × πθ ( a|s ) , in which πθ ( a|s ) stands for the policy under parameter θ . Trust Region Policy Optimization ( TRPO ) . Schulman et al . ( 2015a ) proposes an iterative trust region method that effectively optimizes policy by maximizing the per-iteration policy improvement . The optimization problem proposed in TRPO can be formalized as follows : max θ LTRPO ( θ ) ( 1 ) s.t . E s∼ρθ̃ ( s ) [ DKL ( πθ̃ ( a|s ) ||πθ ( a|s ) ) ] ≤ ( 2 ) in which ρθ̃ ( s ) = ∑∞ t=0 γ tP ( st = s ) . θ denotes the parameter of the new policy while θ̃ is that of the old one . Trajectory is represented by τ = s1 , a1 , s2 , a2 , .... The objective function LTRPO ( θ ) can be given out in the form of expeted return : LTRPO ( θ ) = E s , a∼ρθ̃ ( s , a ) [ πθ ( a|s ) πθ̃ ( a|s ) Aθ̃ ( s , a ) ] ( 3 ) Hindsight Policy Gradient ( HPG ) . After generalizing the concept of hindsight , Rauber et al . ( 2019 ) combines the idea with policy gradient methods . Though goal-conditioned reinforcement learning has been explored for a long time and actively investigated in recent works ( Peters & Schaal , 2008 ; Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Nachum et al. , 2018 ; Held et al. , 2018 ; Nair et al. , 2018 ; Veeriah et al. , 2018 ) , HPG firstly extends the idea of hindsight to goal-conditioned policy gradient and shows that the policy gradient can be computed in expectation over all goals . The goal-conditioned policy gradient is derived as follows : ∇θη ( θ ) = E g [ E τ∼pθ ( τ |g ) [ T−1∑ t=1 ∇θ log πθ ( at | st , g ) Aθ ( st , at , g ) ] ] ( 4 ) Then , by applying hindsight formulation , it rewrites goal-conditioned policy gradient with trajectories conditioned on some other goal g′ using importance sampling ( Bishop , 2016 ) to improve sample efficiency in sparse-reward scenarios . In this paper , we propose an approach that introduces the idea of hindsight to TRPO , called Hindsight Trust Region Policy Optimization ( HTRPO ) , aiming to further improve policy performance and sample efficiency for reinforcement learning with sparse rewards . In Section 3 and Section 4 , we demonstrate how to redesign the objective function and the constraints starting from TRPO respectively . 3 EXPECTED RETURN AND POLICY GRADIENTS OF HTRPO . In order to apply hindsight methodology , this section presents the main steps for the derivation of HTRPO objective function . Starting from the original optimization problem in TRPO , the objective function can be written in the following variant form : Lθ̃ ( θ ) = E τ∼pθ̃ ( τ ) [ ∞∑ t=0 γt πθ ( at|st ) πθ̃ ( at|st ) Aθ̃ ( st , at ) ] ( 5 ) The derivation process of this variant form is shown explicitly in Appendix A.1 and in Schulman et al . ( 2015a ) . Given the expression above , we consider the goal-conditioned objective function of TRPO as a premise for hindsight formulation . Similar to equation 4 , Lθ̃ ( θ ) can be correspondingly given out in the following form : Lθ̃ ( θ ) = E g [ E τ∼pθ̃ ( τ |g ) [ ∞∑ t=0 γt πθ ( at|st , g ) πθ̃ ( at|st , g ) Aθ̃ ( st , at , g ) ] ] ( 6 ) For the record , though it seems that equation 6 makes it possible for off-policy learning , it can be used as the objective only when policy πθ is close to the old policy πθ̃ , i.e . within the trust region . Using severely off-policy data like hindsight experience will make the learning process diverge . Therefore , importance sampling need to be integrated to correct the difference of the trajectory distribution caused by changing the goal . Based on the goal-conditioned form of the objective function , the following theorem gives out the hindsight objective function conditioned on some goal g′ with the distribution correction derived from importance sampling . Theorem 3.1 ( HTRPO Objective Function ) . For the original goal g and an alternative goal g′ , the object function of HTRPO Lθ̃ ( θ ) is given by : Lθ̃ ( θ ) = E g′ [ E τ∼pθ ( τ |g ) [ ∞∑ t=0 t∏ k=1 πθ̃ ( ak|sk , g′ ) πθ̃ ( ak|sk , g ) γt πθ ( at|st , g′ ) πθ̃ ( at|st , g′ ) Aθ̃ ( st , at , g ′ ) ] ] , ( 7 ) in which , τ = s1 , a1 , s2 , a2 , ... , st , at . Appendix A.2 presents an explicit proof on how the hindsight-form objective function derives from equation 6 . It will be solved under a KL divergence expectation constraint , which will be discussed in detail in Section 4 . Intuitively , equation 7 provides a way to compute the expected return in terms of the advantage with new-goal-conditioned hindsight experiences which are generated from interactions directed by old goals . Naturally , Theorem 3.2 gives out the gradient of HTRPO objective function that will be applied to solve the optimization problem . Detailed steps of computing the gradient is presented in Appendix A.3 . Theorem 3.2 ( Gradient of HTRPO Objective Function ) . For the original goal g and an alternative goal g′ , the gradient∇θLθ̃ ( θ ) of HTRPO object function with respect to θ is given by the following expression : ∇θLθ̃ ( θ ) = E g′ [ E τ∼pθ ( τ |g ) [ ∞∑ t=0 t∏ k=1 πθ̃ ( ak|sk , g′ ) πθ̃ ( ak|sk , g ) γt ∇θπθ ( at|st , g′ ) πθ̃ ( at|st , g′ ) Aθ̃ ( st , at , g ′ ) ] ] , ( 8 ) in which τ = s1 , a1 , s2 , a2 , ... , st , at .

Summary: This paper augments the TRPO policy optimization objective with hindsight data, where the hindsight data is generated from goals based on trajectories. The key contribution of the paper is based on deriving an on-policy adaptation of hindsight based TRPO, that can be useful for sparse reward environments. The paper draws ideas from existing papers such as HPG and considers the IS based variant of HPG for on-policy, similar to TRPO, that can achieve monotonic performance improvements. Furthermore, the authors introduce a logarithmic form of constraint, by re-deriving the KL constraint and leading to a f-divergence based constraint, which is argued to have useful effects in terms of lowering the variance. Experimental results are compared with baselines including HPG and its variants on standard sparse reward benchmark tasks.