Supervised Policy Update for Deep Reinforcement Learning

1 INTRODUCTION . The policy gradient problem in deep reinforcement learning ( DRL ) can be defined as seeking a parameterized policy with high expected reward . An issue with policy gradient methods is poor sample efficiency ( Kakade , 2003 ; Schulman et al. , 2015a ; Wang et al. , 2016b ; Wu et al. , 2017 ; Schulman et al. , 2017 ) . In algorithms such as REINFORCE ( Williams , 1992 ) , new samples are needed for every gradient step . When generating samples is expensive ( such as robotic environments ) , sample efficiency is of central concern . The sample efficiency of an algorithm is defined to be the number of calls to the environment required to attain a specified performance level ( Kakade , 2003 ) . Thus , given the current policy and a fixed number of trajectories ( samples ) generated , the goal of the sample efficiency problem is to construct a new policy with the highest performance improvement possible . To do so , it is desirable to limit the search to policies that are close to the original policy πθk ( Kakade , 2002 ; Schulman et al. , 2015a ; Wu et al. , 2017 ; Achiam et al. , 2017 ; Schulman et al. , 2017 ; Tangkaratt et al. , 2018 ) . Intuitively , if the candidate new policy πθ is far from the original policy πθk , it may not perform better than the original policy because too much emphasis is being placed on the relatively small batch of new data generated by πθk , and not enough emphasis is being placed on the relatively large amount of data and effort previously used to construct πθk . This guideline of limiting the search to nearby policies seems reasonable in principle , but requires a distance η ( πθ , πθk ) between the current policy πθk and the candidate new policy πθ , and then attempt to solve the constrained optimization problem : maximize θ Ĵ ( πθ | πθk , new data ) ( 1 ) subject to η ( πθ , πθk ) ≤ δ ( 2 ) where Ĵ ( πθ | πθk , new data ) is an estimate of J ( πθ ) , the performance of policy πθ , based on the previous policy πθk and the batch of fresh data generated by πθk . The objective ( 1 ) attempts to maximize the performance of the updated policy , and the constraint ( 2 ) ensures that the updated policy is not too far from the policy πθk that was used to generate the data . Several recent papers ( Kakade , 2002 ; Schulman et al. , 2015a ; 2017 ; Tangkaratt et al. , 2018 ) belong to the framework ( 1 ) - ( 2 ) . Our work also strikes the right balance between performance and simplicity . The implementation is only slightly more involved than PPO ( Schulman et al. , 2017 ) . Simplicity in RL algorithms has its own merits . This is especially useful when RL algorithms are used to solve problems outside of traditional RL testbeds , which is becoming a trend ( Zoph & Le , 2016 ; Mingxing Tan , 2018 ) . We propose a new methodology , called Supervised Policy Update ( SPU ) , for this sample efficiency problem . The methodology is general in that it applies to both discrete and continuous action spaces , and can address a wide variety of constraint types for ( 2 ) . Starting with data generated by the current policy , SPU optimizes over a proximal policy space to find an optimal non-parameterized policy . It then solves a supervised regression problem to convert the non-parameterized policy to a parameterized policy , from which it draws new samples . We develop a general methodology for finding an optimal policy in the non-parameterized policy space , and then illustrate the methodology for three different definitions of proximity . We also show how the Natural Policy Gradient and Trust Region Policy Optimization ( NPG/TRPO ) problems and the Proximal Policy Optimization ( PPO ) problem can be addressed by this methodology . While SPU is substantially simpler than NPG/TRPO in terms of mathematics and implementation , our extensive experiments show that SPU is more sample efficient than TRPO in Mujoco simulated robotic tasks and PPO in Atari video game tasks . Off-policy RL algorithms generally achieve better sample efficiency than on-policy algorithms ( Haarnoja et al. , 2018 ) . However , the performance of an on-policy algorithm can usually be substantially improved by incorporating off-policy training ( Mnih et al . ( 2015 ) , Wang et al . ( 2016a ) ) . Our paper focuses on igniting interests in separating finding the optimal policy into a two-step process : finding the optimal non-parameterized policy , and then parameterizing this optimal policy . We also wanted to deeply understand the on-policy case before adding off-policy training . We thus compare with algorithms operating under the same algorithmic constraints , one of which is being on-policy . We leave the extension to off-policy to future work . We do not claim state-of-the-art results . 2 PRELIMINARIES . We consider a Markov Decision Process ( MDP ) with state space S , action space A , and reward function r ( s , a ) , s ∈ S , a ∈ A . Let π = { π ( a|s ) : s ∈ S , a ∈ A } denote a policy , let Π be the set of all policies , and let the expected discounted reward be : J ( π ) , E τ∼π [ ∞∑ t=0 γtr ( st , at ) ] ( 3 ) where γ ∈ ( 0 , 1 ) is a discount factor and τ = ( s0 , a0 , s1 , . . . ) is a sample trajectory . Let Aπ ( s , a ) be the advantage function for policy π ( Levine , 2017 ) . Deep reinforcement learning considers a set of parameterized policies ΠDL = { πθ|θ ∈ Θ } ⊂ Π , where each policy is parameterized by a neural network called the policy network . In this paper , we will consider optimizing over the parameterized policies in ΠDL as well as over the non-parameterized policies in Π . For concreteness , we assume that the state and action spaces are finite . However , our methodology also applies to continuous state and action spaces , as shown in the Appendix . One popular approach to maximizing J ( πθ ) over ΠDL is to apply stochastic gradient ascent . The gradient of J ( πθ ) evaluated at a specific θ = θk can be shown to be ( Williams , 1992 ) : ∇θJ ( πθk ) = Eτ∼πθk [ ∞∑ t=0 γt∇θ log πθk ( at|st ) Aπθk ( st , at ) ] . ( 4 ) We can approximate ( 4 ) by sampling N trajectories of length T from πθk : ∇θJ ( πθk ) ≈ 1 N N∑ i=1 T−1∑ t=0 ∇θ log πθk ( ait|sit ) Aπθk ( st , at ) , gk ( 5 ) Additionally , define dπ ( s ) , ( 1 − γ ) ∑∞ t=0 γ tPπ ( st = s ) for the the future state probability distribution for policy π , and denote π ( ·|s ) for the probability distribution over the action space A when in state s and using policy π . Further denote DKL ( π ‖ πθk ) [ s ] for the KL divergence from π ( ·|s ) to πθk ( ·|s ) , and denote the following as the “ aggregated KL divergence ” . D̄KL ( π ‖ πθk ) = E s∼dπθk [ DKL ( π ‖ πθk ) [ s ] ] ( 6 ) 2.1 SURROGATE OBJECTIVES FOR THE SAMPLE EFFICIENCY PROBLEM . For the sample efficiency problem , the objective J ( πθ ) is typically approximated using samples generated from πθk ( Schulman et al. , 2015a ; Achiam et al. , 2017 ; Schulman et al. , 2017 ) . Two different approaches are typically used to approximate J ( πθ ) − J ( πθk ) . We can make a first order approximation of J ( πθ ) around θk ( Kakade , 2002 ; Peters & Schaal , 2008a ; b ; Schulman et al. , 2015a ) : J ( πθ ) − J ( πθk ) ≈ ( θ − θk ) T∇θJ ( πθk ) ≈ ( θ − θk ) T gk ( 7 ) where gk is the sample estimate ( 5 ) . The second approach is to approximate the state distribution dπ ( s ) with dπθk ( s ) ( Achiam et al. , 2017 ; Schulman et al. , 2017 ; Achiam , 2017 ) : J ( π ) − J ( πθk ) ≈ Lπθk ( π ) , 1 1− γ E s∼dπθk E a∼πθk ( ·|s ) [ π ( a|s ) πθk ( a|s ) Aπθk ( s , a ) ] ( 8 ) There is a well-known bound for the approximation ( 8 ) ( Kakade & Langford , 2002 ; Achiam et al. , 2017 ) . Furthermore , the approximation Lπθk ( πθ ) matches J ( πθ ) − J ( πθk ) to the first order with respect to the parameter θ ( Achiam et al. , 2017 ) . 3 RELATED WORK . Natural gradient ( Amari , 1998 ) was first introduced to policy gradient by Kakade ( Kakade , 2002 ) and then in ( Peters & Schaal , 2008a ; b ; Achiam , 2017 ; Schulman et al. , 2015a ) . referred to collectively here as NPG/TRPO . Algorithmically , NPG/TRPO finds the gradient update by solving the sample efficiency problem ( 1 ) - ( 2 ) with η ( πθ , πθk ) = D̄KL ( πθ ‖ πθk ) , i.e. , use the aggregate KL-divergence for the policy proximity constraint ( 2 ) . NPG/TRPO addresses this problem in the parameter space θ ∈ Θ . First , it approximates J ( πθ ) with the first-order approximation ( 7 ) and D̄KL ( πθ ‖ πθk ) using a similar second-order method . Second , it uses samples from πθk to form estimates of these two approximations . Third , using these estimates ( which are functions of θ ) , it solves for the optimal θ∗ . The optimal θ∗ is a function of gk and of hk , the sample average of the Hessian evaluated at θk . TRPO also limits the magnitude of the update to ensure D̄KL ( πθ ‖ πθk ) ≤ δ ( i.e. , ensuring the sampled estimate of the aggregated KL constraint is met without the second-order approximation ) . SPU takes a very different approach by first ( i ) posing and solving the optimization problem in the non-parameterized policy space , and then ( ii ) solving a supervised regression problem to find a parameterized policy that is near the optimal non-parameterized policy . A recent paper , Guided Actor Critic ( GAC ) , independently proposed a similar decomposition ( Tangkaratt et al. , 2018 ) . However , GAC is much more restricted in that it considers only one specific constraint criterion ( aggregated reverse-KL divergence ) and applies only to continuous action spaces . Furthermore , GAC incurs significantly higher computational complexity , e.g . at every update , it minimizes the dual function to obtain the dual variables using SLSQP . MPO also independently propose a similar decomposition ( Abbas Abdolmaleki , 2018 ) . MPO uses much more complex machinery , namely , Expectation Maximization to address the DRL problem . However , MPO has only demonstrates preliminary results on problems with discrete actions whereas our approach naturally applies to problems with either discrete or continuous actions . In both GAC and MPO , working in the non-parameterized space is a by-product of applying the main ideas in those papers to DRL . Our paper demonstrates that the decomposition alone is a general and useful technique for solving constrained policy optimization . Clipped-PPO ( Schulman et al. , 2017 ) takes a very different approach to TRPO . At each iteration , PPO makes many gradient steps while only using the data from πθk . Without the clipping , PPO is the approximation ( 8 ) . The clipping is analogous to the constraint ( 2 ) in that it has the goal of keeping πθ close to πθk . Indeed , the clipping keeps πθ ( at|st ) from becoming neither much larger than ( 1 + ) πθk ( at|st ) nor much smaller than ( 1 − ) πθk ( at|st ) . Thus , although the clipped PPO objective does not squarely fit into the optimization framework ( 1 ) - ( 2 ) , it is quite similar in spirit . We note that the PPO paper considers adding the KL penalty to the objective function , whose gradient is similar to ours . However , this form of gradient was demonstrated to be inferior to Clipped-PPO . To the best of our knowledge , it is only until our work that such form of gradient is demonstrated to outperform Clipped-PPO . Actor-Critic using Kronecker-Factored Trust Region ( ACKTR ) ( Wu et al. , 2017 ) proposed using Kronecker-factored approximation curvature ( K-FAC ) to update both the policy gradient and critic terms , giving a more computationally efficient method of calculating the natural gradients . ACER ( Wang et al. , 2016a ) exploits past episodes , linearizes the KL divergence constraint , and maintains an average policy network to enforce the KL divergence constraint . In future work , it would of interest to extend the SPU methodology to handle past episodes . In contrast to bounding the KL divergence on the action distribution as we have done in this work , Relative Entropy Policy Search considers bounding the joint distribution of state and action and was only demonstrated to work for small problems ( Jan Peters , 2010 ) .

Summary: The authors formulate policy optimization as a two step iterative procedure: 1) solving a constrained optimization problem in the non-parameterized policy space, 2) using supervised regression to project this onto a parameterized policy. This approach generally applies to both continuous and discrete action spaces and can handle a variety of constraints. Their primary claims is that this approach improves sample-efficiency over TRPO on Mujoco tasks and over PPO on Atari games.