A Strong On-Policy Competitor To PPO

1 INTRODUCTION . With the development of deep reinforcement learning , lots of impressive results have been produced in a wide range of fields such as playing Atari game ( Mnih et al. , 2015 ; Hessel et al. , 2018 ) , controlling robotics ( Lillicrap et al. , 2015 ) , Go ( Silver et al. , 2017 ) , neural architecture search ( Tan et al. , 2019 ; Pham et al. , 2018 ) . The basis of a reinforcement learning algorithm is generalized policy iteration ( Sutton & Barto , 2018 ) , which states two essential iterative steps : policy evaluation and improvement . Among various algorithms , policy gradient is an active branch of reinforcement learning whose foundations are Policy Gradient Theorem and the most classical algorithm REINFORCEMENT ( Sutton & Barto , 2018 ) . Since then , handfuls of policy gradient variants have been proposed , such as Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2015 ) , Asynchronous Advantage Actor-Critic ( A3C ) ( Mnih et al. , 2016 ) , Actor-Critic using Kronecker-factored Trust Region ( ACKTR ) ( Wu et al. , 2017 ) , and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) . Improving the strategy monotonically had been nontrivial until Schulman et al . ( 2015 ) proposed Trust Region Policy Optimization ( TRPO ) , in which Fisher vector product is utilized to cut down the computing burden . Specifically , Kullback–Leibler divergence ( KLD ) acts as a hard constraint in place of objective , because its corresponding coefficient is difficult to set for different problems . However , TRPO still has several drawbacks : too complicated , inefficient data usage . Quite a lot of efforts have been devoted to improving TRPO since then and the most commonly used one is PPO . PPO can be regarded as a first-order variant of TRPO and have obvious improvements in several facets . In particular , a pessimistic clipped surrogate objective is proposed where TRPO ’ s hard constraint is replaced by the clipped action probability ratio . In such a way , it constructs an unconstrained optimization problem so that any first-order stochastic gradient optimizer can be directly applied . Besides , it ’ s easier to be implemented and more robust against various problems , achieving an impressive result on Atari games ( Brockman et al. , 2016 ) . However , the cost of data sampling is not always cheap . Haarnoja et al . ( 2018 ) design an off-policy algorithm called Soft Actor-Critic and achieves the state of the art result by encouraging better exploration using maximum entropy . In this paper , we focus on the on-policy improvement to improve PPO and answer the question : how to successfully leverage penalized optimization to solve the constrained one which is formulated by Schulman et al . ( 2015 ) . 1 . It proposes a simple variant of TRPO called POP3D along with a new surrogate objective containing a point probability penalty item , which is symmetric lower bound to the square of the total variance divergence of policy distributions . Specifically , it helps to stabilize the learning process and encourage exploration . Furthermore , it escapes from penalty item setting headache along with penalized version TRPO , where is arduous to select one fixed value for various environments . 2 . It achieves state-of-the-art results among on-policy algorithms with a clear margin on 49 Atari games within 40 million frame steps based on two shared metrics . Moreover , it also achieves competitive results compared with PPO in the continuous domain . It dives into the mechanism of PPO ’ s improvement over TRPO from the perspective of solution manifold , which also plays an important role in our method . 3 . It enjoys almost all PPO ’ s advantages such as easy implementation , fast learning ability . We provide the code and training logs to make our work reproducible . 2 PRELIMINARY KNOWLEDGE AND RELATED WORK . 2.1 POLICY GRADIENT . Agents interact with the environment and receive rewards which are used to adjust their policy in turn . At state st , one agent takes strategy π and transfers to a new state st+1 , rewarded rt by the environment . Maximizing discounted return ( accumulated rewards ) Rt is its objective . In particular , given a policy π , Rt is defined as Rt = ∞∑ n=0 ( rt + γrt+1 + γ 2rt+2 + ... + γ nrt+n ) . ( 1 ) γ is the discounted coefficient to control future rewards , which lies in the range ( 0 , 1 ) . Regarding a neural network with parameter θ , the policy πθ ( a|s ) can be learned by maximizing Equation 1 using the back-propagation algorithm . Particularly , given Q ( s , a ) which represents the agent ’ s return in state s after taking action a , the objective function can be written as max θ Es , a log πθ ( a|s ) Q ( s , a ) . ( 2 ) Equation 2 lays the foundation for handfuls of policy gradient based algorithms . Another variant can be deduced by using A ( s , a ) = Q ( s , a ) − V ( s ) ( 3 ) to replace Q ( s , a ) in Equation 2 equivalently , V ( s ) can be any function so long as V depends on s but not a . In most cases , state value function is used for V , which not only helps to reduce variations but has clear physical meaning . Formally , it can be written as max θ Es , a log πθ ( a|s ) A ( s , a ) . ( 4 ) 2.2 ADVANTAGE ESTIMATE . A commonly used method for advantage calculation is one-step estimation , which follows A ( st , at ) = Q ( st , at ) − V ( st ) = rt + γV ( st+1 ) − V ( st ) . ( 5 ) However , a more accurate method called generalized advantage estimation is proposed in Schulman et al . ( 2016 ) , where all time steps of estimation are combined and summarized using λ-based weights , . The generalized advantage estimator ÂGAE ( γ , λ ) t is defined by Schulman et al . ( 2016 ) as Â GAE ( γ , λ ) t : = ( 1− λ ) ∗ ( Â ( 1 ) t + λÂ ( 2 ) t + λ 2Â ( 3 ) t + . . . ) = ∞∑ l=0 ( γλ ) lδVt+l δVt+l = rt+l + γV ( st+l+1 ) − V ( st+l ) . Â ( k ) t : = k−1∑ l=0 γlδVt+l = −V ( st ) + rt + γrt+1 + · · ·+ γk−1rt+k−1 + γkV ( st+k ) ( 6 ) The parameter λ meets 0 ≤ λ ≤ 1 , which controls the trade-off between bias and variance . All methods in this paper utilize ÂGAE ( γ , λ ) t to estimate the advantage . 2.3 TRUST REGION POLICY OPTIMIZATION . Schulman et al . ( 2015 ) propose TRPO to update the policy monotonically . In particular , its mathematical form is max θ Et [ πθ ( at|st ) πθold ( at|st ) Ât ] − CEt [ KL [ πθold ( ·|st ) , πθ ( ·|st ) ] ] = max s Ea∼πθ ( a|s ) [ Aπθold ( s , a ) ] ) ( 7 ) where C is the penalty coefficient , C = 2 γ ( 1−γ ) 2 . In practice , the policy update steps would be too small if C is valued as Equation 7 . In fact , it ’ s intractable to calculate C beforehand since it requires traversing all states to reach the maximum . Moreover , inevitable bias and variance will be introduced by estimating the advantages of old policy while training . Instead , a surrogate objective is maximized based on the KLD constraint between the old and new policy , which can be written as below , max θ Et [ πθ ( at|st ) πθold ( at|st ) Ât ] s.t . Et [ KL [ πθold ( ·|st ) , πθ ( ·|st ) ] ] ≤ δ ( 8 ) where δ is the KLD upper limitation . In addition , the conjugate gradient algorithm is applied to solve Equation 8 more efficiently . Two major problems have yet to be addressed : one is its complexity even using the conjugate gradient approach , another is compatibility with architectures that involve noise or parameter sharing tricks ( Schulman et al. , 2017 ) . 2.4 PROXIMAL POLICY OPTIMIZATION . To overcome the shortcomings of TRPO , PPO replaces the original constrained problem with a pessimistic clipped surrogate objective where KL constraint is implicitly imposed . The loss function can be written as LCLIP ( θ ) = Et [ min ( rt ( θ ) Ât , clip ( rt ( θ ) , 1− , 1 + ) Ât ) ] rt ( θ ) = πθ ( at|st ) πθold ( at|st ) , ( 9 ) where is a hyper-parameter to control the clipping ratio . Except for the clipped PPO version , KL penalty versions including fixed and adaptive KLD . Besides , their simulation results convince that clipped PPO performs best with an obvious margin across various domains . 3 POLICY OPTIMIZATION WITH PENALIZED POINT PROBABILITY DISTANCE . Before diving into the details of POP3D , we review some drawbacks of several methods , which partly motivate us . 3.1 DISADVANTAGES OF KULLBACK-LEIBLER DIVERGENCE . TRPO ( Schulman et al. , 2015 ) induced the following inequality1 , η ( πθ ) ≤ Lπθold ( πθ ) + 2 γ ( 1− γ ) 2 α2 α = DmaxTV ( πθold , πθ ) DmaxTV ( πθold , πθ ) = maxs DTV ( πθold ||πθ ) ( 10 ) TRPO replaces the square of total variation divergence DmaxTV ( πθold , πθ ) by D max KL ( πθold , πθ ) = maxsDKL ( πθold ||πθ ) . 1Note that η means loss instead of return as the ICML version ( Schulman et al. , 2015 ) . Given a discrete distribution p and q , their total variation divergence DTV ( p||q ) is defined as DTV ( p||q ) : = 1 2 ∑ i |pi − qi| ( 11 ) in TRPO ( Schulman et al. , 2015 ) . Obviously , DTV is symmetric by definition , while KLD is asymmetric . Formally , given state s , KLD of πθold ( ·|s ) for πθ ( ·|s ) can be written as DKL ( πθold ( ·|s ) ||πθ ( ·|s ) ) : = ∑ a πθold ( a|s ) ln πθold ( a|s ) πθ ( a|s ) . ( 12 ) Similarly , KLD in the continuous domain can be defined simply by replacing summation with integration . The consequence of KLD ’ s asymmetry leads to a non-negligible difference of whether choose DKL ( πθold ||πθ ) or DKL ( πθ||πθold ) . Sometimes , those two choices result in quite different solutions . Robert compared the forward and reverse KL on a distribution , one solution matches only one of the modes , and another covers both modes ( Murphy , 2012 ) . Therefore , KLD is not an ideal bound or approximation for the expected discounted cost .

Summary: This paper introduces POP3D, an on-policy policy gradient algorithm that is a variant of TRPO and PPO. While TRPO uses a particular penalty function to keep the policy from being updated too aggressively, POP3D uses an alternative objective function that lower bounds the square of the total variance divergence between two policy distributions. The authors argue that this alternative formulation results in an algorithm that is sample-efficient, like PPO, but that is more effective at keeping policy updates from overshooting. The authors also argue that this new formulation helps users to avoid the arguably challenging process of selecting penalty constants, as required (for instance) by TRPO.