Hinge Policy Optimization: Rethinking Policy Improvement and Reinterpreting PPO

1 INTRODUCTION . Reinforcement learning ( RL ) has served as a powerful framework for achieving optimal sequential decision making by directly interacting with the environment and learning from the underlying random process . Policy optimization , as a fundamental design principle of RL algorithms , iteratively searches for an optimal policy by alternating between a policy evaluation step and a policy improvement subroutine . Most of the popular policy optimization approaches , including the policy gradient methods ( Sutton et al. , 1999 ; Mnih et al. , 2016 ; Silver et al. , 2014 ; Lillicrap et al. , 2016 ) , Trust Region Policy Optimization ( Schulman et al. , 2015 ) , and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) , aim to achieve policy improvement in terms of the expected total reward via gradient-based parameter updates . Despite the empirical success of the above approaches , these policy improvement schemes inherently lead to the following fundamental issues : ( i ) Global convergence is usually difficult to establish due to the non-convexity of the underlying objective function ( e.g. , total discounted reward ) and the use of gradient methods , even for the tabular cases ; ( ii ) The policy update direction for improvement is tightly coupled with the true discounted state visitation distribution , which is usually intractable to obtain and needs to be addressed via sampling in practice . To address the above issues , we propose to rethink policy improvement in RL from the perspective of state-wise policy improvement , which aims for improvement directly in the policy space through a partial ordering of the policies , as adopted by policy iteration for the optimal control of Markov decision processes ( MDPs ) . We start by identifying two useful sufficient conditions for state-wise policy improvement and thereafter pinpoint that improvement can be achieved based solely on the sign of the advantage function . Based on this insight , we propose Hinge Policy Optimization ( HPO ) by connecting state-wise policy improvement to solving a large-margin classification problem , where we regard the process of policy improvement as training a binary classifier with hinge loss via empirical risk minimization . As policy improvement of HPO is not enforced in the expected total reward but in a state-wise manner , the policy update of HPO is completely decoupled from the state visitation distribution , which is by contrast required by many existing popular policy optimization methods . By leveraging various types of classifiers , the proposed HPO framework opens up a whole new family of policy-based RL algorithms . Interestingly , the popular PPO algorithm with a surrogate clipped objective ( PPO-clip ) can be shown to be a special case of HPO with a specific type of classifier . Given the policy improvement property of the proposed classification-based scheme , we are able to establish global convergence to an optimal policy for the HPO algorithms . To the best of our knowledge , our analysis provides the first global convergence guarantee for a variant of PPO-clip . This paper is meant to provide a clear picture of the RL algorithms in the new HPO family with global convergence guarantees and thereby introduce a new perspective for reinterpreting the popular PPO-clip algorithm . The main contributions of this paper can be summarized as follows : • We propose HPO , a new policy optimization framework where the policy update is built on statewise policy improvement via hinge loss . The members in the HPO family share a generic loss function , and their differences lie in the choice of the margin and the classifier . We also show that the widely-used PPO-clip algorithm can be viewed as a special case in this family . • This paper is the first ever that can prove state-wise policy improvement and global convergence for a variant of PPO-clip algorithm1 . Specifically , we first present a variant of the PPO algorithm with an adaptive clipped objective , which can be viewed as an HPO algorithm with adaptive margin , called HPO-AM . We prove the convergence to an optimal policy for HPO-AM , and proceed to generalize the global convergence result to some other HPO-AM like algorithms equipped with other classifiers . We also empirically validate the proposed theoretical framework through experiments and thereby corroborate the performance of various HPO algorithms with different classifiers and margins ( with the experimental results provided in Appendices E and F ) . 2 MAIN RESULTS . In this section , we first review the theoretical background about RL and view PPO-clip update as a classification problem . Then , we establish global convergence for the proposed HPO-AM . 2.1 BACKGROUND . A discounted Markov Decision Process ( MDP ) is defined by the tuple ( S , A , P , r , γ ) , where S is a finite state space , A is a finite action space , P : S ×A×S → [ 0 , 1 ] is the state transition probability matrix , r : S × A → R is the reward function , and γ ∈ ( 0 , 1 ) is the discounted factor . A stochastic policy π : S → ∆ ( A ) specifies the action distribution based on the current state , where ∆ ( A ) is the probability simplex over A , i.e. , ∆ ( A ) = { x ∈ R|A| | x0 + ... + x|A|−1 = 1 , xi ≥ 0 , ∀i = 0 , ... , |A| − 1 } . Given any policy π , the state value function V π , the state-action value function Qπ , and the advantage function Aπ are defined as follows . V π ( s ) : = E at∼π ( ·|st ) , st+1∼P ( ·|st , at ) [ ∞∑ t=0 γtr ( st , at ) ∣∣∣s0 = s ] , ( 1 ) Qπ ( s , a ) : = E at∼π ( ·|st ) , st+1∼P ( ·|st , at ) [ ∞∑ t=0 γtr ( st , at ) ∣∣∣s0 = s , a0 = a ] , ( 2 ) Aπ ( s , a ) : = Qπ ( s , a ) − V π ( s ) . ( 3 ) Define dπs ( s ′ ) : = ( 1 − γ ) ∑∞ t=0 γ t Pr ( st = s ′|s0 = s , π ) as the normalized discounted state visitation frequency , which represents the probability of visiting state s′ in a trajectory of π , given that s0 = s. With the above definition , ( Kakade & Langford , 2002 ) quantified the difference in performance between two policies as follows . Given policies π1 and π2 , V π1 ( s ) − V π2 ( s ) = 1 1− γ ∑ s′∈S dπ1s ( s ′ ) ∑ a∈A π1 ( a|s′ ) Aπ2 ( s′ , a ) . ( 4 ) 1Regarding the convergence of PPO , ( Liu et al. , 2019 ) proves global convergence in expected total reward for a neural variant of PPO with adaptive Kullback-Leibler penalty ( PPO-KL ) . Given the salient algorithmic difference between PPO-KL and PPO-clip , to the best of our knowledge , there remains no proof of global convergence to an optimal policy for PPO-clip . The TRPO algorithm proceeds to maximize the expected value of surrogate function of ( 4 ) over the initial state distribution under the constraint of KL divergence . State-wise policy improvement is formalized based on the following partial ordering relation . Definition 1 ( Partial ordering over policies ) . Let π1 and π2 be two policies . Then , π1 ≥ π2 , called π1 improves upon π2 , if and only if V π1 ( s ) ≥ V π2 ( s ) , ∀s ∈ S. Moreover , we say π1 > π2 , called π1 strictly improves upon π2 , if and only if π1 ≥ π2 and there exists at least one state s such that V π1 ( s ) > V π2 ( s ) . Definition 2 ( An optimal policy ) . A policy π∗ is said to be an optimal policy if π∗ ≥ π′ , for every policy π′ . Moreover , we let V ∗ ( s ) denote the optimal value of each state s , i.e. , V ∗ ( s ) = V π ∗ ( s ) . We first introduce two sufficient conditions for state-wise policy improvement in Propositions 1-2 . These two conditions have been identified by ( Hu et al. , 2020 , Section 3.1 ) . Proposition 1 . Given policies π1 and π2 , π1 improves upon π2 if the following condition holds : ∑ a∈A π1 ( a|s ) Aπ2 ( s , a ) ≥ 0 , ∀s ∈ S. ( 5 ) Proposition 2 . Given policies π1 and π2 , π1 improves upon π2 if the following condition holds : ( π1 ( a|s ) − π2 ( a|s ) ) Aπ2 ( s , a ) ≥ 0 , ∀ ( s , a ) ∈ S ×A . ( 6 ) Proposition 1 holds for the following reason : Since all dπ1s ( s ′ ) are non-negative , all values in ( 4 ) are non-negative , and hence π1 improves upon π2 . Proposition 2 can be derived directly from Proposition 1 and the fact that ∑ a∈A π2 ( a|s ) Aπ2 ( s , a ) = 0 , ∀s ∈ S. Notably , Proposition 2 offers a useful insight that state-wise policy improvement can be achieved by determining the sign of the advantage of each state-action pair ( regardless of its magnitude ) and adjusting the action probabilities accordingly . In this way , no additional constraints , such as the KL divergence constraint used in TRPO ( Schulman et al. , 2015 ) , are needed to ensure policy improvement . This also naturally motivates the design of using the signs of the advantage function as labels in determining the direction of the policy update . More specifically , we can draw an analogy between ( 6 ) in Proposition 2 and the training of a linear classifier : ( i ) The state-action pair serves as the feature vector of a training sample ; ( ii ) The sign ofAπ2 ( s , a ) plays the role of a binary label ; ( iii ) π1 ( a|s ) − π2 ( a|s ) resembles the prediction of a linear classifier . In the next section , we substantiate this insight and present the proposed HPO framework . In the rest of this paper , our analysis relies on the following assumptions : Assumption 1 ( Bounded reward ) . To avoid trivial cases , we assume not all rewards are zero . Since both state and action spaces are finite , it holds naturally that there exists a positive constant R = sup ( s , a ) ∈S×A|r ( s , a ) | > 0 . Assumption 2 ( Tabular policies ) . Policies are parameterized by π ( a|s ) = θs , a , where θs ∈ ∆ ( A ) refers to the vector of θs.· for some fixed state s , and θ ∈ ∆ ( A ) |S| , i.e. , θ is subject to θs , a ≥ 0 and∑ a∈A θs , a = 1 , ∀s ∈ S , ∀a ∈ A . Notations . Throughout this paper , we let 〈a , b〉 and a◦ b denote the inner product and the Hadamard product of two real vectors a , b , respectively .

Summary:  This paper reinterprets the theory of PPO-clip based on the hinge policy optimization. They prove the global convergence of PPO by introducing some assumptions. Besides, they generalize the algorithm to a new family of policy-based algorithms by regarding the policy as a generalized classifier.