Hinge Policy Optimization: Rethinking Policy Improvement and Reinterpreting PPO

1 INTRODUCTION . Reinforcement learning ( RL ) has served as a powerful framework for achieving optimal sequential decision making by directly interacting with the environment and learning from the underlying random process . Policy optimization , as a fundamental design principle of RL algorithms , iteratively searches for an optimal policy by alternating between a policy evaluation step and a policy improvement subroutine . Most of the popular policy optimization approaches , including the policy gradient methods ( Sutton et al. , 1999 ; Mnih et al. , 2016 ; Silver et al. , 2014 ; Lillicrap et al. , 2016 ) , Trust Region Policy Optimization ( Schulman et al. , 2015 ) , and Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) , aim to achieve policy improvement in terms of the expected total reward via gradient-based parameter updates . Despite the empirical success of the above approaches , these policy improvement schemes inherently lead to the following fundamental issues : ( i ) Global convergence is usually difficult to establish due to the non-convexity of the underlying objective function ( e.g. , total discounted reward ) and the use of gradient methods , even for the tabular cases ; ( ii ) The policy update direction for improvement is tightly coupled with the true discounted state visitation distribution , which is usually intractable to obtain and needs to be addressed via sampling in practice . To address the above issues , we propose to rethink policy improvement in RL from the perspective of state-wise policy improvement , which aims for improvement directly in the policy space through a partial ordering of the policies , as adopted by policy iteration for the optimal control of Markov decision processes ( MDPs ) . We start by identifying two useful sufficient conditions for state-wise policy improvement and thereafter pinpoint that improvement can be achieved based solely on the sign of the advantage function . Based on this insight , we propose Hinge Policy Optimization ( HPO ) by connecting state-wise policy improvement to solving a large-margin classification problem , where we regard the process of policy improvement as training a binary classifier with hinge loss via empirical risk minimization . As policy improvement of HPO is not enforced in the expected total reward but in a state-wise manner , the policy update of HPO is completely decoupled from the state visitation distribution , which is by contrast required by many existing popular policy optimization methods . By leveraging various types of classifiers , the proposed HPO framework opens up a whole new family of policy-based RL algorithms . Interestingly , the popular PPO algorithm with a surrogate clipped objective ( PPO-clip ) can be shown to be a special case of HPO with a specific type of classifier . Given the policy improvement property of the proposed classification-based scheme , we are able to establish global convergence to an optimal policy for the HPO algorithms . To the best of our knowledge , our analysis provides the first global convergence guarantee for a variant of PPO-clip . This paper is meant to provide a clear picture of the RL algorithms in the new HPO family with global convergence guarantees and thereby introduce a new perspective for reinterpreting the popular PPO-clip algorithm . The main contributions of this paper can be summarized as follows : • We propose HPO , a new policy optimization framework where the policy update is built on statewise policy improvement via hinge loss . The members in the HPO family share a generic loss function , and their differences lie in the choice of the margin and the classifier . We also show that the widely-used PPO-clip algorithm can be viewed as a special case in this family . • This paper is the first ever that can prove state-wise policy improvement and global convergence for a variant of PPO-clip algorithm1 . Specifically , we first present a variant of the PPO algorithm with an adaptive clipped objective , which can be viewed as an HPO algorithm with adaptive margin , called HPO-AM . We prove the convergence to an optimal policy for HPO-AM , and proceed to generalize the global convergence result to some other HPO-AM like algorithms equipped with other classifiers . We also empirically validate the proposed theoretical framework through experiments and thereby corroborate the performance of various HPO algorithms with different classifiers and margins ( with the experimental results provided in Appendices E and F ) . 2 MAIN RESULTS . In this section , we first review the theoretical background about RL and view PPO-clip update as a classification problem . Then , we establish global convergence for the proposed HPO-AM . 2.1 BACKGROUND . A discounted Markov Decision Process ( MDP ) is defined by the tuple ( S , A , P , r , γ ) , where S is a finite state space , A is a finite action space , P : S ×A×S → [ 0 , 1 ] is the state transition probability matrix , r : S × A → R is the reward function , and γ ∈ ( 0 , 1 ) is the discounted factor . A stochastic policy π : S → ∆ ( A ) specifies the action distribution based on the current state , where ∆ ( A ) is the probability simplex over A , i.e. , ∆ ( A ) = { x ∈ R|A| | x0 + ... + x|A|−1 = 1 , xi ≥ 0 , ∀i = 0 , ... , |A| − 1 } . Given any policy π , the state value function V π , the state-action value function Qπ , and the advantage function Aπ are defined as follows . V π ( s ) : = E at∼π ( ·|st ) , st+1∼P ( ·|st , at ) [ ∞∑ t=0 γtr ( st , at ) ∣∣∣s0 = s ] , ( 1 ) Qπ ( s , a ) : = E at∼π ( ·|st ) , st+1∼P ( ·|st , at ) [ ∞∑ t=0 γtr ( st , at ) ∣∣∣s0 = s , a0 = a ] , ( 2 ) Aπ ( s , a ) : = Qπ ( s , a ) − V π ( s ) . ( 3 ) Define dπs ( s ′ ) : = ( 1 − γ ) ∑∞ t=0 γ t Pr ( st = s ′|s0 = s , π ) as the normalized discounted state visitation frequency , which represents the probability of visiting state s′ in a trajectory of π , given that s0 = s. With the above definition , ( Kakade & Langford , 2002 ) quantified the difference in performance between two policies as follows . Given policies π1 and π2 , V π1 ( s ) − V π2 ( s ) = 1 1− γ ∑ s′∈S dπ1s ( s ′ ) ∑ a∈A π1 ( a|s′ ) Aπ2 ( s′ , a ) . ( 4 ) 1Regarding the convergence of PPO , ( Liu et al. , 2019 ) proves global convergence in expected total reward for a neural variant of PPO with adaptive Kullback-Leibler penalty ( PPO-KL ) . Given the salient algorithmic difference between PPO-KL and PPO-clip , to the best of our knowledge , there remains no proof of global convergence to an optimal policy for PPO-clip . The TRPO algorithm proceeds to maximize the expected value of surrogate function of ( 4 ) over the initial state distribution under the constraint of KL divergence . State-wise policy improvement is formalized based on the following partial ordering relation . Definition 1 ( Partial ordering over policies ) . Let π1 and π2 be two policies . Then , π1 ≥ π2 , called π1 improves upon π2 , if and only if V π1 ( s ) ≥ V π2 ( s ) , ∀s ∈ S. Moreover , we say π1 > π2 , called π1 strictly improves upon π2 , if and only if π1 ≥ π2 and there exists at least one state s such that V π1 ( s ) > V π2 ( s ) . Definition 2 ( An optimal policy ) . A policy π∗ is said to be an optimal policy if π∗ ≥ π′ , for every policy π′ . Moreover , we let V ∗ ( s ) denote the optimal value of each state s , i.e. , V ∗ ( s ) = V π ∗ ( s ) . We first introduce two sufficient conditions for state-wise policy improvement in Propositions 1-2 . These two conditions have been identified by ( Hu et al. , 2020 , Section 3.1 ) . Proposition 1 . Given policies π1 and π2 , π1 improves upon π2 if the following condition holds : ∑ a∈A π1 ( a|s ) Aπ2 ( s , a ) ≥ 0 , ∀s ∈ S. ( 5 ) Proposition 2 . Given policies π1 and π2 , π1 improves upon π2 if the following condition holds : ( π1 ( a|s ) − π2 ( a|s ) ) Aπ2 ( s , a ) ≥ 0 , ∀ ( s , a ) ∈ S ×A . ( 6 ) Proposition 1 holds for the following reason : Since all dπ1s ( s ′ ) are non-negative , all values in ( 4 ) are non-negative , and hence π1 improves upon π2 . Proposition 2 can be derived directly from Proposition 1 and the fact that ∑ a∈A π2 ( a|s ) Aπ2 ( s , a ) = 0 , ∀s ∈ S. Notably , Proposition 2 offers a useful insight that state-wise policy improvement can be achieved by determining the sign of the advantage of each state-action pair ( regardless of its magnitude ) and adjusting the action probabilities accordingly . In this way , no additional constraints , such as the KL divergence constraint used in TRPO ( Schulman et al. , 2015 ) , are needed to ensure policy improvement . This also naturally motivates the design of using the signs of the advantage function as labels in determining the direction of the policy update . More specifically , we can draw an analogy between ( 6 ) in Proposition 2 and the training of a linear classifier : ( i ) The state-action pair serves as the feature vector of a training sample ; ( ii ) The sign ofAπ2 ( s , a ) plays the role of a binary label ; ( iii ) π1 ( a|s ) − π2 ( a|s ) resembles the prediction of a linear classifier . In the next section , we substantiate this insight and present the proposed HPO framework . In the rest of this paper , our analysis relies on the following assumptions : Assumption 1 ( Bounded reward ) . To avoid trivial cases , we assume not all rewards are zero . Since both state and action spaces are finite , it holds naturally that there exists a positive constant R = sup ( s , a ) ∈S×A|r ( s , a ) | > 0 . Assumption 2 ( Tabular policies ) . Policies are parameterized by π ( a|s ) = θs , a , where θs ∈ ∆ ( A ) refers to the vector of θs.· for some fixed state s , and θ ∈ ∆ ( A ) |S| , i.e. , θ is subject to θs , a ≥ 0 and∑ a∈A θs , a = 1 , ∀s ∈ S , ∀a ∈ A . Notations . Throughout this paper , we let 〈a , b〉 and a◦ b denote the inner product and the Hadamard product of two real vectors a , b , respectively .

Summary: The paper proposes hinge policy optimization, a new theoretical framework for interpreting policy gradient algorithms as classification problems to be solved with a hinge loss. In this perspective, the sign of the advantage function becomes the label, and the difference in action probabilities between policy after and before an update becomes the classifier's output. The paper shows the equivalence between such a formulation and the popular PPO-clip objective and provides global converge guarantees on the blueprint of (Agarwal, 2020). It also proposes a range of policy optimization algorithms, depending on the details of the classification algorithm that is used, and empirically evaluates some of them.