Ranking Policy Gradient

1 INTRODUCTION . One of the major challenges in reinforcement learning ( RL ) is the high sample complexity ( Kakade et al. , 2003 ) , which is the number of samples must be collected to conduct successful learning . There are different reasons leading to poor sample efficiency of RL ( Yu , 2018 ) . Because policy gradient algorithms directly optimizing return estimated from rollouts ( e.g. , REINFORCE ( Williams , 1992 ) ) could suffer from high variance ( Sutton & Barto , 2018 ) , value function baselines were introduced by actor-critic methods to reduce the variance and improve the sample-efficiency . However , since a value function is associated with a certain policy , the samples collected by former policies can not be readily used without complicated manipulations ( Degris et al. , 2012 ) and extensive parameter tuning ( Nachum et al. , 2017 ) . Such an on-policy requirement increases the difficulty of sample-efficient learning . On the other hand , off-policy methods , such as one-step Q-learning ( Watkins & Dayan , 1992 ) and variants of deep Q networks ( DQN ) ( Mnih et al. , 2015 ; Hessel et al. , 2017 ; Dabney et al. , 2018 ; Van Hasselt et al. , 2016 ; Schaul et al. , 2015 ) , enjoys the advantage of learning from any trajectory sampled from the same environment ( i.e. , off-policy learning ) , are currently among the most sampleefficient algorithms . These algorithms , however , often require extensive searching ( Bertsekas & Tsitsiklis , 1996 , Chap . 5 ) over the large state-action space to estimate the optimal action value function . Another deficiency is that , the combination of off-policy learning , bootstrapping , and function approximation , making up what Sutton & Barto ( 2018 ) called the `` deadly triad '' , can easily lead to unstable or even divergent learning ( Sutton & Barto , 2018 , Chap . 11 ) . These inherent issues limit their sample-efficiency . Towards addressing the aforementioned challenge , we approach the sample-efficient reinforcement learning from a ranking perspective . Instead of estimating optimal action value function , we concentrate on learning optimal rank of actions . The rank of actions depends on the relative action values . As long as the relative action values preserve the same rank of actions as the optimal action values ( Q-values ) , we choose the same optimal action . To learn optimal relative action values , we propose the ranking policy gradient ( RPG ) that optimizes the actions ’ rank with respect to the long-term reward by learning the pairwise relationship among actions . Ranking Policy Gradient ( RPG ) that directly optimizes relative action values to maximize the return is a policy gradient method . The track of off-policy actor-critic methods ( Degris et al. , 2012 ; Gu et al. , 2016 ; Wang et al. , 2016 ) have made substantial progress on improving the sample-efficiency of policy gradient . However , the fundamental difficulty of learning stability associated with the bias-variance trade-off remains ( Nachum et al. , 2017 ) . In this work , we first exploit the equivalence between RL optimizing the lower bound of return and supervised learning that imitates a specific optimal policy . Build upon this theoretical foundation , we propose a general off-policy learning framework that equips the generalized policy iteration ( Sutton & Barto , 2018 , Chap . 4 ) with an external step of supervised learning . The proposed off-policy learning not only enjoys the property of optimality preserving ( unbiasedness ) , but also largely reduces the variance of policy gradient because of its independence of the horizon and reward scale . Besides , we empirically show that there is a trade-off between optimality and sample-efficiency . Last but not least , we demonstrate that the proposed approach , consolidating the RPG with off-policy learning , significantly outperforms the state-of-the-art ( Hessel et al. , 2017 ; Bellemare et al. , 2017 ; Dabney et al. , 2018 ; Mnih et al. , 2015 ) . 2 RELATED WORKS . Sample Efficiency . The sample efficient reinforcement learning can be roughly divided into two categories . The first category includes variants of Q-learning ( Mnih et al. , 2015 ; Schaul et al. , 2015 ; Van Hasselt et al. , 2016 ; Hessel et al. , 2017 ) . The main advantage of Q-learning methods is the use of off-policy learning , which is essential towards sample efficiency . The representative DQN ( Mnih et al. , 2015 ) introduced deep neural network in Q-learning , which further inspried a track of successful DQN variants such as Double DQN ( Van Hasselt et al. , 2016 ) , Dueling networks ( Wang et al. , 2015 ) , prioritized experience replay ( Schaul et al. , 2015 ) , and RAINBOW ( Hessel et al. , 2017 ) . The second category is the actor-critic approaches . Most of recent works ( Degris et al. , 2012 ; Wang et al. , 2016 ; Gruslys et al. , 2018 ) in this category leverage importance sampling by re-weighting the samples to correct the estimation bias and reduce variance . Its main advantage is in the wall-clock times due to the distributed framework , firstly presented in ( Mnih et al. , 2016 ) , instead of the sample-efficiency . As of the time of writing , the variants of DQN ( Hessel et al. , 2017 ; Dabney et al. , 2018 ; Bellemare et al. , 2017 ; Schaul et al. , 2015 ; Van Hasselt et al. , 2016 ) are among the algorithms of most sample efficiency , which are adopted as our baselines for comparison . RL as supervised learning . Numerous amount of works have developed the connections between RL and supervised learning such as Expectation-Maximization algorithms ( Dayan & Hinton , 1997 ; Peters & Schaal , 2007 ; Kober & Peters , 2009 ; Abdolmaleki et al. , 2018 ) , Entropy-Regularized RL ( Oh et al. , 2018 ; Haarnoja et al. , 2018 ) , and Interactive Imitation Learning ( IIL ) ( Daumé et al. , 2009 ; Syed & Schapire , 2010 ; Ross & Bagnell , 2010 ; Ross et al. , 2011 ; Sun et al. , 2017 ; Hester et al. , 2018 ; Osa et al. , 2018 ) . EM-based approaches utilize the probabilistic framework to transfer RL maximizing lower bound of return as a re-weighted regression problem while it requires on-policy estimation on the expectation step . Entropy-Regularized RL optimizing entropy augmented objectives can lead to off-policy learning without the usage of importance sampling while it converges to soft optimality ( Haarnoja et al. , 2018 ) . Of the three tracks in prior works , the IIL is most closely related to our work . The IIL works firstly pointed out the connection between imitation learning and reinforcement learning ( Ross & Bagnell , 2010 ; Syed & Schapire , 2010 ; Ross et al. , 2011 ) and explore the idea of facilitating reinforcement learning by imitating experts . However , most of imitation learning algorithms assume the access to the expert policy or demonstrations . Our off-policy learning framework can be interpreted as an online imitation learning approach that constructs expert demonstrations during the exploration without soliciting experts , and conducts supervised learning to maximize return at the same time . In conclusion , our approach is different from the prior work in terms of at least one of the following aspects : objectives , oracle assumptions , the optimality of learned policy , and on-policy requirement . More concretely , the proposed method is able to learn both deterministic and stochastic optimal policy in terms of long-term reward , without access to the oracle ( such as expert policy or expert demonstration ) and it can be trained both empirically and theoretically in an off-policy fashion . Due to the space limits , we defer the detailed discussion of the related work in the Appendix Section 10.1 . 3 NOTATIONS AND PROBLEM SETTING . In this paper , we consider a finite horizon T , discrete time Markov Decision Process ( MDP ) with a finite discrete state space S and for each state s ∈ S , the action space As is finite . The environment dynamics is denoted as P = { p ( s′|s , a ) , ∀s , s′ ∈ S , a ∈ As } . We note that the dimension of action space can vary given different states . We use m = maxs ‖As‖ to denote the maximal action dimension among all possible states . Our goal is to maximize the expected sum of rewards , or return J ( θ ) = Eτ , πθ [ ∑T t=1 r ( st , at ) ] , where |r ( s , a ) | < ∞ , ∀s , a . In this case , the optimal deterministic Markovian policy always exists ( Puterman , 2014 ) [ Proposition 4.4.3 ] . The upper bound of trajectory reward ( r ( τ ) ) is denoted as Rmax = maxτ r ( τ ) . A comprehensive list of notations are elaborated in the Appendix Table 1 . 4 RANKING POLICY GRADIENT . Value function estimation is widely used in advanced RL algorithms ( Mnih et al. , 2015 ; 2016 ; Schulman et al. , 2017 ; Gruslys et al. , 2018 ; Hessel et al. , 2017 ; Dabney et al. , 2018 ) to facilitate the learning process . In practice , the on-policy requirement of value function estimations in actorcritic methods has largely increased the difficulty of sample-efficient learning ( Degris et al. , 2012 ; Gruslys et al. , 2018 ) . With the advantage of off-policy learning , the DQN ( Mnih et al. , 2015 ) variants are currently among the most sample-efficient algorithms ( Hessel et al. , 2017 ; Dabney et al. , 2018 ; Bellemare et al. , 2017 ) . For complicated tasks , the value function can align with the relative relationship of action ’ s return , but the absolute values are hardly accurate ( Mnih et al. , 2015 ; Ilyas et al. , 2018 ) . The above observations motivate us to look at the decision phase of RL from a different prospect : Given a state , the decision making is to perform a relative comparison over available actions and then choose the best action , which can lead to relatively higher return than others . Therefore , an alternative solution is to learn the optimal rank of the actions . In this section , we show how to optimize the rank of actions to maximize the return , and thus avoid the necessity of accurate estimation for optimal action value function . To learn the rank of actions , we focus on learning relative action value ( λ-values ) , defined as follows : Definition 1 ( Relative action value ( λ-values ) ) . For a state s , the relative action values of m actions ( λ ( s , ak ) , k = 1 , ... , m ) is a list of scores that denotes the rank of actions . If λ ( s , ai ) > λ ( s , aj ) , then action ai is ranked higher than action aj . The optimal relative action values should preserve the same optimal action as the optimal action values : arg max a λ ( s , a ) = arg max a Qπ∗ ( s , a ) where Qπ∗ ( s , ai ) and λ ( s , ai ) represent the optimal action value and the relative action value of action ai , respectively . We omit the model parameter θ in λθ ( s , ai ) for concise presentation . Remark 1 . The λ-values are different from the advantage function Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . The advantage functions quantitatively show the difference of return taking different actions following the current policy π . The λ-values only determine the relative order of actions and its magnitudes are not the estimations of returns . To learn the λ-values , we can construct a probabilistic model of λ-values such that the best action has the highest probability to be selected than others . Inspired by learning to rank ( Burges et al. , 2005 ) , we consider the pairwise relationship among all actions , by modeling the probability ( denoted as pij ) of an action ai to be ranked higher than any action aj as follows : pij = exp ( λ ( s , ai ) − λ ( s , aj ) ) 1 + exp ( λ ( s , ai ) − λ ( s , aj ) ) , ( 1 ) where pij = 0.5 means the relative action value of ai is same as that of the action aj , pij > 0.5 indicates that the action ai is ranked higher than aj . Given the independent Assumption 1 , we can represent the probability of selecting one action as the multiplication of a set of pairwise probabilities in Eq ( 1 ) . Formally , we define the pairwise ranking policy in Eq ( 2 ) . Please refer to Section 10.10 in the Appendix for the discussions on feasibility of Assumption 1 . Definition 2 . The pairwise ranking policy is defined as : π ( a = ai|s ) = Πmj=1 , j 6=i pij , ( 2 ) where the pij is defined in Eq ( 1 ) . The probability depends on the relative action values q = [ λ1 , ... , λm ] . The highest relative action value leads to the highest probability to be selected . Assumption 1 . For a state s , the set of events E = { eij |∀i 6= j } are conditionally independent , where eij denotes the event that action ai is ranked higher than action aj . The independence of the events is conditioned on a MDP and a stationary policy . Our ultimate goal is to maximize the long-term reward through optimizing the pairwise ranking policy or equivalently optimizing pairwise relationship among the action pairs . Ideally , we would like the pairwise ranking policy selects the best action with the highest probability and the highest λ-value . To achieve this goal , we resort to the policy gradient method . Formally , we propose the ranking policy gradient method ( RPG ) , as shown in Theorem 1 . Theorem 1 ( Ranking Policy Gradient Theorem ) . For any MDP , the gradient of the expected longterm reward J ( θ ) = ∑ τ pθ ( τ ) r ( τ ) w.r.t . the parameter θ of a pairwise ranking policy ( Def 2 ) can be approximated by : ∇θJ ( θ ) ≈ Eτ∼πθ [ ∑T t=1 ∇θ ( ∑m j=1 , j 6=i ( λi − λj ) /2 ) r ( τ ) ] , ( 3 ) and the deterministic pairwise ranking policy πθ is : a = arg maxi λi , i = 1 , . . . , m , where λi denotes the relative action value of action ai ( λθ ( st , at ) , ai = at ) , st and at denotes the t-th stateaction pair in trajectory τ , λj , ∀j 6= i denote the relative action values of all other actions that were not taken given state st in trajectory τ , i.e. , λθ ( st , aj ) , ∀aj 6= at . The proof of Theorem 1 is available in Appendix Section 10.2 . Theorem 1 states that optimizing the discrepancy between the relative action values of the best action and all other actions , is optimizing the pairwise relationships that maximize the return . One limitation of RPG is that it is not convenient for the tasks where only optimal stochastic policies exist since the pairwise ranking policy takes extra efforts to construct a probability distribution [ see Section 10.3 in Appendix ] . In order to learn the stochastic policy , we introduce Listwise Policy Gradient ( LPG ) that optimizes the probability of ranking a specific action on the top of a set of actions , with respect to the return . In the context of RL , this top one probability is the probability of action ai to be chosen , which is equal to the sum of probability all possible permutations that map action ai in the top . Inspired by listwise learning to rank approach ( Cao et al. , 2007 ) , the top one probability can be modeled by the softmax function . Therefore , LPG is equivalent to the REINFORCE ( Williams , 1992 ) algorithm with a softmax layer . LPG provides another interpretation of REINFORCE algorithm from the perspective of learning the optimal ranking and enables the learning of both deterministic policy and stochastic policy . Due to the space limit , we defer the detailed description of LPG in Appendix Section 10.4 . To this end , seeking sample-efficiency motivates us to learn the relative relationship ( RPG ( Theorem 1 ) and LPG ( Theorem 4 ) ) of actions , instead of seeking accurate estimation of optimal action values and then choosing action greedily . However , both of the RPG and LPG belong to policy gradient methods , which suffers from large variance and the on-policy learning requirement ( Sutton & Barto , 2018 ) . Therefore , the direct implementation of RPG or LPG is still far from sample-efficient . In the next section , we will describe a general off-policy learning framework empowered by supervised learning , which provides an alternative way to accelerate learning , preserve optimality , and reduce variance .

Summary: This paper presents a new view on policy gradient methods from the perspective of ranking. The end goal in policy learning is to achieve the right ranking of actions at a state (in the case when deterministic policies are optimal), and the paper proposes a method of doing this inspired from the work on learning to rank. They further argue that in the case with stochastic optimal policies, REINFORCE with softmax policies is rank wise optimal, which is not surprising, but at the same time interesting as well. The other main part of the paper is casting off-policy RL as supervised learning similar to work on self-imitation learning and reward weighted regression methods. This section is presented differently from the past analyses of self-imitation methods and requires the existence of UNOP, which seems like a strong assumption. They then instantiate the framework with GPI based exploration, and show that it achieves better performance than IQN and Rainbow on a subset of atari games.