Ranking Policy Gradient
Sample inefficiency is a long-lasting problem in reinforcement learning. The state-of-the-art uses action value function to derive policy while it usually involves an extensive search over the state-action space and unstable optimization.Towards the sample-efficient RL, we propose ranking policy gradient, a policy gradient method that learns the optimal rank of a set of discrete actions. To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles.These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency.We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.
We propose ranking policy gradient that learns the optimal rank of actions to maximize return. We propose a general off-policy learning framework with the properties of optimality preserving, variance reduction, and sample-efficiency.This paper proposes to reparameterize the policy using a form of ranking to convert the RL problem into a supervised learning problem.This paper presents a new view on policy gradient methods from the perspective of ranking. 