Generalized Proximal Policy Optimization with Sample Reuse

1 Introduction . In recent years , model-free deep reinforcement learning has been used to successfully solve complex simulated control tasks [ 4 ] . Unfortunately , real-world adoption of these techniques remains limited . High-stakes real-world decision making settings demand methods that deliver stable , reliable performance throughout training . In addition , real-world data collection can be difficult and expensive , so learning must make efficient use of limited data . The combination of these requirements is not an easy task , as stability and sample efficiency often represent competing interests . Existing model-free deep reinforcement learning algorithms often focus on one of these goals , and as a result sacrifice performance with respect to the other . On-policy reinforcement learning methods such as Proximal Policy Optimization ( PPO ) [ 19 ] deliver stable performance throughout training due to their connection to theoretical policy improvement guarantees . These methods are motivated by a lower bound on the expected performance loss at every update , which can be approximated using samples generated by the current policy . The theoretically supported stability of these methods is very attractive , but the need for on-policy data and the highvariance nature of reinforcement learning often requires significant data to be collected between every update , resulting in high sample complexity and slow learning . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Off-policy algorithms address the issue of high sample complexity by storing samples in a replay buffer , which allows data to be reused to calculate multiple policy updates . The ability to reuse samples improves learning speed , but also causes the distribution of data to shift away from the distribution generated by the current policy . This distribution shift invalidates the standard performance guarantees used in on-policy methods , and can lead to instability in the training process . Popular off-policy algorithms often require various implementation tricks and extensive hyperparameter tuning to control the instability caused by off-policy data . By combining the attractive features of on-policy and off-policy methods in a principled way , we can balance the competing goals of stability and sample efficiency required in real-world decision making . We consider the popular on-policy algorithm PPO as our starting point due to its theoretically supported stable performance , and develop an off-policy variant with principled sample reuse that we call Generalized Proximal Policy Optimization with Sample Reuse ( GePPO ) . Our algorithm is based on the following main contributions : 1 . We extend existing policy improvement guarantees to the off-policy setting , resulting in a lower bound that can be approximated using data from all recent policies . 2 . We develop connections between the clipping mechanism used in PPO and the penalty term in our policy improvement lower bound , which motivates a generalized clipping mechanism for off-policy data . 3 . We propose an adaptive learning rate method based on the same penalty term that more closely connects theory and practice . We provide theoretical evidence that our algorithm effectively balances the goals of stability and sample efficiency , and we demonstrate the strong performance of our approach through experiments on high-dimensional continuous control tasks in OpenAI Gym ’ s MuJoCo environments [ 3 , 21 ] . 2 Related work . On-policy policy improvement methods The goal of monotonic policy improvement was first introduced by Kakade and Langford [ 14 ] in Conservative Policy Iteration , which maximizes a lower bound on policy improvement that can be constructed using samples from the current policy . This theory of policy improvement has served as a fundamental building block in the design of on-policy deep reinforcement learning methods , including the popular algorithms Trust Region Policy Optimization ( TRPO ) [ 17 ] and Proximal Policy Optimization ( PPO ) [ 19 ] . TRPO achieves approximate policy improvement by enforcing a Kullback-Leibler ( KL ) divergence trust region , while PPO does so by clipping the probability ratio between current and future policies . Due to the strong performance of TRPO and PPO , there has been substantial interest in better understanding these methods . Engstrom et al . [ 5 ] and Andrychowicz et al . [ 2 ] both performed extensive empirical analysis on the various implementation choices in these algorithms , while other research has focused on the clipping mechanism used in PPO . Wang et al . [ 22 ] and Wang et al . [ 23 ] both proposed modifications to the clipping mechanism based on a KL divergence trust region . To the best of our knowledge , we are the first to directly relate the clipping mechanism in PPO to the total variation distance between policies . Wang et al . [ 23 ] also proposed a rollback operation to keep probability ratios close to the clipping range . We accomplish a similar goal by considering an adaptive learning rate . Sample efficiency with off-policy data A common approach to improving the sample efficiency of model-free reinforcement learning is to reuse samples collected under prior policies . Popular off-policy policy gradient approaches such as Deep Deterministic Policy Gradient ( DDPG ) [ 15 ] , Twin Delayed DDPG ( TD3 ) [ 8 ] , and Soft Actor-Critic ( SAC ) [ 11 ] accomplish this by storing data in a replay buffer and sampling from this buffer to calculate policy updates . Note that these methods are not motivated by policy improvement guarantees , and do not directly control the bias introduced by off-policy data . Other approaches have combined on-policy and off-policy policy gradients , with the goal of balancing the variance of on-policy methods and the bias of off-policy methods [ 7 , 9 , 10 , 16 , 24 ] . Gu et al . [ 10 ] demonstrated that the bias introduced by off-policy data is related to the KL divergence between the current policy and the behavior policy used to generate the data . Fakoor et al . [ 7 ] considered a related KL divergence as a penalty term in their objective , while Wang et al . [ 24 ] approximately controlled this KL divergence by applying a trust region around a target policy . These methods are related to the penalty term that appears in our generalized policy improvement lower bound , which can be bounded by a penalty that depends on KL divergence . Finally , there have been heuristic attempts to incorporate off-policy data into PPO [ 13 , 20 ] . However , unlike our approach , these methods do not account for the distribution shift caused by off-policy data that invalidates the theoretical support for PPO . 3 Preliminaries . Reinforcement learning framework We consider an infinite-horizon , discounted Markov Decision Process ( MDP ) defined by the tuple ( S , A , p , r , ρ0 , γ ) , where S is the set of states , A is the set of actions , p : S×A → ∆S is the transition probability function , r : S×A → R is the reward function , ρ0 is the initial state distribution , and γ is the discount rate . We model the agent ’ s decisions as a stationary policy π : S → ∆A . Our goal is to choose a policy that maximizes the expected total discounted rewards J ( π ) = Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) ] , where τ ∼ π represents a trajectory sampled according to s0 ∼ ρ0 , at ∼ π ( · | st ) , and st+1 ∼ p ( · | st , at ) . A policy π also induces a normalized discounted state visitation distribution dπ , where dπ ( s ) = ( 1− γ ) ∑∞ t=0 γ tP ( st = s | ρ0 , π , p ) . We write the corresponding normalized discounted state-action visitation distribution as dπ ( s , a ) = dπ ( s ) π ( a | s ) , where we make it clear from the context whether dπ refers to a distribution over states or state-action pairs . We denote the state value function of π as V π ( s ) = Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) | s0 = s ] , the state- action value function as Qπ ( s , a ) = Eτ∼π [ ∑∞ t=0 γ tr ( st , at ) | s0 = s , a0 = a ] , and the advantage function as Aπ ( s , a ) = Qπ ( s , a ) − V π ( s ) . Policy improvement lower bound The starting point in the design of many popular on-policy algorithms is the following policy improvement lower bound , which was first developed by Kakade and Langford [ 14 ] and later refined by Schulman et al . [ 17 ] and Achiam et al . [ 1 ] : Lemma 1 ( Achiam et al . [ 1 ] ) . Consider a current policy πk . For any future policy π , we have J ( π ) − J ( πk ) ≥ 1 1− γ E ( s , a ) ∼dπk [ π ( a | s ) πk ( a | s ) Aπk ( s , a ) ] − 2γC π , πk ( 1− γ ) 2 Es∼dπk [ TV ( π , πk ) ( s ) ] , ( 1 ) where Cπ , πk = maxs∈S ∣∣Ea∼π ( ·|s ) [ Aπk ( s , a ) ] ∣∣ and TV ( π , πk ) ( s ) represents the total variation distance between the distributions π ( · | s ) and πk ( · | s ) . We refer to the first term of the lower bound in Lemma 1 as the surrogate objective , and the second term as the penalty term . Note that we can guarantee policy improvement at every step of the learning process by choosing the next policy πk+1 to maximize this lower bound . Because the expectations in Lemma 1 depend on the current policy πk , we can approximate this lower bound using samples generated by the current policy . Proximal Policy Optimization PPO , which has become the default on-policy policy optimization algorithm due to its strong performance and simple implementation , is theoretically motivated by the policy improvement lower bound in Lemma 1 . Rather than directly maximizing this lower bound , PPO considers the goal of maximizing the surrogate objective while constraining the next policy to be close to the current policy . In particular , PPO heuristically accomplishes this by considering the following objective at every policy update : LPPOk ( π ) = E ( s , a ) ∼dπk [ min ( π ( a | s ) πk ( a | s ) Aπk ( s , a ) , clip ( π ( a | s ) πk ( a | s ) , 1− ϵ , 1 + ϵ ) Aπk ( s , a ) ) ] , ( 2 ) where clip ( x , l , u ) = min ( max ( x , l ) , u ) . As seen in the second term of this objective , PPO constrains the difference between consecutive policies by removing the incentive for the probability ratio π ( a|s ) /πk ( a|s ) to leave the clipping range [ 1− ϵ , 1+ ϵ ] . Finally , the outer minimization guarantees that ( 2 ) is a lower bound to the surrogate objective in Lemma 1 . In practice , ( 2 ) is approximated using samples generated by the current policy πk , and the resulting empirical objective is approximately optimized at every policy update using minibatch stochastic gradient ascent . For a sufficiently small learning rate and sufficiently large number of samples , PPO results in stable policy improvement throughout the learning process . However , it is well-known that high variance is a major issue in reinforcement learning , so often the number of samples must be large in order for the empirical objective to be an accurate estimator of the true objective ( 2 ) . Because these samples must be collected under the current policy between every policy update , PPO can be very sample intensive .

Summary: This work generalizes PPO to the off-policy setting. First the policy improvement bound from Achiam et al. (2017) was generalized to accomodate for several prior policies. The authors also made a connection between the total variation distance and the PPO clipping term. A new algorithm GePPO is proposed which is shown to be more sample efficient compared to standard PPO.