Generalized Proximal Policy Optimization with Sample Reuse
Authors: James Queeney
Ioannis Paschalidis
Christos Cassandras
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
policy optimization

Abstract: In real-world decision making tasks, it is critical for data-driven reinforcement learning methods to be both stable and sample efficient. On-policy methods typically generate reliable policy improvement throughout training, while off-policy methods make more efficient use of data through sample reuse. In this work, we combine the theoretically supported stability benefits of on-policy algorithms with the sample efficiency of off-policy algorithms. We develop policy improvement guarantees that are suitable for the off-policy setting, and connect these bounds to the clipping mechanism used in Proximal Policy Optimization. This motivates an off-policy version of the popular algorithm that we call Generalized Proximal Policy Optimization with Sample Reuse. We demonstrate both theoretically and empirically that our algorithm delivers improved performance by effectively balancing the competing goals of stability and sample efficiency.

Generalized Proximal Policy Optimization with Sample Reuse
James Queeney Division of Systems Engineering Boston University jqueeney@bu.cm edu
Ioannis Ch. Paschalidis
Department of Electrical and Computer Engineering Division of Systems Engineering
Boston University yannispābu edu
Christos G. Cassandras
Department of Electrical and Computer Engineering Division of Systems Engineering Boston University cgcêbu. edu
Abstract
In real-world decision making tasks, it is critical for data-driven reinforcement learning methods to be both stable and sample efficient. On-policy methods typi- cally generate reliable policy improvement throughout training, while off- off-policy methods make more efficient use data through sample reuse. In this work, we combine the theoretically supported stability benefits of on-policy algorithms with the sample efficiency of off-policy algorithms. We develop policy improvement guarantees that suitable for the off-policy setting, and connect these bounds to the clipping mechanism used in Proximal Policy Optimization. This motivates off-policy version of the popular algorithm that we call Generalized Proxi- mal Policy Optimization with Sample Reuse. We demonstrate both theoretically and empirically that our algorithm delivers improved performance by effectively balancing the competing goals of stability and sample efficiency.
Introduction
In recent years, model-free deep reinforcement learning has been used to successfully solve complex simulated control tasks [4]. Unfortunately. real-world adoption of these techniques remains limited. High-stakes real-world decision making settings demand methods that deliver stable, reliable perfor- mance throughout training. In addition, real-world data collection can be difficult and expensive, so learning must make efficient use of limited data. The combination of these requirements is not an easy task, as stability and sample efficiency often represent competing interests. Existing model-free deep reinforcement learning algorithms often focus on one of these goals, and as a result sacrifice performance with respect the other.
On-poli On-policy reinforcement learning methods such as Proximal Policy Optimization (PPO) 19 deliver stable performance throughout training due their connection to theoretical policy improvement guarantees. These methods are motivated by a lower bound on the expected performance loss at every update, which can be approximated using samples generated by the current policy. The theoretically supported stability of these methods is very attractive, but the need for on-policy data and the high- variance nature of reinforcement learning often requires significant data be collected between every update, resulting in high sample complexity and slow learning.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Off-policy algorithmss address the issue of high sample complexity by storing samples in replay buffer, which allows data be reused to calculate multiple policy updates. The ability to reuse samples improves learning speed, but also causes the distribution of data to shift away from the distribution generated by the current policy. This distribution shift invalidates the standard performance guarantees used in on-policy methods, and can lead to instability in the training process. Popular off-policy algorithms often require various implementation tricks and extensive hyperparameter tuning to control the instability caused by off-policy data.
By combining the attractive features of on on-policy and off- off-policy methods in a principled way, we can balance the competing goals of stability and sample efficiency required in real-worll decision making. We consider the popular on- on-policy algorithm PPO as our starting point due to its theoretically supported stable performance, and develop an off off-pc variant with principled sample reuse that we call Generalized Proximal Policy Optimization with Sample Reuse (GePPO). Our algorithm is based on the following main contributions:
1. We extend existing policy improvement guarantees the off-policy setting, resulting in lower bound that can be approximated using data from all recent policies.
2. We develop connections between the clipping mechanism used in PPO and the penalty term in our policy improvement lower bound, which motivates a generalized clipping mechanism for off- off-policy data.
3. We propose an adaptive learning rate method based on the same penalty term that more closely connects theory and practice.
We provide theoretical evidence that our algorithm effectively balances the goals of stability and sample efficiency, and we demonstrate the strong performance of our approach through experiments on high-dimensional continuous control tasks in OpenAl Gym's MuJoCo environments 21).
2 Related work
On-policy policy improvement methods The goal of monotonic policy improvement was first introduced by Kakade and Langford [14] in Conservative Policy Iteration, which maximizes lower bound on policy improvement that can be constructed using samples from the current policy. This theory of policy improvement has served
fundamental building block in the design of on-policy deep reinforcement learning methods, including the popular algorithms Trust Region Policy Optimization (TRPO) [17] and Proximal Policy Optimization (PPO) [19]. TRPO achieves approximate policy improvement by enforcing a Kullback-Leible (KL) divergence trust region, while PPO does so by clipping the probability ratio between current and future policies.
Due to the strong performance of TRPO and PPO, there has been substantial interest better understanding these methods. Engstrom et al. [5] and Andrychowicz et al. 2 both performed extensive empirical analysis on the various implementation choices in these algorithms, while other research has focused the clipping mechanism used in PPO. Wang et al. [22 and Wang et al. [23 both proposed modifications to the clipping mechanism based on a KL divergence trust region. To the best of our knowledge, we are the first to directly relate the clipping mechanism in PPO to the total variation distance between policies. Wang et al. [23] also proposed a rollback operation to keep probability ratios close to the clipping range. We accomplish similar goal by considering an adaptive learning rate.
Sample efficiency with off-policy data A common approach to improving the sample efficiency of model-free reinforcement learning is to reuse samples collected under prior policies. Popular off-policy policy gradient approaches such Deep Deterministic Policy Gradient (DDPG) [15), Twin Delayed DDPG (TD3) [8) and Soft Actor-Critic (SAC) [T]] accomplish this by storing data in replay buffer and sampling from this buffer to calculate policy updates. Note that these methods are not motivated by policy improvement guarantees, and do not directly control the bias introduced by off- off-policy data.
Other approaches have combined on-policy and off off-policy policy gradients, with the goal of balancing
the variance of on-policy methods and the bias of off-policy methods (7 9 10 16 24 Gu et al. [I0] demonstrated that the bias introduced by off-policy data is related to the KI. divergence between the current policy and the behavior policy used to generate the data. Fakoor et al. 7 considered related

---

KL divergence penalty term in their objective, while Wang et al. (24 approximately controlled this KL divergence by applying a trust region around target policy. These methods are related the penalty term that appears in our generalized policy improvement lower bound, which can be bounded by a penalty that depends on KI. divergence.
Finally, there have been heuristic attempts to incorporate off off-policy data into PPO 13, 20 However, unlike our approach, these methods do not account for the distribution shift caused by off-policy data that invalidates the theoretical support for PPO.
3 Preliminaries
Reinforcement learning framework We consider an -nfniit-horizoo,, discounted Markov Decision Process (MDP) defined by the tuple (S, A. P. r. 7), where is the set of states, AA is the set of actions, p x + As is the transition probability function, r S A + R is the reward function, Po is the initial state distribution, and 7 is the discount rate.
We model the agent's decisions stationary policy S A. Our gual is to choose policy that maximizes the expected total discounted rewards J(TT) = Eror [_i= 7'r(5); dt)], where T ~ I represents trajectory sampled according to so ~ Po, d₁ T(. 8t), and $1+1 p(- St, at). A policy π also induces a normalized discounted state visitation distribution d", where d' (s)
(1 - ?) E "'(sss = Po: P). We write the corresponding normalized discounted state-action visitation distribution as d (s, a) = d" (aa)
(a| s), where we make it clear from the context whether d' refers to distribution over states state-action pairs.
We denote the state value function of x as (*(s) = ENAT 1Ei= 7'r(s,, (t)) | 80 = s]. the state- action value function as QT a) Er-x Et o ('r(s), 800 5,00 - = a]. and the advantage function as A a) = Q" a) (s).
Policy improvement lower bound The starting point in the design of many popular on-policy algorithms is the following policy improvement lower bound, which was first developed by Kakade and Langford
(14) and later refined by Schulman al. [17 and Achiam et al. [I]
Lemma (Achiam et al. I.. Consider a current policy Tk. For any future policy n, we have
T(a () (2
m
E ((T) - ((T)) 2 7 (0.0)rut*
A 4T (8,a)
E ani
mV)),, (1)
(1
where 5 = max,es [Ea-x(-le) distance between the distributions T( 8) and m s).
[AT a)]l and TV(T, (8) represents the total variation
We refer to the first term of the lower bound in Lemma | as the surrogate objective, and the second term as the penalty term. Note that we can guarantee policy improvement at every step of the learning process by choosing the next policy T to maximize this lower bound. Because the expectations Lemma depend on the current policy Ty, we can approximate this lower bound using samples generated by the current policy.
Proximal Policy Optimization PPO, which has become the default on-policy policy optimization algorithm due to its strong performance and simple implementation, is theoretically motivated by the policy improvement lower bound in Lemmal) [ Rather than directly maximizing this lower bound, PPO considers the goal of maximizing the surrogate objective while constraining the next policy to be close to the current policy. In particular, PPO heuristically accomplishes this by considering the following objective at every policy update:
T(a s) AT (8, (e,,a),cii (T(aa ()
T(a 8) T 8) -,, 1e)+ A (s, (e..a)
T.PPO (TT)
E
min
(a.0)~d=k
(2)
where clip(x, 1, = min(max(x, I), u). As seen the second term of this objective, PPO constrains the difference between consecutive policies by removing the incentive for the probability ratio x(als)/<<<a)>)) to leave the clipping range [1 - e, + +e.. Finally, the outer minimization guarantees that (2) is lower bound to the surrogate objective in Lemma In practice, (2) is approximated using samples generated by the current policy TR, and the resulting empirical objective is approximately optimized at every policy update using minibatch stochastic gradient ascent.
3

---

For sufficiently small learning rate and sufficiently large number of samples, PPO results in stable policy improvement throughout the learning process. However, is well-known that high variance is major issue in reinforcement learning, so often the number of samples must be large in order for the empirical objective to an accurate estimator of the true objective (2). Because these samples must be collected under the current policy between every policy update, PPO can be very sample intensive.
4 Generalized policy improvement lower bound
A logical approach to improve the sample efficiency of PPO is to reuse samples from prior policies, as done in off-policy algorithms. Unfortunately, the distribution shift between policies invalidates the policy improvement lower bound in Lemma I which provides the theoretical support for PPO's reliable performance. In order to retain the stability benefits of PPO while reusing samples from prior policies, we must incorporate these off- off-policy samples
principled way. We accomplish this by developing a generalized policy improvement lower bound that can be approximated using samples from the last M policies, rather than requiring samples be generated only from the current policy Tei
Theorem 1 (Generalized Policy Improvement Lower Bound), Consider prior policies Tg- O,... M- 1, where Tr represents the current policy. For any choice of distribution v = [Po ... 1/M-1] over the prior M policies and any future policy n, we have
T(a 8)
J(z) -J(TK) 2
E sovv
E
ATM (s.0))
(s.a)ddtk-s
(a
8)
270
E iowa Lend
TV n [
(3)
71-
where CTT and (TT, (s) are defined as in Lemma
Proof. We generalize Lemma [ to depend on expectations with respect any reference policy, and we apply this result M times where the reference policy is each of πR-i. = O..... respectively. Then, the convex combination determined by v of the resulting M policy improvement lower bounds is also lower bound. See the Appendix for full proof.
Because the expectations in Theorem I depend on distributions related to the last M policies, this lower bound provides theoretical support for extending PPO to include off-policy samples. Note that Theorem ii still a lower bound on the policy improvement between the current policy T and future policy This is true because the advantage function in the surrogate objective and the constant in the penalty term still depend on the current policy Ty. However, the visitation distribution, probability ratio in the surrogate objective, and total variation distance in the penalty term now depend on prior policies. Finally, the standard policy improvement lower bound in Lemma I can be recovered from Theorem by setting M 1.
The penalty term in Theorem I suggests that we should control the expected total variation distances between the future policy æ and the last M policies. By applying the triangle inequality for total variation distance to each component of the penalty term, we see that
E
E Bovd Bovdk-1
[V TK
<
E
E
[TV TH) [+)]
M-1
Σ
E and
(TT (4)
The first term on the right-hand side of 4 represents an expected total variation distance between the current policy тe and the future policy R,, while each component of the second term represents an expected total variation distance between consecutive prior policies. This demonstrates that we can effectively control the expected performance loss at every policy update by controlling the expected total variation distance between consecutive policies. We see next that the clipping mechanism in PPO approximately accomplishes this task.

---

Clipping mechanism
Connection to penalty term As discussed previously, the clipping mechanism present in the PPO objective removes the incentive for the probability ratio (ta)))/m/aaas to leave the clipping range [1-c, 1 + Written differently, the clipping mechanism removes the incentive for the magnitude
T (a TT (a 8)
8)
to exceed We now that (5) is closely related to the penalty term of the standard policy improvement lower bound in Lemma as follows:
Lemma 2. The expected total variation distance between the current policy and the future policy that appears in Lemma can be rewritten as
1
T s)
(T)()) k
(6)
(s.a)aatk
Tx(a 8)
Proof. See the Appendix.
Therefore, the clipping mechanism in PPO can be viewed as heuristic that controls the magnitude of a sample-based approximation of the expectation on the right-hand side of Lemma 2 It accomplishes this by removing the incentive for (5) to exceed all state-action pairs sampled from the state-action visitation distribution d As a result, the clipping mechanism in PPO approximately bounds the expected total variation distance between the current policy TT and the future policy by 5/2.
Generalized clipping mechanism for off-policy data We clipping mechanism and penalty term in PPO to derive a generalized clipping mechanism suitable for the off-policy setting. In particular, the decomposition of the off-policy penalty term from Theorem T that appears in suggests that we should control the expected total variation distance
this connection between the
E
E LeNd
(TV(),
each policy update, which can be rewritten follows: Lemma 3. The expected total variation distance between the current policy nTg and the future policy in can be rewrilten as
Tia
Tx(a 18)
E
E ke-
(T,
=
E
(s,a)~a
E
TTk (a
TT-iiaa c(a (8)
(8)
Proof. Apply the same techniques as in the proof of Lemma
2 See the Appendix for details.
The right-hand side of Lemma 3 provides insight into the appropriate clipping mechanism to be applied in the off-policy setting in order to approximately control the penalty term in the generalized policy improvement lower bound:
Definition 1 (Generalized Clipping Mechanism). Consider a current policy Tu and clipping pa- rameter e. For a state-action pair generated using prior policy Tk-ii the generalized clipping mechanism is defined
Ta(a 1)) Th TT-i(a 8) Th TT-s(a
T(a
aa(a 8) e, mk-s(a s) + +
clip
Note that the probability ratio each state-action pair the off-policy setting begins the center of the clipping range for every policy update just as in PPO, where now the center is given by me(ols)/m :(als). Also note that we recover the standard clipping mechanism used in PPO when samples generated by the current policy mŞ.
5

---

Impact of learning rate on clipping mechanism Due to the heuristic nature of the clipping mech- anism, it can only upproximately bound the penalty term in the corresponding policy improvement lower bound if the learning rate used for policy updates is sufficiently small. To see why this is true, note that the clipping mechanism has no impact at the beginning of each policy update since each probability ratio begins at the center of the clipping range [5] If the learning rate is too large, the initial gradient steps of the policy update can result in probability ratios that are far outside the clipping range. In addition, the sensitivity of the probability ratio to gradient updates can change as training progresses, which suggests that the learning rate may need to change over time in order for the clipping mechanism to approximately enforce total variation distance trust region throughout the course of training.
In order to address these issues, we propose a simple adaptive learning rate method that is directly connected to our goal of controlling a total variation distance penalty term via the clipping mechanism. Using Lemma 3 we can approximate the expected total variation distance of interest using sample- based estimate. We reduce the learning rate if the estimated total variation distance exceeds our goal of 9/2, and we increase the learning rate if the estimated total variation distance is significantly lower than 4/2. This approach more closely connects the implementation of PPO to the policy improvement lower bound on which the algorithm is based. In addition, the adaptive learning rate prevents large policy updates that can lead to instability, while also increasing the speed of learning when policy updates are too small. We formally describe this method in Algorithm
6 Algorithm
The surrogate objective from the generalized policy improvement lower bound in Theorem 1 coupled with the generalized clipping mechanism in Definition] that controls the penalty term in Theorem [ motivate the following generalized PPO objective that directly considers the use of off-policy data:
Taa 8) T (a ())
GePPO Lk
(x) = E bowa
E (d.a)rdkkk-
min
A** (s,a),
Taa
| 8)
xx(a 8) TA (a 1 8) Tx-i(a (s +c) A
clip
(s, 1
(10)
Tk-i(a
() Tk-i(a 8)
Our algorithm, which we call Generalized Proximal Policy Optimization with Sample Reuse (GePPO), approximates this objective using samples collected from each of the last M policies and approxi- mately optimizes it using minibatch stochastic gradient ascent. In addition, we update the learning rate at every iteration using the adaptive method described in the previous section. GcPPO, which we represents a principled approach to improving the sample efficiency of PPO
detail in Algorithm[] | while retaining its approximate policy improvement guarantees.
7 Sample efficiency analysis
Generalized clipping parameter In order to compare GePPO to PPO, we first must determine the appropriate choice of clipping parameter GePPO that results in the same worst-case expected performance loss at every update:
Lemma 4. Consider PPO with clipping parameter e If
PFO and GePPO with clipping parameter GaTO
PPO
GePPO
(11)
Eisy i 1+]
then the worst-case expected performance loss at every update is the same under both algorithms.
Proof. See the Appendix.
Lemma E shows that we must make smaller policy updates compared to PPO in terms of total variation distance, which a result utilizing samples from prior policies. However, these additional samples stabilize policy updates by increasing the batch size used to approximate the true objective, which allows us to make policy updates more frequently. Ultimately, this trade-off results in faster and more stable learning, as we detail next.

---

Algorithm 1: Generalized Proximal Policy Optimization with Sample Reuse (GePPO)
Input: initial policy No; number of prior policies M; policy weights vi clipping parameter e; batch size n; initial learning rate y) adaptive factor a 2 0; minimum threshold factor
0 S B S 1.
for k 0, 1,,,,,.... do
Collect samples with Te.
Update policy:
Use n samples from each of Tk-i, i 0,....,,,.. - 1, to approximate I (T)).
Approximately maximize the empirical objective using minibatch stochastic gradient ascent.
Update learning rate:
Calculate sample-basee basel estimate TV of expected total variation distance in Lemma 3
TV > 5/2 then 1j = 17 1/(1+a)
else if TV < 6c/2 then 1 00-11 11+00 o).
end
Balancing stability and sample efficiency For concreteness, in this section restrict attention to uniform policy weights over the last .M policies, i.e., V; = 1/M for = 0, o,..., .M 1. We provide additional details in the Appendix how these policy weights be optimized to further improve upon the results shown for the uniform case.
Assume require N Bn samples for the empirical objective to be sufficiently accurate estimate of the true objective, where n is the smallest possible batch size we can collect and B is some positive integer. In this setting, PPO makes one policy update per N samples collected, while GePPO leverages data from prior policies to make B updates per N samples collected as long as M 2 B. First, that GePPO
increase the change in total variation distance of the policy throughout training compared to PPO without sacrificing stability in terms sample size:
Theorem Set M B and consider uniform policy weights. Then, GePPO increases the change in total variation distance of the policy throughout training by a factor of (B+(1) compared to PPO, while using the same number of samples for each policy update as PPO.
Proof. See the Appendix.
Alternatively, we that GePPO can increase the sample size used approximate each policy update compared to PPO, while maintaining the same change in total variation distance throughout training:
Theorem 3. Set M 2B 1 and consider uniform policy weights. Then, GePPO increases the sample size used for each policy update by factor of (2B-1)/B compared to PPO, while maintaining the same change in total variation distance of the policy throughout training as PPO.
Proof. See the Appendix.
we see that GePPO with uniform policy weights improves the trade-off between stability and sample efficiency in PPO for any choice of B < M < 2B Also note that the benefit of GePPO increases with B. where a larger value of B indicates more complex problem that requires additional samples to estimate the true objective at every policy update. This is precisely the scenario where the trade-off between stability and sample efficiency becomes critical.
By combining the results in Theorem 2 and Theorem B
8 Experiments
In addition the theoretical support for our algorithm in the previous section, we aim to investigate the stability and sample efficiency of GePPO experimentally through simulations several MuJoCo
7

---

Table 1: Performance comparison across MuJoCo tasks.
Average Performance Over 1M Steps
Steps (M) to Final PPO Performance
Final Performance
Environment
PPO
GePPO
%"
PPO
GePPO
PPO
GePPO
Swimmer-v3
98
161
65%
136
195
44%
1.00
0.23
Hopper-v3
2,362
2,544
8%
3,126
3,450
10%
1.00
0.41
HalfCheetah-v3
1,764
2,439
38%
3,223
3,903
21%
1.00
0.54
Walker2d-v3
1,817
2,199
21%
3,041
3,502
15%
1.00
0.63
Ant-v3
545
762
40%
1,227
1,576
28%
1.00
0.79
Humanoid-v3
584
665
14%
972
1,345
38%
1.00
0.85
Represents percent improvement GePPO compared to PPO.
Swimmer-v3
Hopper-v3
HallCheetahh-0
Average Performance
4000
3000
100
1000
PPO GoPPO
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Walker2d-v3
Ant-33
Humancidiv3
Average Performance
1500
1602
3000
1000
2000
1000
800
0.0
0.2
0.6
0.6
1.0
0.0
0.2
0.4
0.8
0.0
0.0
0.4
0.6
0.8
Steps
Steps
Steps (M)
Figure Performance throughout training across MuJoCo tasks. Shading denotes half of one standard error. Horizontal dotted lines represent the final performance of PPO, and vertical dotted lines represent the time at which GePPO achieves the same performance.
environments [2] in OpenAI Gym [3] In particular, we consider six continuous control locomotion tasks which vary in dimensionality: Swimmer-v3, Hopper-v3, HalfCheetah-v3, Walker2d-v3, Ant-v3, and Humanoid-v3. We compare the performance of our algorithm to PPO, which is the default on-policy policy optimization algorithm. We do not consider comparison with popular off-policy algorithms since they lack approximate policy improvement guarantees, and as a result the risk associated with each policy update is not comparable.
We consider the default implementation choices used by Henderson al. 112 PPO. In particular, we represent the policy и a multivariate Gaussian distribution, where the mean action for given state is parameterized by neural network with two hidden layers of 64 units each and tanh activations. The iute-indeeendent standard deviation is parameterized separately. The default value for the clipping parameter is PPO 0.2, and the default batch size is N = 2,048. Sample trajectories for the tasks consider can contain up one thousand steps, as n = 1,024 and B using the notation from the previous section.
represent the default batch size
For GePPO, we select M and the corresponding policy weights to maximize the effective batch size used for policy updates while maintaining the same change in total variation distance throughout training as PPO. The clipping parameter GePPO is chosen according to Lemma 4 which in our experiments results in GePPO 0. We estimate Th (s, a) with an off-policy variant of Generalized Advantage Estimation
[8] that uses the V-trace value function estimator (6). We run each experiment

---

Default Policy Network: (64,64)
Standard Wide Policy Network: (400,300)
Average Performance
PPO
GePPO
3000
2000
1000
1000
0.0
0.3
Average TV Distenoe
0.2
0.2
0.1
0.1
0.0
0.0
0.8
Steps (M)
Steps (M)
Figure 2: Evaluation on Walker2d-v3 with different policy networks. Hidden layer sizes in paren- theses. Shading denotes half of one standard error. Top: Performance throughout training. Bottom:
Change in average total variation distance per policy update. Horizontal dotted lines represent target total variation distances for PPO and GePPO, respectively.
for total one million steps over five random seeds, See the Appendix for additional implementation details, including the values of all hyperparameters] [
As shown in Table [ and Figure GePPO results in fast and reliable learning. We see that GePPO leads to improved final performance across all environments. In addition to final performance, we assess the sample efficiency of our algorithm by considering the average performance over the course of training as well as the number of samples required for GePPO to match the final performance of PPO. Despite the fact that the default implementation of PPO achieves stable learning with a small batch size (B 2), our results still demonstrate the sample efficiency benefits of GePPO. Compared to PPO, GePPO improves the average performance over training by between 8% and 65%. Moreover, GePPO requires between 15% and 77% fewer samples to reach the final performance level of PPO,
In addition to improving sample efficiency, GePPO also ensures that the total variation distance between consecutive policies remains close to the target determined by the clipping parameter through the use an adaptive learning rate. This is not the case in PPO, where we observe that the change in total variation distance per policy update increases throughout training. As shown on the left-hand side of Figure the change total variation distance for PPO under default settings almost 40% higher than desired after one million steps on Walker2d-v3. We demonstrate that this can lead to instability in the training process by considering standard wide policy network with two hidden layers of 400 and 300 units, respectively. We see the right-hand side of Figure that this minor implementation change exacerbates the trend observed in PPO under the default settings, resulting in unstable learning where performance declines over time due excessively large policy updates. GePPO, on the other hand, successfully controls this source of instability through its adaptive learning rate, resulting in stable policy improvement that is robust to implementation choices.
Conclusion
We have presented a principled approach incorporating off aff-policy samples into PPO that is theo- retically supported by novel off- off-policy policy improvement lower bound. Our algorithm, GePPO, improves the sample efficiency of PPO, and can be viewed as a more reliable approach to sample reuse than standard off-polio algorithms that are not based on approximate policy improvement guarantees. This represents an important step towards developing stable, sample efficient reinforcement learning methods that can be applicd in high-stakes real- weal-world decision making.
"Code available httts:: htap::///gttuut http:///hhuuu..co///eeeee/////// comm//queeeeey//eppo

---

Despite this progress, there remain limitations that must be addressed in order for reinforcement learning to achieve widespread real-world adoption. Because our algorithm is based on PPO, policy improvement guarantees are only approximately achieved due to the use of the clipping mechanism heuristic. addition, the constant factor in the penalty term on which our algorithm is based may be too large to deliver practical guarantees. Finally, we considered PPO as our starting point due to its theoretical support and stable performance, but there may exist other approaches that more effectively balance the goals of stability and sample efficiency. These represent interesting avenues for future work in order to develop reinforcement learning methods that can be trusted to improve real-world decision making.
Acknowledgments and Disclosure of Funding
This research was partially supported by the NSF under grants ECCS- 193160,,,,00, DMS-1664644,, CNS- 1645681, 1, and 111-9914792, by the ONR under grants 400014-19-1-257 and N00014-21-1-2844 by the NIH under grants RO1 GM1 35930 and UL55 TR004130, 130, by the DOE under grants DE-AR- 0001282 and NETL-EE00096% by AFOSR under grant PA9550-19-1-011 the Boston University Kilachand Fund for Integrated Life Science and Engineering, and by the Qatar National Research Fund, a member of the Qatar Foundation (the statements made herein are solely the responsibility of the authors).
by the MathWorks, by
References
[1] Achiam, D. Held, A. Tamar, and Abbeel. Constrained policy optimization. Proceedings of the 34th International Conference Machine Learning, volume 70, pages 22-31. PMLR, 2017.
[2] M. Andrychowicz, A. Raichuk, P. Stańczyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski. S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? A large-scale study. In Ninth International Conference on Learning Representations, 2021.
[3] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAl Gym. arXiv preprint, 2016. arXiv:1606.011 015400
[4] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In Proceedings of the 33rd International Conference on Machine Learning, volume 48, pages 1329-1338 PMLR, 2016.
[5] I. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry. Im- plementation matters deep RL: case study on PPO and TRPO. In Fighth International Conference Learning Representations, 2020.
[6] L. Espeholt, H. Soyer, R. Munos, K. Simonyan,
V. Mnih, T. Ward, Y. Doron, V. Firoiu, Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-leamner architectures. In Proceedings of the 35th International Conference on Machine Learning. volume 80, pages 1407-1416. PMLR, 2018
[7] R. Fakoor, P. Chaudhari, and A. J. Smola. P30: Policy-on policy-off policy optimization. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115, pages 1017-1027. PMLR, 2020.
[8] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 1587-1596. PMLR, 2018
[9] S. Gu, T. Lillicrap, Z. Ghahramani, R. F. Turner, and Levine. Q-Prop: Sample-efficient policy gradient with an off off-policy critic. In 5th International Conference on Learning Representations, 2017.
10

---

[10] S. Gu, T. Lillicrap. R. E. Tumnr, Z. Ghahramani, B. Schölkopf, and S. Levine. Interpolated policy gradient: Merging on-policy and off-policy gradient estimation for deep reinforcement learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[11] T. Haarnoja, A. Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning. volume 80, pages 1861-1870. PMLR, 2018
[12] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 3207-3214, AAAI Press, 2018.
[13] M. S. Holubar and M. A. Wiering. Continuous-action reinforcement learning for playing racing games: Comparing SPG to PPO, arXiv preprint, 2020. wEXiv:2001.052770
[14] S. Kakade and Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the 19th International Conference on Machine Learning, pages 267-274. Morgan Kaufmann Publishers Inc., 2002.
[15] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference Learning Representations, 2016.
[16] B. O'Donoghue, R. Munos, K. Kavukcuoglu, and V. Mnih. Combining policy gradient and Q-learning. In 5th International Conference on Learning Representations, 2017.
[17] Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1889-1897. PMIR, 2015.
[18] Schulman, P. Moritz, Levine, M. I. Jordan, and Abbeel. High-dimensional continuous control using generalized advantage estimation. In 4th International Conference on Learning Representations, 2016.
[19] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017. arXiv:1707.06344
[20] Sovrano. Combining experience replay with exploration by random network distillation. In 2019 IEEE Conference on Games (CoG), pages 1-8, 2019. doi: 10. 20.1109//G..2019884884448
[21] E. Todorov, T. Frez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEFE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 10. 1111199/RRS..0
[22] Y. Wang, H. He, X. Tan, and Y. Gan. Trust region-guided proximal policy optimization. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[23] Y. Wang, H. He, and X. Tan. Truly proximal policy optimization. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115, pages 113-122. PMLR, 2020.
[24] Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu, and N. de Freitas. Sample efficient actor-critic with experience replay. In 5th International Conference on Learning Representations, 2017.


Reviews:
[{"review_id": "pyhabe9awFs", "reviewer": "Reviewer_2GSR", "summary": "This work generalizes PPO to the off-policy setting. First the policy improvement bound from Achiam et al. (2017) was generalized to accomodate for several prior policies. The authors also made a connection between the total variation distance and the PPO clipping term. A new algorithm GePPO is proposed which is shown to be more sample efficient compared to standard PPO.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this is a very well-written paper. To the best of my knowledge, this work is the first to draw a direct connection between the monotonic policy improvement theorem and the clipped PPO algorithm, including deriving a relationship between the clipping mechanism and the total variational distance. The generalization of the policy improvement bound to past policies is also quite innovative. Most of the proofs in this paper directly extend upon the proofs in Achiam et al. (2017) and do appear to be correct.  Different components of the new algorithm were thoroughly analyzed. Experiments presented in this paper are mostly quite well designed though I think a comparison with some popular off-policy algorithms and also TRPO (which is a more direct application of the policy improvement bound) would be of interest. I also particularly like the fact that the authors are very forthcoming with the limitations of their paper, since the proposed algorithm is a variation of PPO, I do not expect it to achieve SOTA performance but I believe this work does provide additional useful insights into the PPO algorithm and serves as a useful jumping off point for the community for further research.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See main review", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5"}, "extra_text": {}}, {"review_id": "SwSLkdZSYVm", "reviewer": "Reviewer_CJcP", "summary": "\nThis paper extends PPO to an off-policy case. They develop policy improvement bound for the off-policy setting. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nGenerally, I like the idea of the paper. Extending PPO to off-policy case is an interesting and promising aspect. The paper has provided concise theorem derivation and an elegant realization of off-policy PPO. I think it's important and valuable.\n\nI have gone through the detail of the derivation and the theorems seem correct for me. I like the insightful derivation in Theorem 1, Lemma 3, and Definition 1. \n\nHowever, I do not give a high rating as the paper does not demonstrate the effectiveness of the methods (see the detail of the comment below).\n\n\n\n\n\n\n\nThe main weakness of the paper is the experiment.\n- There are abundant environments in Mujoco and Atari. Since the author has provided a very general algorithm, I expect the author to conduct experiments on more environments to verify the effectiveness of the newly proposed methods.\n- The comparison methods is missing. \nA) Comparison with other PPO variants. There are several methods (as you have stated in related work) which can improve PPO while require equal computation as PPO. I expect the author to compare with these methods or extend their methods to these settings (since the method in the paper is very general and can also be extended to these PPO variants. I should not blame the author for this, but the author should verify the effectiveness of the proposed method). \nB) Comparison with other off-policy methods, such as DDPG, SAC. One shortcoming of PPO is its on-policy limitation. I expect the proposed off-policy PPO could be competitive with other off-policy methods.\n- Alation study on the parameter \\epsilon. By the way, I don't understand why \\epsilon is not presented in Algorithm 1 (Input) and APPENDIX Table 2.\n\n\n\n\nMinor questions:\nIn my view, one critical component for success is the estimation of A^{\\pi_k}(s,a) using the off-policy data of \\pi_{k-i}. They use the V-trace method for estimation. I expect the author to discuss on this (not necessary).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "DVivi57LwA", "reviewer": "Reviewer_nKjh", "summary": "This paper presents a novel policy improvement lower bound that provides theoretical evidence for the off-policy sample reuse of PPO algorithms. Specifically, the theorem justifies the clip \nterm in PPO as a regularization in the lower bound. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Methodology\n\nThe proposed learning rate schedule is similar to the adaptive Adaptive KL penalty coefficient in PPO. Could the author comment on which schema has better control on the TV? \n\n\nPresumably, the older policy will have larger TV divergence comparing to the more updated policies, \nis there a better approach than uniform weight? Does this weighting strategy affect a lot?\n\n### Experiments\n\nThe experimental results of this work are relatively weak. 1) The common practice in the literature of sample-efficiency algorithms [continuous action space] benchmark six mujoco environments. 2) The only baseline in this work is PPO. It would be helpful to also compare with off-policy algorithms [TD3, SAC, DDPG] as this work enables the sample reuse. \n\nWhen comparing to the baselines, I think the fair comparison would be set the replay ratio same as the baseline. Currently, the replay ratio is B times of PPO. For off-policy approaches, the replay ratio can affect performance a lot. \n\nFurthermore, PPO is not restricted to continuous action environments. It would be more solid to also benchmark the proposed algorithm on discrete action space environments [e.g., Atari]. \n\nIt would be interesting to show the balance between wall clock time and sample efficiency. How does the proposed regularization affect the wall clock time? \n\nFigure 2 indicates the performance drop is aligned with the change in total variation distance is inspiring. \n\nIt looks like most of the performance gain is from the adaptation of learning rate based on the results of \nPPO-Adapt in the appendix. It would be helpful to see what is the performance of GePPO without learning rate adaptation. \n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "AP6npKGrRbZ", "reviewer": "Reviewer_FiaV", "summary": "The paper proposes a generalization of PPO to the off-policy setting.\n\nPPO has been extensively used with multiple policy updates, making it inherently off-policy and it was no clear how the algorithm should be modified in order to preserve its theoretical guarantees. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The adaptation or the bound from Achiam follows easily when considering a distribution over the last M policies. The additional theoretical results help understanding how one should choose this distribution or adapt the learning rate of the policy in a more principled way. The authors perform an ablation, showing that the benefits in performance come both from the adapted clipping mechanism and from the reuse of past data.\n\nI think it would be interesting to compare this also to the case when the generalized clipping mechanism is missing\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The authors' answers solved most of the doubts exposed in the reviews, and the reviewers agree that this paper can be accepted for publication.
I want to congratulate the authors and invite them to modify their paper following the reviewers' suggestions.
Decision: Accept (Poster)