Batch size-invariance for policy optimization
Authors: Jacob Hilton
Karl Cobbe
John Schulman
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
policy gradient
learning rate

Abstract: We say an algorithm is batch size-invariant if changes to the batch size can largely be compensated for by changes to other hyperparameters. Stochastic gradient descent is well-known to have this property at small batch sizes, via the learning rate. However, some policy optimization algorithms (such as PPO) do not have this property, because of how they control the size of policy updates. In this work we show how to make these algorithms batch size-invariant. Our key insight is to decouple the proximal policy (used for controlling policy updates) from the behavior policy (used for off-policy corrections). Our experiments help explain why these algorithms work, and additionally show how they can make more efficient use of stale data.

Under review as a conference paper at ICLR 2022
BATCH SIZE-INVARIANCE FOR POLICY OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We say algorithm is batch size-invariant if changes to the batch size can largely be compensated for by changes to other hyperparameters. Stochastic gradient descent is well-known to have this property at small batch sizes, via the learning rate. However, some policy optimization algorithms (such as PPO) do not have this property, because of how they control the size of policy updates. In this work we show how to make these algorithms batch size-invariant. Our key insight is decouple the proximal policy (used for controlling policy updates) from the behavior policy (used for off-policy corrections). Our experiments help explain why these algorithms work, and additionally show how they can make more efficient use of stale data.
INTRODUCTION
Policy gradient-based methods for reinforcement learning have enjoyed great success in recent years. The stability and reliability of these methods is typically improved by controlling the size of policy updates, using either "trust region" (TRPO)
surrogate objective (PPO) (Schulman et al. 2015 2017). The usual justification for this is that we cannot trust updates that take us too far from the policy used collect experience, called the behavior policy. In this work identify a subtle flaw with this: the behavior policy is irrelevant to the justification. Instead, what matters that we control how fast the policy is updated, or put another way, that we approximate the natural policy gradient (Kakade 2001).
Our key insight is that the "old" policy in these methods serves two independent purposes. The first purpose is for off-policy corrections, via importance sampling, for which the old policy must be the behavior policy. The second purpose is to control the size of policy updates, for which the old policy can be any recent policy, which we call the proximal policy. It does not matter whether the proximal policy is also the behavior policy; only matters how old the proximal policy is. We demonstrate this by running PPO with stale data collected using policy from multiple iterations ago, which causes performance to quickly degrade unless the proximal policy is decoupled from the behavior policy.
Our insight allows us to make PPO batch size-invariant, meaning that when the batch size is changed, we can preserve behavior, as a function of the number of examples processed, by changing other hyperparameters (as long as the batch size is sufficiently small). We achieve this by using an exponentialll-weighhe
moving average (EWMA) of the policy network's 's weights as the network for the proximal policy. Batch size-invariance has been studied many times before (see Section 33.). sometimes under the name "perfect scaling".. It is of practical benefit when we wish to increase the batch size reduce gradient variance, but computational resources such GPU memory do not allow this. In such a situation, we can instead adjust other hyperparameters formulaically, thereby spreading out the increased computational load over time.
The remainder of the paper is structured as follows.
In Section
2 we explain the difference between the proximal and behavior policies, and show how to decouple them in PPO's objectives.
we explain the concept of batch size-invariance, and how it applies to SGD and Adam Kingmma & Ba, 2014).
In Section 3
In Section ct al.] 2020) that make use of our decoupled objectives, and show how to make them batch size-invariant small batch sizes.
4 introduce PPO-EWMA and PPG-EWMA, variants of PPO and PPG (Cobbe

---

Under review as a conference paper at ICLR 2022
provide experimental evidence for our central claims: that decoupling the proximal policy from the behavior policy can be beneficial, and that it allows us to achieve batch size-invariant policy optimization.
In Section 5
Finally, in Section 6
we discuss the theoretical and practical implications of our results.
DECOUPLED POLICY OBJECTIVES
In this section we explain the difference between the proximal and behavior policies, and introduce new versions of PPO's objectives in which they have been decoupled.
PPO alternates between sampling data through interaction with the environment, and optimizing a surrogate objective. The policy used for sampling is denoted Toma and is used by the objective in two different ways. This is casiest to see with the KL penalized objective (Schulman et al. 2017 equation (8)):
L KLPEN (0) :- E.
Th (at 5t)
A. 3KL [To (- 8t) no ( 86)]
Touia (at 8t)
where As is am estimator of the advantage at timestep t, and Eg [...] indicates the empirical average finite batch of timesteps f. The first of No.. in this expression is part of importance sampling ratio. order for the policy gradient estimate to be unbiased, this policy needs to be the call this the behavior policy Te The second use of Teaa
one that was used for sampling, is as recent target to pull the current policy towards, so we call this the proximal policy T0
Our key insight is that the proximal policy need not equal the behavior policy. As we will show experimentally, matters how old the proximal policy is, but it does not matter whether or not the proximal policy was used for sampling.
We therefore define the decoupled KL penalized objective
(a: 8t) TT (at St) A. 3 KL (T (T (.1 me (- st)]
L decoupled
(0) = Ee
where Thhshar is the policy used for sampling, and ppprcx is a recent policy yet to be specified.
It is less obvious how to decouple the clipped PPO objective, because Tesia only appears in that expression (Schulman et al. 2017] equation (7)):
L CLIP (0) == E, min (rx (9) A., clip (r. (0), (c)) c,
where ri (0) = na Ta(al)))) Na (82)
However, we can rewrite this objective as
I, CLIP (0) = R2
min (T clip (No. (1 - -c) Toai (1 mo A.)
T
(omitting the policy arguments (ar st) for brevity). Now the first use of Taaa is as part of an importance sampling ratio, for which we must use the behavior policy, and the second and third uses are applying the implicit KL. penalty, for which we can use the proximal policy.
We therefore define the decoupled clipped objective
prrox (at St))
min (r2 (8) A., clip (r, (0), 10),1-ch e, c,
L decoupeed (8) : E,
Tobaba (at sc)
To (at, 88)
where
rt (0)
Te prace (at )t)
2

---

Under review as a conference paper at ICLR 2022
As sanity check, note that if we set the KL penalty coefficient 3 or the clipping parameter oc, then the dependence on the proximal policy disappears, and we recover the vanilla (importance- sampled) policy gradient objective (Schulman al. 2017 equation (6)).
similar decoupled policy objective also propsed Mirror Descent Policy Optimization (Tomar al. 2020 Section 5.1).
BATCH SIZE-INVARIANCE
say an algorithm batch size-invariant to mean that when batch size is changed, the original behavior can be approximately recovered by adjusting other hyperparameters to compensate. Here we consider behavior as a function of the total number of examples processed, so another way to put this is that doubling the batch size halves the number of steps needed. (Shallue al. (2018) and Zhang et al. (2019) refer to this as "perfect scaling".
We treat batch size-invariance binary property. In practice, the original behavior can never be recovered perfectly. and the extent to which it can be recovered depends both how much and the direction in which the batch size is changed.
descriptive property that hold some degree, rather than
3.1 BATCH
FOR STOCHASTIC GRADIENT DESCENT
Stochastic gradient descent (SGD) is batch size- invarinnrrinnt, up until the batch size approaches some critical batch size. This is the batch size which the gradient has signal-to-noiss, ratio of around 1. At smaller batch sizes than this, changes to the batch size can be compensated for by a directly proportional adjustment to the learning rate. This core observation
been made many times before (Mandt et al.] 2017 Goyal et al.) 2017: [Smith al.) 20117] Hardinn 2017, Ma et al., [0018] (Saalue ct al. 2018 McCandlish et al. 2018). A discussion of this and other previous work can be found in Appendix
Sketch explanation. For the benefit of the reader's ' intuition, we sketch the explanation for SGD's batch size-invariance. For much more thorough explanation, refer the reader to Mandt et al. (017)).
Consider running SGD on loss function L. (0; of parameter vector 0 and data point Two steps with batch size n and learning rate a corresponds to the update rule
0.+2
0:
Σ VoL (0;z) FEB,
Σ xEB
VoL
TI
n.
where B, and B the next two batches of size n. On the other hand, single step with batch size 2n and learning rate 2a corresponds to the update rule
2a
U.+2
0r
Σ
VoL (0;2)..
2n
ZEB,LBI+1
These update rules are very similar, the only difference being whether the gradient for B is evaluated at O: or 0.+1. If batch size small compared to the critical batch size, then the difference between 0g and 0c+1 is mostly noise, and moreover this noise is small compared to the total noise accumulated by 6g over previous updates. Hence the two update rules behave very similarly.
A good mental model of SGD in this amall-batch regime is of the parameter vector making small, mostly random steps around the loss landscape. Over many steps, the noise is canceled out and the parameter vector gradually moves in the direction of steepest descent. But a single additional step makes almost no difference to gradient evaluations.
In more formal terms, SGD is numerically integrating stochastic differential equation (SDE). Changing the learning rate in proportion the batch size leaves the SDE unchanged, and only affects the step size of the numerical integration. Once the step size is small enough (the condition that gives
3

---

Under review as a conference paper at ICLR 2022
rise to the critical batch size), the discretization error is dominated by the noise, and the step size stops mattering.
3.2 BATCH
FOR ADAM
Adam (Kingma & Ba 2014) is popular variant of SGD, and is also batch size-invariant until the batch size approaches critical batch size (which may be different to the critical batch size for SGD) (Zhang et al. 2019). To compensate for the batch size being divided by some constant c, one must
make the following adjustments (Hardin 2017):
Divide the step size by Ve.
(Raise the exponential decay rates 31 and Ba to the power of 1/c.)
The first adjustment should be contrasted with the linear learning rate adjustment for vanilla SGD. We discuss the reason for this difference and provide empirical support for the square root rule in Appendix D
The second adjustment is much less important in practice, since Adam is fairly robust to the B1 and 32 hyperparameters (hence it has been parenthesized). Note also that 81 also affects the relationship between the current policy and the proximal policy in policy optimization. For simplicity, we omitted this adjustment in most of our experiments, but included it in some additional experiments that are detailed in Appendix
3.3 BATCH SIZE-INVARIANCE FOR POLICY OPTIMIZATION
In policy optimization algorithms like PPO, there are two different batch sizes: the number of environment steps in each gradient step, which we call the optimization batch size, and the number of environment steps in each alternation between sampling and optimization, which we call the iteration batch size. When both batch sizes by the same factor simultaneously can be compensated for. The motivation for this definition is that this is the effect of changing the degree of data-parta aata-parallelism.
say that such an algorithm is batch size-invariant,
mean that changes to
If the optimization algorithm (such as SGD or Adam) used by PPO is batch size- invarinnari then by definition this makes PPO optimization batch size-invariant. In the next section, show how to make PPO iteration batch size-invariant, and therefore batch size-invariant outright.
To measure batch size- innarinnveeeece for policy optimization, there are many features of the algorithm's behavior we could look at. As simple metric, we use the final performance of the algorithm, since this is the primary quantity of interest most practitioners. If an algorithm has high degree of batch size-invariance, then the difference in final performance at different batch sizes should be small.
4 PPO-EWMA AND PPG-EWMA
We now introduce simple modification that can be made to any PPO-based algorithm:
Maintain an exponentially-weieee
moving average (EWMA) of the policy network, updat- ing it after every policy gradient step using some decay rate Sprox-
Use this as the network for the proximal policy in one of the decoupled policy objectives.
The motivation for using EWMA is as follows. We would like to be able to use policy from some fixed number of steps ago as the proximal policy, this requires storing a copy the network from every intermediate step, which is prohibitive if this number of steps is large. Using an EWMA allows us to approximate this policy using more reasonable memory requirements. Although this approximation is not exact, averaging in parameter space may actually improve the proximal policy (Izmailov et al. 2018), and the age of the proximal policy can still be controlled by adjusting Sprox-
We refer to this modification using the -EWMA suffix. Thus from PPO obtain PPO-EWMA, and from Phasic Policy Gradient (PPG) (Cobbe et al. (020)) we obtain PPG- EWMA. Pseudocode for PPO-EWMA MA may be found in Appendix || and code may be found at [redacted for anonymity).
4

---

Under review as a conference paper at ICLR 2022
To see how this modification helps us to achieve batch inzariinaa,, note that the main effect of changing the iteration batch size in PPO is to change the age of the behavior and proximal policies (which are coupled). The age of the behavior policy affects how on-policy the data is. but this does not matter much, as long as it is not too large. However, the age of the proximal policy affects the strength of the KI. penalty (or the implicit KI. penalty in the case of the clipped objective), which influences how fast the policy can change. We therefore need to maintain the age of the proximal policy the iteration batch size is changed, which is what our modification enables.
More specifically, to achieve batch size-invariance for PPO- and PPG-EWMA, we make the following adjustments compensate for the optimization and iteration batch sizes being divided by some constant c:
Adjust the optimization hy perparameters described in the previous section, i.c., divide the vanilla SGD learning rate by or the Adam step size by vC. (We use Adam.)
Multiply
1-9prox 1 by c. (This expression is the center of mass of the proximal policy EWMA, measured in gradient steps.) This adjustment is what keeps the age of the proximal policy constant, measured environment steps.
If using advantage normalization, multiply the number of iterations used to estimate the advantage mean variance by c. (In practice, we use EWMAs to estimate the mean and variance, and multiply their effective sample sizes, measured in iterations, by e This keeps the overall sample sizes of these estimates constant, preventing their standard errors becoming too large.
For PPG, multiply the number of policy iterations per phase N₂ by c. (We use PPG.)
For these adjustments to work, we require that the optimization batch size is sufficiently small. We also require that the number of policy epochs (denoted E in PPO or E in PPG) is 1. This is because when the iteration batch size is very small, using multiple policy epochs essentially amounts to training on the same data multiple times in a row, which is redundant (modulo changing the learning rate). Our batch lize-invariance experiments therefore use PPG-EWMA, where E,- is the default.
Note that PPG has a third batch size: the number of environment steps in each alternation between phases, which we call the phase batch size. The effect of our adjustment to N₂ is to simply hold the phase batch size constant, thereby preserving the dynamics of the policy and auxiliary phases.
5 EXPERIMENTS
To validate our analysis, we ran several experiments Procgen Benchmark (Cobbe et al. 2019). which we found to serve as a useful testbed due to the difficulty and diversity of the environments. Hyperparameters for all of our experiments can be found in Appendix B and full results each of the individual environments can be found in Appendix
5.1 ARTIFICALL STALENESS
To investigate our decoupled policy objectives, we introduced artificial staleness. By this we mean that once data has been sampled through interacting with the environment, it immediately used for optimization, but is instead placed in a buffer to be used fixed number of steps later. Despite being artificial, similar staleness often encountered in asynchronous training setups, where it is known to cause problems for on-policy algorithms like PPO (OpenAl et al.) 2019). We measure staleness in iterations, with one iteration being a single alternation between sampling and optimization.
With artificial staleness, the original PPO objectives are underspecified, since there are two natural choices for no.s the policy immediately preceding the current iteration, denoted T and the behavior policy Thecher However, the decoupled objectives allow us to take the proximal policy Trox to be the recent policy, while continuing to use the behavior policy for importance sampling. This allows the KL penalty (or clipping) to have consistent effect in terms of controlling how fast the policy changes, while avoiding harmful bias from incorrect importance sampling.
TThe effective sample size, sometimes called the span, of an F.WAA with decay / is equal _2 1.
5

---

Under review as a conference paper at ICLR 2022
PPO with Raan
PPO with decoupled objective
PPO with To.i Ronnbar
0.6
Mean normalized return
0.4
Stuleness
ticration
0.0
lierations
iteaaiions
iteations
itennions
-0.2
31 Iterations
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
01.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Ravironment steps
x10
Environment steps
x10
Environment steps
10
(a) Using the recent policy for im- portance sampling introduces bias that makes training highly unstable for even small amounts of staleness.
(b) The decoupled objective allows the correct importance sampling ra- tio to be used while maintaining the age of the proximal policy, pre- venting performance from degrad- ing much until the data is very stale.
(c) Using the behavior policy to control the size of policy updates holds back learning unnecessarily for small amounts of staleness, but the additional stability is helpful for very stale data.
Figure 1: PPO with artificial staleness, averaged over all 16 Procgen environments. One iteration corresponds to 65, 536 environment steps with our hyperparameters. Mean and standard deviation over 4 seeds shown.
In our experiments, we compare the decoupled objective to both choices for the original objective. Our results shown in Figure [ With both choices for the original objective, even small amount of staleness hurts performance. However, with the decoupled objective, performance is robust to surprising amount of staleness, with minimal degradation until staleness of around iterations (over 500,000 000 environment steps). This demonstrates that the decoupling the proximal policy from the behavior policy can be beneficial.
5.2 BATCH SIZE-INVARIANCE
We tested our method achieving batch size-invariance for PPG-EWMA described Section 4 Since the optimization batch size is required be sufficiently small, started from our default batch size, which uses 256 parallel copies of the environment, and reduced it by factors of until we running just single parallel copy of the environment.
We were able to achieve a high degree of batch size- invariance, with difference in final mean normalized return between the largest and smallest batch sizes of 0.052. Moreover, there was a single outlier environment, Heist, without which this difference reduced 0.019. We conducted further experiments try explain this outlier, which discuss in Appendix but we were not successful.
Our results are shown in Figures 2 and 3
We conducted ablations which all but one the adjustments was removed, the results of which are also shown in Figures and Figure 3 The Adam step size adjustment is the most important at every batch size, and training becomes highly unstable at the smallest batch sizes without this. The advantage normalization adjustment does not matter at the largest batch sizes, but matters a lot at the smallest batch sizes some environments, which are the ones with particularly noisy advantage standard deviation estimates (see Figure 13 in Appendix G.2). The adjustment the EWMA matters a little at every batch size, which reflects the fact that PPG is relatively robust to changes to the KL penalty. We did not an ablation for the adjustment to the PPG hyperparameter N,, but infer from Cobbe et al. (2020) Figure 5) that it comes immediately after the Adam step size adjustment in importance.

---

Under review as a conference paper at ICLR 2022
PPG-EWMA with all batch size-invariance adjustments
(a) No Adam step size adjustment
(b) No adv. norm. adjustment
0.8
0.6
0.6
0.0
0.0
normalizall
Batch size
0.0
0.2
0.8
1.0
0.0
0.2
0.8
1.0
Default
x10*
x10
Default
(c) No EWMA adjustment
(d)
No EWMA all, just PPG
Default
0.8
0.8
0.2
Default 64
Default 256
PPG with KL penalty
0.0
0.2
0.2
B 1 122
0.2
0.4
0.6
0.8
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.6
0.8
1.0
Environment steps
x10
Environment steps
x10
Environment steps
x10
Figure 2: PPG-EWMA at different batch sizes, with hyperparameters adjusted to achieve batch size-invariance, averaged over all 16 Procgen environments. For reference, we also show PPG (at the default batch size) with the KI. penalty coefficient (3 in the J.KPPEN policy objective) reduced to 1/256, which serves as an approximate lower bound on PPG's performance with a KL, penalty that weak. On the right we show ablations with all but one of the adjustments. Mean and standard deviation over seeds shown.
Dillerence in normalized return
1.25
1,00
0.75
(square root scale)
0.50
0.25
0.00
PPG-EWMA with all batch size- invariance adjustments
(a) No Adam step size adjustment
(b) No adv. norm. adjustment
(c) No EWMA adjustment
(d) No EWMA, just PPG
Experimental setup and environment
Figure 3: For the results shown in Figure 2 we measure the degree of batch vize-invariance for each individual environment by calculating the difference in normalized return between the largest and smallest batch sizes, averaged over the last 4 million timesteps (the length of single PPG phase). We square root scale make small differences more visible. Mean and standard error over seeds shown.
Environments from left to right: CoinRun, StarPilot, CaveFlyer, Dodgeball, FruitBot, Chaser, Miner, Jumper, Leaper, Maze, BigFish, Heist, Climber, Plunder, Ninja, BossFight.
7

---

Under review as a conference paper at ICLR 2022
0.20
0.15
Difference in nornalized return
Mean normalized return
0.10
(square rooK scale)
0.05
0.1
0.00
0.2
PPO
PPO-EWMA
PPG
PPG-EWMA
-0.05
0.0
0.2
0.4
0.6
0.8
1.0
PPO-EWMA vs PPO
PPG-EWMA PPG
Environment steps
x10
Algorithm and enviroment
Figure 4: Performance of all 4 algorithms on Procgen. Left: learning curves, mean and standard deviation over seeds shown. Right: difference normalized return by environment, averaged over the last 4 million timesteps, mean and standard error over 4 seeds shown.
To check the statistical significance of the effects produced by our ablations, we conducted hypothesis tests, which describe in Appendix [F] Our null hypothesis that the ablation had effect on the difference in final normalized return at different batch sizes in any of the environments. For the comparison between the largest and smallest batch sizes, we rejected the null hypothesis for all of the ablations at the 0.1 % level.
5.3 EWMA COMPARISON
Finally, we tested the outright benefit of the EWMA modification by doing a head-to-head comparison of PPO against PPO-EWMA and of PPG against PPG-EWMA. It is important to note that the EWMA introduces an additional hyperparameter Sprox. but that this was tuned only on the first of the 16 Procgen environments (and only on PPG), and so the algorithms are "complete" on the last 8 environments in the sense of Jordan et al. 2020). Our results are shown in Figure 4
We found the benefit of the EWMA be small but remarkably consistent across environments and algorithms, outperforming the baseline on all of the last 8 environments for both PPO and PPG. We believe that this is the result of the EWMA reducing the variance of the proximal policy. Further evidence that the variance of the proximal policy matters is discussed in Appendix H
Note that this benefit comes the of additional memory store the weights of the EWMA network, and an additional forward pass of the EWMA network for each policy gradient step. With our hyperparameters, this increases the computational cost of PPO by 30% and of PPG by 2.3%, not including the cost of stepping the environment 2
6 DISCUSSION
6.1 PPO AS A NATURAL POLICY GRADIENT METHOD
Our experiments provide strong empirical support that decoupling the proximal policy from the behavior policy can be beneficial: it can be used to make more efficient use of stale data, achieve batch invarinnee,, and to slightly improve sample efficiency outright. This implies that the usual
"These costs are calculated as follows. PPO has forward-oniy and 3 forward-baad backward passes per environment step, to which PPO-EWMA adds 3 forward-only passes. PPG has I forward-ond only and 7 forward- backward passes of both networks per environment step, to which PPG-EWMA adds I forward-only pass of the policy network. forward-backward pass has times the of forward-only pass.
'In practice, including the time taken to step the environment, the EWMA increased wall-clock time by 19% for PPO and by 3% for PPG, but our PPG-EWMA implementation included an additional unnecessary forward pass of EWMA of the value network.
8

---

Under review as a conference paper at ICLR 2022
justification for PPO's surrogate objectives, that they approximate trust region methods, is subtly flawed. Trust region methods keep the policy close to the behavior policy, but it does not matter how far from the behavior policy move specifically, only that stay close to some recent policy, or in other words, that we do not move too fast. Instead, we speculate that PPO is better viewed as a natural policy gradient method (Kakade (00T). These methods select updates that efficiently improve performance relative to how much the policy is changed.
This conflicts with the results of Schullan et al. (2015), which found constraining updates relative to the behavior policy to be beneficial. With the benefit of hindsight, we believe that at that time, constraint methods had hyperparameters that were easier to tune, but that with the advent of PPO's clipped objective and various normalization schemes, this tends to no longer be the case.
6.2 PRACTICAL ADVICE FOR POLICY OPTIMIZATION AT SMALL BATCH SIZES
When solving challenging problems using reinforcement learning, it is often beneficial increase the batch size to reduce gradient variance. But this is often prohibited by computational resources such GPU memory, especially with the trend of increasingly large models. The benefit of batch size-invariance is that we can instead train for longer while adjusting other hyperparameters.
However, when working in a new domain, we may need use small batch size without knowing which hyperparameters would have worked well at larger batch sizes. We therefore attempt to distill our findings into practical advice for getting policy optimization to work well in new domain at small batch sizes. Our advice, much of which is already folklore, is as follows:
By far the most important hyperparameter to tune is the learning rate (or Adam step size). Once it has been tuned for a certain batch size, it can be adjusted formulaically for use at other batch sizes using the rules given in Section 3, as long as the batch size remains small.
Consider setting the number of policy epochs (E in PPO or Fx in PPG) to 1, at least initially, This is the easiest way maintain stability even if not enough ratios are being clipped. Furthermore, multiple policy epochs are less likely to be beneficial when the iteration batch size is small.
If using clipping, monitor the fraction of ratios clipped. If it is much less than 1%, then it is probably beneficial to increase the iteration batch size or to use PPO-EWMA with a high Sprox- If it is much more than 10% with 1 policy epoch or 20% with multiple policy epochs, then this is often a sign that the learning rate is too high.
If using advantage normalization, monitor the advantage standard deviation estimates. If estimates oscillate by factor of 10 or more, then it is probably beneficial to perform normalization using data from more iterations.
7 CONCLUSION
Policy optimization algorithms such as PPO typically control the size of policy updates using a recent policy we call the proximal policy. We have shown that this policy can be safely decoupled from the behavior policy, which is used to collect experience. We introduced PPO-EWMA and PPG-EWMA, weiightt moving average
variants of PPO and PPG in which the proximal policy is an of the current policy. These variants allow stale data to be used more efficiently, and are slightly more sample efficient outright. Finally, we showed how make these algorithms batch size-invariant, meaning that when the batch size is changed, we can preserve behavior, as a function of the number of examples processed, by changing other hyperparameters (as long as the batch size is not too large). We discussed our findings, which have both theoretical and practical implications for policy optimization.
'The iteration batch size can be increased without changing the sampling or optimization batch size by simultaneously increasing the number of timesteps per rollout (2') and the number of minibatches per epoch. However, 7 also affects the amount of hootstrapping performed, and so the GAF bootstrapping parameter (A) may also need to he adjusted compensate.

---

[redacted for Under review as a conference paper at ICLR 2022
8 ACKNOWLEDGMENTS
We thank the anonymous reviewers for their detailed and thoughtful feedback. anonymity]
REPRODUCIBILITY STATEMENT
All of our experiments be re-run and all figures re-created using the code at [redacted for anonymity].
REFERENCES
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint larXiv: /512.07788 2019.
Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. arXiv preprint 20X1:2009.044466 2020.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In International Conference Machine Learning, pp. 1407-1416. PMLR, 2018.
Noah Golmant, Nikita Vemuri, Zhewei Yao, Vladimir Feinberg, Amir Gholami, Kai Rothauge, Michael W Mahoney, and Joseph Gonzalez. On the computational inefficiency of large batch sizes for stochastic gradient descent. arXiv preprint arXiv:1811.1294 2018
Priya Goyal. Piotr Dollár, Ross Girshick, Pieter Noordhuis. Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch SGD: Training ImageNet in hour. arXiv preprint arXiv: T 1706.02677 2017.
Chris Hardin. Does batch size matter?, 2017. URL
janest com/
Seatch-size-mare
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint /80.050003 2018.
Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforcement learning algorithms. In International Conference on Machine Learning, pp. 4962-4973, PMLR, 2020.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14:
1531-1538, 2001.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1mXiv:1412.69800 2014.
Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applica- fions, volume 35. Springer Science & Business Media, 2003.
Lisa M La and Gary G Koch. Rank score tests. Circulation, 114(23)2252--2533 2006.
Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation:
Understanding the effectiveness of sgd in modern per-parametrized learning. International Conference on Machine Learning, pp.
PMLR, 2018.
Stephan Mandt, Matthew D Hoffman, and David M Blei. Stochastic gradient descent approximate Bayesian inference. arXiv preprint arXiv:1704.0000 04289 2017.
Sam McCandlish, Jared Kaplan, Dario Amodei, and the OpenAl Dota Team. An empirical model of latce-batch training. arXiv preprint arXix: /8/2.06122
2018.
10

---

Under review as a conference paper at ICLR 2022
OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysaw Dbiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, llya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0668) 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 07710077777777 2017.
Christopher Shallue, Jaehoon Lee, Joseph Antognini, Jaseha Sohl- Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint aoXiiv
2018.
Samuel L. Smith and Quoe V Le. bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXi: 1700:01710 2017.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arxiv:1711.00445 2017.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror descent policy optimization. arXiv preprini a000:: 09814 2020.
PH van Elteren. On the combination of independent two sample tests wilcoxon. Bull Inst Intern Staist, 37:351-361, 1960.
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger Grosse. Which algorithmic choices matter at which batch sizes? Insights from noisy quadratic model. Advances in neural information processing systems, 32:8196 1900777, 2019.

---

Under review as a conference paper at ICLR 2022
A
PSEUDOCODE FOR PPO-EWMA
We provide pseudocode PPO as well as PPO-EWMA to make clear what changes need to be made:
Algorithm PPO
Algorithm 2 PPO-EWMA
for iteration = 1,2,... do
for iteration =,, 2, ... do for actor = 2,. N do
for actor .,,, .N do
Run policy 00.M in environment for T timesteps Compute advantage estimates Ar
Run policy th. in environment for T timesteps
Compute advantage estimates
A1. end for for epoch = 1, 2,..., E do
Å Ag. Ar end for for epoch = 1.2...., E do
for each minibatch do
for each minibatch do
Compute T for minibatch Optimize objective Liecouped with respect to 0 on minibatch
Optimize objective with respect to 0 on minibatch
pprox f- EWMA Sprox
end for end for Ool f- end for
end for end for
Oechas 0 end for
The expression EWMA Sprox (0) is shorthand for
0, + Sprox 0--1
3 Op 0,
Prrox 9o
+ Sprox
+ Pprox
3 prox
where 0o. 01,.. Og are the values of 0 after each gradient step. In practice, we compute this incre- mentally by initializing & pprox and w t- and treating the update pprox + EWMA Sprox (0) as shorthand for
Wnew f- 1 + Sprox W
1
prox
+ Sprox
0 prox
Wnow
Wnew
W f- Wnew-
For PPG-EWMA, we make the same changes to the policy phase, while leaving the auxiliary phase unchanged. However, the EWMA should be reinitialized at the start of each policy phase, since changes a lot during the auxiliary phase.
Code for both PPO-EWMA and PPG-EWMA may be found at [redacted for anonymity].
12

---

Under review as a conference paper at ICLR 2022
B HYPERPARAMETERS
All experiments were on Procgen's hard difficulty, without frame stack, using the convolutional neural network from IMPALA (Espeholt et al. 2018). Unless stated otherwise, experiments lasted for 100 million environment steps.
Table I: Default hyperparameters shared between PPO and PPG.
Hyperparameter
Value
Workers
4
Parallel environments per worker
64
Timesteps per rollout (7)
256
Minibatches per epoch
8
Adam step size (a)
5 10-4
Value function coefficient Entropy coefficient
0,5
0.01
PPO clipping parameter (e)
0.2
GAE discount rate (?)
0.999
GAE bootstrapping parameter (A)
0.95
Reward normalization?
Yes
Advantage normalization?
Yes
Table 2: Default PPO-specific hyperparameter.
Hyperparameter
Value
Epochs (F)
3
Table 3: Default PPG-specific hyperparameters.
Hyperparameter
Value
Policy iterations per phase (N₂)
32
Policy phase policy epochs (Ex)
Policy phase value function epochs (Ev)
Auxiliary phase epochs (Eaux)
6
Auxiliary phase minibatches per epoch
16.V
Auxiliary phase cloning coefficient (Betone)
1
For the purpose of the artificial staleness experiments, we clipped The to keep the ratio Xe below 100, for numerical stability.
For PPG-EWMA, we chose the default EWMA decay rate Sprox such that the center mass of the EWMA,
1, cqualed the number of minibatches per policy phase iteration (8), so that the maximum age of the proximal policy is the same in PPG and PPG-EWMA, We tuned this on the 2 and T- 1-8erox
2 prax
32, but did not find these to perform better. We did not re-tune Sprox on the last 8 Procgen environments or on PPO-EWMA.
first 8 of the 16 Procgen environments by also trying T-Pprex
Table 4: Default PPO-EWMA and PPG-EWMA specific hyperparameter.
Hyperparameter
Value
Proximal policy EWMA decay rate (Sprox) 0.889
For the batch size-invariance experiments, made the following changes the above defaults:
13

---

Under review as a conference paper at ICLR 2022
We reduced the number of parallel environments, first by reducing the number of workers from 4 to I, and then by reducing the number of parallel environments per worker from 64 to 16 to 4 to 1.
For the policy phase, we adjusted the Adam step size (a), the proximal policy EWMA decay rate (Sprox), advantage normalization, and the number of policy iterations per phase (N₂) in the way described in Section 4
For the auxiliary phase, we initially tried adjusting the Adam step size in the same way as for the policy phase. This worked well in terms of batch size-invariance, prohibitively large wall-clock times at small batch sizes, due to the large number of auxiliary epochs. We therefore simply kept the auxiliary phase minibatch size per worker constant, and only adjusted the Adam step size when reducing the number of workers, not when reducing the number of parallel environments per worker.
resulted in
14

---

Under review as a conference paper at ICLR 2022
C PREVIOUS WORK ON BATCH SIZE-INARRIANEE
There has been much previous work on batch size-invariance. The underlying idea of modeling SGD numerically integrating stochastic differential equation (SDE) is long-established, going least as far back as Kushner & Yin (2003), More recently, Mandt et al. (2017) and Hardin (2017) observed that changing the learning rate proportion the batch size leaves the SDE unchanged, and therefore that SGD is batch size-invar inzarianti at small batch sizes. Meanwhile, Goyal et al. (2017) empirically validated this invariance on ImageNet, |Smth & Le (2017) and [mmihh et al.) (2017) provided further empirical validation for this, as well as for rules describing how the optimal learning rate changes with momentum and training set size.
The term critical batch size for the batch size beyond which SGD is no longer batch size-invariant was introduced by Ma et al. (2018), Since then a number of works have studied how the critical batch size varies between problems, mostly with the motivation of improving training efficiency at large batch sizes (in contrast to our work, which focuses on maintaining performance at small batch sizes). Shallue et al. (0188) studied the effect of architectures, datasets and different forms of momentum the critical batch size, and introduced the term perfect scaling for the regime batch ieeiiaariineee Golmant al. (2018) also studied the effect of dataset complexity and size the critical batch size. McCandlish et al. (2018) measured the critical batch size in a range of domains, and showed that can be predicted using measure of the noise-to-signal ratio of the gradient known the gradient noise scale. Finally, Zhang et al. (2019) studied the effect of curvature on the critical batch size using a noisy quadratic model, and showed that preconditioning can be used increase the critical batch size.
15

---

Under review as a conference paper at ICLR 2022
D ADAM SQUARE ROOT STEP SIZE ADJUSTMENT
In Section 3
we stated that SGD and Adam have different learning rate adjustment rules. To compensate for the batch size being divided by constant must divide the SGD learning rate by c, but divide the Adam step size by /c (Hardin 2017).
The reason for the difference is that Adam divides the gradient by rumning estimate of the root mean square gradient. If the gradient vector at the current step is Jt. then this denominator approximately
Var [9c] E [
E Lool
E [oo] + Var [
E [9d]
E [gd
n
where all operations including the variance operator are applied componentwise, is the batch size, and B is componentwise version of the gradient noise scale defined by McCandlish et al. (2018). measure of the noise-to-signal ratio of the gradient that approximates the critical batch size. Hence if the batch size is small compared to the critical batch size, then B > for most components, and so the Adam denominator is approximately proportional v'i
follows that the batch size is divided by some constant e, then the Adam denominator multiplied by approximately /c (providing the batch size is small compared to the critical batch size). Hence Adam is effectively dividing the learning rate by /c automatically, and so the step size a only needs to be adjusted by additional v to effectively divide the learning rate by overall.
This all ignores Adam's c hyperparameter, which is usually negligible, but is sometimes used to interpolate between Adam and momentum SGD.
To verify the square root rule for Adam, we conducted an ablation of our batch inzarinnree adjustments, except that divided the Adam step
experiments, in which made the exact size by e instead of by vf. Our results are shown in Figure When compared with Figure 2 this clearly shows that the square root rule is superior in setting. Full results each of the individual environments can be found in Appendix G.2
PPG-EWMA with linear instead of square Adam step size adjustment
0.8
Moan normalized retum
Batch size
Default
Defiult 4
Default 16
Default/ 64
0.0
Default / 256
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps
x10*
Figure 5: PPG-EWMA at different batch sizes, averaged over all 16 Procgen environments, with hyperparameters adjusted as in Figure except with linear rather than square root adjustment to the Adam learning rate. Mean and standard deviation over 3 seeds shown.
Our results are in tension with those of Smith al. (2017), who verified batch size- inveeiiieriinee for Adam using the linear rather than the square root rule. However, they achieved a lower degree of batch size-invariance with Adam than with SGD (see Figure 4 in that work), and moreover, results
16

---

Under review as a conference paper at ICLR 2022
show that the batch size needs to be reduced significantly before the difference between the two rules noticeable. We believe that this accounts for their experimental results, and that the square root rule is superior in general (with the exception of when Adam's 's hyperparameter is high enough for it to behave like momentum SGD).
17

---

Under review as a conference paper at ICLR 2022
E ADAM 31 AND B2 ADJUSTMENTS
As discussed in Section there is an additional adjustment one should make when using Adam, other than the step size a. To compensate for the batch size being divided some constant c, one should also raise the exponential decay rates 31 and B2 to the power of l/c (Hardin 2017).
We omitted this adjustment in most of experiments, and of batch size invariance. For all except one enviromment, the difference in normalized return between the largest and smallest batch sizes the end of training was at most 0.11 (see Figure 3). For these environments, it would probably have required many additional experiments to detect any further improvement that adjusting 31 and B3 might provide. However, for the Heist environment, this difference was 0.55. We hypothesized that this might be explained by the fact that we did not adjust 31 and 32.
still able achieve high degree
We therefore conducted version of our batch size-invariance experiments in which we either adjusted only 32 using the above rule, or adjusted both 31 and 82. Our results are shown Figure both cases there was still large difference in performance the largest and smallest batch sizes.
No further adjustments
Adjustment to 8g only
Adjustments to & and 32
Batch size
Default
Default /4
Default / 16
Retuen
Default 64
Default / 256
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
Environment steps
x10"
Environment steps
x10*
Environment steps
x10*
Figure 6: PPG-EWMA at different batch sizes on Heist, with hyperparameters adjusted as in Figure 2. together with further adjustments to Adam's 31 and 83 hyperparameters as indicated. Mean and standard deviation over 3 seeds shown.
18

---

Under review as a conference paper at ICLR 2022
F HYPOTHESIS TESTS FOR ABLATIONS
As discussed in Section 3 we conducted hypothesis tests to check the statistical significance of the effects produced by the ablations to our batch size-invariance experiments.
Our primary metric for measuring batch size-invariance was the difference in final performance of the algorithm at different batch sizes. To get complete picture of how important ablations were different batch sizes, we compared our default (largest) batch size with each of the other batch sizes, and tested the hypothesis that the difference was larger for the ablation. This resulted in 16 hypotheses. corresponding to the 4 ablations and the non-default batch sizes. To test each hypothesis, we used van Elteren test (van Elteren 1960). stratified version of the Mann-Ve Mann-Whitney Whitney U-test, treating the different environments strata. This gives non non-paramttric Z-test of the null hypothesis that for each of the environments, the probability the ablation outperforming the original experiment is the same as the probability of the original experiment outperforming the ablation Vange & Koch 2006). To reduce noise (and thereby increase statistical power), we measured average performance over the last 4 million timesteps (the length of single PPG phase). We used a significance level of 0.1 1% and applied a Bonferroni correction to account for multiple comparisons.
Our results shown in Table For ablation (b), the difference is only significant at the smallest batch size. For all other ablations, the difference is significant at every batch size, exeept for the largest batch size for ablation (d).
Table 5: Effect sizes (and Z-scores, in parentheses) for each of our hypotheses. The effect size is difference of differences in final normalized return, between the ablation and the original experiment and between the different batch sizes. bold are the effect sizes found be signifcant at the 0.1% % level after applying Bonferroni correction (i.e., with a one-tailed p-value below 0.001/1, or equivalently, Z-score above 3.84).
Batch sizes:
(a) No Adam step size adjustment
(b) No adv. norm. adjustment
(c) No EWMA adjustment
(d) No EWMA, just PPG
Default vs
Default /
0.046
(5.02)
00.04
(-1.53)
0.020
(4.58)
0.019
(3.27)
Default 16
0.347
(7.31)
-0.03
(-1.53)
0.021
(4.47)
0.016
(4.04)
Default / 64
0.621
(6.98)
0.054
(-0.11)
0.027
(5.02)
0.042
(5.13)
Default 256
0.709
(6.87)
0.224
(4.47)
0.041
(4.80)
0.030
(4.58)
19

---

Under review as a conference paper at ICLR 2022
G
RESULTS ON INDIVIDUAL ENVIRONMENTS
G.1 ARTIFICIAL STALENESS
CoinRun
StarPilot
CaveFlyer
Dodgeball
6
Retum
10
2-
2
2-
0-
FruitBot
Chaser
Miner
Jumper
7.5
Return
10-
6.0
2.5
0-
Leaper
Maze
BigFish
Heist
6-
15
Retur
10
Climber
Plunder
Ninja
BossFight
10.0
7.5
Return
Staleness None iseration isorations
5.0
5
2.5
2-
inerationns
0.0
01
16 irerations
32 inerations
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment steps
x
Environment steps
x10F
Environment steps
x10#
Environment steps
10
Figure 7: Results from Figure (PPO with Telia = Te split across the individual environ- ments, Mean and standard deviation over 4 seeds shown.
20

---

Under review as a conference paper at ICLR 2022
CoinRun
StarPilot
CaveFlyer
Dodgeball
10
Return
FruitBot
Chaser
Miner
Jumper
20
10
15
10-
Leaper
Maze
BigFish
Heist
10-
30
Climber
Plunder
Ninja
BossFight
10
20
Retum
Staleness None lieration
10
ieeratioss
derations
16 iterations
32 serations
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment steps
x10
Environment steps
x10
Environment steps
x10
Environment steps
x10
Figure Results from Figure [b) (PPO with decoupled objective) split across the individual environments. Mean and standard deviation 4 seeds shown.
21

---

Under review as a conference paper at ICLR 2022
CoinRun
StarPilot
CaveFlyer
Dodgeball
5-
2-
FruitBot
Chaser
Miner
Jumper
10
Return
10
Leaper
Maze
BigFish
Heist
15
Return
10
3-
2
Climber
Plunder
Ninja
BossFight
12.5
10.0
Retum
6
Staleness None lieration
ieeaatioss
ieratiioss
2-
16 iterations
32 serations
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment steps
x10
Enviromment steps
x10#
Environment steps
x10
Environmeut steps
x10
Figure 9: Results from Figure (c) (PPO with Teas = To ) split across the individual environ- ments. Mean and standard deviation over 4 seeds shown.
22

---

Under review as a conference paper at ICLR 2022
G.2
BATCH SIZE-INVARIANCE
CoinRun
StarPilot
CaveFlyer
Dodgeball
12.5
10
10.0
7-
7.5
10
5.0
5
2.5
FruitBot
Chaser
Miner
Jumper
10
25
12.5
20
6
10.0
Retum
15
2-
2.3
Leaper
Maze
BigFish
Heist
10
30
Retum
20
4-
10
2.
Climber
Plunder
Ninja
BossFight
10
10
10.0
20
8-
Retur
7.5
15
Batch size
5.0
Default
Defanh /4
2.5
Default 64
2-
Default
0.0
Environment steps
x x10
Environment steps x <0
Environment steps
x10
Environment steps
100
Figure 10: Results from Figure 2 (PPG-EWMA with all batch size-invariance adjustments) split across the individual environments. Mean and standard deviation over 3 seeds shown.
23

---

Under review as a conference paper at ICLR 2022
CoinRun
StarPilot
CaveFlyer
Dodgeball
12.5
10
20
10.0
7.5
10.
5.0
2,5
0-
FruitBot
Chaser
Miner
Jumper
25
20
7.0
10.0
Return
5.0
7.5
10-
2.5
0.0
Leaper
Maze
BigFish
Heist
8
01
Climber
Plunder
Ninja
BossFight
10
12.0
10.0
6-
Batch size
Detault
5
Default 16
Defailt 64
0
2
Detauit 256
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Environment steps
x10
Environment steps
x x100
Environment steps
x10
Eavironment steps
x10
Figure 11: Results from Figure a) (no Adam step size adjustment) split across the individual environments. Mean and standard deviation over 3 seeds shown.
24

---

Under review as a conference paper at ICLR 2022
CoinRun
StarPilot
CaveFlyer
Dodgeball
10
10.0
Return
7.5
2.5
0
FruitBot
Chaser
Miner
Jumper
25-
20
10
Return
6
10
0.
Leaper
Maze
BigFish
Heist
10
30
Relurn
20
2-
Climber
Plunder
Ninja
BossFight
25
10
Retum
Batch size
Default
Defuul/ 16
Defaol 54
Defantt 256
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment steps
x10
Environment steps
x10*
Environment steps
x10*
Environmeut steps
x10
Figure 12: Results from Figure b (no advantage normalization adjustment) split across the individual environments. Mean and standard deviation over seeds shown.
25

---

Under review as a conference paper at ICLR 2022
CoinRun
CaveFlyer
Dodgeball
extimate
109
100
10a
10
10
10
10"2
10
BruitBot
Chaser
Miner
Jumper
Adranage estimate
10
10a
100
10P
10-2
10-1
10-1
10
10
10
Leaper
Maze
Bighish
Heist
10P
100
100
Advantage std. enimase
10-3
10-1_
10
10
10
10-2
10
10
30-9
Climber
Plunder
Ninja
BossFight
Advantage st2, estmmale
10
10
10-1
Batch size
Default
10
10
10-2_
Default/ 16
Default/
10-2
Default 256
0.0
0.5
1.0
0.0
0.5
1.0
11.0
0.5
1.0
0.0
02.5
1.0
Environment steps
x100*
Environment steps
10
Environment stegs
1100
Eaviroument stops
100
Figure 13: Advantage standard deviation estimates for the results from the previous figure. We plot estimates from the first seed only and perform smoothing, since interested in the amount of oscillation. Note that performance degrades with no advantage normalization adjustment only once the estimates oscillate by factor of around 10 more.
26

---

Under review as a conference paper at ICLR 2022
CoinRun
StarPilot
CaveFlyer
Dodgeball
12.6
25
10
10.0
16
2.5
2-
FruitBot
Chaser
Miner
Jumper
10
25
12.5
20
10.0
Return
15-
10
Leaper
Maze
BigFish
Heist
10
30
Relurn
20
4-
3.
4-
2-
Climber
Plunder
Ninja
BossFight
12.5
25
Retum
7.5
15
Batch size
Default
10
Defauh/ 16
Default 54
2
Defanit/ 256
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment steps
x
Environment steps
x10
Environment steps
x10*
Environment steps
x10
Figure 14: Results from Figure 2 c) (no EWMA adjustment) split across the individual environments. Mean and standard deviation over seeds shown.
27

---

Under review as a conference paper at ICLR 2022
CoinRun
StarPilot
CaveFlyer
Dodgeball
25
12.5
20
10.0
7-
15
7.5
10
5.0
5--
2.5
0-
FruitBot
Chaser
Miner
Jumper
10
12.5
20
10.0
Return
10
Leaper
Maze
BigFish
Heist
30
Return
8-
20
4-
2-
Climber
Plunder
Ninja
BossFight
10
12.5
10
25
30.0
20
Return
7.5
Batch size
5.0
Default
2.5
Defauh/ 16
Default 54
2-
2
Defanit/ 256
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment steps
x
Environment steps
x10
Environment steps
x10*
Environment steps
x10
Figure 15: Results from Figure d) (no EWMA at all, just PPG) split across the individual environ- ments. Mean and standard deviation over 3 seeds shown.
28

---

Under review as a conference paper at ICLR 2022
CoinRun
StarPilot
CaveFlyer
Dodgeball
25
15
7.5
10
5.0
5-
5-
2.5
FruitBot
Chaser
Miner
Jumper
12.5
20
10.0
6-
Return
15
7.5
10
4-
5.0
2-
2.5
Leaper
Maze
BigFish
Heist
10
7
8-
Return
20
5
4-
10
3-
4-
2-
Climber
Plunder
Ninja
BossFight
10
Retur
Batch size
10
Default
Default
Defauh/ 16
Default 54
Defanit/ 256
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Environment steps
x 1
Eavironment steps
x10
Environment steps
x10
Environment steps
x10
Figure 16: Results from Figure 5 (PPGEEWMA with linear instead of square root Adam step size adjustment) split across the individual environments. Mean and standard deviation over seeds shown.
29

---

Under review as a conference paper at ICLR 2022
G.3
EWMA COMPARISON
CoinRun
StarPilot
CaveFlyer
Dodgeball
25
12.5
10-
20
10.0
15
7.0
5.0
FruitBot
Chaser
Miner
Jumper
12.5
20
Retum
4
10
4-
2-
2.5
2-
Leaper
Maze
BigFish
Heist
8
10
4-
Climber
Plunder
Ninja
BossFight
10
10
20
15
5.0
10
PPO-EWMA
2.5
PPG
PPG-EWMA
0.0
Environment steps
x 1
Environment steps
x10
Environment steps
x10*
Environment steps
x10
Figure 17: Results from Figure (performance of all 4 algorithms) split across the individual environments. Mean and standard deviation over 4 seeds shown.
30

---

Under review as a conference paper at ICLR 2022
H ROLE OF THE PROXIMAL POLICY EWMA DECAY RATE
In this section we discuss the role of the hyperparameter Sprox PPO-EWMA and PPG-EWMA in more depth.
Recall that in PPO-EWMA and PPG-EWMA, the proximal policy network parameter vector prox an exponentially-weigheed moving average (EWMA) of the policy network parameter vector meaning that
Or + Sprox + 8 prox 0.-2 +
3
0o
8 prox
prox
+ Sprox
+ Pprox
3
prox
where 0o, .1,0..,.0................ O2 the values of 0 after each gradient step.
Rather than working with the decay rate Sprax of this EWMA directly, it is conceptually clearer to work with the center of mass of this EWMA,
+ Sprox! 1 + Pprnx + ... S proxt
COMprox == lim
Sprox 1.
t-+00 + Sprox + $33 1.rox +.. + Pprox
This is average age of term in the EWMA in the limit as t + oc, and so if were to follow straight line path for example, then 0 Pprox would be approximately proportional to COMprec
Suppose then that we halve COMprax- What is the effect of this?
Consider the gradient of the KL divergence from the current policy to the proximal policy, function of the proximal policy parameter vector,
Grad-KL. (0prox) == V KL [Tpprxx (e) [s)]
Since KL divergence is always greater than or equal O, with equality if and only if the input distributions are equal, Grad-KL (0) = O, and hence, to first-order, Grad-KL (fprox) is a linear function of f peox. Therefore halving COMprox should have similar effect halving the KL penalty coefficient B. In other words, we should be able to compensate for halving COMprox by doubling the KI penalty coefficient.
Intuitively, the KL penalty acts like rubber band pulling the policy towards the proximal policy. Halving COMprox is analogous to attaching the rubber band point half as far away, while doubling the KL. penalty coefficient is analogous to doubling the thickness of the rubber band. Doing both simultaneously results in the same overall force.
We tested this hypothesis using hyperparameter grid search over the EWMA center of mass and the KL penalty coefficient, for PPG-EWMA on StarPilot. We used our smallest batch size, along with corresponding batch size-invariance adjustments, to allow the greatest scope for reducing Oprox without making the EWMA degenerate into averaging over a single data point.
Our results shown in Figure [8] The diagonal banding clearly demonstrates the expected effeet. However, the effect only holds locally: as COMprox is continually halved and the KL penalty coefficient is continually doubled, performance gradually degrades.
We believe that this is because reducing COMprox has a second-order effect, which is to increase the smaller effective sample
variance of 0 prex.. This is both because the EWMA is averaging size, and because 0r f._ has a lower signal-to- gegnal-to-noise no-noise ratio as decreases. Therefore if COM prox has been halved too many times, should expect longer be able to fully compensate for this by continuing to double the KL penalty coefficient.
31

---

Under review as a conference paper at ICLR 2022
Return
256
17.5
128
KL penalty coefficient (3)
12.5
32
16
10.0
7.5
5.0
2
2.5
512
EWMA center of muss
I- Sprox
Figure 18: Performance on PPG-EWMA StarPilot after 20 million environment timesteps, using single parallel copy of the environment along with our batch invariiicee adjustments. The default hyperparameter settings correspond to square in the bottom right corner. Mean over 2 seeds shown.
32


Reviews:
[{"review_id": "pQJg50SpRJ3", "reviewer": "Reviewer_kR2g", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Firstly, the paper is clearly written, is simple to read, and flows well. I particularly like that the authors do not oversell any of the results. In terms of motivation, batch size invariance is an important and interesting property and insights specific to RL could be useful for future research. \n\nOn the other hand, I do not understand the main takeaway from the paper. There are two clear contributions here, one of the decoupling objective and the other of achieving batch size invariance. Currently, I am unable to place either of these as the main objective/problem of the paper. If it's the decoupling, then it should lead to better batch size invariance than the standard, coupled policy agent. I think this is plotted in Figure 2 (d), where the coupled PPG agent is used with all optimization adjustments but achieves similar batch invariance as the decoupled PPG agent as in Figure 2. So it seems like batch invariance can be achieved irrespective of the algorithm, i.e. if it's coupled or decoupled. Furthermore, as it is, the decopling only leads to a marginal improvement over the coupled agent. Similarly, if batch size invariance is the main objective, then I don't see how the decoupling idea is worth looking at. Any comments from the authors on this would be really helpful.\n\nMinor Comments/Questions:\n\n1. Typo at line 4 of section 3.1.\n2. Last line of section 3.1 \"Once the step size is small enough...\": Does this small enough batch size have anything to do with the critical batch size? I assume not right? \n3. Last line of section 6.1 \"We speculate that updates should improve...\": Are you trying to say that more the policy differs a lot from the behavior policy, the more performance improvement we'll see? I do not completely understand why the conclusion here is different from that of the PPO paper.\n4. Second paragraph of section 6.2: The first sentence talks about large batch sizes but the next line mentions advice regarding small batch sizes. They seem contradictory to me. Can you clarify the wording here?\n5. The decoupling between behavior and proximal policies is also used in the recently introduced MDPO [1] algorithm (Section 5.1). Another useful reference could be [2]. Particularly, if one is familiar with the NPG or Mirror Descent connection of PPO/PPG, it should be fairly straightforward to see that coupling the behavior and proximal policies is not a hard requirement (something that maybe is not as clear from the PPO paper). \n\nReferences:\n\n[1] Mirror Descent Policy Optimization.\n\n[2] A functional mirror ascent view of policy gradient methods with function approximation.", "summary_of_the_paper": "The paper proposes decoupling the behavior and proximal policies used in policy optimization algorithms such as PPO and PPG. Typically, the behavior policy itself is used as the proximal policy, i.e. the policy to which updates need to be close to (maintain trust region). In such a case, using stale or old data can lead to bad performance, leading to on-policy methods using data from the only most recent sampling iteration. The paper then talks about the batch invariance property, which refers to the performance remaining as is when changing the batch size. This is typically achieved by adjusting the optimization parameters such as learing rate (ex. if batch size is doubled, learning rate should be doubled as well). Finally, the authors perform experiments to test how the decoupling affects the usage of stale data, how batch invariance can be achieved with the decoupled objective, and how much better does the decoupling perform as compared to when the same policy is used as the behvior and proximal policies. ", "summary_of_the_review": "The paper does a good job in explaining the proposed method, in running appropriate experiments, and in terms of writing. However, it fails to motivate a single objective or problem which is addressed by the proposed method. I do not see what the main contribution is and so am currently recommending a weak reject. I am happy to revise the review if the authors can address these concerns.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UWFyBiiqIlC", "reviewer": "Reviewer_VVcj", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weaknesses:\n\n1. In experiment 5.1, the author created artificial staleness data to evaluate their method. By comparing Figure 1(b) and Figure 1(c), we see that the vanilla PPO algorithm performs better than their proposed algorithm! If PPO with $\\pi_{old}=\\pi_{behavior}$ is much better than the proposed algorithm, why does the author want us to use their method?\n\n2. By comparing Figure 2(a) and Figure 2(d),  we see that the most critical factor is the Adam step size instead of the proposed EWMA method. I have no idea the meaning of the proposed EWMA method, and we can obtain nearly nothing when using the EWMA method. And the way of how to adjust the learning rate of Adam is not new. So, what is the contribution of this paper?\n\n3. In Figure 3,  we see EWMA has minimal improvement on current deep RL methods. I doubt the value of this work.\n\n", "summary_of_the_paper": "This paper proposed a new method to deal with batch size-invariance for policy optimization.", "summary_of_the_review": "This paper wants to propose a new deep RL method, which makes use of their decoupled objectives. However, I can see slight improvement over their methods. And the proposed EWMA algorithm contributes little to the batch size invariance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SHGAT2k2TJA", "reviewer": "Reviewer_tiiD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper takes on an essential issue in reinforcement learning in designing algorithms that require less hyperparameter tuning. The ideas in this paper could be beneficial to the community and make it easier for practitioners to use these algorithms. This paper could be excellent, but as currently written needs some significant improvements. The main two areas where the paper can be improved are the clarity of presentation and the design of the experiments.\n\nClarity: \nAs it is currently presented, many of the methods and ideas are described with vague or verbose language, making it more challenging to interpret the paper's main ideas. I will detail some of these below, but the central theme of these changes is that the paper needs to be more precise in its statements. These improvements can largely be achieved by introducing mathematical definitions and providing formal statements of the methods. \n\n\"we can preserve behavior, as a function of the number of examples processed, by changing other hyperparameters (as long as the batch size is not too large)\" -- what does the batch size being too large mean? The batch size limit should be quantified so that a user can understand what large enough actually means. \n\nDefinition of L^{KLPEN} uses the symbol \\hat E_t. What is the \\hat E? This is not a common mathematical symbol and is never defined. \n\nThe critical step size is mentioned often in the paper, and a description of it is given in section 3.1, but a formal definition would remove any ambiguity of the interpretation. \n\nIn Section 3.1, the derivational derivative symbol \\nabla_\\theta is used for what is meant to be a partial derivative. Note that the gradient is the vector of partial derivatives with respect to all inputs to a function, not just a single one. Following standard mathematical symbols would make this clearer. \n\n\"If the batch size is small compared to the critical batch size, then the difference between θ t and θ t+1 is mostly noise, and moreover this noise is small compared to the total noise accumulated by θ_t over previous updates.\" -- what does mostly noise mean? Do second-order effects not impact the process? For a paper studying batch size invariance, it is light on providing precise details of the source of errors in these approximations. \n\n\"To compensate for the batch size being divided by c,\" -- \"c\" is never defined in the main paper or the appendix. The rest of the statements cannot be correctly interpreted. \n\n\"Hence Adam is effectively dividing the learning rate by √ c automatically, and so the step size α only needs to be adjusted by an additional √ c to effectively divide the learning rate by c overall.\" -- This is the perfect example where having mathematical definitions of the operations or pseudo code of the invariant update would drastically improve clarity. \n\n\"PPO is automatically optimization batch size-invariant, as long as the optimization algorithm it uses (such as SGD or Adam) is as well.\" -- This is unproven and contradicts statements that the proximal policy is needed for batch size invariance. \n\n\n\"This demonstrates that the decoupling the proximal policy from the behavior policy is safe and useful.\" -- The use of the word \"safe\" here is probably best avoided since it has many meanings that can be misinterpreted. \n\n\n\nExperiments: \nThe primary issue with the experiments is that they are not designed to show batch size invariance directly. Instead, they focus on seeing if the algorithm has similar performance for different batch size parameters. While having a similar performance for different batch sizes is an expected property of the methods, these experiments cannot determine when batch size invariance should be expected or when it fails. This shortcoming of the experiment design is revealed on page 6 when the paper provides a guess at the failures and offers no empirical evidence. The results do indicate that there is some batch size invariance present. The paper should make it clear what the limits of batch size invariance are. Other comments on the experiments are provided below. \n\nThe experiments use the Procgen environments, which are computationally expensive to use due to image-based features. The result is that only a few trials of each algorithm are run on each environment. Since it is widely known that the performance of RL algorithms has high variance, there are not enough trials being executed to make any claim of significance. Furthermore, since the contributions of this paper have nothing to do with performing well in such complex environments, the paper could greatly benefit from using smaller, more controllable environments and averaging out the effects of noise (Ceron and Castro, 2021). \n\nFigure 2b) \"The decoupled objective allows the correct importance sampling ratio to be used while maintaining the age of the proximal policy, preventing performance from degrading much until the data is very stale.\" -- This is not a result but a hypothesis. No experiment shows that the age of the proximal policy is maintained. There could be many other factors that lead to performance not being degraded. \n\n\"We were able to achieve complete batch size-invariance (with essentially indistinguishable learning curves at every batch size)\" -- This statement is not proven and imprecise. While the learning curves are close together, there is no uncertainty quantification, so these results cannot be trusted. To make such a claim, one needs to have a proper definition of batch size invariance, the corresponding metric, a definition of what complete means, and proper uncertainty quantification. \n\n\"One possible reason is that we did not adjust the Adam β1 and β2 hyperparameters, although we did not ﬁnd the adjustment proposed in Section 3 to noticeably help. We leave further analysis of this to future work.\" -- Were experiments run that show this, or is this a guess? If there is an experiment, then it should be shown. This is also a fundamental experiment that should beconducted to show what is necessary for batch size invariance. Skipping it shows that this work is incomplete. \n\n\"Next most important is the advantage normalization adjustment, which does not matter much in many environments, but matters a lot at the smallest batch sizes in environments for which the advantage standard deviation estimates are particularly noisy.\" -- Was advantage normalization discussed anywhere in the paper? I could not find it. \n\nThere are ablations trying to show what components of the proposed algorithms are necessary for batch size invariance. However, it is unclear how it is being determined if a method has a significant impact on batch size invariance. The effect and uncertainty of each choice needs to be quantified. If the EWMA is unnecessary, how is any discussion about the \"proximal\" policy-relevant to batch size invariance? \n\nThe performance comparison of the algorithms is irrelevant and completely dependent on the choice of hyperparameters. If one wants to compare the performance of these algorithms, then this should be taken into account. The complete algorithm definition idea proposed by Jordan et al. (2020) seems like a good fit for conducting the experiments in this type of work. \n\nIn section 6.2, much \"advice\" is given on these algorithms, but it is unclear if any experiments support this advice. These heuristics need proper evaluation before they should be recommended in a formal academic context. \n\n\nCeron, Johan Samir Obando, and Pablo Samuel Castro. \"Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research.\" International Conference on Machine Learning. PMLR, 2021.\n\nJordan, Scott, et al. \"Evaluating the performance of reinforcement learning algorithms.\" International Conference on Machine Learning. PMLR, 2020.\n", "summary_of_the_paper": "Standard reinforcement learning algorithms such as PPO have several \"batch size\" hyperparameters parameters, and changing them impacts how one needs to choose the step size. To better understand these algorithms and their hyperparameters, this paper investigates batch size invariance, a property that allows the algorithm's behavior to remain constant for changes to the batch size through the modification of other hyperparameters. Batch size invariance has been studied significantly in supervised learning, but these PPO-style algorithms have updates that break the previous batch size invariance techniques. This paper develops two new algorithm variants that provide a way to achieve batch size invariance. Empirical results show that the methods are somewhat effective at providing batch size invariance. \n", "summary_of_the_review": "Due to the lack of precision and weaknesses in the experiments, I cannot recommend this paper for acceptance, but I do acknowledge that the ideas may be helpful. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nzac0vb8zya", "reviewer": "Reviewer_pfgK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\nThis paper study an interesting problem of batch size-invariant.\nThe writing is clear, and the method is easy to understand.\nThe experimental results are sufficient to demonstrate the effectiveness of the proposed method. \n\n\n\nWeaknesses:\n- The paper does not provide any formal theoretical analysis of the batch size-invariant property. Instead, the description is very vague (e.g., Sec 3.1 Sec 4).\"Hence the two update rules behave very similarly.\" To what extent do they behave similarly? Can you give a formal analysis?\n- Using moving average policy (instead of behavior policy) as proxy policy $\\pi_{proxy}$ is interesting. However, this method seems not to be theoretical-justified. What is the motivation of averaging over parameter space? Is averaging over parameter space equivalent to averaging over policy space? Moreover, does this method still enjoy the monotonical improvement property (as in [1])? \n\n[1] Liu B, Cai Q, Yang Z, et al. Neural proximal/trust region policy optimization attains globally optimal policy[J]. Advances in Neural Information Processing Systems, 2019, 32.\n", "summary_of_the_paper": "This paper studies the method to achieve the batch size-invariant for policy gradient algorithms (PPO, PPG). The paper achieves this by decoupling the proximal policy from the behavior policy. The experiments demonstrate the effectiveness of the method.\n", "summary_of_the_review": "This paper presented some interesting ideas of batch size-invariance of policy gradient. \nAlthough the experimental results well demonstrated the effectiveness of their method. \nThe method lacks theoretical justification, and the claim of the batch size-invariant property is vague and weak.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the method to achieve the batch size-invariant for policy gradient algorithms (PPO, PPG). The paper achieves this by decoupling the proximal policy from the behavior policy. Empirical results show that the methods are somewhat effective at providing batch size invariance.

After reading the authors' feedback, the reviewer discussed the paper and they did not reach a consensus. On the one hand, the rebuttal made some reviewers change their minds who appreciated the explanations provided by the authors and the new Figure that better highlights the batch size invariance property.
On the other hand, some reviewers think that there is still significant work to be done to get this paper ready for publication. In particular, it is necessary to improve the theoretical analysis and the evaluation of the empirical results.

I encourage the authors to follow the reviewers' suggestions while they will update their paper for a new submission.
Decision: Reject