Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Standard reinforcement learning algorithms such as PPO have several "batch size" hyperparameters parameters, and changing them impacts how one needs to choose the step size. To better understand these algorithms and their hyperparameters, this paper investigates batch size invariance, a property that allows the algorithm's behavior to remain constant for changes to the batch size through the modification of other hyperparameters. Batch size invariance has been studied significantly in supervised learning, but these PPO-style algorithms have updates that break the previous batch size invariance techniques. This paper develops two new algorithm variants that provide a way to achieve batch size invariance. Empirical results show that the methods are somewhat effective at providing batch size invariance. 

main_review: This paper takes on an essential issue in reinforcement learning in designing algorithms that require less hyperparameter tuning. The ideas in this paper could be beneficial to the community and make it easier for practitioners to use these algorithms. This paper could be excellent, but as currently written needs some significant improvements. The main two areas where the paper can be improved are the clarity of presentation and the design of the experiments.

Clarity: 
As it is currently presented, many of the methods and ideas are described with vague or verbose language, making it more challenging to interpret the paper's main ideas. I will detail some of these below, but the central theme of these changes is that the paper needs to be more precise in its statements. These improvements can largely be achieved by introducing mathematical definitions and providing formal statements of the methods. 

"we can preserve behavior, as a function of the number of examples processed, by changing other hyperparameters (as long as the batch size is not too large)" -- what does the batch size being too large mean? The batch size limit should be quantified so that a user can understand what large enough actually means. 

Definition of L^{KLPEN} uses the symbol \hat E_t. What is the \hat E? This is not a common mathematical symbol and is never defined. 

The critical step size is mentioned often in the paper, and a description of it is given in section 3.1, but a formal definition would remove any ambiguity of the interpretation. 

In Section 3.1, the derivational derivative symbol \nabla_\theta is used for what is meant to be a partial derivative. Note that the gradient is the vector of partial derivatives with respect to all inputs to a function, not just a single one. Following standard mathematical symbols would make this clearer. 

"If the batch size is small compared to the critical batch size, then the difference between θ t and θ t+1 is mostly noise, and moreover this noise is small compared to the total noise accumulated by θ_t over previous updates." -- what does mostly noise mean? Do second-order effects not impact the process? For a paper studying batch size invariance, it is light on providing precise details of the source of errors in these approximations. 

"To compensate for the batch size being divided by c," -- "c" is never defined in the main paper or the appendix. The rest of the statements cannot be correctly interpreted. 

"Hence Adam is effectively dividing the learning rate by √ c automatically, and so the step size α only needs to be adjusted by an additional √ c to effectively divide the learning rate by c overall." -- This is the perfect example where having mathematical definitions of the operations or pseudo code of the invariant update would drastically improve clarity. 

"PPO is automatically optimization batch size-invariant, as long as the optimization algorithm it uses (such as SGD or Adam) is as well." -- This is unproven and contradicts statements that the proximal policy is needed for batch size invariance. 


"This demonstrates that the decoupling the proximal policy from the behavior policy is safe and useful." -- The use of the word "safe" here is probably best avoided since it has many meanings that can be misinterpreted. 



Experiments: 
The primary issue with the experiments is that they are not designed to show batch size invariance directly. Instead, they focus on seeing if the algorithm has similar performance for different batch size parameters. While having a similar performance for different batch sizes is an expected property of the methods, these experiments cannot determine when batch size invariance should be expected or when it fails. This shortcoming of the experiment design is revealed on page 6 when the paper provides a guess at the failures and offers no empirical evidence. The results do indicate that there is some batch size invariance present. The paper should make it clear what the limits of batch size invariance are. Other comments on the experiments are provided below. 

The experiments use the Procgen environments, which are computationally expensive to use due to image-based features. The result is that only a few trials of each algorithm are run on each environment. Since it is widely known that the performance of RL algorithms has high variance, there are not enough trials being executed to make any claim of significance. Furthermore, since the contributions of this paper have nothing to do with performing well in such complex environments, the paper could greatly benefit from using smaller, more controllable environments and averaging out the effects of noise (Ceron and Castro, 2021). 

Figure 2b) "The decoupled objective allows the correct importance sampling ratio to be used while maintaining the age of the proximal policy, preventing performance from degrading much until the data is very stale." -- This is not a result but a hypothesis. No experiment shows that the age of the proximal policy is maintained. There could be many other factors that lead to performance not being degraded. 

"We were able to achieve complete batch size-invariance (with essentially indistinguishable learning curves at every batch size)" -- This statement is not proven and imprecise. While the learning curves are close together, there is no uncertainty quantification, so these results cannot be trusted. To make such a claim, one needs to have a proper definition of batch size invariance, the corresponding metric, a definition of what complete means, and proper uncertainty quantification. 

"One possible reason is that we did not adjust the Adam β1 and β2 hyperparameters, although we did not ﬁnd the adjustment proposed in Section 3 to noticeably help. We leave further analysis of this to future work." -- Were experiments run that show this, or is this a guess? If there is an experiment, then it should be shown. This is also a fundamental experiment that should beconducted to show what is necessary for batch size invariance. Skipping it shows that this work is incomplete. 

"Next most important is the advantage normalization adjustment, which does not matter much in many environments, but matters a lot at the smallest batch sizes in environments for which the advantage standard deviation estimates are particularly noisy." -- Was advantage normalization discussed anywhere in the paper? I could not find it. 

There are ablations trying to show what components of the proposed algorithms are necessary for batch size invariance. However, it is unclear how it is being determined if a method has a significant impact on batch size invariance. The effect and uncertainty of each choice needs to be quantified. If the EWMA is unnecessary, how is any discussion about the "proximal" policy-relevant to batch size invariance? 

The performance comparison of the algorithms is irrelevant and completely dependent on the choice of hyperparameters. If one wants to compare the performance of these algorithms, then this should be taken into account. The complete algorithm definition idea proposed by Jordan et al. (2020) seems like a good fit for conducting the experiments in this type of work. 

In section 6.2, much "advice" is given on these algorithms, but it is unclear if any experiments support this advice. These heuristics need proper evaluation before they should be recommended in a formal academic context. 


Ceron, Johan Samir Obando, and Pablo Samuel Castro. "Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research." International Conference on Machine Learning. PMLR, 2021.

Jordan, Scott, et al. "Evaluating the performance of reinforcement learning algorithms." International Conference on Machine Learning. PMLR, 2020.

summary_of_the_review: Due to the lack of precision and weaknesses in the experiments, I cannot recommend this paper for acceptance, but I do acknowledge that the ideas may be helpful.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposed a new method to deal with batch size-invariance for policy optimization.
main_review: Weaknesses:

1. In experiment 5.1, the author created artificial staleness data to evaluate their method. By comparing Figure 1(b) and Figure 1(c), we see that the vanilla PPO algorithm performs better than their proposed algorithm! If PPO with $\pi_{old}=\pi_{behavior}$ is much better than the proposed algorithm, why does the author want us to use their method?

2. By comparing Figure 2(a) and Figure 2(d),  we see that the most critical factor is the Adam step size instead of the proposed EWMA method. I have no idea the meaning of the proposed EWMA method, and we can obtain nearly nothing when using the EWMA method. And the way of how to adjust the learning rate of Adam is not new. So, what is the contribution of this paper?

3. In Figure 3,  we see EWMA has minimal improvement on current deep RL methods. I doubt the value of this work.


summary_of_the_review: This paper wants to propose a new deep RL method, which makes use of their decoupled objectives. However, I can see slight improvement over their methods. And the proposed EWMA algorithm contributes little to the batch size invariance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes decoupling the behavior and proximal policies used in policy optimization algorithms such as PPO and PPG. Typically, the behavior policy itself is used as the proximal policy, i.e. the policy to which updates need to be close to (maintain trust region). In such a case, using stale or old data can lead to bad performance, leading to on-policy methods using data from the only most recent sampling iteration. The paper then talks about the batch invariance property, which refers to the performance remaining as is when changing the batch size. This is typically achieved by adjusting the optimization parameters such as learing rate (ex. if batch size is doubled, learning rate should be doubled as well). Finally, the authors perform experiments to test how the decoupling affects the usage of stale data, how batch invariance can be achieved with the decoupled objective, and how much better does the decoupling perform as compared to when the same policy is used as the behvior and proximal policies. 
main_review: Firstly, the paper is clearly written, is simple to read, and flows well. I particularly like that the authors do not oversell any of the results. In terms of motivation, batch size invariance is an important and interesting property and insights specific to RL could be useful for future research. 

On the other hand, I do not understand the main takeaway from the paper. There are two clear contributions here, one of the decoupling objective and the other of achieving batch size invariance. Currently, I am unable to place either of these as the main objective/problem of the paper. If it's the decoupling, then it should lead to better batch size invariance than the standard, coupled policy agent. I think this is plotted in Figure 2 (d), where the coupled PPG agent is used with all optimization adjustments but achieves similar batch invariance as the decoupled PPG agent as in Figure 2. So it seems like batch invariance can be achieved irrespective of the algorithm, i.e. if it's coupled or decoupled. Furthermore, as it is, the decopling only leads to a marginal improvement over the coupled agent. Similarly, if batch size invariance is the main objective, then I don't see how the decoupling idea is worth looking at. Any comments from the authors on this would be really helpful.

Minor Comments/Questions:

1. Typo at line 4 of section 3.1.
2. Last line of section 3.1 "Once the step size is small enough...": Does this small enough batch size have anything to do with the critical batch size? I assume not right? 
3. Last line of section 6.1 "We speculate that updates should improve...": Are you trying to say that more the policy differs a lot from the behavior policy, the more performance improvement we'll see? I do not completely understand why the conclusion here is different from that of the PPO paper.
4. Second paragraph of section 6.2: The first sentence talks about large batch sizes but the next line mentions advice regarding small batch sizes. They seem contradictory to me. Can you clarify the wording here?
5. The decoupling between behavior and proximal policies is also used in the recently introduced MDPO [1] algorithm (Section 5.1). Another useful reference could be [2]. Particularly, if one is familiar with the NPG or Mirror Descent connection of PPO/PPG, it should be fairly straightforward to see that coupling the behavior and proximal policies is not a hard requirement (something that maybe is not as clear from the PPO paper). 

References:

[1] Mirror Descent Policy Optimization.

[2] A functional mirror ascent view of policy gradient methods with function approximation.
summary_of_the_review: The paper does a good job in explaining the proposed method, in running appropriate experiments, and in terms of writing. However, it fails to motivate a single objective or problem which is addressed by the proposed method. I do not see what the main contribution is and so am currently recommending a weak reject. I am happy to revise the review if the authors can address these concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the method to achieve the batch size-invariant for policy gradient algorithms (PPO, PPG). The paper achieves this by decoupling the proximal policy from the behavior policy. The experiments demonstrate the effectiveness of the method.

main_review: 
Strengths:

This paper study an interesting problem of batch size-invariant.
The writing is clear, and the method is easy to understand.
The experimental results are sufficient to demonstrate the effectiveness of the proposed method. 



Weaknesses:
- The paper does not provide any formal theoretical analysis of the batch size-invariant property. Instead, the description is very vague (e.g., Sec 3.1 Sec 4)."Hence the two update rules behave very similarly." To what extent do they behave similarly? Can you give a formal analysis?
- Using moving average policy (instead of behavior policy) as proxy policy $\pi_{proxy}$ is interesting. However, this method seems not to be theoretical-justified. What is the motivation of averaging over parameter space? Is averaging over parameter space equivalent to averaging over policy space? Moreover, does this method still enjoy the monotonical improvement property (as in [1])? 

[1] Liu B, Cai Q, Yang Z, et al. Neural proximal/trust region policy optimization attains globally optimal policy[J]. Advances in Neural Information Processing Systems, 2019, 32.

summary_of_the_review: This paper presented some interesting ideas of batch size-invariance of policy gradient. 
Although the experimental results well demonstrated the effectiveness of their method. 
The method lacks theoretical justification, and the claim of the batch size-invariant property is vague and weak.
review_title: null

Meta-review: This paper studies the method to achieve the batch size-invariant for policy gradient algorithms (PPO, PPG). The paper achieves this by decoupling the proximal policy from the behavior policy. Empirical results show that the methods are somewhat effective at providing batch size invariance.

After reading the authors' feedback, the reviewer discussed the paper and they did not reach a consensus. On the one hand, the rebuttal made some reviewers change their minds who appreciated the explanations provided by the authors and the new Figure that better highlights the batch size invariance property.
On the other hand, some reviewers think that there is still significant work to be done to get this paper ready for publication. In particular, it is necessary to improve the theoretical analysis and the evaluation of the empirical results.

I encourage the authors to follow the reviewers' suggestions while they will update their paper for a new submission.
Decision: Reject