Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper investigates the impact of implementation "details", with existing implementations of TRPO and PPO as examples. The main takeaway is that the performance gains observed in PPO (compared to TRPO) are actually caused by differences in implementation, and not by the differences between the two learning algorithms. In particular, adding to the TRPO code the same implementation changes as in PPO makes TRPO on par with (and possibly even better than) PPO. The clipping objective of PPO is also found to have no significant impact on its performance. This calls for more careful comparisons between algorithms (by minimizing implementation changes and more in-depth ablation studies) than has typically been done until now in the RL research community.

Although this paper is pretty straightforward and does not bring meaningful algorithmic improvements, I still believe it should be accepted as reproducibility and evaluation are a major issue in RL, and people need to be aware of these kinds of implementation differences that can affect the reported results.

My only important concern is that I could not find a link to the code, which I believe is a must for such a paper focusing on implementation. Could the authors please confirm that they will release their code?

Other small remarks:
	- Fig. 1 is hard to read, I think more synthetic results could have easily conveyed more clearly the intended message
	- When referring to Fig. 2 and 3 please specify "left", "middle" or "right"
	- Fig. 2's caption should describe the plots in left to right order (also what does "maximum versus mean KL" mean?)
	- Fig. 3's caption lists mean KL twice on its first line
	- "The trust region for PPO-NoClip bounds KL to a lesser degree": this is confusing as it sounds like it is "less bounded" while it is actually "more bounded" (as said in Fig. 3's caption)
	- It would help comparing Fig. 2 and Fig. 3 if they both used the same y axis range
	- Typo: "enforcing" => enforces

Update after author feedback: increasing score to "Accept" thanks to the release of the code
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: 
# Summary
The papers studies the effects of code-level optimization on the performance of TRPO and PPO. Details, usually considered as implementation-level particularities, are shown to be of crucial importance for the algorithms' performance.

# Decision
The paper makes an important point, it is written clearly, and the body of evidence is convincing. Therefore, I recommend this paper for publication.

# Suggestions
Make it more clear what is meant by code-level optimizations.
    - In Sec. 2, there is a link to Appendix A.2 for a "full list", but the list in A.2 does not contain all points from Sec. 2.
    - For PPO-M, it is said "implements only the core of the algorithm". What exactly does that mean?
    - PPO-NoClip is defined as "PPO without clipping". Does it mean that it includes all other tricks apart from clipping? Please, be explicit in such places.

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Summary

This paper calls to attention the importance of specifying all performance altering implementation details that are current inherent in the state-of-the-art deep policy gradient community. Specifically, this paper builds very closely on the work started by  Henderson et al. 2017, building a conversation around the importance of more rigorous and careful scientific study of published algorithms. This paper identifies many "code-level optimizations" that account for the differences between the popular TRPO and PPO deep policy gradient algorithms. The paper then subselects four of these optimizations and carefully investigates their impact on the final performance of each algorithm. The clear conclusion from the paper is that the touted algorithmic improvement of PPO over TRPO has negligible effect on performance, and any previously reported differences are due only to what were considered unimportant implementation details.

Review

This paper investigates the claims made by Schulman et al. 2017 carefully, by investigating the impact of PPO's clipping mechanism on maintaining a valid trust-region; the central claim made by PPO's originating paper. The empirical results suggest that PPO is not sufficient for maintaining a valid trust-region, however the "code-level optimizations" that differ between the TRPO implementation the PPO implementation are sufficient. The ablation study of the four optimizations studied by the paper shows dramatic and clear results suggesting that annealing stepsizes and normalize rewards make very strong differences in learning performance; much more effect than demonstrated by the differences between TRPO and PPO's core algorithmic contribution as demonstrated in Figure 2 and even more strongly in Figure 3. I find the work included in this paper to be novel and a valuable contribution to the field.

For the above reasons, I recommend to accept this paper for publication at ICLR. In the following paragraphs I will discuss why I only recommend a weak accept instead of a strong accept.

My primary concern with the empirical study is the use of only three random seeds. As demonstrated in Henderson et al. 2017 (which is heavily cited in this paper), using such a small number of random seeds can have very misleading results. Although the effects appear very strong in the empirical studies in this paper, the effects likewise appear strong in Henderson et al.'s Figure 6 where 10 random seeds were split into two groups for the same algorithm. For this paper to make such strong claims about the negligence of the careful scientific study on TRPO and PPO, it would be best if this paper included far more random seeds in its investigation.

My second concern is with the discussion and conclusions drawn from Tables 1 and 2. It appears that the inclusion of clipping plays a strong role in the variance of each algorithm on every domain except Hopper. Specifically, the algorithms that include clipping appear to be much lower variance than the algorithms including clipping. Admittedly using only 3 seeds means that investigating the variance appropriately is near impossible (see the above paragraph), however variance should be considered and discussed in a conversation about the effects of the core contribution of PPO. If clipping leads to more consistent results across runs, even if those results are a little worse, it is still a valid and important contribution.

The paper cites Henderson et al. 2017 in several places. I would point out (perhaps in the introduction) that this paper builds on work already done in Henderson et al. 2017. Specifically, Henderson et al. 2017 investigates the effects of using different codebases for TRPO and shows that these different codebases result in dramatically different performance. The similarity to the investigation in this paper to too close to be unreported. However, I find that the investigation in this paper is much more complete and insightful than that of Henderson et al. 2017 (this paper has a more narrow focus), thus contributes significantly and meaningfully to this ongoing conversation.

Additional Comments (do not affect score)

It might be worthwhile to move the related work section to the beginning of the paper, either merged with the introduction or immediately after. This section is of critical importance to understanding the scope of this paper and for understanding why you are studying what you study. In fact, there is already a bit of duplication between the related works and introduction sections, so the paper could likely gain some additional real-estate by combining these.

I disagree with the terminology "code-level optimizations" and I find that it is misleading. This caused a bit of confusion on my first pass reading the paper, as I originally was expected the code differences to be more akin to using Tensorflow vs PyTorch or switching hash table functions, etc. Instead the changes focused on in this paper are changes to the problem specification and algorithm implementation. These are not simply implementation details as "code-level optimizations" suggests, but are rather details that necessarily must be included in peer-reviewed works. I don't have a suggested name to switch to, but felt strongly enough to mention it.
review_title: Official Blind Review #1

Meta-review: This paper provides a careful and well-executed evaluation of the code-level details of two leading policy search algorithms, which are typically considered implementation details and therefore often unstated or brushed aside in papers. These are revealed to have major implications for the performance of both algorithms.

The reviewers are all in agreement that this paper has important reproducibility and evaluation implications for the field, and adds substantially to our body of knowledge on policy gradient algorithms. I therefore recommend it be accepted.

However, a serious limitation is that only 3 random seeds were used to get average performance in the first, key experiment. Experiments are expensive, but that result is not meaningful without more runs, and arguably could be misleading rather than informative. The authors should increase the number of runs as much as possible, at least to 10 but ideally more.
Decision: Accept (Talk)