Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper carries out a large-scale study for understanding of on-policy deep actor-critic. The study looks into a large choices of many implementation settings and design decisions, and investigate their impact on the task performance. The evaluations are done with 250000 RL agents on 5 different continuous control tasks. For each evaluation category, there is a finding summary that provides practical recommendations.

In overall, this study is exhaustive and helpful to both RL researchers and practitioners. The experiment organization which separates all design choices into 7 main categories is very excellent in a systematic way. They cover most design choices in recent works of on-policy RL methods. The reports and the interpretation of results are very interesting and easy to read. The main and important findings are summarized concisely and expected to play important hints.


The only performance metric studied in the paper is a score that is proportional to the area under the learning curve. I was wondering if there should be an additional metric, i.e the final policy or an average reward of the final 100 policies? Would the final or best policy be of more interest to the choice of a practitioner?

As many recent work investigates the design choice of only on-policy RL methods, it would be interesting if in introduction there is discussion on why off-policy methods are not considered or should it be addressed in a different way in another research?

Beside the focus on only the performance in terms of rewards, it would be interesting if the discussion can be expanded to look at other matters, e.g. numerical stability of design/hyper-parameter choices, convergence behaviors (it might requires plot to see if a method show premature convergence, fast learning but sub-optimal, fluctuating, etc.). 

Although the paper only uses Mujoco simulator, would the hyperparameters' domains be subjective to it, e.g. inertia, fiction, joint limits, contacts, etc.? It would be helpful if the discussion can show if such those factors play any role in the results? It would lead to more helpful finding summary.


As a final comment, this is a solid work and will be very helpful to the community. Given that it is implemented on the new SEED RL framework, so it would be better if the implementation code can be published.
review_title: Solid work, and interesting results.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: ##########################################################################

Summary:
 
This paper conducted a large-scale empirical study that provides insights and practical recommendations for the training of on-policy deep actor-critic RL agents

##########################################################################

Reasons for score: 
 
Overall, I'd vote for acceptance to the paper. The paper is informative and practical; however, I'm not sure that the paper meets ICLR's requirement.

Pros:
	1. Reproducibility is one of the main issues for various RL algorithms. This paper conducts a large-scale empirical study for popular on-policy algorithms.
	2. The paper is well-written, and the suggestion is useful to me.


Sorry, but I didn't go through all the details in the appendix.

review_title: Recommendation to accept

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors survey a wide variety of implementation-level and hyperparameter decisions in reinforcement learning for continuous control tasks. They train over 250.000 agents with different settings and suggest empirical guidelines.

As the authors indicate, there's not too much related work so one could call this work pioneering: The sort of work conducted by the authors is crucially important for a field afloat with tricks and tweaks, many of which are typically not discussed in the scientific literature due to a misplaced conceit around this being "not research, just engineering" entirely absent from established fields of science such as experimental Physics. It is also typically not done as it is just plain hard to do. Combined with the often lackluster response from the community, the cost-benefits trade-off has just not been worth it, especially for junior researchers. It's all the more commendable that the authors have engaged with this formidable task of bringing some of the "secret sauce" out of the heads of senior engineers in the various labs and into published and peer-reviewed science.

The authors compare various choices of configurations obtained from the Cartesian product of 8 factors which they call thematic groups: Policy Losses (Sec. 3.1), Networks architecture (Sec. 3.2), Normalization and clipping (Sec. 3.3), Advantage Estimation (Sec. 3.4), Training setup (Sec. 3.5), Timesteps handling (Sec. 3.6), Optimizers (Sec. 3.7), and Regularization (Sec. 3.8). The high-variance nature of training RL agents makes it such that the individual factors in these configurations often have surprising non-linear cross-relations such that the problem space cannot be evaluated incrementally (i.e., it's often not possible to establish "the best" architecture first and select the right learning rate afterwards). The authors propose a novel approach of considering for each choice the distribution of values among the top 5% configurations trained in that experiment. Their experimental design is such that the values for each choice are distributed uniformly at random and thus if certain values are over-represented in the top models this indicates that the specific choice is important in guaranteeing good performance.

As for improvements on the paper, I have one major and only a few minor comments. My major comment is that the paper does not indicate anywhere that the research code is released, only that it's based on SEED RL. I believe an authoritative public implementation of the configurations considered would be extremely worthwhile, both for the community and the authors. If they haven't already done so (there's no supplements to this submission and I refrained from doing any web searches to preserve anonymity), I'd urge the authors to invest the time to release a (possibly cleaned-up) version of their code.

As for minor comments, I'm not clear about the philosophical distinction of something being "due to the algorithms or due to their implementations" (in the Introduction). I very much see the point the authors are making, which is an important one -- what makes RL results work is often "nitty-gritty" details not mentioned in the main part of the relevant publications (and often just barely mentioned in appendices). However, in the strictest sense, the algorithm very much is the implementation -- that's what produces a given result. It's worthwhile to keep the distinction between an idea (say, PPO) and a given implementation of that idea (e.g., presumably the authors had to re-implement PPO in TF2 when using SEED RL and couldn't use OpenAI's original implementation). It's also fine to call the idea "the algorithm", but I'd have preferred to see this distinction more clearly defined.

Somewhat related: The authors are very much correct about what they call "standard modus operandi of algorithms [...] such as PPO", namely iterating between generating experience using the current policy and using the experience to improve the policy. I'd add that strictly speaking no _iteration_ is necessary, as for instance IMPALA, coming from the A3C line of development, does both asynchronously in parallel, and I suspect so do the authors given their use of SEED RL. My suggestion would be to slightly rephrase this sentence and mention IMPALA along with PPO. Perhaps there could also be a comment somewhere about what constitutes "PPO" (or "IMPALA") -- e.g., IMPALA consists of (1) an asynchronous actor/learner split [with further choices of when/how the weights are copied from learner to actor, see e.g. [this comment](https://github.com/deepmind/scalable_agent/blob/master/experiment.py#L508)], (2) a specific type of policy-gradient loss, v-trace, (3) a specific neural network architecture, optionally including recurrence via an LSTM and potentially even (4) a specific type of preprocessing for environments such as Atari or DMLab [according to some papers, swapping out the implementation of the frame-downscaling algorithm in Atari has measurable impact on final performance -- this will matter a lot when evaluating re-implementations of an "algorithm"]. I'd like for the authors to take on this opportunity and propose a common language to discuss these distinctions, which in practise are often confusing to junior researchers (and some senior researchers, too).

Further related to IMPALA and v-trace, I was surprised about the word "unsurprisingly" and the explanation in "Perhaps unsurprisingly, PG and V-trace perform worse on all tasks. This is likely caused by their inability to handle data that become off-policy in one iteration, either due to multiple passes over experience [...] or a large experience buffer in relation to the batch size." While the results speak for themselves, my understanding of v-trace was that it was specifically designed for the very goal of dealing with the "slight" off-policiness produced by asynchronous actor/learner splits in a PG setting. Perhaps the authors have an intuition I'm lacking at this point, but if so I'd appreciate further elaboration.

As a final and perhaps trivial comment, I was slightly irritated by the notation/typography for the inverse cumulative density function of a binomial distribution. In $\rm\LaTeX$, the symbols $icdf$ read as the in-context nonsensical $i\cdot c\cdot d\cdot f$ while the authors would presumably want to use $\mathrm{icdf}$ (compare $exp(x)$ vs $\exp(x)$ or $sin(x)$ vs $\sin(x)$). I'd propose `\mathrm{icdf}`  as the correct syntax for this in $\rm\LaTeX$.

In follow-up work, I'd like to see a similar paper for various "discrete RL" tasks (a subset of Atari, VizDoom, DMLab, MiniGrid, BabyAI, ProcGen, and perhaps even Obstacle Tower, Minecraft, StarCraft (I or II) or the recent NetHack environment) with similar factors of configurations. I assume this is a task yet more daunting, but no less useful to the overall community of researchers.

Overall, this is a strong paper and I recommend it for publication.
review_title: An excellent survey of a very hard empirical field

final_score: rating: 7: Good paper, accept
review_content: The paper presents an empirical evaluation of many algorithmic choices made in the implementations of on-policy actor-critic algorithms in deep reinforcement learning (RL). The authors group those choices in clusters in which they expect some interactions. For each cluster, they test sets of randomly made choices while assuming that choices outside a cluster are set to competitive default values. Based on those experimental results, the authors formulate recommendations about how to make those choices for each cluster.


PROS
The paper is well-written and clear. This paper is part of the string of recent papers that discuss the difficulty of evaluating deep RL algorithms. I appreciate the breadth of the choices that the authors consider. The justification for their overall experimental design (i.e., evaluating per choice clusters) is reasonable. While some findings are as expected, others are indeed unexpected and not discussed in the deep RL literature. 


CONS
I have a doubt about the robustness of the results. The authors decided to use the median over 3 seeds for the evaluation. Although the median is used, is it reliable given the observations made by Henderson et al., which implies that performance can vary a lot with respect to seeds? Could the authors comment on that point?

I think one important missing experiment is the evaluation of the combinations of all the recommendations made in the paper. Do the recommendations depend on the default setting for other choices, do they have a synergetic effect or could there be some negative interactions?

review_title: Empirical evaluation of many algorithmic choices in on-policy DRL with actionable takeaways

Meta-review: There is a clear consensus over all reviewers that this is a very strong empirical analysis, with actionable insights that should prove quite useful both to researchers and practitioners. I have no doubt that many will use it as a reference when implementing and using RL algorithms (especially since the authors said they would release their code).

This is thus a clear accept, that in my opinion would deserve an oral presentation, so as to better disseminate its key findings.
Decision: Accept (Oral)