Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The paper proposes to retrospectively benchmark the sample-efficiency on widely used simulated deep RL benchmarks such as Atari and DMControl across the years. The paper shows some interesting trends with respect to how both the algorithmic improvements as well as the use of increased number of frames have driven the progress in the SoTA scores reported on the Atari HNS benchmark. The paper also shows that there has been exponential progress in the sample-efficiency on both Atari and DMControl with nice log-linear plots. This is an interesting analysis, on the lines of OpenAI's papers on Scaling Laws, but done for Deep RL.

Pros:

Useful review article for the community.
Communicates progress in very visible and scientifically precise manner.
Disentangles progress made in algorithmic efficiency and using more frames (distributed training), and wall clock time reduction.
Cons:

The DMControl results could be presented better.. Specific points below:
DeepMind Control results: (i) In DMControl plots, using SLACv2 and other methods on the same plot is incorrect because SLACv2 performs more gradient updates per batch while other methods do not and it has been known that it is a contributing factor to sample efficiency. The author is encouraged to check an ablation in CURL Appendix E.2 where CURL is shown to be superior to SLAC-v2 on 3/4 envs when using similar update frequencies for 500k, and 2/4 for 100k steps respectively. (ii) Performance numbers used for CURL have been taken from DrQ which do not reflect the accurate numbers. Based on the numbers reported in RAD, for both CURL and RAD, the performance for Walker from CURL seems to be at 403 +/- 24, while DrQ reports 344 +/- 132. Would recommend taking numbers directly from updated versions of CURL/RAD on arXiv which are consistent with each other. Would also recommend adding multiple environments to do the analysis on DMControl as opposed to just one arbitrary hand-picked environment without a clear justification. (iii) As pointed out in DrQ, SLAC uses 100k exploration steps which are not counted in the reported sample-efficiency values. Check Table 1 in DrQ. I believe this is an issue with SLACv2 as well.

The paper should talk a little more detail about how the hardware has changed the game in terms of wall clock time reduction, what kind of hardware, etc. A plot wrt computation flops would be useful to have.

The amount of actionable insight gathered from the paper - in terms of - after reading this paper, what can I do to make my DRL systems more efficient? - is not clear to me. I would be curious if the authors can explain this to me given there's a chance I missed it.

Rating: Weak Reject - I believe the paper has a lot of room for improvement and has to provide some strong good insights out of the analysis, but I am open to updating my views based on author response.
review_title: Review

final_score: rating: 2: Strong rejection
review_content: This paper aims to measure sample efficiency of available RL  methods instead of algorithmic improvement.  It draws a conclusion about continuing improved RL sample efficiency in the past few years.  This conclusion is rather thin to fill a complete paper. 
And the evaluation procedure is rather limited: it directly uses data reported from the original papers without re-producing them at the same environment. These different papers came across a span of at least four years;  although Mujoco is a standard benchmark,  it also has evolved multiple times, basically, there are so many uncontrolled variables in the plot reported by authors. I consider these results not eligible for interpretation. And the algorithms considered here for sample efficiency are not on the same ground, some are off-policy, some are on-policy, off-policy algorithms are naturally far more sample efficient than on-policy, they can't be compared in this way.  Also, authors only report results for score 400 and 2000, which is also inadequate.    In the pixel experiments, it mentions SLAC and dreaming, they are not purely pixel-based RL, rather a latent space is trained at first; so it's so a valid comparison between them and methods like CuRL. In the state experiments, SLAC is not purely state-based, it trains VAE first for a low-dim latent space. 

In all, the material presented in this paper does not fill a complete paper, and the evaluation protocol and conclusions are not valid.  
review_title: Not enough material for a paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper is trying to make extensive and systematic investigation in deep RL papers to measure the recent progress in the broad literatures. The authors look into published state-of-the-art results in Atari, state-based continuous control and pixel-based continuous control. Inferring from papers' published training curves, the authors estimated and found sample efficiency doubling time is 10-18 months for Atari, 5-24 months for continuous control and 4-9 months for pixel-based control tasks.

Pros:
+ The reviewer really likes the extensive efforts put on the paper, which includes deep survey on published DRL literatures, analysis on progress of agents, open discussions on limitations of the analysis in Appendix part D, etc;
+ The authors state super clearly about their methodology in the study in Section 3, where they explain how they measure the sample efficiency, how they select publications and SOTA result for analysis and what's the source of numbers used for analysis;

Cons: 
- The biggest concern the reviewer has, which is also discussed and stated in Appendix D, is that many of the published papers, especially those on Atari, were tuned and focused on the performance *score* at 200M frames (or other frame limits), instead of optimising for data efficiency. In fact, as also be mentioned in the paper, van Hasselt et al. 2019 and Kielak 2020 noted that if you tune  the hyperparms for data efficiency, you may draw a very different conclusion. At the same time, the reviewer acknowledged that reproducing all RL results and tuning them separately for data efficiency would require dramatic amount of time and compute resources;
- Related to the above point, when taking the training curve as reference to plot the data efficiency progress figures like Figure 1, It may be ok for the 200M frames evaluation agents, but when it comes to high frame regimes which R2D2 or MuZero uses, they were aiming for absolute performance score instead of aiming for data efficiency. IMO, it's hard to draw conclusions on data efficiency with those agents which consume billions of frames and they didn't consider data efficiency;
-  The continuous control tasks might be a better place to conduct study on data efficiency where more data efficiency related publications go into and those literatures and applications care much more about data efficiency than the ones on Atari. The reviewer found that the survey put more effort and content on the Atari results instead of the control tasks could be improved;
- The reviewer would love to see more discussions like the Appendix part C on how different agents improved data efficiency. It might be worth expanding these sections and put those into the main paper instead of the Appendix. Answering how DRL has progressed on data efficiency is an essential addition to measuring how much it has progressed;
 


Minor:
- For Figure 1 (a) some agent was labelled at a wrong published year, e.g. FRODO was published in 2020 instead of 2018.
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##################################

Summary: 
This paper conducts a meta-analysis of the trend in sample efficiency in deep RL. The authors argue that this is an informative measure of the progress in the field, in addition to the usual metrics of reward for given tasks, as it is an important consideration when applying deep RL to real world problems. They measure sample efficiency by the number of samples from the environment that is needed to achieve some reward. The amount of time to double sample efficiency is computed for Atari and continuous control environments with state or pixel input.

##################################

Pros: 
1. In deep RL, usually evaluation is done by reporting the reward achieved for some set of environments S after some number of steps N during training. Since S and N vary between papers, it is difficult to measure progress in this field. By using sample efficiency as a metric, this paper is able to give a retrospective of the progress for environments with pixel input, which appears to be missing from the literature.
2. The paper is able to estimate the fraction of progress due to 1) increased sample sizes 2) algorithmic improvements. I found this to be the most interesting conclusion from the paper.
3. Based on the history of the algorithms, the paper provides discussion of the main breakthroughs, which is very helpful.

Cons:
1. On the technical side, the paper does not seem to have much novelty. The analysis approach is the same as that of Hernandez & Brown (2020), with minor changes to adapt to the deep RL setting.
2. There are quite a few limitations to the analysis, which the authors discuss in Section 5 and Appendix D. My main concern is about the hyper-parameters. Deep RL is notoriously sensitive, and the way that researchers select hyper-parameters can have great effect on the final results. Therefore, I am concerned that differences in the amount of effort used to tune hyper-parameters may either mask improvements in sample efficiency or lead to spurious conclusions that sample efficiency has improved. Could the authors discuss how this issue could affect the conclusions in the paper?

##################################

Overall: 
I am not sure that this work in its current form is the right fit for ICLR. It does not have much technical novelty, and its reliance on reported training curves, in my opinion, decreases the credibility of the results. In addition, since it is a meta-analysis, I would suggest that a venue like the "ML Retrospectives, Surveys & meta-Analyses (ML- RSA)" workshop at NeurIPS would be a better fit.

##################################

Further comments and questions:
1. I think a more in-depth discussion of the factors leading to progress in sample efficiency, currently in Appendix C, would be very helpful to researchers thinking about new ideas. Therefore, I suggest that it be moved to the main paper. 
2. In Section 4.2, how large is the variation in the amount of frames used in the unrestricted benchmark, and do most of those algorithms use >>200M frames? I think that a lot of variation in the amount of frames would imply a less accurate fit of the trend.

##################################

Update after reading other reviews and author responses:

Thanks to the authors for answering my questions. However, my main concerns were not addressed, so I will keep my score. 
review_title: A meta-analysis of sample efficiency in deep RL

Meta-review: Although all reviewers agree that this is an interesting analysis of sample efficiency in Deep RL over the past few years, there is also a consensus that it is not enough material for an ICLR paper. I also share this sentiment, which motivates the "Reject" decision. This work could have been made stronger by reproducing previous results (even partially) and sharing the corresponding code, so as to provide a fair and controlled comparison of algorithms, and setting the stage for future progress in this area. In its form, it is better suited for a presentation at a workshop than for the main conference.
Decision: Reject