Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The comparisons between RL algorithms are always based on the mean/median obtained from several runs for each task. Because of the limited number of experiments, the comparison can be overwhelmed by the uncertainty. This paper proposes using new metrics such as IQM to replace mean/median; furthermore, it also argues that reporting performance distributions would be a better choice. 

main_review: Strength:
- This paper has an extensive number of experiments, which is very valuable. 
- I appreciate the open-source code, such as colab, that is easy to implement. 
- The problem is well-motivated, it is important to find a more robust way to make comparisons when the computation resources are limited. 

Weakness:
The paper proposes using robust aggregate statistics (e.g. IQM) with interval estimates and performance distributions as measures to make comparisons among RL tasks. I have three concerns/questions regarding this (I'll mainly consider the case N=5, because usual medium and small labs don't have resources beyond that)
- From Fig2 and Fig7, we can see that the interval estimates and performance distributions between different algorithms may have significant overlaps, e.g., DrQ and SPR in Fig2 right panel for 3/5 runs, then can we claim a winner between these two algorithms? Or should we eventually have to compare the single aggregate statistics in this case?
- The authors argue that IQM is a more robust statistics compared with its surrogates such as median, first I don't think they're more robust to outliers than median since median uses less data points than IQM does, so the chance of median getting affected by outliers is less than IQM. It looks obvious that IQM will have a much smaller variance because it averages across more data points, however mean can have even smaller variance. So I think the point here is really how "strange" the normalized score distribution can be: if it's like a Gaussian distribution, mean will be the best choice; if the other extreme situation, median may be more appropriate. If in the middle, IQM may play its role. But we never know what the true distribution looks like (depending on the algorithm and environment), so there is no guideline for this. 
- Generalization: as pointed above, the choices and conclusions are made on Atari games mainly because the authors have $100$ runs of each algorithm in every environment (which we can consider them as the ground truth). However, I don't know how much it can be generalized to other task sets or another set of algorithms. 

Minor Remarks:
- I would suggest to clarify the tasks for each plot. If all plots are using all the tasks, should the heteroskedasticity be considered when doing the test?
- l.77: It's not independent of the rest of the tasks because the order matters. 
- Fig2: Is there any explanation why SPR with a larger $N$ has a larger variance than DER with a smaller $N$?
- Fig2: Why is Median and IQM different for $N=3$? I think both of them would use the middle value in this case. 
- Fig6: I'm confused by the description here, I thought each task has 100 runs and using bootstrap with replacement?
- Fig6: It is interesting that Median seems to do a better job than IQM when the cases are small (N<5)
- l.163: It would be helpful to provide details on how this values are calculated. For example, what is the power that requires 50-100 runs?
- eq(1): It should be $\hat{F}_i(\tau)$ for summation


limitations_and_societal_impact: yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper analyzes a large set of existing deep RL methods, proposes a variety of protocols towards more rigorous evaluation even in the presence of a small count ("handful") of runs, and, through an extensive set of analyses, demonstrates a variety of erroneous conclusions drawn from previous evaluation in prior work. The main protocols proposed are (1) to use confidence intervals via bootstrap estimation, (2) to use so-called "performance profiles" / "run-score distributions" for any given randomized score (scalar performance metric), (3) to use additional metrics, mainly the Interquartile Mean (IQM), but also "optimality gap", "probability of improvement", "difficulty progress", and "probability of being superhuman".
main_review: Main comments
--
This paper is another step in the growing area of research concerned with establishing a more rigorous evaluation of deep RL algorithms. The paper's main strengths are the originality of the proposed protocols (particularly the performance profile), as well as an extensive performance analysis of a large set of deep RL methods. The paper is of high-quality -- the experiments are illuminating because they are very relevant and well-done, and the writing is overall quite clear. The paper could prove quite significant if its proposals are adopted.

One of the paper's weaknesses is that its conclusions are interspersed throughout the paper and appendix, which is more difficult to digest than a clearer "executive summary" of proposed protocols and new findings. While I think it's likely that most readers may agree with the spirit of more rigorous analysis, I think it's likely that many readers that do not closely read the paper may miss the important recommendations and findings. The abstract mentions "we present a more rigorous evaluation methodology", but the reality of the paper is that the methodological recommendations are scattered throughout, as opposed to consolidated into a single compact proposed methodology. Relatedly, the abstract claims that the paper "reveal[s] discrepancies in prior comparisons", but these are not really consolidated or summarized in one place. These issues could be addressed by either making enumerated lists or tables of both (1) the proposed protocols / methodology (e.g. use CIs, use performance profiles, use IQM), and (2) the new findings that shed more light on previous findings (those described in 5.1, as well as those in the appendix).

Another weakness is the underdeveloped discussion of how this paper relates to prior work (L318 [11,15,16,32,37]). For example, [15,16,32,37] also discuss CIs for deep RL.

Other comments
--
The bars for CURL in Fig2. is very faint and hard to see -- the bars of the other methods are much easier to see.

L96 The collection of scores, $y_{1:K}$, is undefined in terms of $x$.

L165-169 The purpose of this experiment is not clearly explained at the beginning of the paragraph. The paper should describe why this experiment is run; from my understanding, it might be something like "Consider a setting in which an algorithm is known to be better -- what is the reliability of median and IQM scores for accurately assessing performance differences as the true performance difference and number of runs varies?"

Fig 6: The definition of "true coverage %" appears to only exist in the caption?  

Looking at [19], there are a few difference between the score distributions proposed herein and the originally proposed performance profiles. These differences should be explicitly named and discussed. Two important differences are (1) performance profiles were defined without any bootstrapping to estimated CIs (2) performance profiles are empirical CDFs, whereas score distributions are their complements (tail distributions).

The argument that median is not a good indicator because it is calculated using "at most two tasks" (L256) is dubious. The median is an aggregate metric that is calculated using 100% of the runs, not 1 or 2.

The "IQM" acronym appears long before it is actually defined on L249.

L254 Is the following formal claim? "IQM has considerably less bias than median". If so, then this claim requires evidence in the form of a derivation or citation, particularly because the audience for this paper is likely not intimately familiar with the IQM.

The ability to read the medians from the performance profiles would be better highlighted if the Fig 7 y=0.5 lines were thicker.

L297 "While publications make binary claims" -> "While publications sometimes make binary claims". The former statement is too broad to be true.

L267 It's not clear how P(X_m > Y_m) is estimated. More details are needed. After looking through the appendix, it appears to be on page 20. This definition should be referenced in the main paper near L267.

Appendix P20, it's not clear if "Alternative alternative Metrics" is a typo or intentional

L323 "... the problem is not solved by fixing random seeds, as has sometimes been proposed" -- consider adding citations here for epistemological completeness.

limitations_and_societal_impact: There was no centralized discussion of limitations. This needs to be included, although I'm not sure what the limitations might be. Perhaps one of the major limitations in the proposed protocol is the unclear incentives future deep RL papers have in actually employing the proposed protocol. I speculate that a lack of rigor in some prior work is partly due to a missing / weak incentive for papers to be more rigorous, as it generally entails more nuanced, tempered claims. I think that it might make sense to explicitly mention this phenomenon as a barrier to the adoption of this work.

The societal impacts aren't really addressed: section A.5 actually discussed 'research community impacts', which is a small subset of 'societal impacts'. Perhaps one sentence is needed to explicitly state that the paper poses no foreseeable strongly negative impacts (if the authors agree), and another sentence that discusses how the paper could positively impact society by constituting a step forwards in rigorous small run count-based evaluation, which reduces computational burden (and is therefore "greener" than using large run counts).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents an empirical analysis of the failures of evaluation protocols in deep RL research and proposes reporting interval estimates, the use of performance profiles, and the use of more robust aggregate metrics. The empirical analysis focuses on the Atari 100k benchmark (but also looks at Atari 200M, procgen, and the deepmind control suite) and finds that previously reported median scores are not very robust by using more compute to estimate the distribution of normalized scores. The ordering of various algorithms is often different depending on the precise evaluation protocol. The paper proposes bootstrapped interval estimates and performance profiles as better ways to report results that show the variability across runs, and proposes the interquartile mean, optimality gap and probability of improvement as alternatives to the median as aggregate metrics. 

main_review: ### Strengths:

1. The paper raises an important problem in deep RL research. While issues of reproducibility have been raised before, this paper focuses specifically on the robustness of evaluation procedures and dives deeper into the issue. The findings of inconsistencies in the literature on this specific part of the pipeline seems like an important problem for the community to consider. 
2. The large empirical evaluation provides solid evidence for the paper's claims. This evaluation shows things like the ordering changing across different protocols, the high variability of prior metrics, and the reduced variability in the proposed metrics. Relative to prior work on reproducibility and evaluation protocols, this paper provides a substantially larger evaluation across many seeds and many benchmark suites in different settings with both finite and continuous action spaces that considers many highly performing algorithms in each benchmark. This large-scale evaluation can be useful for the community beyond this paper since the authors plan to release the data from the training runs. 
3. The paper's proposed solutions seem to help reduce the problems raised with prior work. The main idea to report intervals or performance profiles instead of point estimates is a clearly better practice. Then the IQM does indeed seem to reduce variability. 
4. The paper provides useful tools for the RL community. The paper plans to open source code in a colab notebook to compute all of the proposed confidence intervals, performance profiles, and metrics. It also plans to release all of the data from the large-scale experiments. 

---

### Weaknesses:

1. The organization and writing of the paper sometimes lacks focus making it difficult to follow. I understand that evaluation is a complicated issue and that the paper wants to make sure it covers a lot of different aspects and cases which makes it difficult to organize, but the paper could benefit substantially from a more coherent framing. As it is currently written, the paper reads as a laundry list of issues followed by a laundry list of proposals that each solve some parts of the issues . Something like a table that explicitly shows the issues with each method under consideration could help to provide some high level clarity for the reader of the relationship between the shortcomings of prior evaluation procedures versus the new proposals. It would also be useful to answer questions like: what are the tradeoffs between different methods? Can these be presented in a concise visualization/table or theorem/lemma? 
2. The paper lacks a more formal argument for the proposed metrics. Adding some theory about when and how the proposed metrics are more robust could help clarify the contribution. Even some illustrative examples (perhaps in the appendix) could help to show what sorts of failure modes the new methods avoid. 

---

### Recommendation:

Accept

This paper raises an important problem for deep RL and presents a large empirical study along with some new proposals like reporting bootstrapped confidence intervals. While the organization and theory in the paper could be improved, I think the contribution is good enough to warrant acceptance. 

---

### Typos/minor comments/questions:

1. Why does the paper not compute all of the proposed metrics for each task? For example, there is no performance profile or IQM for the control suite, the probability of improvement seems to only be computed on procgen, and . If all of these metrics are included in the paper, then these results should be reported in the appendix. Alternatively, maybe things would be cleaner if one or two of the metrics were removed from the paper. 
2. The clause "we assert reporting interval estimates of aggregate performance" on line 15 does not make sense. Perhaps the authors meant something like "advocate for" instead of "assert" here?
3. This is sort of a nitpick, but I don't think the title is very well-aligned with the content of the paper. It would be useful to at least say something about "evaluation" in the title so that the reader gets an idea of what the paper is about.
4. The proposed metrics often seems to order the metrics differently, eg. IQM vs. optimality gap in Figure 9. Of course this is to be expected since they are measuring different things, but more of a discussion about how to deal with this inconsistency could be beneficial. 
limitations_and_societal_impact: The paper discusses societal impact in the appendix. The discussion acknowledges the complexity of reproducibility, but could be a bit more forthcoming about how the proposed methods can help illustrate issues of variability in results, but do not solve this fundamental issue.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper discusses the evaluation protocol dominant in deep reinforcment learning (evaluating 3-5 runs across a range of games/environments and reporting point estimates of performance such as mean or median), and describes several severe limitations of that protocol. The protocol ignores the uncertainty present in evaluating performance, especially with so few runs. They demonstrate the various failure modes of this protocol with a case study on the Atari@100k benchmark, showing that the reported results of previous methods are misleading and don't capture the variability possible in evaluation. The paper then proposes several methods for more robust evaluation measures which have tighter confidence intervals, including using inter-quartile mean (IQM) instead of mean or median, calculating interval estimates with bootstrap confidence intervals with stratified sampling, and introducing run-score distributions and performance profiles. The authors then demonstrate the use of these tools on Atari@200M, OpenAI Procgen and DeepMind Control from pixels, showing that previous evaluations are sometimes misleading or confusing, and don't properly capture the full nature of the differences between methods.
main_review: ## Overview

I believe this paper is an important contribution to the field of RL. It points out a failure mode common in reinforcment learning evaluation and comparison. While similar failures of evaluation have been reported before, it seems (as demonstrated by the case studies in the paper) that these failures haven't been adaquately addressed. Further, the degree of detail and rigour taken in presenting the experimental results (and visualising those results) is a credit to the authors and means the work has an excellent clarity in it's presentation. There are a few small clarifications or adjustments to the paper which I believe would be beneficial.

## Strengths

Originality: As noted in the paper (e.g. L318) there has been previous work looking at the evaluation protocol of deep reinforcement learning, but this paper evaluates these problems more thoroughly, finds problems in previous recommendations of increasing the number of random seeds, and proposes new methods for tackling these issues in deep reinforcement learning evaluation. As noted in the Weaknesses section below, I do think a better comparison of the results, recommendations and methods in this work to previous work would be beneficial to clarify the novelty of the contribution.

Clarity: The paper is very well-written, and explains it's points clearly in all parts. The visualisations are well explained and provide good intuition for the variety of statistical ideas presented in the text of the paper, in a way that is rigorous. The paper's structure is well-organised the motivates the main messages of the paper well.

Quality: The technical quality of the experiments and visualisations is very high, and the techniques used and choices made are well-motivated. The claims made are well supported by the experiments.

Significance: The results presented in the paper of very high significance to the reinforcement learning community. The demonstration of spurious or misleading reporting of previous results is very important and should inform the field as to which methods are actually improvements on previous methods, and in what way. The improved approaches for reporting results, including representing the full distribution of results in multiple ways, should be taken up by current researchers in RL to improve the robustness of their results and determining whether real progress is being made.

## Weaknesses

It would be useful to have a (small) related work section (possibly in the appendix, given the space constraints) on previous work and what it demonstrated, to clarify the contributions in this paper. For example, the papers listed in L318 could be explained in more detail.

### Suggestions for improvements

- L186: I think it would be more fair to say that the increase in the number of runs which would be required to address the statistical uncertainty issues is infeasible for computationally demanding deep RL benchmarks.
- L233: I think the "(1)" should go after "... is also" for this list to read correctly.
- From my understanding of dopamine, using the Rainbow implementation from dopamine isn't the full Rainbow algorithm (it misses out on three components - dueling DQN, double DQN and NoisyNets) - this should be made clear in the paper, as it changes the results on the Atari@200M benchmark. For example, see the Rainbow results in table 2 of [Muesli: Combining Improvements in Policy Optimization](https://arxiv.org/abs/2104.06159), which has a much higher median than reported in your paper (and in the DreamerV2 paper, which also uses the dopamine implementation of Rainbow).

## Edit

Upon reading the response and other reviews, I have raised my confidence from 3 to 4, maintaining my score of 9.
limitations_and_societal_impact: I believe limitations and societal impacts are adaquately addressed in the paper.
review_title: null

Meta-review: I thank the authors for their submission and active participation in the discussions. The reviewers unanimously agree that this is a very strong paper with a convincing motivation [9zHZ] and thorough experiments [9zHZ,GYfs,9Qic]. It raises an important issue in deep RL research that could significantly help future work [KFpZ,9Qic]. Furthermore, reviewers appreciate the release of open source code and tools that will help future researchers [9zHZ,GYfs] with their evaluation protocol. Overall, this paper presents an important step in more rigorous evaluation of deep RL research. It identified shortcomings of prior evaluation protocols, questioning some of the conclusions made in recent years, as well as providing clear guidance on how to improve evaluation in the future. I agree with the reviewers and strongly recommend acceptance. I also encourage the authors to use the feedback by the reviewers to further improve their paper, in particular it's clarity.
Decision: Accept (Oral)