Deep Reinforcement Learning at the Edge of the Statistical Precipice
Authors: Rishabh Agarwal
Max Schwarzer
Pablo Samuel Castro
Aaron Courville
Marc G Bellemare
Venue: NeurIPS 2021 Oral (2021)
Keywords: Reinforcement Learning
Evaluation
Benchmarking
Scientific Progress
Reliability

Abstract: Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable, to prevent unreliable results from stagnating the field. This work received an outstanding paper award at NeurIPS 2021.

Deep Reinforcement Learning at the Edge of the Statistical Precipice
Rishabh Agarwal"
Max Schwarzer
Pablo Samuel Castro Google Research, Brain Team
Google Research, Brain Team MILA, Université de Montréal
MILA, Université de Montréal
Aaron Courville
Marc G. Bellemare Google Research, Brain Team
MILA, Université de Montréal
Abstract
Deep reinforcement learning (RL) algorithms are predominantly evaluated by com- paring their relative performance on a large suite of tasks. Most published results deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally -emmoonndlly- benchmarks has led to the practice of evaluating only small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few-run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using
study the Atari 100k benchmark, where find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval esti- mates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metries, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable", to prevent unreliable results from stagnating the field.
Introduction
Research in artificial intelligence, and particularly deep reinforcement learning (RL), relies on evaluating aggregate performance on diverse suite of tasks to assess progress. Quantitative evaluation on suite of tasks, such Atari games [5], reveals strengths and limitations of methods while simultaneously guiding researchers towards methods with promising results. Performance of RL algorithms is usually summarized with point estimate of task performance measure, such as mean and median performance across tasks, aggregated over independent training runs.
A small number of training runs (Figure 1) coupled with high variability in performance of deep RL algorithms [16. 17, 41, 68, 70], often leads substantial statistical uncertainty in reported point
Outstanding Paper Award. Correspondence to Rishabh ccrishabhggrraa@@gggle..oom www.stttbooooooeeeeeeoooooootttt
/tttps://github.. com wewpggogeeeeeeerrr/rraiaaaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

estimates. While evaluating more runs per task has been prescribed to reduce uncertainty and obtain reliable estimates [20, 41, 49], 3-10 runs are prevalent in deep RL as it is often computationally prohibitive to evaluate more runs. For example, 5 runs each on 50+ Atari 2600 games in ALE using standard protocol requires more than 1000 GPU training days [15]. As we move towards more challenging and complex RL benchmarks (e.g., StarCraft [110]), evaluating more than handful of runs will become increasingly demanding due to increased amount of compute and data needed to tackle such tasks. Additional confounding factors, such as exploration in the low-data regime, exacerbates the performance variability in deep RL as seen on the Atari 100k benchmark [50] - often requiring many more runs to achieve negligible statistical uncertainty in reported estimates.
Ignoring the statistical uncertainty in deep RL results gives a false impression of fast scien- tific progress in the field. inevitably evades the question: "Would similar findings be ob-
-T--tammng * Cooking
Number of Runs
100
+Opitions
Frerts
ALLE
30-
Poon Alptaca -a
PPD SAc Rantion Sogt
Opgnill Five
Balloons
10-
paw
tained with new independent runs under differ- ent random conditions?" This could steer re- searchers towards superficially beneficial meth- ods [11, 12, 25], often at the expense of bet- ter methods being neglected or even rejected early [67, 74] as such methods fail to outper- form inferior methods simply due to less fa- vorable random conditions. Furthermore, only reporting point estimates obscures nuances in comparisons [85] and can erroneously lead the field to conclude which methods are state-of- the-art [63, 84), ensuing wasted effort when applied in practice [ [108]. Moreover, not report- ing the uncertainty in deep RL results makes them difficult to reproduce except under the exact same random conditions, which could lead to a reproducibility crisis similar to the one that plagues other fields [4, 44, 78]. Finally, unreliable results could erode trust deep RL research itself [45].
3-
Batch-
Muzero
14
2005
2015
2017 Year
2018
2019
Figure 1: Number of runs in RL over the years. Be- ginning with DQN [75] on the ALE, or less runs are common in the field. Here, we show representative RL papers with empirical results, in the order of their publi- cation year: TD-leaming [99], Sparse coding [100], Op- tions [102], Tetris (CEM) [103], Batch-Q [31], ALE (5],
DQN [75]. AlphaGo [96]. A3C [76], DDPG [62]. ES [88].
PPO [92]. SAC [36], Rainbow [42]. AlphaStar [110], Go-
Explore [28], OpenAl Five [8], Balloon navigation [7]
and MuZero [91].
In this work, we show that recent deep RL papers compare unreliable point estimates, which are dominated by statistical uncertainty, as well exploit non-standard evaluation protocols, using case study on Atari 100k (Section 3). Then, we illustrate how to reliably evaluate performance with only a handful of runs using more rigorous evaluation methodology that accounts for uncertainty in results (Section 4). To exemplify the necessity of such methodology, we scrutinize performance evaluations of existing algorithms on widely used benchmarks, including the ALE [5] (Atari 100k, Atari 200M), Procgen [18] and DeepMind Control Suite [104], again revealing discrepancies in prior comparisons (Section 5). Our findings call for change in how we evaluate performance in deep RL, for which we present a better methodology to prevent umreliable results from stagnating the field.
How do we reliably evaluate performance on deep RL benchmarks with only a handful of runs? As practical solution that is easily applicable with 3-10 runs per task, we identify three statistical tools (Table 1) for improving the quality of experimental reporting. Since any performance estimate based on a finite number of runs is a random variable, we argue that it should be treated as such. Specifically, we argue for reporting aggregate performance measures using interval estimates via stratified bootstrap confidence intervals, as opposed to point estimates. Among prevalent aggregate measures, mean can be easily dominated by performance on few outlier tasks, while median has high variability and zero performance on nearly half of the tasks does not change it. To address these deficiencies, we present more efficient and robust alternatives, such as interquartile mean, which are not unduly affected by outliers and have small uncertainty even with a handful of runs. Furthermore, to reveal the variability in performance across tasks, we propose reporting performance distributions across all runs. Compared to prior work [5, 83), these distributions result in performance profiles [26] that are statistically unbiased, more robust to outliers, and require fewer runs for smaller uncertainty.
2 Formalism
We consider the setting in which a reinforcement learning algorithm is evaluated on M tasks. For each of these tasks, we perform N independent runs" which each provide a scalar, normalized score
2 A run can be different from using a fixed random seed. Indeed, fixing the seed may not be able to control all sources of randomness such as -ondettrrmiiimm of ML frameworks with GPUs (e.g.. Figure 13).
2

---

Table 1:
Our recommendations for reliable evaluation, easily applicable with a handful of runs. Refer to Section 4 for details about recommendations and Section 5 for their application to widely-used RL benchmarks.
Desidcratum
Current Evaluation Protocol
Our Recommendation
Point estimates
Uncertainty
Ignore statistical uncertainty
Interval estimates via stratified bootstrap confidence intervals
in aggregate performance
Hinder results reproducibility
Tables with mean scores per task
Performance profiles (score distribution)
Variability in performance
Overwhelming beyond few tasks
Show tail distribution of scores on com- bined runs across tasks
Standard deviations often omitted
across tasks and runs
Incomplete picture for multimodal
Allow qualitative comparisons
and hcavy-tailed distributions
Easily read any score percentile
Mean
Often dominated by performance outlier tasks
Interquartile Mean (IQM) across all runs
Aggregate
Performance on middle 50% of com- bined runs
metrics
Median
sum-
Requires large number of runs to claim improvements
Robust to outlier scores but more statis- tically efficient than median
marizing
performance
Poor indicator of overall perfor-
To show other aspects of performance gains, report average probability of improvement and optimality gap.
across tasks
mance: scores nearly half the tasks not affect
mm. m I,..., M and n = 1,... N. These normalized scores are obtained by linearly rescaling per-task scores* based on two reference points; for example, performance on the Atari games is typically normalized with respect to random agent and an average human, who are assigned normalized score of 0 and respectively [75]. We denote the set of normalized scores by M1:M,
In most experiments, there is inherent randomness in the scores obtained from different runs. This randomness can arise from stochasticity in the task, exploratory choices made during learning, ran- domized initial parameters, but also software and hardware considerations such non-determinnim in GPUs and in machine learning frameworks [116]. Thus, we model the algorithm's 's normalized score on the th task as real-valued random variable X Then, the score Jmn. is realization of the random variable Xm.., which is identically distributed as Xm- For T E RR, we define the tail distribution function of Xm (T) P(X,,. > r). For any collection of scores y1-k, the empirical tail distribution function given by f(T:yi.)) = Dk 1yy > T]. In particular, we write Fm [T; I...::).
The aggregate performance of algorithm maps the of normalized scores FI:M, I:N scalar value. Two prevalent aggregate performance metrics are the mean and median normalized scores. If we denote by F. N En n== Im.n the average score on task m across N runs, then these aggregate metrics are Mean ( 1.M) and Median (..M). More precisely, we call these sample mean and sample median over the task means since they are computed from a finite set of N runs. Since Im is realization of the random variable X,, = N Σn Xm, the sample mean and median scores point estimates of the random variables Mcan(X (X (.M) and Median(X) Coddian(Xi.ba XI:M) respectively. We call true mean and true median the metrics that would be obtained if we had unlimited experimental capacity (N + oo), given by MEan(E]X1..u)) and Median (E) [[[X1:M]) respectively.
Confidence intervals (CIs) for a finite-sample score can be interpreted as an estimate of plausible values for the true score. A a x 100% CI computes an interval such that if we rerun the experiment and construct the CI using a different set of runs, the fraction of calculated CIs (which would differ for each set of runs) that contain the true score would tend towards a x 100%, where E [0, 1] is the nominal coverage rate. 95% Cls are typically used in practice. If the true score lies outside the 95% CI, then a sampling event has occurred which had probability of 5% of happening by chance.
'Often the average undiscounted return obtained during an episode (see Sutton and Barto [101] for an explanation of the reinforcement learning setting).
3

---

Median
IQM
Reported
50 25 10 5
DER
Fraction of runs
3.0%
OTR
CURL
DrQ
DrQ(e)
1.5%
SPR
0.0%
0.2
0.3
0.4
0.5
01 0.2 0.3 0.4 0.2
0.5
0.
Median Human Normalized Score
95% Confidence Interval
Figure 2: Left. Distribution of median normalized scores computed using 100,000 different sets of N runs subsampled uniformly with replacement from 100 runs. For a given algorithm, the sampling distribution shows the variation in the median scores when re-estimated using different set of runs. The reported point estimates of median in publications, median scores and severely overestimate or underestimate the expected median. We use the same number of runs as reported by publications: N 5 runs for DER, OTR and DrQ, N = 10 runs for SPR and N 20 runs for CURL. Right. 95% CIs for median and IQM scores (Section 4.3) for varying N. There is substantial uncertainty median scores with 50 runs. IQM has much smaller Cls than median. Note that when CIs overlap, properly accounting for uncertainty entails computing CIs for score differences (Figure A.15).
shown by dashed lines, do provide any information about the variability in
Remark. Following Amrhein et al. [2], Romer [87], Wasserstein et al. [112), we recommend using confidence intervals for measuring the uncertainty in results and showing effect sizes (e.g., per- formance improvements over baseline) that are compatible with the given data, Furthermore, we emphasize using statistical thinking but avoid statistical significance tests (e.g., p-value < 0.05) because of their dichotomous nature (significant vs. not significant) and common misinterpreta- tions [33, 35, 73] such as 1) lack of statistically significant results does not demonstrate the absence of effect (Figure 2, right), and 2) given enough data, any trivial effect can be statistically significant but may not be practically significant.
3 Case Study: The Atari 100k benchmark
We begin with a case study to illustrate the pitfalls arising from the naive use of point estimates in the few-run regime. Our case study concerns the Atari 100k benchmark [50], offshoot of the ALE for evaluating data-efficiency in deep RL. In this benchmark, algorithms are evaluated on only 100k steps (2-3 hours of game-play for each of its 26 games, versus 200M frames in the ALE benchmark.
Prior reported results on this benchmark have been computed mostly from 3 [39, 55, 59, 72, 89, 95] or 5 runs [50, 51, 53, 54, 64, 66, 86, 107, 115], and more rarely, 10 [65, 93] or 20 runs [56].
Our case study compares the performance of five recent deep RL algorithms, namely: (1) DER [107] and (2) OTR [51], (3) Dro" [53], (4) CURL [56]. and (5) SPR [93]. We chose these methods as representative of influential algorithms within this benchmark. Since good performance on one game can result in unduly high sample means without providing much information about performance on other games, it is common to measure performance on Atari 100k using sample medians. Refer to Appendix A.2 for more details about the experimental setup.
We investigate statistical variations in the few-run regime by evaluating 100 independent runs for each algorithm, where the score for run is the average returns obtained in 100 evaluation episodes taking place after training. Each run corresponds to training one algorithm on each of the 26 games in Atari 100k. This provides us with 26 x 100 scores per algorithm, which we then subsample with replacement to 3-100 runs. The subsampled scores are then used to produce a collection of point estimates whose statistical variability be measured. We begin by using this experimental protocol to highlight statistical concerns regarding median normalized scores.
High variability in reported results. Our first observation is that the sample medians reported in the literature exhibit substantial variability when viewed as random quantities that depend on a
$ 'DrQ codebase uses non-standard evaluation hyperparameters. Instead, Dro(s) corresponds to DrQ with standard e-greedy parameters [14, Table 1] in ALE. See Appendix for more details.
4

---

small number of sample runs (Figure 2, left). This shows that there is a fairly large potential for drawing erroneous conclusions based on point estimates alone. As concrete example, our analysis suggests that DER may in fact be better than OTR, unlike what the reported point estimates suggest. We conclude that in the few-run regime, point estimates are unlikely to provide definitive answers to the question: "Would we draw the same conclusions were we to re-evaluate our algorithm with different set of runs?"
Substantial bias in sample medians. The sample median is a biased estimator of the true median: E[Median(XX) (X1:M) * in general. In the few-run regime, we find that this bias can dominate the comparison between algo- rithms, as evidenced in Fig- ure 3. For example, the score difference between sample medians with 5 and 100 runs for SPR (+0.03 points) is about 36% of its mean improvement over DrQ(e) (+0.08 points). Adding to the issue, the magnitude and sign of this bias strongly depends on the algorithm being evaluated.
Expected Median
DER
OTR
CURL
DrQ14)
SPR
with N rure
0.195
0.130
0.108
0.300
0.390
0.190
0.125
0.285
0.375
0.105
0.360
0.100
0.036
0.270
0.345
55,9/8866
3
Number
Figure 3: Expected sample median of task means. The expected score for N runs is computed by repeatedly subsampling N runs with replacement out of 100 runs for 100,000 times.
Statistical concerns cannot be satisfactorily addressed with few runs. While claiming improve- ments with 3 or fewer runs may naturally raise eyebrows, folk wisdom in experimental RL suggests that 20 or 30 runs are enough. By calculating 95% confidence interval on sample medians for varying number of runs (Figure 2, right), we find that this number closer to 50-100 runs in Atari 100k- far too many to be computationally feasible for most research projects.
Consider setting in which an algorithm is known be better what is the reliability of median and IQM (Section 4.3) for accurately assessing performance differences as the number of runs varies? Specifically, we consider two identical N-run experiments involving SPR, except that we artificially inflate one of the experiments' scores by fixed fraction or lift of +5% (Figure 4). In particular, & = 0 corresponds to running the same experiment twice but with different runs. We find that statistically defensible improvements with median scores is only achieved for 25 runs (f = 25) and 100 runs (f = 10). With é = O, even 100 runs are insufficient, with deviations of 20% possible.
Changes in evaluation protocols invalidates comparisons to prior work. A typical and relatively safe approach for measuring the performance of an RL algorithm is to average the scores received in their final training episodes [69]. However, the field has seen number of alternative protocols used, including reporting the maximum evaluation score achieved during training [1, 3, 75] across multiple runs [32, 47, 82]. A similar protocol is also used by CURL and SUNRISE [59] (Appendix A.4).
Results produced under alternative protocols involving maximum perfrerforeance reported results. On Atari 100k, we find that the two protocols produce substantially different results (Figure 5), of magnitude greater than the actual difference in score. In particular, evaluating DER with CURL'S protocol results in scores far above those reported for CURL. In other words, this gap in evaluation procedures resulted in CURL being assessed as achieving greater true median than DER, where our experiment gives strong support to DER being superior. Similarly, we find that a lot of SUNRISE's improvement over DER can be explained by the change in evaluation protocol (Figure 5). Refer to Appendix A.4 for discussion on pitfalls of such alternative protocols.
generally incomparable with
4 Recommendations and Tools for Reliable Evaluation
Our case study shows that the increase in the number of runs required to address the statistical uncertainty issues is typically infeasible for computationally demanding deep RI. benchmarks. In this section, we identify three tools for improving the quality of experimental reporting in the few-run regime, all aligned with the principle of accounting for statistical uncertainty in results.
4.1 Stratified Bootstrap Confidence Intervals
We first reaffirm the importance of reporting interval estimates to indicate the range within which an algorithm's 's aggregate performance is believed lic. Coneretely, we propose using bootstrap Cls [29]
'Specifically, we use the m/n bootstrap [9] to calculate the interval between "2.5*,,7.55" percentiles of the distribution of sample medians (95% Cls).
5

---

0% Lift [Mpper ()
10% Elpperr
25% dippprr CI)
95% Median Cls
95% IQM Cls
0% Lil Howerr (X)
10% Lift (Lower O)
-
25% Lif (Lowwe CD
Coverage %
95
96
95% Os for Ni Observed Lift
1200%
Median Lift
120%
IOM Life
True
100%
100%
85
85
80%
80%
60%
60%
75
75
40%
40%
0.10
0.10
beskc
20%
20%
Average Gi Width
bc
0%
0%
0.05
0.05
boa
20%
20%
percantlle
-00%
-00%
5
10
25 50 100
5
10
25 50 100
0.00
0.00
Number of Runs (M)
Number of Runs (N)
3.5 10 20 50100 Number Runs
3.5 10 20 50100 Number Runs
Figure 4: Detecting score lifts. Left. 95% CIs for oh- served lift with median scores, and Right. 95% CIs for observed lift with IQM (Section 4.3) when comparing SPR with an algorithm that performs 0% better. IQM requires fewer runs than median for small uncertainty.
Figure 6: Validating 95% Stratified Bootstrap CIs for varying number of runs for median and IQM scores for DER. The true coverage % com- puted by sampling 10,000 sets of K runs without replacement from 200 runs and checking the frac- tion of 95% CIs that contains the true estimate approximation based 200 runs. Note that we evaluate additional 100 runs for DER for an accu- rate point estimate. Percentile CIs has the best coverage while achieving a small width compared other methods. Also. CI widths for IQM are much smaller than that of median. We also note that with 3 runs, bootstrap Cls underestimate the true 95% CIs and might require a larger nominal coverage rate to achieve true 95% coverage.
DER (Fira)
DER ISUNRISE'S protocol)
Reported
Reported
Reported
Median
Mean
0.2
0.3
0.3
0.4
0.5
95% Confidence Interval
Figure 5: Normalized DER scores with non-standard evaluation protocols. Gains from SUNRISE and CURL over DER can mostly be explained by such protocols.
with stratified sampling for aggregate performance, a method that can be applied to small sample sizes and is better justified than reporting sample standard deviations in this context. While prior work has recommended using bootstrap CIs for reporting uncertainty in single task mean scores with N runs [16, 20, 41], this is less useful when N is small (Figure 18) as bootstrapping assumes that re-sampling from the data approximates sampling from the true distribution. We can do better by aggregating samples across tasks, for a total of M N random samples.
To compute the stratified bootstrap CIs, we re-sample runs with replacement independently for each task to construct an empirical bootstrap sample with N runs each for M tasks from which we calculate statistic and repeat this process many times to approximate the sampling distribution of the statistic. We measure the reliability of this technique in Atari 100k for variable N, by comparing the nominal coverage of 95% the "true" coverage from the estimated CIs (Figure 6) for different bootstrap methods (see [30] and Appendix A.5). We find that percentile CIs provide good interval estimates for as few as N 10 runs for both median and IQM scores (Section 4.3).
4.2 Performance Profiles
Most deep RI. benchmarks yield scores that vary widely between tasks and may be heavy-tailed, multimodal, or possess outliers (e.g., Figure 14).). In this regime, both point estimates, such as mean and median scores, and interval estimates of these quantities paint an incomplete picture of an algorithm's 's performance [24, Section 3]. Instead, we recommend the use of performance profiles [26], commonly used in benchmarking optimization software. While performance profiles from Dolan and Moré [26] correspond to empirical cumulative distribution functions without any uncertainty estimates, profiles proposed herein visualize the empirical tail distribution function (Section 2) of a random score (higher curve is better), with pointwise confidence bands based on stratified bootstrap.
By representing the entire set of normalized scores 31:M,1:N visually, performance profiles reveal per- formance variability across tasks much better than interval estimates of aggregate metrics. Although tables containing per-task mean scores and standard deviations can reveal this variability, such tables tend to be overwhelming for more than few tasks.? In addition, performance profiles are robust to outlier runs and insensitive to small changes in performance across all tasks [26].
In this paper, we propose the use of a performance profile we call run-score distributions or sim- ply score distributions (Figure 7, right), particularly well-suited to the few-run regime. A score distribution shows the fraction of runs above certain normalized score and is given by
In addition, standard deviations sometimes omitted from tables due to space constraints.

---

SimPLe
DER
OTR
CURL
DrQtel
SPR
1.00
1.00
Fraction of runs with score
Fraction tasks with score
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Human Normalized Score
Human Normalized Score (T)
Figure 7: Performance profiles on Atari 100k based on score distributions (left), which recommend, and average score distributions (right). Shaded regions show pointwise 95% confidence bands based percentile bootstrap with stratified sampling. The profiles
left are more robust outliers have smaller confidence bands. We use 10 runs to show the robustness of profiles with a few runs. For SimPLe [50]. we use the 5 runs from their reported results. The value where the profiles intersect y = 0.5 shows the median while for non-negative random variable, area under the performance profile corresponds to the mean.
M
M Σ
Fx(r) = ((T:LLMMN)
Σ Fm M
1
1 N
[ > T..
(I)
M
n=1
One advantage of the score distribution that is an unbiased estimator of the underlying distribution F(T) _m=1 Fm (T). Another advantage is that an outlier run with extremely high score can change the output of score distribution for any by most value of MN.
is useful contrast score distributions the context of the ALE [5] as a generalization of the median score. Average-score distributions correspond to the performance profile of a random variable X, (T) = ((r: T1:M), which shows the fraction of tasks on which an algorithm performs better than a certain score. However, such distributions are a biased estimate of the thing they seek to represent. Run-score distributions are more robust than average-score distributions, as they are a step function in 1/MN N versus MM 1/M intervals, and typically has less variance: ox MN m-1 (T) Fm (T)) versus Em=1 Fx,,, (1) Fx.. (T)). Figure illustrates these differences.
everage-score distributions, originally proposed in
x
4.3 Robust and Efficient Aggregate Metries
Performance profiles allow us to compare different methods at glance. one curve strictly above another, the better method is said to stochastically dominate" the other [27, 61]. In RL benchmarks with a large number of tasks, however, stochastic dominance is rarely observed: performance profiles often in- tersect at multiple points. Finer quantitative comparisons must therefore entail aggregate metrics.
1.00
Optimality Gap
PIX =)
0.00 0.0
We can extract number of aggregate metrics from score dis- tributions, including median (mixing runs and tasks) and mean normalized scores (matching our usual definition). As we al- ready argued that these metrics are deficient, we now consider interesting alternatives also derived from score distributions.
0.5
1.0
1.5
2.0
Normalized score (T)
Figure Aggregate metries. For neg-tigative random variable X, IQM corresponds to the red shaded region while optimality gap corresponds to the orange shaded region in the perfor- mance profile of X.
As an alternative to median, we recommend using the in- terquartile mean (IQM). Also called 25% trimmed mean, IQM discards the bottom and top 25% the runs and cal- culates the mean score of the remaining 50% runs [=[NM/2 for N runs each M tasks), IQM interpolates between mean and median across runs, which are 0% and almost 50% trimmed means
A random variable X has stochastic dominance over random variable Y if P(X > 7) 2 P(Y > r) for all and for some P(X 7) P(Y > r).
7

---

Median
IQM
Mean
Optimality Gap
DreamerV2
M-IQN
Rainbow
IQN
REM
C51
DON (Adam)
DQN (Nature)
1.0 1.5 2.0
1.0
1.5
2.0
2.5 5.0
7.5
10.0
0.24
0.32
0.40
Human Normalized Score
Figure 9: Aggregate metrics on Atari 200M with 95% CIs based on 55 games with sticky actions [69]. Higher mean, median and IQM scores and lower optimality gap better. The CIs are estimated using the percentile bootstrap with stratified sampling. IQM typically results in smaller CIs than median scores. Large values of mean scores relative to median and IQM indicate being dominated by few high performing tasks, for example, DreamerV2 V2 and M-IQN obtain normalized scores above 50 on the game JAMES BOND. Optimality gap is less susceptible to outliers compared to mean scores. We compare DQN (Nature) [75], DQN with Adam optimizer, C51 [6], REM [1], Rainbow [42], IQN [22], Munchausen-I00 (M-IQN) [109], and DreamerV2 [38], All results are based on rums per game except for M-IQN and Dreamer which report results with and 11 runs.
respectively. Compared to sample median, IQM is better indicator of overall performance as calculated using 50% of the combined runs while median only depends on the performance ordering across tasks and not on the magnitude except at most 2 tasks. For example, zero scores on nearly half of the tasks does not affect the median while IQM exhibits severe degradation. Compared to mean, IQM robust to outliers, yet has considerably less bias than median (Figure A.17). While median is more robust to outliers than IQM, this robustness comes the expense of statistical efficiency, which is crucial in the few-run regime: IQM results in much smaller CIs (Figure (right) and 6) and is able to detect a given improvement with far fewer runs (Figures 4 and A.1
As robust alternative to mean, we recommend using the optimality gap: the amount by which the algorithm fails to meet a minimum score of 1.0 (orange region in Figure 8). This assumes that score of 1.0 is desirable target beyond which improvements are not very important, for example when the aim is to obtain human-level performance [e.g., 3, 23]. Naturally, the threshold y may be chosen differently, which we discuss further in Appendix A.7.
If one is interested in knowing how robust an improvement from an algorithm X over an algorithm Y is, another possible metric consider the average probability of improvement this metric shows how likely it is for X to outperform Y on a randomly selected task. Specifically, P(X > Y) M - P(Xm > Yn), where P(Xm, Ym) (Equation A.2) is the probability that X is better than Y on task m. Note that, unlike IQM and optimality gap. this metric does not account for the size of improvement. While finding the best aggregate metric is still an open question and is often dependent on underlying normalized score distribution, our proposed alternatives avoid the failure modes of prevalent metrics while being robust and requiring fewer runs reduce uncertainty.
Re-evaluating Evaluation on Deep RL Benchmarks
Arcade Learning Environment. Training RL agents for 200M frames on the ALE [5, 69] is the most widely recognized benchmark in deep RI. We revisit some popular methods which demonstrated progress on this benchmark and reveal discrepancies in their findings as a consequence of ignoring the uncertainty in their results (Figure 9). For example, DreamerV2 [38] exhibits large amount of uncertainty in aggregate scores. While M-IQN [109] claimed better performance than Dopamine Rainbow" [42] in terms of median normalized scores, their interval estimates strikingly overlap. Similarly, while C51 [5] is considered substantially better than DQN [75], the interval estimates as well as performance profiles for DQN (Adam) and C51 overlap significantly.
Figure 9 reveals an interesting limitation of aggregate metrics: depending on the choice of metric, the ordering between algorithms changes (e.g., Median vs. IQM). The inconsistency in ranking across aggregate metrics arises from the fact that such metrics only capture specific aspect of overall performance across tasks and runs. Additionally, the change of algorithm ranking between optimality gap and IQM/median scores reveal that while recent algorithms typically show performance gains relative to humans on average, their performance seems to be worse on games below human
*Dopamine Rainbow differs from that of Hessel et al. [42] by not including double DQN, dueling architecture and noisy networks. Also, results in [42] were reported using single without sticky actions.
8

---

DQN (Nature)
DQN (Adam)
C51
REM
IQN
Rainbow
M-IQN
DreamerV2
1.00
1-0-
1QM Human Normalized Score
Fraction ofnus with score
0.8
2.0
0.7
0.6
1.5
0.5
0.50
0.25
0:50
0:75
1.00
1.25
1.0
0.25
0.5
0.0
0.00
0
50
100
150
200
Human Normalized Score (V)
Number of Frames (in millions)
Figure 10: Atari 200M evaluation. Left. Score distributions using human momaan-ormmalzed scores obtained after training for 200M frames. Right. Sample-effiienncy of agents as a function of number of frames measured via IQM human-normalized scores. Shaded regions show pointwise 95% percentile stratified hootstrap Cls.
1002 steps
100k stops
SO0K stops
CURL-D2RL
Fraction (in 96
SUSE PISAC
SLAC
0.00
SAC+AF
CURL RAD
Frastion airuns
Drcamer
1.001
PISAC
SLAC
0.75
RAD
SAC+AE
0.50
0.4
0.6
0.8
0.8
0.9
0:23
20-
Normalized Scores
Normalized Scores
0.0
0.4
0.5
0.8
Normelized Score
(a) Mean Scores (= 1- Optimality Gap)
(b) Score Distributions. (c) Rank comparisons.
Figure 11: DeepMind Control Suite evaluation results, averaged across tasks, the 100k and 500k benchmark. We compare SAC+AE [114], SLAC [58], Dreamer [37]. CURL [98], RAD [57]. DrQ [53]. PISAC [60]. SUNRISE [59]. and CURL-D2RL [97]. The ordering of the algorithms in the left figure is based on their claimed relative performance all algorithus except Dreamer claimed improvement over at least algorithm placed below them. (a) Interval estimates show 95% stratified bootstrap CIs for methods with individual runs provided by their respective authors 95% studentized Cls CURL, CURL-D2RL, and SUNRISE. Normalized scores are computed by dividing by the maximum score (=1000), (b) Score distributions. (c) The 1t column in the rank distribution plots show the probability that a given method is assigned rank i, averaged across all tasks. The ranks estimated using 200,000 stratified bootstrap re-samples.
performance. Since performance profiles capture the full picture, they would often illustrate why such inconsistencies exist. For example, optimality gap and IQM can be both read as areas in the profile (Figure 8). The performance profile in Figure 10 (left) illustrates the nuances present when comparing different algorithms. For example, IQN seems to be better than Rainbow for T 2 2, but worse for 2. Similarly, the profiles of DreamerV2 and M-IQN for 8 intersect multiple points. To compare sample efficiency of the agents, we also present their IQM scores as a function of number of frames in Figure 10 (right).
DeepMind Control Suite. Recent continuous control papers benchmark performance on 6 tasks in DM Control [104] at 100k and 500k steps. Typically, such papers claim improvement based on higher mean scores per task regardless of the variability those scores. However, we find that when accounting for uncertainty in results, most algorithms do not consistently rank above algorithms they claimed to improve upon (Figure
and 11b). Furthermore, there are huge overlaps 95% CIs of mean normalized scores for most algorithms (Figure a1. These findings suggest that a lot of the reported improvements are spurious, resulting from randomness in the experimental protocol.
Procgen benchmark. Procgen [18] is popular benchmark, consisting of 16 diverse tasks, for evaluating generalization in RL. Recent papers report mean PPO-normalized scores on this benchmark to emphasize the gains relative to PPO [92] as most methods are built on top of it. However, Figure 12 (left) shows that PPO-normalizer scores typically have heavy-tailed distribution making the mean scores highly dependent on performance on a small fraction of tasks. Instead, we recommend using normalization based on the estimated minimum and maximum scores on ProcGen (18) and reporting aggregate metrics based on such scores (Figure A.32). While publications sometimes make binary claims about whether they improve over prior methods, such improvements are inherently probabilistic. To reveal this discrepancy, investigate the following question: "What is the

---

Fraction of with >
1.00
PPO
MixReg
0.75
UCB-DrAC
Algorithm X
Algorithm Y
PLR
IDAAC
PPG UCB-DrAC PPO PPO MixReg MixReg
PPG
IDAAC
0.50
IDAAC
IDAAC
PPG
UCB-DrAC
0.25
PLR
UCB-DrAC
MixReg
PPO
0.00
0.4 0.5
0.8 0.9 1.0
0.5
1.0
Min-Max Normalized Score (t)
1.5
2.0
2.5 3.0
P(X Y)
Figure 12: Proegen evaluation results based on easy mode comparisons [80] with 16 tasks. Left. Score distributions which compare PPO [92], MixReg [111], UCB-DrAC [81]. PLR [48], PPG [19] and IDAAC [80]. Shaded regions indicate 95% percentile stratified bootstrap Cls. Right. Each row shows the probability of improvement, with 95% bootstrap CIs, that the algorithm X on the left outperforms algorithm Y on the right, given that X was claimed to better than Y. For all algorithms, results are based on 10 runs per task.
probability that an algorithm which claimed improvement over prior algorithm performs better than it?" (Figure 12, right). While this probability does not distinguish between two algorithms which uniformly improve on all tasks by 1% and 100%, it does highlight how likely improvement is. For example, there is only 40 50% chance that UCB-Dra DrAC [81] improves upon PLR [48]. We note that a number of improvements reported in the existing literature only 50 70% likely.
6 Discussion
We saw, both in our case study on the Atari 100k benchmark and with our analysis of other widely- RL benchmarks, that statistical issues can have a sizeable influence on reported results, in particular when point estimates are used or evaluation protocols are not kept constant within comparisons. Despite earlier calls for more experimental rigor in deep RL [16, 20, 21, 41, 49, 83] (diseussed in Appendix A.3), analysis shows that the field has not yet found sure footing in this regards.
complex one; where our work is concerned with our confidence about and interpretation of reported results (what Goodman et al. [34] calls results reproducibilitty) others [79] have highlighted that there might be missing information about the experiments themselves (methods reproducibility). We remark that the problem is not solved by fixing random seeds, as has sometimes been proposed [52, 77], since it does not really address the question of whether an algorithm would perform well under similar conditions but with different seeds. Furthermore, fixed seeds might benefit certain algorithms more than others. Nor can the problem be solved by the use of dichotomous statistical significance tests, discussed in Section 2.
In part, this is because the issue of reproducibility
One way minimize the risks associated with statistical effects is to report results more complete fashion, paying close attention to bias and uncertainty within these estimates. To this end, our recommendations are summarized in Table To further support RL researchers in this endeavour, we released an easy-to-use Python library, rliable along with Colab notebook for implementing our recommendations, as well as all the individual runs used in our experiments 10, Again, we emphasize the importance of published papers providing results for all runs to allow for future statistical analyses.
A barrier adoption of evaluation protocols proposed in this work, and more generally, rigorous evaluation, is whether there are clear incentives for researchers to do so, as more rigor generally entails more nuanced and tempered claims. Arguably, doing good and reproducible science is one such incentive. We hope that our findings about erroncous conclusions in published papers would encourage researchers avoid fooling themselves, even if that requires tempered claims. That said, a more pragmatic incentive would be if conferences and reviewers required more rigorous evaluation for publication, e.g., NeurIPS 2021 checklist asks whether error bars are reported. Moving towards reliable evaluation is an ongoing process and we believe that this paper would greatly benefit it.
Given the substantial influence of statistical considerations in experiments involving 40-year old Atari 2600 video games and low-DOF robotic simulations, we argue that it is unlikely that an increase in available computation will resolve the problem for the future generation of RL benchmarks. Instead, just as a well-prepared rock-climber can skirt the edge of the steepest precipices, it seems likely that ongoing progress in reinforcement learning will require greater experimental discipline.
CoCoa:: bit
Individual runs: htt////l-bencheee
10

---

Societal Impacts
This paper calls for statistical sophistication in deep RL research by accounting for statistical uncertainty in reported results. However, statistical sophistication can introduce new forms of statistical abuses and monitoring the literature for such abuses should he an ongoing priority for the research community. Moving towards reliable evaluation and reproducible research is ongoing process and this paper only partly addresses it by providing tools for more reliable evaluation. That said, while accounting for uncertainty in results trustworthy results on which the community can build upon, with increased confidence. In terms of broader societal impact of this work, do not see any foreseeable strongly negative impacts. However, this paper could positively impact society by constituting a step forwards in rigorous few-rum evaluation regime, which reduces computational burden on researchers and is "greener" than evaluating a large number of runs.
a panacea, provides strong foundation for
Acknowledgments
We thank Xavier Bouthillier, Dumitru Erhan, Marlos C. Machado, David Ha, Fabio Viola, Fernando Diaz, Stephanie Chan, Jacob Buckman, Danijar Hafner and anonymous NeurIPS' reviewers for providing valuable feedback for an earlier draft of this work. We also acknowledge Matteo Hessel, David Silver, Tom Schaul, Csaba Szepesvári, Hado van Hasselt, Rosanne Liu, Simon Kornblith, Aviral Kumar, George Tucker, Kevin Murphy, Ankit Anand, Aravind Srinivas, Matthew Botvinick, Clare Lyle, Kimin Lee, Misha Laskin, Ankesh Anand, Joelle Pineau and Braham Synder for helpful discussions. We also thank all the authors who provided individual runs for their corresponding publications. We are also grateful for general support from Google Research teams in Montréal and elsewhere.
References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, 2020.
[2] Valentin Amrhein, Sander Greenland, and Blake McShane. Scientists rise up against statistical significance. Nature, 2019.
[3] Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski. Pablo Sprechmann, Alex Vitvitskyi, Zhao han Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507-517. PMLR, 2020.
[4] Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature News, 2016.
[5] Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[6] Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[7] Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong. Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning Nature, 2020.
[8] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer. Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning, arXiv preprint arXiv: 191::1912,0000 2,66600, 2019.
[9] Peter Bickel, Friedrich Götze, and Willem R van Zwet. Resampling fewer than n observations: gains, losses, and remedies for losses. In Selected works of Willem van Zwer, pages 267-297. Springer, 2012.
[10] Mauro Birattari and Marco Dorigo. How to and report the performance of stochastic algorithm on benchmark problem; mean or result on a number of runs? Optimization letters, 2007.
[11] Xavier Bouthillier, César Laurent, and Pascal Vincent. Unreproducible research is reproducible. In International Conference Machine Learning, pages 725-734, 2019.

---

[12] Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand. Edward Raff. Kanika Madan, Vikram Voleti, al. Accounting for variance in machine learning benchmarks. Proceedings of Machine Learning and Systems, 3. 2021.
[13] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github com/ google/ jax.
[14] Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G Bellemare. Dopamine: research framework for deep reinforcement learning, arXiv preprint arXiv:1812.061 10, 2018.
[15] Johan Samir Obando Ceron and Pablo Samuel Castro. Revisiting rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In International Conference on Machine Learning, 2021.
[16] Stephanie CY Chan, Samuel Fishman, Anoop Korattikara, John Canny, and Sergio Guadarrama. Mea- suring the reliability of reinforcement learning algorithms. In International Conference on Learning Representations, 2020.
[17] Kaleigh Clary, Emma Tosch, John Foley, and David Jensen. Let's play again: Variability of deep reinforcement learning agents in atari environments. arXiv preprint 1944v:190 2019.
[18] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pages 2048-2056. PMLR, 2020.
[19] Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. arXiv preprint PrXii:2009.044111 2020.
[20] Cédric Colas. Olivier Sigaud, and Pierre- Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv: 1806,0829, 2018.
[21]] Cédric Colas, Olivier Sigaud, and Pierre-Y ves Oudeyer. A hitchhiker's guide to statistical comparisons of reinforcement learning algorithms. arXiv preprint arXiv:1904.00 06979, 2019.
[22] Will Dahney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distri- butional reinforcement learning. In International conference on machine learning, pages 1096-1105. PMLR, 2018
[23] Will Dabney, Mark Rowland, Mare G Bellemare, and Rémi Munos. Distributional reinforeement learning with quantile regression. In Thirty-Second AAAI Conference Artificial Intelligence, 2018
[24] Mostafa Dehghani, Yi Tay, Alexey A Gritsenko. Zhe Zhao, Neil Houlsby, Fernando Diaz, Donald Metzler, and Oriol Vinyals. The benchmark lottery. arXiv preprint arXiv:2107.07000) 2021.
[25] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine- tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint 2XXi::002.063300 2020.
[26] Elizabeth D Dolan and Jorge Moré. Benchmarking optimization software with performance profiles. Mathematical programming, (2)22011-2,,, 2002.
[27] Rotem Dror, Segev Shlomov, and Roi Reichart. Deep dominance-how to properly compare deep neural models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[28] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for hard-exploration problems. arXiv preprint arXiv:1901. 2019.
[29] Bradley Efron. Bootstrap methods: another look at the jackknife. The Annals of Statistics. 7:1-26, 1979.
[30] Bradicy Efron. Better bootstrap confidence intervals. Journal of the American statistical Association, 1987.
[31] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-555 -5000-55666000 2005.
12

---

[32] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley. lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning. PMLR, 2018.
[33] Gerd Gigerenzer. Statistical rituals: The replication delusion and how we got there. Advances in Methods and Practices in Psychological Science, 1(2):198-218, 2018.
[34] Steven N Goodman, Daniele Fanelli, and John Ioannidis. What does research reproducibility mean? Science transiational medicine, ((344):341ss12-3411
2016.
[35] Sander Greenland, Stephen J Senn, Kenneth Rothman, John B Carlin, Charles Poole, Steven N Goodman, and Douglas G Altman. Statistical tests, p values, confidence intervals, and power: guide to misinterpretations. European journal of epidemiology, 2016.
[36] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[37] Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations 2019.
[38] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
[39] Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In International Conference on Learning Reprexentations, 2020.
[40] Nathaniel Helwig. Bootstrap Confidence Intervals. 01 2021. URL httphtt///sssssstae umn wwn.edu/ "helriggnotee//ppoott notes notes. html tttt:ooootttrr--cffiiddee deddeeenneeeaaa....
[41] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup. and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence. 2018.
[42] Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, 2018.
[43] Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, and Hado Hasselt. Muesli: Combining improvements in policy optimization. Proceedings of the 38th International Conference on Machine Learning, 2021.
[44] John PA Toannidis. Why most published research findings are false. PLoS medicine, 2(8)e 124, 2005.
[45] Alex Irpan. Deep reinforcement lcarning doesn't 't work yet. 14/r1-hard. html, 2018
[46] Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of benchmarked deep reinforcement learning tasks for continuous control. arXiv preprint arXiv:1708.04133 2017.
[47] Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czamecki, Tom Schaul, Jocl Z Lcibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv: 1. 05397, 2016.
[48] Minqi Jiang, Ed Grefenstette, and Tim Rocktäschel. Prioritized level replay. International Conference on Machine Learning, 2021.
[49] Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforcement learning algorithms. In International Conference on Machine Learning. pages 4962-4973. PMIR, 2020.
[50] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy Campbell. Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint 1rXiv:1903.0003 2019.
[51] Kacper Kielak. Do recent advancements in model- based deep reinforcement learning really improve data efficiency? arXiv preprint arXiv:2003.1013 2020.
13

---

[52] Sergey Kolesnikov and Oleksii Hrinchuk. Catalyst. rl: distributed framework for reproducible rl research. arXiv preprint 1rXiv:1903.00027, 2019.
[53] Πly Kostrikov Denis Yarats", and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021.
[54] Piotr Kozakowski, Lukasz. Kaiser, Henryk Michalewski, Afroz Mohiuddin, and Katarzyna Kańska. Q- value weighted regression: Reinforcement learning with limited data. arXiv preprint arXiv:2102.06788 2021.
[55] Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for percoption and control. NeurlPS, 32:10724-10733 10734, 2019.
[56] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020.
[57] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Picter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data, Advances Neural Information Processing Systems, 2020.
[58] Alex Lce, Anusha Nagabandi, Pieter Abbecl, and Sergey Levine. Stochastic latent actor-critic: Deep reinforeement learning with latent variable model. Advances in Neural Information Processing Systems, 33, 2020.
[59] Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abheel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. International Conference on Machine Learning, 2021.
[60] Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning in rl. Advances in Neural Information Processing Systems, 2020.
[61] Haim Levy. Stochastic dominance and expected utility: Survey and analysis. Management science, 38(4): 555-593, 1992.
[62] Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 150900297. 2015.
[63] Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Significant improvements over the state of the art? a case study the ms marco document ranking leaderboard. arXiv preprint arXiv:2 102. 12887, 2021.
[64] Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Li Jian, Nenghai Yu, and Tie-Yan Liu. Retum-basedd contrastive representation learning for reinforeement learning. In International Conference on Learning Representations, 2021.
[65] Hao Liu and Pieter Abbecl. Behavior from the void: Unsupervised active pre-training. arXiv preprint 10X.v:210300455) 2021.
[66] Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[67] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? large-scale study. arXiv preprint 1rXiv: 2017.
[68] Nicolai A Lynnerup, Laura Nolling, Rasmus Hasle, and John Hallam. A survey on reproducibility by evaluating deep reinforcement learning algorithms on real-world robots. In Conference on Robot Learning. 2020.
[69] Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning enviromment: Evaluation protocols and open problems for general agents, Journal of Artificial Intelligence Research, 2018.
[70] Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides competitive approach reinforcement learning, arXiv preprint arXiv: 1803.07055 2018.
[71] Henry B Mann and Donald R Whitney. On test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50 60, 1947.
14

---

[72] Muhammad Rizki Maulana and Wee Sun Lee. Ensemble and auxiliary tasks for data-efficient deep reinforcement learning, arXiv preprint arXiv:2107.01904, 2021.
[73] Blakelcy B McShane, David Gal, Andrew Gelman, Christian Rohert, and Jennifer I. Tackett. Ahandon statistical significance. The American Statistician, 2019.
[74] Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations, 2018.
[75] Volodymyr Mnih, Koray Kavukeuogiu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
[76] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement lcarning. In International conference on machine learning, pages 1938-1937. PMLR, 2016.
[77] Prabhat Nagarajan, Garrett Wamell, and Peter Stone. Deterministic implementations for reprodueibility in deep reinforcement learning. arXiv preprint arXiv:189.006670 2018.
[78] Harold Pashler and Eric-Jan Wagenmakers. Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on psychological science, 7(6):528-530, 2012.
[79] Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program), arXiv preprint htt:::2000.222000 2020.
[80] Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. International Conference on Muchine Learning, 2021.
[81] Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006. /2862, 2020.
[82] David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matl Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long- pog-tterm credit assignment. arXiv preprint 1iXiv:2122.1222: 2021.
[83] Ben Recht, Benchmarking Machine Learning with Performance Profiles, 03 2018. URL hoLLht/p//ww/ argmin.net/2012. httt./ee///////////rarta///// http:///epprroooceeeeeeeeee profiees/.
[84] Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study lstm-networks for sequence tagging. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338-348, 2017.
[85] Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In International conference on machine learning, 2017.
[86] Jan Robine, Tobias Uelwer, and Stefan Harmeling. Smaller world models for reinforcement learning. arXiv preprint arXiv:0010.05767, 2020.
[87]
David Romer. In praise of confidence intervals. In AEA Papers and Proceedings, volume 110, pages 55 60, 2020.
[88] Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Tlya Sutskever. Fvolution strategies a scalable alternative reinforcement learning. arXiv preprint 1703.v/703..3300 2017.
[89] Rohan Saphal, Balaraman Ravindran, Dheevatsa Mudigere, Sasikant Avancha, and Bharat Kaul. Seerl: Sample efficient ensemble reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1000 108, 2021.
[90] Tom Schaul, Georg Ostrovski, Iurii Kemaev, and Diana Borsa. Return-based scaling: Yet another normalisation trick for deep rl. arXiv preprint arXiv:2105.05347 2021.
[91] Julian Schritwieser,, Toannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 2020.
[92] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
15

---

[93] Max Schwarzer. Ankesh Anand, Rishab Goel, R Devon Hjelm. Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2021.
[94] Thibault Sellam, Steve Yadlowsky, Jason Wei, Naomi Saphra, Alexander Amour, Linzen, Jasmijn Bastings, Iulia Ture, Jacoh Fisenstein, Dipanjan Das, et al. The multiberts: Bert reproductions for rohustness analysis. arXiv preprint arXiv:2106. /6163, 2021.
[95] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In Proceedings of the 38th International Conference Machine Learning. 2021.
[96] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam. Mare Lanctot, al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):4844485 2016.
[97] Samarth Sinha, Homanga Bharadhwaj, Aravind Srinivas, and Animesh Garg. D2rl: Deep dense architec- tures in reinforcement learning. arXiv preprint arXiv:2010.0916: 2020.
[98] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, arXiv preprint 3rXiv:2004.041666 2020.
[99] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1): 9 44, 1988.
[100] Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. Advances in neural information processing systems, 1996.
[101] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 2nd edition, 2018
[102] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1999.
[103] István Szita and András Lörinez. Learning tetris using the noisy cross cross-entropy method. Neural computa- tion, 2006.
[104] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801. 180000900 2018.
[105] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. Is deep reinforcement learning really superhu- man on atari? leveling the playing field. arXiv preprint arXiv:1908. 04683, 2019.
[106] John W Tukey. survey of sampling from contaminated distributions. Contributions to probability and statistics, pages 448-485, 1960.
[107] Hado van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? NeurlPS, 2019.
[108] Gaël Varoquaux and Veronika Cheplygina. How failed machine learning in medical imaging- shortcomings and recommendations. arXiv preprint 2xXi:22103.1029) 2021.
[109] Nino Vieillard, Olivier Pietquin, and Matthieu Geist, Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
[110] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathicu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 2019.
[111] Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. arXiv preprint 200iv:2010. 10814, 2020.
[112] Ronald L. Wasserstein, Allen L. Schirm, and Nicole A. Lazar. Moving to a world beyond "p < 0.05". The American Statistician, 2019.
[113] Bernard L Welch. The generalization ofstudent's problem when several different population variances are involved. Biometrika, 34(1/2):28 -5, 1947.
16

---

[114] Denis Yarats, Amy Zhang, 1lya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency model-free reinforcement learning from images. arXiv preprini arXiv:1910.01741. 2019.
[115] Jinhua Zhu, Yingee Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, and Houqiang Li. Masked contrastive representation learning reinforcement learning. arXiv preprint arXiv:2010.07470, 2020.
[116] Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. arXiv preprint arXiv:2106. 2021.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] See Ap- pendix A.I.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.2
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A.2
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Apache License, Version 2.0
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
17


Reviews:
[{"review_id": "aWYOVqljJNN", "reviewer": "Reviewer_9zHZ", "summary": "The comparisons between RL algorithms are always based on the mean/median obtained from several runs for each task. Because of the limited number of experiments, the comparison can be overwhelmed by the uncertainty. This paper proposes using new metrics such as IQM to replace mean/median; furthermore, it also argues that reporting performance distributions would be a better choice. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- This paper has an extensive number of experiments, which is very valuable. \n- I appreciate the open-source code, such as colab, that is easy to implement. \n- The problem is well-motivated, it is important to find a more robust way to make comparisons when the computation resources are limited. \n\nWeakness:\nThe paper proposes using robust aggregate statistics (e.g. IQM) with interval estimates and performance distributions as measures to make comparisons among RL tasks. I have three concerns/questions regarding this (I'll mainly consider the case N=5, because usual medium and small labs don't have resources beyond that)\n- From Fig2 and Fig7, we can see that the interval estimates and performance distributions between different algorithms may have significant overlaps, e.g., DrQ and SPR in Fig2 right panel for 3/5 runs, then can we claim a winner between these two algorithms? Or should we eventually have to compare the single aggregate statistics in this case?\n- The authors argue that IQM is a more robust statistics compared with its surrogates such as median, first I don't think they're more robust to outliers than median since median uses less data points than IQM does, so the chance of median getting affected by outliers is less than IQM. It looks obvious that IQM will have a much smaller variance because it averages across more data points, however mean can have even smaller variance. So I think the point here is really how \"strange\" the normalized score distribution can be: if it's like a Gaussian distribution, mean will be the best choice; if the other extreme situation, median may be more appropriate. If in the middle, IQM may play its role. But we never know what the true distribution looks like (depending on the algorithm and environment), so there is no guideline for this. \n- Generalization: as pointed above, the choices and conclusions are made on Atari games mainly because the authors have $100$ runs of each algorithm in every environment (which we can consider them as the ground truth). However, I don't know how much it can be generalized to other task sets or another set of algorithms. \n\nMinor Remarks:\n- I would suggest to clarify the tasks for each plot. If all plots are using all the tasks, should the heteroskedasticity be considered when doing the test?\n- l.77: It's not independent of the rest of the tasks because the order matters. \n- Fig2: Is there any explanation why SPR with a larger $N$ has a larger variance than DER with a smaller $N$?\n- Fig2: Why is Median and IQM different for $N=3$? I think both of them would use the middle value in this case. \n- Fig6: I'm confused by the description here, I thought each task has 100 runs and using bootstrap with replacement?\n- Fig6: It is interesting that Median seems to do a better job than IQM when the cases are small (N<5)\n- l.163: It would be helpful to provide details on how this values are calculated. For example, what is the power that requires 50-100 runs?\n- eq(1): It should be $\\hat{F}_i(\\tau)$ for summation\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "S7FsNCyONXS", "reviewer": "Reviewer_9Qic", "summary": "This paper discusses the evaluation protocol dominant in deep reinforcment learning (evaluating 3-5 runs across a range of games/environments and reporting point estimates of performance such as mean or median), and describes several severe limitations of that protocol. The protocol ignores the uncertainty present in evaluating performance, especially with so few runs. They demonstrate the various failure modes of this protocol with a case study on the Atari@100k benchmark, showing that the reported results of previous methods are misleading and don't capture the variability possible in evaluation. The paper then proposes several methods for more robust evaluation measures which have tighter confidence intervals, including using inter-quartile mean (IQM) instead of mean or median, calculating interval estimates with bootstrap confidence intervals with stratified sampling, and introducing run-score distributions and performance profiles. The authors then demonstrate the use of these tools on Atari@200M, OpenAI Procgen and DeepMind Control from pixels, showing that previous evaluations are sometimes misleading or confusing, and don't properly capture the full nature of the differences between methods.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Overview\n\nI believe this paper is an important contribution to the field of RL. It points out a failure mode common in reinforcment learning evaluation and comparison. While similar failures of evaluation have been reported before, it seems (as demonstrated by the case studies in the paper) that these failures haven't been adaquately addressed. Further, the degree of detail and rigour taken in presenting the experimental results (and visualising those results) is a credit to the authors and means the work has an excellent clarity in it's presentation. There are a few small clarifications or adjustments to the paper which I believe would be beneficial.\n\n## Strengths\n\nOriginality: As noted in the paper (e.g. L318) there has been previous work looking at the evaluation protocol of deep reinforcement learning, but this paper evaluates these problems more thoroughly, finds problems in previous recommendations of increasing the number of random seeds, and proposes new methods for tackling these issues in deep reinforcement learning evaluation. As noted in the Weaknesses section below, I do think a better comparison of the results, recommendations and methods in this work to previous work would be beneficial to clarify the novelty of the contribution.\n\nClarity: The paper is very well-written, and explains it's points clearly in all parts. The visualisations are well explained and provide good intuition for the variety of statistical ideas presented in the text of the paper, in a way that is rigorous. The paper's structure is well-organised the motivates the main messages of the paper well.\n\nQuality: The technical quality of the experiments and visualisations is very high, and the techniques used and choices made are well-motivated. The claims made are well supported by the experiments.\n\nSignificance: The results presented in the paper of very high significance to the reinforcement learning community. The demonstration of spurious or misleading reporting of previous results is very important and should inform the field as to which methods are actually improvements on previous methods, and in what way. The improved approaches for reporting results, including representing the full distribution of results in multiple ways, should be taken up by current researchers in RL to improve the robustness of their results and determining whether real progress is being made.\n\n## Weaknesses\n\nIt would be useful to have a (small) related work section (possibly in the appendix, given the space constraints) on previous work and what it demonstrated, to clarify the contributions in this paper. For example, the papers listed in L318 could be explained in more detail.\n\n### Suggestions for improvements\n\n- L186: I think it would be more fair to say that the increase in the number of runs which would be required to address the statistical uncertainty issues is infeasible for computationally demanding deep RL benchmarks.\n- L233: I think the \"(1)\" should go after \"... is also\" for this list to read correctly.\n- From my understanding of dopamine, using the Rainbow implementation from dopamine isn't the full Rainbow algorithm (it misses out on three components - dueling DQN, double DQN and NoisyNets) - this should be made clear in the paper, as it changes the results on the Atari@200M benchmark. For example, see the Rainbow results in table 2 of [Muesli: Combining Improvements in Policy Optimization](https://arxiv.org/abs/2104.06159), which has a much higher median than reported in your paper (and in the DreamerV2 paper, which also uses the dopamine implementation of Rainbow).\n\n## Edit\n\nUpon reading the response and other reviews, I have raised my confidence from 3 to 4, maintaining my score of 9.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe limitations and societal impacts are adaquately addressed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4xH37PivD3_", "reviewer": "Reviewer_KFpZ", "summary": "This paper analyzes a large set of existing deep RL methods, proposes a variety of protocols towards more rigorous evaluation even in the presence of a small count (\"handful\") of runs, and, through an extensive set of analyses, demonstrates a variety of erroneous conclusions drawn from previous evaluation in prior work. The main protocols proposed are (1) to use confidence intervals via bootstrap estimation, (2) to use so-called \"performance profiles\" / \"run-score distributions\" for any given randomized score (scalar performance metric), (3) to use additional metrics, mainly the Interquartile Mean (IQM), but also \"optimality gap\", \"probability of improvement\", \"difficulty progress\", and \"probability of being superhuman\".", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main comments\n--\nThis paper is another step in the growing area of research concerned with establishing a more rigorous evaluation of deep RL algorithms. The paper's main strengths are the originality of the proposed protocols (particularly the performance profile), as well as an extensive performance analysis of a large set of deep RL methods. The paper is of high-quality -- the experiments are illuminating because they are very relevant and well-done, and the writing is overall quite clear. The paper could prove quite significant if its proposals are adopted.\n\nOne of the paper's weaknesses is that its conclusions are interspersed throughout the paper and appendix, which is more difficult to digest than a clearer \"executive summary\" of proposed protocols and new findings. While I think it's likely that most readers may agree with the spirit of more rigorous analysis, I think it's likely that many readers that do not closely read the paper may miss the important recommendations and findings. The abstract mentions \"we present a more rigorous evaluation methodology\", but the reality of the paper is that the methodological recommendations are scattered throughout, as opposed to consolidated into a single compact proposed methodology. Relatedly, the abstract claims that the paper \"reveal[s] discrepancies in prior comparisons\", but these are not really consolidated or summarized in one place. These issues could be addressed by either making enumerated lists or tables of both (1) the proposed protocols / methodology (e.g. use CIs, use performance profiles, use IQM), and (2) the new findings that shed more light on previous findings (those described in 5.1, as well as those in the appendix).\n\nAnother weakness is the underdeveloped discussion of how this paper relates to prior work (L318 [11,15,16,32,37]). For example, [15,16,32,37] also discuss CIs for deep RL.\n\nOther comments\n--\nThe bars for CURL in Fig2. is very faint and hard to see -- the bars of the other methods are much easier to see.\n\nL96 The collection of scores, $y_{1:K}$, is undefined in terms of $x$.\n\nL165-169 The purpose of this experiment is not clearly explained at the beginning of the paragraph. The paper should describe why this experiment is run; from my understanding, it might be something like \"Consider a setting in which an algorithm is known to be better -- what is the reliability of median and IQM scores for accurately assessing performance differences as the true performance difference and number of runs varies?\"\n\nFig 6: The definition of \"true coverage %\" appears to only exist in the caption?  \n\nLooking at [19], there are a few difference between the score distributions proposed herein and the originally proposed performance profiles. These differences should be explicitly named and discussed. Two important differences are (1) performance profiles were defined without any bootstrapping to estimated CIs (2) performance profiles are empirical CDFs, whereas score distributions are their complements (tail distributions).\n\nThe argument that median is not a good indicator because it is calculated using \"at most two tasks\" (L256) is dubious. The median is an aggregate metric that is calculated using 100% of the runs, not 1 or 2.\n\nThe \"IQM\" acronym appears long before it is actually defined on L249.\n\nL254 Is the following formal claim? \"IQM has considerably less bias than median\". If so, then this claim requires evidence in the form of a derivation or citation, particularly because the audience for this paper is likely not intimately familiar with the IQM.\n\nThe ability to read the medians from the performance profiles would be better highlighted if the Fig 7 y=0.5 lines were thicker.\n\nL297 \"While publications make binary claims\" -> \"While publications sometimes make binary claims\". The former statement is too broad to be true.\n\nL267 It's not clear how P(X_m > Y_m) is estimated. More details are needed. After looking through the appendix, it appears to be on page 20. This definition should be referenced in the main paper near L267.\n\nAppendix P20, it's not clear if \"Alternative alternative Metrics\" is a typo or intentional\n\nL323 \"... the problem is not solved by fixing random seeds, as has sometimes been proposed\" -- consider adding citations here for epistemological completeness.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There was no centralized discussion of limitations. This needs to be included, although I'm not sure what the limitations might be. Perhaps one of the major limitations in the proposed protocol is the unclear incentives future deep RL papers have in actually employing the proposed protocol. I speculate that a lack of rigor in some prior work is partly due to a missing / weak incentive for papers to be more rigorous, as it generally entails more nuanced, tempered claims. I think that it might make sense to explicitly mention this phenomenon as a barrier to the adoption of this work.\n\nThe societal impacts aren't really addressed: section A.5 actually discussed 'research community impacts', which is a small subset of 'societal impacts'. Perhaps one sentence is needed to explicitly state that the paper poses no foreseeable strongly negative impacts (if the authors agree), and another sentence that discusses how the paper could positively impact society by constituting a step forwards in rigorous small run count-based evaluation, which reduces computational burden (and is therefore \"greener\" than using large run counts).", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2HrvadMdvy", "reviewer": "Reviewer_GYfs", "summary": "This paper presents an empirical analysis of the failures of evaluation protocols in deep RL research and proposes reporting interval estimates, the use of performance profiles, and the use of more robust aggregate metrics. The empirical analysis focuses on the Atari 100k benchmark (but also looks at Atari 200M, procgen, and the deepmind control suite) and finds that previously reported median scores are not very robust by using more compute to estimate the distribution of normalized scores. The ordering of various algorithms is often different depending on the precise evaluation protocol. The paper proposes bootstrapped interval estimates and performance profiles as better ways to report results that show the variability across runs, and proposes the interquartile mean, optimality gap and probability of improvement as alternatives to the median as aggregate metrics. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\n1. The paper raises an important problem in deep RL research. While issues of reproducibility have been raised before, this paper focuses specifically on the robustness of evaluation procedures and dives deeper into the issue. The findings of inconsistencies in the literature on this specific part of the pipeline seems like an important problem for the community to consider. \n2. The large empirical evaluation provides solid evidence for the paper's claims. This evaluation shows things like the ordering changing across different protocols, the high variability of prior metrics, and the reduced variability in the proposed metrics. Relative to prior work on reproducibility and evaluation protocols, this paper provides a substantially larger evaluation across many seeds and many benchmark suites in different settings with both finite and continuous action spaces that considers many highly performing algorithms in each benchmark. This large-scale evaluation can be useful for the community beyond this paper since the authors plan to release the data from the training runs. \n3. The paper's proposed solutions seem to help reduce the problems raised with prior work. The main idea to report intervals or performance profiles instead of point estimates is a clearly better practice. Then the IQM does indeed seem to reduce variability. \n4. The paper provides useful tools for the RL community. The paper plans to open source code in a colab notebook to compute all of the proposed confidence intervals, performance profiles, and metrics. It also plans to release all of the data from the large-scale experiments. \n\n---\n\n### Weaknesses:\n\n1. The organization and writing of the paper sometimes lacks focus making it difficult to follow. I understand that evaluation is a complicated issue and that the paper wants to make sure it covers a lot of different aspects and cases which makes it difficult to organize, but the paper could benefit substantially from a more coherent framing. As it is currently written, the paper reads as a laundry list of issues followed by a laundry list of proposals that each solve some parts of the issues . Something like a table that explicitly shows the issues with each method under consideration could help to provide some high level clarity for the reader of the relationship between the shortcomings of prior evaluation procedures versus the new proposals. It would also be useful to answer questions like: what are the tradeoffs between different methods? Can these be presented in a concise visualization/table or theorem/lemma? \n2. The paper lacks a more formal argument for the proposed metrics. Adding some theory about when and how the proposed metrics are more robust could help clarify the contribution. Even some illustrative examples (perhaps in the appendix) could help to show what sorts of failure modes the new methods avoid. \n\n---\n\n### Recommendation:\n\nAccept\n\nThis paper raises an important problem for deep RL and presents a large empirical study along with some new proposals like reporting bootstrapped confidence intervals. While the organization and theory in the paper could be improved, I think the contribution is good enough to warrant acceptance. \n\n---\n\n### Typos/minor comments/questions:\n\n1. Why does the paper not compute all of the proposed metrics for each task? For example, there is no performance profile or IQM for the control suite, the probability of improvement seems to only be computed on procgen, and . If all of these metrics are included in the paper, then these results should be reported in the appendix. Alternatively, maybe things would be cleaner if one or two of the metrics were removed from the paper. \n2. The clause \"we assert reporting interval estimates of aggregate performance\" on line 15 does not make sense. Perhaps the authors meant something like \"advocate for\" instead of \"assert\" here?\n3. This is sort of a nitpick, but I don't think the title is very well-aligned with the content of the paper. It would be useful to at least say something about \"evaluation\" in the title so that the reader gets an idea of what the paper is about.\n4. The proposed metrics often seems to order the metrics differently, eg. IQM vs. optimality gap in Figure 9. Of course this is to be expected since they are measuring different things, but more of a discussion about how to deal with this inconsistency could be beneficial. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses societal impact in the appendix. The discussion acknowledges the complexity of reproducibility, but could be a bit more forthcoming about how the proposed methods can help illustrate issues of variability in results, but do not solve this fundamental issue. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussions. The reviewers unanimously agree that this is a very strong paper with a convincing motivation [9zHZ] and thorough experiments [9zHZ,GYfs,9Qic]. It raises an important issue in deep RL research that could significantly help future work [KFpZ,9Qic]. Furthermore, reviewers appreciate the release of open source code and tools that will help future researchers [9zHZ,GYfs] with their evaluation protocol. Overall, this paper presents an important step in more rigorous evaluation of deep RL research. It identified shortcomings of prior evaluation protocols, questioning some of the conclusions made in recent years, as well as providing clear guidance on how to improve evaluation in the future. I agree with the reviewers and strongly recommend acceptance. I also encourage the authors to use the feedback by the reviewers to further improve their paper, in particular it's clarity.
Decision: Accept (Oral)