Deep Reinforcement Learning at the Edge of the Statistical Precipice

1 Introduction . Research in artificial intelligence , and particularly deep reinforcement learning ( RL ) , relies on evaluating aggregate performance on a diverse suite of tasks to assess progress . Quantitative evaluation on a suite of tasks , such as Atari games [ 5 ] , reveals strengths and limitations of methods while simultaneously guiding researchers towards methods with promising results . Performance of RL algorithms is usually summarized with a point estimate of task performance measure , such as mean and median performance across tasks , aggregated over independent training runs . A small number of training runs ( Figure 1 ) coupled with high variability in performance of deep RL algorithms [ 16 , 17 , 41 , 68 , 70 ] , often leads to substantial statistical uncertainty in reported point ∗Outstanding Paper Award . Correspondence to Rishabh < rishabhagarwal @ google.com > . 2https : //github.com/google-research/rliable 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . estimates . While evaluating more runs per task has been prescribed to reduce uncertainty and obtain reliable estimates [ 20 , 41 , 49 ] , 3-10 runs are prevalent in deep RL as it is often computationally prohibitive to evaluate more runs . For example , 5 runs each on 50+ Atari 2600 games in ALE using standard protocol requires more than 1000 GPU training days [ 15 ] . As we move towards more challenging and complex RL benchmarks ( e.g. , StarCraft [ 110 ] ) , evaluating more than a handful of runs will become increasingly demanding due to increased amount of compute and data needed to tackle such tasks . Additional confounding factors , such as exploration in the low-data regime , exacerbates the performance variability in deep RL – as seen on the Atari 100k benchmark [ 50 ] – often requiring many more runs to achieve negligible statistical uncertainty in reported estimates . Ignoring the statistical uncertainty in deep RL results gives a false impression of fast scientific progress in the field . It inevitably evades the question : “ Would similar findings be obtained with new independent runs under different random conditions ? ” This could steer researchers towards superficially beneficial methods [ 11 , 12 , 25 ] , often at the expense of better methods being neglected or even rejected early [ 67 , 74 ] as such methods fail to outperform inferior methods simply due to less favorable random conditions . Furthermore , only reporting point estimates obscures nuances in comparisons [ 85 ] and can erroneously lead the field to conclude which methods are state-ofthe-art [ 63 , 84 ] , ensuing wasted effort when applied in practice [ 108 ] . Moreover , not report- ing the uncertainty in deep RL results makes them difficult to reproduce except under the exact same random conditions , which could lead to a reproducibility crisis similar to the one that plagues other fields [ 4 , 44 , 78 ] . Finally , unreliable results could erode trust in deep RL research itself [ 45 ] . In this work , we show that recent deep RL papers compare unreliable point estimates , which are dominated by statistical uncertainty , as well as exploit non-standard evaluation protocols , using a case study on Atari 100k ( Section 3 ) . Then , we illustrate how to reliably evaluate performance with only a handful of runs using a more rigorous evaluation methodology that accounts for uncertainty in results ( Section 4 ) . To exemplify the necessity of such methodology , we scrutinize performance evaluations of existing algorithms on widely used benchmarks , including the ALE [ 5 ] ( Atari 100k , Atari 200M ) , Procgen [ 18 ] and DeepMind Control Suite [ 104 ] , again revealing discrepancies in prior comparisons ( Section 5 ) . Our findings call for a change in how we evaluate performance in deep RL , for which we present a better methodology to prevent unreliable results from stagnating the field . How do we reliably evaluate performance on deep RL benchmarks with only a handful of runs ? As a practical solution that is easily applicable with 3-10 runs per task , we identify three statistical tools ( Table 1 ) for improving the quality of experimental reporting . Since any performance estimate based on a finite number of runs is a random variable , we argue that it should be treated as such . Specifically , we argue for reporting aggregate performance measures using interval estimates via stratified bootstrap confidence intervals , as opposed to point estimates . Among prevalent aggregate measures , mean can be easily dominated by performance on a few outlier tasks , while median has high variability and zero performance on nearly half of the tasks does not change it . To address these deficiencies , we present more efficient and robust alternatives , such as interquartile mean , which are not unduly affected by outliers and have small uncertainty even with a handful of runs . Furthermore , to reveal the variability in performance across tasks , we propose reporting performance distributions across all runs . Compared to prior work [ 5 , 83 ] , these distributions result in performance profiles [ 26 ] that are statistically unbiased , more robust to outliers , and require fewer runs for smaller uncertainty . 2 Formalism . We consider the setting in which a reinforcement learning algorithm is evaluated on M tasks . For each of these tasks , we perform N independent runs3 which each provide a scalar , normalized score 3A run can be different from using a fixed random seed . Indeed , fixing the seed may not be able to control all sources of randomness such as non-determinism of ML frameworks with GPUs ( e.g. , Figure A.13 ) . xm , n , m = 1 , . . . , M and n = 1 , . . . , N . These normalized scores are obtained by linearly rescaling per-task scores4 based on two reference points ; for example , performance on the Atari games is typically normalized with respect to a random agent and an average human , who are assigned a normalized score of 0 and 1 respectively [ 75 ] . We denote the set of normalized scores by x1 : M,1 : N . In most experiments , there is inherent randomness in the scores obtained from different runs . This randomness can arise from stochasticity in the task , exploratory choices made during learning , randomized initial parameters , but also software and hardware considerations such as non-determinism in GPUs and in machine learning frameworks [ 116 ] . Thus , we model the algorithm ’ s normalized score on the mth task as a real-valued random variable Xm . Then , the score xm , n is a realization of the random variable Xm , n , which is identically distributed as Xm . For τ ∈ R , we define the tail distribution function of Xm as Fm ( τ ) = P ( Xm > τ ) . For any collection of scores y1 : K , the empirical tail distribution function is given by F̂ ( τ ; y1 : K ) = 1K ∑K k=1 1 [ yk > τ ] . In particular , we write F̂m ( τ ) = F̂ ( τ ; xm,1 : N ) . The aggregate performance of an algorithm maps the set of normalized scores x1 : M,1 : N to a scalar value . Two prevalent aggregate performance metrics are the mean and median normalized scores . If we denote by x̄m = 1N ∑N n=1 xm , n the average score on task m across N runs , then these aggregate metrics are Mean ( x̄1 : M ) and Median ( x̄1 : M ) . More precisely , we call these sample mean and sample median over the task means since they are computed from a finite set of N runs . Since x̄m is a realization of the random variable X̄m = 1N ∑N n=1Xm , n , the sample mean and median scores are point estimates of the random variables Mean ( X̄1 : M ) and Median ( X̄1 : M ) respectively . We call true mean and true median the metrics that would be obtained if we had unlimited experimental capacity ( N →∞ ) , given by Mean ( E [ X1 : M ] ) and Median ( E [ X1 : M ] ) respectively . Confidence intervals ( CIs ) for a finite-sample score can be interpreted as an estimate of plausible values for the true score . A α× 100 % CI computes an interval such that if we rerun the experiment and construct the CI using a different set of runs , the fraction of calculated CIs ( which would differ for each set of runs ) that contain the true score would tend towards α× 100 % , where α ∈ [ 0 , 1 ] is the nominal coverage rate . 95 % CIs are typically used in practice . If the true score lies outside the 95 % CI , then a sampling event has occurred which had a probability of 5 % of happening by chance . 4Often the average undiscounted return obtained during an episode ( see Sutton and Barto [ 101 ] for an explanation of the reinforcement learning setting ) . Remark . Following Amrhein et al . [ 2 ] , Romer [ 87 ] , Wasserstein et al . [ 112 ] , we recommend using confidence intervals for measuring the uncertainty in results and showing effect sizes ( e.g. , performance improvements over baseline ) that are compatible with the given data . Furthermore , we emphasize using statistical thinking but avoid statistical significance tests ( e.g. , p-value < 0.05 ) because of their dichotomous nature ( significant vs. not significant ) and common misinterpretations [ 33 , 35 , 73 ] such as 1 ) lack of statistically significant results does not demonstrate the absence of effect ( Figure 2 , right ) , and 2 ) given enough data , any trivial effect can be statistically significant but may not be practically significant . 3 Case Study : The Atari 100k benchmark . We begin with a case study to illustrate the pitfalls arising from the naïve use of point estimates in the few-run regime . Our case study concerns the Atari 100k benchmark [ 50 ] , an offshoot of the ALE for evaluating data-efficiency in deep RL . In this benchmark , algorithms are evaluated on only 100k steps ( 2-3 hours of game-play ) for each of its 26 games , versus 200M frames in the ALE benchmark . Prior reported results on this benchmark have been computed mostly from 3 [ 39 , 55 , 59 , 72 , 89 , 95 ] or 5 runs [ 50 , 51 , 53 , 54 , 64 , 66 , 86 , 107 , 115 ] , and more rarely , 10 [ 65 , 93 ] or 20 runs [ 56 ] . Our case study compares the performance of five recent deep RL algorithms , namely : ( 1 ) DER [ 107 ] and ( 2 ) OTR [ 51 ] , ( 3 ) DrQ5 [ 53 ] , ( 4 ) CURL [ 56 ] , and ( 5 ) SPR [ 93 ] . We chose these methods as representative of influential algorithms within this benchmark . Since good performance on one game can result in unduly high sample means without providing much information about performance on other games , it is common to measure performance on Atari 100k using sample medians . Refer to Appendix A.2 for more details about the experimental setup . We investigate statistical variations in the few-run regime by evaluating 100 independent runs for each algorithm , where the score for a run is the average returns obtained in 100 evaluation episodes taking place after training . Each run corresponds to training one algorithm on each of the 26 games in Atari 100k . This provides us with 26× 100 scores per algorithm , which we then subsample with replacement to 3–100 runs . The subsampled scores are then used to produce a collection of point estimates whose statistical variability can be measured . We begin by using this experimental protocol to highlight statistical concerns regarding median normalized scores . High variability in reported results . Our first observation is that the sample medians reported in the literature exhibit substantial variability when viewed as random quantities that depend on a 5DrQ codebase uses non-standard evaluation hyperparameters . Instead , DrQ ( ε ) corresponds to DrQ with standard ε-greedy parameters [ 14 , Table 1 ] in ALE . See Appendix for more details . small number of sample runs ( Figure 2 , left ) . This shows that there is a fairly large potential for drawing erroneous conclusions based on point estimates alone . As a concrete example , our analysis suggests that DER may in fact be better than OTR , unlike what the reported point estimates suggest . We conclude that in the few-run regime , point estimates are unlikely to provide definitive answers to the question : “ Would we draw the same conclusions were we to re-evaluate our algorithm with a different set of runs ? ” Substantial bias in sample medians . The sample median is a biased estimator of the true median : E [ Median ( X̄1 : M ) ] 6= Median ( E [ X1 : M ] ) in general . In the few-run regime , we find that this bias can dominate the comparison between algorithms , as evidenced in Fig- ure 3 . For example , the score difference between sample medians with 5 and 100 runs for SPR ( +0.03 points ) is about 36 % of its mean improvement over DrQ ( ε ) ( +0.08 points ) . Adding to the issue , the magnitude and sign of this bias strongly depends on the algorithm being evaluated . Statistical concerns can not be satisfactorily addressed with few runs . While claiming improvements with 3 or fewer runs may naturally raise eyebrows , folk wisdom in experimental RL suggests that 20 or 30 runs are enough . By calculating 95 % confidence interval6 on sample medians for a varying number of runs ( Figure 2 , right ) , we find that this number is closer to 50–100 runs in Atari 100k – far too many to be computationally feasible for most research projects . Consider a setting in which an algorithm is known to be better – what is the reliability of median and IQM ( Section 4.3 ) for accurately assessing performance differences as the number of runs varies ? Specifically , we consider two identical N -run experiments involving SPR , except that we artificially inflate one of the experiments ’ scores by a fixed fraction or lift of + ` % ( Figure 4 ) . In particular , ` = 0 corresponds to running the same experiment twice but with different runs . We find that statistically defensible improvements with median scores is only achieved for 25 runs ( ` = 25 ) and 100 runs ( ` = 10 ) . With ` = 0 , even 100 runs are insufficient , with deviations of 20 % possible . Changes in evaluation protocols invalidates comparisons to prior work . A typical and relatively safe approach for measuring the performance of an RL algorithm is to average the scores received in their final training episodes [ 69 ] . However , the field has seen a number of alternative protocols used , including reporting the maximum evaluation score achieved during training [ 1 , 3 , 75 ] or across multiple runs [ 32 , 47 , 82 ] . A similar protocol is also used by CURL and SUNRISE [ 59 ] ( Appendix A.4 ) . Results produced under alternative protocols involving maximum are generally incomparable with end-performance reported results . On Atari 100k , we find that the two protocols produce substantially different results ( Figure 5 ) , of a magnitude greater than the actual difference in score . In particular , evaluating DER with CURL ’ s protocol results in scores far above those reported for CURL . In other words , this gap in evaluation procedures resulted in CURL being assessed as achieving a greater true median than DER , where our experiment gives strong support to DER being superior . Similarly , we find that a lot of SUNRISE ’ s improvement over DER can be explained by the change in evaluation protocol ( Figure 5 ) . Refer to Appendix A.4 for discussion on pitfalls of such alternative protocols .

Summary: This paper discusses the evaluation protocol dominant in deep reinforcment learning (evaluating 3-5 runs across a range of games/environments and reporting point estimates of performance such as mean or median), and describes several severe limitations of that protocol. The protocol ignores the uncertainty present in evaluating performance, especially with so few runs. They demonstrate the various failure modes of this protocol with a case study on the Atari@100k benchmark, showing that the reported results of previous methods are misleading and don't capture the variability possible in evaluation. The paper then proposes several methods for more robust evaluation measures which have tighter confidence intervals, including using inter-quartile mean (IQM) instead of mean or median, calculating interval estimates with bootstrap confidence intervals with stratified sampling, and introducing run-score distributions and performance profiles. The authors then demonstrate the use of these tools on Atari@200M, OpenAI Procgen and DeepMind Control from pixels, showing that previous evaluations are sometimes misleading or confusing, and don't properly capture the full nature of the differences between methods.