Measuring the Reliability of Reinforcement Learning Algorithms
Authors: Stephanie C.Y. Chan
Samuel Fishman
Anoop Korattikara
John Canny
Sergio Guadarrama
Venue:  (2020)
Keywords: reinforcement learning
metrics
statistics
reliability

Abstract: Lack of reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts to improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a set of metrics that quantitatively measure different aspects of reliability. In this work, we focus on variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general-purpose, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. The metrics and accompanying statistical tools have been made available as an open-source library. We apply our metrics to a set of common RL algorithms and environments, compare them, and analyze the results.

Published as a conference paper at ICLR 2020
MEASURING THE RELIABILITY OF REINFORCEMENT LEARNING ALGORITHMS
Stephanie C.Y. Chan," 1 Samuel Fishman," John Canny, 42 Anoop Korattikara,' & Sergio Guadarrama' Google Research Berkeley EECS
[scychan, sfishman, canny, xbanoop, sguada /agpaadda@@gggge....ooo @egoogll. com
ABSTRACT
Lack of reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a set of metrics that quantitatively measure different aspects of reliability. In this work, we focus on variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general -purpos,, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. The metrics and accompanying statistical tools have been made available an open-source library T We apply our metries to set of common RL algorithms and environments, compare them, and analyze the results.
INTRODUCTION
Reinforcement learning (RL) algorithms, especially Deep RL algorithms, tend to be highly variable in performance and considerably sensitive to range of different factors, including implementation details, hyper-parameters, choice of environments, and even random seeds (Henderson et al. (017). This variability hinders reproducible research, and can be costly even dangerous for real-world applications. Furthermore, it impedes scientific progress in the field when practitioners cannot reliably evaluate or predict the performance of any particular algorithm, compare different algorithms, or even compare different implementations of the same algorithm.
Recently, Henderson al. (017) has performed a detailed analysis of reliability for several policy gradient algorithms, while Duan et al. (2016) has benchmarked average performance of different citiinoous-onntror algorithms. other related work./Colas et al. (2018) have provided detailed anal- ysis on power analyses for mean performance in RL, and Colas et al. (2019) provide a comprehensive primer on statistical testing for mean and median performance R
In this work, we aim to devise a set of metrics that measure reliability of RL algorithms.. Our analysis distinguishes between several typical modes to evaluate RL performance: "evaluation during training". which is computed over the course of training, fixed policy after it has been trained. These metrics are also designed to measure different aspects of reliability, e.g. reproducibility (variability across training and variability across rollouts of fixed policy) or stability (variability within training runs). Additionally, the metrics capture multiple aspects of variability dispersion (the width of distribution), and risk (the heaviness and extremity of the lower tail of a distribution).
"evaluation after learning", which is evaluated on
Standardized measures of reliability can benefit the field of RL by allowing RL practitioners to compare algorithms in rigorous and consistent way. This in turn allows the field to measure
Work done as part of the Google Al Residency http://gghhhlcccmmooooggrrrrettt

---

Published as a conference paper at ICLR 2020
progress, and also informs the selection of algorithms for both research and production environments. By measuring various aspects reliability, we can also identify particular strengths and weaknesses of algorithms, allowing users to pinpoint specific areas of improvement.
In this paper, in addition to describing these reliability metrics, we also present practical recommen- dations for statistical tests compare metric results and how to report the results more generally. As examples, we apply these metrics to a set of algorithms and environments (discrete and continuous, off-policy and on-policy). We have released the code used in this paper as an open-source Python package to ease the adoption of these metrics and their complementary statistics.
Dispersion (D)
Risk (R)
Across Time (T) (within training runs)
IQR" within windows, after detrending
Short-term: CVaRt on
DURING TRAINING
first-order differences
Long-term: CVaRl
Drawdown
Across Runs (R)
IQR* across training runs, after low-pass filtering.
CVaRt across
AFTER LEARNING
Across rollouts on a Fixed Policy (F)
IQR* across rollouts for fixed policy
CVaRt across rollouts for a fixed policy
Table 1: Summary of our proposed reliability metrics. For evaluation DURING TRAINING, which measures reliability over the course of training an algorithm, the inputs the metrics are the performance curves of an algorithm, evaluated at regular intervals during a single training run (or on of training runs). For evaluation AFTER LEARNING, which measures reliability of an already- trained policy, the inputs to the metrics are the performance scores of set of rollouts of that fixed policy. *IQR: inter-quartile range. CCVaR: conditional value risk.
2 RELIABILITY METRICS
We target three different axes of variability, and two different measures of variability along each axis. We denote each of these by a letter, and each metric as a combination of an axis a measure, e.g. "DR" for "Dispersion Across Runs". See Table for summary. Please see Appendix A for more detailed definitions of the terms used here.
2.1 AXES VARIABILITY
Our metrics target the following three axes of variability. The first two capture reliability "during training", while the last captures reliability of fixed policy "after learning".
During training: Across
Time (T) In the setting of evaluation during training, one desirable property for RL algorithm is be stable "across time" within each training run. In general, smooth monotonic improvement is preferable to noisy fluctuations around positive trend, or unpredictable swings performance.
This type of stability is important for several reasons. During learning, especially when deployed for real applications, it can costly or even dangerous for an algorithm to have unpredictable levels of performance. Even in cases where bouts of poor performance do not directly cause harm, e.g. if training in simulation, high instability implies that algorithms have to be check check-pointed and evaluated more frequently in order catch the peak performance of the algorithm, which can be expensive. Furthermore. while training, can be a waste of computational resources to train an unstable algorithm that tends to forget previously learned behaviors.
2

---

Published as a conference paper at ICLR 2020
During training: Across Runs (R) During training, RL algorithms should have easily and consis- tently reproducible performances across multiple training runs. Depending on the components that we allow to vary across training runs, this variability can encapsulate the algorithm's 's sensitivity to a variety of factors, such as: random seed and initialization of the optimization, random seed and initialization of the environment, implementation details, and hyper-paramett the goals of the analysis, these factors can be held constant or allowed to vary, in order to disentangle the contribution of each factor to variability in training performance. High variability on any of these dimensions leads to unpredictable performance, and also requires a large search in order to find a model with good performance.
settings. Depending on
After learning: Across rollouts of fixed policy (F) When evaluating a fixed policy, natural concern is the variability in performance across multiple rollouts of that fixed policy. Each rollout may be specified e.g. in terms of number of actions, environment steps, episodes. Generally, this metric measures sensitivity to both stochasticity from the environment and stochasticity from the training procedure (the optimization). Practitioners may sometimes wish keep one the other constant if is important to disentangle the two factors (e.g. holding constant the random seed of the environment while allowing the random seed controlling optimization vary across rollouts).
2.2 MEASURES OF VARIABILITY
For each axis of variability, we have two kinds of measures: dispersion and risk.
Dispersion Dispersion is the width of the distribution. To measure dispersion, we use "robust statisties" such as the Inter-quartile range (IQR) (i.e. the difference between the 75th and 25th percentiles) and the Median absolute deviation from the median (MAD), which are more robust We prefer use IQR over MAD, because it is more appropriate for asymmetric distributions (Rousseeuw & Croux 1993).
statisties and don't require assuming normality of the distributions. [
Risk In many cases, we are concerned about the worst-case scenarios. Therefore, we define risk as the heaviness and extent of the lower tail of the distribution. This is complementary to measures of dispersion like IQR, which cuts off the tails of the distribution. To measure risk, we use the Conditional Value at Risk (CVaR), also known as "expected shortfall". CVaR measures the expected loss in the worst-case scenarios, defined by some quantile a. It is computed as the expected value in the left-most tail of a distribution (Acerbi & Tasche| 2002). We use the following definition for the CVaR of random variable X for given quantile a:
CVaR, EX S VaRo
(1)
where a E (0, 1) and the VaR, (Value at Risk) is just the a-quantile of the distribution of X. Originally developed in finance, CVaR has also seen recent adoption in Safe RL as an additional component of the objective function by applying it to the cumulative returns within an episode, e.g. Bäuerle & Ott (20TT): Chow & Ghavamzadeh 22014); Tamar et al. (2015). this work, we apply CVaR to the dimensions of reliability described in Section 2.1
2.3 DESIDERATA
In designing our metrics and statistical tests, we required that they fulfill the following criteria:
A minimal number of configuration parameters - to facilitate standardization as well as to minimize "researcher degrees of freedom" (where ficxibility may allow users to tune settings to produce more favorable results, leading to an inflated rate of false positives) (Simmons et al. 101
Robust statistics, when possible. Robust statistics are less sensitive outliers and have more reliable performance for a wider range of distributions. Robust statistics are especially
NNote that our aim here is to measure the variability of the distribution, rather than to characterize the uncertainty in estimating a statistical parameter of that distribution. Therefore, confidence intervals and other similar methods not suitable for the aim of measuring dispersion.
3

---

Published as a conference paper at ICLR 2020
important when applied training performance, which tends to be highly Con-Gausiian, making metrics such as variance and standard deviation inappropriate. For example, training performance is often bi-modal, with concentration of points near the starting level and another concentration at the level of asymptotic performance.
Invariance sampling frequency results should not be biased by the frequency which an algorithm was evaluated during training. See Section 2.5 for further discussion.
Enable meaningful statistical comparisons on the metrics, while making minimal assump- tions about the distribution of the results. We thus designed statistical procedures that are non parameemttticc (Section
OpenAI Gym During Training
Dispersion across Time
Short-term Risk across Time
Long-term Risk across Time
normalized mean rank
3-
SAC
103
I1O
DDPG REINP
DDPG
REiNE
Dispersion across Runs
Risk across Runs
Median Performance
normalized mean rank
2-
3--
5-
SAC
1D0
1PO
DDPG TEENN
SAC
T03
100
DDPG REINP
SAC
703
DDPG BEINT
OpenAl Gym After Learning
Dispersion of Fixed Policy
Risk of Fixed Policy
Median Performance
normallzzd mean rank
3-
4-
190 DUPO KEINE
SAC
DDPG REINE
SAC
PPO
DOING
REINP
Figure 1: RReiability metries and median performance for continuous control RL algorithms (DDPG, TD3, SAC, RHINFORCE, and PPO) tested on OpenAl Gym environments. Rank always indicates "best" reliability, e.g. lowest IQR across runs. Error bars are 95% bootstrap confidence intervals (# bootstraps = 1,000). Significant pairwise differences in ranking between pairs of algorithms are indicated by black horizontal lines above the colored bars. correction, permutation test with # permutations = 1,000). Note that the best algorithms by median performance are not always the best algorithms on reliability.
0.05 with Benjamini- Yekutilli
2.4 METRIC DEFINITIONS
Dispersion across Time (DT): IQR across Time To measure dispersion across time (DT), we wished to isolate righer-frequence variability, rather than capturing longer-term trends. We did not want our metrics to be influenced by positive trends of improvement during training, which are in fact desirable sources of variation in the training performance. Therefore, apply detrending before
4

---

Published as a conference paper at ICLR 2020
Atari -- During Training
Dispersion across Time
Short-term Risk across Time
Long-term Risk across Time
normalized mean rank
1-
3-
RBW
KON
CS1
DQN
10W
ION
C51
DON
K0N
CD1
DQN
Dispersion across Runs
Risk across Runs
Median Performance
normalized mean rank
2-
2000
10N
DON
C31
DON
PSW
C51
DQN
Atari -- After Learning
Dispersion of Fixed Policy
Risk of Fixed Policy
Median Performance
2-
normalizd
0000
000
1000
000
287V
RGN
C33
DQN
Figure 2: Reliability metrics and median performance for four DON-variants (C51, DQN: Deep Q-network, IQ: Implicit Quantiles, and RBW: Rainbow) tested on 60 Atari games. Rank I always indicates "best" reliability, e.g. lowest IQR across runs. Significant pairwise differences in ranking between pairs of algorithms are indicated by black lines above the colored circles. (a = 0.05 with Benjamini-Yekein Yekutieli correction, permutation test with permutations 1.000). Note that the best algorithms by median performance are not always the best algorithms on reliability. Error bars are 95% bootstrap confidence intervals (#f bootstraps = 1,000).
computing dispersion metrics. For detrending, we used differencing (i.e. / measure consisted of inter-quartile range (IQR) within a sliding window along the detrended training curve.
y: -1- 1) The final
Short-term Risk across Time (SRT): CVaR on Differences For this measure, we wish to measure the most extreme short-term drop over time. To do this, we apply CVaR to the changes in performance from one evaluation point to the next. l.e., in Eq. [ X represents the differences from one evaluation time-point to the next. We first compute the time-point to time-point differences on each training run. These differences are normalized by the distance between time-points, to ensure invariance to evaluation frequency (see Section 2.5). Then, we obtain the distribution of these differences, and find the a- a-quantile. Finally, we compute the expected value of the distribution below the a-quantile. This gives us the worst-case expected drop in performance during training, from one point of evaluation to the next.
'leasse see Appendix B for more detailed discussion of different types of detrending, and the rationale for choosing differencing here.
5

---

Published as a conference paper at ICLR 2020
Long-term Risk across Time (LRT): CVaR Drawdown For this measure, would also like to be able to capture whether an algorithm has the potential to lose a lot of performance relative to its peak, even if longer timescale, e.g. over an accumulation of small drops. For this measure, we apply CVaR to the Drawdown. The Drawdown at time 7 is the drop in performance relative to the highest peak so far, and is another measure borrowed from economics (Chekhlov al., 22055) I.e. Drawdowny = RT max Rg Like the SRT metric, the LRT can capture unusually large short-term drops in performance, but also capture unusually large drops that occur over longer timescales.
Dispersion across Runs (DR): IQR across Runs Unlike the rest of the metrics described here, the dispersion across training runs has previously been used to characterize performance (e.g. Duan et al. (2016); Islam et al. (2017): Bellemare et (201)); Fortunato [ et al. (2017): } Nagarajan et al. (2018)) This usually measured by taking the variance or standard deviation across training runs at set of evaluation points. We build on the existing practice by recommending first performing low-pass filtering of the training data, to filter out high-frequency variability within runs (this instead measured using Dispersion across Time, DT). We also replace variance or standard deviation with robust statistics like IQR.
Risk across Runs (RR): CVaR across Runs In order to measure Risk across Runs (RR), apply CVaR to the final performance of all the training runs. This gives a measure of the expected performance of the worst runs.
Dispersion across Fixed-Policy Rollouts (DF): IQR across Rollouts When evaluating a fixed policy, we are interested variability in performance when the same policy is rolled out multiple times. To compute this metric, simply compute the IQR the performance of the rollouts.
Risk across Fixed-Policy Rollouts (RF): CVaR across Rollouts except that we apply CVaR on the rollout performances.
This metric is similar to DF,
2.5 INVARIANCE TO FREQUENCY OF EVALUATION
Different experiments and different tasks may produce evaluations at different frequencies during training. Therefore, the reliability metrics should be unbiased by the choice of evaluation frequency. As long as there are no cyclical patterns in performance, the frequency evaluation will not bias any of the metrics except Long- Term Risk across Time (LRT). For all other metrics, changes in the frequency of evaluation will simply lead to more or less noisy estimates of these metrics. For LRT, comparisons should only be made if the frequency of evaluation is held constant across experiments.
RECOMMENDATIONS FOR REPORTING METRICS AND PARAMETERS
Whether evaluating algorithm for practical use for research, recommend evaluating all of the reliability metrics described above. Each metric measures a different aspect of reliability, and can help pinpoint specific strengths and weaknesses of the algorithm. Evaluating the metrics easy with the open-source Python package that we have released.
Reporting parameters. Even given our purposeful efforts to minimize the number of parameters in the reliability metrics, few remain to be specified by the user that affect the results, namely: window size (for Dispersion across Time), frequency threshold for low-pass and high-pass filtering (Dispersion across Time, Dispersion across Runs), evaluation frequency (only for Long-term Risk across Time), and length of training runs. Therefore, when reporting these metrics, these parameters need to be clearly specified, and must also be held constant across experiments for meaningful comparisons. The same is true for any other parameters that affect evaluation, e.g., the number of roll-outs per evaluation, the parameters of the environment, whether on-line or off-line evaluation is used, and the random seeds chosen.
Collapsing across evaluation points. Some of the in in training reliability metrics (Dispersion across Runs, Risk across Runs, and Dispersion across Time) need to be evaluated at multiple evaluation

---

Published as a conference paper at ICLR 2020
points along the training runs. If it is useful to obtain a small number of values to summarize each metric. we recommend dividing the training run into "time frames" (e.g. beginning, middle, and end), and collapsing across all evaluation points within each time frame.
Normalization by performance. Different algorithms can have vastly different ranges of perfor- mance even on the same task, and variability in performance tends to scale with actual performance. Thus, we normalize our metrics in post-processing by measure of the range of performance for each algorithm. For "during training" reliability, we recommend normalizing by the median range of performance, which we define the PP,5 Pt=o. where PP,5 is the 95th percentile and Pt=0 is the starting performance. For "after learning" reliability, the range performance may not be available, the median performance directly.
in which case
Ranking the algorithms. Because different environments have different ranges and distributions of reward, we must be careful when aggregating across environments or comparing between envi- ronments. Thus, if the analysis involves more than one environment, the per-environment median results for the algorithms are first converted to rankings, by ranking all algorithms within each task. To summarize the performance of single algorithm across multiple tasks, compute the mean ranking across tasks.
Per-environment analysis. The same algorithm can have different patterns of reliability for dif- ferent environments. Therefore, we recommend inspecting reliability metrics on per-environment basis, as well as aggregating across environments as described above.
4 CONFIDENCE INTERVALS AND STATISTICAL. SIGNIFICANCE TESTS FOR COMPARISON
4.1 CONFIDENCE INTERVALS
We assume that the metric values have been converted to mean rankings, as explained in Section 3 To obtain confidence intervals on the mean rankings for each algorithm, we apply bootstrap sampling on the runs, by resampling runs with replacement (Efron & Tibshirani 1986).
For metries that are evaluated per-run (e-g. Dispersion across Time), we can resample the metric values directly, and then recompute the mean rankings on each resampling to obtain distribution over the rankings; this allow us to compute confidence intervals. For metrics that are evaluated across-runs, need to resample the runs themselves, then evaluate the metrics each resampling, before recomputing the mean rankings to obtain a distribution on the mean rankings.
4.2 SIGNIFICANCE TESTS FOR COMPARING ALGOIIHTM
Commonly, we would like to compare algorithms evaluated on a fixed set of environments. To determine whether any two algorithms have statistically significant differences in their metric rankings, we perform an exact permutation test on each pair of algorithms. Such tests allow us to compute p-value for the null hypothesis (probability that the methods are in fact indistinguishable on the reliability metric).
We designed our permutation tests based the null hypothesis that runs exchangeable across the two algorithms being compared. In brief, let and B be sets of performance measurements for algorithms and b. Let Metric(X X) be reliability metric, e.g. the inter-quartile range across runs, computed on a set measurements X. Metric CadricRanking((XX X) is the mean ranking across tasks on X, compared the other algorithms being considered. We compute test statistic
Metriclankinng (A, B) MetricRankinng(A) MetricRanking(D).
Next we compute the distribution for SMetricRankinng under the null hypothesis that the meth- ods are equivalent, i.e. that performance measurements should have the same distribution for a and b. We do this by computing random partitions A', B' of {A B). and computing the test statistic SMrtriin (A', B') on each partition. This yields a distribution for 8 MetricRaaniing (for sufficiently many samples), and the p-value can be computed from the percentile value of
7

---

Published as a conference paper at ICLR 2020
MMetricRankinng4 (A, B) in this distribution. As with the confidence intervals, different procedure required for per-run vs across-run metrics. Please see Appendix for diagrams illustrating the permutation test procedures.
When performing pairwise comparisons between algorithms, it is critical to include corrections for multiple comparisons. This is because the probability of incorrect inferences increases with greater number of simultaneous comparisons. We recommend using the Benjamini- Yenjumnni-Yekutilli method, which controls the false discovery rate (FDR), i.e., the proportion of rejected null hypotheses that are false
4.3 REPORTING ON STATISTICAL TESTS
It is important to report the details of any statistical tests performed, e.g. which test was used, the significance threshold, and the type of multiple-comparioons correction used.
ANALYSIS OF RELIABILITY FOR COMMON ALGORITHMS AND ENVIRONMENTS
In this section, we provide examples of applying the reliability metrics to number of RL algorithms and environments, following the recommendations described above.
5.1 CONTINUOUS CONTROL ALGORITHMS ON OPENAI GYM
We applied the reliability metrics to algorithms tested on seven continuous control environments from the Open-Al Gym (Greg Brockman et al., (016)) run on the MuloCo physics simulator (Todorov et al.
2 We tested REINFORCE (Sutton et al.] 2000), DDPG (Lillicrap et al. 2015), PPO (Schulman et al. 2017). TD3 (Fujimoto et al. 2018). and SAC (Haarnoja et al. 2018 8 on the following Gym environments: Ant-v2, HalfCheetah-- Humanoid-v2. Reacher-v2, Swimmer-v2, and Walker2d-v2. We used the implementations of DDPG, TD3, and SAC from the TF-Agents library (Guadarrama ctt al.
2018). Each algorithm was run on each environment for 30 independent training runs.
We used a black-box optimizer (Golovin et al. 2017) to tune selected hyperparameters on per-task basis. optimizing for final performance. The remaining hyperparameters were defined as stated in the corresponding original papers. See Appendix E details of the hyperparameter search space and the final set of hyperparameters. During training, we evaluated the policies at a frequency of 1000 training steps. Each algorithm run for a total of two million environment steps. For the "online" evaluations we used the generated training curves, averaging returns over recent training episodes collected using the exploration policy as it evolves. The raw training curves shown in Appendix D For evaluations after learning on a fixed policy, we took the last checkpoint from each training run the fixed policy for evaluation. Each of these policies then evaluated for 30 roll-outs, where each roll-out was defined as 1000 environment steps.
5.2 DISCRETE CONTROL: DQN VARIANTS ON ATARI
We also applied the reliability metrics the RL algorithms and training data released as part of the Dopamine package (Castro et al. 2018). The data comprise the training runs of four RL algorithms, each applied 60 Atari games. The RL algorithms are: DQN (Mnih et al.) (015), Implicit Quantile (IQN) (Dabney et al. 2018), C51 (Bellemare et al. (017). and a variant of Rainbow implementing the three most important components (Hessel Modayil 2018). The algorithms were trained on each game for 5 training runs. Syper-parameter follow the original papers, but were modified as necessary to follow Rainbow (Hessel & Modayil 2018), to ensure poples-to-apple comparison. See Appendix E for the hyperparameters.
During training, the algorithms were evaluated in an "online" fashion every 1 million frames, averaging across the training episodes recommended for evaluations the ALE (Machado et al.) 2018). Each training run consisted of approximately 200 million Atari frames (rounding to the nearest
Forr situations in which user wishes instead to control the family-wise error rate (FWER; the probability of incorrectly rejecting least null hypothesis), we recommend using the Holm Bonferroni method.
8

---

Published as a conference paper at ICLR 2020
episode boundary every 1 million frames) For evaluations after learning on fixed policy ("after learning"), " we took the last checkpoint from each training run as the fixed policies for evaluation. We then evaluated each of these policies for 125,000 environment steps.
5.3 PARAMETERS FOR RELIABILITY METRICS, CONFIDENCE INTERVALS, AND STATISTICAL TESTS
For the MuJoCo environments, applied sliding window of 100000 training steps for Dispersion across Time. For the Atari experiments, we used a sliding window size of 25 on top of the evaluations for the Dispersion across Time. For metries with multiple evaluation points, we divided each training run into 3 time frames and averaged the metric rankings within each time frame. Because the results were extremely similar for all three time frames, here report just for the final time frames.
Statistical tests for comparing algorithms were performed according to the recommendations in Section 4 We used pairwise permutation tests using 10,000 permutations per test, with a significance threshold of 0.05 and Benjamini Yekutieli multiple-comparison correction.
5.4 MEDIAN PERFORMANCE
The median performance of an algorithm is not reliability metric, but interesting see side-by- side with the reliability metrics. For analyzing median performance for the DQN variants, we used the normalization scheme of (Mnih al., 2015), where an algorithm's performance normalized against lower baseline (e.g. the performance of a random policy) and an upper baseline (e.g. the P Byna
performance of human): P normalized continuous control algorithms.
Median performance was not normalized for the
Bupa Bune.
5.5 RESULTS
The reliability metric rankings are shown in Fig. for the MuJoCo results. We see that, according to Median Performance during training, SAC and TD3 have the best performance and perform similarly well, while REINFORCE performs the worst. However, SAC outperforms TD3 on all reliability metrics during training. Furthermore, both SAC and TD3 perform relatively poorly all reliability metrics after learning, despite performing best on median performance.
The reliability metric rankings shown in Fig. 2 for the Atari results. Here that, even though Rainbow performs significantly better than IQN in Median Performance, IQN performs numerically significantly better than Rainbow on many of the reliability metrics.
similar result
The differing patterns in these metrics demonstrates that reliability is a separate dimension that needs be inspected separately from mean or median performance two algorithms may have similar median performance but may nonetheless significantly differ in reliability, as with SAC and TD3 above. Additionally, these results demonstrate that reliability along one axis does not necessarily correlate with reliability on other axes, demonstrating the value of evaluating these different dimensions so that algorithms can be compared and selected based on the requirements of the problem at hand.
To see metric results evaluated on a per-environment basis, please refer to Appendix Rank order of algorithms was often relatively consistent across the different environments evaluated. However, different environments did display different patterns across algorithms. For example, even though SAC showed the same better Dispersion across Runs for most of the MuJoCo environments evaluated, it did show slightly worse Dispersion across Runs for the HalfCheetah environment (Fig Tap. This kind of result emphasizes the importance of inspecting reliability (and other performance metrics) on a per-environment basis, and also of evaluating reliability and performance on the environment of interest, if possible.
'The raw training curves can be viewed http://wggggggiiiioooopipiieeeee

---

Published as a conference paper at ICLR 2020
6 CONCLUSION
We have presented number of metrics, designed measure different aspects of reliability of RL algorithms. We motivated the design goals and choices made in constructing these metrics, and also presented practical recommendations for the measurement of reliability for RL. Additionally, we presented examples of applying these metrics common RL algorithms and environments, and showed that these metrics can reveal strengths and weaknesses of an algorithm that are obscured when we only inspect mean or median performance.
ACKNOWLEDGMENTS
Many thanks to the following people for helpful discussions during the formulation of these metrics and the writing of the paper: Mohammad Ghavamzadeh, Yinlam Chow, Danijar Hafner, Rohan Anil, Archit Sharma, Vikas Sindhwani, Krzysztof Choromanski, Joelle Pineau, Hal Varian, Shyue-Minn Loh, and Tim Hesterberg. Thanks also to Toby Boyd for his assistance the open-sourcing process, Oscar Ramirez for code reviews, and Pablo Castro for his help with running experiments using the Dopamine baselines data.
REFERENCES
Carlo Acerbi and Dirk Tasche. Expected Shortfall: A Natural Coherent Alternative to Value at Risk. Economic Notes, 31 31(2):399-388, July 2002. ISSN 0391-5026, 1468-0300. doi: 10.1111/ 1468-03000 106-00300.0009.
wiley www..oomm.....m wttp://///..11111000000000011110 00091
Mare G. Bellemare, Will Dabney, and Rémi Munos. A Distributional Perspective on Reinforcement Learning. arXiv:1707 170706777 [cs, stat], July 2017. URL
06887 arXiv:
Nicole Bäuerle and Jonathan Ott. Markov Decision Processes with Average-Value-at criteria. Mathematical Methods of Operations Research, 74(3):361-31) December 201 ISSN 1432-2994, doi: 10. 10.1007/500188--11-03333
1432-5217.
URL http: /tEtp://likk springer com/ 10
1007 00077/00186--1111 1007/800186-0110037---
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A research framework for deep reinforcement learning. CoRR, abs/18 htw//811.00111 10, 2018.
URL http:///xxxxv..rg/aa//
06110
Alexei Chekhlov, Stanislav Uryasev, and Michael Zaharankin. Drawdown measure in portfolio optimization. International Journal of Theoretical and Applied Finance, 8(1):46, 2005.
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR Optimization in MDPs. Advances in Neural Information Processing Systems, pp. 9, 2014.
Cédric Colas, Olivier Sigaud, and Pierre-Yvel Oudeyer. How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. 1806v: 08295 [cs, stat], June C8295
2018. URL
arXiv: 1806.08295.
Cédric Colas, Olivier Sigaud, and Pierre-Yve Oudeyer. A Hitchhiker's 's Guide Statistical Com- parisons of Reinforcement Learning Algorithms. arXiv: 190Xiv/904. 0904006979 [cs, stat], April 2019, URL
http://arxxv.org//aaa 1904 06979 arXiv: 1904.06979.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit Quantile Networks for Distributional Reinforcement Learning. Thirty-fith International Conference on Machine Learning, pp. 10, 2018.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking Deep Rein- forcement Learning for Continuous Control. In International Conference on Machine Learning, pp. 1329 1339-1338, June 2016.
mlr press / v48 duw///dannnn 6. html
B. Efron and R. Tibshirani. Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy. Staristical Science, 1(1):54-75, February 1986. ISSN 0883-4237, 2768-8745. doi: 10. 00.1214/ss/11770131111 URL
ss/ 0s/1177013815
10

---

Published as a conference paper at ICLR 2020
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy Networks for Exploration. 1706. 10295 [cs, stat], June 2017. http: arXiv: 1706. 10295.
arg/abs/1706. 10295
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. arXiv:/802.09441 [cs, stat], February 2018. URL [tL[hhtpp//axx
org arg/aba/Da 1802. 09477
arXiv: 1802.09477.
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and Sculley, Google Vizier: A Service for Black-Box Optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining KDD '17, pp. 1487-1495, Halifax. NS, Canada, 2017. ACM Press. ISBN 978 978-1-4503-4487-44 3098043. URL http: acm. oeapeoorgcciie wttggiiiaiiinn.oom wfm:doidddddeeoooo 3097983.... 3098043
doi: 10. 15/30999997
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym, 2016.
Sergio Guadarrama, Anoop Korattikara, Pablo Castro Oscar Ramirez, Ethan Holly, Sam Fishman, Ke Wang, Chris Harris Ekaterina Gonina, Vincent Vanhoucke, and Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow. /bttps:/ wtt.://soo/oeeeoloooooooooooo aggntts) 2018. URL https: http://wwww....ooccooooooooo///a aggetss [Online; accessed 20-No8v
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. arXiv:1801.01200 (cs, star), January 2018. URL bttp:://aaax wrg. 01293 arXiv: 1801.01290.
James D. Hamilton. Time Series Analysis. Princeton University Press, 1994.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep Reinforcement Learning that Matters. arXiv:1709.000 06060 (cs, stat], September 2017. URL httt::/aaxiv..r hwtp://wwwrr.bb/boaaooaobbosss/ abs abs//709. 06560
arXiv: 1709.06560.
Matteo Hessel and Joseph Modayil. Rainbow: Combining Improvements in Deep Reinforcement Learning. AAAI, pp. 8, 2.18.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of Bench- marked Deep Reinforcement Learning Tasks for Continuous Control. arXiv:1708. 10000000783 [cs]. August 2017. URL http:
org. abs/1708. 04133 arXiv: 1708.04133.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. 1509.02509 [es, stat), September 2015. URL httt::://axiv..rr//eaeaa wwt..bggg//oooo.oe arXiv: 1509.02971.
02971
Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journal of Artificial Intelligence Research, 61:523-562. March
2018. ISSN 1076-9757. doi: 10.1613/jair.569) URL http::ttt////t htttps::/www.. /wwp://wwww... jair .or.oogggioo... wrt//idexxxnnnoooooooooooooooeee index php/
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7550) 533, February 2015. ISSN 1476-4687. doi: 1 14236. URL https: www nature com/ art wtw.:i/eew..auuue15555566
Prabhat Nagarajan, Garrett Warnell, and Peter Stone. Deterministic Implementations for Repro- ducibility in Deep Reinforcement Learning. arXiv:1809.0566 [cs], September 2018. URL
http://arxis> http:/awwwwww.oora////a/aaaa//// 05676
arXiv: 1809.05676.

---

Published as a conference paper at ICLR 2020
Charles R Nelson and Charles I Plosser. Trends and random walks in macrocconomic time series. Journal of Monetary Economics, 10:139-162, 1982.
Peter Rousseeuw and Christophe Croux, Alternatives to the MedianAbsolute Deviation. Journal of the American Statistical Association, 1993.
Said E. Said and David A. Dickey. Testing for unit roots in autoregressive-moving average models of unknown order. Biometrika, 71(3):599-60) 1984.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. ftdi::7777.060. [cs], July 2017. URL
0011170763311 arXiv: 1707.06347.
Joseph Simmons, Leif D. Nelson, and Uri Simonsohn. Palse-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psy- chological Science, 22(11):1359-136) November 2011. ISSN 0956-7976, 1467-9280. doi: 10.1 10.1177/9956777111477777 URL httt:://ttttr/e/ s. sagepub. com/doi /10./001110000 117.7
0956797611417632
Richard Sutton, David A McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In NIPS'99 Proceedings of the 12th International Conference on Neural Information Processing Systems, pp. 7, 2000.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via Sampling. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 7, 2015.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control.
In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 Vilamoura-Alaarve Portugal, October 2012. IEEE ISBN 978-1 778-1-4673-173--3 978-1-463--7773 978-1-4673-1735-11 doi: 10.1109/ 2012.63861999 htc//docuumen//3661000
URL
ieee
A ASSUMPTIONS AND DEFINITIONS
Reinforcement Learning algorithms vary widely in design, and our metrics are based on certain notions that should span the gamut of RL algorithms.
Policy policy me (as|s;) a distribution over actions a; given a current (input) state 8,. assume policies are parameterized by parameter e.
Agent An agent defined
distribution over policies (or equivalently distribution over pa- rameters +). In many cases, an agent will be a single policy but for population-based RL methods, the agent is discrete set of policies.
Window A window a collection states over which the agent is assumed to have small variation. A window could be sequence of consecutive time steps for sequential RL algorithm, or collection of states at the same training step of a distributed RL algorithm with a parameter server (all agents share +).
Performance
The performance of an agent is the mean or median per-epoch reward from running that agent. If the agent is a single policy, then the performance p(Te) is the mean or median per-epoch reward for that agent. If the agent is a distribution D(O) of policies, then the performance is the median of p(Te) with ② D.
Training Run A training is sequence of updates to the agent D(O) from running reinforce-
ment learning algorithm. It leads to trained agent (imaa(()) Multiple training runs share information with each other.
We cannot directly measure performance since it is a statistic across an infinite sample of evaluation runs of an agent. Instead we windows compute sample medians to approximate performance.
12

---

Published as a conference paper at ICLR 2020
B DETRENDING BY DIFFERENCING
Typically, de-trending can performed in two main ways (Nelson & Plosser 1982 Hamilton 1994).
Differencing (i.e. yd = J/2 yt-1) is more appropriate for difference-stational (DS) processes (e.g. a random walk: 1/ = 1 + b + et), where the shocks accumulate over time. For trend-stationary (TS) processes, which are characterized by stationary fluctuations around a deterministic trend, e.g.
+ E,, it is more appropriate to fit and subtract that trend.
We performed an analysis of real training runs and verified that the data are indeed approximately DS, and that differencing does indeed remove the majority of time-dependent structure. For this analysis we used the training runs on Atari described in 5.2 Before differencing, the Augmented Dickey-Fulle test (ADF test, also known difference-staiee stationarity test: [Said E. Said & David A Dickey (1984) rejects the null hypothesis of unit root on only 72% the runs; after differencing. the ADF test rejects the null hypothesis
92% of the runs (p-value threshold 0.05). For the ADF test, the rejection of a unit root (of the autoregressive lag polynomial) implies the alternate hypothesis, which is that the time series is trend-stationar
Therefore, our training curves are better characterized as an accumulation shocks, i.c. as DS processes, rather than mean-reverting TS processes. They are not actually purely DS because the shocks 6g are not stationary over time, but because we compute standard deviation within sliding windows, we can capture the non-stationarity and change variability over time. Thus, we chose to detrend using differencing.
As further note in favor of detrending differencing, it useful observe that many measures of variability are defined relative to the central tendency of the data, e.g. the median absolute deviation MAD = median( (, - X)) where X is the median of X. On the raw data (without differencing), the MAD would be defined relative X median performance, performance are included in that computation of variability. On the other hand, if we compute MAD) on the 1st-order differences, we are using X that represents the median change in performance, which is a more reasonable baseline to compute variability against, when we are in fact concerned with the variability of those changes.
that any improvemetts in
final benefit of differencing that it is parameter-free.
C ILLUSTRATIONS OF PERMUTATION TEST PROCEDURES
We illustrate the procedure for computing permutation tests to compare pairs of algorithms on specified metric, in Figs. (for per-run metrics) and an across-run metrics).
D RAW TRAINING CURVES FOR OPENAI MuJoCo TASKS
In Figure 5 we show the raw training curves for the TF-Agents implementations of continuous-
control algorithms, applied to the OpenAl MuloCo tasks. These are compared against baselines from the literature, where available (DDPG and TD3: Fujimoto et al. (208)) PPO: [Schulman et al. (0017) SAC: Haarnoja et al. (2018)
E HYPERPARAMETER SETTINGS
For the continuous control experiments, hyperparameters were chosen on a per-environment ba- sis according the black-box optimization algorithm described in Golovin et al. (2017). The hyperparameter search space shown in Table
For the discrete control experiments, hyperparameter selection is described in (Castro et al.) 2018). Hyperparameters are shown in Table [] duplicated for reference from http://gnnncooooooooggommmmeeeee
13

---

Published as a conference paper at ICLR 2020
Comparing algorithms on per-run metrics
Raw values
egg runs per (task, algo)
algoA
-1,7.3
2..7.3
77.90 4
algoB
2,2
1.9. 0.344
5,32, 15
algoC
3,
64.
task1
task2
task3
Evaluate per-run
metrics for each
Metric values
0.1,0.7, 3
23,4.1,3
3, 6,6.9
algorilhms
.9.
0,-02..4
0.9. 1.6
8, 0.3
tasks
Rank within each task
Metric rankings
Permuted rankings
algorithms
4.5.8
5. 7.
Permute rankings across
1,6,7
7,5.3
3. 7.9
1,6,7
3,
algoy and algoB, within task
algorithin
permuted
3.2.
tasks
tasks
Get difference in mean ranking: algoA - algoB = 6.8 4.7
mean ranking:
algoA algoB 5.3 6.0
Repeat n permutations times
null distribution
actual difference
Figure 3: Diagram illustrating the computation of the permutation tests for per-run metrics (Disper- sion across Time, Short-term Risk across Time, Long-term Risk across Time). In this example, we are comparing Algorithm A and Algorithm B, and there are only 3 algorithms. 3 tasks, and 3 runs per (task, algo) pair. To compute the difference in average rankings for two algorithms, follow the gray arrows. To compute null distribution of difference in average rankings (by permuting the runs), follow the blue arrows a number of times (e.g. 1,000 times). Once the null distribution has been computed, the actual value of the difference can be compared with the null distribution to obtain p-value.
14

---

Metric values on permuted runs Published as a conference paper at ICLR 2020
Comparing algorithms on across-run metrics
Raw values
egg.
runs per (task, algo)
Permuted values
-1,7,3
2,3
77, 90.00, 4
07,
25.7.3
Permute runs across
algonthmm
permuted
2,
1.3,0.3,
5,32, 15
algoÅ algoB, within task
4. 2.0
1.9.0.344
5,32, 15
algoC
3.
6.10.5
2,
6. 10,5
52, 64,
task1
lask2
lask3
tasks
Evaluate across-run
Evaluate
metrics for each (task, algo)
metrica for each (lask, algo)
Metric values
0.1
2.3
1.4
algorithms
algorithms
-0.4
2.5
0.6
1.2
2.1
0.8
-0.2
1.9
0.3
-0.2
1.9
0.3
tasks
tasks
Rank within each task
Rank within each task
Metric rankings
Metric rankings permuted
algonttmm
algorithms
3
3
tasks
Get difference mean ranking: alguA algoB 2 1.67
Get difference mean ranking: algoA algoB 5.3 6.0
Repeat n_ peonutatioonnns
distribution
difference
Figure 4: Diagram illustrating the computation of the permutation tests for across-run or across- rollout metries (Dispersion across Runs, Risk Across Runs, Dispersion across Fixed-policy rollouts, Risk across Fixed-Policy rollouts). In this example, we are comparing Algorithm A and Algorithm B, and there are only algorithms, tasks, and runs per (task, algo) pair. To compute the difference in average rankings for two algorithms, follow the gray arrows. To compute a null distribution of difference in average rankings (by permuting the runs), follow the blue arrows number of times (e.g. 1. 1,000 times). Once the null distribution has been computed, the actual value of the difference can be compared with the null distribution to obtain p-value.
15

---

Published as a conference paper at ICLR 2020
8000
12923
30000
2400
6933
8023
2002
-1000
-003
2502030
Mumanaic
$030
10-
4038
-15
20
-22
2030
SXX0c
1300cce
1500cco
200000
503003
1000000
1500000
2000033
2599298
4020
2020
2528
30
43
1058
SUMMM
J35.800
Ibcreet
AMMMU
ZARMM
303230
DAMMAU
LSMMAU
0933393
EX30962
Figure 5: Raw training curves for OpenAI MuJoCo tasks. The x-axes indicate enviromment steps, and the y-axes indicate average per-episode return. Dotted lines indicate baseline performance from the literature, where available.
16

---

Published as a conference paper at ICLR 2020
Table 2: Hyperparameter search space for continuous control algorithms.
Algorithm
Hy plperaaraeeeer
Search min
Search max
SAC
actor learning rate
0.000001
0.001
learning rate
0.000001
0.001
critic learning rate
0.000001
0.001
target update 7
0.00001
1.0
TD3
actor learning rate
0.000001
0.001
critic learning rate
0.000001
0.001
target update
0.00001
1.0
PPO
learning rate
0.000001
0.001
DDPG
actor learning rate
0.000001
0.001
critic learning rate
0.000001
0.001
target update
0.00001
1.0
REINFORCE
learning rate
0.000001
0.001
# episodes before each train step
1.0
10
Table 3: Final hyperparameters for SAC.
actor learning rate
learning rate
critic learning rate
target update
Ant-v2
0.000006
0.000009
0.0009
0.0002
HalfCheetah-v2
0.0001
0.000005
0.0004
0.02
Humanoid-v2
0.0003
0.0008
0.0006
0.8
Reacher-v2
0.00001
0.000002
0.0005
0.00002
Swimmer-v2
0.000004
0.000009
0.0002
0.009
Walker2d-v2
0.0002
0.0009
0.0008
0.01
Table 4: Final hyperparameters for TD3.
actor learning rate
critic learning rate
target update т
Ant-v2
0.000001
0.0002
0.0003
HalfCheetah-v2
0.0003
0.0005
0.02
Humanoid-v2
0.0001
0.0001
0.0002
Reacher-v2
0.000001
0.00003
0.00003
Swimmer-v2 v2
0.0004
0.0002
0.01
Walker2d-v2
0.00006
0.00009
0.001
Table 5: Final hyperparameters for PPO.
learning rate
Ant-v2
0.0008
HalfCheetah-v2
0.0008
Humanoid-v2
0.0008
Reacher-v2
0.00002
Swimmer-v2 v2
0.0004
Walker2d-v2
0.0002
17

---

Published as a conference paper at ICLR 2020
Table 6: Final hyperparameters for DDPG.
actor learning rate
critic learning rate
target update т
Ant-v2
0.00003
0.0004
0.0002
HalfCheetah-v2
0.00006
0.0005
0.02
Humanoid-v2
0.00006
0.00009
0.01
Reacher-v2
0,00005
0.0005
0.005
Swimmer-v2
0.0005
0.0003
0.004
Walker2d-v2
0.0003
0.0004
0.03
Table Final hyperparameters for REINFORCE.
learning rate
episodes before
each train step
Ant-v2
0.00002
HalfCheetah-v2
0.0004
7
Humanoid-v2
0.0005
Reacher-v2
0.000004
6
Swimmer-v2
0.000005
3
Walker2d-v2
0.0001
6
Table 8: Hyperparameters for discrete control algorithms.
Training
Evaluation
decay schedule
Min. history
Target network
to start learning
update frequency
0.01
0.001
1,000,000 frames
80,000 frames
32,000 frames
PER-TASK METRIC RESULTS
Metric results are shown on per-task basis in Figs. 6 to for the OpenAl Gym MuJoCo tasks, and Figs. to 23 for the Atari environments. Note that because we are longer aggregating across tasks this analysis, we do not need to convert the metric values to rankings.
18

---

Published as a conference paper at ICLR 2020
00009
Aut
0000445
Wilker24
CO004S
Humonool
BLDOKESS
00007
CUOOOS
00063
000025
C.0023
00083
0000015
000013
0.0003
00001
000065
0.00003-
500000
3000000
1500000000000
300000 300000000000000000
300060 1000000000220000
300000 S00000C C000000O0000
6.607
0/0018
000054
000012
00004
00083
00662
000066
dtpp
oc0062
500000 200000 1500000000000
so0ces 100000510000iiooo
(a) Dispersion across Time. Better reliability indicated by less positive values. The x-axes indicate the number of environment steps.
HaleCheetah
Walker2d
Humanoid
-0.0011
000022
-001
-0/0022
-002
-00004
4000
-0.005
0000
-006
10006
-0.0055
-8.5
SAC
TDO ITo zorcruisr
SÃC 1DO IPO DorGzzINe
SàC 10I moo rercacivv
SAC Tho noo zororeier
Reacher
Swimmer
0.000
00000
+0.00 -0.0010
-1 660
-0,000
-0.004
SAC T03 ino mirerebrr
sic
1do nio
Dorcazzbe
(b) Short-term Risk across Time. Better reliability is indicated by more positive values.
Ant
HalfCheetah
Walker2d
Humanoid
60000
6.002-
04-
0,5
50002-
600
55C
Tho sairo seivee
SAC
nPo moro aeiorr
SiC
zorc stive
SÃC
nño
zorro rzive
Reacher
Swimmer
06-
04-
01
sic
mho soiro seinee
550
123
mño
(c) Long-term Risk across
Time. Better reliability is indicated by less positive values.
Figure 6: Across-time reliability metrics for continuous control RL algorithms tested OpenAl Gym environments, evaluated on per-environment basis.
19

---

Published as a conference paper at ICLR 2020
Ast
Waikerdd
68-
64
020
010
02-
01-
300000
1000000 200000e 2000000
xc0ess 1000000 1393000 zocecss
2000ce sceéecs inedoss 200000
100000 so00ce moodece 2000000
Beacher
Swtereer
12.
08.-
10.00
65.-
08-
Odps
minforce
05. -TR
08
500000
1300000
00-g
1000000
1580000 200000
(a) Dispersion across Runs. Better reliability is indicated by less positive values.
HolChootah
WalkerDd
Hamaneid
-0
-10
02-
04
200933 woloo 1091000 2000330
500000 isséeco 100000 2000003
300000 sodéces 1000000 2000000
500090 Iceéoso sseéece 2000000
500
-10
-33
-10
-1
reinforce
300000
2000000
1000000
2000000
500000 sseéece 1000000 2000000
(b) Risk across Runs. Better reliability is indicated by more positive values.
HellCheeiah
Walkerlld
Hamsnnit
10000
4090
2000
-1000
roedess soicee m003333
-2000
sookos sooion stedass nocécce
500ba3 sceéess yooicce an0daso
Reacher
140
Sentensrr
-00
ppo
-30
-35
relnforce
3000ee soedeso 250000 2090000
so9900 2000000 toed0s zooccce
(c) Median performance during training. Better performance indicated by more positive values.
Figure 7: Across-run reliability metrics and median performance for continuous control RL algorithms tested on OpenAl Gym environments, evaluated on a per-environment basis. The x-axes indicate the number of environment steps.
20

---

Published as a conference paper at ICLR 2020
Ant
HallCheetah
Walker2d
Humanold
5-
4-
3-
2-
SAC Hoo Diioo seivve
SAC
TE3 nho
DOPO reivve
DIPO
Reacher
Swimmer
00.
sáC 193
no mieo veiier
Dispersion Fixed-policy rollouts. Better reliability is indicated by less positive values.
Ant
HalfCheetah
Walker2d
Humanold
1.-
0.5 -0000e
04
00 200000
no zoro recser
-3066
bic The no DovOREIST
SÃC
mo
EDORO rezber
Reachor
Swimmer
0.00
0.53
0.55
DÃC
no zorro reier
sic
Tho
mo móvombir
(h) Risk on Fixed-policy rollouts. Better reliability is indicated by more positive values.
HalfCheetah
Walker2d
Humanold
10000
4000
5000
3000
4000
2000
2000
1000
5-
sniro
0-
Reacher
Swimmer
140-
100-
-2-
-12-
<10
nÃo
(c) Median performance Fixed-policy rollouts. Better performance is indicated by more positive values.
Figure 8: Reliability metrics and median performance on fixed-policy rollouts for continuous control
RL algorithms tested on OpenAl Gym environments, evaluated on per-environment basis.
21

---

Published as a conference paper at ICLR 2020
025
ArReed
012
0.3--
Ansiee
0.08
Asswalt
020
010
0.54
0.15
606.
0.50
0.22
0.22
666.
0.08
0.88
0.4
0.00
0.54
000-
0.05
0.2
0.02
6.00
0.00
0.00
t5o
082
0.14
Asterts
04-
Astermed
0.22
Alantts
0.54
BanicHisin
0.12
02.
0.52
0..
0.00
0.50
6.00
0.08
0.00
0.44
0.04
0.41
0.2-
0.2
100
130
0.20
DettleZone
DesmRider
0.0-
Derzerk
1.-
Dowling
0.53
0.10
0.10
0.14
0.12
0.58
0.10
0.10
0.4
0:00
0.83
6.-
0.6-
0:04
0.80
1/0
150
300
0.02
0.0g
50
559
0.50
Boxtng
Dreekont
0.10
Cornival
Cestipode
0.5
0.88
6.06
0.4
0.44
0.04
82-
0.82
0.03
0.02
0.00
-0
100
1io
200
0.00
310
130
0.0g
10
0.14
ChopperCenmand
0.1
CrazyClmbber
0.18
DemonoAtackk
0.3-
DeableDunk
630
0.10
0.08
0.22
0.00
0.4
0.55
0.5
0.4-
008
0.55
002-
0.0
0.02
0.00
0.50
100
300
ElnnterActton
0.0-
Enduro
0.00
PaitnngDebby
0.32
Freewsy
038-
0.08
0.30
014
0.00-
0.00
-0.1
012
0.03
6.00
008-
-02-
003
0.0-
0.02
001
300
350
200
50
100
150
200
30
300
150
200
50
100
150
Figure 9: Dispersion across Time for DON-variants tested on 60 Atari games, evaluated on a per- environment basis (page 1). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
22

---

Published as a conference paper at ICLR 2020
0.20
Provtbbee
Captec
0.00
Graviiar
Hero
0.83
0.15
010
0.5-
0.84
0
0.33
0:0-
0.15
0.2
0.33
0.0
0.03
0.00
100
0.00
000
0.00
100
ido
kollockey
Jamesbond
0.12
JoarneySscape
0.14
Sargaroo
0.15
0.0-
0.10
0.00
610
0.10
0.00
0.00
0.00
0.04
0.00
634-
0.0-
0.02
0.22
100
110
200
0.00
7
0.00
0.00
30
100
100
309
Kndl
KanghuMasterr
0.05
Mertertunoorroagg
0.5
McPecmoon
0.12
0.04
0.03
0.10
0.82
0.57
-3.02
0.5
0.54
0.02
50
100
130
0.03
Phorotz
Pitfal
0.50
Pong
0.55
0.10
0.04
0.02
30
110
200
120
130
30
150
100
120
0:30
Pooyaz
08-
PrivataBye
0.44
Oteet
Ioverruid
0.25
07-
06.
0.53
0.00
0.20
0.33
0.6
0.8-
0.04
0.10
02-
084
0.55
0.1
682
0.0
50
100
30
100
150
200
300
150
203
50
100
150
0.07
Robetank
028-
Sking
rarbow
0.00
0.05
0.03
0.1
130
30
100
250
200
350
130
303
30
100
150
350
Figure 10: Dispersion across Time for DQN-variants tested 60 Atari games, evaluated per-environment basis (page 2). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
23

---

Published as a conference paper at ICLR 2020
Sclans
Speceiaraders
StarGhnnner
023
030
0.00
0.0
0.3
0.03
015
300
150
300
300
150
300
100
150
UyNDowa
M
020
0.0
010
012
000
000
150
300
50
150
50
500
150
100
150
VidesPaabell
WaandOWWor
0.2
YarsRevenge
Zacons
(0)
0.14
0.50
0.88
0.05
0.00
0.09
0.4
0.10
0.04
0.02
0.2
0.0
0.00
0.00
8.50
160
Figure 11: Dispersion across Time for DQN-variants tested 60 Atari games, evaluated per-environment basis (page 3). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
24

---

Published as a conference paper at ICLR 2020
AirRaid
Alien
Amidar
Assault
-0.07
-0.14
-0.00
-0.10
-0.00
-0.8
-0.8
-0.12
-0.32
-400
-0.10
-0.1d
-0.44
-8.2
-0.1
+0.00
10.0-
-424
-0.6
-0.12
-0.8
-0.8
-2.8
DON
-0.3
Riw
KIN
-920-
-020-
KIN
cht
DON
Asterix
Asterolds
Atlantis
BankHelst
-0.0
0-
-004
-6.10
-0.13
00.00
-0.12
-10-
-0.00
-0.13
-0.10
-6.44
-6.55
-0.32
-0.8
-0.4
DON
KIN
Co1
M0N
DUN
KJN
BattleZone
BeamRider
Berzerk
Bowling
-0.00
-0.00
-007
-0.10
-000
-0.00
-0.5
-017
-020
-0.1-
-0.11
-220
-0.19
-0.13
cit DON
Boxing
Breakout
Carnival
Contipedo
-001
0010
-2015
-0.12
-0.4
-014
-1.00
-0.3
-0.0
-0.10
122
-007
3.030
-220-
KIN
C31
DON
DON
NBW
10N
DON
NBW
Cb1 Dij
ChopperCommand
CrazyClimber
DemonAttack
DoubleDunk
-007
3.00
00.3
-0.5
-0.00
-111
-0.9
-020
4220
-013
4029
DOR
xie
ElevatorAction
Faduro
PishingDerby
Freeway
60
-2.00
-1.00-
-03
-2:00
-0.22
-1.000
-2000-
-13
-3.10
-23-
2236
DON
H3W
DQN
DON
N0W
K0N
D0N
Figure 12: Short-term Risk across Time for DQN-variants tested on 60 Atari games, evaluated a envirometnn basis (page 1). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
25

---

Published as a conference paper at ICLR 2020
Prostbite
Copher
Gravilar
Hero
30-
-082
-030
-0.12
-01
-0.3-
-020
-0.33
-82
-03
0.15
-04-
-0.0-
-0.0
-03
-00-
0087
BBW
12M
DON
70W
K2N
DON
10W
KIN
DON
cb
DON
Hockoy
Jameshond
Journey
Kangaroo
-0.00
-0.0
-0.00
0.00
00.00
9.00
-0.12
0.20
4.07
-0.00
-0.4
-0.30
00.5
-0.00
-0.16
DON
DON
12N
DON
PDW
Krull
Kung FuMaster
MontezumnRReenge
MsPamman
-0.6
-0.0
-0.44
-0.1
-0.13
Thisaaee
Phoenix
Pitfall
00.4
-0.00
-22
00.00
00.00
-14
0.30
0.20
-034
-2.30
N8W
DON
HWW
DON
KIN
DON
Hiw
KIN
cia
DON
Pooyan
PrivateEye
Qbort
Fiverraid
-0.00
-0.9
-0.55
-0.11
-0.10
10.00
-0.33
-0.12
-8.07
-0.4
-0.33
-0.00
-0.15
-0.17
-2.
-0.16
DON
DUN
RBW
DON
Raw
DON
RoadRunner
Robotank
Seaquest
Sking
-0.02
-2.00-
-2.44
-0.23
-0.35-
-0.0
-2-
-2.00-
-0.04
2.05
-0.10
-6
-0.12
-0.3
-0.70-
-0.00
-3.073
-0.0
+22
-3.00
-018
-34
-0.77
-2.033
.0.8
-0.00
031
DON
R0W
DON
DON
HWW
DON
Figure 13: Short-term Risk across Time for DQN-variants tested on 60 Atari games, evaluated on a per-environment basis (page 2). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
26

---

Published as a conference paper at ICLR 2020
Solaris
Spacelavaders
StarGunner
Tennis
-604
-0.05
-0.10
-0.0
-0.00
-0.15
-015
-0.20
110
-0.35
-000
-0.30
-0.3
200
-0.10
101
12N
DON
DON
30W
DQN
KIN
DON
TimePllot
Tutankham
UpNDoww
Venture
00
-0.12
00-
-0.44
-05
-0.16
-033
-0.88
-920-
-028
-0.-
-022
-023
1816
12N
C31
DON
nBW
DON
38W
320
DON
KIN
DON
VideoPinball
WizardOfWor
YarsRevenge
Zaxxxon
-0.33
0.12
-002-
-01
-0.5
-0.16
-006-
-02-
-0.3
-0.6
-0.35
-0.00
-0.10
-05
-045
-024
-011
10W
DON
10W
1ON
DON
DON
12w
DON
Figure 14: Short-term Risk across Time for DQN-variants tested on 60 Atari games, evaluated on per-environment basis (page 3). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
27

---

Published as a conference paper at ICLR 2020
AirRald
Allen
Amidar
Assault
9.2-
002-
0.2-
3.00
0.0-
0.00
0.00
-3.02
020
-0.00-
-0.02
-3.04-
-0.04
-0.04
-0.04
-3.00
DON
DCN
-3.00
DQN
-0.06
Asterix
Asteroids
Atlantis
BankHeist
000-
0.5
0.000
0.5
0.2
0.15-
0.4
0.00-
4.02
02-
-.00
0.000
-3.04-
-0.04
-1.00
ch
chi
DCN
-0.00
ch
DON
K2N ch
ndot
BattleZone
BeamRider
Berzerk
Bowling
0.6
0.00
00033-
0.44
0.01
00030-
0.0025 -
6.2-
0.02
0.0200
6.00
0.00
0.0155
02-
-0.02
+0.00-
-0.04
-0.05
0.000-
-0.00
-0.00
DIN
DON
025- KUN
DON
Boxing
Breakout
Carnival
Centipede
0.22
0.22
0.00
0.00
0.30
0.10
0.5-
000-
10N
ch
DON
cis
DiN
cha
DQN
ChopperCommand
CrazyCllmber
DemonAttack
DoubleDunk
8.0
035-
0.00
8.00
0.00
-6.00
-0.02-
-0.02
-3
-6.05
-6.05
04.0
12W
d1
DON
10W
chi
DQN
N0W
ch
DOX
1/N4
da DQS
ElevatorAction
Enduro
FishingDerby
Freeway
0.012
0.55
0.00
0.00
0.0-
0.4-
0.003 -
0.04
-0.22-
-0.02
8002-
-005-
-0.04
-3
0200
+0.05
0..00
Figure 15: Long-term Risk across Time for DQN-variants tested on 60 Atari games, evaluated on a per-environment basis (page 1). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
28

---

Published as a conference paper at ICLR 2020
Frostbite
Gopher
Gravitar
Hero
0.025-
200-
6.0-
0.5-
0,020-
0.4-
6.0-
0.1-
0.0033
012-
6.02
002-
00010-
200
6.00
0.00
-0.02
-0.03
-0 02
-0.04
-0.04
-0.04
0.000 +
-0.66
-0.06
-0.00
10TH
12M
BBW
10M
DOX
101
DQN
101
KM
IceHockey
Jamoshond
JourneyEscape
Kangaroo
0.00
2.0
008-
0.94
2.4
004-
0.82
2.2
0.000
0.2-
0.00
2.00
0.00
-0.02
-0.02
0.00-
-0.04
-0.04
-0.0-
DON
DON
chi Dóx
Krull
KungPuMaster
MontezumaReveagee
MsPacman
0.00
0.00
0.44
0.1-
0.01
0.01
640
0.82
0.52
0.02-
640
0.00
0.00
0.00
001
-0.02-
-0.02
-0.00
-0.00
-3.00
-0.04
0.00
23.05
-0.00
10W
RIM
C3
DQN
35TV
XIN
C31
DON
XIM
C31
DQN
3000
Ch1
DON
NameThisGame
Phoenix
Pitsfll
Pong
0.00 +
0.044
0.00
0.64
0.02-
0.01
0.02
0.20-
0.10-
0.02
6.00
0015
0.00
-0.82
0.00-
0.04-
-0.02
0053-
-0.64
-0.04
0.00-
0.00
1010
I0N
chs
DDN
RMW
KIN
chi
DŌN
R0W
I0N
chs
DDN
R0W
300
cha DON
Pooyan
PriveteEye
Qbart
Riverraid
0.00
0.00
0.07-
0.-
0.04-
0.04
0.02
0.62
0.0
02-
011
10WM
0.5- cl
DŌN
DON
C3
DON
di
ndw
RoadRunner
Robotank
Seaquest
Skinng
0.50
40-
35-
30-
1.5
1.0-
-0.04
121
10N
Cl
DON
101M
DON
C51
DXN
ch
Figure 16: Long-term Risk across Time for DQN-variants tested on 60 Atari games, evaluated on a per-environment basis (page 2). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
29

---

Published as a conference paper at ICLR 2020
Solarts
Spacelnvaders
StarGunner
Tonnts
005
0.1-
001-
-0.02
-0.00
-0.04
-0.04
-0.04
0+
-2.00
-0.06
T0W
10M
DON
101
100
ch1
DON
2BW
K2H
c31
DQN
107V
100
chs
DQN
TimoPilot
Tutankham
UpNDown
Venture
0.023
0.0-
023
0.020
0.4
042
0010
00010
0.0-
0005
-0.00
-0.04
00030
0000
10W
10N
18W
KIN
10W
KNN
C51
DXN
K2M
DON
VideoPinball
WizardOFWor
YarsRevange
Zaxxon
0.08
0.0
0.05
0.033
0.64
0.4
0.61
082
0.02
0.22
0.15
0.0
0.00
0.10
-0.00
-0.02-
0.033
-0.06
-0.04
-0.02
0.00
BBW
120
DOX
10W
10N
C31
DCN
181
30N
C31
DQN
12M
DDN
Figure 17: Long-term Risk across Time for DQN-variants tested on 60 Atari games, evaluated on a per-environment basis (page 3). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
30

---

Published as a conference paper at ICLR 2020
AirRaid
Alien
Amidar
Ascoult
09
04
DON
KIN
DON
RBW
cb1
DON
AsterTx
Asterolds
Atlantis
BankHetst
25
16-
000
0.35
20
12.
004
0.23
15
0.55
06-
10
0.19
05
60
600
000
BON
BattleZone
BeamRlder
Berzerk
Bowling
0.43
09-
07-
0.40
08-
66-
0.10 05-
0.35
66.
BBW
DON
RBW
Ch
DKN
RBW
DCN
HBW
RIN Centipede
DCN
Boxing
Breakout
Carnival
20-
0.35
13.
0.3
14.
12-
0.3
10-
0.0
01
08
ass
60-
66
a.o
naW
1004
cbu
DON
K0N
DCN
DBW
DCN
ndw
10.0
DON
ChopperCommand
CrazyClimber
DemonAttack
DoubleDunk
13-
0.40
10-
30-
12-
25.
1..
0.30
12
20.
1.
1.
0.2
67
015
06-
05.
00.
010
-05-
64.
085
n0-
-10
RBW
DON
DON
Elevatorcction
Enduro
Pishing
Freeway
8.40
6005
10-
6400
-2-
8815
0.5
-0.02
6465
-5-
8000
Figure 18: Dispersion across Fixed-policy Rollouts for DQN-variants tested on 60 Atari games, evaluated on per-environment basis (page 1). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
31

---

Published as a conference paper at ICLR 2020
Prosthite
Gopher
Gravitar
Hero
09-
0.00
38-
0.3
06-
0.20
65-
1.-
0.15
05
0.0
02-
005-
00
IOM
DQN
nBW
12N
DON
DQN
10W
KMN
DDN
IcoHockey
Jameshond
Journey
Kangaroo
00010
-0.5
0.8
0.003
-14-
00000
0.4
-2.0
-0 0005
08
-0.0010
BQN
HaW
DDN
HWW
KUN
DON
DON
Krull
KungttMaster
MontozumaRevengee
MsPacman
0.00
0.55
0.50
0.52
0.6
0.00
05
0.40
0.1
0.35
-0.02
04
0.30
63-
02
-0.04
02.
020
-0.00
011
100
DON
100
c31
DQN
I0M
C51
DON
TSW
X0N
DON
Tamemee
Phoenix
Pitfall
Pong
035-
10-
0.86
0.8
0.54
0.30
0.6
0.52
0.35
0.50
0.00
0.10
DON
c3
Pooyan
PrivateRye
Qbert
Riverrald
065
30
10-
0.50
025
85-
020-
60-
DON
css
DOx
NON
cba
DOX
new
DON
RoadRunner
Robotank
Seaquest
Skiing
0..
0.35
12
60-
18-
0.6 025
-02-
-0.4
6.
0.4
013
64-
010
-0.8
085-
-1.0
0.0-
Figure 19: Dispersion across Fixed-policy Rollouts for tested on 60 Atari games, evaluated on per-environment basis (page 2). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
32

---

Published as a conference paper at ICLR 2020
Solaris
Spacelavaders
StarGunner
Tennis
67-
15
0.00
10
63.
0.60
221V
DDN
RBW
cb1
RBW
DQN
NBW
DQN
TimoPllot
Tutankham
UpNDown
Venture
18
10-
02
08
60-
0.0
04
DON
PBW
cb1
DUN
DDN
HBW
DON
VideoPinball
WizardOWWor
YarsRevenge
Zaxxon
33-
23
14-
05-
25-
1..
07
1.3
13-
03
60
02
Figure 20: Dispersion across Fixed-policy Rollouts for DQN-variants tested on 60 Atari games, evaluated on a per-environment basis (page 3). Better reliability is indicated by less positive values. The x-axes indicate millions of Atari frames.
33

---

Published as a conference paper at ICLR 2020
AirRaid
Allen
Amidar
Assoult
65.
0:45
06
64-
0.40
60-
03
0.35
04
02
0.30
03
033
01
023
02
00
DQN
020
DON
01
chs
011
DÓN
Astertx
Asteroids
Atlantis
BankHeist
07
0.30
0:25
0.5
0.10
003
chs
DON
0.00
BattleZone
BeamRider
Berzerk
Bowling
6.40
0:30
635
620
03
0.35
DON
DON
chi
DON
Boxing
Breakout
Cariiall
Contipode
10-
6:50
1.00
0.00
0.50
8:40
0.00
0:58
0:52
630
02
625
0.80
028
0.00
00
075
020
-02
6:5
0.7
024
ChopperCommandd
CrazyCliber
DemonAttack
DoubleDunk
10.
39-
03-
38-
36-
35-
01-
00-
-38-
DevatorAction
Enduro
Feshing
Freeway
10-
L0-
5-
1000
38.
30.
1750
37-
055
2575
DON
10%
KTW
R0N
ION
Figure 21: Risk across Fixed-policy Rollouts for DON-variants tested on 60 Atari games, evaluated on per-environment basis (page 1). Better reliability is indicated by more positive values. The x-axes indicate millions of Atari frames.
34

---

Published as a conference paper at ICLR 2020
Frostbite
Gopher
Gravitar
Hero
07
03
02-
02
DON
3BW
KIN
chd
DUN
ch
DON
NBW
DGN
tootockyy
Jamesbond
Journey Escape
Kangaroo
67
05-
00
06-
00-
67
05-
-35
06-
04-
-10-
03
03-
-15
04
-12
02-
-20- -
-16
10W
-25
chs
Kungteuasste
MontezumaRevenge
MsPamman
0.0
0.3
043
60-
0.5
100
DON
101V
XIN
Ch
DON
IOM
C51
DON
DBW
ION
DON
NameThisGame
Phoenix
Pitfall
Pong
0.00
0.04
06-
87-
05
24
DON
T0T
10W
DQN
12W
I0M
C51
DON
RBW
KNN
DON
Fooyan
PrivateEye
Qbert
Riverrald
835
0.70
0.65
830
0.00
0.5
545
030
0.45
340
-10
01
0.40
-12
033
335 -
14-
0.30
DON
RoadRunner
Robotank
Seaquest
Skiing
0.5 05
02-
60-
chi
DŌN
TON
KJN
DÓN
c51
Figure 22: Risk across Fixed-policy Rollouts for DQN-variants tested 60 Atari games, evaluated on per-environment basis (page 2). Better reliability is indicated by more positive values. The x-axes indicate millions of Atari frames.
35

---

Published as a conference paper at ICLR 2020
Solarts
Spacelnvaders
StarGunner
Tonnts
07-
0.5-
035
06-
040
0.33
030 0.3
0.25
020
0.60
005-
01
0.22
DCN
DON
DOo
DON
TimePilot
Tutankbam
UpNDown
Venture
0.50 01
0
DON
VideoPinball
WizardotWor
YarsRovenge
Zaxxom
07-
07-
06
0.5
04
DON
KIN
DUN
DON
DÓN
Figure 23: Risk across Fixed-policy Rollouts for DQN-variants tested on 60 Atari games, evaluated on per-environment basis (page 3). Better reliability is indicated by more positive values. The x-axes indicate millions of Atari frames.
36


Reviews:
[{"review_id": "SkxBZirTFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary*\n\nAuthors proposed a variety of metrics to measure the reliability of an RL algorithm. Mainly looking at Dispersion and Risk across time and runs while learning, and also in the evaluation phase. \nAuthors have further proposed ranking and also confidence intervals based on bootstrapped samples. They also compared the famous continuous control and discrete actions algorithms on Atari and OpenAI Gym on the metrics they defined.\n\n*Decision*\n\nI believe the paper is discussing a very important issue, and some possible solutions to it, even if not perfect it's an important step toward paying more attention to maybe similar metrics. I am in favor of the paper in general,  but I have some concerns.\n\n1. My main concern is that why authors think that community will adopt these metrics and report them? I like how authors have proposed different metrics, but having one or two easy to compute metric is much more likely to be adopted, than 6 different metrics, which I’m not sure how easy it is to use the python package? It’s of main importance, because if community don’t use these metrics in the future, the contribution of the paper is minimal. \n\n2. There is no question of the importance of reliability of RL algorithms, but we need to be careful that RL algorithms are not optimizing for metics like CVaR, so maybe a better learning algorithm (in the sense of expectation learning) might not have better reliability metrics because it is not the main objective.  \nSo following this, how would authors think their metrics can be used to design a more reliable algorithms? For example there is good literature on CVaR learning for safe policies. Do you think there exists a proxy for metrics you introduced that can be used to for the objective of the optimization?\n\n3. Another main concern is the effect of exploration strategy: All these metrics can be highly affected  by different exploration strategy in different environments. For example if an environment has a chain like structure, then given the exploration strategy you may have an extremely high CVaR or IQR. How do authors think they can strip off this effect? (Running all algorithm with the same exploration strategy is not sufficient, since the interplay of learning algorithm and exploration may be important)\n\n4. Generalizability: How do authors think these metrics are generalizable. For example if algorithm A has better metrics than algorithm B on open AI Gym task for continuous control, how much we expect the same ranking applies while learning on a new environment. I am asking this, because to me, some of these metrics are very environment dependent, and being reliable in some environments may not imply reliability in other environments.\n\n\n*Note*:\nCode and modularity of it: The main contribution of this paper will be shown when other researchers start using it and report the metric, if the code is hard to use, the contribution of the paper is hard to be significant. \n\n==== Post Rebuttal ====\nThanks for the responses authors posted, I think there is a good chance that the community will benefit from this experimental metrics in the future, so I increase my rating to accept.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkloOhgEKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors study an important problem in the area of reinforcement learning (RL). Specifically, the authors focus on how to evaluate the reliability of RL algorithms, in particular of the deep RL algorithms. The paper is well motivated by providing convincing justification of evaluating the RL algorithms properly. In particular, the authors define seven specific evaluation metrics, including 'Dispersion across Time (DT): IQR across Time', 'Short-term Risk across Time (SRT): CVaR on Differences', 'Long-term Risk across Time (LRT): CVaR on Drawdown', 'Dispersion across Runs (DR): IQR across Runs', 'Risk across Runs (RR): CVaR across Runs', 'Dispersion across Fixed-Policy Rollouts (DF): IQR across Rollouts' and 'Risk across Fixed-Policy Rollouts (RF): CVaR across Rollouts', from a two-dimension analysis shown in Table 1.\n\nMoreover, the authors apply the proposed evaluation metrics to some typical RL algorithms and environments, and provide some insightful discussions and analysis.\n\nOverall, the paper is well presented though it is somehow different from a typical technical paper.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJxox6QRYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provides a unified way to provide robust statistics in evaluating RL algorithms in experimental research. Though I don't believe the metrics are particularly novel, I believe this work would be useful to the broader community and was evaluated on a number of environments. I do have a few concerns, however, about experimental performance per environment being omitted from both the main paper and the appendix.\n\nComments: \n\n+ I think this is a valuable work and the ideas/metrics are useful, though I'm not sure I would call them novel (CVar and the like have been seen before).  I think the value comes in the unification of the metrics to give more robust pictures of algorithmic performance. \n+ The details of all of these evaluations and individual performance should be provided in the appendix, however, it seems only MuJoco curves were included. Moreover, it says that a blackbox optimizer was used to find hyperparameters, but these hyperparameters were not provided in the appendix or anywhere else as far as I can tell. I think it's important for a paper which recommends evaluation methodology in particular to be more explicit regarding all details within the appendix. I hope to see additional details in future revisions -- including per-environment performance.\n+ I believe clustering results across environments can be potentially misleading. Say that we have an environment where the algorithm always fails but is very consistent and an environment where it excels. These are blended together in the current Figures. While it requires more space, I believe it is important to separate these two. I am concerned that a recommendation paper like this one will set a precedent for only including the combined metrics of algorithmic performance across environments, masking effects.  I would suggest splitting out results per environment as well and pointing out particular cross-environment phenomena. \n\nThere is a missing discussion of prior work on statistical testing of RL evaluation:\n+ Colas, Cédric, Olivier Sigaud, and Pierre-Yves Oudeyer. \"A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms.\" arXiv preprint arXiv:1904.06979 (2019).\n+ Colas, Cédric, Olivier Sigaud, and Pierre-Yves Oudeyer. \"How many random seeds? statistical power analysis in deep reinforcement learning experiments.\" arXiv preprint arXiv:1806.08295 (2018).\n\nEDIT: Score boosted after significant updates to the paper.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

This paper provides a unified way to provide robust statistics in evaluating the reliability of RL algorithms, especially deep RL algorithms. Though the metrics are not particularly novel, the investigation should be useful to the broader community as it compares seven specific evaluation metrics, including 'Dispersion across Time (DT): IQR across Time', 'Short-term Risk across Time (SRT): CVaR on Differences', 'Long-term Risk across Time (LRT): CVaR on Drawdown', 'Dispersion across Runs (DR): IQR across Runs', 'Risk across Runs (RR): CVaR across Runs', 'Dispersion across Fixed-Policy Rollouts (DF): IQR across Rollouts' and 'Risk across Fixed-Policy Rollouts (RF): CVaR across Rollouts'. The paper further proposed ranking and also confidence intervals based on bootstrapped samples, and compared against continuous control and discrete actions algorithms on Atari and OpenAI Gym.

--

Discussion:

The reviews clearly agree on accepting the paper, with a weak accept coming from a reviewer who does not know much about this subarea. Comments are mostly just directed at clarifications and completeness of description, which the authors have addressed.

--

Recommendation and justification:

This paper should be accepted due to its useful contributions toward doing a better job of measuring performance of RL.
Decision: Accept (Spotlight)