Measuring the Reliability of Reinforcement Learning Algorithms

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms , especially Deep RL algorithms , tend to be highly variable in performance and considerably sensitive to a range of different factors , including implementation details , hyper-parameters , choice of environments , and even random seeds ( Henderson et al. , 2017 ) . This variability hinders reproducible research , and can be costly or even dangerous for real-world applications . Furthermore , it impedes scientific progress in the field when practitioners can not reliably evaluate or predict the performance of any particular algorithm , compare different algorithms , or even compare different implementations of the same algorithm . Recently , Henderson et al . ( 2017 ) has performed a detailed analysis of reliability for several policy gradient algorithms , while Duan et al . ( 2016 ) has benchmarked average performance of different continuous-control algorithms . In other related work , Colas et al . ( 2018 ) have provided a detailed analysis on power analyses for mean performance in RL , and Colas et al . ( 2019 ) provide a comprehensive primer on statistical testing for mean and median performance in RL . In this work , we aim to devise a set of metrics that measure reliability of RL algorithms . Our analysis distinguishes between several typical modes to evaluate RL performance : `` evaluation during training '' , which is computed over the course of training , vs. `` evaluation after learning '' , which is evaluated on a fixed policy after it has been trained . These metrics are also designed to measure different aspects of reliability , e.g . reproducibility ( variability across training runs and variability across rollouts of a fixed policy ) or stability ( variability within training runs ) . Additionally , the metrics capture multiple aspects of variability – dispersion ( the width of a distribution ) , and risk ( the heaviness and extremity of the lower tail of a distribution ) . Standardized measures of reliability can benefit the field of RL by allowing RL practitioners to compare algorithms in a rigorous and consistent way . This in turn allows the field to measure ∗Work done as part of the Google AI Residency 1https : //github.com/google-research/rl-reliability-metrics progress , and also informs the selection of algorithms for both research and production environments . By measuring various aspects of reliability , we can also identify particular strengths and weaknesses of algorithms , allowing users to pinpoint specific areas of improvement . In this paper , in addition to describing these reliability metrics , we also present practical recommendations for statistical tests to compare metric results and how to report the results more generally . As examples , we apply these metrics to a set of algorithms and environments ( discrete and continuous , off-policy and on-policy ) . We have released the code used in this paper as an open-source Python package to ease the adoption of these metrics and their complementary statistics . 2 RELIABILITY METRICS . We target three different axes of variability , and two different measures of variability along each axis . We denote each of these by a letter , and each metric as a combination of an axis + a measure , e.g . `` DR '' for `` Dispersion Across Runs '' . See Table 1 for a summary . Please see Appendix A for more detailed definitions of the terms used here . 2.1 AXES OF VARIABILITY . Our metrics target the following three axes of variability . The first two capture reliability `` during training '' , while the last captures reliability of a fixed policy `` after learning '' . During training : Across Time ( T ) In the setting of evaluation during training , one desirable property for an RL algorithm is to be stable `` across time '' within each training run . In general , smooth monotonic improvement is preferable to noisy fluctuations around a positive trend , or unpredictable swings in performance . This type of stability is important for several reasons . During learning , especially when deployed for real applications , it can be costly or even dangerous for an algorithm to have unpredictable levels of performance . Even in cases where bouts of poor performance do not directly cause harm , e.g . if training in simulation , high instability implies that algorithms have to be check-pointed and evaluated more frequently in order to catch the peak performance of the algorithm , which can be expensive . Furthermore , while training , it can be a waste of computational resources to train an unstable algorithm that tends to forget previously learned behaviors . During training : Across Runs ( R ) During training , RL algorithms should have easily and consistently reproducible performances across multiple training runs . Depending on the components that we allow to vary across training runs , this variability can encapsulate the algorithm ’ s sensitivity to a variety of factors , such as : random seed and initialization of the optimization , random seed and initialization of the environment , implementation details , and hyper-parameter settings . Depending on the goals of the analysis , these factors can be held constant or allowed to vary , in order to disentangle the contribution of each factor to variability in training performance . High variability on any of these dimensions leads to unpredictable performance , and also requires a large search in order to find a model with good performance . After learning : Across rollouts of a fixed policy ( F ) When evaluating a fixed policy , a natural concern is the variability in performance across multiple rollouts of that fixed policy . Each rollout may be specified e.g . in terms of a number of actions , environment steps , or episodes . Generally , this metric measures sensitivity to both stochasticity from the environment and stochasticity from the training procedure ( the optimization ) . Practitioners may sometimes wish to keep one or the other constant if it is important to disentangle the two factors ( e.g . holding constant the random seed of the environment while allowing the random seed controlling optimization to vary across rollouts ) . 2.2 MEASURES OF VARIABILITY . For each axis of variability , we have two kinds of measures : dispersion and risk . Dispersion Dispersion is the width of the distribution . To measure dispersion , we use `` robust statistics '' such as the Inter-quartile range ( IQR ) ( i.e . the difference between the 75th and 25th percentiles ) and the Median absolute deviation from the median ( MAD ) , which are more robust statistics and don ’ t require assuming normality of the distributions . 2 We prefer to use IQR over MAD , because it is more appropriate for asymmetric distributions ( Rousseeuw & Croux , 1993 ) . Risk In many cases , we are concerned about the worst-case scenarios . Therefore , we define risk as the heaviness and extent of the lower tail of the distribution . This is complementary to measures of dispersion like IQR , which cuts off the tails of the distribution . To measure risk , we use the Conditional Value at Risk ( CVaR ) , also known as “ expected shortfall '' . CVaR measures the expected loss in the worst-case scenarios , defined by some quantile α . It is computed as the expected value in the left-most tail of a distribution ( Acerbi & Tasche , 2002 ) . We use the following definition for the CVaR of a random variable X for a given quantile α : CVaRα ( X ) = E [ X|X ≤ V aRα ( X ) ] ( 1 ) where α ∈ ( 0 , 1 ) and the V aRα ( Value at Risk ) is just the α-quantile of the distribution of X . Originally developed in finance , CVaR has also seen recent adoption in Safe RL as an additional component of the objective function by applying it to the cumulative returns within an episode , e.g . Bäuerle & Ott ( 2011 ) ; Chow & Ghavamzadeh ( 2014 ) ; Tamar et al . ( 2015 ) . In this work , we apply CVaR to the dimensions of reliability described in Section 2.1 . 2.3 DESIDERATA . In designing our metrics and statistical tests , we required that they fulfill the following criteria : • A minimal number of configuration parameters – to facilitate standardization as well as to minimize “ researcher degrees of freedom '' ( where flexibility may allow users to tune settings to produce more favorable results , leading to an inflated rate of false positives ) ( Simmons et al. , 2011 ) . • Robust statistics , when possible . Robust statistics are less sensitive to outliers and have more reliable performance for a wider range of distributions . Robust statistics are especially 2Note that our aim here is to measure the variability of the distribution , rather than to characterize the uncertainty in estimating a statistical parameter of that distribution . Therefore , confidence intervals and other similar methods are not suitable for the aim of measuring dispersion . important when applied to training performance , which tends to be highly non-Gaussian , making metrics such as variance and standard deviation inappropriate . For example , training performance is often bi-modal , with a concentration of points near the starting level and another concentration at the level of asymptotic performance . • Invariance to sampling frequency – results should not be biased by the frequency at which an algorithm was evaluated during training . See Section 2.5 for further discussion . • Enable meaningful statistical comparisons on the metrics , while making minimal assumptions about the distribution of the results . We thus designed statistical procedures that are non-parametric ( Section 4 ) . 2.4 METRIC DEFINITIONS . Dispersion across Time ( DT ) : IQR across Time To measure dispersion across time ( DT ) , we wished to isolate higher-frequency variability , rather than capturing longer-term trends . We did not want our metrics to be influenced by positive trends of improvement during training , which are in fact desirable sources of variation in the training performance . Therefore , we apply detrending before computing dispersion metrics . For detrending , we used differencing ( i.e . yt′ = yt − yt−1 ) .3 The final measure consisted of inter-quartile range ( IQR ) within a sliding window along the detrended training curve . Short-term Risk across Time ( SRT ) : CVaR on Differences For this measure , we wish to measure the most extreme short-term drop over time . To do this , we apply CVaR to the changes in performance from one evaluation point to the next . I.e. , in Eq . 1 , X represents the differences from one evaluation time-point to the next . We first compute the time-point to time-point differences on each training run . These differences are normalized by the distance between time-points , to ensure invariance to evaluation frequency ( see Section 2.5 ) . Then , we obtain the distribution of these differences , and find the α-quantile . Finally , we compute the expected value of the distribution below the α-quantile . This gives us the worst-case expected drop in performance during training , from one point of evaluation to the next . 3Please see Appendix B for a more detailed discussion of different types of detrending , and the rationale for choosing differencing here . Long-term Risk across Time ( LRT ) : CVaR on Drawdown For this measure , we would also like to be able to capture whether an algorithm has the potential to lose a lot of performance relative to its peak , even if on a longer timescale , e.g . over an accumulation of small drops . For this measure , we apply CVaR to the Drawdown . The Drawdown at time T is the drop in performance relative to the highest peak so far , and is another measure borrowed from economics ( Chekhlov et al. , 2005 ) . I.e . DrawdownT = RT −maxt < =T Rt . Like the SRT metric , the LRT can capture unusually large short-term drops in performance , but can also capture unusually large drops that occur over longer timescales . Dispersion across Runs ( DR ) : IQR across Runs Unlike the rest of the metrics described here , the dispersion across training runs has previously been used to characterize performance ( e.g . Duan et al . ( 2016 ) ; Islam et al . ( 2017 ) ; Bellemare et al . ( 2017 ) ; Fortunato et al . ( 2017 ) ; Nagarajan et al . ( 2018 ) ) . This is usually measured by taking the variance or standard deviation across training runs at a set of evaluation points . We build on the existing practice by recommending first performing low-pass filtering of the training data , to filter out high-frequency variability within runs ( this is instead measured using Dispersion across Time , DT ) . We also replace variance or standard deviation with robust statistics like IQR . Risk across Runs ( RR ) : CVaR across Runs In order to measure Risk across Runs ( RR ) , we apply CVaR to the final performance of all the training runs . This gives a measure of the expected performance of the worst runs . Dispersion across Fixed-Policy Rollouts ( DF ) : IQR across Rollouts When evaluating a fixed policy , we are interested in variability in performance when the same policy is rolled out multiple times . To compute this metric , we simply compute the IQR on the performance of the rollouts . Risk across Fixed-Policy Rollouts ( RF ) : CVaR across Rollouts This metric is similar to DF , except that we apply CVaR on the rollout performances .

Summary: In this paper, the authors study an important problem in the area of reinforcement learning (RL). Specifically, the authors focus on how to evaluate the reliability of RL algorithms, in particular of the deep RL algorithms. The paper is well motivated by providing convincing justification of evaluating the RL algorithms properly. In particular, the authors define seven specific evaluation metrics, including 'Dispersion across Time (DT): IQR across Time', 'Short-term Risk across Time (SRT): CVaR on Differences', 'Long-term Risk across Time (LRT): CVaR on Drawdown', 'Dispersion across Runs (DR): IQR across Runs', 'Risk across Runs (RR): CVaR across Runs', 'Dispersion across Fixed-Policy Rollouts (DF): IQR across Rollouts' and 'Risk across Fixed-Policy Rollouts (RF): CVaR across Rollouts', from a two-dimension analysis shown in Table 1.