Package: rliable
rliableis an open-source Python library for reliable evaluation, even with ahandful
of runs, on reinforcement learning and machine learnings benchmarks.DesideratumCurrent evaluation approachOur RecommendationUncertainty in aggregate performancePoint estimates:Ignore statistical uncertaintyHinderresults reproducibilityInterval estimates usingstratified bootstrap confidence intervals(CIs)Performance variability across tasks and runsTables with task mean scores:Overwhelming beyond a few tasksStandard deviations frequently omittedIncomplete picture for multimodal and heavy-tailed distributionsScore distributions(performance profiles):Show tail distribution of scores on combined runs across tasksAllow qualitative comparisonsEasily read any score percentileAggregate metrics for summarizing benchmark performanceMean:Often dominated by performance on outlier tasksMedian:Statistically inefficient (requires a large number of runs to claim improvements)Poor indicator of overall performance: 0 scores on nearly half the tasks doesn't change itInterquartile Mean (IQM)across all runs:Performance on middle 50% of combined runsRobust to outlier scores but more statistically efficient than medianTo show other aspects of performance gains, reportProbability of improvementandOptimality gaprliableprovides support for:Stratified Bootstrap Confidence Intervals (CIs)Performance Profiles (with plotting functions)Aggregate metricsInterquartile Mean (IQM) across all runsOptimality GapProbability of ImprovementInteractive colabWe provide a colab atbit.ly/statistical_precipice_colab,
which shows how to use the library with examples of published algorithms on
widely used benchmarks including Atari 100k, ALE, DM Control and Procgen.PaperFor more details, refer to the accompanyingNeurIPS 2021paper (Outstanding PaperAward):Deep Reinforcement Learning at the Edge of the Statistical Precipice.InstallationTo installrliable, run:pipinstall-UrliableTo install latest version ofrliableas a package, run:pipinstallgit+https://github.com/google-research/rliableTo importrliable, we suggest:fromrliableimportlibraryasrlyfromrliableimportmetricsfromrliableimportplot_utilsAggregate metrics with 95% Stratified Bootstrap CIsIQM, Optimality Gap, Median, Meanalgorithms=['DQN (Nature)','DQN (Adam)','C51','REM','Rainbow','IQN','M-IQN','DreamerV2']# Load ALE scores as a dictionary mapping algorithms to their human normalized# score matrices, each of which is of size `(num_runs x num_games)`.atari_200m_normalized_score_dict=...aggregate_func=lambdax:np.array([metrics.aggregate_median(x),metrics.aggregate_iqm(x),metrics.aggregate_mean(x),metrics.aggregate_optimality_gap(x)])aggregate_scores,aggregate_score_cis=rly.get_interval_estimates(atari_200m_normalized_score_dict,aggregate_func,reps=50000)fig,axes=plot_utils.plot_interval_estimates(aggregate_scores,aggregate_score_cis,metric_names=['Median','IQM','Mean','Optimality Gap'],algorithms=algorithms,xlabel='Human Normalized Score')Probability of Improvement# Load ProcGen scores as a dictionary containing pairs of normalized score# matrices for pairs of algorithms we want to compareprocgen_algorithm_pairs={..,'x,y':(score_x,score_y),..}average_probabilities,average_prob_cis=rly.get_interval_estimates(procgen_algorithm_pairs,metrics.probability_of_improvement,reps=2000)plot_utils.plot_probability_of_improvement(average_probabilities,average_prob_cis)Sample Efficiency Curvealgorithms=['DQN (Nature)','DQN (Adam)','C51','REM','Rainbow','IQN','M-IQN','DreamerV2']# Load ALE scores as a dictionary mapping algorithms to their human normalized# score matrices across all 200 million frames, each of which is of size# `(num_runs x num_games x 200)` where scores are recorded every million frame.ale_all_frames_scores_dict=...frames=np.array([1,10,25,50,75,100,125,150,175,200])-1ale_frames_scores_dict={algorithm:score[:,:,frames]foralgorithm,scoreinale_all_frames_scores_dict.items()}iqm=lambdascores:np.array([metrics.aggregate_iqm(scores[...,frame])forframeinrange(scores.shape[-1])])iqm_scores,iqm_cis=rly.get_interval_estimates(ale_frames_scores_dict,iqm,reps=50000)plot_utils.plot_sample_efficiency_curve(frames+1,iqm_scores,iqm_cis,algorithms=algorithms,xlabel=r'Number of Frames (in millions)',ylabel='IQM Human Normalized Score')Performance Profiles# Load ALE scores as a dictionary mapping algorithms to their human normalized# score matrices, each of which is of size `(num_runs x num_games)`.atari_200m_normalized_score_dict=...# Human normalized score thresholdsatari_200m_thresholds=np.linspace(0.0,8.0,81)score_distributions,score_distributions_cis=rly.create_performance_profile(atari_200m_normalized_score_dict,atari_200m_thresholds)# Plot score distributionsfig,ax=plt.subplots(ncols=1,figsize=(7,5))plot_utils.plot_performance_profiles(score_distributions,atari_200m_thresholds,performance_profile_cis=score_distributions_cis,colors=dict(zip(algorithms,sns.color_palette('colorblind'))),xlabel=r'Human Normalized Score $(\tau)$',ax=ax)The above profile can also be plotted with non-linear scaling as follows:plot_utils.plot_performance_profiles(perf_prof_atari_200m,atari_200m_tau,performance_profile_cis=perf_prof_atari_200m_cis,use_non_linear_scaling=True,xticks=[0.0,0.5,1.0,2.0,4.0,8.0]colors=dict(zip(algorithms,sns.color_palette('colorblind'))),xlabel=r'Human Normalized Score $(\tau)$',ax=ax)DependenciesThe code was tested underPython>=3.7and uses these packages:arch == 5.3.0scipy >= 1.7.0numpy >= 0.9.0absl-py >= 1.16.4seaborn >= 0.11.2CitingIf you find this open source release useful, please reference in your paper:@article{agarwal2021deep,
  title={Deep Reinforcement Learning at the Edge of the Statistical Precipice},
  author={Agarwal, Rishabh and Schwarzer, Max and Castro, Pablo Samuel
          and Courville, Aaron and Bellemare, Marc G},
  journal={Advances in Neural Information Processing Systems},
  year={2021}
}Disclaimer: This is not an official Google product.