Package: openrlbenchmark-nightly
Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement LearningOpen RL Benchmark is a comprehensive collection of tracked experiments for RL. It aims to make it easier for RL practitioners to pull and compare all kinds of metrics from reputable RL libraries like Stable-baselines3, Tianshou, CleanRL, and others.💾GitHub Repo: source code and more docs.📜Design docs: our motivation and vision.🔗Open RL Benchmark reports:  W&B reports with tracked Atari, MuJoCo experiments from SB3, CleanRL, and others.InstallationYou can install it via pip or the dev setup.Pip installpipinstallopenrlbenchmark--upgradeDev SetupPrerequisites:Python >=3.7.1,<3.10 (not yet 3.10)Poetry 1.2.1+gitclonehttps://github.com/openrlbenchmark/openrlbenchmark.gitcdopenrlbenchmark
poetryinstallGet startedOpen RL Benchmark provides an RLops CLI to pull and compare metrics from Weights and Biases. The following example shows how to compare the performance of SB3's ppo, a2c, ddpg, ppo_lstm, sac, td3, ppo, trpo, CleanRL's sac onHalfCheetahBulletEnv-v0.python-mopenrlbenchmark.rlops\--filters'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return'\'ppo_continuous_action?tag=v1.0.0-27-gde3f410&cl=CleanRL PPO'\--filters'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return'\'baselines-ppo2-mlp?cl=openai/baselines PPO2'\--env-idsHalfCheetah-v2Hopper-v2Walker2d-v2\--env-idsHalfCheetah-v2Hopper-v2Walker2d-v2\--no-check-empty-runs\--pc.ncols3\--pc.ncols-legend3\--rliable\--rc.score_normalization_methodmaxmin\--rc.normalized_score_threshold1.0\--rc.sample_efficiency_plots\--rc.sample_efficiency_and_walltime_efficiency_methodMedian\--rc.performance_profile_plots\--rc.aggregate_metrics_plots\--rc.sample_efficiency_num_bootstrap_reps10\--rc.performance_profile_num_bootstrap_reps10\--rc.interval_estimates_num_bootstrap_reps10\--output-filenamestatic/0compare\--scan-historyHere, we created multiple filters. The first string in the first filter is'?we=openrlbenchmark&wpn=sb3&ceik=env&cen=algo&metric=rollout/ep_rew_mean', which is a query string that specifies the following:we: the W&B entity namewpn: the W&B project nameceik: the custom key for the environment idcen: the custom key for the experiment namemetric: the metric we are interested inSo we are fetching metrics fromhttps://wandb.ai/openrlbenchmark/sb3. The environment id is stored in theenvkey, and the experiment name is stored in thealgokey. The metric we are interested in isrollout/ep_rew_mean.Similarly, we are fetching metrics fromhttps://wandb.ai/openrlbenchmark/cleanrl. The environment id is stored in theenv_idkey, and the experiment name is stored in theexp_namekey. The metric we are interested in ischarts/episodic_return. You can also customize the legend with theclquery string, such asppo_lstm?cl=PPO w/ LSTM.The labels of the figure can be customized with the--pc.xlabeland--pc.ylabelflags. The--pc.ncolsflag specifies the number of columns in the figure. The--pc.ncols-legendflag specifies the number of columns in the legend. The--output-filenameflag specifies the filename of the output figureThe command above generates the following plot:cleanrl vs. Stable Baselines 3cleanrl vs. Stable Baselines 3 (Time)The--reporttag also generates awandb reportThe command also generates acompare.png, acompare.md, and acompare.csvin the current directory.Learning curves:thecompare.pngshows the learning curves which subsamples 10000 data points and and interpolate. The curves are smoothed by a rolling average with a window size 100 and their shaded region represents the standard deviation.Result table:thecompare.mdandcompare.csvshows the average episodic return of the last 100 episodes. For each random seed $i$ (we have 3 random seeds for each set of experiments), we calculate the average episodic return of the last 100 training episodes as $a_i$. We then average the $a_i$'s over all random seeds to get the final average episodic return and report its standard deviation. This type of evaluation is known as an implicit evaluation method (Machado et al., 2017) which aligns better with the general goal of RL which is continual learning. This method also detects issues with catastrophic forgetting compared to the evaluation method that evaluates the best model.WarningYou may get slightly different curves every time you run the commands. This is because we sample 500 data points from the track experiments to save bandwidth. If you are usingopenrlbenchmarkrepeatedly or you wanto to generate consistentcompare.mdand learning curves, we recommend you to use--scan-historyto get all of the data points, but initially it may take a while to run.Currently supported librariesCleanRLceik:env_idcen:exp_name(e.g.,sac_continuous_action,ppo_continuous_action,ppo_atari)metric:charts/episodic_returnStable-baselines3ceik:envcen:algo(e.g.,sac,ppo,a2c)metric:rollout/ep_rew_meanoreval/mean_rewardikostrikov/jaxrlceik:env_namecen:algo(e.g.,sac)metric:training/returnorevaluation/average_returnsbaselinesceik:envcen:alg(e.g.,ppo2)metric:charts/episodic_returnoreprewmeansbxceik:envcen:alg(e.g.,sac,tqc)metric:rollout/ep_rew_meanoreval/mean_rewardTianshouceik:taskcen:algo_name(e.g.,ppo,iqn)metric:test/rewardMORL-Baselinesceik:env_idcen:algo(e.g.,PGMORL,PCN)metric:eval/hypervolume,eval/igd,eval/sparsity,eval/eum,eval/mulThe following libraries have some recorded experiments:openai/phasic-policy-gradient(has some metrics)ceik:env_namecen:arch(shared)metric:charts/episodic_returnsfujim/TD3(has some metrics)ceik:envcen:policy(e.g.,TD3)metric:charts/episodic_returnMore examplesCompare CleanRL's PPO withopenai/baselines's PPO2 on Atari games:Sometimes the same environments could have different names in different libraries. For example,openai/baselinesusesBreakoutNoFrameskip-v4whileEnvPoolusesBreakout-v5. To compare the two libraries, we need to specify theenv_idforCleanRLandenvforopenai/baselines. In this case, can specify the correspondingenv_idsfor each filter.For Atari games, we have additional batteries includedopenrlbenchmark.rlops_hnsto show human normalized-scores and further statistical analysis throughrliable.python-mopenrlbenchmark.rlops_hns\--filters'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return''baselines-ppo2-cnn'\--filters'?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return''ppo_atari_envpool_xla_jax_truncation'\--env-idsAlienNoFrameskip-v4AmidarNoFrameskip-v4AssaultNoFrameskip-v4AsterixNoFrameskip-v4AsteroidsNoFrameskip-v4AtlantisNoFrameskip-v4BankHeistNoFrameskip-v4BattleZoneNoFrameskip-v4BeamRiderNoFrameskip-v4BerzerkNoFrameskip-v4BowlingNoFrameskip-v4BoxingNoFrameskip-v4BreakoutNoFrameskip-v4CentipedeNoFrameskip-v4ChopperCommandNoFrameskip-v4CrazyClimberNoFrameskip-v4DefenderNoFrameskip-v4DemonAttackNoFrameskip-v4DoubleDunkNoFrameskip-v4EnduroNoFrameskip-v4FishingDerbyNoFrameskip-v4FreewayNoFrameskip-v4FrostbiteNoFrameskip-v4GopherNoFrameskip-v4GravitarNoFrameskip-v4HeroNoFrameskip-v4IceHockeyNoFrameskip-v4PrivateEyeNoFrameskip-v4QbertNoFrameskip-v4RiverraidNoFrameskip-v4RoadRunnerNoFrameskip-v4RobotankNoFrameskip-v4SeaquestNoFrameskip-v4SkiingNoFrameskip-v4SolarisNoFrameskip-v4SpaceInvadersNoFrameskip-v4StarGunnerNoFrameskip-v4SurroundNoFrameskip-v4TennisNoFrameskip-v4TimePilotNoFrameskip-v4TutankhamNoFrameskip-v4UpNDownNoFrameskip-v4VentureNoFrameskip-v4VideoPinballNoFrameskip-v4WizardOfWorNoFrameskip-v4YarsRevengeNoFrameskip-v4ZaxxonNoFrameskip-v4JamesbondNoFrameskip-v4KangarooNoFrameskip-v4KrullNoFrameskip-v4KungFuMasterNoFrameskip-v4MontezumaRevengeNoFrameskip-v4MsPacmanNoFrameskip-v4NameThisGameNoFrameskip-v4PhoenixNoFrameskip-v4PitfallNoFrameskip-v4PongNoFrameskip-v4\--env-idsAlien-v5Amidar-v5Assault-v5Asterix-v5Asteroids-v5Atlantis-v5BankHeist-v5BattleZone-v5BeamRider-v5Berzerk-v5Bowling-v5Boxing-v5Breakout-v5Centipede-v5ChopperCommand-v5CrazyClimber-v5Defender-v5DemonAttack-v5DoubleDunk-v5Enduro-v5FishingDerby-v5Freeway-v5Frostbite-v5Gopher-v5Gravitar-v5Hero-v5IceHockey-v5PrivateEye-v5Qbert-v5Riverraid-v5RoadRunner-v5Robotank-v5Seaquest-v5Skiing-v5Solaris-v5SpaceInvaders-v5StarGunner-v5Surround-v5Tennis-v5TimePilot-v5Tutankham-v5UpNDown-v5Venture-v5VideoPinball-v5WizardOfWor-v5YarsRevenge-v5Zaxxon-v5Jamesbond-v5Kangaroo-v5Krull-v5KungFuMaster-v5MontezumaRevenge-v5MsPacman-v5NameThisGame-v5Phoenix-v5Pitfall-v5Pong-v5\--no-check-empty-runs\--pc.ncols5\--pc.ncols-legend2\--output-filenamestatic/cleanrl_vs_baselines\--scan-history--rliableIn the individual learning curves below, the right y-axis is the human normalized score. The left y-axis is the raw episodic return.The script also generatescleanrl_vs_baselines_hns_median.png, the learning curves for median human-normalized scores.Furthermore, the--rliableintegration generatescleanrl_vs_baselines_iqm_profile.png, the  Interquartile Mean (IQM) and performance profile (Agarwal et al., 2022), andcleanrl_vs_baselines_hns_aggregate.png, the aggregate human-normalized scores with Stratified Bootstrap Confidence Intervals (see @araffin's excellent blog postexplainer).Offline modeSometimes even with caching--scan-historythe script can still take a long time if there are too many environments or experiments. This is because we are still calling manywandb.Api().runs(..., filters)under the hood.No worries though. When running with--scan-history, we also automatically build a localsqlitedatabase to store the metadata of runs, enabling us to run the script without internet connection.python-mopenrlbenchmark.rlops\--filters'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return''baselines-ppo2-mlp'\--filters'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return''ppo_continuous_action?tag=v1.0.0-27-gde3f410'\--filters'?we=openrlbenchmark&wpn=jaxrl&ceik=env_name&cen=algo&metric=training/return''sac'\--env-idsHalfCheetah-v2Walker2d-v2Hopper-v2InvertedPendulum-v2Humanoid-v2Pusher-v2\--no-check-empty-runs\--pc.ncols3\--pc.ncols-legend3\--output-filenamestatic/baselines_vs_cleanrl_vs_jaxrl\--scan-history\--offlineCompare CleanRL's PPO withopenai/baselines's PPO2 andjaxrl's SAC on Mujoco:python-mopenrlbenchmark.rlops\--filters'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return''baselines-ppo2-mlp'\--filters'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return''ppo_continuous_action?tag=v1.0.0-27-gde3f410'\--filters'?we=openrlbenchmark&wpn=jaxrl&ceik=env_name&cen=algo&metric=training/return''sac'\--env-idsHalfCheetah-v2Walker2d-v2Hopper-v2InvertedPendulum-v2Humanoid-v2Pusher-v2\--no-check-empty-runs\--pc.ncols3\--pc.ncols-legend3\--output-filenamestatic/baselines_vs_cleanrl_vs_jaxrl\--scan-historyCompare Tianshou's algorithms withopenai/baselines's PPO2 on Atari:python-mopenrlbenchmark.rlops\--filters'?we=tianshou&wpn=atari.benchmark&ceik=task&cen=algo_name&metric=test/reward''iqn''ppo''rainbow''fqf''c51''dqn''qrdqn'\--filters'?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return''baselines-ppo2-cnn'\--env-idsBreakoutNoFrameskip-v4SpaceInvadersNoFrameskip-v4SeaquestNoFrameskip-v4MsPacmanNoFrameskip-v4EnduroNoFrameskip-v4PongNoFrameskip-v4QbertNoFrameskip-v4\--no-check-empty-runs\--pc.ncols4\--pc.ncols-legend4\--output-filenamestatic/baselines_vs_tianshou--scan-historyCompare CleanRL's PPG and PPO withopenai/phasic-policy-gradient's PPG on procgen:python-mopenrlbenchmark.rlops\--filters'?we=openrlbenchmark&wpn=phasic-policy-gradient&ceik=env_name&cen=arch&metric=charts/episodic_return''shared'\--filters'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return''ppo_procgen?tag=v1.0.0b1-4-g4ea73d9''ppg_procgen?tag=v1.0.0b1-4-g4ea73d9'\--env-idsstarpilotbossfightbigfish\--no-check-empty-runs\--pc.ncols3\--pc.ncols-legend3\--output-filenamestatic/ppg_vs_cleanrl\--scan-historyCompare CleanRL's TD3 withsfujim/TD3's TD3 on Mujoco:python-mopenrlbenchmark.rlops\--filters'?we=openrlbenchmark&wpn=sfujim-TD3&ceik=env&cen=policy&metric=charts/episodic_return''TD3'\--filters'?we=openrlbenchmark&wpn=cleanrl&ceik=env_id&cen=exp_name&metric=charts/episodic_return''td3_continuous_action_jax?tag=pr-285''ddpg_continuous_action_jax?tag=pr-298'\--env-idsHalfCheetah-v2Walker2d-v2Hopper-v2\--no-check-empty-runs\--pc.ncols3\--pc.ncols-legend3\--output-filenamestatic/td3_vs_cleanrl\--scan-historyCompare MORL Baselines algorithms on deterministic environmentspython-mopenrlbenchmark.rlops\--filters'?we=openrlbenchmark&wpn=MORL-Baselines&ceik=env_id&cen=algo&metric=eval/hypervolume'\'Pareto Q-Learning?cl=Pareto Q-Learning'\'MultiPolicy MO Q-Learning?cl=MultiPolicy MO Q-Learning'\'MultiPolicy MO Q-Learning (OLS)?cl=MultiPolicy MO Q-Learning (OLS)'\--env-idsdeep-sea-treasure-v0deep-sea-treasure-concave-v0\--pc.ncols2\--pc.ncols-legend1\--pc.xlabel'Training steps'\--pc.ylabel'Hypervolume'\--output-filenamemorl_deterministic_envs\--scan-historyCalculate human normalized scorespython-mopenrlbenchmark.hns--filesstatic/cleanrl_vs_baselines.csvstatic/machado_10M.csvstatic/machado_50M.csvbaselines-ppo2-cnn ({})
┣━━ median hns: 0.7959851540635047
┣━━ mean hns: 4.54588939893709
ppo_atari_envpool_xla_jax_truncation ({})
┣━━ median hns: 0.9783505154639175
┣━━ mean hns: 6.841083973256849
ppo_atari_envpool_xla_jax_truncation_machado_10M ({})
┣━━ median hns: 0.7347972972972973
┣━━ mean hns: 2.919095857954249
ppo_atari_envpool_xla_jax_truncation ({'metric': ['charts/avg_episodic_return']})
┣━━ median hns: 0.9783505154639175
┣━━ mean hns: 6.841083973256849
ppo_atari_envpool_xla_jax_truncation_machado ({'metric': ['charts/avg_episodic_return']})
┣━━ median hns: 1.5679929625118418
┣━━ mean hns: 8.352308370550299What's going on right now?This is a project we are slowly working on. There is no specific timeline or roadmap, but if you want to get involved. Feel free to reach out to me or open an issue. We are looking for volunteers to help us with the following:Add experiments from other librariesRun more experiments from currently supported librariesDocumentation and designing standardsDownload the tensorboard metrics from the tracked experiments and load them locally to save time