Package: cleanrl
CleanRL (Clean Implementation of RL Algorithms)CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:📜 Single-file implementationEvery detail about an algorithm variant is put into a single standalone file.For example, ourppo_atari.pyonly has 340 lines of code but contains all implementation details on how PPO works with Atari games,so it is a great reference implementation to read for folks who do not wish to read an entire modular library.📊 Benchmarked Implementation (7+ algorithms and 34+ games athttps://benchmark.cleanrl.dev)📈 Tensorboard Logging🪛 Local Reproducibility via Seeding🎮 Videos of Gameplay Capturing🧫 Experiment Management withWeights and Biases💸 Cloud Integration with docker and AWSYou can read more about CleanRL in ourJMLR paperanddocumentation.CleanRL only contains implementations ofonlinedeep reinforcement learning algorithms. If you are looking forofflinealgorithms, please check outtinkoff-ai/CORL, which shares a similar design philosophy as CleanRL.ℹ️Support for Gymnasium:Farama-Foundation/Gymnasiumis the next generation ofopenai/gymthat will continue to be maintained and introduce new features. Please see theirannouncementfor further detail. We are migrating togymnasiumand the progress can be tracked invwxyzjn/cleanrl#277.⚠️NOTE: CleanRL isnota modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's varaint or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).Get startedPrerequisites:Python >=3.7.1,<3.11Poetry 1.2.1+To run experiments locally, give the following a try:gitclonehttps://github.com/vwxyzjn/cleanrl.git&&cdcleanrl
poetryinstall# alternatively, you could use `poetry shell` and do# `python run cleanrl/ppo.py`poetryrunpythoncleanrl/ppo.py\--seed1\--env-idCartPole-v0\--total-timesteps50000# open another temrminal and enter `cd cleanrl/cleanrl`tensorboard--logdirrunsTo use experiment tracking with wandb, runwandblogin# only required for the first timepoetryrunpythoncleanrl/ppo.py\--seed1\--env-idCartPole-v0\--total-timesteps50000\--track\--wandb-project-namecleanrltestIf you are not usingpoetry, you can install CleanRL withrequirements.txt:# core dependenciespipinstall-rrequirements/requirements.txt# optional dependenciespipinstall-rrequirements/requirements-atari.txt
pipinstall-rrequirements/requirements-mujoco.txt
pipinstall-rrequirements/requirements-mujoco_py.txt
pipinstall-rrequirements/requirements-procgen.txt
pipinstall-rrequirements/requirements-envpool.txt
pipinstall-rrequirements/requirements-pettingzoo.txt
pipinstall-rrequirements/requirements-jax.txt
pipinstall-rrequirements/requirements-docs.txt
pipinstall-rrequirements/requirements-cloud.txtTo run training scripts in other games:poetry shell

# classic control
python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1

# atari
poetry install -E atari
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
poetry install -E envpool
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
poetry run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3

# procgen
poetry install -E procgen
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot

# ppo + lstm
poetry install -E atari
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4You may also use a prebuilt development environment hosted in Gitpod:Algorithms ImplementedAlgorithmVariants Implemented✅Proximal Policy Gradient (PPO)ppo.py,docsppo_atari.py,docsppo_continuous_action.py,docsppo_atari_lstm.py,docsppo_atari_envpool.py,docsppo_atari_envpool_xla_jax.py,docsppo_atari_envpool_xla_jax_scan.py,docs)ppo_procgen.py,docsppo_atari_multigpu.py,docsppo_pettingzoo_ma_atari.py,docsppo_continuous_action_isaacgym.py,docs✅Deep Q-Learning (DQN)dqn.py,docsdqn_atari.py,docsdqn_jax.py,docsdqn_atari_jax.py,docs✅Categorical DQN (C51)c51.py,docsc51_atari.py,docsc51_jax.py,docsc51_atari_jax.py,docs✅Soft Actor-Critic (SAC)sac_continuous_action.py,docssac_atari.py,docs✅Deep Deterministic Policy Gradient (DDPG)ddpg_continuous_action.py,docsddpg_continuous_action_jax.py,docs✅Twin Delayed Deep Deterministic Policy Gradient (TD3)td3_continuous_action.py,docstd3_continuous_action_jax.py,docs✅Phasic Policy Gradient (PPG)ppg_procgen.py,docs✅Random Network Distillation (RND)ppo_rnd_envpool.py,docsOpen RL BenchmarkTo make our experimental data transparent, CleanRL participates in a related project calledOpen RL Benchmark, which contains tracked experiments from popular DRL libraries such as ours,Stable-baselines3,openai/baselines,jaxrl, and others.Check outhttps://benchmark.cleanrl.dev/for a collection of Weights and Biases reports showcasing tracked DRL experiments. The reports are interactive, and researchers can easily query information such as GPU utilization and videos of an agent's gameplay that are normally hard to acquire in other RL benchmarks. In the future, Open RL Benchmark will likely provide an dataset API for researchers to easily access the data (seerepo).Support and get involvedWe have aDiscord Communityfor support. Feel free to ask questions. Posting inGithub Issuesand PRs are also welcome. Also our past video recordings are available atYouTubeCiting CleanRLIf you use CleanRL in your work, please cite our technicalpaper:@article{huang2022cleanrl,author={Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},title={CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},journal={Journal of Machine Learning Research},year={2022},volume={23},number={274},pages={1--18},url={http://jmlr.org/papers/v23/21-1342.html}}