Package: openrl
OpenRL-v0.2.0 is updated on Dec 20, 2023The main branch is the latest version of OpenRL, which is under active development. If you just want to have a try with
OpenRL, you can switch to the stable branch.Welcome to OpenRLDocumentation|中文介绍|中文文档Crafting Reinforcement Learning Frameworks with Passion, Your Valuable Insights Welcome.OpenRL is an open-source general reinforcement learning research framework that supports training for various tasks
such as single-agent, multi-agent, offline RL, self-play, and natural language.
Developed based on PyTorch, the goal of OpenRL is to provide a simple-to-use, flexible, efficient and sustainable
platform for the reinforcement learning research community.Currently, the features supported by OpenRL include:Asimple-to-use universal interfacethat supports training for all tasks/environmentsSupport for both single-agent and multi-agent tasksSupport for offline RL training with expert datasetSupport self-play trainingReinforcement learning training support for natural language tasks (such as dialogue)SupportDeepSpeedSupportArena, which allows convenient evaluation of
various agents (even submissions forJiDi) in a competitive environment.Importing models and datasets fromHugging Face. Supports loadingStable-baselines3 models from Hugging Facefor testing and training.Tutorialon how to integrate user-defined environments into OpenRL.Support for models such as LSTM, GRU, Transformer etc.Multiple training acceleration methods including automatic mixed precision training and data collecting wth half
precision policy networkUser-defined training models, reward models, training data and environment supportSupport forgymnasiumenvironmentsSupport forCallbacks, which can be used to
implement various functions such as logging, saving, and early stoppingDictionary observation space supportPopular visualization tools such
aswandb,tensorboardXare supportedSerial or parallel environment training while ensuring consistent results in both modesChinese and English documentationProvides unit testing and code coverage testingCompliant with Black Code Style guidelines and type checkingAlgorithms currently supported by OpenRL (for more details, please refer toGallery):Proximal Policy Optimization (PPO)Dual-clip PPOMulti-agent PPO (MAPPO)Joint-ratio Policy Optimization (JRPO)Generative Adversarial Imitation Learning (GAIL)Behavior Cloning (BC)Advantage Actor-Critic (A2C)Self-PlayDeep Q-Network (DQN)Multi-Agent Transformer (MAT)Value-Decomposition Network (VDN)Soft Actor Critic (SAC)Deep Deterministic Policy Gradient (DDPG)Environments currently supported by OpenRL (for more details, please refer toGallery):GymnasiumMuJoCoPettingZooMPEChat BotAtariStarCraft IISMACv2Omniverse Isaac GymDeepMind ControlSnakegym-pybullet-dronesGridWorldSuper Mario BrosGym RetroThis framework has undergone multiple iterations by theOpenRL-Labteam which has
applied it in academic research.
It has now become a mature reinforcement learning framework.OpenRL-Lab will continue to maintain and update OpenRL, and we welcome everyone to join
ouropen-source communityto contribute towards the development of reinforcement learning.For more information about OpenRL, please refer to thedocumentation.OutlineWelcome to OpenRLOutlineWhy OpenRL?InstallationUse DockerQuick StartGalleryProjects Using OpenRLFeedback and ContributionMaintainersSupporters↳ Contributors↳ Stargazers↳ ForkersCiting OpenRLLicenseAcknowledgmentsWhy OpenRLHere we provide a table for the comparison of OpenRL and existing popular RL libraries.
OpenRL employs a modular design and high-level abstraction, allowing users to accomplish training for various tasks
through a unified and user-friendly interface.LibraryNLP/RLHFMulti-agentSelf-Play TrainingOffline RLDeepSpeedOpenRL:heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark::heavy_check_mark:Stable Baselines3:x::x::x::x::x:Ray/RLlib:x::heavy_check_mark::heavy_check_mark::heavy_check_mark::x:DI-engine:x::heavy_check_mark:not fullly supported:heavy_check_mark::x:Tianshou:x:not fullly supportednot fullly supported:heavy_check_mark::x:MARLlib:x::heavy_check_mark:not fullly supported:x::x:MAPPO Benchmark:x::heavy_check_mark::x::x::x:RL4LMs:heavy_check_mark::x::x::x::x:trlx:heavy_check_mark::x::x::x::heavy_check_mark:trl:heavy_check_mark::x::x::x::heavy_check_mark:TimeChamber:x::x::heavy_check_mark::x::x:InstallationUsers can directly install OpenRL via pip:pipinstallopenrlIf users are using Anaconda or Miniconda, they can also install OpenRL via conda:condainstall-copenrlopenrlUsers who want to modify the source code can also install OpenRL from the source code:gitclonehttps://github.com/OpenRL-Lab/openrl.git&&cdopenrl
pipinstall-e.After installation, users can check the version of OpenRL through command line:openrl--versionTips: No installation required, try OpenRL online through
Colab:Use DockerOpenRL currently provides Docker images with and without GPU support.
If the user's computer does not have an NVIDIA GPU, they can obtain an image without the GPU plugin using the following
command:sudodockerpullopenrllab/openrl-cpuIf the user wants to accelerate training with a GPU, they can obtain it using the following command:sudodockerpullopenrllab/openrlAfter successfully pulling the image, users can run OpenRL's Docker image using the following commands:# Without GPU accelerationsudodockerrun-itopenrllab/openrl-cpu# With GPU accelerationsudodockerrun-it--gpusall--nethostopenrllab/openrlOnce inside the Docker container, users can check OpenRL's version and then run test cases using these commands:# Check OpenRL version in Docker containeropenrl--version# Run test caseopenrl--modetrain--envCartPole-v1Quick StartOpenRL provides a simple and easy-to-use interface for beginners in reinforcement learning.
Below is an example of using the PPO algorithm to train theCartPoleenvironment:# train_ppo.pyfromopenrl.envs.commonimportmakefromopenrl.modules.commonimportPPONetasNetfromopenrl.runners.commonimportPPOAgentasAgentenv=make("CartPole-v1",env_num=9)# Create an environment and set the environment parallelism to 9.net=Net(env)# Create neural network.agent=Agent(net)# Initialize the agent.agent.train(total_time_steps=20000)# Start training and set the total number of steps to 20,000 for the running environment.Training an agent using OpenRL only requires four simple steps:Create Environment=>Initialize Model=>Initialize Agent=>Start Training!For a well-trained agent, users can also easily test the agent:# train_ppo.pyfromopenrl.envs.commonimportmakefromopenrl.modules.commonimportPPONetasNetfromopenrl.runners.commonimportPPOAgentasAgentagent=Agent(Net(make("CartPole-v1",env_num=9)))# Initialize trainer.agent.train(total_time_steps=20000)# Create an environment for test, set the parallelism of the environment to 9, and set the rendering mode to group_human.env=make("CartPole-v1",env_num=9,render_mode="group_human")agent.set_env(env)# The agent requires an interactive environment.obs,info=env.reset()# Initialize the environment to obtain initial observations and environmental information.whileTrue:action,_=agent.act(obs)# The agent predicts the next action based on environmental observations.# The environment takes one step according to the action, obtains the next observation, reward, whether it ends and environmental information.obs,r,done,info=env.step(action)ifany(done):breakenv.close()# Close test environmentExecuting the above code on a regular laptop only takesa few secondsto complete the training. Below shows the visualization of the agent:Tips:Users can also quickly train theCartPoleenvironment by executing a command line in the terminal.openrl--modetrain--envCartPole-v1For training tasks such as multi-agent and natural language processing, OpenRL also provides a similarly simple and
easy-to-use interface.For information on how to perform multi-agent training, set hyperparameters for training, load training configurations,
use wandb, save GIF animations, etc., please refer to:Multi-Agent Training ExampleFor information on natural language task training, loading models/datasets on Hugging Face, customizing training
models/reward models, etc., please refer to:Dialogue Task Training ExampleFor more information about OpenRL, please refer to thedocumentation.GalleryIn order to facilitate users' familiarity with the framework, we provide more examples and demos of using OpenRL
inGallery.
Users are also welcome to contribute their own training examples and demos to the Gallery.Projects Using OpenRLWe have listed research projects that use OpenRL in theOpenRL Project.
If you are using OpenRL in your research project, you are also welcome to join this list.Feedback and ContributionIf you have any questions or find bugs, you can check or ask in
theIssues.Join the QQ group:OpenRL Official Communication GroupJoin theslackgroup to discuss
OpenRL usage and development with us.Join theDiscordgroup to discuss OpenRL usage and development with us.Send an E-mail to:huangshiyu@4paradigm.comJoin theGitHub Discussion.The OpenRL framework is still under continuous development and documentation.
We welcome you to join us in making this project better:How to contribute code: Read theContributors' GuideOpenRL RoadmapMaintainersAt present, OpenRL is maintained by the following maintainers:Shiyu Huang(@huangshiyu13)Wenze Chen(@Chen001117)Yiwen Sun(@YiwenAI)Welcome more contributors to join our maintenance team (send an E-mail
tohuangshiyu@4paradigm.comto apply for joining the OpenRL team).Supporters↳ Contributors↳ Stargazers↳ ForkersCiting OpenRLIf our work has been helpful to you, please feel free to cite us:@misc{openrl2023,
    title={OpenRL},
    author={OpenRL Contributors},
    publisher ={GitHub},
    howpublished ={\url{https://github.com/OpenRL-Lab/openrl}},
    year={2023},}Star HistoryLicenseOpenRL under the Apache 2.0 license.AcknowledgmentsThe development of the OpenRL framework has drawn on the strengths of other reinforcement learning frameworks:Stable-baselines3:https://github.com/DLR-RM/stable-baselines3pytorch-a2c-ppo-acktr-gail:https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gailMAPPO:https://github.com/marlbenchmark/on-policyGymnasium:https://github.com/Farama-Foundation/GymnasiumDI-engine:https://github.com/opendilab/DI-engine/Tianshou:https://github.com/thu-ml/tianshouRL4LMs:https://github.com/allenai/RL4LMs