Package: fast-safe-rl
Key Features|Documentation|Installation|Quick Start|ContributingThe Fast Safe Reinforcement Learning (FSRL) package provides modularized implementations
of Safe RL algorithms based on PyTorch and theTianshouframework. Safe RL is a rapidly evolving subfield of RL, focusing on ensuring the safety of learning agents during the training and deployment process. The study of Safe RL is essential because it addresses the critical challenge of preventing unintended or harmful actions while still optimizing an agent's performance in complex environments.This project offers high-quality and fast implementations of popular Safe RL algorithms, serving as an ideal starting point for those looking to explore and experiment in this field. By providing a comprehensive and accessible toolkit, the FSRL package aims to accelerate research in this crucial area and contribute to the development of safer and more reliable RL-powered systems.Please note that this project is still under active development, and major updates might be expected.Your feedback and contributions are highly appreciated, as they help us improve the FSRL package.🌟 Key FeaturesFSRL is designed with several key aspects in mind:High-quality implementations. For instance, theCPOimplementation by SafetyGym fails to satisfy constraints according to theirbenchmark results. As a result, many safe RL papers that adopt these implementations may also report failure results. However, we discovered that with appropriate hyper-parameters and our implementation, it can achieve good safety performance in most tasks as well.Fast training speed. FSRL cares about accelerating experimentation and benchmarking processes, providing fast training times for popular safe RL tasks. For example, most algorithms can solve theSafetyCarCircle-v0task in 10 minutes with 4 cpus. TheCVPOalgorithm implementation can also achieve 5x faster training than the original implementation.Well-tuned hyper-parameters. We carefully studied the effects of key hyperparameters for these algorithms and plan to provide a practical guide for tuning them. We believe both implementations and hyper-parameters play a crucial role in learning a successful safe RL agent.Modular design and easy usability. FSRL is built upon the elegant RL frameworkTianshou. We provide an agent wrapper, refactored loggers for both Tensorboard and Wandb, andpyrallisconfiguration support to further facilitate usage. Our algorithms also support multiple constraints and standard RL tasks (like Mujoco).The implemented safe RL algorithms include:AlgorithmTypeDescriptionCPOon-policyConstrained Policy OptimizationFOCOPSon-policyFirst Order Constrained Optimization in Policy SpacePPOLagrangianon-policyPPOwithPID LagrangianTRPOLagrangianon-policyTRPOwithPID LagrangianDDPGLagrangianoff-on-policy(1)DDPGwithPID LagrangianSACLagrangianoff-on-policy(1)SACwithPID LagrangianCVPOoff-policyConstrained Variational Policy Optimization(1):Off-on-policymeans that the base learning algorithm is off-policy, but the Lagrange multiplier is updated in an on-policy fashion. Our previousfindingsuggested that using off-policy style Lagrange update may result in poor performanceThe implemented algorithms are well-tuned for many tasks in the following safe RL
environments, which cover the majority of tasks in recent safe RL papers:BulletSafetyGym, FSRL will install
this environment by default as the testing ground.SafetyGymnasium, note that you need
to install it from the source because our current version adopts thegymnasiumAPI.Note that the latest versions of FSRL and the above environments use thegymnasium >= 0.26.3API. But if you want to use the oldgymAPI such as thesafety_gym, you can simply change the example scripts fromimport gymnasium as gymtoimport gym.🔍 DocumentationThe tutorials and API documentation are hosted onfsrl.readthedocs.io.The majority of the API design in FSRL follows Tianshou, and we aim to reuse their modules as much as possible. For example, theEnv,Batch,Buffer,
and (most)Netmodules are used directly in our repo. This means that you can refer to their comprehensive documentation to gain a good understanding of the code structure. We highly recommend you read the following Tianshou tutorials:Get Started with Jupyter Notebook. You can get a quick overview of different modules through this tutorial.Basic concepts in Tianshou. Note that the basic concepts in FSRL are almost the same as Tianshou.Understanding Batch. Note that the Batch data structure is extensively used in this repo.We observe that for most existing safe RL environments, a few layers of neural networks can solve them quite effectively. Therefore, we provide an 'Agent' class with default MLP networks to facilitate the usage.
You can refer to thetutorialfor more details.Example training and evaluation scripts for both default MLP agent and customized networks are available at theexamplesfolder.🛠️ InstallationFSRL requires Python >= 3.8. You can install it from source by:gitclonehttps://github.com/liuzuxin/fsrl.gitcdfsrl
pipinstall-e.You can also directly install it with pip through GitHub:pipinstallgit+https://github.com/liuzuxin/fsrl.git@main--upgradeYou can check whether the installation is successful by:importfsrlprint(fsrl.__version__)🚀  Quick StartTraining with default MLP agentThis is an example of training a PPO-Lagrangian agent with a Tensorboard logger and default parameters.First, import relevant packages:importbullet_safety_gymimportgymnasiumasgymfromtianshou.envimportDummyVectorEnvfromfsrl.agentimportPPOLagAgentfromfsrl.utilsimportTensorboardLoggerThen initialize the environment, logger, and agent:task="SafetyCarCircle-v0"# init loggerlogger=TensorboardLogger("logs",log_txt=True,name=task)# init the PPO Lag agent with default parametersagent=PPOLagAgent(gym.make(task),logger)# init the envstraining_num,testing_num=10,1train_envs=DummyVectorEnv([lambda:gym.make(task)for_inrange(training_num)])test_envs=DummyVectorEnv([lambda:gym.make(task)for_inrange(testing_num)])Finally, start training:agent.learn(train_envs,test_envs,epoch=100)You can check the experiment results in thelogs/SafetyCarCircle-v0folder.Training with the example scriptsWe provide easy-to-use example training script for all the agents in theexamples/mlpfolder. Each training script is by default use theWandblogger andPyrallisconfiguration system. The default hyper-parameters are located thefsrl/configfolder.
You have three alternatives to run the experiment with your customized hyper-parameters:M1. Directly override the parameters via the command line:pythonexamples/mlp/train_ppol_agent.py--argvalue--arg2value2...where--argspecify the parameter you want to override. For example,--task SafetyAntRun-v0. Note that if you specify--use_default_cfg 1, the script will automatically use the task's default parameters for training. We plan to release more default configs in the future.M2. Use pre-definedyamlorjsonortomlconfigs.For example, you want to use a different learning-rate and training epochs from our default ones, create amy_cfg.yaml:task:"SafetyDroneCircle-v0"epoch:500lr:0.001Then you can starting training with above parameters by:pythonexamples/mlp/train_ppol_agent.py--configmy_cfg.yamlwhere--configspecify the path of the configuration parameters.M3. Inherent the config dataclass in thefsrl/configfolder.For example, you can inherent thePPOLagAgentconfig by:fromdataclassesimportdataclassfromfsrl.config.ppol_cfgimportTrainCfg@dataclassclassMyCfg(TrainCfg):task:str="SafetyDroneCircle-v0"epoch:int=500lr:float=0.001@pyrallis.wrap()deftrain(args:MyCfg):...Then, you can start training with your own default configs:pythonexamples/mlp/train_ppol_agent.pyNote that our example scripts support theauto_namefeature, meaning that it can automatically compare your specified hyper-parameters with our default ones, and create the experiment name based on the difference. The default training statistics are saved in thelogsdirectory.Training with cutomized networksWhile the pre-defined MLP agent is sufficient for solving many existing safe RL benchmarks, for more complex tasks, it may be necessary to customize the value and policy networks. Our modular design supports Tianshou's style training scripts. Example training scripts can be found in theexamples/customizedfolder. For more details on building networks, please refer to Tianshou'stutorial, as our algorithms are mostly compatible with their networks.Evaluate trained modelsTo evaluate a trained model, for example, a pre-trained PPOLag model in thelogs/exp_namefolder, run:pythonexamples/mlp/eval_ppol_agent.py--pathlogs/exp_name--eval_episodes20It will load the savedconfig.yamlfromlogs/exp_name/config.yamland pre-trained model fromlogs/exp_name/checkpoint/model.pt, run 20 episodes and print the average reward and cost. If thebestmodel is saved during training, you can evaluate it by setting--best 1.Related ProjectsFSRL is heavily inspired by theTianshouproject. In addition, there are several other remarkable safe RL-related projects:Safety-Gymnasium, a well-maintained and customizable safe RL environments based on Mujoco.Bullet-Safety-Gym, a tuned and fast safe RL environments based on PyBullet.Safe-Multi-Agent-Mujoco, a multi-agent safe RL environments based on Mujoco.Safe-Control-Gym, a learning-based control and RL library with PyBullet.OmniSafe, a well-maintained infrastructural framework for safe RL algorithms.SafePO, another benchmark repository for safe RL algorithms.ContributingThe main maintainers of this project are:Zuxin Liu,Zijian Guo.If you have any suggestions or find any bugs, please feel free to submit an issue or a pull request. We welcome contributions from the community!