Package: jaisalab
jaisalab: A Toolkit for Safe RLjaisalab is a framework for developing and evaluating reinforcement learning algorithms that take into account constraints that should be satisfied in OpenAI gym environments, that also provides PyTorch implementations for a few state-of-the-art algorithms in Constrained RL.
Namely, the algorithms implemented areConstrained Policy Optimization (CPO) <https://github.com/jaimesabalimperial/jaisalab/blob/master/jaisalab/algos/cpo.py>, theSafety Augmented RL (SAUTE) <https://github.com/jaimesabalimperial/jaisalab/blob/master/jaisalab/envs/saute_env.py>wrapper that incorporates safety to any RL algorithm in a plug-n-play manner, andDistributional Constrained Policy Optimization, a modification to CPO that exploits uncertainty in the cost value function to better satisfy constraints.The framework builds on thegarage <https://github.com/rlworkgroup/garage>_ toolkit to add functionality to safe RL settings (i.e. where there is a cost associated with each state-action pair on top of the reward) and thus provides an auxiliary set of modular tools for implementing constrained RL algorithms, analogously to howgarageprovides these for regular RL settings.Running Unit and Integration TestsThis project uses pytest for unit and integration testing (included in the
developer dependencies). The tests may be run from the root directory as
follows:.. code-block:: console$ pytest
...
===== x passed, x warnings in x.xx seconds =====Instructions for Experiment ReproductionThe final results for DCPO, Saute TRPO, Vanilla TRPO, and CPO showed in the final report are equivalent to the experiments injaisalab.experiments.backlog <https://github.com/jaimesabalimperial/jaisalab/blob/master/jaisalab/experiments/backlog.py>_, ran across seeds 1-10. The evaluation of the trained algorithms is done through the SeedEvaluator object for 30 epochs and the plots are made using the methods of the RLPlotter object.ReferencesThe code implementations for the algorithms present injaisalabare based partly or fully on the repositories of their respective papers. ForCPO, the original TensorFlow implementation can be found inhere <https://github.com/jachiam/cpo>, but this had to be translated to PyTorch to fit our framework. ForSAUTE, the implementation was much more straightforward since it's simply a wrapper around the environment, rather than a separate algorithm that should fit our framework as a whole, so the code present injaisalab/envs/safe_envandjaisalab/envs/saute_envwas taken directly fromHuawei's repository <https://github.com/huawei-noah/HEBO>(more specificallyhere <https://github.com/huawei-noah/HEBO/tree/405dc4ceb93a79f0d1f0eaa24f5458dd26de1d05/SAUTE/envs/wrappers>_).Citing jaisalabIf you use jaisalab for academic research, please cite the repository using the
following BibTeX entry. You should update thecommitfield with the commit or
release tag your publication uses... code-block:: console@misc{jaisalab,
author = {Jaime Sabal Bermúdez},
title = {jaisalab: A garage-based framework for reproducible constrained reinforcement learning research},
year = {2022},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/jaimesabalimperial/jaisalab}},
commit = {cc5bc6b4dc7074af1f47d21c6d312429b2ccb931}NotesDeveloped in partial fulfillment of the individual project of Jaime Sabal Bermúdez for the MSc degree in Artificial Intelligence of Imperial College London.I would like to thank Dr. Calvin Tsay for his guidance and advice throughout the project.Disclaimer:While jaisalab is an open-source project and contributions are
welcome, we cannot guarantee that the codebase will be actively maintained in
the future.