Package: bellman
BellmanWebsite|Twitter|Documentation (latest)What does Bellman do?Bellman is a package for model-based reinforcement learning (MBRL) in Python, usingTensorFlowand building on top of model-free reinforcement learning packageTensorFlow Agents.Bellman provides a framework for flexible composition of model-based reinforcement learning algorithms. It offers two major classes of algorithms: decision time planning and background planning algorithms. With each class any kind of supervised learning method can be easily used to learn certain component of the environment. Bellman was designed with modularity in mind - important components can be flexibly combined, such as type of decision time planning method (e.g. a cross entropy method or a random shooting method) and type of model for state transition (e.g. a probabilistic neural network or an ensemble of neural networks). Bellman also provides implementation of several popular state-of-the-art MBRL algorithms, such as PETS, MBPO and METRPO. Theonline documentation (latest)contains more details.Bellman requires Python 3.7 onwards and usesTensorFlow 2.4+for running computations, which allows fast execution on GPUs.MaintainersBellman was originally created by (in alphabetical order)Vincent Adam,Jordi Grau-Moya,Felix Leibfried,John A. McLeod,Hrvoje Stojic, andPeter Vrancx,
atSecondmind Labs.It is now actively maintained by (in alphabetical order)Felix Leibfried,John A. McLeod,Hrvoje Stojic,
andPeter Vrancx.Bellman is an open source project. If you have relevant skills and are interested in contributing then please do contact us (see"The Bellman Community" sectionbelow).We are very grateful to our Secondmind Labs colleagues, maintainers ofGPflowandTriestein particular, for their help with creating contributing guidelines, instructions for users and open-sourcing in general.Install BellmanFor usersFor latest (stable) release fromPyPIyou can usepipto install the toolbox$pipinstallbellmanUsepipto install the toolbox from latest source from GitHub. Check-out thedevelopbranch of theBellman GitHub repository, and in the repository root run$pipinstall-e.This will install the toolbox in editable mode.For contributorsIf you wish to contribute please usePoetryto manage dependencies in a local virtual environment. Poetryconfiguration filespecifies all the development dependencies (testing, linting, typing, docs etc) and makes it much easier to contribute. To install Poetry,follow the instructions in the Poetry documentation.To install this project in editable mode, run the commands below from the root directory of thebellmanrepository.poetryinstallThis command creates a virtual environment for this project
in a hidden.venvdirectory under the root directory. You can easily activate it withpoetryshellYou must also run thepoetry installcommand to install updated dependencies when
thepyproject.tomlfile is updated, for example after agit pull.Installing MuJoCo (Optional)Many benchmarks in continuous control in MBRL use the MuJoCo physics engine. Some of the TF-Agents examples have been tested against Mujoco environments as well. MuJoCo is proprietary software that requires a license(see MuJoCo website). As a result installing it is optional, but because of its importance to the research community it is highly recommended. Don't worry if you decide not to install MuJoCo though, all our examples and notebooks rely on standard environments available in OpenAI Gym.We interface with MuJoCo through a python librarymujoco-pyvia OpenAI Gym(mujoco-py github page). Check the installation instructions there on how to install MuJoCo. Note that you should install MuJoCo 1.5 since OpenAI Gym supports that version. After that you can install mujoco-py library with an additional Poetry command:poetryinstall-Emujoco-pyIf this command fails, please check troubleshooting sections atmujoco-pygithub page, you might need to satisfy othermujoco-pydependencies (e.g. Linux system libraries) or set some environment variables.The Bellman CommunityGetting helpBugs, feature requests, pain points, annoying design quirks, etc:Please useGitHub issuesto flag up bugs/issues/pain points, suggest new features, and discuss anything else related to the use of Bellman that in some sense involves changing the Bellman code itself. We positively welcome comments or concerns about usability, and suggestions for changes at any level of design. We aim to respond to issues promptly, but if you believe we may have forgotten about an issue, please feel free to add another comment to remind us."How-to-use" questions:Please useStack Overflow (Bellman tag)to ask questions that relate to "how to use Bellman", i.e. questions of understanding rather than issues that require changing Bellman code. (If you are unsure where to ask, you are always welcome to open a GitHub issue; we may then ask you to move your question to Stack Overflow.)Slack workspaceWe have a publicBellman slack workspace. Please use thisinvite linkif you'd like to join, whether to ask short informal questions or to be involved in the discussion and future development of Bellman.ContributingAll constructive input is very much welcome. For detailed information, see theguidelines for contributors.Citing BellmanTo cite Bellman, please reference ourarXiv paperwhere we review the framework and describe the design. Sample Bibtex is given below:@article{bellman2021,
    author = {McLeod, John and Stojic, Hrvoje and Adam, Vincent and Kim, Dongho and Grau-Moya, Jordi and Vrancx, Peter and Leibfried, Felix},
    title = {Bellman: A Toolbox for Model-based Reinforcement Learning in TensorFlow},
    year = {2021},
    journal = {arXiv:2103.14407},
    url = {https://arxiv.org/abs/2103.14407}
}LicenseApache License 2.0