Package: bsuite
Behaviour Suite for Reinforcement Learning (bsuite)Introductionbsuiteis a collection of carefully-designed experiments that investigate core
capabilities of a reinforcement learning (RL) agent with two main objectives.To collect clear, informative and scalable problems that capture key issues
in the design of efficient and general learning algorithms.To study agent behavior through their performance on these shared
benchmarks.This library automates evaluation and analysis of any agent on these benchmarks.
It serves to facilitate reproducible, and accessible, research on the core
issues in RL, and ultimately the design of superior learning algorithms.Going forward, we hope to incorporate more excellent experiments from the
research community, and commit to a periodic review of the experiments from a
committee of prominent researchers.For a more comprehensive overview, see the accompanyingpaper.Technical overviewbsuiteis a collection ofexperiments, defined in theexperimentssubdirectory. Each subdirectory corresponds to one experiment and contains:A file defining an RL environment, which may be configurable to provide
different levels of difficulty or different random seeds (for example).A sequence of keyword arguments for this environment, defined in theSETTINGSvariable found in the experiment'ssweep.pyfile.A fileanalysis.pydefining plots used in the provided Jupyter notebook.bsuiteworks by logging results from "within" each environment, when loading
environment via aload_and_record*function.
This means any experiment will automatically output data in the correct format
for analysis using the notebook, without any constraints on the structure of
agents or algorithms.We collate all of the results and analysis in a pre-made jupyter notebookbit.ly/bsuite-colab.Getting startedIf you are new tobsuiteyou can get started in ourcolab tutorial.
This Jupyter notebook is hosted with a free cloud server, so you can start
coding right away without installing anything on your machine. After this, you
can follow the instructions below to getbsuiterunning on your local machine.InstallationWe have testedbsuiteon Python 3.6 & 3.7. To install the dependencies:Optional: We recommend using aPython virtual environmentto manage your dependencies, so as not to clobber your system installation:python3-mvenvbsuitesourcebsuite/bin/activate
pipinstall--upgradepipsetuptoolsInstallbsuitedirectly fromPyPI:pipinstallbsuiteOptional: To also install dependencies for thebaselinesexamples
(excluding OpenAI and Dopamine examples), run:pipinstallbsuite[baselines]EnvironmentsComplete descriptions of each environment and their corresponding experiments
are found in theanalysis/results.ipynbJupyter notebook.These environments all have small observation sizes, allowing for reasonable performance with a small network on a CPU.Loading an environmentEnvironments are specified by absuite_idstring, for example"deep_sea/7".
This string denotes the experiment and the (index of the) environment settings
to use, as described in thetechnical overview section.For a full description of each environment and its corresponding experiment settings, see thepaper.importbsuiteenv=bsuite.load_from_id('catch/0')The sequence ofbsuite_ids required to run all experiments can be accessed
programmatically via:frombsuiteimportsweepsweep.SWEEPThis module also containsbsuite_ids for each experiment individually via
uppercase constants corresponding to the experiment name, for example:sweep.DEEP_SEAsweep.DISCOUNTING_CHAINIn addition, sequences ofbsuite_ids with the same tag can be loaded via:frombsuiteimportsweepsweep.TAGSTheTAGSvariable groupsbsuiteenvironments together by their underlying
tag, so all thebasictasks orscaletasks can be loaded with:sweep.TAGS['basic']sweep.TAGS['scale']Loading an environment with logging includedWe include two implementations of automatic logging, available via:bsuite.load_and_record_to_csv. This outputs one CSV file perbsuite_id, so is suitable for running a set of bsuite experiments split
over multiple machines. The implementation is inlogging/csv_logging.pybsuite.load_and_record_to_sqlite. This outputs a single file, and is
best suited when running a set of bsuite experiments via multiple processes
on a single workstation. The implementation is inlogging/sqlite_logging.py.We also include a terminal logger inlogging/terminal_logging.py, exposed
viabsuite.load_and_record_to_terminal.It is easy to write your own logging mechanism, if you need to save results to a
different storage system. See the CSV implementation for the simplest reference.Interacting with an environmentOur environments implement the Python interface defined indm_env.More specifically, all our environments accept a discrete, zero-based integer
action (or equivalently, a scalar numpy array with shape()).To determine the number of actions for a specific environment, usenum_actions=env.action_spec().num_valuesEach environment returns observations in the form of a numpy array.We also expose absuite_num_episodesproperty for each environment in bsuite.
This allows users to run exactly the number of episodes required for bsuite's
analysis, which may vary between environments used in different experiments.Example run loop for a hypothetical agent with astep()method.for_inrange(env.bsuite_num_episodes):timestep=env.reset()whilenottimestep.last():action=agent.step(timestep)timestep=env.step(action)agent.step(timestep)Usingbsuitein 'OpenAI Gym' formatTo usebsuitewith a codebase that uses theOpenAI Gyminterface, use theGymFromDMEnvclass inutils/gym_wrapper.py:importbsuitefrombsuite.utilsimportgym_wrapperenv=bsuite.load_and_record_to_csv('catch/0',results_dir='/path/to/results')gym_env=gym_wrapper.GymFromDMEnv(env)Note thatbsuitedoes not include Gym in its default dependencies, so you may
need to pip install it separately.Baseline agentsWe include implementations of several common agents in the [baselines/]
subdirectory, along with a minimal run-loop.See theinstallationsection for how to include the required
dependencies at install time. These
dependencies are not installed by default, sincebsuitedoes not require users
to use any specific machine learning library.Running the entire suite of experimentsEach of the agents in thebaselinesfolder contains arunscript which
serves as an example which can run against a single environment or against the
entire suite of experiments, by passing the--bsuite_id=SWEEPflags; this will
start a pool of processes with which to run as many experiments in parallel as
the host machine allows. On a 12 core machine, this will complete overnight for
most agents. Alternatively, it is possible to run on Google Compute Platform
usingrun_on_gcp.sh, steps of which are outlined below.Running experiments on Google Cloud Platformrun_on_gcp.shdoes the following in order:Create an instance with specified specs (by default 64-core CPU optimized).git clonesbsuiteand installs it together with other dependencies.Runs the specified agent (currently limited to/baselines) on a specified
environment.Copies the resulting SQLite file to/tmp/bsuite.dbfrom the remote
instance to you local machine.Shuts down the created instance.In order to run the script, you first need to create a billing account. Then
follow the instructionshereto setup and
initialize Cloud SDK. After completinggcloud init, you are ready to runbsuiteon Google Cloud.For this makerun_on_gcp.shexecutable and run it:chmod+xrun_on_gcp.sh
./run_on_gcp.shAfter the instance is created, the instance name will be printed. Then you can
ssh into the instance by selectingCompute Engine -> Instancesand clickingSSH. Note that this is not necessary, as the result will be copied to your
local machine once it is ready. However,sshing might be convenient if you
want to make local changes to agent and environments. In this case, aftersshing, do~/bsuite_env/bin/activateto activate the virtual environment. Then you can run agents viapython~/bsuite/bsuite/baselines/dqn/run.py--bsuite_id=SWEEPfor instance.Analysisbsuitecomes with a ready-made analysis Jupyter notebook included inanalysis/results.ipynb. This notebook loads and processes logged data, and
produces the scores and plots for each experiment. We recommend using this
notebook in conjunction withColaboratory.We provide an example of a suchbsuitereporthere.bsuiteReportYou can usebsuiteto generate an automated 1-page appendix, that summarizes
the core capabilities of your RL algorithm. This appendix is compatible with
most major ML conference formats. For example output run,pdflatexbsuite/reports/neurips_2019/neurips_2019.texMore examples of bsuite reports can be found in thereports/subdirectory.CitingIf you usebsuitein your work, please cite the accompanyingpaper:@inproceedings{osband2020bsuite,title={Behaviour Suite for Reinforcement Learning},author={Osband, Ian andDoron, Yotam andHessel, Matteo andAslanides, John andSezener, Eren andSaraiva, Andre andMcKinney, Katrina andLattimore, Tor and{Sz}epesv{\'a}ri, Csaba andSingh, Satinder andVan Roy, Benjamin andSutton, Richard andSilver, David andvan Hasselt, Hado},booktitle={International Conference on Learning Representations},year={2020},url={https://openreview.net/forum?id=rygf-kSYwH}}