Behaviour Suite for Reinforcement Learning
Authors: Ian Osband
Yotam Doron
Matteo Hessel
John Aslanides
Eren Sezener
Andre Saraiva
Katrina McKinney
Tor Lattimore
Csaba Szepesvari
Satinder Singh
Benjamin Van Roy
Richard Sutton
David Silver
Hado Van Hasselt
Venue:  (2020)
Keywords: reinforcement learning
benchmark
core issues
scalability
reproducibility

Abstract: This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is a collection of carefully-designed experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning algorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source this http URL, which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research on the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAI Baselines, Dopamine as well as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent researchers.

Published as a conference paper at ICLR 2020
BEHAVIOUR SUITE FOR REINFORCEMENT LEARNING
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, Hado Van Hasselt
DecpMind
ABSTRACT
This paper introduces the Behaviour Suite for Reinforcement Learning, or bsuite for short. bsuite is collection of carefully-dssigne experiments that investigate core capabilities of reinforcement learning (RL) agents with two objectives. First, to collect clear, informative and scalable problems that capture key issues in the design of general and efficient learning al- gorithms. Second, to study agent behaviour through their performance on these shared benchmarks. To complement this effort, we open source github. com deepmind/ sappiindbbsuitet which automates evaluation and analysis of any agent on bsuite. This library facilitates reproducible and accessible research the core issues in RL, and ultimately the design of superior learning algorithms. Our code is Python, and easy to use within existing projects. We include examples with OpenAl Baselines, Dopamine as new reference implementations. Going forward, we hope to incorporate more excellent experiments from the research community, and commit to a periodic review of bsuite from a committee of prominent rescarchers.
well
INTRODUCTION
The reinforcement learning (RL) problem descríbes agent. interacting with an environ- ment with the goal of maximizing cumulative reward through time (Sutton & Barto 2017). Unlike other branches of control, the dynamics of the environment are not fully known the agent, but can be learned through experience. Unlike other branches of statistics and ma- chine learning, an RL agent must consider the effects of its actions upon future experience. An efficient RL agent must address three challenges simultancously:
1. Generalization: be able to learn elficiently from data collects.
2. Exploration: prioritize the right experience to learn from.
3. Long-term consequences: consider effects beyond a single timestep.
The great promise of reinforcement learning agents that can learn solve wide range of important problems. According to some definitions, an agent that can learn to perform at or above human level across wide varicty of tasks is artificial general intelligence (AGI) (Minsky 1961 Legg et al. 2007
Interest in artificial intelligence has undergone resurgence in recent years. Part of this interest is driven by the constant stream of innovation and success on high profile challenges previously deemed impossible for computer systems. Improvements in image recognition a clear example of these accomplishments, progressing from individual digit recognition (Le- Cun al. 1998). to mastering ImageNet in only few years (Deng et al. 2009 Krizhevsky ct al.] 2012 The advances in RL systems have been similarly impressive: from checkers [Samuel] 1959), to Backgammon (Tesauro) 1995), to Atari games (Mnih et al. (015a)), to
competing with professional players at DOTA (Pachocki et al., 2019) or StarCraft (Vinyals et al. 2019 and beating world champions at Go (Silver et al. 2016). Outside of playing
games, decision systems are increasingly guided by AT systems
Evans & Gao 2016
*Corresponding author ioshands@goolee.omm

---

Published as a conference paper at ICLR 2020
As we look towards the next great challenges for RL and AI, we need to understand our systems better (Henderson et al. 2017). This includes the scalability of our RL algorithms, the environments where we expect them to perform well, and the key issues outstanding in the design of a general intelligence system. We have the existence proof that a single self-learning RT. agent can master the game of Go purely from self-play (Silver et al.) 2018). We do not have a clear picture of whether such a learning algorithm will perform well at driving car, or managing a power plant. If we want take the next leaps forward, we need to continue to enhance our understanding.
1.1 PRACTICAL THEORY OFTEN LAGS PRACTICAL ALGORITHMS
The practical success of RT algorithms has built upon base of theory including gradient descent (Bottou 2010), temporal difference learning (Sutton 1988) and other foundational algorithmss Good theory provides insight into our algorithms beyond the particular, and a route towards general improvements beyond ad-hoc tinkering. As the psychologist Kurt Lewin said, 'there is nothing as practical as good theory' (Lewin) 1943). If we hope to use RL to (ackle important problems, then we must continue lo solidity these foundations. This need is particularly clear for RL with noulinear function approximation, or 'deep RL: At the same time, theory often lags practice, particularly in dilficult problems. We should not. avoid practical progress that can be made before we reach a full theoretical understanding. The successful development of algorithms and theory typically moves in tandem, with each side enriched by the insights of the other.
The evolution of neural network rescarch, deep learning, provides a poignant illustration of how theory and practice can develop together (LeCun et al. 2015). Many of the key ideas for deep learning have been around, and with successful demonstrations, for many years be- fore the modern deep learning explosion Rosenblati 1958 Tvakhnenko 1968 Fukushima 1979). However, most of these techniques remained outside the scope of developed learn- ing theory, partly due to their complex and non- non-convex loss functions. Much of the field turned away from these techniques in a 'ncural network winter', focusing instcad of function approximation under convex loss (Cortes & Vapnik 1995).. These convex methods were almost completely dominant until the emergence of benchmark problems, mostly for image recognition, where deep learning methods were able to clearly and objectively demonstrate their superiorily (LeCun el al., 1998. Krizhevsky el al., 2012). is only now, several years after these high profile successes, that learning theory has begun to turn its attention back to deep learning (Kawaguchi) 2016, Bartlett et al.) 2017 Belkin et al. 2018). The current theory of deep RI. is still in its infancy. In the absence of comprehensive theory, the com- munity needs principled benchmarks that help to develop an understanding of the strengths and weakenesses of our algorithms.
1.2 AN 'MNIST' FOR REINFORCEMENT LEARNING
In this paper we introduce the Behaviour Suite for Reinforcement Learning (or bsuite for short): a collection of experiments designed to highlight key aspects of agent scalability. Our aim is that these experiments can help provide a bridge between theory and practice, with benefits to both sides. These experiments embody fundamental issues, such as 'exploration' or 'memory' in a way that can be easily tested and iterated. For the development of theory, they force us to instantiate measurable and falsifiable hypotheses that we might later formalize into provable guarantees. While a full theory of RL may remain out of reach, the development of clear experiments that instantiate outstanding challenges for the field is powerful driver for progress. We provide description of the current suile of experiments and the key issues they identify in Section 2
Our work on bsuite is part of research process, rather than a final offering. We do not claim to capture all, or even most, of the important issues in RL. Instead, we hope to provide a simple library that collects the best available experiments, and makes them casily accessible to the community. As part of an ongoing commitment, we are forming bsuite committee that will periodically review the experiments included in the official bsuite release. We provide more details on what makes an 'excellent' experiment in Section 2 and on how to engage in their coestnn for future iterations in Section
2

---

Published as a conference paper at ICLR 2020
The Behaviour Suite for Reinforcement Learning is not replacement for 'grand challenge' undertakings in artificial intelligence, or a leaderboard to climb. Instead is a collection of diagnostic experiments designed to provide insight into key aspects of agent behaviour. Just as the MNTST dataset offers a clean, sanitised, test of image recognition as a stepping stone to advanced computer vision; too bsuite aims instantiate targeted experiments for the development of key RL capabilities.
The successful use of illustrative benchmark problems is not unique machine learning, and our work is similar in spirit to the Mixed Integer Programming Library (MIPLIB) (miplib2017). In mixed integer programming, and unlike linear programming. the majority of algorithmic advances have (so far) eluded theoretical analysis. In this field, MIPLIB serves to instantiate key properties of problems (or types of problems), and evaluation on MIPLIB is typical component of any new algorithm. We hope that bsuite can grow to perform a similar role RL research, least for those parts that continue to clude a unified theory of artificial intelligence. We provide guidelines for how researchers can use bsuite effectively in Section 3
1.3 OPEN SOURCE CODE, REPRODUCIBLE RESEARCH
As part of this project we open source github. com/ /esppmidd/bbuite. which instantiates all experiments in code and automales the evaluation and analysis of any RL agent on bsui te. This library serves to facilitale reproducible and accessible research on the core issues in reinforcement learning. includes:
Canonical implementations of all experiments, as described in Section 2
Reference implementations of several reinforcement learning algorithms.
Example usage of bsuite with alternative codcbases, including 'OpenAI Gym
Launch scripts for Google cloud that automate large scale compute at low cost A ready-made bsuite Jupyter notebook with analyses for all experiments.
Automated IАTX appendix, suitable for inclusion in conference submission.
We provide more details on code and usage in Section
We hope the Behaviour Suite for Reinforcement Learning, and its open source code, will provide significant value to the RL research community, and help to make key conceptual issues concrete and precise. bsuite can highlight bottlenecks in generul algorithms that are not amenable to hacks, and reveal properties and scalings of algorithms outside the scope of current analytical techniques. We believe this offers an avente towards great leaps on key issues, separate to the challenges of large-scale engineering (Nair et al. 2015). Further, bsuite facilitates clear, targeted and unified experiments across different. code frameworks. something that help to remedy issues of reproducibility in RL research (Tanner & White 2009
Henderson et al. 2017).
1.4
RELATED WORK
The Behaviour Suite for Reinforcement Learning fits into long history of RL benchmarks. From the beginning, rescarch into general learning algorithms has been grounded by the performance on specific environments (Sutton & Barto 2017). At first, these environments were typically motivated by small MDPs that instantiale the general learning problem. "CartPole" (Barto et al.) 1983) and 'MountainCar' (Moore 1990) are examples of classic benchmarks that has provided a testing ground for RL algorithm development. Similarly, when studying specific capabilities of learning algorithms, it has often been helpful to design diagnostic environments with that capability in mind. Examples of this include 'RiverSwim' for exploration (Strehll & Littman, 2008) or 'Taxi' for temporal abstraction (Dietterich) 2000). Performance in these environments provide a targeted signal for particular aspects of algorithm development.
As the capabilities or RL algorithms have advanced, so has the complexity of the benchmark problems. The Arcade Learning Environment (ALE) has been instrumental in driving
At August 2019 pricing, full bauite evaluation for our DQN implementation cost under $6.
3

---

Published as a conference paper at ICLR 2020
progress in deep RLL through surfacing dozens of Atari 2600 games as learning environments
Bellemare et al. 2013). Similar projects have been crucial to progress in contimuous control Duan et al. 2016 Tassa et al. 2018), proell-based RL (Wang et al. 2019) and even rich 3D games (Beattie et al.) 2016]- Performing well in these complex environments requires the integration of many core agent capabilities. We might think of these benchmarks as natural successors to 'CartPole" or 'MountainCar'.
The Behaviour Suite for Reinforcement Learning offers a complementary approach exist- ing benchmarks in RL, with several novel components:
bsuite experiments enforee a specific methodology for agent evaluation beyond just the environment definition. This is crucial for scientific comparisons and something that has become a major problem for many benchmark suites (Machado et al., 2017) (Section 2
2. bsuite aims to isolate core capabilities with largeled 'unit tests", rather her than integrale general learning ability. Other benchmarks evolve by increasing complexity, bsuite aims to remove all confounds from the core agent capabilities of interest (Section 3
3. bsuite experiments are designed with an emphasis on scalability rather than final per- formance. Previous 'unit. tests' (such as 'Taxi' or 'RiverSwim') are of fixed size, bsuite experiments are spocifically designed to vary the complexity smoothly (Section 2]
4. github. htmm//deepind/bbuuieee compatibility with RL agents not specifically designed for bsuite. Evaluating an agent on bsuite is practical even for agents designed for different benchmark (Section
has an extraordinary emphasis on the ease of use, and
2
EXPERIMENTS
This section outlines the experiments included in the Behaviour Suite for Reinforcement Learning 2019 releasc. In the context of bsuite, an ezperiment consists of three parts:
1. Environments: fixed set of environments determined by some parameters,
2. Interaction: a fixed regime of agent/environtt /environment interaction (e.g. 100 episodes).
3. Analysis: fixed procedure that maps agent behaviour to results and plots.
One crucial part of cach bsuite analysis defines 'score' that maps agent performance on the task to j0. 1]. This score allows for agent comparison 'at a glance', the Jupyter notebook includes further detailed analysis for each experiment. All experiments in bsuite only measure behavioural aspects of RL agents. This means that they only measure properties that can be observed in the environment, and are not internal to the agent. It is this choice that allows bsuite to easily generate and compare results across different algorithms and codebases. Researchers may still find it useful to investigate internal aspects of their agents on bsuite environments, but it is not part of the standard analysis.
Every current and future bsuite experiment should target some key issuc in RL. We aim for simple behavioural experiments, where agents that implement some concept well score better than those that don't. For an experiment to be included in bsuite it should embody five key qualities:
Targeted: performance in this task corresponds to a key issue in RT.
mufoouning/connfuiingg Simple: strips away confounding/ confusing factors in research.
Challenging: pushes agents beyond the normal range.
Scalable: provides insight on scalability, not performance on one environment.
Fast: itcration from launch to results in under 30min on standard CPU.
Where our current experiments fall short, we see this as an opportunity to improve the Behaviour Suite for Reinforcement Learning in future iterations. We can do this both through replacing experiments with improved variants, and through broadening the scope of issues that we consider.
We maintain the full description of each of our experiments through the code and accom- panying documentation at github. com//éeepinn//bautte In the following subsections, we pick two bsuite experiments to review in detail: 'memory length' and 'deep sea', and review these examples in detail. By presenting these experiments examples, we emphasize what we think makes bsuite a valuable tool for investigating core RL issues. We do provide high level summary of all other current experiments in Appendix A
4

---

Published as a conference paper at ICLR 2020
To accompany experiment descriptions, we present results and analysis comparing three baseline algorithms on bsuite: DQN (Mnih et al. 2015a), A2C (Mnih et al. 2016) and Bootstrapped DQN (Osband et al. 2016 As part of our open source eftort, we include full code for these agents and more at couite/baselinn through the automated bsuite Jupyter notebook, and give a flavour for the sort of agent comparisons that are made easy by bsuite.
All plots and analysis are generated
2.1 EXAMPLE EXPERIMENT: MEMORY LENGTH
Almost everyone agrees that a competent learning system requires memory, and almost everyone finds the concept memory intuitive. Nevertheless, it can be difficult to provide rigorous definition for memory. Even in human minds, there is evidence for distinct types of "memory' handled by distinct regions the brain (Milner et al. 1998). The assessment of memory only becomes more difficult to analyse in the context of general learning algorithms, which may differ greatly from human models of cognition. Which types of memory should analyse? How we inspect belief models for arbitrary learning systems? Our approach in bsuite is to sidestep these debates through simple behavioural experiments.
We refer to this experiment memory length; it is designed to test the number of sequential steps an agent can remember a single bit. The underlying environment is based on a stylized T-maze (O'Keefe & Dostrovsky 1971), parameterized by length N E N. Each episode lasts N steps with observation
{ (c,, t/N) ) for t = I....N N and action space A = [-1, +1}. The context C₁ ~ (.) and c for all 2. The reward rı 0 for all < N, but Sign(ax = c.). For the bsuite experiment we run the agent on sizes N = 1..., 100 exponentially spaced and look the average regret compared to optimal after 10k episodes. The summary 'score' is the percentage of runs for which the average regret is less than 75% of that achieved by a uniformly random policy.
cuc
choice
Figure 1: Illustration of the 'memory length' environment
Memory length a good bsuite experiment because is targeted, simple, challenging, scalable and fast. By construction, an agent that performs well on this task has mastered some use of memory over multiple timesteps. Our summary 'score' provides quick and dirty way to compare agent performance at high level. Our sweep over different lengths N provides empirical evidence about the scaling properties of the algorithm beyond a simple /fail Figure 2a gives quick snapshot of the performance of bascline algorithms. Unsurprisingly, actor-critic with a recurrent neural network greatly outperforms the feed- forward DQN and Bootstrapped DQN. Figure 2b gives us more detailed analysis of the same underlying data. Both DQN and Bootstrapped DQN are unable to learn anything for length
they lack functioning memory. A2C performs well for all N s 30 and essentially random for all N > 30, with quite a sharp cutoff. While it is not surprising that the recurrent agent outperforms feedforward architectures on memory task, Figure 2b gives an excellent insight into the scaling properties of this architecture. In this case, we have a clear explanation for the observed performance: the RNN agent was trained via backprop-through-
with length 30. bauite recovers empirical ivaluation of the scaling properties we would expect from theory.
2.2 EXAMPLE EXPERIMENT: DEEP SEA
Reinforcement learning calls for a sophisticated form of exploration called deep exploration (Osband et al. 2017). Just as agent seeking to 'exploit' must. consider the long term
5

---

Published as a conference paper at ICLR 2020
0.70
everage regret Trax
1015 ESP
(a) Summary score
(b) Examining learning scaling.
Figure 2: Selected output from bsuite evaluation on 'memory length..
consequences of its actions towards cumulative rewards, an agent seeking to 'explore' must consider how its actions position it to learn more effectively in future timesteps. The literature on efficient exploration broadly states that only agents that perform deep explo- ration expect polynomial sample complexity in learning (Kearns & Singh 2002). This literature has focused, for the most part, on uncovering possible strategics for deep explo- ration through studying the tabular setting analytically (Jaksch et al. 2010 Azar al. 2017 Our approach in bsuite is to complement this understanding through a series of behavioural experiments that highlight the need for efficient exploration.
The deep sea problem is implemented as an N x N grid with a one-hot encoding for state. The agent begins each episode in the top left corner of the grid and descends one row per timestep. Each episode terminates after N steps, when the agent reaches the bottom row. In each state there is a random but fixed mapping between actions A {0, 1} and the transitions 'left' and 'right'. At each timestep there is a small cost = -0.01/N of moving right, and r 0 for moving left. However, should the agent transition right every limestep of the episode it will be rewarded with an additional reward of +1. This presents particularly challenging exploration problem for two reasons. First, following the 'gradient" of small intermediate rewards leads the agent away from the optimal policy. Second, policy that explores with actions uniformly at random has probability 2- of reaching the rewarding state in any episode. For the bsuite experiment we the agent sizes N = 10, 12, .., 50 and look at the average regret compared to optimal after 10k episodes. The summary 'score' computes the percentage of runs for which the average regret drops below 0.9 faster than the 2 episodes expected by dithering.
2
3
N
N
Figure Deep-sea exploration: a simple example where deep exploration is critical.
Deep Sea a good bsuite experiment because is targeted, simple, challenging, scalable and fast. By construction, agent that performs well on this task has mastered some key properties of deep exploration. Our summary score provides a 'quick and dirty' way to compare agent performance at high level. Our sweep over different sizes N can help pro- vide empirical evidence of the scaling properties of an algorithm beyond a simple pass/fal fail. Figure presents example output comparing A2C, DQN and Bootstrapped DQN this

---

Published as a conference paper at ICLR 2020
gives quick snapshot of performance. As expected, only Bootstrapped DQN, which was developed for efficient exploration, scores well. Figure 4b gives a more de- tailed analysis of the same underlying data. When compare the scaling of learning with problem size N is clear that only Bootstrapped DQN scales gracefully to large problem sizes. Although experiment was only to size 50, the regular progression of learning times suggest we might expect this algorithm to scale towards N > 50.
task. Figure 4a
esperitenn actor CEEC THE
expersennt boot dip
IepertmenT
10000
7300
solved
50009
0.00
3000
Tiro
2500
wdoi
op
10
20
20
80
10
problem
00
50
10
20
30
10
(a) Summary score
(b) Examining learning scaling. Dashed line at 2 for reference.
Figure 4: Selected output from bsuite evaluation on 'deep sea'.
3 How TO USE BSUITE
This section describes some of the ways you can use bauite in your research and develop- ment of RL algorithms. Our aim is to present high-level description of some research and engineering use cases, rather than a tutorial for the code installation and use. We provide examples of specific investigat ions using bsuite in Appendixes D and E Section 4 pro- vides an outline of our code and implementation. Full details and tutorials are available at github. com/ hemm//depmnndbbuutte
A bsuite experiment is defined by a set of environments and number of episodes of inter- action. Since loading the environment via bsuite handles the logging automatically, any agent interacting with that environment will generate the data required for required for analysis through the Jupyter notebook we provide (Pérez & Granger] 2007j. Generating plots and analysis via the notebook only requires useTs to provide the path to the logged data. The 'radar plot' (Figure 5 at the start the notebook provides a snapshot agent behaviour, based on summary scores. The notebook also contains complete description of every experiment, summary scoring and in-depth analysis of each experiment. You can interact with the full report at bit. www.1yyybuit-aggnn.
Exploration
Credit Assigrment
Generalization
Busic
Memory
agent
actor criic
boot dgg
Scale
random
Nuise
Figure 5: We aggregate experiment performance with snapshot of core capabilities.
If you are developing an algorithm to make progress on fundamental issues in RL, rumning on bsuite provides simple way to replicate benchmark experiments in the field. Although
7

---

Published as a conference paper at ICLR 2020
many of these problems are 'small', in the sense that their solution does not necessarily require large neural architecture, they are designed to highlight key challenges in RL, Fur- ther, although these experiments do offer summary 'score', the plots and analysis are designed to provide much more information than just a leaderboard ranking. By using this common code and analysis, it is easy to benchmark your agents and provide reproducible and verifiable research.
If you are using RL as a tool to crack 'grand challenge' in AI, such as beating world champion at Go, then taking on bsuite gridworlds might seem like small fry. We argue that one of the most valuable uses of bsuite is as a diagnostic 'unit-test' for large-scale algo- rithm development. Imagine you believe that 'better exploration' is key to improving your performance on some challenge, but when you try your 'improved' agent, the performance does not improve. Does this mean your agent does not do good exploration? ? Or maybe that exploration is not the bottleneck in this problem? Worse still, these experiments might take days and thousands of dollars of compute to run, and even then the information you get might not be targeted to the key RL issues. Running on bauite, you can test key capabili- tics of your agent and diagnose potential improvements much faster, and more cheaply. For example, you might see that your algorithm completely fails at credit assignment beyond 20 steps. If this is the case, maybe this lack of credit-assignmet over long horizons is the bottleneck and not necessarily exploration. This can allow for much faster, and much better informed agent development just like a good suite of tests for software development.
Another benefit of bsuite is to disseminate your results more easily and engage with the research community. For example, if you write a conference paper targeting some improve- ment to hierarchical reinforcement learning, you will likely provide some justification for your results in terms of theorems or experiments targeted to this setting However, it is typically a large amount of work to evaluate your algorithm according to alternative metrics, such as exploration. This means that some fields may evolve without realising the connections and inst between related concepts. If you run on bsuite, you can automatically gener- ate a one- Appendix, with a link to a notebook report hosted online. This can help provide a scientific evaluation of your algorithmic changes, and help share your results in an casily-digestible format, compatible with ICML, ICLR and NeurIPS formatting. We
provide examples of these experiment reports in Appendices B and E
4 CODE STRUCTURE
To avoid discrepancies between this paper and the source code, we suggest that you take practical tutorials directly from github. aethub..cm/dee. wtt.ccmm/////eeeee dewp///depminn/uttt A good starting point is bit.1yybsuite--utorii
a Jupyter notebook where you can play the code right from your browser, without installing anything. The purpose of this section is to provide a high-level overview of the code that we open source. In particular, we want to stress is that bsuite is designed to be a library for RL research, not a framework. We provide implementations for all the environments, analysis, run loop and even baseline agents. However, it is not necessary that you make use of them all in order to make use of bsuite.
The recommended method is to implement your RL agent as a class that implements policy method for action selection, and an update method for learning from transitions and rewards. Then, simply pass your agent to our run loop, which enumerates all the necessary bsuite experiments and logs all the data automatically. the experiments and analysis will be handled automatically and generate your results via the included Jupyter notebook. We provide examples of running these scripts locally, and via Google cloud through our tutorials.
you do this, then all
If you have an existing codebase, you can still use bauite without migrating to our run loop or agent structure. Simply replace your environment with environment bsuite.
(bsuiteeee)e and add the flag bsuite-id to your code. You can then complete full bsuite evaluation by iterating over the bsuite ids defined in
2 A notable omission from the bsuite2019 release is the lack of any targeted experiments for 'hieaarchical reinforcement learning' (HRL). We invite the community to help curate excellent. experiments that can evaluate quality of HRL.
8

---

Published as a conference paper at ICLR 2020
sweep. SWEEP. Since the environments handle the logging themselves, your don't need any additional logging for the standard analysis. Although full bsuite includes many sepa- rate evaluations, single bsuite environment takes more than 30 minutes to run and the sweep is naturally parallel. As such, we recommend launching in parallel using multiple processes or multiple machines. Our examples include simple approach using Python's multiprocessing module with Google cloud compute. We also provide examples of running bsuite from OpenAl baselines (Dhariwal al. 2017) and Dopamine (Castro et al. 2018
Designing single RL agent compatible with diverse environments can cause problems, particularly for specialized neural networks. bsuite alleviates this problem by specifying an observation spec that surfaces the necessary information for adaptive network creation. By default, bsuite environments implement the dm env standards (Muldal et al. 2017]; but we also include wrapper for use through Openai gym (Brockman et al. 2016). However, if your agent hardcoded for a format, bsuite offers the option to output each environment with the observatio an_apec of your choosing via linear interpolation. This means that, if you are developing network suitable for Atari and particular observation.sen you can choose to swap in bsuite without any changes to your agent.
FUTURE ITERATIONS
This paper introduces the Behaviour Suite for Reinforcement Learning, and marks the start of its ongoing development. With our opensource effort, we chose specific collection of experiments as the bsuite2019 release, but expect this collection to evolve in future iterations. We are reaching out to researchers and practitioners to help collate the most informative, targeted, scalable and clear experiments possible for reinforcement lcarning. To do this, submissions should implement a sweep that determines the selection of environments to include and logs the necessary data, logether with an analysis that parses this data.
In order to review and collate these submissions we will be forming bsuite committee. The committee will meet anmually during the NeurIPS conference to decide which experiments will be included in the bsuite release. We are reaching out to a select group of researchers, and hope to build strong core formed across industry and academia. If you would like to submit an experiment bsuite or propose a committce member, you can do this via github pull request, or via email to bsuite. commi temmmitte@@mmil..c.mmmm com.
We believe that bsuite can be valuable tool for the RL community, and particularly for research in deep RL. So far, the great success of deep RL has been to leverage large amounts of computation to improve performance. With bsuite, we hope to leverage large- scale computation for improved understanding. By collecting clear, informative and scalable experiments; and providing accessible tools for reproducible evaluation we hope to facilitate progress in reinforcement learning research.
REFERENCES
Martín Abadi et al. FensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http://tensorfloz org/
Software available from tensorflow.org
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforce- ment learning. In Proc. of ICML, 2017.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. nporaail margin bounds for neural networks. In Advances in Newral Information Processing Systems 30, pp. 6241 6250, 2017.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions systems, man, and cybernetics, (5):834-844) 846, 1983.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. or Xiv preprint arXio: 1612 03801, 2016.
9

---

Published as a conference paper at ICLR 2020
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the birs-variance trade-off. arXiv preprint arXi::1812.1114 2018.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Envi- ronment: An Evaluation Platform for General Agents, Journal of Artificial Intelligence Research, 47:253-275 2013.
Léon Botton. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010 pp. 177 186. Springer, 2010.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. CoRR, abs/1606.0154// 2016. URL httt:://axxti org/ /bbb//
01540
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare. Dopamine: A Research Framework for Deep Reinforcement Learning. 2018. Jatt://///aaxx org/abs/1812.06110
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pallern recognition, pp. 248 255. Icce, 2009.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https:
//ithbb.. www.///peaa/aaaseleaaaaaaa 2017.
Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. Journal of artificial intelligence research, 13:227-300 2000.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016.
Richard Evans and Jim Gao. Deepmind Al reduces google data centre cooling bill by 40 htttsss: //deepmind. wwt.ccoom//bbeeeeeceeo aibeppminn--i--eduucee daateee---goooee-----a---e--eee deteleeddata--ttteeeeeeeeeeeeeee centre- boggtrrr--ool-----l11--0-000 2016.
Kunihiko Fukushima. Neural network model for mechanism of pattern recognition unaffected by
shift in position-neocconnitoon IEICE Technical Report, A, 62(10):658 665, 1979.
John C Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society: Series B (Methodological), (2):148-11) 164, 1979.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reiuforcement learning that matters. CoRR, abs/1709.06560, 2017. URL //atii//axivv hrrg/abs/10e 1109/1779...0 .06660)
Alexey Grigorevich Ivakhnenko. The group method of data of handling; rival of the method of stochastic approximation. Soviet Automatic Control, 13:43-55, 1968.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11(Apr):1563-1100, 2010.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 586-594, 2016.
M. Kearus and S. Singh. Near-optinal reinforcement learning in polyuomial time. Machine Learn- ing, 49, 2002.
Jeannette Kiefer and Jacob Wolfowitz. Stochastic estimation of the maximum of regression function. 1952.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL. http:://aaxivoorg/bbbbasse
10

---

-or-tioo--oo--dee--rreeee- Published as a conference paper at ICLR 2020
Alex Krizhevsky, Ilya Sulskever, and Geoffrey E Hinton. Imagenel classification with deep convolu- tional neural networks. In Advances Neural Information Processing Systems 25, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haflner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521 2015.
Shane Legg, Marcus Hutter, et al. A collection of definitions of intelligence. Frondiers in Artificial Intellgence and applications, 137:17, 2007.
Kurt Lewin, Psychology and the process of group living. The Journal of Social Psychology. 17(1): 113-131, 1943.
Xiuyuan and Benjamin Van Roy. Ensemble sampling. Advances in Neural Information Processing Systems, pp. 3260-3268, 2017.
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation protocols and open problems for general agents. Xiv preprint arviv: 709. 06009, 2017.
Breuda Milner, Larry R. Squire, and Eric R Kandel. Coguitive neuroscience and the study of memory. Neuron, 20(3):445-460 1998.
Marvin Minsky. Steps towards artificial intelligence. Proceedings of the IRF, 1961.
miplib2017. MIPLIB 2017, 2018. http://mipii..aibdde..
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level Control through Deep Reinforcement Learning. Nalure, 518(7540):559 533, 2015a.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep reinforcement learning. Nature, 518(7540))529 533, 2015b.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proc. of JCML, 2016.
Andrew William Moore. Efficient memory-base based learning for robot control. 1990.
Alistair Muldal, Yotam Doron, and John Aslanides. dm env. https:///gitabb.com//aaaaaaaaa env 2017.
Arun Nair, Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, et al. Massively Parallel Methods for Deep Reinforcement Learning. In JCML. Workshop on Deep Learning, 2015.
John O'Keefe and Jonathan Dostrovsky. The hippocampus from unit activity in the freely-moving rat. Brain research, 1971.
spatial map: preliminary evidence
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In Advances In Neural Information Processing Systems 29, pp. 4026-4034 2016.
lan Osband, Daniel Russo, Zheng Wen, and Benjamin Van Roy. Deep exploration via randomized value functions. aixiv preprint erXiv: Nal:1703.07604 2017.
Ian Osband, John Aslanides, and Albin Cassirer. reinforcement learning. In Advances in Neural Information Processing Systems 31, pp. 8627-8629. 8629. Curran Associates, Inc., 2018. URL htttp://papers wnnerrsiiiss..... ccps.cc/papprr 2080-randomized-ppa
Randomized prior functions for deep
-eernnteeeeennn.n..e pdf
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado Van Hasselt. Behaviour suite for reinforcement learning. 2019.
Jakub Pachocki, David Farhi, Szymon Sidor, Greg Brockman, Filip Wolski, Henrique PondAr, Jie Tang, Jonathan Raiman, Michael Petrov, Christy Dennison, Brooke Chan, Susan Zhang, RafaAC J Azzefowicz, and PrzemysAćaw DAZbiak. Openai five. luttpa://opemai. cow.iiiieicmmieiee//ee//a/a 2019.
11

---

Published as a conference paper at ICLR 2020
Fernando Pérez and Brian E. Granger. IPythom: system for interactive scientific compuling.
Computing Science and Engineering, 9(3):21-29, May 2007. ISSN 1521-9615. doi: 10.1109/ MCSE.2007.53. URL https:// org
Frank Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological revieu, 65(6):386, 1958.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, and lan Osband. A tutorial on Thompson sampling, arXie preprint arXiv:1707.00 02038, 2017.
AL Samuel. Some studies oin machine learning using the game chockers. IBM Journal of Researchand Development, 3:211-229, 1959.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Joannis Antonoglou, Veda Paaneershelvam,, Mare Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, (79(7787):484 489, 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6119):11100 1144, 2018. ISSN 0036-8075. doi:
sciencemag..... httpe://sciennee.cciinneeeg
a.r661....a66400.. URL https://scienc:
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 74(8):1309- 1333-133 2008.
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MTT Press, 2017.
R.S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3, 1988.
Brian Tanner and Adam White Rl-glue: languugg--ndeepuueett software for reinforeement-lerningg experiments. Journal of Machine Learning Research, 10(Sep):2133 12123-2113,,, 2136, 2009.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. XrXie preprint Xiv: 881.00690, 2018.
Gerald Tesauro. Temporal difference learning and ID-gammon. Communications of the ACM, 38 (8):5--68, 1995.
T. Ticleman and G. Hinton. Lecture 6.5 RmssPrpp Divide the gradient by rumning average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep Reinforcement Learning with Double Q-Learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
Oriol Vinyals, Igor Babuscliin, Junyoung Chung, Michael Mathieu, Jaderberg, al. AlphaS- tar: Mastering the Real-Time Strategy Game StarCraft II. https: /tdp:::///eemmidd com/blog/
maaastta-mmsterinngg real- weeae-ttiee time- strategy wwwegaataaeeaaaeeeeee starcraft- 2019.
-time-straeegy
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Éric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. CoRR, abs/1907.02057, 2019. URL http: //arxiv. http://aavv.oo/bb//1//0000 org/abs//901 02057
12

---

Published as a conference paper at ICLR 2020
A EXPERIMENT SUMMARY
This appendix outlines the experiments that make up the bauite 2019 release. In the interests of brevity, we provide only an outline of each experiment here. Full documentation for the environments, interaction and analysis are kept with code at github. www. http:/ccommmmmmd///uuutto
A.1 BASIC LEARNING
We begin with collection of very simple decision problems, and standard analysis that confirms an agent's competence al learning rewarding policy within them. We call these experiments 'basic', since they not particularly largeled at specific core issues in RL, but instead test general base level of competence we expect all general agents to attain.
A.1.1 SIMPLE BANDIT
component
description
environments
Finite-armed bandit with deterministic re- wards [0, 0... (Gittins 1979) 20 seeds.
interaction
10k episodes, record regret optimal.
score
regret normalized [random, oplimal] - [0.1]
issues
basic
A.1.2 MNIST
2
component
description
3
environments
Contextual bandit classification with rewards (LeCun et al. 1998). 20 seeds.
MNIST
3
86 < + 9
5
2 0
interaction
10k episodes, record average regret.
7 0 7
5 7
score
regret normalized [random, optimal] - [0.1]
4 4
&
O 4
issues
basic, generalization
A.1.3
CATCH
component
description
environments
10x5 Tetris-grid with single block falling per column. The agent can move left ffig/rhht the bottom row to 'catch' the block. 20 seeds.
interaction
10k episodes, record average regret.
score
regret normalized [random, optimal] + [0,1]
issues
basic, credit assignment
A.1.4 CARTPOLE
component
description
environments
Agent can move cart left reft/right on plane to keep balanced pole upright (Barto et al. 1983), 20 seeds.
interaction
10k episodes, record average regret.
regret normalized [random, optimal] + [0.1]
issues
basic, credit assignment, generalization
13

---

Published as a conference paper at ICLR 2020
A.1.5
MOUNTAIN CAR
component
description
environments
Agent drives underpowered car up hill Moore 1990), 20 seeds.
interaction
10k episodes, record average regret.
Score
regret normalized [random, optimal] + [0,1]
issues
basic, credit assignment, generalization
A.2 STOCHASTICITY
To investigate the robustness of RL agents to noisy rewards, we repeat the experiments from Section A.1 under differing levels of Gaussian noise. This time we allocate the 20 different seeds across 5 levols of Gaussian noise N(0, o²) for a = [0.1, 0.3, 1, 10] with seeds cach.
A.3 PROBLEM SCALE
To investigate the robustness of RL agents problem scale, repeat the experiments from Section A. under differing reward scales. This time we allocate the 20 different seeds across 5 levels of reward scaling, where multiply the observed rewards by À [0.01, 0.1, 1, 10, 100] with 4 seeds each.
A.4 EXPLORATION
As an agent interacts with its environment, it observes the outcomes that result from previ- ous states and actions, and learns about the system dynamics. This leads to a fundamental tradcoff: by exploring poorly-undersoood states and actions the agent can learn improve future performance, but it may attain better short-run performance by exploiting its exist- ing knowledge. Exploration is the challenge of prioritizing useful information for learning, and the experiments in this section are designed to necessitate efficient exploration for good performance.
A.4.1
DEEP SEA
component
description
environments
Deep sea chain environments size N=(5..50).
interaction
10k episodes, record average regret.
score
% of runs with regret 90% random
issues
exploration
A.4.2 STOCHASTIC DEEP SEA
component
description
environments
Deep chain environments with stochastic transitions, N(0,1) reward noise, size (0]
interaction
10k episodes, record average regret.
% of runs with regret 90% random
issues
exploration, stochasticity
14

---

Published as a conference paper at ICLR 2020
A.4.3
CARTPOLE SWINGUP
component
description
environments
Cartpole 'swing up' problem with sparse ward (Barto et 1983). heigh limit x= 0.5, 0.95].
interaction
1K episodes, record average regret.
score
of runs with average return exploration, generalization
issues
A.5 CREDIT ASSIGNMENT
Reinforcement learning extends contextual bandit. decision problem allow long term con- sequences in decision problems. This means that actions in one timestep effect dynamics in future timesteps. One of the challenges of this setting that of credit assignment, and the experiments in this section are designed to highlight these issucs.
A.5.1
UMBRELLA LENGTH
component
description
Fooeat
Unbrdke
environments
Stylized 'umbrella problem', where only the first decision matters and long chain of con- founding variables. Vary length 1..100 loga- rithmically.
interaction
1k episodes, record average regret.
score
regret normalized [random, optimal] + [0,1]
issues
credit assignment, stochasticity
A.5.2 UMBRELLA PEATURES
component
description
Feooot
Unbrdke
environments
Stylized 'umbrella problem', where only the first decision matters and long chain of con- founding variables. Vary features 10.100 loga- rithmically.
interaction
1k episodes, record average regret.
score
regret normalized [random, optimal] + [0.1]
Skine
Unclanl
issues
credit assignment, stochasticity
A.5.3
DISCOUNTING CHAIN
component
description
Ronadl
environments
Experiment designed to highlight issues of dis-
counting horizon.
Rewwall wirit
interaction
1k episodes, record average regret.
Ronadl aneddo
score
regret normalized [random, optimal] + [0.1]
issues
credit assignment
15

---

Published as a conference paper at ICLR 2020
A.6 MEMORY
Memory is the challenge that an agent should be able to curate an effective state represen- lation from a series of observations. In this section we review series of experiments in which agents with memory can perform much better than those that only have access to the immediate observation.
A.6.1 MEMORY LENGTII
component
description
environments
T-maze with single binary context, grow length 1..100 logarithmically.
interaction
1K episodes, record average regret.
score
regret normalized [random, optimal] + [0,1]
issues
credit assignment
A.6.2
MEMORY BITS
component
description
environments
T-maze with length vary number of bits remember 1..100 logarithmically.
dags
interaction
1k episodes, record average regret.
Makk exqeiee
PN
score
regret normalized [random, optimal] + [0.1]
issues
credit assignment
B BSUITE REPORT AS CONFERENCE APPENDIX
If you run an agent on bsuite, and you want to share these results as part of a conference submission, we make it easy share a single-page 'bsuite report' as part of your appendix. We provide a simple L'TeXfile that you can copy copy/pase /paste into your paper, and is compatible out-the-box with ICLR, ICML and NeurIPS style files. This single page summary displays the summary scores for experiment evaluations for one or more agents, with plots generated automatically from the included ipython notebook. In each report, two sections are left for the authors to fill in: one describing the variants of the agents examined and another to give some brief commentary on the results. We suggest that authors promote more in-depth anal- ysis to their main papers, simply link to hosted version of the full bsuite analysis online. You can find more details on our automated reports at llggg github.... com/ deepmi nd /bsuutet
The sections that follow example bsuite reports, that give report appendixes might be used. We bolieve that these simple reports can be a good complement to conference submissions in RL research, that 'sanity check' the elementary properties of algorithmic implementations. An added bonus of bsuite is that it is easy to set up a like for like experiment between agents from different 'Trameworks' in way that would be extremely laborious for an individual researcher. If you are writing a conference paper on a new RL algorithm, we believe that makes sense for you include bsuite report in the appendix by default.
example of how these
16

---

Published as a conference paper at ICLR 2020
bsuite report: benchmarking baseline agents
C
The Behaviour Suite for Reinforeement Learning, bsuite for short, is collection of carefully-desigued experiments that investigate core capabilities reinforcement learning (RL) agent. The aim of the bauite project is to collect clear, informative and scalable problems that capture key issues in the design of efficient and general learning algorithms and study agent behaviour through their performance
these shared benchmarks. This report provides a snapshot of agent performance on bauite2019, obtained by running experiments from github. com//deepminn//buuite (Osband et al. 2019).
C.1 AGENT DEFINITION
In this experiment inplementations are laken from
with default configura-
lions. We provide a brief summary of the agents run on bauite2019:
random: selects action uniformly at random each timestep.
dqn: Deep Q-networks (Mnih et al. 2015b).
boot_de dqn: bootstrapped DQN with prior networks (Osband al. 2016 2018
actor_critic_rve an actor critic with recurrent neural network (Miih et al. 2016
C.2 SUMMARY SCORES
Each bauite experiment outputs summary in [0.1]. We aggregate these scores according lo key experiment. lype, according the standard analysis notebook. detailed analysis each of these experiments may be found in a notebook hosted on Colaboratory bit. http//buiteaaggee
Deporation
Credit Aetigmment
Becis
Memory
best dep
Scili
ruddon
Figure snapshot of agent behaviour.
Figure
Score for cach bsuite experiment.
C.3 RESULTS COMMENTARY
random performs uniformly poorly, confirming the scores working as intended.
dqn performs well on basic tasks, and quite well on credit assignment, generalization, noise and scale. DQN performs extremely poorly across memory and exploration tasks. The feedforward
MLP has no mechanism for memory, and e=5%-greedy action selection is inefficient exploration. boot.dqn mostly identically to DQN, except for exploration where it greatly outperforms. This result matches our understanding of Bootstrapped DQN as a variant DQN designed to estimate uncertainty and use this to guide deep exploration.
actor actor-critic rnn typically performs worse than either DQN or Bootstrapped DQN on all tasks apart from memory. This agent is the only one able to perform better than random due to its recurrent network architecture.
17

---

Published as a conference paper at ICLR 2020
D
bsuite report:
optimization algorithm in DQN
The Behaviour Suite for Reinforcement Learning, bsuite for short, is collection of carefully-designed experiments that investigate core ca capabilities reinforcement learning (RL. agent. The aim of the bauite project is to collect clear, informative and scalable problems that capture key issues in the design of efficient and general learning algorithms and study agent behaviour through their performance these shared benchmarks. This report provides a snapshot agent performance on bauite2019, obtained by running experiments from github.
(Osband et al. 2019).
D.1 AGENT DEFINITION
All agents correspond to different instantiations of the DQN agent (Miih al. 2015b). as imple- mented in
but with differnet optimizers from Tensorllow (Abai et al. each case we tune learning rate to optimize performance on "basic' tasks from [le-1, 1e-2, 1e-3), keeping all other parameters constant default value.
2015 In
sgd: vanilla stochastic gradient descent with learning rate 1e-2 Kieffr & Wolfowitz, 1952
rmsprop: RMSProp with learning rate 1e-3 (Tieleman Hinton 2012).
adam: Adam with learning rate le-3 [Kingma & Ba 2015
D.2 SUMMARY SCORES
Each bsuite experiment outputs a summary score in [0,1]. We aggregate these scores by according
to key experiment type, according to the standard analysis notebook. A detailed analysis of each of these experiments may be found in notebook hosted on Colaboratory: bit bit..y/bbuute-optinn
Explocation
Credit Ansignment
Gesernlication
Monmory
optimiior
edwm
Scale
Neioo
Figure 8: A snapshot of agent behaviour.
Figure 9: Score for each bsuite experiment.
D.3 RESULTS COMMENTARY
Both RMSProp and Adam perform belter than SGD in every category. In most categories, Adam slightly outperforms RMSprop, although this difference much more minor. SGD performs par- ticularly badly on enviromments that require generalization and/or scale. This is not particularly surprising, since we expect the non-adaptive SGD may be more sensitive to learning rate optimiza- tion annealing.
that the differences particularly pronounced the cartpole domains. We hypothesize that this task requires more efficient neural network optimization, and non- adaptive SGD is prone to numerical issues.
In Figure 1
18

---

Published as a conference paper at ICLR 2020
bsuite report: ensemble size in Bootstrapped DQN
E
The Behaviour Suite for Reinforcement Learning, bsuite for short, is collection of carefully-designed experiments that investigate core ca capabilities reinforcement learning (RL. agent. The aim of the bauite project is to collect clear, informative and scalable problems that capture key issues in the design of efficient and general learning algorithms and study agent behaviour through their performance
these shared benchmarks. This report provides a snapshot agent performance on bauite2019, obtained by running experiments from github. com/deepmind/benee (Osband et al. 2019).
E.1 AGENT DEFINITION
this experiment, all agents correspond to different instantiations of a Bootstrapped DQN with prior networks (Osband et al. 2016 2018). We take the default implenentation from bsuite/ Dsuite/baseline baseeleeeeeeeeeeeeeeeeee We investigate the effect of the number of models used in the ensemble, sweep- ing over {1, 10, 30).
E.2 SUMMARY SCORES
Each bauite experiment outputs summary score in [0,1]. We aggregate these scores by according key experiment type, according the standard analysis notebook. A detailed analysis of each of these experiments may found a notebook hosted on Colaboratory: bit
Iapleration
Crodit Assigmment
Gerenaisatiin
Kemory
mim ensemble
Scale
10
Nusse
Figure 10: snapshot agent behaviour. Figure 11: Score for each bsuite experiment.
E.3 RESULTS COMMENTARY
Generally, increasing the size of the ensemble improves bauite performance across the board. However, we do see signficantly decreasing returns to ensemble size, that ensemble 30 does not perform much better than size 10. These results not predicted by the theoretical scaling of proven bounds (Lu & Van Rov 2017). but are consistent with previous empirical findings (Osband ett al.) 2017 Russo et al. 2017). The gains are most extreme the exploration tasks, where ensemble sizes less than 10 are not able to solve large 'deep sea' tasks, but larger ensembles solve them reliably.
Even for large ensemble sizes, implementation does not completely solve every cartpole swingup instance. Further examination learning curves suggests may be due some instability issues, which might be helped by using Double DQN to combat value overestimation (van Hasselt el al, 2016
19


Reviews:
[{"review_id": "rkxk2BR3YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Behaviour Suite for Reinforcement Learning\n\nIn this paper the authors provide a set of light-weighted but dedicated designed environments, so that researchers can use the environments as a quick indication of the ability of the proposed (or existing) algorithms.\nI think the paper is well-written, with the intuition clearly demonstrated.\n\nI tend to vote for rejection though, given that the novelty in the project is relatively limited.\nBut I believe in general it is a very valuable project that will be beneficial to future research and I would like to recommend for a workshop publication.\n\nPros:\n- The paper is well written, easy to understand. \n- Provide an industry level code base that can be used efficiently and easily.\nThe project will be of great value to the research community in the near future.\n\nCons:\n- The novelty of the project is relatively limited. \nThe proposed and implemented environments have been studied before.\n- No explicit conclusion from the evaluation.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rJxjmH6otS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a set of benchmarks for evaluating different aspects of reinforcement learning algorithms such as generalisation, exploration, and memory. The aim is to provide a set of simple environments to better understand the RL algorithms and also to provide a set of scores that summarise the performance in each respect. The code of the benchmark is also released.\n \nThe paper is well written and clear, and generally can provide a useful contribution. In particular, I like the idea of having a set of benchmarks which can be used for the diagnosis of RL algorithms. Having said this, I have the following concerns which are mostly related to the presentation of the paper. Given clarifications in an author response, I would be willing to increase the score. \n\n- Based on section 1.1 and elsewhere, it seems that the main driver for developing this benchmark has been connecting theory to practical algorithms (which in my opinion is an important step). However, how this can be achieved using the proposed benchmark is not shown in the paper. This can be for example showing how the generalisation score proposed here is linked to theoretical accounts. Or for example in section 2.1, by showing that the memory length 30 for RNN is related to the theoretical expectations. Alternatively, if linking theory and experiments is not the main driver of this work, then it seems a bit unclear what the point of presenting section 1.1 (and other related discussions) is within the context of the paper.\n\n- In terms of novelty, currently the differences between the current work and previous attempts to develop benchmarks is unclear (some examples are mentioned below). In general, a related work section is vital here, but missing in the paper. It should clearly state what the previous attempts in developing benchmarks are, their shortcomings, and how the current work addresses them. \n\n- Some statements in the paper sound more like opinions (which I happen to agree with) rather than something being based on the results of the paper. For example, \"We should not turn away from deep RL just because our current theory is not yet developed\". It is unclear how this statement is related to the results obtained in this work.\n\n- In section 3, I would like to see some real examples in which bsuite can be used for diagnosis. I find this application of bsuite (diagnosis) very interesting, but as it stands section 3 is more like a tutorial rather than providing a concrete example.\n\n- There are some aspects of RL which are specific to certain classes of RL. For example, in model-based RL, aspects such as the dynamics bottleneck and the planning horizon dilemma have been previously looked at, but are not presented in bsuite. How do the authors envision incorporating such aspects into their framework?\n\nMinor:\n- \"anything for length ¿ 1\" -> replace ¿\n\n- what is the dashed grey line in Fig 4b?\n\nReferences:\nDuan, Yan, et al. \"Benchmarking deep reinforcement learning for continuous control.\" International Conference on Machine Learning. 2016.\n\nBenchmarking Model-Based Reinforcement Learning, Wang et al, 2019.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgEVpbAFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents the « Behavior Suite for Reinforcement Learning » (bsuite), which is a set of RL tasks (called « experiments ») meant to evaluate an algorithm’s ability to solve various key challenges in RL. Importantly, these experiments are designed to run fast enough that one can benchmark a new algorithm within a reasonable amount of time (and money). They can thus be seen as a « test suite » for RL, limited to small toy problems but very useful to efficiently debug RL algorithms and get an overview of some of their key properties. The paper describes the motivation behind bsuite, shows detailed results from some classical RL algorithms on a couple of experiments, and gives a high-level overview of how the code is structured.\n\nI really believe such a suite of RL tasks can indeed be extremely useful to RL researchers developing new algorithms, and as a result I would like to encourage this initiative and see it published at ICLR to help it gain additional traction within the RL community.\n\nThe paper is easy to read, motivates well the reasons behind bsuite, and shows some convincing examples. However, in my opinion there remain a few important issues with this submission:\n\n1.\tThere is no « related work » section to position bsuite within the landscape of RL benchmarks (ex: DMLab, ALE / MinAtar, MuJoCo tasks, etc.). I believe it is important to add one.\n\n2.\tThe current collection of experiments appears to be quite limited. The authors acknowledge the lack of hierarchical RL, but what about other aspects like continuous control, parameterized actions, multi-agent, state representation learning, continual learning, transfer learning, imitation learning / inverse RL, self-play, etc? It is unclear to me whether the goal is to grow bsuite in all these directions (and more) over time, or if there is some kind of « boundary » the authors have in mind regarding the scope of bsuite. Regardless, the fact is that in its current form, bsuite appears to be suited only to a limited subset of current RL research.\n\n3.\tI wish an anonymized version of the code had been provided, so that reviewers could test it. In particular I wonder (a) if it is easy to setup and run under Windows, and (b) if it is straighforward to plug a bsuite experiment within an algorithm based on the popular OpenAI gym API (I think the latter is true from what is said at the end of Section 4, but I would have appreciated being able to try it out myself).\n\nAdditional minor remarks:\n•\tI noticed two anoymity-related issues with the provided links: (1) the Google Colab notebook revealed to me the name of its author when clicking the « Open in Playground » link to be able to run it, and (2) the bsuite-tutorial link asks for permission, which might let the authors access reviewer info. I would not hold it against the authors though as I believe these are genuine mistakes and they did their best to preserve anonymity.\n•\tt > 2 in Section 2.1 should probably be t >= 2\n•\tIn FIg. 2b the label for the y axis seems incorrect since good results are near 0\n•\tPlease explain what is the dashed grey line in Fig. 4b\n•\tI was unable to understand the last 2 sentences of Section 4\n•\tSections C.2, D.2 and E.2 all have the same plots\n•\tA few typos: incomplete sentence near bottom of p.3 (« the internal workings… »), « These assessment », « expeirments », « recurrant », « length ? 1 », « together with an analysis parses this data », « anonimize », « bsuite environments by implementing », « even if require »\n\nReview update: the authors have addressed my concerns, and I look forward to using bsuite in my research => review score increased to \"Accept\"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a platform for benchmarking and evaluating reinforcement learning algorithms.  While reviewers had some concerns about whether such a tool was necessary given existing tools, reviewers who interacted with the tool found it easy to use and useful. Making such tools is often an engineering task and rarely aligned with typical research value systems, despite potentially acting as a public good. The success or failure of similar tools rely on community acceptance and it is my belief that this tool surpasses the bar to be promoted to the community at a top tier venue.  
Decision: Accept (Spotlight)