Behaviour Suite for Reinforcement Learning

1 Introduction . The reinforcement learning ( RL ) problem describes an agent interacting with an environment with the goal of maximizing cumulative reward through time ( Sutton & Barto , 2017 ) . Unlike other branches of control , the dynamics of the environment are not fully known to the agent , but can be learned through experience . Unlike other branches of statistics and machine learning , an RL agent must consider the effects of its actions upon future experience . An efficient RL agent must address three challenges simultaneously : 1 . Generalization : be able to learn efficiently from data it collects . 2 . Exploration : prioritize the right experience to learn from . 3 . Long-term consequences : consider effects beyond a single timestep . The great promise of reinforcement learning are agents that can learn to solve a wide range of important problems . According to some definitions , an agent that can learn to perform at or above human level across a wide variety of tasks is an artificial general intelligence ( AGI ) ( Minsky , 1961 ; Legg et al. , 2007 ) . Interest in artificial intelligence has undergone a resurgence in recent years . Part of this interest is driven by the constant stream of innovation and success on high profile challenges previously deemed impossible for computer systems . Improvements in image recognition are a clear example of these accomplishments , progressing from individual digit recognition ( LeCun et al. , 1998 ) , to mastering ImageNet in only a few years ( Deng et al. , 2009 ; Krizhevsky et al. , 2012 ) . The advances in RL systems have been similarly impressive : from checkers ( Samuel , 1959 ) , to Backgammon ( Tesauro , 1995 ) , to Atari games ( Mnih et al. , 2015a ) , to competing with professional players at DOTA ( Pachocki et al. , 2019 ) or StarCraft ( Vinyals et al. , 2019 ) and beating world champions at Go ( Silver et al. , 2016 ) . Outside of playing games , decision systems are increasingly guided by AI systems ( Evans & Gao , 2016 ) . ∗Corresponding author iosband @ google.com . As we look towards the next great challenges for RL and AI , we need to understand our systems better ( Henderson et al. , 2017 ) . This includes the scalability of our RL algorithms , the environments where we expect them to perform well , and the key issues outstanding in the design of a general intelligence system . We have the existence proof that a single self-learning RL agent can master the game of Go purely from self-play ( Silver et al. , 2018 ) . We do not have a clear picture of whether such a learning algorithm will perform well at driving a car , or managing a power plant . If we want to take the next leaps forward , we need to continue to enhance our understanding . 1.1 Practical theory often lags practical algorithms . The practical success of RL algorithms has built upon a base of theory including gradient descent ( Bottou , 2010 ) , temporal difference learning ( Sutton , 1988 ) and other foundational algorithms . Good theory provides insight into our algorithms beyond the particular , and a route towards general improvements beyond ad-hoc tinkering . As the psychologist Kurt Lewin said , ‘ there is nothing as practical as good theory ’ ( Lewin , 1943 ) . If we hope to use RL to tackle important problems , then we must continue to solidify these foundations . This need is particularly clear for RL with nonlinear function approximation , or ‘ deep RL ’ . At the same time , theory often lags practice , particularly in difficult problems . We should not avoid practical progress that can be made before we reach a full theoretical understanding . The successful development of algorithms and theory typically moves in tandem , with each side enriched by the insights of the other . The evolution of neural network research , or deep learning , provides a poignant illustration of how theory and practice can develop together ( LeCun et al. , 2015 ) . Many of the key ideas for deep learning have been around , and with successful demonstrations , for many years before the modern deep learning explosion ( Rosenblatt , 1958 ; Ivakhnenko , 1968 ; Fukushima , 1979 ) . However , most of these techniques remained outside the scope of developed learning theory , partly due to their complex and non-convex loss functions . Much of the field turned away from these techniques in a ‘ neural network winter ’ , focusing instead of function approximation under convex loss ( Cortes & Vapnik , 1995 ) . These convex methods were almost completely dominant until the emergence of benchmark problems , mostly for image recognition , where deep learning methods were able to clearly and objectively demonstrate their superiority ( LeCun et al. , 1998 ; Krizhevsky et al. , 2012 ) . It is only now , several years after these high profile successes , that learning theory has begun to turn its attention back to deep learning ( Kawaguchi , 2016 ; Bartlett et al. , 2017 ; Belkin et al. , 2018 ) . The current theory of deep RL is still in its infancy . In the absence of a comprehensive theory , the community needs principled benchmarks that help to develop an understanding of the strengths and weakenesses of our algorithms . 1.2 An ‘ MNIST ’ for reinforcement learning . In this paper we introduce the Behaviour Suite for Reinforcement Learning ( or bsuite for short ) : a collection of experiments designed to highlight key aspects of agent scalability . Our aim is that these experiments can help provide a bridge between theory and practice , with benefits to both sides . These experiments embody fundamental issues , such as ‘ exploration ’ or ‘ memory ’ in a way that can be easily tested and iterated . For the development of theory , they force us to instantiate measurable and falsifiable hypotheses that we might later formalize into provable guarantees . While a full theory of RL may remain out of reach , the development of clear experiments that instantiate outstanding challenges for the field is a powerful driver for progress . We provide a description of the current suite of experiments and the key issues they identify in Section 2 . Our work on bsuite is part of a research process , rather than a final offering . We do not claim to capture all , or even most , of the important issues in RL . Instead , we hope to provide a simple library that collects the best available experiments , and makes them easily accessible to the community . As part of an ongoing commitment , we are forming a bsuite committee that will periodically review the experiments included in the official bsuite release . We provide more details on what makes an ‘ excellent ’ experiment in Section 2 , and on how to engage in their construction for future iterations in Section 5 . The Behaviour Suite for Reinforcement Learning is a not a replacement for ‘ grand challenge ’ undertakings in artificial intelligence , or a leaderboard to climb . Instead it is a collection of diagnostic experiments designed to provide insight into key aspects of agent behaviour . Just as the MNIST dataset offers a clean , sanitised , test of image recognition as a stepping stone to advanced computer vision ; so too bsuite aims to instantiate targeted experiments for the development of key RL capabilities . The successful use of illustrative benchmark problems is not unique to machine learning , and our work is similar in spirit to the Mixed Integer Programming Library ( MIPLIB ) ( miplib2017 ) . In mixed integer programming , and unlike linear programming , the majority of algorithmic advances have ( so far ) eluded theoretical analysis . In this field , MIPLIB serves to instantiate key properties of problems ( or types of problems ) , and evaluation on MIPLIB is a typical component of any new algorithm . We hope that bsuite can grow to perform a similar role in RL research , at least for those parts that continue to elude a unified theory of artificial intelligence . We provide guidelines for how researchers can use bsuite effectively in Section 3 . 1.3 Open source code , reproducible research . As part of this project we open source github.com/deepmind/bsuite , which instantiates all experiments in code and automates the evaluation and analysis of any RL agent on bsuite . This library serves to facilitate reproducible and accessible research on the core issues in reinforcement learning . It includes : • Canonical implementations of all experiments , as described in Section 2 . • Reference implementations of several reinforcement learning algorithms . • Example usage of bsuite with alternative codebases , including ‘ OpenAI Gym ’ . • Launch scripts for Google cloud that automate large scale compute at low cost.1 • A ready-made bsuite Jupyter notebook with analyses for all experiments . • Automated LATEX appendix , suitable for inclusion in conference submission . We provide more details on code and usage in Section 4 . We hope the Behaviour Suite for Reinforcement Learning , and its open source code , will provide significant value to the RL research community , and help to make key conceptual issues concrete and precise . bsuite can highlight bottlenecks in general algorithms that are not amenable to hacks , and reveal properties and scalings of algorithms outside the scope of current analytical techniques . We believe this offers an avenue towards great leaps on key issues , separate to the challenges of large-scale engineering ( Nair et al. , 2015 ) . Further , bsuite facilitates clear , targeted and unified experiments across different code frameworks , something that can help to remedy issues of reproducibility in RL research ( Tanner & White , 2009 ; Henderson et al. , 2017 ) . 1.4 Related work . The Behaviour Suite for Reinforcement Learning fits into a long history of RL benchmarks . From the beginning , research into general learning algorithms has been grounded by the performance on specific environments ( Sutton & Barto , 2017 ) . At first , these environments were typically motivated by small MDPs that instantiate the general learning problem . ‘ CartPole ’ ( Barto et al. , 1983 ) and ‘ MountainCar ’ ( Moore , 1990 ) are examples of classic benchmarks that has provided a testing ground for RL algorithm development . Similarly , when studying specific capabilities of learning algorithms , it has often been helpful to design diagnostic environments with that capability in mind . Examples of this include ‘ RiverSwim ’ for exploration ( Strehl & Littman , 2008 ) or ‘ Taxi ’ for temporal abstraction ( Dietterich , 2000 ) . Performance in these environments provide a targeted signal for particular aspects of algorithm development . As the capabilities or RL algorithms have advanced , so has the complexity of the benchmark problems . The Arcade Learning Environment ( ALE ) has been instrumental in driving 1At August 2019 pricing , a full bsuite evaluation for our DQN implementation cost under $ 6 . progress in deep RL through surfacing dozens of Atari 2600 games as learning environments ( Bellemare et al. , 2013 ) . Similar projects have been crucial to progress in continuous control ( Duan et al. , 2016 ; Tassa et al. , 2018 ) , model-based RL ( Wang et al. , 2019 ) and even rich 3D games ( Beattie et al. , 2016 ) . Performing well in these complex environments requires the integration of many core agent capabilities . We might think of these benchmarks as natural successors to ‘ CartPole ’ or ‘ MountainCar ’ . The Behaviour Suite for Reinforcement Learning offers a complementary approach to existing benchmarks in RL , with several novel components : 1. bsuite experiments enforce a specific methodology for agent evaluation beyond just the environment definition . This is crucial for scientific comparisons and something that has become a major problem for many benchmark suites ( Machado et al. , 2017 ) ( Section 2 ) . 2. bsuite aims to isolate core capabilities with targeted ‘ unit tests ’ , rather than integrate general learning ability . Other benchmarks evolve by increasing complexity , bsuite aims to remove all confounds from the core agent capabilities of interest ( Section 3 ) . 3. bsuite experiments are designed with an emphasis on scalability rather than final performance . Previous ‘ unit tests ’ ( such as ‘ Taxi ’ or ‘ RiverSwim ’ ) are of fixed size , bsuite experiments are specifically designed to vary the complexity smoothly ( Section 2 ) . 4. github.com/deepmind/bsuite has an extraordinary emphasis on the ease of use , and compatibility with RL agents not specifically designed for bsuite . Evaluating an agent on bsuite is practical even for agents designed for a different benchmark ( Section 4 ) .

Summary: This paper presents the « Behavior Suite for Reinforcement Learning » (bsuite), which is a set of RL tasks (called « experiments ») meant to evaluate an algorithm’s ability to solve various key challenges in RL. Importantly, these experiments are designed to run fast enough that one can benchmark a new algorithm within a reasonable amount of time (and money). They can thus be seen as a « test suite » for RL, limited to small toy problems but very useful to efficiently debug RL algorithms and get an overview of some of their key properties. The paper describes the motivation behind bsuite, shows detailed results from some classical RL algorithms on a couple of experiments, and gives a high-level overview of how the code is structured.