Behaviour Suite for Reinforcement Learning

1 Introduction . The reinforcement learning ( RL ) problem describes an agent interacting with an environment with the goal of maximizing cumulative reward through time ( Sutton & Barto , 2017 ) . Unlike other branches of control , the dynamics of the environment are not fully known to the agent , but can be learned through experience . Unlike other branches of statistics and machine learning , an RL agent must consider the effects of its actions upon future experience . An efficient RL agent must address three challenges simultaneously : 1 . Generalization : be able to learn efficiently from data it collects . 2 . Exploration : prioritize the right experience to learn from . 3 . Long-term consequences : consider effects beyond a single timestep . The great promise of reinforcement learning are agents that can learn to solve a wide range of important problems . According to some definitions , an agent that can learn to perform at or above human level across a wide variety of tasks is an artificial general intelligence ( AGI ) ( Minsky , 1961 ; Legg et al. , 2007 ) . Interest in artificial intelligence has undergone a resurgence in recent years . Part of this interest is driven by the constant stream of innovation and success on high profile challenges previously deemed impossible for computer systems . Improvements in image recognition are a clear example of these accomplishments , progressing from individual digit recognition ( LeCun et al. , 1998 ) , to mastering ImageNet in only a few years ( Deng et al. , 2009 ; Krizhevsky et al. , 2012 ) . The advances in RL systems have been similarly impressive : from checkers ( Samuel , 1959 ) , to Backgammon ( Tesauro , 1995 ) , to Atari games ( Mnih et al. , 2015a ) , to competing with professional players at DOTA ( Pachocki et al. , 2019 ) or StarCraft ( Vinyals et al. , 2019 ) and beating world champions at Go ( Silver et al. , 2016 ) . Outside of playing games , decision systems are increasingly guided by AI systems ( Evans & Gao , 2016 ) . ∗Corresponding author iosband @ google.com . As we look towards the next great challenges for RL and AI , we need to understand our systems better ( Henderson et al. , 2017 ) . This includes the scalability of our RL algorithms , the environments where we expect them to perform well , and the key issues outstanding in the design of a general intelligence system . We have the existence proof that a single self-learning RL agent can master the game of Go purely from self-play ( Silver et al. , 2018 ) . We do not have a clear picture of whether such a learning algorithm will perform well at driving a car , or managing a power plant . If we want to take the next leaps forward , we need to continue to enhance our understanding . 1.1 Practical theory often lags practical algorithms . The practical success of RL algorithms has built upon a base of theory including gradient descent ( Bottou , 2010 ) , temporal difference learning ( Sutton , 1988 ) and other foundational algorithms . Good theory provides insight into our algorithms beyond the particular , and a route towards general improvements beyond ad-hoc tinkering . As the psychologist Kurt Lewin said , ‘ there is nothing as practical as good theory ’ ( Lewin , 1943 ) . If we hope to use RL to tackle important problems , then we must continue to solidify these foundations . This need is particularly clear for RL with nonlinear function approximation , or ‘ deep RL ’ . At the same time , theory often lags practice , particularly in difficult problems . We should not avoid practical progress that can be made before we reach a full theoretical understanding . The successful development of algorithms and theory typically moves in tandem , with each side enriched by the insights of the other . The evolution of neural network research , or deep learning , provides a poignant illustration of how theory and practice can develop together ( LeCun et al. , 2015 ) . Many of the key ideas for deep learning have been around , and with successful demonstrations , for many years before the modern deep learning explosion ( Rosenblatt , 1958 ; Ivakhnenko , 1968 ; Fukushima , 1979 ) . However , most of these techniques remained outside the scope of developed learning theory , partly due to their complex and non-convex loss functions . Much of the field turned away from these techniques in a ‘ neural network winter ’ , focusing instead of function approximation under convex loss ( Cortes & Vapnik , 1995 ) . These convex methods were almost completely dominant until the emergence of benchmark problems , mostly for image recognition , where deep learning methods were able to clearly and objectively demonstrate their superiority ( LeCun et al. , 1998 ; Krizhevsky et al. , 2012 ) . It is only now , several years after these high profile successes , that learning theory has begun to turn its attention back to deep learning ( Kawaguchi , 2016 ; Bartlett et al. , 2017 ; Belkin et al. , 2018 ) . The current theory of deep RL is still in its infancy . In the absence of a comprehensive theory , the community needs principled benchmarks that help to develop an understanding of the strengths and weakenesses of our algorithms . 1.2 An ‘ MNIST ’ for reinforcement learning . In this paper we introduce the Behaviour Suite for Reinforcement Learning ( or bsuite for short ) : a collection of experiments designed to highlight key aspects of agent scalability . Our aim is that these experiments can help provide a bridge between theory and practice , with benefits to both sides . These experiments embody fundamental issues , such as ‘ exploration ’ or ‘ memory ’ in a way that can be easily tested and iterated . For the development of theory , they force us to instantiate measurable and falsifiable hypotheses that we might later formalize into provable guarantees . While a full theory of RL may remain out of reach , the development of clear experiments that instantiate outstanding challenges for the field is a powerful driver for progress . We provide a description of the current suite of experiments and the key issues they identify in Section 2 . Our work on bsuite is part of a research process , rather than a final offering . We do not claim to capture all , or even most , of the important issues in RL . Instead , we hope to provide a simple library that collects the best available experiments , and makes them easily accessible to the community . As part of an ongoing commitment , we are forming a bsuite committee that will periodically review the experiments included in the official bsuite release . We provide more details on what makes an ‘ excellent ’ experiment in Section 2 , and on how to engage in their construction for future iterations in Section 5 . The Behaviour Suite for Reinforcement Learning is a not a replacement for ‘ grand challenge ’ undertakings in artificial intelligence , or a leaderboard to climb . Instead it is a collection of diagnostic experiments designed to provide insight into key aspects of agent behaviour . Just as the MNIST dataset offers a clean , sanitised , test of image recognition as a stepping stone to advanced computer vision ; so too bsuite aims to instantiate targeted experiments for the development of key RL capabilities . The successful use of illustrative benchmark problems is not unique to machine learning , and our work is similar in spirit to the Mixed Integer Programming Library ( MIPLIB ) ( miplib2017 ) . In mixed integer programming , and unlike linear programming , the majority of algorithmic advances have ( so far ) eluded theoretical analysis . In this field , MIPLIB serves to instantiate key properties of problems ( or types of problems ) , and evaluation on MIPLIB is a typical component of any new algorithm . We hope that bsuite can grow to perform a similar role in RL research , at least for those parts that continue to elude a unified theory of artificial intelligence . We provide guidelines for how researchers can use bsuite effectively in Section 3 . 1.3 Open source code , reproducible research . As part of this project we open source github.com/deepmind/bsuite , which instantiates all experiments in code and automates the evaluation and analysis of any RL agent on bsuite . This library serves to facilitate reproducible and accessible research on the core issues in reinforcement learning . It includes : • Canonical implementations of all experiments , as described in Section 2 . • Reference implementations of several reinforcement learning algorithms . • Example usage of bsuite with alternative codebases , including ‘ OpenAI Gym ’ . • Launch scripts for Google cloud that automate large scale compute at low cost.1 • A ready-made bsuite Jupyter notebook with analyses for all experiments . • Automated LATEX appendix , suitable for inclusion in conference submission . We provide more details on code and usage in Section 4 . We hope the Behaviour Suite for Reinforcement Learning , and its open source code , will provide significant value to the RL research community , and help to make key conceptual issues concrete and precise . bsuite can highlight bottlenecks in general algorithms that are not amenable to hacks , and reveal properties and scalings of algorithms outside the scope of current analytical techniques . We believe this offers an avenue towards great leaps on key issues , separate to the challenges of large-scale engineering ( Nair et al. , 2015 ) . Further , bsuite facilitates clear , targeted and unified experiments across different code frameworks , something that can help to remedy issues of reproducibility in RL research ( Tanner & White , 2009 ; Henderson et al. , 2017 ) . 1.4 Related work . The Behaviour Suite for Reinforcement Learning fits into a long history of RL benchmarks . From the beginning , research into general learning algorithms has been grounded by the performance on specific environments ( Sutton & Barto , 2017 ) . At first , these environments were typically motivated by small MDPs that instantiate the general learning problem . ‘ CartPole ’ ( Barto et al. , 1983 ) and ‘ MountainCar ’ ( Moore , 1990 ) are examples of classic benchmarks that has provided a testing ground for RL algorithm development . Similarly , when studying specific capabilities of learning algorithms , it has often been helpful to design diagnostic environments with that capability in mind . Examples of this include ‘ RiverSwim ’ for exploration ( Strehl & Littman , 2008 ) or ‘ Taxi ’ for temporal abstraction ( Dietterich , 2000 ) . Performance in these environments provide a targeted signal for particular aspects of algorithm development . As the capabilities or RL algorithms have advanced , so has the complexity of the benchmark problems . The Arcade Learning Environment ( ALE ) has been instrumental in driving 1At August 2019 pricing , a full bsuite evaluation for our DQN implementation cost under $ 6 . progress in deep RL through surfacing dozens of Atari 2600 games as learning environments ( Bellemare et al. , 2013 ) . Similar projects have been crucial to progress in continuous control ( Duan et al. , 2016 ; Tassa et al. , 2018 ) , model-based RL ( Wang et al. , 2019 ) and even rich 3D games ( Beattie et al. , 2016 ) . Performing well in these complex environments requires the integration of many core agent capabilities . We might think of these benchmarks as natural successors to ‘ CartPole ’ or ‘ MountainCar ’ . The Behaviour Suite for Reinforcement Learning offers a complementary approach to existing benchmarks in RL , with several novel components : 1. bsuite experiments enforce a specific methodology for agent evaluation beyond just the environment definition . This is crucial for scientific comparisons and something that has become a major problem for many benchmark suites ( Machado et al. , 2017 ) ( Section 2 ) . 2. bsuite aims to isolate core capabilities with targeted ‘ unit tests ’ , rather than integrate general learning ability . Other benchmarks evolve by increasing complexity , bsuite aims to remove all confounds from the core agent capabilities of interest ( Section 3 ) . 3. bsuite experiments are designed with an emphasis on scalability rather than final performance . Previous ‘ unit tests ’ ( such as ‘ Taxi ’ or ‘ RiverSwim ’ ) are of fixed size , bsuite experiments are specifically designed to vary the complexity smoothly ( Section 2 ) . 4. github.com/deepmind/bsuite has an extraordinary emphasis on the ease of use , and compatibility with RL agents not specifically designed for bsuite . Evaluating an agent on bsuite is practical even for agents designed for a different benchmark ( Section 4 ) .

Summary: In this paper, the authors propose a set of benchmarks for evaluating different aspects of reinforcement learning algorithms such as generalisation, exploration, and memory. The aim is to provide a set of simple environments to better understand the RL algorithms and also to provide a set of scores that summarise the performance in each respect. The code of the benchmark is also released.