Continual World: A Robotic Benchmark For Continual Reinforcement Learning

1 Introduction . Change is ubiquitous . Unsurprisingly , due to evolutionary pressure , humans can quickly adapt and creatively reuse their previous experiences . In contrast , although biologically inspired , deep learning ( DL ) models excel mostly in static domains that satisfy the i.i.d . assumption , as for example in image processing [ 28 , 49 , 10 , 40 ] , language modelling [ 52 , 11 ] or biological applications [ 47 ] . As the systems are scaled up and deployed in open-ended settings , such assumptions are increasingly questionable ; imagine , for example , a robot that needs to adapt to the changing environment and the wear-and-tear of its hardware . Continual learning ( CL ) , an area that explicitly focuses on such problems , has been gaining more attention recently . The progress in this area could offer enormous advantages for deep neural networks [ 19 ] and move the community closer to the long-term goal of building intelligent machines [ 20 ] . ⇤equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Evaluation of CL methods is challenging . Due to the sequential nature of the problem that disallows parallel computation , evaluation tends to be expensive , which has biased the community to focus on toy tasks . These are mostly in the domain of supervised learning , often relying on MNIST . In this work , we expand on previous discussions on the topic [ 45 , 16 , 30 , 46 ] and introduce a new benchmark , Continual World . The benchmark is built on realistic robotic manipulation tasks from Meta-World [ 54 ] , benefiting from its diversity but also being computationally cheap . Moreover , we provide shorter auxiliary sequences , all of which enable a quick research cycle . On the conceptual level , a fundamental difficulty of evaluating CL algorithms comes from the different desiderata for a CL solution . These objectives are often opposing each other , forcing practitioners to explicitly or implicitly make trade-offs in their algorithmic design that are data-dependent . Continual World provides more meaningful relationships between tasks , answering recent calls [ 19 ] to increase attention on forward transfer . Additionally , we provide an extensive evaluation of a spectrum of commonly used CL methods . It highlights that many approaches can deal relatively well with catastrophic forgetting at the expense of other desiderata , in particular forward transfer . This emphasizes our call for focusing on forward transfer and the need for more benchmarks that allow for common structure among the tasks . The main contribution of this work is a CL benchmark that poses optimizing forward transfer as the central goal and shows that existing methods struggle to outperform simple baselines in terms of the forward transfer capability . We release the code2 both for the benchmark and 7 CL methods , which aims to provide the community helpful tools to better understand the performance of existing and future solutions . We encourage to visit the website3 of the project and participate in the Continual World Challenge . 2 Related work . The field of continual learning has grown considerably in the past years , with numerous works forming new subfields [ 23 ] and finding novel applications [ 48 ] . For brevity , we focus only on the papers proposing RL-based benchmarks and point to selected surveys of the entire field . [ 19 ] provide a high-level overview of CL and argue that learning in a non-stationary setting is a fundamental problem for the development of AI , highlighting the frequent connections to neuroscience . On the other hand , [ 13 , 38 ] focus on describing , evaluating , and relating CL methods to each other , providing a taxonomy of CL solutions that we use in this work . The possibility of applying CL methods in reinforcement learning scenarios has been explored for a long time , see [ 25 ] for a recent review . However , no benchmark has been widely accepted by the community so far , which is the aim of this work . Below we discuss various benchmarks and environments considered in the literature . Supervised settings MNIST has been widely used to benchmark CL algorithms in two forms [ 27 ] . In the permuted MNIST , the pixels of images are randomly permuted to form new tasks . In the split MNIST , tasks are defined by classifying non-overlapping subsets of classes , e.g . 0 vs. 1 followed by 2 vs. 3 . A similar procedure has been applied to various image classification tasks like CIFAR-10 , CIFAR-100 , Omniglot or mini-ImageNet [ 2 , 46 , 5 ] . Another benchmark is CORe50 [ 31 ] , a dataset for continuous object recognition . Recent work [ 29 ] proposes a benchmark based on language modeling . We find that many of these benchmarks are challenging and allow to measure forgetting . However , we argue they are not geared towards measuring forward transfer or for highlighting important RL-specific characteristics of the CL problem . Atari The Atari 2600 suite [ 8 ] is a widely accepted RL benchmark . Sequences of different Atari games have been used for evaluating continual learning approaches [ 43 , 27 ] . Using Atari can be computationally expensive , e.g. , training a sequence of ten games typically requires 100M steps or more . More importantly , as [ 43 ] notes , these games lack a meaningful overlap , limiting their relevance for studying transfers . Continuous control [ 32 , 24 ] use continuous control tasks such as Humanoid or Walker2D . However , the considered sequences are short , and the range of experiments is limited . [ 34 ] use Meta-World tasks , similarly to us , for evaluations of their continual learning method , but the work is not aimed at building a benchmark . As such , it uses the Meta-World ’ s MT10 preset and 2https : //github.com/awarelab/continual_world 3https : //sites.google.com/view/continualworld/home does not provide an in-depth analysis of the tasks or other CL methods . Maze navigation A set of 3D maze environments is used in [ 43 ] . The map structure and objects that the agent needs to collect change between tasks . It is not clear , though , if the tasks provide enough diversity . [ 30 ] propose CRLMaze , 3D navigation scenarios for continual learning , which solely concentrate on changes of the visual aspects . StarCraft [ 45 ] present a StarCraft campaign ( 11 tasks ) to evaluate a high-level transfer of skills . The main drawback of this benchmark is excessive computational demand ( often more than 1B frames ) . Minecraft [ 50 ] propose simple scenarios within the Minecraft domain along with a hierarchical learning method . The authors phrase the problem as lifelong learning and do not use typical CL methods . Lifelong Hanabi [ 36 ] consider a multi-agent reinforcement learning setting based on Hanabi , a cooperative game requiring significant coordination between agents . On the other hand , we focus on the single agent setting with changing environment , which allows us to bypass the computational complexity needed to model interactions between agents and highlight issues connected to learning in a changing world . Causal World [ 3 ] propose an environment for robotic manipulation tasks which share causal structure . Although they investigate issues deeply connected to learning in a changing world , such as generalization to new tasks and curricula , they do not directly consider continual learning . Jelly Bean World [ 39 ] provide interesting procedurally generated grid world environments . The suite is configurable and can host a non-stationary setting . It is unclear , however , if such environments reflect the characteristics of real-world challenges . 3 Continual learning background . Continual learning ( CL ) is an area of research which focuses on building algorithms capable of handling non-stationarity . They should be able to sequentially acquire new skills and solve novel tasks without forgetting the previous ones . Such systems are desired to accommodate over extended periods swiftly , which is often compared to human capabilities and alternatively dubbed as lifelong learning . CL is intimately related to multi-task learning , curriculum learning , meta-learning , with some key differences . Multi-task assumes constant access to all tasks , thus ignoring non-stationarity . Curriculum learning focuses on controlling the task ordering and often the learning time-span . Metalearning , a large field of its own , sets the objective to develop procedures that allow fast adaptation within a task distribution and usually ignores the issue of non-stationarity . The CL objective is operationalized by the training and evaluation protocols . The former typically consists of a sequence of tasks ( their boundaries might be implicit and smooth ) . The latter usually involves measuring catastrophic forgetting , forward transfer , and backward transfer . The learning system might also have constrained resources : computations , memory , size of neural networks , and the volume of data samples . A fundamental observation is that the above aspects and desiderata are conflicting . For example , given unlimited resources , one might mitigate forgetting simply by storing everything in memory and paying a high computational cost of rehearsing all samples from the past . icant limitations of the current deep ( reinforcement ) learning methods . We conjecture that forward transfer might greatly improve the situation and possibly one day enable us to create systems with human-level cognition capabilities , in line with similar thoughts expressed in [ 19 ] . 4 Continual World benchmark . Continual World is a new benchmark designed to be a testbed for evaluating RL agents on the challenges advocated by the CL paradigm , described in Section 3 , as well as highlighting the RLspecific algorithmic challenges for CL ( see Section 6.1 ) . As such it is aimed at being valuable to both the CL and RL communities . Continual World consists of realistic robotic manipulation tasks , aligned in a sequence to enable the study of forward transfer . It is designed to be challenging while computationally accessible.4 The benchmark is based on Meta-World , a suite of robotic tasks already established in the community . This enables easy comparisons with the related fields of multi-task and meta-learning reinforcement learning , potentially highlighting one benefit of CL framing , namely that of dealing with different reward scales as we discuss more in detail in Appendix H. Continual World comes with open-source code that allows for easy development and testing of new algorithms and provides implementations of 7 existing algorithms . Finally , it allows highlighting RL-specific challenges for the CL setting . We believe that our work is a step in the right direction towards reliable benchmarks of CL . We realize , however , that it will need to evolve as the field progresses . We leave a discussion on future directions and limitations to Section 4.4 . 4.1 Metrics . To facilitate further discussion , we start with defining metrics . These are rather standard in the CL setting [ 42 ] . Assume pi ( t ) 2 [ 0 , 1 ] to be the performance ( success rate ) of task i at time t. As a measure of performance , we take the average success rate of achieving a goal specified by a given task when using randomized initial conditions and stochastic policies ( see also Section 4.3 ) .5 Each task is trained for = 1M steps . The main sequence has N = 20 tasks and the total sample budget is T = N · = 20M . The i-th task is trained during the interval t 2 [ ( i 1 ) · , i · ] . We report the following metrics : Average performance . The average performance at time t is ( see Figure 3 ) P ( t ) : = 1 N NX i=1 pi ( t ) . ( 1 ) Its final value , P ( T ) , is a traditional metric used in the CL research . This is the objective we use for tuning hyperparameters . We have P ( t ) 2 [ 0 , 1 ] for each t. Forward transfer . We measure the forward transfer of a method as the normalized area between its training curve and the training curve of the reference , single-task , experiment , see Figure 1 . Let pbi 2 [ 0 , 1 ] be the reference performance6 then the forward transfer for the task i , denoted by FTi , is FTi : = AUCi AUCbi 1 AUCbi , AUCi : = 1 Z i· ( i 1 ) · pi ( t ) dt , AUCbi : = 1 Z 0 pbi ( t ) dt , The average forward transfer for all tasks , FT , is defined as FT = 1 N NX i=1 FTi . ( 2 ) We note that FTi 1 and they might be negative . In our experiments , we also measure backward transfer . As it is negligible , see Appendix E.1 . Forgetting . For task i , we measure the decrease of performance after ending its training , i.e . Fi = pi ( i · ) pi ( T ) . ( 3 ) Similarly to FT , we report F = 1N PN i=1 Fi . We have Fi 1 for any i and consequently FT 1 . It is possible that Fi are negative , which would indicate backward transfer . We do not observe this in practice , see Appendix E.1 . 4We use 8-core machines without GPU . Training the CW20 sequence of twenty tasks takes about 100 hours . We also provide shorter 10 and 3 task sequences to speed up the experimental loop further . 5Stochastic evaluations are slightly more smooth and have little difference to the deterministic ones . 6Note that we avoid trivial tasks , for which AUCbi = 1 is making the metric ill-defined . Additionally , we acknowledge the dependency on the hyperparameters of the learning algorithm and that there are alternative quantities of interest like relative improvement in performance rather than faster learning .

Summary: A benchmark for continual reinforcement learning is provided, where the main task contains 20 subtasks. A certain number of choices are made such as the agent being aware of task change, not carrying forward the replay buffer, and learning policies with separate output heads per task. The evaluation was overall performance, forgetting, and forward transfer. Seven learning methods are compared extensively and results are provided.