Continual World: A Robotic Benchmark For Continual Reinforcement Learning

1 Introduction . Change is ubiquitous . Unsurprisingly , due to evolutionary pressure , humans can quickly adapt and creatively reuse their previous experiences . In contrast , although biologically inspired , deep learning ( DL ) models excel mostly in static domains that satisfy the i.i.d . assumption , as for example in image processing [ 28 , 49 , 10 , 40 ] , language modelling [ 52 , 11 ] or biological applications [ 47 ] . As the systems are scaled up and deployed in open-ended settings , such assumptions are increasingly questionable ; imagine , for example , a robot that needs to adapt to the changing environment and the wear-and-tear of its hardware . Continual learning ( CL ) , an area that explicitly focuses on such problems , has been gaining more attention recently . The progress in this area could offer enormous advantages for deep neural networks [ 19 ] and move the community closer to the long-term goal of building intelligent machines [ 20 ] . ⇤equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Evaluation of CL methods is challenging . Due to the sequential nature of the problem that disallows parallel computation , evaluation tends to be expensive , which has biased the community to focus on toy tasks . These are mostly in the domain of supervised learning , often relying on MNIST . In this work , we expand on previous discussions on the topic [ 45 , 16 , 30 , 46 ] and introduce a new benchmark , Continual World . The benchmark is built on realistic robotic manipulation tasks from Meta-World [ 54 ] , benefiting from its diversity but also being computationally cheap . Moreover , we provide shorter auxiliary sequences , all of which enable a quick research cycle . On the conceptual level , a fundamental difficulty of evaluating CL algorithms comes from the different desiderata for a CL solution . These objectives are often opposing each other , forcing practitioners to explicitly or implicitly make trade-offs in their algorithmic design that are data-dependent . Continual World provides more meaningful relationships between tasks , answering recent calls [ 19 ] to increase attention on forward transfer . Additionally , we provide an extensive evaluation of a spectrum of commonly used CL methods . It highlights that many approaches can deal relatively well with catastrophic forgetting at the expense of other desiderata , in particular forward transfer . This emphasizes our call for focusing on forward transfer and the need for more benchmarks that allow for common structure among the tasks . The main contribution of this work is a CL benchmark that poses optimizing forward transfer as the central goal and shows that existing methods struggle to outperform simple baselines in terms of the forward transfer capability . We release the code2 both for the benchmark and 7 CL methods , which aims to provide the community helpful tools to better understand the performance of existing and future solutions . We encourage to visit the website3 of the project and participate in the Continual World Challenge . 2 Related work . The field of continual learning has grown considerably in the past years , with numerous works forming new subfields [ 23 ] and finding novel applications [ 48 ] . For brevity , we focus only on the papers proposing RL-based benchmarks and point to selected surveys of the entire field . [ 19 ] provide a high-level overview of CL and argue that learning in a non-stationary setting is a fundamental problem for the development of AI , highlighting the frequent connections to neuroscience . On the other hand , [ 13 , 38 ] focus on describing , evaluating , and relating CL methods to each other , providing a taxonomy of CL solutions that we use in this work . The possibility of applying CL methods in reinforcement learning scenarios has been explored for a long time , see [ 25 ] for a recent review . However , no benchmark has been widely accepted by the community so far , which is the aim of this work . Below we discuss various benchmarks and environments considered in the literature . Supervised settings MNIST has been widely used to benchmark CL algorithms in two forms [ 27 ] . In the permuted MNIST , the pixels of images are randomly permuted to form new tasks . In the split MNIST , tasks are defined by classifying non-overlapping subsets of classes , e.g . 0 vs. 1 followed by 2 vs. 3 . A similar procedure has been applied to various image classification tasks like CIFAR-10 , CIFAR-100 , Omniglot or mini-ImageNet [ 2 , 46 , 5 ] . Another benchmark is CORe50 [ 31 ] , a dataset for continuous object recognition . Recent work [ 29 ] proposes a benchmark based on language modeling . We find that many of these benchmarks are challenging and allow to measure forgetting . However , we argue they are not geared towards measuring forward transfer or for highlighting important RL-specific characteristics of the CL problem . Atari The Atari 2600 suite [ 8 ] is a widely accepted RL benchmark . Sequences of different Atari games have been used for evaluating continual learning approaches [ 43 , 27 ] . Using Atari can be computationally expensive , e.g. , training a sequence of ten games typically requires 100M steps or more . More importantly , as [ 43 ] notes , these games lack a meaningful overlap , limiting their relevance for studying transfers . Continuous control [ 32 , 24 ] use continuous control tasks such as Humanoid or Walker2D . However , the considered sequences are short , and the range of experiments is limited . [ 34 ] use Meta-World tasks , similarly to us , for evaluations of their continual learning method , but the work is not aimed at building a benchmark . As such , it uses the Meta-World ’ s MT10 preset and 2https : //github.com/awarelab/continual_world 3https : //sites.google.com/view/continualworld/home does not provide an in-depth analysis of the tasks or other CL methods . Maze navigation A set of 3D maze environments is used in [ 43 ] . The map structure and objects that the agent needs to collect change between tasks . It is not clear , though , if the tasks provide enough diversity . [ 30 ] propose CRLMaze , 3D navigation scenarios for continual learning , which solely concentrate on changes of the visual aspects . StarCraft [ 45 ] present a StarCraft campaign ( 11 tasks ) to evaluate a high-level transfer of skills . The main drawback of this benchmark is excessive computational demand ( often more than 1B frames ) . Minecraft [ 50 ] propose simple scenarios within the Minecraft domain along with a hierarchical learning method . The authors phrase the problem as lifelong learning and do not use typical CL methods . Lifelong Hanabi [ 36 ] consider a multi-agent reinforcement learning setting based on Hanabi , a cooperative game requiring significant coordination between agents . On the other hand , we focus on the single agent setting with changing environment , which allows us to bypass the computational complexity needed to model interactions between agents and highlight issues connected to learning in a changing world . Causal World [ 3 ] propose an environment for robotic manipulation tasks which share causal structure . Although they investigate issues deeply connected to learning in a changing world , such as generalization to new tasks and curricula , they do not directly consider continual learning . Jelly Bean World [ 39 ] provide interesting procedurally generated grid world environments . The suite is configurable and can host a non-stationary setting . It is unclear , however , if such environments reflect the characteristics of real-world challenges . 3 Continual learning background . Continual learning ( CL ) is an area of research which focuses on building algorithms capable of handling non-stationarity . They should be able to sequentially acquire new skills and solve novel tasks without forgetting the previous ones . Such systems are desired to accommodate over extended periods swiftly , which is often compared to human capabilities and alternatively dubbed as lifelong learning . CL is intimately related to multi-task learning , curriculum learning , meta-learning , with some key differences . Multi-task assumes constant access to all tasks , thus ignoring non-stationarity . Curriculum learning focuses on controlling the task ordering and often the learning time-span . Metalearning , a large field of its own , sets the objective to develop procedures that allow fast adaptation within a task distribution and usually ignores the issue of non-stationarity . The CL objective is operationalized by the training and evaluation protocols . The former typically consists of a sequence of tasks ( their boundaries might be implicit and smooth ) . The latter usually involves measuring catastrophic forgetting , forward transfer , and backward transfer . The learning system might also have constrained resources : computations , memory , size of neural networks , and the volume of data samples . A fundamental observation is that the above aspects and desiderata are conflicting . For example , given unlimited resources , one might mitigate forgetting simply by storing everything in memory and paying a high computational cost of rehearsing all samples from the past . icant limitations of the current deep ( reinforcement ) learning methods . We conjecture that forward transfer might greatly improve the situation and possibly one day enable us to create systems with human-level cognition capabilities , in line with similar thoughts expressed in [ 19 ] . 4 Continual World benchmark . Continual World is a new benchmark designed to be a testbed for evaluating RL agents on the challenges advocated by the CL paradigm , described in Section 3 , as well as highlighting the RLspecific algorithmic challenges for CL ( see Section 6.1 ) . As such it is aimed at being valuable to both the CL and RL communities . Continual World consists of realistic robotic manipulation tasks , aligned in a sequence to enable the study of forward transfer . It is designed to be challenging while computationally accessible.4 The benchmark is based on Meta-World , a suite of robotic tasks already established in the community . This enables easy comparisons with the related fields of multi-task and meta-learning reinforcement learning , potentially highlighting one benefit of CL framing , namely that of dealing with different reward scales as we discuss more in detail in Appendix H. Continual World comes with open-source code that allows for easy development and testing of new algorithms and provides implementations of 7 existing algorithms . Finally , it allows highlighting RL-specific challenges for the CL setting . We believe that our work is a step in the right direction towards reliable benchmarks of CL . We realize , however , that it will need to evolve as the field progresses . We leave a discussion on future directions and limitations to Section 4.4 . 4.1 Metrics . To facilitate further discussion , we start with defining metrics . These are rather standard in the CL setting [ 42 ] . Assume pi ( t ) 2 [ 0 , 1 ] to be the performance ( success rate ) of task i at time t. As a measure of performance , we take the average success rate of achieving a goal specified by a given task when using randomized initial conditions and stochastic policies ( see also Section 4.3 ) .5 Each task is trained for = 1M steps . The main sequence has N = 20 tasks and the total sample budget is T = N · = 20M . The i-th task is trained during the interval t 2 [ ( i 1 ) · , i · ] . We report the following metrics : Average performance . The average performance at time t is ( see Figure 3 ) P ( t ) : = 1 N NX i=1 pi ( t ) . ( 1 ) Its final value , P ( T ) , is a traditional metric used in the CL research . This is the objective we use for tuning hyperparameters . We have P ( t ) 2 [ 0 , 1 ] for each t. Forward transfer . We measure the forward transfer of a method as the normalized area between its training curve and the training curve of the reference , single-task , experiment , see Figure 1 . Let pbi 2 [ 0 , 1 ] be the reference performance6 then the forward transfer for the task i , denoted by FTi , is FTi : = AUCi AUCbi 1 AUCbi , AUCi : = 1 Z i· ( i 1 ) · pi ( t ) dt , AUCbi : = 1 Z 0 pbi ( t ) dt , The average forward transfer for all tasks , FT , is defined as FT = 1 N NX i=1 FTi . ( 2 ) We note that FTi 1 and they might be negative . In our experiments , we also measure backward transfer . As it is negligible , see Appendix E.1 . Forgetting . For task i , we measure the decrease of performance after ending its training , i.e . Fi = pi ( i · ) pi ( T ) . ( 3 ) Similarly to FT , we report F = 1N PN i=1 Fi . We have Fi 1 for any i and consequently FT 1 . It is possible that Fi are negative , which would indicate backward transfer . We do not observe this in practice , see Appendix E.1 . 4We use 8-core machines without GPU . Training the CW20 sequence of twenty tasks takes about 100 hours . We also provide shorter 10 and 3 task sequences to speed up the experimental loop further . 5Stochastic evaluations are slightly more smooth and have little difference to the deterministic ones . 6Note that we avoid trivial tasks , for which AUCbi = 1 is making the metric ill-defined . Additionally , we acknowledge the dependency on the hyperparameters of the learning algorithm and that there are alternative quantities of interest like relative improvement in performance rather than faster learning .

Summary: This paper proposes a new benchmark for continual reinforcement learning. It modifies the Meta-World robotics benchmark that’s designed for meta- and multi-task RL evaluation by learning the tasks in sequence rather than together. This work is very timely as there lacks a common benchmark for continual/lifelong RL.  The benchmark consists of 10 tasks from Meta-World. To evaluate catastrophic forgetting, these 10 tasks are repeated for a sequence of length 20. For quicker iteration, there are also different triplets of tasks where the middle task might inhibit transfer to the last task.