Continual World: A Robotic Benchmark For Continual Reinforcement Learning
Authors: Maciej Wolczyk
Michał Zając
Razvan Pascanu
Łukasz Kuciński
Piotr Miłoś
Venue: NeurIPS 2021 Poster (2021)
Keywords: continual learning
reinforcement learning
benchmark

Abstract:   Continual learning (CL) --- the ability to continuously learn, building on previously acquired knowledge --- is a natural requirement for long-lived autonomous reinforcement learning (RL) agents. While building such agents, one needs to balance opposing desiderata, such as constraints on capacity and compute, the ability to not catastrophically forget, and to exhibit positive transfer on new tasks. Understanding the right trade-off is conceptually and computationally challenging, which we argue has led the community to overly focus on catastrophic forgetting.  In response to these issues, we advocate for the need to prioritize forward transfer and propose Continual World, a benchmark consisting of realistic and meaningfully diverse robotic tasks built on top of Meta-World  as a testbed. Following an in-depth empirical evaluation of existing CL methods, we pinpoint their limitations and highlight unique algorithmic challenges in the RL setting. Our benchmark aims to provide a meaningful and computationally inexpensive challenge for the community and thus help better understand the performance of existing and future solutions. Information about the benchmark, including the open-source code, is available at https://sites.google.com/view/continualworld.

Continual World: A Robotic Benchmark For Continual Reinforcement Learning
Maciej Wolezyk"
Michal Zajac" Jagiellonian University Kraków, Poland emzajacQgmail.com com
Jagiellonian University
Kraków, Poland
maciej wolczykkdocttoal. uj edu.pl
Razvan Pascanu DeepMind London, UK razp@google.com com
Lukasz Kuciński
Piotr Miloš
Polish Academy of Sciences
Polish Academy of Sciences, University of Oxford,
Warsaw, Poland 1kucinski @impan.
deepsense.Rt
Warsaw, Poland pmilos@impan pl
Abstract
Continual learning (CL) the ability to continuously learn, building on previ- ously acquired knowledge is natural requirement for long-lived autonomous reinforcement learning (RL) agents. While building such agents, one needs to balance opposing desiderata, such as constraints on capacity and compute, the ability to not catastrophically forget, and to exhibit positive transfer new tasks. Understanding the right trade-off is conceptually and computationally challenging, which argue has led the community to overly focus on catastrophic forgetting. response to these issues, we advocate for the need prioritize forward transfer and propose Continual World, benchmark consisting of realistic and meaningfully diverse robotic tasks built on top of Werld [A4I] as a testbed. Following an in-depth empirical evaluation of existing CL methods, we pinpoint their limitations and highlight unique algorithmic challenges the RL setting. Our benchmark aims provide meaningful and computationally inexpensive challenge for the community and thus help better understand the performance of existing and future solutions. Information about the benchmark, including the open-source code, is available at https:: :/sps::///stee //sitesssooglle com/ homm/view/ae wtt.ee//oootiimawwoolllll
Introduction
Change is ubiquitous. Unsurprisingly, due to evolutionary pressure, humans can quickly adapt and creatively reuse their previous experiences. contrast, although biologically inspired, deep learning (DL) models excel mostly in static domains that satisfy the i.i.d. assumption, as for example in image processing 2
49
TO 40 language modelling [52] | biological applications [47] As
the systems are scaled up and deployed in open- open-ended settings, such assumptions are increasingly questionable; imagine, for example, robot that needs adapt to the changing environment and the wear-and-tear of its hardware. Continual learning (CL), an area that explicitly focuses on such problems, has been gaining more attention recently. The progress in this area could offer enormous advantages for deep neural networks [19 and move the community closer to the long-ten termttemmmm goal of
building intelligent machines 20
equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Evaluation of CL methods is challenging. Due to the sequential nature of the problem that disallows parallel computation, evaluation tends to be expensive, which has biased the community to focus on toy tasks. These are mostly in the domain of supervised learning, often relying on MNIST. In this work, we expand on previous discussions on the topic [45]
4d and introduce a new benchmark, Continual World. The benchmark is built on realistic robotic manipulation tasks from Meta-World [54], benefiting from its diversity but also being computationally cheap. Moreover, we provide shorter auxiliary sequences, all of which enable a quick research cycle. On the conceptual level, a fundamental difficulty of evaluating CL algorithms comes from the different desiderata for CL solution. These objectives are often opposing each other, forcing practitioners to explicitly or implicitly make trade-offs in their algorithmic design that are data-dependent. Continual World provides more meaningful relationships between tasks, answering recent calls [9] to increase attention on forward transfer.
Additionally, we provide an extensive evaluation of a spectrum of commonly used CL methods. It highlights that many approaches can deal relatively well with catastrophic forgetting at the expense of other desiderata, in particular forward transfer. This emphasizes our call for focusing on forward transfer and the need for more benchmarks that allow for common structure among the tasks.
The main contribution of this work is a CL benchmark that poses optimizing forward transfer as the central goal and shows that existing methods struggle to outperform simple baselines in terms of the forward transfer capability. We release the code both for the benchmark and 7 CL methods, which aims to provide the community helpful tools to better understand the performance of existing and future solutions. We encourage to visit the website of the project and participate in the Continual World Challenge.
2 Related work
The field of continual learning has grown considerably in the past years, with numerous works forming new subfields (23 and finding novel applications 4 For brevity, we focus only on the papers proposing RL-based benchmarks and point to selected surveys of the entire field. [19] provide a high-level overview of CL. and argue that learning in a non-stationary setting a fundamental problem for the development of AI, highlighting the frequent connections to neuroscience. On the
other hand, [1 38 focus on describing, evaluating, and relating CL methods to each other, providing a taxonomy of CL solutions that we use in this work.
The possibility of applying CL methods in reinforcement learning scenarios has been explored for a long time, see [25] for a recent review. However, no benchmark has been widely accepted by the community so far, which the aim of this work. Below we discuss various benchmarks and environments considered in the literature.
Supervised settings MNIST has been widely used benchmark CL. algorithms in two forms 2 In the permuted MNIST, the pixels of images are randomly permuted to form new tasks. In the split MNIST, tasks are defined by classifying non-overlapping subsets of classes, c.g. vs. followed by 2 vs. 3. A similar procedure has been applied to various image classification tasks like CIFAR- 10, CIFAR-100, Omniglot or mini-ImageNet 2 46 Another benchmark is CORe50 BI a dataset for continuous object recognition. Recent work 29 proposes benchmark based on language modeling. We find that many of these benchmarks are challenging and allow to measure forgetting. However, we argue they are not geared towards measuring forward transfer or for highlighting important RL-specific characteristics of the CL problem.
Atari The Atari 2600 suite g is a widely accepted RI. henchmark. Sequences of different Atari games have been used for evaluating continual learning approaches 43 27
Using Atari can be computationally expensive, e.g., training a sequence of ten games typically requires 100M steps or more. More importantly, as [ notes, these games lack a meaningful overlap, limiting their relevance for studying transfers. Continuous control [32] 24 use continuous control tasks such as Humanoid
or Walker2D. However, the considered sequences are short, and the range of experiments is limited. [4l use Meta- Meta-World tasks, similarly to us, for evaluations of their continual learning method, but the work not aimed at building benchmark. As such, it uses the Meta- World's s MT10 10 preset and

---

does not provide an in-depth analysis of the tasks or other CL methods. Maze navigation A set of 3D maze environments is used in [43]. The map structure and objects that the agent needs to collect change between tasks. It is not clear, though, if the tasks provide enough diversity. R0 propose CRL.Maee, 3D navigation scenarios for continual learning, which solely concentrate on changes of the visual aspects. StarCraft :5 present StarCraft campaign (11 tasks) to evaluate high-level
transfer of skills. The main drawback of this benchmark is excessive computational demand (often more than 1B frames). Minecraft [50] propose simple scenarios within the Minecraft domain along with a hierarchical learning method. The authors phrase the problem as lifélong learning and do not use typical CL methods. Lifelong Hanabi [36] consider multi-agent reinforcement learning setting based on Hanabi, a cooperative game requiring sigmificant coordination between agents. On the other hand, we focus the single agent setting with changing environment, which allows us to bypass the computational complexity needed to model interactions between agents and highlight issues connected to learning in a changing world. Causal World 3 propose environment for robotic manipulation tasks which share causal structure. Although they investigate issues deeply connected to learning in changing world, such generalization new tasks and currieula, they do not directly consider continual learning. Jelly Bean World generated grid world environments. The suite is configurable and can host non-stationary setting. is unclear, however, if such environments reflect the characteristics of real-world challenges.
(39] provide interesting procedurally
3 Continual learning background
Continual learning (CL) is handling non-stationarity. They should be able to sequentially acquire new skills and solve novel tasks without forgetting the previous ones. Such systems are desired to accommodate over extended periods swiftly, which is often compared to human capabilities and alternatively dubbed as lifelong learning. CL is intimately related multi-task learning, curriculum learning, meta-learning, with some key differences. Multi-task assumes constant access to all tasks, thus ignoring non-stationarity. Curriculum learning focuses on controlling the task ordering and often the learning time-span. Meta- learning, large field of its own, sets the objective to develop procedures that allow fast adaptation within a task distribution and usually ignores the issue of non-stationarity.
of research which focuses on building algorithms capable of
The CL objective is operationalized by the training and evaluation protocols. The former typically consists of a sequence of tasks (their boundaries might be implicit and smooth). The latter usually involves measuring catastrophic forgetting, forward transfer, and backward transfer. The learning system might also have constrained resources: compulations, memory, size of neural networks, and the volume of data samples. A fundamental observation is that the above aspects and desiderata are conflicting. For example, given unlimited resources, one might mitigate forgetting simply by storing everything in memory and paying high computational cost of rehearsing all samples from the past.
Another pair of objectives that are problematic for current methods are forgetting and forward transfer. For neural networks, existing methods propose to limit network plasticity. These alle- viate the problem of forgetting, however, at the cost of choking the further learning process. We advocate for more nuanced approaches. Impor- tantly, make the transfer possible, our bench- mark is composed of related tasks. We also put modest bounds on resources. This requirement is in line with realistic scenarios, demanding computationally efficient adaptation and infer- ence. In a broader sense, we hope to address a data efficiency challenge, one of the most signif- icant limitations of the current deep (reinforcement) learning methods. We conjecture that forward transfer might greatly improve the situation and possibly one day enable us to create systems with human-level cognition capabilities, in line with similar thoughts expressed in [19]
FT
Reference Method
Maximum
Figure 1: Left graph shows task BEG-UNPPUG- SIDE- vI and the right graph presents forward transfer from SHELF-PLACE- I to PEG-UNPLUG- SIDE- v.. In this case FT = 0.10.
3

---

4 Continual World benchmark
Continual World is a new benchmark designed to be testbed for evaluating RL agents on the challenges advocated by the CL. paradigm, described in Section 31 as well as highlighting the RL- specific algorithmic challenges for CL (see Section 6. As such it aimed being valuable to both the CL and RL communities. Continual World consists of realistic robotic manipulation tasks, aligned in a sequence to enable the study forward transfer. It designed to be challenging while The benchmark is based on Meta-World, suite of robotic tasks already
computationally accessible I established the community. This enables easy comparisons with the related fields of multi-task and meta-learning reinforcement learning, potentially highlighting one benefit of CL framing, namely that of dealing with different reward scales as we discuss more in detail in Appendix H. Continual World comes with open-source code that allows for easy development and testing of new algorithms and provides implementations of 7 existing algorithms. Finally, it allows highlighting RL-specific challenges for the CL setting. We believe that our work is step in the right direction towards reliable benchmarks of CL. We realize, however, that it will need to evolve as the field progresses. We leave a discussion on future directions and limitations to Section 4.4
4.1 Metrics
To facilitate further discussion, we start with defining metrics. These are rather standard in the CL setting 12]. Assume pi()) E [0, 1] to be the performance (success rate) of task at time t. As measure of performance, we take the average success rate of achieving goal specified by a given task when using randomized initial conditions and stochastic policies (see also Section 4.3 Each task is trained for A = 1M steps. The main sequence has N = 20 tasks and the total sample budget
T N A 20M. The i-th task is trained during the interval t [(i (i-1)) A. A]. We report the following metrics:
Average performance. The average performance time is (see Figure 1 N N
P(t)
Σ (t()).
Its final value, P(T), is traditional metric used in the CL research. This is the objective we use for tuning hyperparameters. We have P(t) E [0, 1] for each t.
Forward transfer. We measure the forward transfer of method the normalized area between its training curve and the training curve of the reference, single-task, experiment, see Figure Let p [0, 1] be the reference performance then the forward transfer for the task i, denoted by FT;, is
AUC;
AUC
A
1 AUC = 2 //² (t)dt,
FT,
AUC, := A
pc(t)dt,
AUC
(1-1--
The average forward transfer for all tasks, FT, is defined as
FT
Σ FT. N
1
(2)
We note that FT, < 1 and they might be negative. In our experiments, we also measure backward transfer. As it is negligible, see Appendix E.
Forgetting. For task i, we measure the decrease of performance after ending its training, i.e. F. = p,(i A) PT()).
Similarly to FT, we report F == N ΣN i=1 F.. We have F; < 1 for any and consequently FT < 1. It is possible that F; are negative, which would indicate backward transfer. We do not observe this in practice, see Appendix E.T
'We use 8-core machines without GPU. Training the CW20 sequence of twenty tasks takes about 100 hours. We also provide shorter 10 and 3 task sequences to speed up the experimental loop further. 'Stochastic evaluations are slightly more smooth and have little difference the deterministic ones. "Note that we avoid trivial tasks, for which AUC, 1 is making the metric ill-defined. Additionally, we acknowledge the dependency on the hyperparameters of the learning algorithm and that there are alternative quantities of interest like relative improvement in performance rather than faster learning.

---

4.2 Continual World tasks
This section describes the composition of Continual World benchmark and the rationale behind its design. We decided to base on Meta-Worte [54). fairly new but already established robotic benchmark for multi-task and meta reinforcement learning. From a practical standpoint, Meta-World utilizes the open-source MuJoCo physics engine [51]. prized for speed and accuracy. Meta-World provides 50 distinct manipulation tasks with everyday objects using a simulated robotic Sawyer arm. Although the tasks vary significantly, the structure and semantics of observation and action spaces remain the same, allowing for transfer between tasks. Each observation is a 12-dimensional vector containing (1r, y, 2) coordinates of the robot's gripper and objects of interest in the scene. The 4- 4-dimensional action space describes the direction of the arm's 's movement in the next step and the gripper actuator delta. Reward functions are shaped make each task solvable. In evaluations, we use a binary success metric based on the distance of the task-relevant object to its goal position. This metric is interpretable and enables comparisons between tasks. For more details about the rewards and evaluation metrics, see
54 Section 4.2, Section 4.3].
CW20, CW10, triplets sequences The core of our benchmark is CW20 sequence. Out of 50 tasks defined in Merl-World, we picked those that are not too easy or too hard in the assumed sample budget A -I.. Aiming to strike a balance between the difficulty of the benchmark and computational requirements, we selected 10 tasks. The tasks and their ordering were based on the transfer matrix (see the next paragraph),
that there is high variation of forward transfers (both in the whole list and locally). We refer to these ordered tasks as CW10, 10, and CW20 is CW10 10 repeated twice. We recommend using CW20 for final evaluation; however, CW 10 is already very informative in most cases. Due to brevity constraints, we present an ablation with an alternative ordering of the tasks and a longer sequence of 30 tasks in Appendix Additionally, to facilitate fast development cycle, we propose a set of triplets, sequences of three tasks which exhibit interesting learning dynamics.
G however, these experiments do not alter our findings.
The CW10 sequence is: HAMMER-VI, FUSH-WALL-VI. FAUCET-CLOSE-VI, PUSH-BACK-VI, STICK- PULL- VI, MANDLE-PRESS, -IDES-- PUSH-VI, SHELF-PLACE-VI, WINDOW-CLOOSE-VA PEG-UNPLAUG-SDDE-V..
Transfer matrix Generally, the relationship between tasks and its impact on learning dynamics of neural networks is hard to quamtify, where semantic similarity does not typically lead to transfer [14] To this end, we consider a minimal setting, in which we finetune on task 12 a model pretrained on ti, using the same protocol the benchmark (e.g., different output heads,
Section 4.3 This
provides neural katwork-centric insight into the relationship between tasks summarized in Figure 2. and allows us measure low-level transfer between tasks, i.e., the ability of the model reuse previously acquired features. See Appendix D for more results and extended discussion.
Notice that there are only few negative forward transfer cases, and those are of a rather small magnitude (perhaps unsurprisingly, as the tasks are related). There are also visible patterns in the matrix. For instance, some tasks such as PEG- UNPLUG-SIDE-V or PUSH BACK-V benefit from relatively large forward transfer, (almost) irrespective of the first task. Furthermore, the average forward transfer given the second task (columns) is more variable than the correspond- ing quantity for the first task (rows).
second task
0.4.2
0.13
0.64
0.58
0.24
0.48
003
0.58
0.32 0.47
0.13 0.030.055.37
0.46
040.88
0.36
0.78
0.00 0.000/5000 0...0000000006666.0....0 0.100.22
0.033 0.150.400..0
0:09
0.42
0.02
0.39
0.32
0.33
frss task
0.....10.45...
0.47
0.19
0-38
04
S0ue 0.00 0.3303300
3.64
0.53
0.140.44022
0.12
4.03
0.20 9,060.47
0.700.8
0.55
0.30000
0:42
0.81
64
Note that some transfers on the diagonal (i.c., between the same tasks) are relatively small. We made a detailed analysis of possible reasons, which revealed that the biggest negative impact is due to the replay buffer resets, which seems, however, unavoidable for off-diagonal cases, Section 6. for details.
0:33
0.42
0.300.40
0.33
8 10.00 0.24 8.69
0.0.
0.20.4
0.36
08.30-0 0500000 0.53
0.74
Each cell Figure 2: Transfer matrix, see Section 4.2
represents the forward transfer from the first task to the second one. We shaded the cells for which 0 belongs to their 90% confidence interval.
Importantly, we use this matrix to estimate what level of forward transfer a good CL. method should be able to achieve. We expect that model which is able to remember all meaningful
5

---

aspects of previously seen tasks would transfer at least as well as if one were just fine-tuning after learning the best choice between the previous tasks. For sequence , ty we set the reference forward transfer, RT, to be
N
1
RT
Σ ==2
max FT t.),
(4)
N
Jái
where FT t.) is the transfer matrix value for for For the CW20 sequence, the value is RT 0.46. Note that a model can do better that this by composing knowledge from multiple previous tasks.
4.3 Training and evaluation details
We adapt the standard Meta-World setting to CL needs. First, we use separate policy heads for each task, instead of the original one-hot task ID inputs (we provide ablation experiments for this choice in Appendix G Second, in each episode. we randomize the positions of objects in the scene to encourage learning more robust policies. We use an MLP network with 4 layers of 256 neurons.
For training, we use soft actor-critic (SAC) [1 a popular and efficient RL method for continuous domains. SAC is an off-policy algorithm using replay buffer, which is an important aspect for CL, particularly for methods relying on rehearsing old trajectories. SAC is based on the so-called maximum entropy principle; this results in policies which explore better and are more robust to changes in the environment dynamics. Both of these qualities might be beneficial in CL.
We note that the size of the neural network and optimization details of the SAC algorithm (like batch size) put constraints on "the amount of compute". Intentionally, these are rather modest, which is in line with CL desiderata, see Section D Similarly, we limit the number of timesteps to 1M, which is a humble amount for modern-day deep reinforcement learning. We picked tasks to be challenging but not impossible within this budget. We note that training the RL setting tends to be less stable than in the supervised one. We recommend using multiple seeds, in our experiments, we typically used 20 and calculate confidence intervals; we used the bootstrap method. We choose hyperparameters that maximize average performance ( In our experiments, we tune common parameters for SAC and the method-specific hyperparameters separately. All details the training and evaluation setup are presented in Appendix A
4.4 Limitations of Continual World
As any benchmark, we are fully aware that ours will not cover the entire spectrum of problems that one might be interested in. Here we summarize few limitations that we hope to overcome in future instantiation of this benchmark:
Input space We use a small 12 dimensional observation space. This is key to achieve modest computational demand. However, richer inputs could allow for potentially more interesting forms of transfer (c.g., based on visual similarity of objects) and would allow inferring the task from the observation, which is currently impossible.
Reliance on SAC We use the SAC algorithm 17 which is considered a standard choice for continuous robotic tasks. However, there is potential risk of overfitting to the particularities of this algorithm and exploring alternative RL algorithms is important.
Task boundaries We rely on task boundaries. One can rely on task inference mechanisms (e.g. B5 41) to resolve this limitation, though we acknowledge the importance to extend the benchmark towards allowing and testing for task inference capabilities. Also, testing for algorithms dealing with continuous distributional drift is not possible in the current format.
Output heads We rely on using a separate head for each new task, similar to many works on continual learning. We opt for this variant based on its simplicity and better performance than using one-hot encoding to indicate a task. We believe that the lack of semantics of the one-hot encoding would further impede transfer, as the relationship between tasks can not be inferred. We carry ablation studies with using one-hot encoding as an input and single head architecture, setting that is already compatible with our benchmark. We regard this aspect as an important future work, and in particular, we are exploring alternative encoding of input to make this choice more natural. A coherent domain, like Continual World, provides unique opportunity to exploit a consistent output layer as its semantics does not change between tasks.

---

Sept Taak Sucoes Ralo
Curert teak Sauura
PMC
Iefermnce
167
167
Avorogo Succoss
0.6
0.0
Shapps
167
line-lnning
Necbbl Hemory
Mulistask Popput
PWC
ieccant CWC20
Figure 3: Training curves for selected CL. methods and multi-task. The upper panel shows performance on the first task for subset of methods throughout the whole training. Note that due to the use of different output heads, we do not see second bump when revisiting this task time 10M. The upper right panel shows the performance the current task being trained for EWC compared to reference model learning only that task from scratch). The bottom plot shows the average performance. Solid lines show the performance of the model training the first 10 tasks (where means being able to solve all of them). Dashed lines show the performance of learning the same tasks in the second half of the benchmark. Note that dashed lower are below solid ones, indicating lower performance the second pass, even if the agent has already previously learned the tasks and has access to relevant features.
The difficulty and number of tasks The number of tasks is relatively small. CW20, the main sequence we use, consists of only different tasks, which are then repeated. We believe the repetition of tasks is important for CL. benchmark, leading to interesting observations. We check also that results are quantitatively similar on sequence of 30 tasks, see Appendix G However, longer sequences, potentially unbounded, are needed to understand the various limitations of existing algorithms. For example, the importance of graceful forgetting or dealing with systems that run out of capacity, scenario where there is no multi-task solution for the sequence of observed tasks. This is particularly of interest for methods such as PackNet [3) Additionally, we provide the number of tasks in advance. Dealing with unknown number of tasks might raise further interesting questions. Finally, in future iterations the benchmark, it important to consider more complex tasks or more complex relationships between tasks to remain a challenge existing methods. Our goal to provide a benchmark that is approachable by existing methods, as not to stifle progress.
Low-level transfer We focus low-level transfers via neural network features and weights. As such, we do not explicitly explore the ability of the learning process to exploit the compositionality of behavior or to rely more interesting semantic level. While we believe such research is crucial, we argue that solving low-level transfer is equally important and might be a prerequisite. So, for now, it is beyond the scope of this work, though future iterations of the benchmark could contain such scenarios.
5 Methods
We now sketch 7 CL methods evaluated on benchmark. Some of them were developed for RL, while others were meant for the supervised learning context and required non-trivial adaptation. We aimed to cover different families of methods; following [13,, we consider three classes: regularization- based, parameter isolation and replay methods. An extended description and discussion of these methods are provided in Appendix B
Regularization- Methods This family builds on the observation that one can reduce forgetting by protecting parameters that are important for the previous tasks. The most basic approach often

---

dubbed L2 (27] simply adds Lg penalty, which regularizes the network not to stray away from the previously learned weights. In this approach, each parameter is equally important. Elastic Weight Consolidation (EWC) 27 uses the Fisher information matrix to approximate the importance of each weight. Memory-Aware Synapses (MAS) A also utilizes weighted penalty, but the importance obtained by approximating the impact each parameter has on the output of the network. Variational Continual Learning (VCL), follows similar path but uses variational inference to minimize the Kullback-Leibler divergence between the current distribution of parameters (posterior) and the distribution for the previous tasks (prior).
This family (also called modularity- forbids any changes to parameters that are important for the previous tasks. may be considered a "hard" equivalent of regularization-batt based methods. PackNet [3 "packs" multiple tasks into a single network by iteratively pruning, freezing, and retraining parts of the network task change. PackNet closely related to progressive neural networks 44, developed in the RI. context.
Parameter Isolation Methods
Replay Methods Methods of this family keep some samples from the previous tasks and use them for training or as constraints to reduce forgetting. We use a Perfect Memory baseline, a modification of our setting which remembers all the samples from the past (i.e., without resetting the buffer the task change). We also implemented Averaged Gradient Episodic Memory (A-GEM) (2). which projects gradients from new samples as to not interfere with previous tasks. We find that A-GEM does not perform well on our benchmark.
Multi-task learning In multi-task learning, field elosely related CL., tasks are trained simultane- ously. By its design, it does not suffer from forgetting, however, it is considered to he hard as multiple tasks "compete for the attention of a single learning system", see 21 44] We find that using reward normalization as in PopArt [ is essential to achieve good performance. See Appendix H
Experiments
Now we present empirical results; these are evaluations of a set of 7 representative CL methods (as described in Section (5) on our Continual World benchmark. We focus on forgetting and transfers while keeping fixed constraints on computation, memory, number of samples, and neural network architecture. Our main empirical contributions are experiments on the long CW20 sequence and following high-level conclusions. For a summary see [ Figure and for an extensive discussion, we refer to Appendix [ provide various ablations and detailed analysis of sensitivity to the CL-specific hyperparameters.
(including results for the shorter sequence, CW10). In Appendix G we
Performance The performance (success rate) averaged over tasks (eq. ( is a typical met- ric for the CL setting. Pack- Net seems to outperform other methods, approaching 0.8 from the maximum of 1.0, outperform- ing multi-task solutions which might struggle with different re- ward scales, problem elegantly avoided in the CL framing. Other methods perform considerably worse. A-GEM and Perfect Memory struggle. We further discuss possible reasons in Sec- tion 6.
method
performance
forgetting
f. transfer
Fine-tuning
0.05 (0.05. 0.06]
0.73 10.72,
0.20 (0.17, 0.23)
L.
0.43 (0.59, 0.47]
0.02 (0.00, 0.03
-0.71 [4887, -0.77
EWC
0.60 (0.57, 0.64]
0.02 [-000, 0.05]
-0.17 [4024, 1011]
MAS
0.51 [0.49, 0.53]
0.00
1-0.1, 0.02)
-0.52 (-0.59, -0471
VCL
0.48 (0.46. 0.50]
0.01
[-0.1., 0.02]
-0.49 [-0.7, 0.42)
PackNet
0.80 (0.79, 0.82)
0.00
[-001,
0.19
(0.15, 0.23]
Perfect Memory
0.12 |0.09, 0.15]
0.07
jo.05, (.05,0.100
-1.34 1-1.42, -1.271
A-GEM
0.07 [0.06. 0,08]
0.71
10:70.
0.13 10.10.00 0.16)
MT
0.51 [0.48, 0.53]
MT (PopArt)
0.65 (0.63, 0.67]
RT
0.46
Table 1:
Results on CW20, for CL methods and multi-task training. Metrics are defined in Scction [.] RT is eq. 4.. We used 20 seeds and provide 90% confidence intervals.
Forgetting We observe that most CL methods are usually efficient in mitigating forgetting. However, we did not notice any boost when revisiting a task (see Figure B). Even if different output head was employed, relearning the internal representation should have had an impact unless it changed considerably when revisiting the task, Additionally, we found A A-GEM difficult to tune; consequently, with the best hyperparameter settings, it is relatively similar to the bascline fine-tuning method (see details in Appendix C
8

---

Transfers For all methods. forward transfer for the second ten tasks (and the same tasks are revisited) drops compared the first ten tasks. This is in stark contrast forgetting, which seems to be well under control. Among all methods, only fine-tuning and PackNet are able to achieve positive forward transfer (0.20 and 0.19, resp.) as well as on the first (0.32 and 0.21, resp.) and the second (0.08 and 0.17, resp.) half of tasks. However, these are considerably smaller than RT' = 0.46, which in principle can even be exceeded, and which should be reached by a model that remembers all meaningful aspects of previously seen tasks, see 4 These results paint fairly grim picture: we would expect improvement, rather than deterioration in performance, when revisiting previously seen tasks. There could be multiple reasons for this state of affairs. It could be attributed to the loss of plasticity, similar to the effect observed in [ Another reason could be related to the interference between CL mechanisms or setting and RL, for instance, hindering exploration. We did not observe any substantial cases of backward transfer, even though the benchmark is well suited to study this question due to the revisiting of tasks. See Appendix
Triplets experiments We illustrate how forgetting and forward transfer inter- act with each other in sim- pler setting of three task se- quences, see Figure 4 and Appendix F We focus on sequences of tasks A - B - C. where A - C has significant positive for- ward transfer and B - C has smaller or even neg- ative transfer. An efficient CL agent should be able to use information from A to get good performance on C. However, interference intro- duced by B reduces the fi- nal forward transfer (see Figure 4. The drive for reducing forgetting in CL agents has been primarily to perform well on previous tasks when we revisit them. With this example, we argue that an equally important reason to improve the memory of CL agents is to efficiently use past experiences to learn faster on new tasks. Currently, the tested CL methods often are not able to outperform the forgetful fine-tuning baseline. Observe that even the modularity-basee task. This possibly indicates that using the activation mask from task enough to deteriorate the performance.
Triple
Triple
1.0
Success Rate
0.8
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.B
1.0
Steps
1e6
Steps
le6
A-
C (Hie-tuninn)
A-> [PackNet)
0
C
A-2 D - C (CWC)
Figure 4: How forgetting impacts the forward transfer. Two different triplets of tasks learnt in sequence. An ideal agent learning sequence A > B -> C should have at least as good performance task C' agent which just learns + C. In reality, an interfering task B reduces this transfer, even when continual learning approaches are used.
PackNet approach struggles with this
PackNet PackNet stands out in our evaluations. We conjecture that developing related methods might be a promising research direction. Besides further increasing performance, one could mitigate the limitations of PackNet. PackNet relies on knowing task identity during evaluation. While this assumption is met in our benchmark, it is an interesting topic for future research to develop methods that cope without task identity. Another nuisance that PackNet assigns some fixed fraction of parameters to task. This necessitates knowledge of the length of the sequence in advance. Additionally, when the second ten tasks of CW20 start, PackNet performance degrades, showing its potentially inefficient use of capacity and past knowledge, given that the second ten tasks are identical with the first ten and hence no additional capacity is needed. In a broader context, we speculate that parameter isolation methods might be promising direction towards better CL methods.
Resources usage In practical applications it is important to consider resources usage. All tested methods have relatively small overheads. For example, PackNet needs only 15% more time than the baseline fine-tuning and it requires 50% more neural network parameters (which is negligible when small networks like ours). See Appendix B.4 for details concerning other methods.
Other observations In stark contrast with the supervised learning setting, found that replay based methods (Perfect Memory and A-GEM) suffer from poor performance. This is even though we allow for generous replay, which could store the whole experience. Explaining and amending this situation is, in our view, an important research question. We conjecture that this happens due to the regularization of the critic network (which was unavoidable for these methods). We found multi-task

---

learning attaining lower scores than PackNet, the best CL method and comparable to the second one, EWC. We think this suggests interesting research directions for multi-tast learning.
6.1 RL-Related Challenges
Reinforcement learning brings a set of issues not present in the supervised learning setting, e.g.. exploration, varying reward scales, and stochasticity of environments. We argue that is imperative to have a reliable benchmark to assess the efficiency of CL algorithms with respect to these problems. We find that some current methods are not well adjusted to the RL setting and require non-tre con-trivial conceptual considerations and careful tuning of hyperparameters, see details in Appendix g
An important design choice is whether not regularize the critic in the actor-critic framework (e.g. in SAC). We find it beneficial to focus on reducing forgetting in the actor while allowing the critic freely adapt to the current task (note that critic is used only in training of the current task), similar to [4.] On the other hand, a forgetful critic is controversial. This can be sharply seen when the same task is repeated and the critic needs learn from scratch. Additionally, not all methods can be trivially adapted to the 'actor-only regularization' setting, as for example replay based methods. In Appendix
C we examine these issues empirically, by showing experiments with critic regularization for EWC.
Another aspect is the exploration and its non-trivial impact on transfers. As it observed, transfers from a given task to the same one are sometimes poor. We show in Appendix [.1 that this results from the fact that the task change the replay buffer is emptied and SAC collects new samples from scratch, usually by using the uniform policy. Learning on these random samples reduces performance on the current task and thus the forward transfer. Experimentally, or using the current policy for exploration improves the transfer on the diagonal.
find that not resetting the buffer
7 Conclusions and Future Work
In this work, we present Continual World, continual reinforcement learning benchmark, and an in-depth analysis of how existing methods perform on it. The benchmark is aimed at facilitating and standardizing the CL system evaluation, and such, is released with code, including implementation of 7 representative CL algorithms. We argue for more attention to forward transfer and the interaction between forgetting and transfer, as many existing methods seem sacrifice transfer to alleviate forgetting. In our opinion, this should not be the aim of CL., and we need to strike different balance between these objectives.
We made several observations, both conceptual and empirical, which open future research directions. In particular, we conjecture that parameter isolation methods are a promising direction. Further, we identified a set of critical issues at the intersection of RL and CL. Resolving critic regularization and efficient use of multi-task replays seem be the most pressing ones. Our benchmark highlights some challenges, which in our view are relevant and tangible now. In the long horizon, achieving high-level transfers, removing task boundaries, and scaling up are among significant goals for future editions of Continual World. Our work is foundational research and does not lead to any direct negative applications.
Acknowledgments and Disclosure of Funding
We would like to thank Stanislaw Jastrzębski for stimulating talks and help while preparing the manuscript. The work of PM was supported by the Polish National Science Center grant UMO- 2017/6/E/STT600622 The work of MW funded by Foundation for Polish Science (grant no 20IR.04.04..00
carried out within the Team-Net program co-financed by the European Union under the European Regional Development Fund. This research was supported by the PL-Grid Infrastructure. Our experiments were managed using httnsttttps:: ttatps://neptuneee ai We would like thank the Neptune team for providing us access the team version and technical support.
10

---

References
[1] Joshua Achiam. Spinning Up in Deep Reinforcement Learning, 2018.
[2] Tameem Adel, Han Zhao, and Richard E. Turner. Continual learning with adaptive weights (CLAW). In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[3] Ossama Ahmed, Frederik Träuble, Anirudh Goyal, Alexander Neitz, Manuel Wuthrich, Yoshua Bengio, Bernhard Schölkopf, and Stefan Bauer. Causalworld: A robotic manipulation bench- mark for causal structure and transfer learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[4] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny. Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8- 14, 2018, Proceedings, Part III, volume 11207 of Lecture Notes in Computer Science, pages 144-161. Springer, 2018
[5] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 11849-11864 2019.
[6] Jordan T. Ash and Ryan P. Adams. On warm-starting neural network training. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[7] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.06455 2016.
[8] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
[9] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1613-1622. JMLR.org. 2015.
[10] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA,, USA, May 6-9, 2019. OpenReview.net, 2019.
[11] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balean, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[12] Arslan Chaudhry, Marc Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
11

---

[13] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting in classification tasks. arXiv preprini (900:0999988 2019.
[14] Yunshu Du, Wojeiech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. CoRR, abs/1812.02224, 2018.
[15] Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
[16] Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. CoRR, abs/1 00//1805.09333 2018.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference Machine Learning, ICMI. 2018, Stockholmssüsssan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865, PMLR, 2018
[18) Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. CoRR, abs/ 10s05982 2018.
[19] Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24(12):1028 1040, 2020.
[20] Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245 258, 2017.
[21] Mattco Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 February AAAI Press, 2019.
1, 2019, pages
[22] Ferenc Huszár. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, page 201717045 2018
[23] Khurram Javed and Martha White. Meta-learning representations for continual learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8 14, 2019, Vancouver, BC, Canada, pages 1818-1822 2019.
[24] Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1242-3251. 1. PMLR, 09-15 Jun 2019.
[25] Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual rein- forcement learning: A review and perspectives, 2020.
[26] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
[27] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- Demis
drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.00796. 2016.
12

---

[28] Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106-1114, 2012.
[29] Germán Kruszewski, Ionut-Teode Sorodoc, and Tomas Mikolov. Evaluating online continual learning with calm, 2021.
[30] Vincenzo Lomonaco, Karan Desai, Eugenio Culurciello, and Davide Maltoni. Continual reinforcement learning in 3d non-stationary environments. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pages 999-1008. IEEE, 2020.
[31] Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In 1st Annual Conference on Robot Learning, CoRL 2017. Mountain View, California, USA, November 13-15, 2017, Proceedings, volume 78 of Proceedings of Machine Learning Research, pages 17-26. PMLR, 2017.
[32] Kevin Lu, Igor Mordatch, and Pieter Abbeel. Adaptive online planning for continual lifelong learning. CoRR, abs/1912.01188, 2019.
[33] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7765-7773, IEEE Computer Society, 2018.
[34] Jorge A. Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[35] Kieran Milan, Joel Veness, James Kirkpatrick, Michael H. Bowling, Anna Koop, and Demis Hassabis. The forget-me-not process. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3702-3710, 2016.
[36] Hadi Nekoei, Akilesh Badrinaaraayanan, Aaron C. Courville, and Sarath Chandar. Continuous coordination as realistic scenario for lifelong learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8024.8024. PMLR, 2021.
[37] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In 6th International Conference Learning Representutions, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net 2018
[38] German Ignacio Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[39] Emmanouil Antonios Platanios, Abulhair Saparov, and Tom M. Mitchell. Jelly bean world: A testbed for never-ending learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[40] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Suiskever. Zero-shot text-to-image generation. CoRR, abs/2102. 2021.
[41] Dushyant Rao, Francesco Visin, Andrei A. Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Had- sell. Continual unsupervised representation learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Gamettt editors, Advances
13

---

in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pages 7645-7655, 2019.
[42] Natalia Díaz Rodríguez, Vincenzo Lomonaco, David Filliat, and Davide Maltoni. Don't forget, there is more than forgetting: new metrics for continual learning. CoRR, 0ts//810013166 2018.
[43] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1600.04671, 2016.
[44] Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: a source of plateaus in deep reinforcement learning. CoRR, abs/1904. 1455, 2019.
[45] Jonathan Schwarz, Daniel Altman, Andrew Dudzik, Oriol Vinyals, Yee Whye Teh, and Razvan Pascanu. Towards a natural benchmark for continual learning. In Continual learning Workshop, Neurips 2018, 2018.
[46] Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka
Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4535 4544. PMLR, 2018.
[47] Andrew W. Senior, Richard Evans, John Jumper. James Kirkpatrick. Laurent Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Koray Kavukcuoglu, and Demis Hassabis. Improved protein structure prediction using potentials from 2020.
deep learning. Nat., 577(7792))7066
[48] Pan-Keng Sun, Cheng- Hao Ho, and Hung- Yi Lee. LAMOL: language modeling for lifelong language learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[49] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pages 6105 61 14. PMLR, 2019.
[50] Chen Tessler, Shahar Givony, Tom Zahavy, Daniel J. Mankowitz, and Shie Mannor. A deep hier- archical approach to lifelong learning in minecraft. In Satinder Singh and Shaul Markovitch, editors, Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 1553-1561. AAAI Press, 2017.
[51] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Poruuaal, October 7-12, 2012, pages 5026-5033. IEEE, 2012.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[53] Jeffrey Scott Vitter. Random sampling with reservoir. ACM Trans. Math. Softw. 11(1):37-57. 1985.
[54] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual
14

---

Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1094- 1100. PMLR, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] The codes is included the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see details in Appendix A
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [Yes] we used 20 random seeds, see also details in Appendix A.6
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] see Appendix A.7
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We base on the MetaWorld benchmark [54], which clearly indicate few times, including the abstract.
(b) Did you mention the license of the assets? [Yes] We use MIT licence; see Appendix A.
(c) Did you include any new assets either in the supplemental material or as a URL? Yes]
(d) Did you discuss whether and how consent obtained from people whose data you'ne using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "YMU3l0-44bf", "reviewer": "Reviewer_EkWJ", "summary": "The paper presents a suite of continuous control tasks for studying continual reinforcement learning. The paper includes a comprehensive study evaluating prior continual learning methods, both the ones designed for RL and for other tasks. It also highlights the challenges of applying existing RL methods to continual learning setting.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Key contribution\n- A suite of continuous control tasks for continual reinforcement learning (CRL) (built on Meta World tasks)\n- Implementations of existing continual learning (CL) methods, both the ones designed for RL and other tasks --- require non trivial adaptation.\n- A fairly comprehensive study using existing methods on the proposed benchmark.\n- Highlights CRL-related challenges (Sec. 6.1) --- 1) existing actor-critic methods may not be the best for CRL settings and 2) we may need to rethink the standard RL training procedure, i.e., exploration and how to populate replay buffers, in the CL setting.\n\nComments\n- From my understanding, the key selling point of the proposed benchmark is “more realistic robotics tasks”, but practically, I’m not sure if they are “realistic” in the sense that they are still individual tasks with clear assumptions about task boundaries. Learning is harder for sure, but I’m not sure about realism.\n- In addition, I'm not sure about why the previous supervised CL benchmarks “are not geared towards measuring forward transfer” (line 65). As far as I could tell, the forward transfer measures the learning speed / sample efficiency as a result of transferring the knowledge from the previous task. It would seem natural to also measure the sample efficiency as a form of forward transfer in supervised CL?\n- It would be interesting to see what the harder tasks bring to the table. Ideally, the paper should conduct a small-scale experiment and apply the same methods on “easier” tasks such as MNIST sequence and see if the conclusion still holds, since the main selling point is that the easier tasks cannot show the challenge of “more realistic settings”\n- Triplet experiment: ideally PackNet should not be affected by the prior tasks (i.e., task B) since it isolates parameters for each task, but it doesn’t seem to be the case in this experiment. More analysis would be helpful.\n\n\nMinor:\n- Fig 1 no axis\n- Including Multi-task learning + PopArt in Fig 3 as a curve is somewhat confusing, since the training curve means an entirely different thing here (learning all tasks simultaneously). Better include the asymptotic performance as a horizontal line?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The \"limitation\" section is adequate for technical contributions. It'd be interesting to see a short discussion on societal impact of studying the continual RL setting.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JeD2zCcEH2", "reviewer": "Reviewer_7Se4", "summary": "This paper proposes a new benchmark for continual reinforcement learning. It modifies the Meta-World robotics benchmark that’s designed for meta- and multi-task RL evaluation by learning the tasks in sequence rather than together. This work is very timely as there lacks a common benchmark for continual/lifelong RL.\n\nThe benchmark consists of 10 tasks from Meta-World. To evaluate catastrophic forgetting, these 10 tasks are repeated for a sequence of length 20. For quicker iteration, there are also different triplets of tasks where the middle task might inhibit transfer to the last task.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n- Continual World modifies the evaluation protocol of the existing Meta-World benchmark. \n\n- As noted by the authors, there is already work that re-purposes Meta-World for the continual RL setting and evaluates various algorithms on it [33]. Additionally, [33] performs the evaluation on the full set of 50 tasks, while Continual World only looks at 10 tasks.\n\nQuality:\n- The 10 tasks in Continual World were carefully selected so that there is a good range in the difficulty of transfer between these tasks. They also include a transfer matrix, which can aid researchers when selecting the ordering of the tasks and understanding which tasks facilitate easier or harder transfer.\n\n- They also evaluate a diverse range of continual learning methods on the benchmark. Interestingly, continuous fine-tuning gives the best forward transfer performance, but even that is far below the reference forward transfer value which is the highest transfer value from a single source task.\n\n- Other methods I’d be curious to see the forward transfer performance of are: progressive neural networks and meta-RL methods adapted to the online setting.\n\nClarity:\n- The paper is overall well-written and organized. The related work section also provides a thorough survey of the existing benchmarks. \n\nSignificance:\n- A benchmark for continual RL is highly relevant to the NeurIPS community as there is no commonly used benchmark yet.\n\n- The forward transfer results, when compared to the forgetting scores, really highlight transfer as an important direction to study in the context of continual RL.\n\nQuestions: \n- In Table 1, is the Performance column reporting the success rate?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. See main review for other limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Buf88sO8BE", "reviewer": "Reviewer_g4u3", "summary": "A benchmark for continual reinforcement learning is provided, where the main task contains 20 subtasks. A certain number of choices are made such as the agent being aware of task change, not carrying forward the replay buffer, and learning policies with separate output heads per task. The evaluation was overall performance, forgetting, and forward transfer. Seven learning methods are compared extensively and results are provided.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The setup is not satisfying but it is also true that finding a continual learning benchmark acceptable is extremely hard. As a first start, I think this benchmark can be acceptable and an interesting place to start.\n\nThere is an emphasis that the different goals in continual learning are conflicting. However, that is neither a surprise nor unrealistic. Animals face the same tradeoffs, and these are exactly why continual learning is so challenging.\n\nIt seems the focus on computational and memory resource capacity is forgone. In the experiments and comparison, it was not taken into account. Different algorithms took different amounts of computations and memory, which were not described. If they took widely different amounts of computations, it is not clear that aligning their performance on a per time step basis for comparison is fair anymore.\n\nIn Table 1, all forgetting numbers are positive. Equation (3) suggests they should be negative. If that is not the case, then can you clarify how this forgetting number is calculated? And, does 0.73 for fine-tuning mean a high level of forgetting?\n\nIt is not clear what is meant by regularizing critic. It is also not clear whether the critic is being learned from scratch or not or whether it depends on the method.\n\n*** post rebuttal ***\n\nThanks for the clarification.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9wiCZ080bkB", "reviewer": "Reviewer_Mh4F", "summary": "Summary. The paper proposes a robotic benchmark for continual reinforcement learning built on top of meta-world benchmark. It analyzes the difficulty in the problem of continual RL (such as catastrophic forgetting) and proposes metrics (average performance, forward transfer, forgetting) to measure performance of continual RL algorithms. Additionally, the paper evaluates existing continual RL algorithms on their benchmark and discusses the limitations of the proposed benchmark in detail as well.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper doesn’t propose any new method but rather presents a new robotic continual RL benchmark.  \n\nQuality: The paper appropriately analyzes the proposed benchmark, evaluates existing methods on it and discusses the limitations of their proposed benchmark in detail. I do think that absence of visual continual RL tasks makes the benchmark limiting as it inhibits study of interesting forms of transfer involving visual features.\n\nClarity: The paper is clearly written.\n\nSignificance: The paper is important because it provides a robotic benchmark for continual RL along with evaluating existing methods on the proposed benchmark which makes it easier to test new continual RL algorithms.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It's not discussed in the paper", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hr"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I think reviewers were generally satisfied that this was a useful contribution.  I found the reviews generally fair and the paper itself generally meritorious.  Obviously there are some concerns, mostly related to the inherent difficulty of producing a satisfactory continual learning benchmark.  This paper is somewhere near the current pareto-frontier of the tradeoff between "doing something useful but hard" and "doing something in a completely satisfactory way".   I don't completely know if that frontier is close enough to the absolute quality threshold for NeurIPS, but I tend to think it is.  
Decision: Accept (Poster)