Sequoia: A Software Framework to Unify Continual Learning Research
Authors: Fabrice Normandin
Oleksiy Ostapenko
Pau Rodriguez
Florian Golemo
Ryan Lindeborg
Matthew Riemer
Lucas Cecchi
Timothee LESORT
Khimya Khetarpal
David Vazquez
Laurent Charlin
Irina Rish
Massimo Caccia
Venue: ICLR 2022 Submitted (2022)
Keywords: Continual Learning
Reinforcement Learning
Software Engineering
Deep learning

Abstract: The field of Continual Learning (CL) seeks to develop algorithms that accumulate knowledge and skills over time through interaction with non-stationary environments. In practice, a plethora of evaluation procedures (settings) and algorithmic solutions (methods) exist, each with their own potentially disjoint set of assumptions. This variety makes measuring progress in CL difficult. We propose a taxonomy of settings, where each setting is described as a set of assumptions. A tree-shaped hierarchy emerges from this view, where more general settings become the parents of those with more restrictive assumptions. This makes it possible to use inheritance to share and reuse research, as developing a method for a given setting also makes it directly applicable onto any of its children. We instantiate this idea as a publicly available software framework called Sequoia, which features a wide variety of settings from both the Continual Supervised Learning (CSL) and Continual Reinforcement Learning (CRL) domains. Sequoia also includes a growing suite of methods which are easy to extend and customize, in addition to more specialized methods from external libraries. We hope that this new paradigm and its first implementation can help unify and accelerate research in CL. You can help us grow the tree by visiting (this GitHub URL).

Under review as a conference paper at ICLR 2022
SEQUOIA: A SOFTWARE FRAMEWORK TO UNIFY CONTINUAL LEARNING RESEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
The field of Continual Learning (CL) seeks to develop algorithms that accumulate knowledge and skills over time through interaction with non-stationary environ- ments. In practice, a plethora of evaluation procedures (sellings) and algorithmic solutions (methods) exist, each with their own potentially disjoint set of assump- tions. This variety makes measuring progress in CL difficult. We propose a tax- onomy of settings, where each setting is described as a set of assumptions. A tree- shaped hierarchy emerges from this view, where more general settings become the parents of those with more restrictive assumptions. This makes it possible to use inheritance share and reuse research, developing method for a given setting also makes it directly applicable onto any of its children. We instantiate this idea publicly available software framework called Sequoia, which features a wide variety of settings from both the Continual Supervised Learning (CSL) and Con- tinual Reinforcement Learning (CRL) domains. Sequoia also includes growing suite of methods which are easy to extend and customize, in addition to more spe- cialized methods from external libraries. We hope that this new paradigm and its first implementation can help unify and accelerate research in CL. You can help us grow the tree by visiting (this GitHub URL).
INTRODUCTION
With the growing interest in developing methods robust to changes in the data distribution, research in continual learning (CL.) has gained traction recent years (Delange et al., 2021; Caccia et al., 2020; Parisi et al., 2019; Lesort et al., 2020). CL enables models to acquire knowledge from non-
Assumppion
Classification
MLP Classieer
Switing
Imputs are mages
stationary data, learning new and possibly more complex tasks, while retaining performance previously-learned tasks.
Image classitiation
ConwNic Choodier
imagus contain or dogs
mages contain digis
Cata Doge
MNIST classifiction
To instantiate CL problem, must first make assumptions about the data distribution and set constraints enforce non-stationary learning. For instance, assumptions are often made about the type and number of tasks, the task boundaries, or the availability of task labels, while constraints often relate to memory, compute, or time allowed to learn a task. Combinations of assumptions, rules, and datasets have resulted in multitude of settings (Khetarpal et al., 2020).
Figure I: Example simple tree of settings that shows the core principle of our framework: research settings can be organized into a hierar- chy based their assumptions. Methods have a set of basic assumptions which correspond to their target setting (dashed arrows), and can be applied onto any of their descendants. In this trivial example, both MLP and ConvNet clas- sifiers are applicable to the MNIST classifica- tion setting and their performances are directly comparable, even though they were created for slightly different settings. Sequoia applies this principle to the field of Continual Learning.
We argue that the increased popularity of CL, combined with lack of unification in part due to the large number of settings and the ab- sence of well-defined applications - has led a "research jungle" that may be slowing down progress.

---

Under review as a conference paper at ICLR 2022
We identify some of the main challenges associated with the lack of unification in CL:
i) Evaluation. Methods in CL are often studied under a small subset of the available settings, making it difficult to evaluate them, as their problem domains don't always overlap. Consequently, it is challenging to determine if method will generalize beyond the setting it was designed for. To add to this, continual reinforcement learning (CRL) poses further challenges in evaluation due the lack of a clear distinction between training and testing phases (Khetarpal et al., 2018). Moreover, resource consumption is a critical factor for evaluation of CL methods which is often overlooked due to the lack of standardized evaluation protocols. Thus, there is need for standardization of the infrastructure used evaluate CL methods.
ii) Reproducibility. In order to analyze specific properties of novel methods, researchers tend to re-implement baselines and adapt them their particular needs (Henderson et al., 2018). These baselines are often not described in enough detail to ensure reproducibility. e.g. prescribed hyper- parameter search strategy, computational requirements, open source libraries, etc.
iii) CSL and CRL evolve in silos. Continual supervised learning (CSL) and continual reinforcement learning (CRL) are considered to be independent settings in the literature and thus, they are evolving separately. However, most methods in one field can be instantiated in the other, resulting in duplicate efforts such as replay for CSL (Rebuffi et al., 2017: Lesort et al., 2019a; Shin et al., 2017; Lesort et al., 2019b; Prabhu et al., 2020) and replay for CRL (Traoré et al., 2019; Rolnick et al., 2019; Kaplanis et al., 2020). To this end, we advocate that the unification of both fields would greatly reduce these duplicate efforts and accelerate CL research.
In this work, we present Sequoia, a unifying software framework for CL research, as a solution for joimtly addressing these issues. We describe how setrings differ from one another in terms of their assumptions (e.g., are task IDs observed or not). This perspective gives rise to a hierarchical organization of CL. settings, through which methods become directly reusable by inheritance, thus greatly reducing the amount of work involved in developing and evaluating methods in CL.
Key Contributions of this work are 1) a general unified framework that systematically organizes CL settings; 2) Sequoia, a software framework that can serve as a universal platform for CL research.
2 A UNIFYING FRAMEWORK FOR CL RESEARCH
To construct our unifying framework, we first represent each CL setting as a set of shared assump- tions. More general settings make fewer assumptions and vice versa. Settings can then be organized in hierarchy where adding/removing an assumption yields a child/parent setting (Fig. 2).
We formalize the framework using mode Markov decision process (HM-MDP), special case of a POMDP (Choi et al., 2000). HM-MDPs comprise an observation space X, an action space A, context space Z (we also refer contexts tasks), and a feedback function the full state space a concatenation of the observation space and task space S X' x 2. As such, the hidden context variable = E Z defines the dynamics of the environment p(æ' [x, a. for observations x', x E X' and action a E A. The feedback function r(æ, a, z) provides an agent (aa(z) (e.g. a supervised model) the value of performing action a after receiving observation æ. The context variable follows a Markov chain p(='(z). The dynamic context enables modelling CL task/context-change A change in the context variable called task boundary.
Here
In the next section ($ 2.1) we show that by restricting the different elements of the HM-MDP we recover different CL settings. Then in 2.2 we discuss differences between continual supervised (CSL) and continual reinforcement learning (CRL). We end in $ 2.3, by presenting additional as- sumptions that are relevant to CL problems.
2.1 CONTINUAL LEARNING ASSUMPTIONS
Assumptions related to CL can be aranged into hierarchy, as illustrated in the the central portion of Fig. 2. These settings cover most, if not all the current CL literature. We start from the most general setting: continuous task-agnostic CL (Zeno et al., 2018) and add assumptions one by one.
Continuous Task-Agnostic CL is our most general setting. The context variable is continuous j E R. This setting allows for different kinds of drifts in the environment, including smooth task
2

---

Under review as a conference paper at ICLR 2022
continuous task-agnostic CL
complete supervision passive environment
supervision active environment
clear task boundaries
continuous lask-agnostic CSL
continuous task-agnostic CRL
discrete task-agnotic CL
observable boundaries
incremental learning
discrete task- agnostic CSL
discrett agnosagoostic CRL
stationary
observable task variable
tasks
traditional leamiig
task-incremental learning
observable task variable
stationary tasks
multi-task
learning
Figure 2: Sequoia The Continual Learning Research Tree. Continual learning research set- tings can be organized into a tree, in which more general settings (parents) are linked with more restricted settings (children) by the differences in assumptions between them. Settings generally become more challenging the higher they are in this hierarchy, as less information becomes avail- able to the method. The central portion of the tree shows the assumptions specific CL, while the highest lateral branches indicate the choice of either supervised or reinforcement learning, which we consider be orthogonal to CL. By combining either with the central assumptions, settings from Continual SL and Continual RL can be recovered to the left and right, respectively.
boundaries, i.e. slow drift (Zeno et al., 2018). This setting is task-agnostic, meaning that the context variable is unobserved. Because the context is allowed to drift slowly, it can be more challenging for the methods to infer when a task has changed enough to compartmentalize the recently acquired knowledge before adapting to the new task. In RL, this setting is analogous the DP-MDP (Xie et al., 2020; Chandak et al., 2020b). In SL, it has also been studied in e.g. Zeno et al. (2018); Aljundi al. (2019a;b).
Discrete Task Task-Agnostic CL assumes clear (or well-defined) task boundaries and so discrete con- text variable Z N. In this setting the context can shift in drastic way, still without the agent being explicitly noticed. Some cases where this setting has been studied are Choi et al. (2000); Riemer al. (2018) for RL and Caccia al. (2020); He al. (2019); Harrison et al. (2019) for SI.
Incremental Learning (IL) relaxes the task-agnostic assumption: the task boundaries are observ- able. This is akin to augmenting the observation with a binary variable that is set to when z' + and 0 otherwise. In doing so, the algorithm does not need to perform lask-boundary detection. In SL, some well-known IL settings include class-IL and domain-IL. distinguished by their disjoint action space and shared action space, respectively. This is discussed in $ 2.3.
At this point in the CL hicrarchy, the tree branches in two directions, depending on the order of remaining assumptions (see Fig. 2). We will first explain the right sub-tree.
Task-Incrrementll Learning (task-IL) IL) assumes a fully obser-bbeereaale context variable available to the agent =<a a/x 2). In the literature, observing z is analogous to knowing the task ID task label. In this simpler CL setting. forgetting can be prevented by freezing a model at the completion of each task and using the task-ID to retrieve it for evaluation.
3

---

Under review as a conference paper at ICLR 2022
The following settings remove the non-stationarity assumption in the contexts/tasks and are often used to set an upper-bound performance for CL methods.
Multi-task Learning removes the non-stationarity in the environment dynamics and the feedback function as it assumes a stationary context variable p(z'|z) p(z'). When the contexts are station- ary, there is catastrophic forgetting (CF) (French, 1999) problem to solve. Multi-task learning assumes a fully-observable task variable.
Traditional Learning branches off incremental CL and assumes stationary environment. It is the vanilla supervised setting machine learning defaults to. In our framework, it can be seen as a multi-task learning problem where the task variable isn't observable. However, more natural view of this setting is to simply assume single task/context.
2.2 SUPERVISED LEARNING AND REINFORCEMENT LEARNING ASSUMPTIONS
So far we have introduced settings and assumptions that revolve mainly around the type and pres- ence of non-stationarity in the environment and the information observed by the agent. These have allowed us to define the CL problem. To bring all of CL research under two assumptions, orthogonal to the previous ones, to recover RL and SL settings. With these as- sumptions, methods for given CL setting are applicable to both its CSL and CRL versions, in Kirkpatrick et al. (2017); Fernando et al. (2017).
umbrella,
introduce
Below we the term observation as a state in RL parlance and the actions predictions in SL parlance. Also, we assume single context or task.
Level of feedback: In RI. the feedback function r(æx, a, 2) returns reward that informs the agent about the value of performing action a after receiving observation æ in context z. In SL however, the feedback function is generally both directly known by the agent and differentiable, which allows the agent to simultaneously consider the value of all actions for a particular observation. This feedback computed based on a label when the action space is discrete (classification) or a target when is continuous (regression). The feedback level is key differentiating feature between RL and SL.
Active vs passive environments: In RL, is generally assumed that the agent's 's action has an effect on the next observation or state.' In other words, the dynamics of the environment p(æ' (e, z) are action-dependant and we call this an active environment. In SI. the agent is generally assumed to not influence the next observation i.e. p(æ' xr, z) = p(=''1a x. 2). The environment is thus referred to as being passive in these cases.
As seen in Figure 2, the two aforementioned assumptions are combined into single assumption for SL (blue, left) and for RL (red, right). By combining either the RL or SL assumption along with those from the the central CL "trunk", settings from CSL and CRL are recovered. Future versions of Sequoia will decouple these assumptions to enable settings such as bandits and imitation learning.
2.3 ADDITIONAL ASSUMPTIONS
Additional assumptions can be added on top of the ones described above to recover additional re- search settings. For example, a useful assumption in CL experiments is the one of disjoint versus joint action space, i.c. whether the contexts/tasks share same action space, or whether that space different for each task. In CSL, this assumption differentiates faass-incrementll learning from comain-incrementie learning (van de Ven & Tolias, 2019). In Farquhar & Gal (2018), where it is re- ferred as the shared output space assumption, a disjoint action space greatly increases the difficulty of setting in terms of forgetting. In CRL however, the studied settings mostly have joint action space, with the notable exception in the work of Chandak et al. (2020a).
Other assumptions could also be relevant in defining continual learning problem. For instance, the action space being either discrete or continuous. resulting in classification and regression CSL problems, respectively; particular structure being required of the method's actions, in image segmentation problems; an episodic vs non-episodic setting in RL; context-dependant (Caccia et al., 2020) versus context-independen feedback functions; and many more.
'The bandit setting one notable exception this rule.
4

---

Under review as a conference paper at ICLR 2022
3 SEQUOIA A SOFTWARE FRAMEWORK
Alongside this unifying perspective, we introduce Sequoia, an open- python framework. Each setting described above is instantiated class in a tree-shape inheritance hierarchy. Sequoia is designed to address some of the issues associated with Continual Learning research, previously described in
First, we establish a clear separation of concerns between research problems and the solutions to such problems. We establish this separation through This decoupling greatly helps to evaluate methods, since the logic for each component is cleanly separated, and extracting component and reusing it elsewhere becomes possible. An example of Method is shown Listing
abstractions: Setting and Met.od
Second, to help bridge the gap between the CRL and CSL domains, Sequoia uses Environmert as the interface between methods and settings. Envi ronments extend the familiar abstractions from OpenAl gym, to also include supervised learning datasets, making it possible to develop methods that are applicable in both the CRL and CSL domains. Environment will be described in $ 3.2.
Finally, Sequoia uses inheritance to make methods directly reusable across settings. By orga- nizing research settings into a tree-shaped inheritance hierarchy, along with their environments, observations, actions, and rewards, Sequoia enables methods developed for any particular setting to be applicable onto any of their descendants, since all the objects the method will interact with will inherit from those they were designed to handle. This mechanism has the potential to greatly improve code reuse and reproducibility in CL research.
This section first describes each of these abstractions more detail, after which the currently avail- able settings and methods are described. $ 4 will then provide a demonstration of the kind of large-scale empirical studies which are made possible through the use this new framework.
Relation with other frameworks: Sequoia is in no way competing with existing tools and libraries which provide standardized bench- marks, models, or algorithm implementations. On the contrary, Sequoia benefits from the de- velopment of such frameworks.
from sequoia. sattings.e import
from sequoia. settings. .rl import sequoia. methods import #asettethod
met BasMMethod (lanrttndeedlarnnnnnnnaaeeeeee
for setting in bonniinnosstaakkknnnttiiie ("mnia=),
In the case of libraries that introduce standard- ized benchmarks, they can be used to enrich existing settings with additional datasets en- vironments (Douillard & Lesort, 2021; Brock- man et al., 2016). or even to create entirely new settings. Likewise, frameworks which in- troduce new models or algorithms (Lomonaco et al., 2021; Raffin et al., 2019; Wolczyk et al., 2021) can also be used to create new Methods to add new backbones to existing Methods within Sequoia. External repositories can reg- ister their own methods through a simple plu- gin system. The end goal for Sequoia is to pro- vide the research community with central- ized catalog of the different research frame- works and their associated methods, settings, environments, etc. The following sections will show examples of such extensions.
Cont aawaauteTaakgggsttoot.......
ing. ccrrtpo ee e*),
siscettsssskknncrrisssssettteeee ("mnist),, ni
TaakinoremontallRisetttttt "taarpolle",
(ic=ffaackttSsttting(aaaaar ("aniat"),
Traditionnl wtt.::oonn.11..ttt.111 ("tnnatttgg""miaa""
TradittooeelRGGettaaa ("castpole"),
results setting..apeeeeeoee apply (mathod)
reaults.summe () besults.meke_plce
Code snippet, using Sequoia evalu- ate a method in multiple settings. This particular example made possible by the has, which is applicable to all settings.
Listing 1:
3.1 SETTINGS
A Setting be viewed configurable evaluation procedure for Method. It creates vari- ous training environments, evaluates the method, and finally returns some Results. These results contain various metrics relevant the setting. The training/testing routine for each setting is imple-
5

---

Under review as a conference paper at ICLR 2022
mented according to the evaluation protocol of that setting in the literature. An example of applying a method onto multiple settings is shown in Listing 1.
Concretely, settings create the training, validation, and testing environments that method interacts with. This interface also makes it possible for methods to leverage PyTocc--Lihhting (Falcon et al., 2019) perform nigh-performance training of their models.² For more information the interactions between Sequoia and Py PyTrch--Lightnnng see App. C.
Settings available for each combination of the CL assumptions, along with the choice of one of RL / SL (as illustrated in Figure 2), for a total of 12 settings.' These two "branches" (one for CRL and the other CSL.) form the basis of Sequoia's eponymous tree of settings. Each setting inherits from one or more parent settings, following the abeve-mentioned organization.
Every Sett ing created by extending a more general Sett Setting and adding additional assumptions. This inheritance relationship from one setting to the next also extends to the setting's 's environments (Environment ), as well as the objects (Observation voos,, Actions, and Rewards) they create. See App. D for an illustration of this principle.
3.2 ENVIRONMENTS
Settings in Sequoia create training, validation, and testing anwironments, which adhere to both the gym. Env and the torch. DataLoader APIs. This makes it easy for SL researchers to transition to RL and vire-versa. These environments receive Act Actions and return Observations and Rewards, Observations contain the input samples x, and may also contain task labels for each sample, depending on the setting. These objects have the same structure in both RI. and SL. settings. However, described in $ 2.2, in SI., Act ons correspond to the predictions, while Rewards correspond to targets or labels. These objects are defined by the Sett ing and follow the same pattern of inheritance as the settings themselves. The structure of these objects are reflected in the environment's observation, action, and reward spaces, which are used within methods to create their models.
Supervised Learning environments Sequoia supports most of the datasets traditionally used in continual supervised learning research, through its use of the Continuun package (Douillard & Lesort, 2021). The list of supported datasets is available in Table 1.
Reinforcement Learning environments Through its close integration with gym, Sequoia is able to use any gym-compatible environment as the "dataset" used by its RL settings. For settings with multiple tasks, Sequoia simply needs a way to sample new tasks for the chosen environment. This mechanism makes it casy to add support for new or existing gym environments. An example of this is included in App. D.
Sequoia creates continuous or discrete tasks, depending on the choice of setting and dataset/en- vironment. For example, when using one of the classic-control environments from gym such as CartPole, tasks are created by sampling new set of values for the environment constants such as the gravity, the length of the pole, the mass of the cart, etc. This is also the case for the well-known 1 fCheetah, h, Wa ker2d, and Hopper MuJoCo environments, where tasks can created by introducing changes in the environmental constants such gravity. Continuous tasks can thus easily be created in this case, as the environment is able to respond dynamically to changes in these values at every step, and the task evolve smoothly by interpolating between different target values.
Other gym environments become available when using RL settings with discrete tasks (i.e. all wasteeestkkkanssaaeeeee
settings that inherit from give Sequoia a list of environments to use for each task, and the constructed Setting will then use them as part of its evaluation procedure.
ing), as it becomes possible simply
We use this feature to construct continual variants of the MT10, MT50 benchmarks from Meta- World (Yu al., 2019), well to replicate the CW10 and CW20 benchmarks introduced in Wolezyk
It is important to that methods are no way required to use PyTorch-L PyTorch-Lightning. 3 'Other common SL settings, such as rely on additional family of assumption, and are thus omitted from the main portion of this paper for sake of brevity and clarity.
learning are also available in Sequoia. but they

---

Under review as a conference paper at ICLR 2022
Methods
SL
BaseMethha.,base, EWC. PackNet }. PNN. replay, HAT, CN-DPM
aaalancheeenaive, AGEM, CWR* " EWC, Gdumb, GEM, LWF, replay, SI}
RL
BaseMethod.(baxx, EWC, PackNet ). PNN
stable_basee www.baaeeeeesee 3...3 DDPG, DQN, PPO, SAC, TD3}
Sontinual_world.$55 AGEM, EWC, VCL, PackNet, 1.2 reg., MAS, replay]
Environments
SI.
continuum (Ktpuummm(K_OO.....0 (K,F.Q, ) MNIST, Cifar10(0), ImageNer100K0), Core50, Synbols]
RL
gym.[Hoppert Half-Chettah, Walker2d, Car(Pole. Pendulum, MontainCar) Monsterkong,
metaworld. ( (MT10. MT50). conting innallll... wwrld. CW20)
Metries
(Transfer Matrix, forward transfer, backward transfer, Average final performance,
Online Training Performance)
SL.
(loss, accuracy)
[loss, total reward, average reward, episode length)
Table I: Sequoia's methods, environments and metries. Most of the RL settings in Sequoia can be passed custom environments to use for each task. This makes it possible to use virtually any gym environment to create custom incremental RL settings. The environments listed here are those explicitly supported in Sequoia, where multiple tasks can be sampled within a single environment.
import gym
sequoia. settings import secting, Environment, Chservations, Actions from sequoia methods import Nethod
class DamaModal:
def (self, observations: Ooservations) >> Actions:
class DamoMethod (Mathad, sertt__ettnnn--ettiiie
*aa #seudocode for Nethod that targets given Setting. **. def configura (self, satting: Setting):
Called by setting before training begins.
self. - DemuModel (eeLing
wweeting.ccbervatliee iiservalion_spede setling accivn_apacs,
setting.reward rww.rrd te.tasks-sttinng nb_sesks)
self. optimizer Adam
def fit (self, train_env: Envirenment, valid_eny: knvironment):
# Train a mode? caiig these envi ronments from the setting. all Bnvirenments for epoch in range (self. e_poche)
gya environmente,
on later.
celf.modes train_epoch woll (tall (valid_env)
def actiinatons (salf, observations: observations) -> Actions:
# Celled by the setting for inference (at teat-time). actions mede lobservations) return actiens
def on task, mook_awlthhee tsell taak_id: optal [fnt]]:
$ Geta called on task boundaries, depending on the setting.
mollal propare_ Yar_new_tast tnew_toakstaak_te
Listing Pseudocode for creating a new Method.
et al. (2021). Sequoia also introduces a non-stationary version of the MonsterKong environment, described in App. E.2. more complete list of the supported environments is shown Table 1.
3.3 METHODS
Methods hold the logic related to the model and training algorithm. When defined, each method selects a "target setting" from those available in the tree. method can be applied setting well any setting which inherits from it (i.e. any setting which is child node of the target setting). We now provide a brief description of the different types of Methods available in Sequoia. An illustration of the Method API can be seen in Listing 2.
target
General methods: Methods in Sequoia can target settings from either the RL or SL branches of the tree. Additionally, it is also possible to select one of the settings from the central CL branch for instance, Incremental Learning. This makes methods applicable to both the CRL and CSL variants of that setting. One such method is the saseMethod, which can be applied to any setting in the tree,

---

Under review as a conference paper at ICLR 2022
and is provided modular, customizable, jumping off point for new users. This BaseMethod is equipped with modules for task inference and multi-head prediction. See App. B for a more in-depth discussion of its features and capabilities.
Sequoia benefits from other CL frameworks such as Avalanche (Lomonaco al., 2021). Avalanche offers both standardized benchmarks well growing set of CL methods, which are referred to as strategies in Avalanche. Sequoia reuses these strategies as Method classes, See Table for complete list of such methods.
Supervised Learning Methods:
Reinforcement Learning Methods: Settings in Sequoia produce Environments, which adhere exactly to the Gym API. is therefore easy import existing RI. tools and libraries and use them to create new methods.
As an example, here we enlist the help of a specialized framework for RL, namely stable- baselines3 (Raffin et al., 2019) The A2C, PPO, DQN, DDPG, TD3 and SAC algorithms from SB3 were easily introduced as new Method classes in Sequoia, without duplicating any code. These methods are applicable onto any of the RL settings in the tree.
In addition to these RL backbones from SB3, Continual reinforcement learning methods are also available. These methods were adapted from the work of Wolczyk et al. (2021), which introduced the CW10 and CW20 benchmarks for continual learning. based on sequences of tasks from Meta- World (Yu et al., 2019). The authors also provided implementations for CRL algorithms, built on top of a SAC(Haanojj et al., 2018) backbone. These algorithms were adapted from their original implementation and made available as CRL methods in Sequoia (see Table for full the list).
4 EXPERIMENTS
Sequoia's design makes it easy to conduct large-scale experiments to compare the performance of different methods on given setting, or to evaluate the performance of given method across multiple settings and datasets. We illustrate this by performing large-scale empirical studies in- volving all the settings and methods available in Sequoia, both in CRL and CSL. Each study involves up to 20 hyper-parameter configurations for each combination of setting, method, and dataset, in both RL and SI., for a combined total of x 8000 individual completed runs. The rest of this section provides brief overview of these experiments, which are also publicly available at https: wando. a sequoia/ All results are reproduced in App. in a larger format and accompanied with further analysis.
Continual Supervised Learning. As part of the CSL study, we use some of the "standard" image classification datasets such as MNIST (LeCun & Cortes, 2010), Cifar10, and Cifar100 (Krizhevsky et al., 2009). Furthermore, we also include the Synbols (Lacoste et al., 2020) dataset, a character dataset composed of two independent labels: the characters and the fonts. (see App. E.1 for the motivation). Exhaustive results can be found at https:/ /wwndb...... , a sequoiquuaaa a cs csssssssset lssddy
A sample of these results is illustrated in Figure 3, which shows results of various methods in the class-IL and task-IL settings in terms of their final performance and runtime. We note that some Avalanche methods achieve lower than chance accuracy in task-IL because they do not use the task label to mask out the classes that lie outside the tested task.
Continual Reinforcement Learning. We apply the RI. methods from SB3 on multiple bench- marks built on HalfCheetah-v2, Hopper-v2, MountainCar-vO, CartPole-vO, Meta World-v2 (details in App. E). We also introduce new discrete domain benchmark, namely ContinullMMonsterKooe that we developed to study forward transfer in a more meaningful way (see App. E.2 for more de- tails). Complete results are available https: / /wwnddb......///.////......b. ai/ sequsiiuiiaaaa htcprrrr__sttttee study.
A sample of these results is in Figure 4. It presents various methods in the traditional and incremental learning settings with their final performance, online performance and normalized runtime.
In Table 2, we apply the continual wworldd methods, built on top of SAC, on a incremental RL benchmark inspired by Mendez et al. (2020) (see App. E.3 for more details). Finally, Figure 11 shows the transfer matrix achieved by one such algorithm, namely PPO (Schulman et al., 2017).
"While most other methods use PyTocch these methods are implemented using Tensorflow.
We will update these sample studies periodically reflect all future improvements made the framework.
8

---

Under review as a conference paper at ICLR 2022
Final Performance
1.0
lask-lncremmnntll MNIST
Final Porformance
1.0
Insemencrrmeental
Final Parfommncc
1.0
Task-Incremental Synbols
BesaMeilodd EWC
Avalanche, Avalanche.GDumb GDumb
PNN
0.5
BasaMethod
Avalanche
Avalanchhe EWC
Avalanchee Si
10"
10'
10"
0.0
10
10"
10"
10
0.0
10"
10"
10"
Avalanche Replay
Runtime (secands)
Runtime (seconds)
Runtime (secandis)
Class Incremental MNIST
Class Intseminnremetllll CIFAR10
1.0
Clasa-incremental Synbols
Avalanche,
Final Performance
Final Perfarmance
Final Final Perforaance
BaseMeihod EWC
BaseMethod
Avalanchhe SI
0.5
0.5
0.5
Avallanche GEM
Avalanche
Replay
0.0
0.0
Avalanche Roplay
10"
10" 10"
10"
10'
10"
10" Runtime (sccands)
10"
Runtime (secands)
Runtime (seconds)
Figure 3: Incremental Supervised Learning results. Final performance (vertical axis) is plotted against runtime (horizontal axis). The methods achieving the best trade-off lie closer to the top-left of the figures. Task-Incremental and Class-Incremental results are presented the top and bottom row, respectively. The dotted line shows chance accuracy for each setting-datasee combination. For each methods, several trials are presented depending on metries composed linear combination of final performance and (normalized) runtime. GEM and GDumb achieve the best tradoff, although the latter cannot make predictions in an online manner and thus serves more reference point.
Traditional HallCheetahh-grawe
Incramental HalfChettah-gravh
Final Performance
10000
Final Performance
6000
86300PG
(F)
OP(1)
R (L)
583.TD0 TD3
tuning
278
17.1
5000
2500
883.22C
EWC
529
230
863.SAC
MAS
637
238
20.0
PackNet
1002
267
2500
5000
7500
2000
4000
Memory
Online Performanc mance
Orlina Parformance
A.GEM
776
281
24.5
Traditional MonsterKKng
Incremental MonsterKang
Firal Perfomance
Perforaacc
8a
Table 2: Incremental RL results. Multiple CRL methods, all built top of SAC, are tested on the Hopper-Bodyee Bodyparts benchmarks. FP and OP stands for final and online performance, respectively, whereas R stands for runtime, reported in hours. Results are averaged over 5 seeds. All CRL methods out- performed the Fine-tuming baseline, validating their efficacy. Experi- ence replay with a Perfect Memory achieves the best retained perfor- mance on all tasks, followed closely by PackNet.
40
SB3
to
30
&
SB3.DON
18
20
40
60
80
Online Performance
Online Performance
Figure 4: Impact of the RL backbone algorithm in Traditional and Incremental RL. Final performance (vertical axis) is plotted against online performance (horizontal axis). The bubbles' size indicates the normalized runtime of the methods. The methods achieving the best trade-off lie closer to the top-right of the fig- ures and have smaller bubble size. Datasets are presented in each row and settings are presented in each column. For each method, several trials are presented depending on metries composed of linear combination of final performance and online performance.
CONCLUSION
In this work, we introduce Sequoia: publicly available framework to organize virtually all research settings from both the fields of Continual Supervised and Continual Reinforcement learning. Se- quoia also makes methods are directly reusable by contract across settings using inheritance. It is our hope that Sequoia will be useful to new and experienced researchers in CL. Further, the princi- ples used construct this framework for CL could very well be applied to other fields of research, effectively growing the tree towards new and interesting directions. We welcome suggestions and contributions that effect in GitHub page at (this GitHub url).

---

Under review as a conference paper at ICLR 2022
Reproducibility statement To facilitate reproducing results of our experiments, include an anonymized version of the Sequoia codebase. All results from the experiments of Š 4 can be ob- served at https:/ /'wandbbbbb/ ai sequoia. Each run includes the exact command used, well as the git state, the complete system specification, hyper-parameter configurations, and random seeds used.
While most sources of randomness are accounted for in Sequoia, we are still in the process of making settings entirely deterministic given random seed. In other words, for some combinations of settings and methods, launching two runs with the exact same arguments and seeds do sometimes produce different results. Making settings and methods entirely deterministic is part of the plans for future work in this project.
REFERENCES
Rahaf Aljundi, Francesca Babiloni. Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. frek-free continual learning. 2019 IEEE/CVF Conference Computer Vision and Patlern Recognition (CVPR), pp. 11246-11255, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew Taylor. Online multi-task learning for policy gradient methods. In International conference on machine learning, pp. 1206-12144 2014.
André Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. arXiv preprint arXiv:1901.1096 2019.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page- Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, al. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. Advances in Neural Information Processing Systems, 33, 2020.
Daniele Calandriello, Alessandro Lazaric, and Marcello Restelli. Sparse multi-task reinforcement learning. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in neural information processing systems 27, pp. 119-827. Curran Associates, Inc., 2014.
Yash Chandak, Georgios Theocharous, Chris Nota, and Philip Thomas. Lifelong learnigg with changing action set. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3373-3380, 2020a.
Yash Chandak, Georgios Theocharous, Shiv Shankar, Sridhar Mahadevan, Martha White, and Philip S Thomas. Optimizing for the future in non-stationary mdps. arXiv preprint wrXiv:2005.08158. 2020b.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with A-GEM. In International Conference of Learning Representations (ICLR), 2019.
Samuel PM Choi, Dit-Yan Yeung, and Nevin Zhang. Hidden-mode markov decision processes for nonstationary sequential decision making. In Sequence Learning, pp. 264-287. Springer, 2000.
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
10

---

Under review as a conference paper at ICLR 2022
Arthur Douillard and Timothée Lesort. learning scenarios, 2021.
Continuum: Simple management of complex continual
William Falcon et al.
Pytorch lightning.
GitHub.
Note:
3, 2019.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:/8/0.00//33 2018.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint WxXv::880.009735 2018.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint VrXiv:/701.000 2017.
Robert M. French. Catastrophic forgetting in connectionis networks. Trends in Cognitive Sciences, 3(4):128-135, 1999. ISSN 13646613, doi: 10100066$113444-6610000000004440 URL https:
www. www.sseeeeeiiiic
com/
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870. PMLR, 2018.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. ArXiv, abs/1912.08866, 2019.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu. Yee Whye Teh, and Razvan Pas- canu. Task agnostic continual learning via meta learning. ArXiv, htss/9906.05201 2019.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Christos Kaplanis, Claudia Clopath, and Murray Shanahan. Continual reinforcement learning with multi-timescale replay. arXiv preprint arXiv:2004.07530, 2020.
Khimya Khetarpal, Zafarali Ahmed, Andre Cianflone, Riashat Islam, and Joelle Pineau. Re- evaluate: Reproducibility in evaluating reinforcement learning algorithms. 2018.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives. arXiv preprint arXiv:2012.1 13490, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):521-33266 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of Technical report, Citeseer, 2009.
features from tiny images.
Alexandre Lacoste, Pau Rodríguez López, Frederic Branchaud-Charron. Parmida Atighe- hchian, Massimo Caccia, Issam Hadj Laradji, Alexandre Drouin, Matthew Craddock, Lau- rent Charlin, and David Vázquez. Synbols: Probing learning algorithms with synthetic datasets. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances Neural Information Processing Systems, volume 33, pp- 134-146. Curran Asso- ciates, Inc., 2020. URL hww.thttss.ccc httt::::s//ppooeediggg. neurips. cc/ httpp/ppppr222200 /ilee/ Paper, haper.pdt. .df.
Nicholas C. Landolfi, Garrett Thomas, and Tengyu Ma. A masel-based approach for sample- efficient multi-task reinforcement learning, 2019.

---

Under review as a conference paper at ICLR 2022
Yann
LeCun and Corinna Cortes. http://yam.lecc..omm/ekdde
MNIST
handwritten
digit
database.
2010.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free free continual learning, 2020.
Timothée Lesort, Hugo Sisellle-Duuré, Michael Garcia- Ortii, -ean-François Goudou, and David Filliat. Generative Models from the perspective Continual Learning. International Joint Conference Neural Networks (IJCNN), 2019a.
Timothée Lesort, Alexander Gepperth, Andrei Stoian, and David Filliat. Marginal replay vs condi- tional replay for continual learning. In International Conference Artificial Neural Networks, pp. 466-480. Springer, 2019b. URL. httpt::t////....... / /ttpp:///aaxivi..// //aaaaivvvooroo.o org/ abs/ /181 12069.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. Continual learning for robotics: Definition, framework, learning strategies, opportunitiies and challenges. Information Fusion, 58:52 68, 2020. ISSN 1566-2535. doi: http://ddo.org111.0116......n.i, science/articon sccannc//arricee///////622222392
URL http: /ttp:://www...// www.sciincedirect com/
HongLin Li, Payam Barnaghi, Shirin Enshacifar, and Frieder Ganz.
Continual learning using
bayesian neural networks, 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint vrXiv:/509.02977 2015.
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Grafficti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: end-to-end library for continual learning, 2021.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NIPS), 2017.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning, 2018.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes The benefit of multitask representation learning. J. Mach. Learn. Res., 17(1):2853-288 January 2016. ISSN 1532-4435.
Jorge Armando Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. abs/2007.07011. 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature. 5 518(7540):529-- 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, PMLR, 2016.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Tumne. Variational continual learn- ing. In International Conference Learning Representations (ICLR), 2018.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In /CLR, 2016.
12

---

Under review as a conference paper at ICLR 2022
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. 2020. URL http:/ httt:://www.obot.... robots. ox ac. uk/ -vv// publications/22
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- mann. Stable baselines3. https: / /tt///gittti github..t.. wotp///iaa--////tttal--aa-----a- beeebiness,,eee.eeeeeee 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Computer Vision and Pattern Recognition (CVPR), 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv: 1910, 2018.
Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77-104,
1997.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, 2019.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive Neural Networks. ArXiv e-prints, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint orXiv:1707.000
2017.
Joan Serrà, Dídac Surís, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. CoRR, Abs//801.01423, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems (NIPS), 2017.
Norman Tasfi. Pygame learning environment. https:///ithhi/ wtt.:/aua.ccommmtti....... LeaeeeLLLLannnn---eeaa
2016.
Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research,
2009.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and autonomous systems, 15(--2):25-46, 1995.
Sebastian Thrun and Anton Schwartz. Finding structure in reinforcement learning. Advances in neural information processing systems, pp. 385-392. 1995.
René Traoré, Hugo Dusre,, Timothée Lesort, Te Sun, Guanghang Cai, Natalia Díaz
Rodríguez, and David Filliat. Discorl: Continual reinforcement learning via policy distillation.
CoRR, abs/1907.nn 005555555 2019. URL http:: / http//axxiie rxiv org/ abs / 05855.
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprint 17X1v:9904.07734, 2019.
Maciej Wolczyk. Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milos. Continual world: A robotic benchmark for continual reinforcement learning. CoRR. abs/2105.10019, 2021. abs / 2 05. 10919.
URL httpt:
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity, arXiv preprint arXiv:2006. 10701, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. CoRR, abs/1910.10897, 2019. URL http: //tpiv/rvvvv//iv////v//ea////aaa org/ abs / 19100..1111. 10897.
Friedeman Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning (ICML), 2017.
Chen Zeno Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes, 2018
13

---

Under review as a conference paper at ICLR 2022
A SUPPORTED METHODS
One of Sequoia's 's biggest strength how casy it to extend. Most methods in Sequoia the result directly reusing existing implementations from other frameworks and repositories, such as Avalanchel.omonaco et al. (2021), Baable-BaslinneRRRffin et al. (2019) and Continual WorldWolczyk et al. (2021). Table shows all the methods currently available in Sequoia.
Method
Target setting
BaseMethod
Setting (all)
BaseMechod. EWC Kirkpatrick et al. (2017)
Setting (all)
Packeet Mallya & Lazebnik (2018)
Incremental Learning (RL + SL)
replay
Incremental SL
CN-DPM Lee et al. (2020)
Continual SL
HAT Serrà al. (2018)
Task-Incremental SL
PNN Rusu et al. (2016)
Incremental SL.
Avalanche.naive Lomonaco al. (2021)
Incremental SL
Avalanche.AGEM Chaudhry et al. (2019)
Incremental SL
Avalanche.ccrrsttrr Lomonaco al. (2021)
Incremental SL
Avalanche.EEWC Kirkpatrick Kirkaattt et al. (2017)
Incremental SL
Aval anchheGGuuuubbb Prabhu et al. (2020)
Incremental SI.
Aval GGM Lopez-Paz & Ranzato (2017)
Incremental SI.
Ava Avalanche.LLWFI1. Li al. (2019)
Incremental SI.
kvalanche.replay
Incremental SL
Sellancha.SS7Zenee et al. (2017)
Incremental SL
Stableebbasallness_A222MM Mnih et al. (2016)
Incremental RL
stablee-bassllinse..DDDD Lillicrap et al. (2015)
Continual RL
stable--baselinesss.D0 Mnih et al. (2015)
Continual RL
3table-bassllnness3PP Schulman et al. (2017)
Continual RL
stable--baseiiinse:...... Haarnoja et al. (2018)
Continual RL
stable-bbaselinnes:......
T.D3 Fujimoto et al. (2018)
Continual RL
continual__wrric...4. Haarnoja al. (2018)
Incremental RL
Contimoua__worrddAGGEE.. Chaudhry et al. (2019)
Incremental RL
continual__oorld.EWW Kirkpatrick al. (2017)
Incremental RL
continual_worldd VCL Nguyen et al. (2018)
Incremental RL
continual_worldd.Me Aljundi et al. (2018)
Incremental RL
continual_worie regularization
Incremental RL
continual_woric_PPe Mallya Lazebnik (2018)
Incremental RL
contimual_woric.Reee
Incremental RL
Table 3: Sequoia's methods support. Each method specifies target setting, listed on the right. Most methods are applicable in either RL or SL, while some can be applied to both. Methods also specify the "level of nonstationarity" they are prepared handle, as choice of one of Continual (which is also referred to as Continuous Task-Agnostic CL in $ 2.1), Discrete, Incremental, Task- Incremental, Traditional, and Multi-Task.
B THE BASE METHOD
While developing new Method in Sequoia, users are encouraged to separate the training logic from the networks used, the former being contained in the Method, and the latter in a model class, advocated by PyTorch-Lightning (Falcon et al., 2019) (PL), powerful research library, which we employ as part of this BascMethod.
The has is accompanied by the BaseModel, which acts as modular and ex- tendable model for CL Methods to use. This BaseModel adheres to PyTorch- Loghhi-Lggtnnng'ss LightningModule interface, making it easy extend and customize with additional callbacks and loggers. Likewise, the EaseMethor hod employs a pl. Trainer, which is able to train the BaseModel the Krvi ronments produced by any setting. Sequoia's Settings also closely related PL's
14

---

Under review as a conference paper at ICLR 2022
gym
Env
Torch
pytorch Jightning
abyervation space. Space=Obus artion, spare Spareclets
DataLoader
DogtnnnDattaoouee
stepláction) TuplejObs, del 0 rexet0 > Gbs
sequoia
stabic (class) attributes
Ensionmmnn
Chrervations: bhervtatt
Lewards Spillirardss
action, space: Sparedittinsy
renard ssace: Spaccellewerdi>
creates
4nhanted Unfenmmbenatbuulee schie0
stepi4ctions actions) mellecbsreevettnn Rewards, bool, Diet
reset) > Observations
val fotaicaderr de Enviromment test, Jataloadbri Envirommnt
Abstram Method
applyDMethod
yields
receives
interacts with
applies
targets
produres
O. Tenso
O V.ppred Tencor
D Tensor
axtlends
Method
Aarget setting Typets)
abstract (requredd methore-
Rosults
actorulobssrvaaionn Obseriations action space. Space!
O dbjective foot
optional methods-
coní gurelacting: 5)
optora][ett)
tesihees, enw. Envinanmendi
can use
Model
fonwerdtinput
Figure 5: UML Diagram showing the main abstractions of Sequoia.
DataMocule abstraction. See App. C for a further discussion of the relationship between Sequoia and Pytorch-Lightning.
Using this BaseModel when creating a new CL method can be particularly useful when transitioning from CL Setting to its parent, it comes equipped with most of the components required to handle such transitions (e.g. task inference, multi-head prediction, etc.) These components, as well as the underlying encoder, output head, loss function, etc. easily be replaced or customized.
Additional losses can also be added to the BaseModel through a modular interface, which was explicitly designed to facilitate exploration of self-supervised learning research.
15

---

Under review as a conference paper at ICLR 2022
C ADDING NEW SETTINGS TO SEQUOIA
The very simple definition the Setting abstract base class means that new Settings are not required place themselves into our existing inheritance hierarchy, and could also be related to continual learning at all! It is however preferable, whenever possible, to find the closest existing Setting within Sequoia and either add additional assumptions or remove extraneous ones, by creating the new Setting either below its closest relative or above it when adding or removing assumptions respectively.
The most general Setting in our current hierarchy also referred to as the "root" setting inherits from this abstract base class, while also building upon the elegant DataModuie abstraction intro- duced by Pytorch-Lightming Falcon et al. (2019), in which the Data is the entity responsible for the preparation of the data, as well as for the creation of the training, validation and testing DataLoaders. Models in pytorch lightning can thus easily train and evaluate themselves through this standardized API, where dataloaders can swapped out between experiments. Sequoia's 's main contributions can thus be viewed taking this idea one step further, by 1) giving control of the "main loop" to this construct (through the addition of the apply method), 2) expanding this idea into the realm of Reinforcement Learning by moving from PyTorch's aat loaders level abstraction (Environments) s), and 3) organizing these modules into an inheritance hierarchy.
higher-
Given how all current Sequoia Settings are instances of PL's LightningDas inDat aModule class, is easy to use pytorch lightning for the training of Methods in Sequoia. This is one of the reasons why, for instance, the BaseMethod uses Pytorch Lightning's Trainer class in its implementation. However, the trainer-based API is not directly usable, due to the very nature of CL problems, in which there are training dataloaders for each task, which isn't currently possible through the standard Pytorch Lightning's API.
16

---

Under review as a conference paper at ICLR 2022
from typing import List, Dict from sequoia.satt sattings.rt import cana__onniimeus_tae from gym envs claasic_conttri import CartPo.eBav import numpy np
A Cont inuous the environment:
dict mapping actributes to volues
ContinueusTask bict (str, flcat]
Snake_ conakeeoottiuouuee.. wtwpsasass.....sstee...e. tes..reeeiateerrtatotolllaa.. (Cartllleev))
def Bakke_takk_oo.ce mor_my_envi
env: CartPoleRnv, step: int, change_steps: sr Mtst [int], seed: 1nt None, kakargg,,
Continuoustass:
NOTE; task sampling should be repraduaible given "seed".
step_seed
None else step (etep_seed)
rng -
return {
"gravity"; 9.8 vng.normal (1, 0.5),
"masscart"; 1.0 rng. (1, 0.51,
"masspola": 0.1 rag.nermal (1, 0.51,
"length": D.5 + nog.no (1, 0.51,
"Corce_maq";
ing.aormallt f1, 0.5),
"tau": 0.02 mng.normal 0.5),
Listing 3: Example of how to add new RI. environments to Sequoia. this example, we register a function which will be used sample continuous tasks for this environment, allowing it become used as part of the Continuous Aask-Agnostic Continual RL Setting and all of its descendants.
import operator
from typing import List, Dict, Union, Callable import numpy as np
import gym
from metaworld envs. mujoco. sawyer_xy,, v2 import SawyerReachZavi2
metaworld import
from sequoia. settings. xl. discrete import imke_diacreto_tasi
the of Discrete Rt settings, "continuous" task as before or a calieble which will be applied the cnvironment when task boundary is reached:
either retuzn
Continucusiask vict [str, flcat]
IncromentalTask = Union Onion(Contes [Contimuuustaak Callable [gym,Env), Nono]]
Saake_discretee Lack. wac..aakkkeee............kk..kt. (Bawy))))ecc
def make__iiscret_ttarke wtw.:eooo__tttttrrll..oooo
SawyorRaachEnwV2 step: int, change_ctops: List (int),
seed: int None, **kwargs,
Incrementallask:
benchmark XL10 (seed rng www.oo.ddaaallee.... (seed)
sone_metawordd_cadd rag. choice (benchmark. wwwpecamark.trannccom train_tasksii
NOTE: Equivalent to the foliowing, but the benefit
being pickleable for vectorized envs:
retura lambda eav: eav, suv.set_tee task (swne_metawwrldddcc.... task) return operator.co trscccscsleee"sst_ttte
some_metarorld_tart
Listing Example of how to add support for new RL environments to Sequoia the case discrete tasks, which applied when task boundary is reached. In this example, we register a function which will be used to sample discrete tasks for this environment, allowing it to become used as part of the Discrete Aano-tg RL setting and its descendants.
D ADDING NEW ENVIRONMENTS
New environments can added to Sequoia by registering new handler creating new tasks, as can be seen in Listing 3 for continuous tasks, and in Listing 4 for discrete tasks.
17

---

Under review as a conference paper at ICLR 2022
base
gym
Rewards
C
Actions
Observations
SettingABC
Erv
continuous
AabstractSettrgg Condrruuusaakkkgnooskkeeee
clear lask, boundaries: bool Falne
discrete
abbsraccteettingg Siirrett
assumptions
clear task, boundartes: Constent(boo)
assumptions
incremental
AbstractSettingg IncrementalSetting
assumptions
Inown, task boundaries, btw.k_kk_nudrriieseemommmm test bime Constert[bool) True
Constant(bool)
Inherited assumptions
clear task boundaries: Len_ttas__boundriiss
Constent[bool]
True
task incremental
traditional
wbbstractSetting Taskincrementaleetirg
<bbstractSetönn- TraditonalSetting
New assumptions
o task labels, at train time: Constantibol] - True o task labels labels at test time: Constant(bool) True
New assumptions
statonary.contod Constan[[bol] True
inherited assumpbons
inherited assumptons
clear task bourdaries: Constantboo)] True known, boundaries, at frain Constent(bool]
clear, task boundaries: Constant[boa||
test__mme Constan([baol]
multi_tass
.bbstractSetting- MulbTaskSSeting
assumptions icsmpared Traditonall.
task labels time: Constantbod) taskllebels testt time: Constant[bosl True
assumptions (compared Taskiccrementa
stationary, Conteit(bool] True
assumption Contextbeool
о task labels
time Constantboo)
о tank labels test time: Constant[boal) True
O clear task Soundaries: Constantsbool] True
O known, task boundaries, at train time: Constantbooll True
о keowwn_tekk boundaries, aeanddries__attett [ Constert(pool) True
Figure 6: UML Diagram of the CL assumptions hierarchy. The CRL and CSL branches are not shown, but follow an identical structure.
18

---

Under review as a conference paper at ICLR 2022
E BENCHMARK DETAILS
E. SPLIT-SYNBOLS DATASET
Currently employed datasets can't 't be used sensibly to construct domain-incrementi learning prob- lems. Some have used MNIST to construct Permuted-MNIST and Rotated-MNIST, however, Far- quhar & Gal (2018) have explained and demonstrated why such benchmarks are flawed and bias their results unfairly towards some methods. Motivated by this, introduce Split-Synbols. Based on the Synbols dataset (Lacoste et al., 2020), character classification dataset in which examples have extra label corresponding to their font, can easily construct sensible domain-incremennal benchmark where e.g. a font would consist of domain.
For the experiments, however, opted for class- dass-inccrmennal version to increase the difficulty. We prescribe a segmentation into 12 tasks to be learned sequentially, each consisting of a 4-way classification problem. Some example of Synbols character are displayed in Figure 7.
S
N
b
S
h
Figure 7: Split-Synbols. Example of Synbols character to classify.
F.2 CONTINUAL MONSTTRR KONG ENVIRONMENT
With rapid advancements in the field of deep RI., continual RL never-ending-RL. has witnessed rekindled interest towards the goal for broad-AT in recent years. While significant progress has been made in related domains such as transfer learning Taylor & Stone (2009), multi-task learning Ammar et al. (2014); Parisotto et al. (2016); Calandriello et al. (2014); Maurer et al. (2016); Barreto et al. RL Farebrother al. (2018), an outstanding
(2019); Landolfi et al. (2019), and generalization bottleneck is the lack of standard tools to develop and evaluate CRL agents Khetarpal et al. (2020). A standardized benchmark will potentially enable rapid research and development of CRL agents. To this end, we propose new CRL benchmark within the unified framework of Sequoia. In particular, we build the CRL benchmark leveraging the Pygame learning environment MonsterKon Tasfi (2016). MonsterKong pixel- lightweight and has casily-customizable domain, making a good choice for evaluating continual learning agents.
Specifically, we design tasks through variety of map configurations. These configurations vary in terms the location of the goal and the location of coins within each level. We introduce ran- domness across runs of task by varying the start locations of the agent. To incorporate the ability to evaluate across specific CRL characteristics, we leverage tasks to define CRL experiments. We design families of tasks leveraging the following abstract concepts: jumping tasks which require the agent to perform jumps across platforms of different lengths in order to collect coins and reach the goal, climbing tasks which require the agent competently navigate ladders in order collect coins and reach the goal, and tasks that combine both of these skills. The specific tasks leveraged as part of the CRL competition are depicted Figure 8. The agent trains on each task for 200,000 steps.
To evaluate the agents on the CRL benchmark, we follow the standard eval- uation introduced above. Final performance reports accumulated reward per episode on all test environments, averaged over all tasks, after the end of training, whereas online performance is mea- sured as the accumulated reward per episode on the training environment of the current task during training of all tasks. For the runtime score, 1.5 hours. Lastly, the agents are allowed a maximum of 200,000 steps per task.
Experiment Details:
set max-runtime of 12 hours and min_runtine to
Ideally, CRL agents must be able to solve tasks by acquiring knowledge in the form of skills, be able to use previously acquired behaviors, and build even more complex behaviours over the course of its lifetime Ring (1997); Thrun & Mitchell (1995); Thrun & Schwartz (1995). While leveraging the MonsterKong environment, it is easy to introduce new environment layouts or modi- fications existing layouts. Configurations could be customized to include arbitrary configurations
Customization:
19

---

Under review as a conference paper at ICLR 2022
Figure 8: Montinua-MMonstrrKong We display the tasks that constitute the benchmark chrono- logical order. The first two tasks the agent's ability to jump between platforms, the second two test its ability to climb ladders and the last four combine both skills.
of coins, ladders, platforms, walls, monsters, fire balls, and spikes. Making custom environment el- ements is straightforward as well, so the environment can be modified to aligned with the properties of the CRL agent that we would like to test.
While in our benchmark mainly focused three families of tasks within the Monsterkong domain, it fairly straightforward to introduce variations map configurations to the framework. Monsterkong provides two degrees of design choices 1. the task definitions and 2. the evolution of tasks referred to as experiment definitions. Due to the nature of how tasks are specified through simple matrices (map configurations), many layers of complexity specification. For example, object addition and removal can induce local variations in reward, nails can be penalizing, diamonds can be bonuses. Additionally, changes the textures of the game like simple changes to the color of the walls, the coins, and the background as well as changes in the lighting are easy to add for users interested to test generalization of the policies learned.
be added through the task
E.3 HALFCHEFTAF-GGRAVITY AND
HalfCheetah-gran -rrvvity and Hopper-Bodypart are two benchmarks introduced in Mendez et al. (2020). In the first, each task consist of a different gravity. In the latter, the agent's 's body parts are changing in size at each tasks. The gravity and body parts values sampled in Mendez et al. (2020). The two benchmarks we study are each composed of 10 tasks.
20

---

Under review as a conference paper at ICLR 2022
F
EXTENDED EXPERIMENTS
Class-Incremeentl MNIST
Task-Incremental MNIST
Avalanche. GDumb
1.0
Final Performance
BaseMethod. EWC
Final Performance
PNN
HAT
0.5
0.5
BaseMethod
Avalanche. GEM
Avalanche. EWC
Avalanche.SI
0.0
10
10"
10"
0.0
10"
10" Runtime (seconds)
2
Avalanche
Replay Replay
Runtime (seconds)
Class-Incremental CIFAR10
Task-Incremental CIFAR10
Avalanche Wuuumb
GDumb
1.0
1.0
BaseMethod.EWV EWC
Final Performance
Final Performance
PNN
HAT
0.5
0.5
BaseMethod
Avalanche GEM
Avalanche. EWC
Avalanche.5
0.0
102
10
10
10
10
10
10
Avalanche. Replay
Runtime (seconds)
Runtime (seconds)
Class-Incremental Synbols
Task-Incremental Synbols
Avalanche. GuulabcheGDDumb
1.0
1.0
BaseMethod. EWC
Final Performance
Final Performance
PNN HAT
0.5
BaseMethod
Avalanche GEM
Avalanche EWC
Avalanche. SI
0.0
0.0
10"
10
10*
10"
10
Avalanche Realay
Runtime (seconds)
Runtime (seconds)
Figure 9: Incremental Supervised Learning results. Transpose of Figure for improved read- ability. Final performance (vertical axis) is plotted against runtime (horizontal axis). The methods achieving the best trade-off lie closer to the top-left of the figures. The dotted line shows chance accuracy for each setting- -attnet combination. For each methods, several trials are presented de- pending on metrics composed of linear combination of final performance and (normalized) runtime. Intuitively, better nerformance in CL normally comes the cost of increased computation. This intuition is reflected in the presented results, as highlighted by the observed correlation between fi- nal performance and runtime. GEM and GDumb achieve the best tradoff, although the latter cannot make predictions in an online manner and thus serves more as reference point.
21

---

Under review as a conference paper at ICLR 2022
Traditional Halrahyetah-gravity
incremental HalfCheetah-gravity
Final Performance
10000
Final Performance
SB3. GDP.
5000
SB3.TD3
5000
SB3.A2C
2500
SB3.PPO
SB3.SAC
2500
5000
7500
0
2000
4000
Online Performance
Online Performance
Traditional MonsterKong
Incremental MonsterKong
Final Performance
Final Performance
60
40
SB3.PPO
40
SB3.A2C
30
SB3.DQN
20
20
18
20
20
40
60
80
Online Performance
Online Performance
Figure 10: Impact of the RL backhone algorithm in Traditional and Incremental RL. Larger version of Figure 4 for improved readability. Final performance (vertical axis) is plotted against online performance (horizontal axis). The bubbles' size indicates the normalized runtime of the methods. Datasets are presented in each row and settings are presented in each column. For each method, several trials are presented depending metrics composed of linear combination of final performance and online performance. The methods achieving the best trade-off lie closer to the top-right of the figures and have smaller bubble size. In general, we observe trade-off between performance and runtime, a tendency also observed in Figure 9. Another interesting trade-off can be observed between final performance and online performance. E.g., in both MonsterKong bench- marks, DQN achieves the best final performance whereas PPO achieves the best online performance. Because the former off- off-policy, it can re-use the previously acquired data to retain its performance on past tasks, increasing final performance. Contrarily, the latter, being on-policy, focuses on the current task and thus learns faster, thereby increasing online performance.
22

---

Under review as a conference paper at ICLR 2022
Values
Task
73.77
16.5
1
4
3
3
Task
67.39
63.19
0
1.5
Task
72.22
13.5
47.27
0
36.36
Task
12.27
4
8
15.5
6.5
4
1.5
61.64
28.08
48.65
26.15
Task
35.75
10.5
33.5
26.36
28.46
Task
3
43.33
14.55
32.35
10
97.95
19
7.5
41.56
21.15
84.38
35.83
Task
Task
Task 2-
Task
Task
Task 5
Task
Task 7-
Figure 11: PPO's Transfer matrix in Soatinual-MonsterKoon Each cell at row and column j indicates the test performance task after having learned tasks 0 through i. The contents of each cell correspond to the average reward per episode obtained in the test environment for the corresponding task. Positive numbers above the diagonal indicate generalization unseen tasks, which is achievable by design in the
benchmark.
23


Reviews:
[{"review_id": "lcYlYv9tU1S", "reviewer": "Reviewer_SrGM", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\nP1) Creating a software package which unifies RL and supervised learning approaches seems like a step in the right direction.\n\nCons:\n\nC1) CL hierarchy - I think there are missing nodes in the hierarchy. For instance, it is important to also consider problem-incremental learning where the task is the same, but the input spaces are different (e.g. the dimensionality of the input changes, not just the input distribution).\n\nC2) Related work: You mention that your framework doesn’t compete with others. It is not clear to me why someone else would not just use the other frameworks. A more in-depth discussion of the previous work is necessary. This way, it would be easier to distinguish your contributions are.\n\nC3) Experiments: The text does not give us information on which methods are implemented by the authors. It gave me the impression that the evaluated methods have all been implemented by another software package.\n\nC4) Experiments: The results are presented but not discussed.\n\nThe provided hierarchy of CL methods might contain an interesting insight, which relates RL and supervised learning approaches to CL. However, due to the lack of comparison to related work, I am not certain of this. \n", "summary_of_the_paper": "The paper attempts to unify all CL research with a single formalism. Next, they present a software implementation of their framework. Finally, experiments are ran which demonstrate that Sequoia can be used to evaluate CL methods.", "summary_of_the_review": "Overall, I don’t see a significant technological or conceptual contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "UwjAKsTbSl", "reviewer": "Reviewer_YwgU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- In section 2, the authors propose a Markovian process that is being ignored in the later parts. It is not clear why this hidden-mode Markov decision process is useful to the proposed framework.\n- While I do appreciate the effort in developing this framework, I am lacking the novelty in the paper. The work is mainly an engineering effort that's appreciated but might not fit this conference.\n", "summary_of_the_paper": "This paper introduces a new continual learning framework that aims to boost the research in the field. This framework is based on a taxonomy of all possible assumptions that are common to CL methods. Moreover, this taxonomy helps in putting supervised and reinforcement methods in a unified framework.\n", "summary_of_the_review": "While I do appreciate the effort behind this work, I doubt the match between this paper and the scope of ICLR.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DvjgSVgmKBi", "reviewer": "Reviewer_pX2k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The goal of the paper is very ambitious. There exist many open source libraries implementing CL methods for supervised learning and reinforcement learning settings.\n\nFRAMEWORK:\nAccording to the paper \"each setting is described as a set of assumptions. A tree-shaped hierarchy emerges from this view\". This shows a limitation of the theoretical framework. Defining settings as a set of assumptions does not result in a tree-shaped hierarchy (it would be a lattice). Therefore, using a tree-shaped hierarchy means that some methods will be compatible in principle but not in practice (in Sequoia) because the tree lacks the connection. This a strong limitation of the framework that is not discussed.\n\nIMPLEMENTATION:\nThe framework heavily relies on several dependencies to implement the heavy lifting (datasets, RL environments, Methods). This is not a big problem by itself, but it is not clear what Sequoia is adding compared to the original libraries.\n\nEXPERIMENTS:\nThe experiments show several baselines in the supervised and reinforcement learning settings. Most methods are not implemented by Sequoia and are inherited from avalanche for SL and stable-baselines and continual-world for RL. One thing that I expected from this section was how the same method could be easily applied to different settings, which is the main claim of the library. Instead, SL and RL settings share zero methods. For example, the EWC implementation is different in the SL and RL settings. Does this mean that users have to implement every method twice? It seems that RL and SL methods are completely separate, which is against the entire spirit of the library. At this point, what is the advantage that Sequoia brings compared to its dependencies (continuum, gym, avalanche, stable-baselines, continual-world)?\n\nADDITIONAL COMMENTS:\n- \"constraints often relate to memory, compute, or time allowed to learn a task\" -> is Sequoia able to check time and memory constraints?\n- is it possible for the end users to define new settings or modify the hierarchy? it would be interesting to see an example.\n- \"We note that some Avalanche methods achieve lower than chance accuracy in task-IL because they do not use the task label to mask out the classes that lie outside the tested task\" -> can you fix this problem? Otherwise a user needs to know the internals of each library to debug the code. This adds a lot of complexity.\n", "summary_of_the_paper": "The paper proposes a theoretical framework to organize research problems in the continual learning (CL) domain according to a hierarchy. This theoretical framework is used as the basic foundation for Sequoia, a software library designed to reuse methods (i.e. training algorithms) across different research problems (settings).", "summary_of_the_review": "I believe the objective of this paper of merging together continual SL and RL is very important and ambitious. However, the paper in the present form has several weaknesses. It is unclear whether the theoretical framework really achieves the paper’s objective. Sequoia (the software) seems to be still at a very alpha stage in its development cycle. I see very little unification in the methods, which is the main scope of the paper. This strongly limits the methods' reuse between the different settings. It is unclear what advantages sequoia is bringing compared to using its dependencies directly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "-QISKIbXYal", "reviewer": "Reviewer_D4MM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n\n&nbsp;\n\n- The continual learning research tree (Figure 2) is well organized and makes a lot of sense to me.\n\n&nbsp;\n\n- I think the motivation of building a unified framework for continual learning is meaningful. Many different continual learning papers evaluate their methods in different benchmark protocols. It is difficult for the following researchers to compare these related methods. \n\n&nbsp;\n\n- Detailed explanations and experimental results are provided in this paper. The authors also provide the results on wandb. It makes obtaining the detailed results for each setting very convenient for the following researchers. \n\n&nbsp;\n\n### Weaknesses\n\n&nbsp;\n\n- Some important continual learning baselines are not included, such as iCaRL (Rebuffi et al., 2017). \n\n&nbsp;\n\n- The authors only provide experiment results on small-scale datasets, such as MNIST and CIFAR. Most continual learning papers, such as iCaRL (Rebuffi et al., 2017) and BiC (Wu et al., 2019), provide the results on large-scale datasets (e.g., ImageNet-1k). As the authors trying to establish a new benchmark protocol, it is important to provide the results on large-scale datasets.\n\n&nbsp;\n\n- This project includes plenty of different methods, but the authors don’t include detailed information (such as the licenses) about the related open-source resources they use. \n\n&nbsp;\n\n- The definition of “incremental learning” in Section 2.1 is ambiguous. I think it is better to use class-incremental learning (or domain-incremental learning) directly. The reasons are as follows. (1) “Incremental learning” is often considered as another name of “continual learning”. It is weird to use it to denote a specific setting of continual learning. (2) You include class-IL and domain-IL in “incremental learning”, but you exclude task-IL. It is not reasonable.\n\n&nbsp;\n\n- The authors include too many details about the code implementation in the paper. I think it is better to move these parts to the appendix and include more experimental results and analyses. ", "summary_of_the_paper": "In this paper, the authors try to establish a unified framework for different continual learning settings. They also provide a Python library, which includes different related methods. Extensive experimental results are also provided. ", "summary_of_the_review": "Overall, I think this project will be a useful tool for the following continual learning researchers. I will recommend acceptance if the authors can address my concerns in the “weaknesses” part. \n\n&nbsp;\n\n### === Post-rebuttal Comments ===\n\nI thought the authors aimed to establish a unified software framework that makes running continual learning experiments easy.\nHowever, after reading the rebuttal, I think Sequoia has the following major issues and the authors failed to address them in the rebuttal:\n\n- ***Sequoia heavily relies on the previous libraries, such as Avalanche and Continuum.*** I don't think this design is very friendly to the researchers. In my personal view, I prefer a framework that is easy to be understood and includes the most popular baselines. I think your framework should be designed for a researcher instead of a software engineer.\n\n- ***Sequoia hasn't been evaluated on large-scale datasets (e.g., ImageNet-1k).*** If I need to use this framework, I need the framework can reproduce the results of the previous baselines correctly. \n\nSo, I don't think Sequoia is very useful to a researcher like me. According to my personal experience, I tend to reject this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript introduces a taxonomy for organizing continual learning research settings and a software framework that realizes this taxonomy. Each continual learning setting is represented by as a set of shared assumptions (e.g., are task IDs observed or not) represented in a hierarchy, and the software is introduced with the hopes of unifying continual learning research.

The manuscript identifies a clear issue in the field: settings and methods for continual learning have proliferated so that there is little coherence in benchmarks, making progress difficult to judge. Reviewers generally agreed that the motivation of building software to help unify continual learning research was a positive.

However, reviewers also pointed to many concerns with the manuscript and software package (Sequoia) that comprises its main contribution. In particular, there is concern that the software is at an early stage of development and makes heavy use of existing libraries to function (e.g. Avalanche and Continuum). This makes it unclear what Sequioa offers over using its dependencies directly. As well, there is concern that multiple standard benchmark tasks and common methods are missing from the implementation — particularly for large scale experiments with, e.g. ImageNet-1k. In theory, the library allows extension and these might be implemented by others in the community. However, this would require that the original manuscript+software are strong enough to draw buy in from other researchers. 
In sum, the manuscript+software does not yet offer a convincing starting point for researchers looking for a starting point to begin their continual learning research.
Decision: Reject