Sequoia: A Software Framework to Unify Continual Learning Research

1 INTRODUCTION With the growing interest in developing methods robust to changes in the data distribution , research in continual learning ( CL ) has gained traction in recent years ( Delange et al. , 2021 ; Caccia et al. , 2020 ; Parisi et al. , 2019 ; Lesort et al. , 2020 ) . CL enables models to acquire knowledge from nonstationary data , learning new and possibly more complex tasks , while retaining performance on previously-learned tasks . To instantiate a CL problem , one must first make assumptions about the data distribution and set constraints to enforce non-stationary learning . For instance , assumptions are often made about the type and number of tasks , the task boundaries , or the availability of task labels , while constraints often relate to memory , compute , or time allowed to learn a task . Combinations of assumptions , rules , and datasets have resulted in a multitude of settings ( Khetarpal et al. , 2020 ) . We argue that the increased popularity of CL , combined with a lack of unification — in part due to the large number of settings and the absence of well-defined applications — has led to a “ research jungle ” that may be slowing down progress . We identify some of the main challenges associated with the lack of unification in CL : i ) Evaluation . Methods in CL are often studied under a small subset of the available settings , making it difficult to evaluate them , as their problem domains don ’ t always overlap . Consequently , it is challenging to determine if a method will generalize beyond the setting it was designed for . To add to this , continual reinforcement learning ( CRL ) poses further challenges in evaluation due to the lack of a clear distinction between training and testing phases ( Khetarpal et al. , 2018 ) . Moreover , resource consumption is a critical factor for evaluation of CL methods which is often overlooked due to the lack of standardized evaluation protocols . Thus , there is a need for standardization of the infrastructure used to evaluate CL methods . ii ) Reproducibility . In order to analyze specific properties of novel methods , researchers tend to re-implement baselines and adapt them to their particular needs ( Henderson et al. , 2018 ) . These baselines are often not described in enough detail to ensure reproducibility , e.g . a prescribed hyperparameter search strategy , computational requirements , open source libraries , etc . iii ) CSL and CRL evolve in silos . Continual supervised learning ( CSL ) and continual reinforcement learning ( CRL ) are considered to be independent settings in the literature and thus , they are evolving separately . However , most methods in one field can be instantiated in the other , resulting in duplicate efforts such as replay for CSL ( Rebuffi et al. , 2017 ; Lesort et al. , 2019a ; Shin et al. , 2017 ; Lesort et al. , 2019b ; Prabhu et al. , 2020 ) and replay for CRL ( Traoré et al. , 2019 ; Rolnick et al. , 2019 ; Kaplanis et al. , 2020 ) . To this end , we advocate that the unification of both fields would greatly reduce these duplicate efforts and accelerate CL research . In this work , we present Sequoia , a unifying software framework for CL research , as a solution for jointly addressing these issues . We describe how settings differ from one another in terms of their assumptions ( e.g. , are task IDs observed or not ) . This perspective gives rise to a hierarchical organization of CL settings , through which methods become directly reusable by inheritance , thus greatly reducing the amount of work involved in developing and evaluating methods in CL . Key Contributions of this work are 1 ) a general unified framework that systematically organizes CL settings ; 2 ) Sequoia , a software framework that can serve as a universal platform for CL research . 2 A UNIFYING FRAMEWORK FOR CL RESEARCH . To construct our unifying framework , we first represent each CL setting as a set of shared assumptions . More general settings make fewer assumptions and vice versa . Settings can then be organized in a hierarchy where adding/removing an assumption yields a child/parent setting ( Fig . 2 ) . We formalize the framework using a hidden-mode Markov decision process ( HM-MDP ) , a special case of a POMDP ( Choi et al. , 2000 ) . HM-MDPs comprise an observation space X , an action space A , a context space Z ( we also refer to contexts as tasks ) , and a feedback function r. Here the full state space is a concatenation of the observation space and task space S “ X ˆ Z . As such , the hidden context variable z P Z defines the dynamics of the environment ppx1|x , a , zq for observations x1 , x P X and action a P A . The feedback function rpx , a , zq provides an agent πpa|xq ( e.g . a supervised model ) the value of performing action a after receiving observation x . The context variable follows a Markov chain ppz1|zq . The dynamic context enables modelling CL task/context-change . A change in the context variable is called a task boundary . In the next section ( § 2.1 ) we show that by restricting the different elements of the HM-MDP we recover different CL settings . Then in § 2.2 we discuss differences between continual supervised ( CSL ) and continual reinforcement learning ( CRL ) . We end in § 2.3 , by presenting additional assumptions that are relevant to CL problems . 2.1 CONTINUAL LEARNING ASSUMPTIONS . Assumptions related to CL can be aranged into a hierarchy , as illustrated in the the central portion of Fig . 2 . These settings cover most , if not all the current CL literature . We start from the most general setting : continuous task-agnostic CL ( Zeno et al. , 2018 ) and add assumptions one by one . Continuous Task-Agnostic CL is our most general setting . The context variable is continuous ‡ P R. This setting allows for different kinds of drifts in the environment , including smooth task boundaries , i.e . slow drift ( Zeno et al. , 2018 ) . This setting is task-agnostic , meaning that the context variable z is unobserved . Because the context is allowed to drift slowly , it can be more challenging for the methods to infer when a task has changed enough to compartmentalize the recently acquired knowledge before adapting to the new task . In RL , this setting is analogous to the DP-MDP ( Xie et al. , 2020 ; Chandak et al. , 2020b ) . In SL , it has also been studied in e.g . Zeno et al . ( 2018 ) ; Aljundi et al . ( 2019a ; b ) . Discrete Task-Agnostic CL assumes clear ( or well-defined ) task boundaries and so a discrete context variable Z P N. In this setting the context can shift in a drastic way , still without the agent being explicitly noticed . Some cases where this setting has been studied are Choi et al . ( 2000 ) ; Riemer et al . ( 2018 ) for RL and Caccia et al . ( 2020 ) ; He et al . ( 2019 ) ; Harrison et al . ( 2019 ) for SL . Incremental Learning ( IL ) relaxes the task-agnostic assumption : the task boundaries are observable . This is akin to augmenting the observation with a binary variable that is set to 1 when z1 ‰ z and 0 otherwise . In doing so , the algorithm does not need to perform task-boundary detection . In SL , some well-known IL settings include class-IL and domain-IL distinguished by their disjoint action space and shared action space , respectively . This is discussed in § 2.3 . At this point in the CL hierarchy , the tree branches in two directions , depending on the order of remaining assumptions ( see Fig . 2 ) . We will first explain the right sub-tree . Task-Incremental Learning ( task-IL ) assumes a fully-observable context variable available to the agent πpa|x , zq . In the literature , observing z is analogous to knowing the task ID or task label . In this simpler CL setting , forgetting can be prevented by freezing a model at the completion of each task and using the task-ID to retrieve it for evaluation . The following settings remove the non-stationarity assumption in the contexts/tasks and are often used to set an upper-bound performance for CL methods . Multi-task Learning removes the non-stationarity in the environment dynamics and the feedback function as it assumes a stationary context variable ppz1|zq “ ppz1q . When the contexts are stationary , there is no catastrophic forgetting ( CF ) ( French , 1999 ) problem to solve . Multi-task learning assumes a fully-observable task variable . Traditional Learning branches off incremental CL and assumes a stationary environment . It is the vanilla supervised setting machine learning defaults to . In our framework , it can be seen as a multi-task learning problem where the task variable isn ’ t observable . However , a more natural view of this setting is to simply assume a single task/context . 2.2 SUPERVISED LEARNING AND REINFORCEMENT LEARNING ASSUMPTIONS . So far we have introduced settings and assumptions that revolve mainly around the type and presence of non-stationarity in the environment and the information observed by the agent . These have allowed us to define the CL problem . To bring all of CL research under one umbrella , we introduce two assumptions , orthogonal to the previous ones , to recover RL and SL settings . With these assumptions , methods for a given CL setting are applicable to both its CSL and CRL versions , as in Kirkpatrick et al . ( 2017 ) ; Fernando et al . ( 2017 ) . Below we use the term observation as a state in RL parlance and the actions as predictions in SL parlance . Also , we assume a single context or task . Level of feedback : In RL , the feedback function rpx , a , zq returns a reward that informs the agent about the value of performing action a after receiving observation x in context z . In SL however , the feedback function is generally both directly known by the agent and differentiable , which allows the agent to simultaneously consider the value of all actions for a particular observation . This feedback is computed based on a label when the action space is discrete ( classification ) or a target when it is continuous ( regression ) . The feedback level is a key differentiating feature between RL and SL . Active vs passive environments : In RL , it is generally assumed that the agent ’ s action has an effect on the next observation or state.1 In other words , the dynamics of the environment ppx1|x , a , zq are action-dependant and we call this an active environment . In SL the agent is generally assumed to not influence the next observation i.e . ppx1|x , a , zq “ ppx1|x , zq . The environment is thus referred to as being passive in these cases . As seen in Figure 2 , the two aforementioned assumptions are combined into a single assumption for SL ( blue , left ) and for RL ( red , right ) . By combining either the RL or SL assumption along with those from the the central CL “ trunk ” , settings from CSL and CRL are recovered . Future versions of Sequoia will decouple these assumptions to enable settings such as bandits and imitation learning .

Summary: The paper attempts to unify all CL research with a single formalism. Next, they present a software implementation of their framework. Finally, experiments are ran which demonstrate that Sequoia can be used to evaluate CL methods.