Closed-loop Control for Online Continual Learning

1 INTRODUCTION . A major challenge in research on artificial neural networks is to develop incremental learning ability to accumulate knowledge over time from a non-stationary stream of data . Most deep learning techniques are designed with the assumption that the training samples are drawn independently and identically from a fixed data distribution , but this assumption is often violated in ever-changing realworld environments . Most successful deep learning results are achieved by training on pre-collected datasets . The resulting model is static and incapable of adapting its behavior to the non-stationary environment over time ( Delange et al. , 2021 ) . When new data becomes available , the training process needs to restart again . In a non-stationary environment , training on the new data can seriously undermine the model ’ s previously acquired knowledge of past data , which may cause what is known as “ catastrophic forgetting ” ( Parisi et al. , 2019 ) . In many real-world applications , a large amount of new data are generated continually , and it has become increasingly important to ensure that the learning agent has lifelong learning capabilities to learn new knowledge and maintain the old knowledge . Our work focuses on online class incremental ( OCI ) continual learning problems to learn a sequence of classification tasks with a single pass over the data . To address the problem of catastrophic forgetting in this setting , continual learning algorithms need to balance between stability and plasticity : the ability to robustly maintain old knowledge while being able to quickly incorporate new knowledge . Previous work on continual learning deals with this problem by modifying the standard training process using memory replay-based , regularization-based , or parameter isolation-based techniques . The work presented in this paper rests on the observation that , besides requiring changes to the training algorithm , the sequential learning process also brings special challenges to the hyperparameter tuning process , which has not been extensively studied so far . In fact , a large number of CL methods have to rely on well-selected hyperparameters to effectively balance the trade-off between stability and plasticity ; to search for the best hyperparameters , most research in this area follows an offline approach to go through the whole data sequence several times measuring performance against a representative , held-out validation set . This violates the fundamental assumption of continual learning , namely that there is no access to previous task data . There is some work on heuristic hyperparameter tuning specifically designed for online CL ( Aljundi et al. , 2019 ; Mai et al. , 2021a ) , but it requires extra held-out task validation data , which may not be available . To address the hyperparameter tuning problem , we propose a closed-loop continual learning framework that is able to balance stability and plasticity without access to an external held-out validation set . Specifically , we build a real-time feedback signal that measures the performance of the CL agent and enables online hyperparameter tuning using reinforcement learning ( RL ) . The proposed closed-loop CL framework is combined with replay-based continual learning to achieve adaptive memory replay for online class-incremental continual learning . Replay-based CL methods prevent forgetting of previously learned knowledge by maintaining a small memory of past data and performing joint training on incoming and memorized data during the optimization process . Despite their simplicity , these methods have significantly outperformed CL approaches without episodic memory in the online class-incremental setting ( Chaudhry et al. , 2019 ) . Compared to standard supervised learning , two main challenges exist in memory replay methods . One is the risk of overfitting the memory , as a relatively small amount of memorized samples are repeatedly rehearsed in the training process . The other challenge is the class imbalance problem in the joint training process : generally , only a relatively small amount of old-class data is available in the memory . Because of these challenges , it is important to be able to modulate the replay dynamics of these methods by hyperparameters such as the replay step size and the number of replay iterations . However , the values assigned to these hyperparameters are often hand-crafted and kept fixed during the whole learning process in previous research on memory replay methods . In this work , we address this problem by proposing a method that is able to dynamically adapt the hyperparameters of memory replay methods in an online manner . Specifically , based on a closedloop CL framework , we apply reinforcement learning to adjust replay hyperparameters to achieve a dynamic adjustment between stability and plasticity . We formulate the problem of adjusting replay hyperparameters as a Markov decision process and use a purpose-built state space to capture the current performance of the CL model and the relevant input data information for replay hyperparameter decision-making . To address the non-stationarity of the environment , we propose a Q function with task-shared and task-specific components . The proposed RL-based methods are used to improve the two replay-based methods ER ( Chaudhry et al. , 2019 ) and SCR ( Mai et al. , 2021b ) , and achieve adaptive memory replay . 2 RELATED WORK . 2.1 CONTINUAL LEARNING . Continual learning is concerned with learning a sequence of tasks . In each task t , data X ( t ) is sampled from distribution D ( t ) ; the corresponding class labels are Y ( t ) . The goal is to minimize the loss on all seen tasks given limited or no access to data X ( t ) , Y ( t ) from previous tasks t < T : ∑T t=1 E ( X ( t ) , Y ( t ) ) [ ` ( f ( X ( t ) ; θ ) , Y ( t ) ) ] with loss function ` , CL parameters θ , T the number of tasks seen so far , and f representing the CL network function . In recent years , continual learning methods have been proposed to deal with different problem settings and evaluation protocols . Van de Ven & Tolias ( 2019 ) summarize three continual learning scenarios based on whether task identity is provided at test time . Notably , there are substantial differences between the difficulties of these three scenarios . The first scenario is task-incremental , which assumes the task identity is provided at test time . Methods proposed for this scenario usually train task-specific components to prevent the interference between new tasks and old task ( Yoon et al. , 2020 ; 2018 ; Mallya & Lazebnik , 2018 ) . A common practice in this setting is to use multi-head evaluation with a separate output layer ( head ) assigned to each task . The task ID is used to determine the head to use for inference . This scenario is the easiest since the model just needs to distinguish classes within a task . The second scenario is domain incremental , where the structure of the task is always the same and the input distribution is changing . The last scenario is class-incremental , with new tasks containing new classes . This setting employs single-head evaluation , meaning the CL agent needs to classify all the classes seen so far , without known task information . This setting can be used as a realistic framework of the common real-world problem of incrementally learning new classes of objects . Another aspect of CL is whether offline or online learning is considered . The offline CL setting assumes full access to the whole data for a task at once . Therefore , training on each single task can be performed over multiple epochs , before moving on to the next task ( Rebuffi et al. , 2017 ) . The online CL setting is more challenging : a stream of samples is seen only once and the sampling distribution is non-IID . In this paper , we focus on this more challenging and realistic setting . To combat catastrophic forgetting , CL methods can be categorized into three families : replay methods , regularization-based methods , and parameter isolation methods ( Delange et al. , 2021 ; Parisi et al. , 2019 ) . The regularization methods , such as EWC++ ( Chaudhry et al. , 2018a ) and LWF ( Li & Hoiem , 2017 ) introduce a regularization term in the loss function to prevent dramatic changes to important parameters and consolidate previous knowledge . Parameter isolation-based methods assign different subsets of model parameters to different tasks to prevent the interference between tasks . Some examples include methods that dynamically expand the network , such as RCL ( Xu & Zhu , 2018 ) , APD ( Yoon et al. , 2020 ) , and DEN ( Yoon et al. , 2018 ) ; alternatively , one can use a fixed network ( Mallya & Lazebnik , 2018 ) . This line of work generally requires knowledge of the task ID at test time and thus is applied in the task-incremental scenario . Replay-based methods retain a memory to store past data for joint training ( Chaudhry et al. , 2019 ; Prabhu et al. , 2020 ) . Some methods in this area focus on how to store and sample informative memory samples , such as MIR ( Aljundi et al. , 2019 ) . Recent work uses supervised contrastive loss in memory-replay to help learn a high-quality feature representation ( Mai et al. , 2021b ) . Some work also combines the regularization and replay approaches , such as MC-SGD ( Mirzadeh et al. , 2020 ) , ICARL ( Rebuffi et al. , 2017 ) . In terms of hyperparameter tuning for continual learning , most work in CL simply employs offline hyperparameter optimization using a held-out validation set . However , there is some recent work that discusses hyperparameter tuning frameworks specifically designed for continual learning settings without a representative validation set . For offline continual learning , Delange et al . ( 2021 ) proposes a method to dynamically balance the stability-plasticity trade-off via a maximal plasticity search followed by stability decay . Instead of requiring a validation set covering all the tasks , this method makes use of a held-out validation set for a new task ; however , some parameters , like the tolerance threshold and decaying speed , need to be manually set in advance . Considering the online continual learning setting , Chaudhry et al . ( 2018b ) and Mai et al . ( 2021a ) employ a hyperparameter tuning protocol targeting this setting . These approaches use external cross-validation data streaming with a small number of tasks . Offline hyperparameter tuning is applied on this external validation data with multiple passes to identify optimal values , which are then used for the actual online continual learning tasks . A limitation of this work is that it relies on external validation data . Depending on the similarity between the validation tasks and the actual continual learning tasks , the chosen hyperparameters may not be appropriate . 2.2 REINFORCEMENT LEARNING . Reinforcement learning ( RL ) concerns the problem of sequential decision-making in a stochastic environment ( Sutton & Barto , 2018 ) . In standard reinforcement learning settings , an RL agent interacts with the environment to maximize the long-term rewards . Typically , this interactive process is formulated as a Markov decision process ( MDP ) described by a tuple < S , A , R , P , γ > , where S is the state space , A is the action space , R : S×A→ R is the reward function , P : S×A×A→ [ 0 , 1 ] is the state transition probability , and γ is the discount factor . The agent behavior is defined as the policy π : S → A that determines how and action is selected during the interaction . The goal of the RL agent is to learn a policy that maximizes the cumulative rewards in an episode . The action-value function Qπ ( s , a ) is used to represent the expected long-term reward of executing action a in state s , where s0 is the initial state distribution . Although reinforcement learning has achieved superior performance in many challenging applications such as board games , continual locomotion control , and complex computer games ( Schrittwieser et al. , 2020 ; Mnih et al. , 2015 ; Vinyals et al. , 2019 ) , its application to adapting the learning process of other machine learning algorithms is relatively limited . Reinforcement learning and bandits algorithms have been used in the area of neural network architecture search and offline hyperparameter optimization in IID environments where a held-out validation set is available and the reward distribution is static ( Li et al. , 2017 ; Parker-Holder et al. , 2020 ; Hoffman et al. , 2014 ) . To our knowledge , the only work applying reinforcement learning in continual learning is in the parameter isolation-based method , which applies it to dynamically expand a network ( Xu & Zhu , 2018 ) . This approach is applied in the offline task-incremental continual learning setting and assumes access to a held-out validation set . There does not appear to be any work that applies reinforcement learning to replay-based continual learning methods or online continual learning settings .

Summary: This submission proposes an RL-based method for tuning hyper-parameters of continual learning (CL) methods. The rationale is that tuning hyper-parameters on the entire set of tasks off-line invalidates the CL assumption of online learning, and therefore online methods are needed. The authors discuss the elements that make up the state space for making hyper-parameter choices, such as various performance metrics, and use use (deep) Q-learning to train an RL agent to make decisions about the hyper-parameter to use at each training step. The approach is applied on top of two existing replay-based methods, varying two hyper-parameters (one each in separate evaluations), and demonstrated to achieve improved performance w.r.t. the _un-tuned_ variants.