Closed-loop Control for Online Continual Learning

1 INTRODUCTION . A major challenge in research on artificial neural networks is to develop incremental learning ability to accumulate knowledge over time from a non-stationary stream of data . Most deep learning techniques are designed with the assumption that the training samples are drawn independently and identically from a fixed data distribution , but this assumption is often violated in ever-changing realworld environments . Most successful deep learning results are achieved by training on pre-collected datasets . The resulting model is static and incapable of adapting its behavior to the non-stationary environment over time ( Delange et al. , 2021 ) . When new data becomes available , the training process needs to restart again . In a non-stationary environment , training on the new data can seriously undermine the model ’ s previously acquired knowledge of past data , which may cause what is known as “ catastrophic forgetting ” ( Parisi et al. , 2019 ) . In many real-world applications , a large amount of new data are generated continually , and it has become increasingly important to ensure that the learning agent has lifelong learning capabilities to learn new knowledge and maintain the old knowledge . Our work focuses on online class incremental ( OCI ) continual learning problems to learn a sequence of classification tasks with a single pass over the data . To address the problem of catastrophic forgetting in this setting , continual learning algorithms need to balance between stability and plasticity : the ability to robustly maintain old knowledge while being able to quickly incorporate new knowledge . Previous work on continual learning deals with this problem by modifying the standard training process using memory replay-based , regularization-based , or parameter isolation-based techniques . The work presented in this paper rests on the observation that , besides requiring changes to the training algorithm , the sequential learning process also brings special challenges to the hyperparameter tuning process , which has not been extensively studied so far . In fact , a large number of CL methods have to rely on well-selected hyperparameters to effectively balance the trade-off between stability and plasticity ; to search for the best hyperparameters , most research in this area follows an offline approach to go through the whole data sequence several times measuring performance against a representative , held-out validation set . This violates the fundamental assumption of continual learning , namely that there is no access to previous task data . There is some work on heuristic hyperparameter tuning specifically designed for online CL ( Aljundi et al. , 2019 ; Mai et al. , 2021a ) , but it requires extra held-out task validation data , which may not be available . To address the hyperparameter tuning problem , we propose a closed-loop continual learning framework that is able to balance stability and plasticity without access to an external held-out validation set . Specifically , we build a real-time feedback signal that measures the performance of the CL agent and enables online hyperparameter tuning using reinforcement learning ( RL ) . The proposed closed-loop CL framework is combined with replay-based continual learning to achieve adaptive memory replay for online class-incremental continual learning . Replay-based CL methods prevent forgetting of previously learned knowledge by maintaining a small memory of past data and performing joint training on incoming and memorized data during the optimization process . Despite their simplicity , these methods have significantly outperformed CL approaches without episodic memory in the online class-incremental setting ( Chaudhry et al. , 2019 ) . Compared to standard supervised learning , two main challenges exist in memory replay methods . One is the risk of overfitting the memory , as a relatively small amount of memorized samples are repeatedly rehearsed in the training process . The other challenge is the class imbalance problem in the joint training process : generally , only a relatively small amount of old-class data is available in the memory . Because of these challenges , it is important to be able to modulate the replay dynamics of these methods by hyperparameters such as the replay step size and the number of replay iterations . However , the values assigned to these hyperparameters are often hand-crafted and kept fixed during the whole learning process in previous research on memory replay methods . In this work , we address this problem by proposing a method that is able to dynamically adapt the hyperparameters of memory replay methods in an online manner . Specifically , based on a closedloop CL framework , we apply reinforcement learning to adjust replay hyperparameters to achieve a dynamic adjustment between stability and plasticity . We formulate the problem of adjusting replay hyperparameters as a Markov decision process and use a purpose-built state space to capture the current performance of the CL model and the relevant input data information for replay hyperparameter decision-making . To address the non-stationarity of the environment , we propose a Q function with task-shared and task-specific components . The proposed RL-based methods are used to improve the two replay-based methods ER ( Chaudhry et al. , 2019 ) and SCR ( Mai et al. , 2021b ) , and achieve adaptive memory replay . 2 RELATED WORK . 2.1 CONTINUAL LEARNING . Continual learning is concerned with learning a sequence of tasks . In each task t , data X ( t ) is sampled from distribution D ( t ) ; the corresponding class labels are Y ( t ) . The goal is to minimize the loss on all seen tasks given limited or no access to data X ( t ) , Y ( t ) from previous tasks t < T : ∑T t=1 E ( X ( t ) , Y ( t ) ) [ ` ( f ( X ( t ) ; θ ) , Y ( t ) ) ] with loss function ` , CL parameters θ , T the number of tasks seen so far , and f representing the CL network function . In recent years , continual learning methods have been proposed to deal with different problem settings and evaluation protocols . Van de Ven & Tolias ( 2019 ) summarize three continual learning scenarios based on whether task identity is provided at test time . Notably , there are substantial differences between the difficulties of these three scenarios . The first scenario is task-incremental , which assumes the task identity is provided at test time . Methods proposed for this scenario usually train task-specific components to prevent the interference between new tasks and old task ( Yoon et al. , 2020 ; 2018 ; Mallya & Lazebnik , 2018 ) . A common practice in this setting is to use multi-head evaluation with a separate output layer ( head ) assigned to each task . The task ID is used to determine the head to use for inference . This scenario is the easiest since the model just needs to distinguish classes within a task . The second scenario is domain incremental , where the structure of the task is always the same and the input distribution is changing . The last scenario is class-incremental , with new tasks containing new classes . This setting employs single-head evaluation , meaning the CL agent needs to classify all the classes seen so far , without known task information . This setting can be used as a realistic framework of the common real-world problem of incrementally learning new classes of objects . Another aspect of CL is whether offline or online learning is considered . The offline CL setting assumes full access to the whole data for a task at once . Therefore , training on each single task can be performed over multiple epochs , before moving on to the next task ( Rebuffi et al. , 2017 ) . The online CL setting is more challenging : a stream of samples is seen only once and the sampling distribution is non-IID . In this paper , we focus on this more challenging and realistic setting . To combat catastrophic forgetting , CL methods can be categorized into three families : replay methods , regularization-based methods , and parameter isolation methods ( Delange et al. , 2021 ; Parisi et al. , 2019 ) . The regularization methods , such as EWC++ ( Chaudhry et al. , 2018a ) and LWF ( Li & Hoiem , 2017 ) introduce a regularization term in the loss function to prevent dramatic changes to important parameters and consolidate previous knowledge . Parameter isolation-based methods assign different subsets of model parameters to different tasks to prevent the interference between tasks . Some examples include methods that dynamically expand the network , such as RCL ( Xu & Zhu , 2018 ) , APD ( Yoon et al. , 2020 ) , and DEN ( Yoon et al. , 2018 ) ; alternatively , one can use a fixed network ( Mallya & Lazebnik , 2018 ) . This line of work generally requires knowledge of the task ID at test time and thus is applied in the task-incremental scenario . Replay-based methods retain a memory to store past data for joint training ( Chaudhry et al. , 2019 ; Prabhu et al. , 2020 ) . Some methods in this area focus on how to store and sample informative memory samples , such as MIR ( Aljundi et al. , 2019 ) . Recent work uses supervised contrastive loss in memory-replay to help learn a high-quality feature representation ( Mai et al. , 2021b ) . Some work also combines the regularization and replay approaches , such as MC-SGD ( Mirzadeh et al. , 2020 ) , ICARL ( Rebuffi et al. , 2017 ) . In terms of hyperparameter tuning for continual learning , most work in CL simply employs offline hyperparameter optimization using a held-out validation set . However , there is some recent work that discusses hyperparameter tuning frameworks specifically designed for continual learning settings without a representative validation set . For offline continual learning , Delange et al . ( 2021 ) proposes a method to dynamically balance the stability-plasticity trade-off via a maximal plasticity search followed by stability decay . Instead of requiring a validation set covering all the tasks , this method makes use of a held-out validation set for a new task ; however , some parameters , like the tolerance threshold and decaying speed , need to be manually set in advance . Considering the online continual learning setting , Chaudhry et al . ( 2018b ) and Mai et al . ( 2021a ) employ a hyperparameter tuning protocol targeting this setting . These approaches use external cross-validation data streaming with a small number of tasks . Offline hyperparameter tuning is applied on this external validation data with multiple passes to identify optimal values , which are then used for the actual online continual learning tasks . A limitation of this work is that it relies on external validation data . Depending on the similarity between the validation tasks and the actual continual learning tasks , the chosen hyperparameters may not be appropriate . 2.2 REINFORCEMENT LEARNING . Reinforcement learning ( RL ) concerns the problem of sequential decision-making in a stochastic environment ( Sutton & Barto , 2018 ) . In standard reinforcement learning settings , an RL agent interacts with the environment to maximize the long-term rewards . Typically , this interactive process is formulated as a Markov decision process ( MDP ) described by a tuple < S , A , R , P , γ > , where S is the state space , A is the action space , R : S×A→ R is the reward function , P : S×A×A→ [ 0 , 1 ] is the state transition probability , and γ is the discount factor . The agent behavior is defined as the policy π : S → A that determines how and action is selected during the interaction . The goal of the RL agent is to learn a policy that maximizes the cumulative rewards in an episode . The action-value function Qπ ( s , a ) is used to represent the expected long-term reward of executing action a in state s , where s0 is the initial state distribution . Although reinforcement learning has achieved superior performance in many challenging applications such as board games , continual locomotion control , and complex computer games ( Schrittwieser et al. , 2020 ; Mnih et al. , 2015 ; Vinyals et al. , 2019 ) , its application to adapting the learning process of other machine learning algorithms is relatively limited . Reinforcement learning and bandits algorithms have been used in the area of neural network architecture search and offline hyperparameter optimization in IID environments where a held-out validation set is available and the reward distribution is static ( Li et al. , 2017 ; Parker-Holder et al. , 2020 ; Hoffman et al. , 2014 ) . To our knowledge , the only work applying reinforcement learning in continual learning is in the parameter isolation-based method , which applies it to dynamically expand a network ( Xu & Zhu , 2018 ) . This approach is applied in the offline task-incremental continual learning setting and assumes access to a held-out validation set . There does not appear to be any work that applies reinforcement learning to replay-based continual learning methods or online continual learning settings .

Summary: This paper takes a particular continual learning setup and addresses the challenge of choosing two important hyper-parameters of standard continual learning approaches based on experience replay: the online-to-memory loss ratio, and the number of replay iterations. A DQN-style reinforcement learning approach is proposed to adapt these two parameters online. An additional small test memory is introduced, which is used to construct the reward and states. The efficacy of the approach is demonstrated using CIFAR datasets.