Mitigating Forgetting in Online Continual Learning with  Neuron Calibration

1 Introduction . While humans and animals exhibit remarkable ability to deal with new tasks by effectively adapting their acquired knowledge without forgetting the previously learned skills , in stark contrast , it is challenging for artificial learning systems to effectively deal with continuously inquiring tasks . When switching to a new task from some previously learned ones , there is sometimes a significant drop in performance , where such a phenomenon is also referred to as catastrophic forgetting . To overcome such an issue , the research on continual learning has emerged , which defines the learning protocol for the scenarios when the training data of different tasks procedurally arrives in a sequential order [ 20 , 23 , 28 , 35 ] . The catastrophic forgetting problem is rooted in the model training process , as the gradients to train the model parameters encode great amount of information about the dynamically shifting data distribution seen by the model , which could keep changing throughout the entire continual learning period . In this paper , we consider a challenging task which poses a more restrictive requirement on the conventional continual learning setting , i.e. , online continual learning [ 1 ] . Inspired by many real life scenarios , online continual learning requires the data for each training task to appear as a ( one-pass ) stream of samples . To tackle such challenges , we focus on the replay-based approach [ 19 , 34 ] , which 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . grants the model with limited access to the data from past tasks for the rehearsal of past experience . To mitigate the catastrophic forgetting issue , such methods leverage the replay memory to consolidate the past knowledge in various forms , such as raw data [ 1 ] and past gradients [ 27 , 32 ] , to facilitate effective transfer of knowledge . Though the research community has also started to pay attention on a number of non replay-based directions to tackle the continual learning problems , such as leveraging energy-based models [ 14 ] and neuron modulation approaches [ 15 ] , the replay-based methods still play a pivot role in continual learning with state-of-the-art performance in many challenging scenarios . However , despite of the promise in their performance , the replay-based attempts easily lead to a data-imbalance issue , which is also formally referred to as the stability-plasticity dilemma [ 21 ] . In particular , high plasticity refers to the case when old experience is drastically forgotten , and high stability refers to the case that too much attention has been paid on stabilizing the previously learned knowledge so that the learning hinders the acquisition of knowledge on the new tasks . Lose of balance between plasticity and stability would deteriorate the performance of continual learning . The existing replay-based approaches have taken into account of different perspectives of the model training process to remedy the stability-plasticity dilemma , such as regularizing the parameter change during training [ 4 , 12 , 23 ] , selective memory storage or replay [ 1 , 11 ] , Bayesian and variational Bayesian training [ 4 , 12 , 22 ] , and task-specific parameterization of the model [ 25 , 36 ] . In this paper , we tackle the problem from a novel angle that is distinct to all the aforementioned attempts , i.e. , seeking a better balance between stability and plasticity with neuron calibration . Specifically , we refer to neuron calibration as a process of mathematically adjusting the transformation functions in various layers of deep neural networks . Considering that besides the task setting , catastrophic forgetting in contemporary deep learning-based continual learning models is also closely related to the vulnerability of deep neural networks , our proposed neuron calibration approach aims to regularize the parameter update against catastrophic forgetting via posing a trainable soft mask on the parameters , which then influences both the model inference process and the model training process through the forward inference path and the backward optimization path . Our work is inspired by the earlier works that seek optimal model generalization through calibrating the neural networks parameters or labels [ 9 , 10 ] , as well as a recent continual learning work that learns task-specific calibrations for retaining task-specific parameters without memory rehearsal [ 36 ] . Compared to those works , the neuron calibration approach we propose shares an enjoyable property of being task-agnostic in terms of parameter sharing among the tasks . That is , instead of reserving task-specific parameters for preserving the task knowledge against forgetting , we train a shared calibrated model where we interleave data from different task distributions to effectively optimize the model . The contributions of our work are three-fold : ( i ) we introduce a general and light-weight neuron calibration approach to tackle task-incremental online continual learning problems where the models are formulated as feed-forward deep neural networks-based function approximations ; ( ii ) we formulate a novel task-incremental learning paradigm to train the calibrated model with an interleaved optimization scheme to achieve a better balance between stability and plasticity ; ( iii ) we show through extensive empirical experiments that our proposed method could outperform the state-of-the-art continual learning algorithms as well as the related layer calibration approach with significant margins on all the evaluation datasets . 2 Related Work . Based on how the existing methods deal with catastrophic forgetting , following [ 7 , 8 , 13 , 25 ] , we classify the continual learning methods into three major categories : ( i ) memory-based , ( ii ) regularization-based and ( iii ) dynamic architectural strategies . Memory rehearsal based methods partially stores the data from past tasks into episodic memories to be later used in a form of knowledge rehearsal [ 29 ] . The old data are often used in a standard way of performing experience replay [ 1 , 3 , 30 , 37 ] , but there are also a number of other purposes of use , such as leveraging the data to perform representation learning [ 26 ] and forming constraints for model optimization [ 6 , 19 ] . To facilitate more effective memory rehearsal , there are also a number of methods that attempt to improve the memory selection process for continual learning , such as iCaRL [ 26 ] which stores the samples close to the center of each class , MIR [ 1 ] that selects most interfered samples for memory rehearsal , and HAL [ 5 ] that selects the anchor points of past tasks and interleaves them with new tasks for future training . Overall the memory-based approaches address catastrophic forgetting issue from training with past data but their performance could easily be degraded under a tight memory restriction . Our proposed neuron calibration approach could effectively scale up the performance standard of memory rehearsal-based approaches under varying memory restrictions . Regularization based methods extend the loss function in continual learning with regularization terms to promote selective consolidation of past knowledge stored in the model parameters . Often , an importance measure over the model parameters need to be established . In [ 12 ] , an elastic weight consolidation ( EWC ) approach is proposed with an importance measure defined as promoting the parameters that have a higher value in terms of the Fisher information matrix . In [ 2 ] , a selfless sequential learning algorithm is proposed which exploits the sparse and de-correlated representations to avoid overlapping of representations . In [ 33 ] , hard attention is learned on the task embeddings to identify important neurons during gradient propagation . We inherit the idea of EWC to regularize the degree of parameter change during online continual learning , but such regularization in our method is to work with the specific calibrated neural network models , whereas applying the EWC regularizer by itself often leads to much inferior performance on conventional online continual learning tasks . Dynamic architectural methods attempts to address catastrophic forgetting through approximating the training of a separate network per task . In [ 31 , 33 , 36 , 38 ] a sub-network is trained for each task and in [ 16 , 39 ] the architecture of model grows over time during training . Though isolating shared model parameters could for sure mitigate forgetting of past knowledge , the drawbacks are in terms of extensive resource usage and scalability . On the other hand , such methods also fail to utilize the multi-task nature of continual learning to exploit the relationship between related tasks for achieving better generalization . Our work is mostly related to [ 25 ] , which engages task-specific embeddings to calibrate the feature output of neural networks and train the task-specific parameters with hold-out validation data to promote generalization . Our method differs from [ 25 ] in terms of the following three points : ( i ) our method is built upon a neuron calibration approach , where such contribution is orthogonal to that from all the previous works ; ( ii ) our proposed method does not engage any task-specific part ; ( iii ) we do not use any hold-out data from the episodic memories during training , considering that holding out data over the episodic memories which typically come with restrictive storage budgets might not be a desirable practice . Our work is also related to [ 36 ] which calibrates neural networks to obtain task-specific parameters . However , the method only trains the task-specific calibrator parameters after the first task and freezes the base parameters thereafter . Such model design makes the model to face a potential risk that when the data distributions of the later tasks are significantly biased from the first task , freezing the base model might lead to inferior performance . Our proposed task-agnostic calibrator shares all the parameters among tasks and it trains all its parameters with interleaved optimization throughout continual learning so that it can better exploit task relationships . 3 NCCL : ( N ) euron ( C ) alibration for online ( C ) ontinual ( L ) earning . Notation . In this section , we first introduce the notations for online continual learning . Formally , we denote the sequence of training tasks in online continual learning as { T1 , · · · , TT } , where Tt is the t-th task . The tasks arrives in order and the training data for each task is observed in an online fashion . Inheriting a memory-based setting , each task is granted with a small amount of storage to save the past data , which is termed as a episodic memory . For the task Tt , we denote its training dataset as Dt and its episodic memory as Mt . The assembly of all the episodic memories for the past tasks prior to t is denoted as M < t . Note that Mt is a subset sampled from Dt and in our work we adopt uniform sampling throughout the learning . The objective of the task is to learn a neural networks-based classifier . Our work tackles the image classification-based continual learning problems and thus the classifier is typically modeled as a feed-forward neural network with L layers ( i.e. , { li } Li=1 ) , with its corresponding parameters denoted as { ✓i } Li=1 . 3.1 Neuron Calibration . We introduce a general neuron calibration mechanism to tackle the online continue learning problems on image classification , where the models are formulated as feed-forward neural networks . By applying neuron calibration , we aim to adapt the transformation functions in the deep neural network layers , which aims to effectively mitigate catastrophic change on the model parameters while accomplishing a stable consolidation of knowledge from different tasks . Specifically , in this work , we formalize the calibration of two types of commonly adopted transformation layers in feed-forward deep neural networks : fully connected layers and convolutional layers1 . Figure 1 provides an illustrative example of our neuron calibration process . The neural networks model before calibration is denoted as the base network . Formally , we introduce two types of general calibration modules to be applied on the base network layers : ( i ) weight calibration module ( WCM ) and ( ii ) feature calibration module ( FCM ) . The weight calibration module learns to scale the weights of the parameters from the transformation function whereas the feature calibration module learns to scale the output feature maps predicted by the transformation function [ 24 ] . When calibrating the i-th layer of a neural network , we use ✓i and e✓i to denote its transformation function parameters before and after applying weight calibration ( WCM ) , and use hi and ehi to denote the output feature maps before and after applying feature calibration ( FCM ) , respectively . We first introduce the formulation for WCM . Let ⌦ i ( · ) denote the weight calibration function employed by the i-th layer of the network , which is parameterized by i . Overall , the weight calibration unit is modularized by an element-wise multiplication operation , which is applied between the base network parameters and the calibrator parameters . Specifically , we refer to the i-th layer parameters from the base network transformation function as ✓i . Then the weight calibration function is defined in the following manner , ⌦ i ( ✓i ) = ( tile ( i ) ✓i , i 2 RO⇥I ( Convolution Layer ) , tile ( i ) ✓i , i 2 RO ( Fully Connected Layer ) , ( 1 ) where denotes element-wise multiplication , O and I denote the number of output channels and input channels , respectively . To reduce the size of the calibrator parameters for efficient computation , we specify the calibration size to be much smaller than the original size of ✓i . To this end , the calibration for the convolutional layers and fully connected layers is specified to work at per-channellevel and per-feature-level , respectively . To scale up the shape of the calibrator parameters i to match that for ✓i , a tile ( · ) function is applied on the calibrator parameters i . With the aforementioned weight calibration approach , the calibrator module plays a crucial role during the model training process : at the forward inference path , it scales the value of the base network parameters to make prediction ; at the backward optimization path , it serves as a prioritized weight to regularize the update 1Calibrating the fully connected and convolutional layers suffices to deal with primary continual learning tasks on image classification , and our method could also be easily extended to calibrate other types of layers . on important parameters ( e.g. , 5✓1Lb is derived in the form of 5e✓1Lb tile ( 1 ) , scaled by the calibrator parameters ) . By applying weight calibration on neural network layers , the transformation function at the i-th layer F✓̃i is parameterized by ✓̃i and produces the output as follows , hi = F✓̃i ( h̃i 1 ) s.t . ✓̃i = ⌦ i ( ✓i ) . ( 2 ) The output hi of the transformation function is then processed by a feature transformation module ( FCM ) to generate the calibrated feature map for that layer . We use ⌦ i ( · ) to denote the feature transformation function at the i-th layer , parameterized by i . With FCM , the calibration parameters also interact with the feature map hi with a multiplicative operation . Specifically , the calibrated feature map is computed as : ⌦ i ( hi ) = ( tile ( i ) hi , i 2 RO and hi 2 RH⇥W⇥O ( Convolution Layer ) , i hi , i 2 RO and hi 2 RO ( Fully Connected Layer ) , ( 3 ) where O is the output ( channel ) size for fully connected layers and the convolutional layers . In the end , the outputs from ( 2 ) and ( 3 ) get added up in an element-wise manner by a residual connection . This is followed by normalization and activation operations to produce a final output for that layer . In summary , the overall calibration process for the i-th layer could be formulated as follows , h̃i = ⇣ BN ⇣ ⌦ i ⇣ F✓̃i ⇣ h̃i 1 ⌘⌘ F✓̃i ⇣ h̃i 1 ⌘⌘⌘ , s.t . ✓̃i = ⌦ i ( ✓i ) ( 4 ) where BN ( · ) denotes the batch normalization , denotes an element-wise addition operator , and ( · ) is an activation function . h̃i is sent as input to the i+ 1-th layer in the feed-forward neural network . All the calibrator parameters are initialized with a value of 1 at the start of training . We demonstrate an example case of applying neuron calibration on a CNN-based model in Figure 1 .

Summary: This work proposes an online continual learning approach that learns an attention map on the weights (termed as weight calibration module) and another on the features (termed as feature calibration module) of a neural network. The training is done by adding a stability regularization term (similar to CTN) to the original EWC loss and interleaving the optimization of network parameters and attention modules as the new tasks are observed continually. A task-incremental learning scenario is considered, and the accuracy is compared against five other approaches with four benchmark datasets using three different metrics.  With a marginal increase in the number of parameters with the attention maps, we can see considerable improvement in accuracy.