Mitigating Forgetting in Online Continual Learning with  Neuron Calibration
Authors: Haiyan Yin
Peng Yang
Ping Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Continual Learning
Online Learning
Deep Learning

Abstract: Inspired by human intelligence, the research on online continual learning aims to push the limits of the machine learning models to constantly learn from sequentially encountered tasks, with the data from each task being observed in an online fashion. Though recent studies have achieved remarkable progress in improving the online continual learning performance empowered by the deep neural networks-based models, many of today's approaches still suffer a lot from catastrophic forgetting, a persistent challenge for continual learning. In this paper, we present a novel method which attempts to mitigate catastrophic forgetting in online continual learning from a new perspective, i.e., neuron calibration. In particular, we model the neurons in the deep neural networks-based models as calibrated units under a general formulation. Then we formalize a learning framework to effectively train the calibrated model, where neuron calibration could give ubiquitous benefit to balance the stability and plasticity of online continual learning algorithms through influencing both their forward inference path and backward optimization path.  Our proposed formulation for neuron calibration is lightweight and applicable to general feed-forward neural networks-based models. We perform extensive experiments to evaluate our method on four benchmark continual learning datasets. The results show that neuron calibration plays a vital role in improving online continual learning performance and our method could substantially improve the state-of-the-art performance on all~the~evaluated~datasets.

Mitigating Forgetting in Online Continual Learning with Neuron Calibration
Haiyan Yin, Peng Yang, Ping Li Cogmitive Computing Lab Baidu Research 10900 NE 8th St. Bellevue, WA 98004, USA Chaiyanyin, pengyang01, 1ipingl1)@baidu com
Abstract
Inspired by human intelligence, the research online continual learning aims push the limits the machine learning models to constantly learn from sequentially encountered tasks, with the data from each task being observed in online fashion. Though recent studies have achieved remarkable progress in improving the online continual learning performance empowered by the deep neural networks based models, many of today's s approaches still suffer a lot from catastrophic forgetting, persistent challenge for continual learning. In this paper, present novel method which attempts to mitigate catastrophic forgetting in online continual learning from new perspective, i.e., neuron calibration. In particular, we model the neurons in the deep neural networks-based models as calibrated units under a general formulation. Then we formalize learning framework effectively train the calibrated model, where neuron calibration could give ubiquitous benefit to balance the stability and plasticity of online continual learning algorithms through influencing both their forward inference path and backward optimization path. Our proposed formulation for neuron calibration is lightweight and applicable to general feed-forward neural networks- based models. We perform extensive experiments evaluate our method on four benchmark continual learning datasets. The results show that neuron calibration plays vital role in improving online continual learning performance and our method could substantially improve the state-of-the-ate performance all the evaluated datasets.
Introduction
While humans and animals exhibit remarkable ability to deal with new tasks by effectively adapting their acquired knowledge without forgetting the previously learned skills, in stark contrast, it is challenging for artificial learning systems to effectively deal with continuously inquiring tasks. When switching new task from some previously learned ones, there is sometimes significant drop in performance, where such a phenomenon is also referred to as catastrophic forgetting. jarastrepphicc_gggiinn.... To overcome such an issue, the research on continual learning has emerged, which defines the learning protocol for the scenarios when the training data of different tasks procedurally arrives in a sequential order [20, 23, 28, 35]. The catastrophic forgetting problem is rooted in the model training process, as the gradients to train the model parameters encode great amount of information about the dynamically shifting data distribution seen by the model, which could keep changing throughout the entire continual learning period.
In this paper, we consider a challenging task which poses more restrictive requirement the conventional continual learning setting, i.e., online continual learning [1]. Inspired by many real life scenarios, online continual learning requires the data for each training task appear as (one-pass) stream of samples. To tackle such challenges, we focus on the replay- based approach [19, 34], which
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

grants the model with limited access to the data from past tasks for the rehearsal of past experience. To mitigate the catastrophic forgetting issue, such methods leverage the replay m to consolidate the past knowledge in various forms, such as raw data [1] and past gradients [27, 32], to facilitate effective transfer of knowledge. Though the research community has also started to pay attention on a number of non replay-based directions to tackle the continual learning problems, such as leveraging energy-based models [14] and neuron modulation approaches [15], the replay-based methods still play pivot role in continual learning with state-of-the-art performance in many challenging scenarios. However, despite of the promise in their performance, the replay-based attempts easily lead to a inttaimbalannce issue, which is also formally referred to as the particular, high plasticity refers to the case when old experience is drastically forgotten, and high stability refers to the case that too much attention has been paid on stabilizing the previously learned knowledge so that the learning hinders the acquisition of knowledge on the new tasks. Lose of balance between plasticity and stability would deteriorate the performance of continual learning.
dilemma [21]. In
The existing replay-based approaches have taken into account of different perspectives of the model training process to remedy the stability-plasticity dilemma, such as regularizing the parameter change during training [4. 12. 23], selective memory storage or replay [I. 11J, Bayesian and variational Bayesian training [4, 12, 22]. and task-specific parameterization of the model [25, 36]. In this paper, we tackle the problem from a novel angle that is distinct to all the aforementioned attempts, i.e. seeking a better balance between stability and plasticity with neuron calibration. Specifically, refer to neuron calibration as a process of mathematically adjusting the transformation functions in various layers of deep neural networks. Considering that besides the task setting, catastrophic forgetting in contemporary deep learning-based continual learning models is also closely related to the vulnerability of deep neural networks, our proposed neuron calibration approach aims to regularize the parameter update against forgetin via posing a trainable soft mask on the parameters, which then influences both the model inference process and the model training process through the forward inference path and the backward optimization path. Our work is inspired by the earlier works that seek optimal model generalization through calibrating the neural networks parameters or labels [9, 10], as well as a recent continual learning work that learns task-specific calibrations for retaining task- specific parameters without memory rehearsal [36]. Compared to those works, the neuron calibration approach we propose shares an enjoyable property of being task-agnostic in terms of parameter sharing among the tasks. That is, instead of reserving task-specific parameters for preserving the task knowledge against forgetting, we train a shared calibrated model where we interleave data from different task distributions to effectively optimize the model.
The contributions of our work are three-fold: (i) we introduce a general and light-weight neuron calibration approach tackle task-incremental online continual learning problems where the models are formulated as feed-forward deep neural networks-based function approximations: (ii) we for- mulate a novel task ieskeincremettl learning paradigm to train the calibrated model with an interleaved optimization scheme to achieve better balance between stability and plasticity; (iii) we show through extensive empirical experiments that our proposed method could outperform the tfte-off-the-ar con- tinual learning algorithms as well as the related layer calibration approach with significant margins on all the evaluation datasets.
2 Related Work
Based on how the existing methods deal with catastrophic forgetting, following [7, 8, 13, 25]. we classify the continual learning methods into three major categories: (i) memory-based, (ii) regularization-basee and (iii) dynamic architectural strategies.
Memory rehearsal based methods partially stores the data from past tasks into episodic memories to be later used in a form of knowledge rehearsal [29]. The old data are often used in standard way of performing experience replay [1, 3, 30, 37J, but there are also number of other purposes of use, such as leveraging the data to perform representation learning [26] and forming constraints for model optimization [6, 19]. To facilitate more effective memory rehearsal, there are also a number of methods that attempt to improve the memory selection process for continual learning, such as iCaRL [26] which stores the samples close to the center of each class, MIR [1] that selects most interfered samples for memory rehearsal, and HAL [5] that selects the anchor points of past tasks and interleaves them with new tasks for future training. Overall the memory-based approaches address catastrophic forgetting issue from training with past data but their performance could casily
2

---

be degraded under a tight memory restriction. Our proposed neuron calibration approach could effectively scale up the performance standard of memory rehearsal-basd -bssed upproaches under varying memory restrictions.
Regularization based methods extend the loss function in continual learning with regularization terms to promote selective consolidation of past knowledge stored in the model parameters. Often, an importance measure over the model parameters need to be established. In [12], an elastic weight consolidation (EWC) approach is proposed with an importance measure defined as promoting the parameters that have higher value in terms of the Fisher information matrix. In [2], selfless sequential learning algorithm is proposed which exploits the sparse and de-correlated representations to avoid overlapping of representations. In [33], hard attention learned on the task embeddings to identify important neurons during gradient propagation. We inherit the idea of EWC to regularize the degree of parameter change during online continual learning but such regularization in our method is to work with the specific calibrated neural network models, whereas applying the EWC regularizer by itself often leads to much inferior performance on conventional online continual learning tasks.
Dynamic architectural methods attempts to address catastrophic forgetting through approximating the training of a separate network per task. In [31, 33, 36, 38] a sub-network is trained for each task and in [16, 39] the architecture of model grows over time during training. Though isolating shared model parameters could for sure mitigate forgetting of past knowledge, the drawbacks are in terms of extensive resource usage and scalability. On the other hand, such methods also fail to utilize the multi-task nature of continual learning to exploit the relationship between related tasks for achieving better generalization. Our work is mostly related to [25], which engages task- embeddings to calibrate the feature output of neural networks and train the lask-specfiic parameters with hold-out validation data to promote generalization. Our method differs from [25] in terms of the following three points: (i) our method is built upon a neuron calibration approach, where such contribution is orthogonal to that from all the previous works; (ii) our proposed method does not engage any task-specific part; (iii) we do not use any hold-out data from the episodic memories during training, considering that holding out data over the episodic memories which typically come with restrictive storage budgets might not be a desirable practice. Our work is also related to [36] which calibrates neural networks to obtain task-specific parameters. However, the method only trains the task-specifir calibrator parameters after the first task and freezes the base parameters thereafter. Such model design makes the model to face potential risk that when the data distributions of the later tasks are significantly biased from the first task, freezing the base model might lead to inferior performance. Our proposed task-agnostic calibrator shares all the parameters among tasks and it trains all its parameters with interleaved optimization throughout continual learning so that it can better exploit task relationships.
3 NCCL: (N)euron (C)alibration for online (C)ontinual (L)earning
Notation. In this section, we first introduce the notations for online continual learning. Formally, we denote the sequence of training tasks in online continual learning as (Ti,.... ] where Te is the t-th task. The tasks arrives in order and the training data for each task is observed in an online fashion. Inheriting a semoory-based setting, each task is granted with a small amount of storage to save the past data, which is termed and its episodic memory as M₁. The assembly of all the episodic memories for the past tasks prior to t is denoted as Mar. Note that M, is subset sampled from D, and in our work adopt uniform sampling throughout the learning. The objective of the task is to learn a neural networks- classifier. Our work tackles the image dlssiiicationn-asee continual learning problems and thus the classifier is typically modeled as a feed-forwart neural network with I, layers (i.e., (4))2) with its corresponding parameters denoted as {0.)
episodic memory. For the task Te, denote its training dataset as D,
3.1 Neuron Calibration
We introduce general neuron calibration mechanism to tackle the online continue learning problems on image classification, where the models are formulated as feed-forward neural networks. By applying neuron calibration, we aim to adapt the transformation functions in the deep neural network layers, which aims to effectively mitigate catastrophic change on the model parameters while accomplishing a stable consolidation of knowledge from different tasks. Specifically, in this work.
3

---

Ls
Php
D2
Input
WCM
FCM
Activation
WCM
FCM
Activation
WCM
FCML
Acttivation
Classifier
Ma
Phc
Weight calibration module (WCM)
Feature calibration module (FCM)
Fg. 4p
1/
0
h
Ix]
IXI
2x3
+
+B
SK1 3X3
3x3
3x3
133
Le
3H3
W
VAgCe
Back Propagation
Forward Pass
Inner
Outer Loop
Figure 1: Overview of proposed Neural Calibration for online Continual Learning (NCCL) framework. NCCL consists of two types calibration modules: weight calibration module (WCM) and feature calibration module (FCM), which sequentially applied to the layers in the base model (as shown in the figure) to calibrate the model weights and feature maps, respectively.
we formalize the calibration of two types of commonly adopted transformation layers in ferward deep neural networks: fully connected layers and convolutional layers'. Figure 1 provides illustrative example of our neuron calibration process.
The neural networks model before calibration is denoted as the base network. Formally, we introduce two types of general calibration modules to be applied on the base network layers: weight calibra- tion module (WCM) and (ii) feature calibration module (FCM). The weight calibration module learns to scale the weights of the parameters from the transformation function whereas the feature calibration module learns scale the output feature maps predicted by the transformation function [24]. When calibrating the i-th layer of neural network, use 0 and 02 denote its transformation function parameters before and after applying weight calibration (WCM), and use h, and h, denote the output feature maps before and after applying feature calibration (FCM), respectively.
We first introduce the formulation for WCM. Let 2,yy (-) denote the weight calibration function employed by the i-th layer of the network, which is parameterized by th. Overall, the weight calibration unit is modularized by an element-wise multiplication operation, which is applied between the base network parameters and the calibrator parameters. Specifically, we refer to the i-th layer parameters from the base network transformation function as 0,. Then the weight calibration function is defined in the following manner,
(0.) (tile(bs)) O2, ii E Rxx
(Convolution Layer),
(1)
(1:) @ 0,, E R
(Fully Connected Layer),
where @ denotes element-wise multiplication, O and denote the number of output channels and input channels, respectively. To reduce the size of the calibrator parameters for efficient computation, we specify the calibration size be much smaller than the original size of 0,, To this end, the calibration for the comvolutional layers and fully connected layers is specified to work at per-channel- level and per-felture-level. respectively. To scale up the shape of the calibrator parameters ti: to match that for θ,, tile(.) function is applied on the calibrator parameters v/, With the aforementioned weight calibration approach, the calibrator module plays a crucial role during the model training process: at the forward inference path, it scales the value of the base network parameters to make prediction; at the backward optimization path, it serves prioritized weight to regularize the update
'Calibrating the fully connected and convolutional layers suffices to deal with primary continual learning tasks image classification, and our method could also be easily extended calibrate other types of layers.

---

on important parameters (e.g., Vo, Lo is derived in the form of Va, Co @ tile(1))). scaled by the calibrator parameters).
By applying weight calibration on neural network layers, the transformation function at the i-th layer Fo. is parameterized by 0 and produces the output follows,
h2 = Fo, (h--1) s.t. 0, Ro, (0:)-
The output hy of the transformation function is then processed by a feature transformation module (FCM) to generate the calibrated feature map for that layer. We use s2,, (-) to denote the feature transformation function at the i-th layer, parameterized by A, With FCM, the calibration parameters also interact with the feature map h; with a multiplicative operation. Specifically, the calibrated feature map is computed as:
2x (h.) (At)(Ae) @ his Vi E RO and h; E IR HxxxO (Convolution Layer),
(3)
O hi, vi E R and h2 E RO (Fully Connected Layer),
where is the output (channel) size for fully connected layers and the convolutional layers.
In the end, the outputs from (2) and (3) get added up in an element- wise manner by a residual connection. This is followed by normalization and activation operations to produce a final output for that layer. In summary, the overall calibration process for the i-th layer could be formulated as follows,
h. = a (BN (1x, (Fa. (mn-1) Fa. ( (m)))))
S.t. 0, = S2y (f.)
where (-) denotes the batch normalization, C denotes an element- wise addition operator, and (-)) is an activation functioo. hy is sent as input to the + 1-th layer in the feed-forward neural network. All the calibrator parameters are initialized with value of I the start of training. We demonstrate an example case of applying neuron calibration on a CNN-based model in Figure 1.
3.2 Learning Calibration Parameters
We propose a new learning paradigm to train the calibrated neural networks model. During model training, we aim to utilize the training of the calibrator parameters to maximally prevent the catas- trophic forgetting issue for online continual learning. Considering that such plasticity in training is often reflected as drastic changes in parameter values, we set the objective for this calibrator learning as to regularize the parameter change after consolidating the new knowledge and past knowledge (from Dr and Mcx)) not be biased too much from the parameter values before the update. Given the formulation of neuron calibration, the parameter change to be regularized is in terms of the calibrated values rather than that for the vanilla base model parameters.
To formulate the loss function for the training of calibrator parameters, we inherit the elastic weight consolidation (EWC) approach proposed in [12]. Specifically, EWC approximates the true posterior distribution for the continual learning parameters by Gaussian distribution given by the mean from the previous tasks and diagonal precision given by the Fisher information matrix. In our work, we formulate weight consolidation process to avoid catastrophic change on calibrated parameters. In particular, the consolidation process takes place after the base model parameters are trained to absorb new knowledge and rehearsing the past knowledge by replaying data from the episodic memory. Then we train the calibrator parameters with the following loss function,
Lo((v, (x. k.)) = vec (0 Øt)
Aevec (0-0') + 3DKL (
S(
24
(5)
2
term (a)
term (b)
where 3 is trade-off parameter, S(.) is the softmax function, 7 is softmax temperature, 2 are the logits of current model's prediction before the softmax function, and &g is the logits predicted by previous models saved the episodic memory. The operator vec (-) stacks the tensor into vector.
The matrix A, in term (a) are the Fisher information matrix, which obtained from knowledge distillation loss for memory rehearsal. To consolidate the weight in terms of the calibrated values,
5

---

Algorithm 1: Neural Calibration for online Continual Learning (NCCL) Algorithm
Input: Base model fo, calibrator ((N,A) learning rate Oánn/ Okouts outer loss weight (3, training dataset (Di", .... Dij', }, testing dataset {Dfe :..,, Dir 1, episodic memory [M ) observation batch size bs,, rehearsal batch size bs, and confusion matrix to store online learning results CM RTxx" x2"
ACC Output: Base model fa, calibrator S(V.A). ACC (1), FM (4) and LA (1).
function train_and, and__vall
Randomly initialize 0, vb and X. Initialize Mr +- 0 for t = {I.
for t 4 tooo do
/- Training
for 6 < to Rbttchh do
Observe now stream of online data (2,, y/)== b8, from task
Update episodic memory for task t: Mr + (Mr, {ai, yi (il)
for 5 1o slep do
/* Inmer step: consolidate new knowledge and old knowledge
*/
Sample bs, samples from task t: Br + MM.
// intra-class sampling
10
Sample bs, samples from old tasks: Bet t Met // inter-class sampling
11
Binner <- BB UB
12
+ 0--aa Oin
10 1),
/+ Outer step: elastic weight consolidation Bouter + Mac
13
a t t Ocaucer V (Bouter, 0', x)
14
15
+ x - Couter (Bonter, 1,
16
end
17
end
18
Update logits prediction for data from current task: 2 t- f(B ;8, /* Evaluation
;8,40, A)
19
for te + to T do
Evaluate testing accuracy (ACC!e) for the current model on Di
20
Update CM CMU ACC!!
22
end
23
end
24
Compute ACC (t), FM (L) and LA (t) from C.M.
25
return Base model fo, calibrator f(w.)) ACC (t), FM (L) and LA (1).
26
end
the Fisher information matrix needs to computed upon the gradients evaluated on the calibrated parameter values. Formally, we derive its gradient by the chain rule as follows,
Vgi DKL
s(
Vo DKL
s( 2
((
S(
V
ge
(6)
tile(w)
Thus the Fisher information matrix distinguishes the parameters that important retain past knowledge and penalizes their weight change appropriately following the term (a). We add an additional knowledge distillation term (b) to let the training take care of the stabilization issue well, where the stabilization issue refers to the case of avoiding the parameters to experience much sudden change which is considered as a potential source of catastrophic forgetting.
3.3 Interleaved Optimization
We formulate the optimization process to train the calibrated model under interleaved optimization schema [17], with the parameters from the base model and those from the calibrator model being opti-

---

mized by their respective loss functions. Formally, the training process can be formulated as follows:
Outer Loop:
(*' x²) arg (o,s) Le(( A), 0r, MM<,), arg ming Lo((1), A); 0, Mgg),
Inner Loop:
0*
(8)
where (35", >*) denote the optimal calibrator parameters when fixing base model 0, and vice versa. At the inner loop, the base model parameters are updated to learn new knowledge from D, and rehearse old knowledge from Meg. Thus the data for the inner loop update comes from Mrs. with Cs being a cross-entropy loss. At the outer loop, the calibrator parameters are updated to consolidate the weight update from the inner loop against catastrophic change by employing a loss L. which is introduced in Section 3.2. During the interleaved optimization process, we first fix (e) and take gradient steps with regard to as follows:
Oc+1 < 02 Oin Vo La ((1),, At), 0c, Mg.).
Then, we go to optimize the calibrator when the inference takes place with the updated base model,
Ve+1 + ++ - Coout Vv Le ((b,, A), (0.1.
(10)
A411 +- X Coutt V Le ((1, A), Mc1).
(11)
where ain and aout are the learning rates for the inner loop update and the outer loop update, respectively. By employing the calibrated parameterization of the neural networks-based model and optimizing with the interleaved learning scheme, our method could potentially tackle the catastrophic forgetring issue in online continual learning with greater effectiveness than many conventional approaches. We present the detailed algorithm in Algorithm 1.
4 Experiments
In this section, we demonstrate the empirical evaluation results on comparing our method with number of closely related baselines under various experimental settings. Our method is implemented using the PaddlePaddle (PArallel Distributed Deep LEarning) framework.
4.1 Benchmark Datasets
We consider the following four continual learning datasets as our evaluation testheds: Permuted MNIST (denoted as PMNIST) [19], Split CIFAR [19], Split minilmageNet [6], and a continual learning benchmark created with the real world dataset Split CORe50 [18]. For fair comparison in Split minilmageNNet and Split CORe50, we adopt identical datasets as those used in [25]. For online continual learning with P.MNIST, we create 20 independent tasks with each task being created by randomly permuting the order of the pixels in an MNIST image. Each task in PMNIST consists of 1000 samples for the 10 MNIST classes and all the tasks share a common label space. Split CIFAR is constructed by evenly splitting the 100 classes for CIFAR into 20 disjoint subsets, where each task takes 2500 training samples. Split minilmageNNe consists 17 tasks, where each task has 5 classes disjoint from other tasks and each task takes 2500 samples. Split CORe50 is constructed by evenly splitting the 50 classes in the dataset into 10 disjoint tasks, where each task has 5 classes. The details on the dataset statistics is available in appendix.
4.2 Architectures
For the PMNIST experiments, we a multi-layer perceptron with two hidden layers of size 256 [25]. For Split CIFAR and Split minilmageNet experiments, we use a reduced ResNet18 with three times fewer CNN filters than the full R8 [19, 25]. For CORe50 experiments, we use pretrained ResNet18 88 where the pretrained parameters are loaded from paddleCV in PaddlePaddle (we do not use new model be trained from scratch in this task because such models yield extremely poor accuracy which is at around 30% during most of the learning period and thus fail to be valid domain to compare the continual learning ability for different approaches). We show the model capacity in terms of trainable parameter sizes for different models in Table 1. The results demonstrate that our proposed calibration approach leads to moderate increases parameter size compared to its corresponding backbone models for all the architectures. The parameter increase is approximately 10% 155 for the CNN-based models, and that for the multi-layer perceptron- -base models are much more smaller, i.c., much less than 1 1%.
7

---

Table 1: Light-weight model complexity w.r.t parameter size for various network architectures. NCCL results in moderate parameter increase over its backbone architectures in all testified datasets.
Backbone
NCCL
Architecture
Structure
Increase
# Params
# Params
MLP (pMNIST)
269,322
Linear
270,366
0.39%
1-reeued- (CIFAR)
1,109,240
Linear, Convolution
1,265,440
14.08%
ResNet 8-reduced (minilMNN
1,106,825
Linear, Convolution
1,263,025
14.11 %
ResNet18-full (CORe50)
11,202,162
Linear, Convolution
12,430,450
10.96%
4.3 Baselines and Evaluation Metrics
For evaluation, we compare our method with the following continual learning baselines: GEM [19] and AGEM [6] which are two representative gradient projection-baser rehearsal approaches which find and store the important sub-spaces for past gradients and let the learning new task to take gradient steps orthogonal to the saved sub-spaces, MER [27] which performs experience replay with reptile update that regularize updated parameter values to be close to the initial values, MIR [1] which retrieves samples that would be maximally interfered by the new samples to update the model, and CTN [25] which is recently published state-of-the-ar method that performs task-specific scaling on the output of the prediction layers. Apart from the established baselines, we also consider an independent baseline, where each of the online continual learning task is trained with an independent model that share the same model architecture with each other, an offline baseline where the task setting assumes the data for all the tasks are available hand in an off-line fashion where we do not specify a limit for the epoches we train the model. We denote our proposed method (N)euron (C)alibration for online (C)ontinual (L)earning as NCCL. During evaluation, we adopt identical datasets and task ordering for each method. It is also important to note that during the online continual learning process, each mini-batch of samples could be used to update the model for several times and each of the considered method shares such permission. Each score reported is derived from 10 independent runs. We present the detailed hyperparameter settings for all the baseline methods as well as NCCL on all the datasets in appendix.
To evaluate the methods, we adopt three standard evaluation metrics on all the datasets: ACC (1) (higher is better) [19], FM ()) (lower is better) [4], and LA (1) (higher is better) [27]:
Averaged Accuracy (ACC 1): is the continual learning accuracy evaluated after the model has been trained on all the tasks, i.e., ACC = + Σ 01,7, where dij denotes the accuracy on task after training on the task j.
Forgetting Measure (FM 1): the average difference between the final performance obtained for each task compared to the best performance on each task, i.c., FM = + Σ |a,, az where of denotes the best performance on task i.
Learning Accuracy (LA 1): the average of best accuracy evaluated through continual learning for each task domain, i.e., LA = ET= di,s-
We present the empirical results on the continual learning benchmark datasets in Table 2. We observe that our method NCCL consistently outperforms the other baseline approaches with large margins and achieves state-of-the-art performance on all the datasets. Even compared with CTN, strong baseline with remarkable performance records, our method still outperform it on all the evaluation domains with noticeable margin. Overall, compared to CTN, the improvement of our method in terms of ACC (t), FM (L) and LA (f) are 4.67%, 1.62% and 2.79%, which are averaged among the four datasets. On Split CIFAR, Split minilmageNer and 50, our method results in (almost) the highest LA (1) which records the best accuracy achieved throughout the continual learning process. We wish to remind that Independent and Offline are two baselines created by abusing the data accessibility and model consistency to demonstrate the potential upper bound of the online continual learning performance. From the results presented in Table we observe that the performance of our method is close to the Independent baseline on the task PMNIST and better than it on the task Split CIFAR and Split minilmageNNe.. This might due to the reason that our model training could effectively exploit the multi-task property among the tasks. Moreover, in terms of the best accuracy achieved for each task throughout the learning (i.e., LA (f)), our method is better than the strong baseline Independent on all the datasets except for 50Re50 where the gap is quite small.
8

---

Table Evaluation results on four continual learning benchmark datasets. The results are obtained with a replay memory size of 50. We adopt identical (backbone) network architecture for all the compared methods on all the datasets. Each score derived from independent runs.
Split CIFAR
Split minilmageNel
Method
(1C(1)
(C)
LA (f)
ACC (1)
FM
LA (t)
GEM
62.85 10.83
9.6010.76
71.9110.78
59.3611.28
6.97:10.93
63.35:10.67
AGEM
60.43:12.35
9.60112.65
69.20110779
52.7011..12
7.52.11.80
58.33:11.01
MER
61.83.11.18
8,4410094
6..0
55,4912.01
9.11122.0
62.33:11.65
MIR
6..51
8.29111.12
69.20111116
57.25.1.00
5.73.10.79
60.2311.16
CTN
68.62:t0.59
5.92++0.33
73.83+0.71
1.625=+1665
3.78++2.31
67.54+2.87
NCCL (ours)
74.39+00.97
4.88+0097
78.28=00553
69.49=00922
3.36++091
70.69=1.06
Independent*
70.78++0.73 76.98.10.43
0.0
70.78=0.73
64.42
0.0
64,42=1.13
Offline
70.47.10.9
PMNIST
CORe50
Method
(CC(1)
FM
LA
ACC
FM (1)
LA (f)
GEM
74.57+00.1
7.40+0.11
79.13+0.42
77.48+2.72
12.12+44.49
87.92++0337
AGEM
69.50=L0.76
13.10+0.63
82.61=0.41
78.13+2.14
11.60+1.62
88.30=0.65
MER
75.75+0065
8.74+0.73
83.52+00.38
82.75+1.30
7.54
88.04+1.21
MIR
78.31100.63
7.1510.67
84.4810.28
83.3411.44
7.30.11.32
89.52.1045
CTN
79.70.10.44
5.0810.44
84.0810.336
83.8711.13
5.71111008
88.93 88.9310.53
NCCL (ours)
83.47.10.43
3.44.10.26
86.5910.22
88.80
2.22.10.49
00.12.10.38
Independent*
83.50:L0.39
0.0
83.50=0.39
90.34 +0.45
0.0
90.34=0.45
Offline
89.05+0027
90.89 #0.44
4.4 Evaluation Results on Continual Learning Benchmarks
We also evaluate our method under the cases when online continual learning is performed with varying memory size per task. We evaluate the methods when the memory size per task is obtained from the following set: (50, 100, 150, 200, 250]. We train the methods on two tasks, Split CIFAR and Split minilmageNet, and show the evaluation results for NCCL, CTN, MIR and GEM Figure 2.
The performance standard for AGEM and MER is far from the presented baselines and we show them in appendix. From the results, we find all the evaluated methods result in consistent performance improvement as the memory size increases. Overall, NCCL results in superior performance than the baseline approaches under all of the evaluated settings for memory size. We also notice that the baseline approaches could hardly outperform the Offline baseline under the restrictions memory size per task, but our method could outperform at the early stages of the curves. We clarify that is possible for the method to pass the Offline method as the memory size increases [25]. The reason
0.800
0.750
0.775
0.725
Average Accuracy
0.750
0.700
0.725
0.700
0.650
0.675
0.650
Offine
0.600
Offine
0.625
Mi
CTN
0.575
CTN Mi
0.550
100
150
200
250
50
100
150
200
250
Memory (per task)
Memory (per task)
(a) Split CIFAR
(b) Split minilmageNet
Figure 2: Evaluation results with varying memory size per task on the following two benchmark datasets: (i) Split CIFAR and (ii) Split minilmageNet. The presented values correspond to the measures in terms of ACC (t). Each score is derived from 10 independent runs.

---

Table 3: Ablation study results on comparison with Calibrating CNNs for Lifelong Learning (CCLL).
Update Calibrator
Update
Split CIFAR
Split minilmageNet
Base
ACC()
FM (
LA(1)
ACC(1)
FM FM)
LA(1)
CCLL
V
X
64.77=00.73
0.0
64.77=0.73
60.56+: 1.26
0.0
60.56++1.22
V
74.49+00779
74.49+0.79
68.80+1.17
0.0
68.80+1.117
NCCL
74.39+0.97
4.88
78.28+0.53
69.49+00.2
3.36+0.91
10.69+F1.06
that method outperforms Offline under partial memory capacity settings could possibly due to that NCCL could effectively mitigate catastrophic forgetting while exhibiting a desirable ability to transfer knowledge during its multi-task training. Such findings reveal that the replay-based online continual learning setting could potentially pose an effective curriculum for the learning of multiple tasks and become desirable choice for multi-task training.
4.5 Additional Results on Comparing with CCLL
We mostly consider well adopted memory rehearsal- -aase algorithms for benchmark comparison. In this section, we extend such comparison and present additional results comparing with another closely related baseline to ours which calibrates CNN models without memory rehearsal, termed Calibrating CNNs for Lifelong Learning (CCLL) [36].
CCLL calibrates the feature outputs of the CNN layers with additional calibrating layers, to adapt the model to new tasks. It trains all the model parameters on the first task, and only trains the calibrator parameters and the classifier parameters on the subsequent tasks with the base parameters being fixed. Thus it needs to store the parameters for each task and it requires task identities at the inference time during testing. Moreover, CCLL is sensitive the choice of the first task, as the base parameters are desired to be trained on informative tasks that are related to later tasks. As such, the motivation for CCLL essentially different from ours, as we train a task-agnostic calibrator with memory rehearsal and our method calibrator is general to be applied on multiple layers.
We present the comparison results with CCLL in Table 3. We carefully tuned the parameters for CCLL and the settings are available in Section C of appendix. Overall, we noticed that though CCLL alleviates forgetting by storing historical parameters which results in zero FM(1), the ACC(1) of CCLL (train calibrator+classffee w/o memory rehearsal) is much inferior than our method NCCL (train all parameters with memory rehearsal). For sanitary checking, we adapt CCLL model to create variant of CCLL that updates both the calibrator parameters and base parameters. We notice that when updating all the model parameters, the performance apparently higher than the original CCLL and the setting is essentially identical to the independent baseline except their difference in model architecture. The CCLL variant that updates all parameters result in ACC(t) scores that are close to our method in both evaluation domains. But that updates partial parameters following the original proposed form of CCLL results much inferior results than NCCL. Overall, the results demonstrate that calibrating CNN weights with memory rehearsal is promising direction to consider.
5 Conclusion
In this paper, we present a novel online continual learning framework for task-incremental learning problems, which sheds light to new direction tackle online continual learning problems with neuron calibration. We formulate a general neuron calibration approach complemented by an interleaved optimization scheme for effective model training. Our proposed solution enjoys considerable level of generality so that it could potentially tackle many online continual learning problems which employ feed-forward neural networks. One possible limitation for our method is that our work focuses on establishing the generality of the method without considering much on the diversity of neural network layers. Specifically, the formulation we propose tackles the calibration of the feed- forward neural network layers without dealing with other types of architectures, such as the networks with recurrent or recursive structures. Extending our work develop a more inclusive neuron calibration framework is promising future research direction to consider. Also, it is worth considering to adapt our method to be applied on another type of more challenging continual learning problems which is under the inss-innremental settings.
10

---

References
[1] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems (NeurIPS), pages 11849-11860 Vancouver, Canada, 2019.
[2] Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selfless sequential learning. In Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, 2019.
[3] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. In Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
[4] Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Tomr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the 15th European Conference on Computer Vision (ECCV), Part XI, pages 556 572,5 Munich, Germany, 2018.
[5] Arslan Chaudhry, Albert Gordo, Puneet K. Dokania, Philip H. S. Torr, and David Lopez- Paz. Using hindsight anchor past knowledge in continual learning. Proceedings of the thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI), pages 6993-7001, virtual, 2021.
[6] Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with A-GEM. In Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, 2019.
[7] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Ku- mar Dokania, Philip H. S. Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv: IrXiv:/902.111
2019.
[8] Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty- guided continual learning with bayesian neural networks. In Proceedings of the 8th International Conference Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
[9] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), pages 1321-1330. Sydney, Australia, 2017.
[10] Susmit Jha, Sunny Raj, Steven Lawrence Fernandes, Sumit Kumar Jha, Somesh Jha, Brian Jalaian, Gunjan Verma, and Ananthram Swami. Attribution caribution-based confidence metric for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 1 11826-1 1837, Vancouver, Canada, 2019.
[11] Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partitioning reservoir sampling. In Proceedings of the 16th European Conference on Computer Vision (ECCV), Part XIII, pages 41 1-428, Glasgow, UK, 2020.
[12] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Crabska-Barwwinkke et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):3521-3522 2017.
[13] Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory G. Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.088 08383, 2019.
[14] Shuang Li, Yilun Du, Gido M. van Ven, Antonio Torralba, and Igor Mordatch. Energy-based models for continual learning. arXiv preprint 1rXiv:20/1./22/6 2020.
[15] Shuang Li, Yilun Du, Gido M. van de Ven, Antonio Torralba, and Igor Mordatch. Neuromod- ulated neural architectures with local error signals for memory-constrained online continual learning, arXiv preprint 200.:0007.0019., 2020.

---

[16] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overeoming catastrophic forgetting. Proceedings of the 36th International Conference on Machine Learning (ICML), pages 3925-3934, Long Beach, CA, 2019.
[17] Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages (2245-12254, virtual, 2020.
[18] Vincenzo Lomonaco and Davide Maltoni. CORe50: a new dataset and benchmark for continuous object recognition. In Proceedings of the 1st Annual Conference on Robot Learning (CoRL), pages 17-26, Mountain View, CA, 2017.
[19] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems (NIPS), pages 6467-6476, Long Beach, CA, 2017.
[20] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. arXiv preprint arXiv:2101.10423 2021.
[21] Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability-plasticity dilemma: In- vestigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4:504, 2013.
[22] Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In Proceedings of the 6th International Conference Learning Representations (ICLR), Vancouver, Canada, 2018
[23] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[24] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 3942-3951, New Orleans, LA, 2018.
[25] Quang Pham, Chenghao Liu, Doyen Sahoo, and Steven CH Hoi. Contextual transformation networks for online continual learning. Proceedings of the 8th International Conference Learning Representations (ICLR), 2020.
[26] Aylves Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pages 5533-5542, Honolulu, HI, 2017.
[27] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, 2019.
[28] Mark B. Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin, TX, 1994.
[29] Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connect. Sci., 123)1123 1995.
[30] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), pages 348-358, Vancouver, Canada, 2019.
[31] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint, arXiv:1606.04677 2016.
12

---

[32] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In Proceedings of the 9th International Conference on Learning Representations (ICLR), virtual, 2021.
[33] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. Proceedings of the 35th International Conference on Machine Learning (ICML), pages 4548-4557, 4457, Stockholm, Sweden, 2018.
[34] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems (NIPS), pages 2990-2999, Long Beach, CA, 2017.
[35] Daniel L. Silver and Robert E. Mercer. The task rehearsal method of life-long learning: Overcoming impoverished data. In Proceedings of the 15th Conference of the Canadian Society for Computationnl Studies of Intelligence on Advances Artificial Intelligence, pages 10-101, Calgary, Canada, 2002.
[36] Pravendra Singh, Vinay Kumar Verma, Pratik Mazumder, Lawrence Carin, and Piyush Rai. Calibrating cnns for lifelong learning. Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
[37] Gido M Van de Ven and Andreas S Tolias. Generative replay with feedback connections general strategy for continual learning. arXiv preprint arXiv:/809 10635, 2018.
[38] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. Proceedings of the 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
[39] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 2018.
13


Reviews:
[{"review_id": "s4aPzEIObXD", "reviewer": "Reviewer_zHgC", "summary": "This paper presents a rehearsal based calibration technique for continual learning. The method learns parameters used to scale the parameters of every layer in the network, as well as parameters to scale the output of every layer. The network is trained in an interleaved fashion: first the scaling parameters are held fixed and the base network is trained on batches from the current task and examples from rehearsal memory. Then the base network is held fixed and the scaling parameters are trained using examples just from the rehearsal memory. This approach is tested on several different benchmarks and shows strong results.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows strong results on several continual learning benchmarks. It introduces a novel calibration technique that expands work in the line regularization based approaches for continual learning. Despite good results, I have a few issues/questions:\n\n- Given that you are using a rehearsal based approach, you should have included the basic [Tiny Episodic Memory baseline](https://arxiv.org/pdf/1902.10486.pdf)\n- I would have liked to see an ablation where you used only one of either parameter calibration or feature calibration, and turned the other off (instead of replacing with dropout), to see whether both are actually necessary.\n- Several of the baseline methods listed (GEM, AGEM) use a task id during inference to specify the output labels. The supplementary lists them, as well as your approach, as not using task ids. Does your approach use task ids during training/inference to specify the valid output space?\n- The description of $\\mathcal{L_c}$ is a bit confusing, and several things are defined significantly after they are used ($S(), \\hat{z}, \\hat{z}_k$). The description of the Fisher information matrix, what it represents/how its being used can also be improved.\n- There is a \"stabilization issue\" mentioned on page 5, but it is a bit unclear what that issue refers to, and how term b in the loss fixes it. \n- For your baselines: does offline simply refer to a multitask baseline? Is there a limit on how many epochs you train the offline for?\n- On page 8, you mention that the learning accuracy of your method is better than or equal to the offline method for all datasets. Did you mean to say independent, since offline does not have a learning accuracy reported? Also, a potentially more useful baseline comparison than independent (only for learning accuracy) would be finetune, as independent might be handicapped since it has to start from random initialization for every task.\n- The checklist mentions you have provided code, but I could not find it. The github link in the supplementary had an empty repo. Could you please update the code?\n\nEdit after discussion: I am raising my score to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "EqaxzICz_vq", "reviewer": "Reviewer_Eiq8", "summary": "The paper proposed a new online continual learning approach using neuron calibration techniques including model weight calibration and feature calibration. These calibration procedures only increase about 1x% model size which is reasonable. The performance on several datasets is impressive compared to other baselines. One surprising result is that the proposed method can even outperform offline learning (Fig. 2). ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe proposed task agnostic calibration techniques are novel for online continual learning.\nThese calibration procedures only increase about 1x% model size which is reasonable.\n\nQuality:\nThe results on many datasets are impressive, especially on the mini-ImageNet dataset.\nMissing reference or experiment\n1. [32] is indeed very similar to the proposed method excepted the method is not originally designed for online continual learning. However, I wonder what its performance is, if once naively apply the method on the online continual learning dataset.\n2. A few latest works are missing in the discussion:\n* Mitigating Forgetting in Online Continual Learning via Instance-Aware Parameterization. Chen et al. NeurIP'20.\n* Continual Learning with Node-Importance-based Adaptive Group Sparse Regularization. Jung et al. NeurIP'20.\n\nSignificance:\nThe most impressive finding is in Fig. 2 where the proposed method can even outperform offline learning. \nRather than referring to [21], it will be even more impressive if some diagnoses can be done to understand the reason better.\n\nClarity:\nThe paper is well written with clear messages, equations, and figures.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It seems to online continual learning is still a very new task to be studied. Hence, experimental settings are not aligned across papers.\nOther than comparing with [21], why not adapting other settings or what's the issue of other settings?", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "B31ExVu85N", "reviewer": "Reviewer_Ng2E", "summary": "This work proposes an online continual learning approach that learns an attention map on the weights (termed as weight calibration module) and another on the features (termed as feature calibration module) of a neural network. The training is done by adding a stability regularization term (similar to CTN) to the original EWC loss and interleaving the optimization of network parameters and attention modules as the new tasks are observed continually. A task-incremental learning scenario is considered, and the accuracy is compared against five other approaches with four benchmark datasets using three different metrics.  With a marginal increase in the number of parameters with the attention maps, we can see considerable improvement in accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "• Online continual learning is generally associated with memory limitations [1,2,3]. Maintaining a buffer from all the previous tasks can be memory intensive.\n\n•The number of gradients updated for each approach is mentioned in the appendix, but it is not clear if all the data (current batch + memory buffer) is used for multiple updates/steps. Using the current batch multiple times violates the online continual learning scenario\n\n• Is the parameter convergence evaluated after one pass over the data?\n\n• It is claimed that the approach is task-agnostic, but the experiments seem to be in a task-incremental setting (one output head per task) where the task label is provided at training and test time.\n\n• Other related methods [4,5] should be considered in the comparison\n\n• It should be made clear in the introduction on what continual learning scenario is considered [1] and some thoughts on the applicability of this approach from a more realist class-IL scenarios.\n\n• There are also OCL approach without memory buffer that outperform memory-based approaches [3,6]. They should also be included in the related work\n\n[1] Mai, Zheda,  et al. \"Online Continual Learning in Image Classification: An Empirical Survey.\" arXiv preprint arXiv:2101.10423 (2021).\n[2] Chen, H.-J., et al. “Mitigating forgetting in online continual learning via instance-aware parameterization” Advances in Neural Information Processing Systems, 33, 2020\n[3] Madireddy, Sandeep et al. “Neuromodulated Neural Architectures with Local Error Signals for Memory-Constrained Online Continual Learning”, arXiv preprint arXiv:2007.08159 (2021)\n[4] Mirzadeh, Seyed Iman, et al. \"Dropout as an implicit gating mechanism for continual learning.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 232-233. 2020.\n[5] Yoon, Jaehong, et.al. \"Scalable and order-robust continual learning with additive parameter decomposition.\" arXiv preprint arXiv:1902.09432 (2019).\n[6] Li, S., et al. \"Energy-Based Models for Continual Learning.\" arXiv preprint arXiv:2011.12216, 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not adequately mentioned,  potential negative societal impact of their work are included in the manuscript", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops an approach to mitigate catastrophic forgetting in continual learning using neuron calibration techniques, including model weight and feature calibration, with rehearsal over the data. The resulting approach has impressive performance on task-incremental learning, especially compared to the other baselines. The paper is well-developed and clear with compelling results.

There are still a few remaining concerns that the authors are encouraged to address.  Most notably, adding class-incremental learning results and comparisons with related work would have provided more insights into the strengths and weakness of the proposed approach.  It would also help improve the strength of the results and broaden the paper. The authors are also encouraged to see if they can provide a deeper explanation of the results in Figure 2, helping to explain why the their method performs so well in comparison to offline learning. The authors' responses during the discussions included a number of clarifications and detail that should be added to the paper, including a promised clarification on what exactly is meant by the task-agnostic aspects of the approach.
Decision: Accept (Poster)