Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
Authors: Elahe Arani
Fahad Sarfraz
Bahram Zonooz
Venue: ICLR 2022 Poster (2022)
Keywords: Continual Learning
Catastrophic Forgetting
Complementary Learning Systems Theory
Experience Replay

Abstract: Humans excel at continually learning from an ever-changing environment whereas it remains a challenge for deep neural networks which exhibit catastrophic forgetting. The complementary learning system (CLS) theory suggests that the interplay between rapid instance-based learning and slow structured learning in the brain is crucial for accumulating and retaining knowledge. Here, we propose CLS-ER, a novel dual memory experience replay (ER) method which maintains short-term and long-term semantic memories that interact with the episodic memory. Our method employs an effective replay mechanism whereby new knowledge is acquired while aligning the decision boundaries with the semantic memories. CLS-ER does not utilize the task boundaries or make any assumption about the distribution of the data which makes it versatile and suited for ``general continual learning''. Our approach achieves state-of-the-art performance on standard benchmarks as well as more realistic general continual learning settings.

Published as a conference paper at ICLR 2022
LEARNING FAST, LEARNING SLOW: A GENERAL CONTINUAL LEARNING METHOD BASED ON COMPLE- MENTARY LEARNING SYSTEM
Elahe Arani", Fahad Sarfraz" & Bahram Zonooz
Advanced Research Lab, NavInfo Europe, Eindhoven, Netherlands (elahe. arani, fahad. sarfraz)
(arfraz//aavinoo..mmmm @eavinno..es bahram. zonooz @gmal com
ABSTRACT
Humans excel at continually learning from an ever-changing environment whereas it remains a challenge for deep neural networks which exhibit catastrophic forget- ting. The complementary learning system (CLS) theory suggests that the interplay between rapid instance-based learning and slow structured learning in the brain is crucial for accumulating and retaining knowledge. Here, we propose CLS-ER, a novel dual memory experience replay (ER) method which maintains short-term and long-term semantic memories that interact with the episodic memory. Our method employs effective replay mechanism whereby new knowledge is ac- quired while aligning the decision boundaries with the semantic memories. CLS- ER does not utilize the task boundaries make any assumption about the dis- tribution of the data which makes it versatile and suited for "general continual learning". Our approach achieves sfate-of-the-art performance on standard bench- marks as well as more realistic general continual learning settings.
INTRODUCTION
Continual learning (CL) refers to the ability of learning agent to continuously interact with dynamic environment and process a stream of information to acquire new knowledge while consoli- dating and retaining previously obtained knowledge (Parisi et al., 2019). This ability to continuously learn from a changing environment is a hallmark of intelligence and a critical missing component in quest towards making models truly intelligent. The major challenge towards enabling CL in deep neural networks (DNNs) is that the continual acquisition of incrementally available informa- tion from non-stationary data distributions leads to catastrophic forgetting whereby the performance of the model on previously learned tasks drops drastically (McCloskey & Cohen, 1989).
Several approaches have been proposed to address the issue of catastrophic forgetting in CL. These can be broadly categorized into regularization-ban based methods (Farajtabar et al., 2020; Kirkpatrick al., 2017; Ritter et al., 2018; Zenke al., 2017) which penalizes changes in the network weights, network expansion-base based methods (Rusu et al., 2016; Yoon et al., 2017) which dedicate a distinct set of network parameters to distinct tasks, and rehearsal-based methods (Chaudhry et al., 2018; Lopez-Paz & Ranzato, 2017) which maintains a memory buffer and replays samples from previous tasks. Amongst these, rehearsal-baae
methods have proven to more effective in challenging CL tasks (Farquhar & Gal, 2018). However, an optimal approach for replaying memory samples and constraining the model update to efficiently consolidate knowledge remains an open question.
In the brain, the ability to continually acquire, consolidate, and transfer knowledge over time is mediated by a rich set of neurophysiological processing principles (Parisi al., 2019; Zenke al., 2017) and multiple memory systems (Hassabis et al., 2017). In particular, the CLS theory (Kumaran et al., 2016) posits that efficient learning requires two complementary learning systems: the hip- pocampus exhibits short-term adaptation and rapid learning of episodic information which is then gradually consolidated to the neocortex for slow learning of structured information. Furthermore, a recent study by Hayes et al. (2021) identified the missing elements of biological reply in the replay
"Contributed equally.
'The code is avaiable httpps://github..comm// wtt.:ttb..c..m...raat.t. -lab CC.SS

---

Published as a conference paper at ICLR 2022
Data Stream D
Update
Episadic Memory
Plastic Model
Working Model
Stable Model
Reservoir
Fast
Slow Jearning
learning
recent experienzes
Merooiizaiion of epieodic-Hte events
siructoraf knowledge
Short-tam adaptation
Storage, retrivval, replay
Storage. retrivval, replay
Long-term retention
Learns statistical strusture of the perceived eveent
Working Model
Consistency Loss
Plastic Model Op
Stable Model
Sonundic Memory
Efficient
Elficient representation
Update with rates
0g
representation
tP r,
Figure 1: CLS-ER employs a dual-memory learning mechanism whereby the episodic memory stores the samples and the semantic memories build short-term and long-term memories of the learned representations of the working model. The two memories interact to enforce a consistency loss on the working model which prevents rapid changes in the parameter space and enables the alignment of the decision boundary with semantic memories for effective knowledge consolidation.
mechanisms employed in DNNs for CL. They highlight that many existing approaches only focus on modeling the prefrontal cortex directly and do not have a fast learning network which plays a critical role in enabling efficient CL in the brain. Inspired by these studies, we hypothesize that mimicking the slow and rapid adaptation of information and having efficient mechanism for incorporating them into the working memory can enable better CL in DNNs.
To this end, we propose novel dual memory experience replay method based the complementary learning systems theory in the brain, dubbed as CLS-ER. In addition to a small episodic memory. method builds long-term and short-term semantic memories which mimic the rapid and slow adaptation of information (Figure 1). As the network weights encode the learned representations of the tasks (Krishnan al., 2019), the semantic memories are maintained by taking the exponential moving average of the working model's weights to consolidate information across the tasks with varying time windows and frequencies. The semantic memories interact with the episodic memory to extract consolidated replay activation patterns and enforce consistency loss on the update of the working model so that new knowledge is acquired while aligning the decision boundary of the working model with the decision boundaries of semantic memories. This maintains a balance between the plasticity and stability of the model for effective knowledge consolidation.
CLS-ER provides a general CL method that does not utilize the task boundaries or make any strong assumption regarding the distribution of the data and tasks. We demonstrate the versatility and effectiveness of our method wide range of CL benchmark tasks as well more challenging scenarios which simulate the complexities of CL in the real world.
2 RELATED WORK
The base method for the rehearsal-based approach, Experience Replay (ER) (Riemer et al., 2018) combines the memory samples with the task samples into the training batch. Several techniques have since been employed on top of ER. Meta Experience Replay (MER) (Riemer al., 2018) considers replay as a meta-learning problem for maximizing the transfer from previous tasks and minimizing the interference. iCARL (Rebuffi et al., 2017) uses the nearest average representation of past exemplars to classify in an incrementally learned representation space. Gradient Episodic Memory (GEM) (Lopez-Paz& & Ranzato, 2017) formulates optimization constraints on the exem- plars in memory. Gradient-based Sample Selection (GSS) (Aljundi et al., 2019) aims for memory sample diversity the gradient space and provides a greedy selection approach. Function Distance Regularization (FDR) (Benjamin et al., 2018) saves the network response at the task boundaries and adds a consistency loss on top of ER. Dark Experience Replay (DER++) applies knowledge distil- lation (Sarfraz et al., 2021) and regularization on logits sampled during the optimization trajectory.
CLS has been used as source of inspiration for dual memory learning systems in earlier works (French, 1999; Robins, 1993) but they have not been shown to scale to current computer vision tasks (Parisi et al., 2019). Recently, Rostami et al. (2019) utilizes a generative model to couple sequential tasks in latent embedding space. Kamra al. (2017) utilizes two generative models
2

---

Published as a conference paper at ICLR 2022
Stable Model
Working Model
Plastic Model
50.7
6.0
0,0
0.5
90.6
03
0.0
0.0
0.0
88:55
0.0
0.0
0.0
84:23
05
0.0
0.0
0.0
87.5
74.6
0.0
58.25
92.65
0.0
0.0
Task
Task
Bask
Bask
Taak
Raak 5
Tack
Tack
Task
Tosk
Rock
Task
Busk
Back
Rask
Task 5
Figure 2: Task-wise performance on S-CIFAR-10 test set with 500 buffer size. The models are evaluated at the end of each task (y-axis) to evaluate how the task performances (x-axis) are af- fected as training progress. The stable model retains information from earlier tasks while the plastic model quickly adapts to the recent task. Note that there is less forgetting the semantic memories compared to the working model. For other buffer sizes and S- Tinnymageeee see Figures S1 and S2.
in dual memory architecture. However, they utilize the task boundaries and generative replay has its own set of challenges as difficult to learn a faithful distribution and performs sub-par in comparison to instance-based replay methods on challenging CL settings. Generally, the inspiration from CLS theory in DNNs has been mostly limited to episodic memory and mimicking the rapid and slow learning mechanism is majorly ignored (Hayes et al., 2021) which we aim to address.
3
METHOD
We first provide an overview of the CLS theory for the brain and how we aim to mimic it for DNNs before introducing the main components of our method and the overall formulation.
3.1
COMPLEMENTARY LEARNING SYSTEM THEORY
The CLS theory posits that effective lifelong learning in the brain requires two complementary learn- ing systems. The hippocampus rapidly encodes novel information as short-term memory which subsequently used to transfer and consolidate knowledge in the neocortex which gradually acquires structured knowledge representation as long-term memory through experience replay. The interplay between the functionality of the hippocampus and ncocortex is crucial for concurrently learning effi- cient representations (for better generalization) and the specifics of instance-bart based episodic memory.
3.2
COMPLEMENTARY LEARNING SYSTEM BASED EXPERIENCED REPLAY
Inspired by the CLS theory, we propose dual memory experience replay method, CLS-ER, which aims to mimic the interplay between fast learning and slow learning mechanisms for enabling ef- fective CL in DNNs. Our method maintains short-term and long-term semantic memories of the encountered tasks which interact with the episodic memory for replaying the associated neural ac- tivities. The working model is updated so that it acquires new knowledge while aligning its decision boundary with the semantic memories to enable the consolidation of structured knowledge across the tasks. Figure highlights the parallels between CLS theory and our method.
Semantic Memories:
Central to our method is the maintenance of two semantic memories which accumulate and consolidate information over long-term and short-term periods. the acquired knowledge of the learned tasks is encoded in the weights of DNNs (Krishnan et al., 2019), aim to form our semantic memories by accumulating the knowledge encoded in the corresponding weights of the model as it sequentially learns different tasks.
An efficient method for aggregating the weights of model is provided by Mean Teacher (Tarvainen & Valpola, 2017) which is a knowledge distillation approach that uses exponential moving aver- age (EMA) of the student's weights during training as a teacher for semi-supervised learning. It can also be considered as forming self-ensemble of the intermediate model states that leads to better
3

---

Published as a conference paper at ICLR 2022
internal representations. We adapt the Mean Teacher approach to build our semantic memories it provides a computational and memory--fficient method for accumulating knowledge over the tasks.
As CL involves learning tasks sequentially, the model weights each training step can be consid- cred as a student model specialized for a particular task. Therefore, averaging the weights during training can be considered as forming tively aggregates information across the tasks and leads to smoother decision boundaries. CLS-ER builds long-term (stable model) and short-term (plastic model) semantic memories by maintaining two EMA-weighted models over the working model's weights. The stable model is updated less frequently with a larger window size so that retains more information from the earlier tasks while the plastic model is updated more frequently with a smaller window size so that it adapts faster to information from new tasks (Figure 2). Section D further demonstrates the benefits of employing two semantic memories instead of a single semantic memory.
ensemble of task-specific student models which effec-
Replay of samples from the previous tasks stored in small episodic memory is common approach in CL that has proven to be effective in mitigating catastrophic forgetting. As we aim to position CLS-ER as a versatile general incremental learning method, we do not utilize the task boundaries or make any strong assumptions about the distribution of the tasks or samples. Therefore, to maintain a fixed episodic memory buffer, we employ Reservoir sampling (Vitter, 1985) which assigns equal probability to each sample in the stream for being represented in the buffer and randomly replaces the existing memory samples (Algorithm 2). It is a global distribution matching strategy that ensures that any given time the distribution of samples in the buffer will approxi- mately match the distribution of all the samples seen so far (Iscle& & Cosgun, 2018).
Episodic Memory:
Consolidation of Information: with the previously acquired information. This requires an effective balance between the stability and plasticity of the model. Furthermore, the sharp change in decision boundary as a new task is learned makes the consolidation of information over tasks more challenging. CLS-ER tackles these challenges through a novel dual memory experience replay mechanism. The long-term and short- term semantic memories interact with the episodic memory to extract the consolidated activations for the memory samples which are then utilized to constrain the update of the working model so that new knowledge is obtained whilst the decision boundary is aligned with the semantic memories. This prevents rapid changes the parameter space as new tasks are learned. Furthermore, aligning the working model's decision boundary with the semantic memories serves two goals: (i) helps in retaining and consolidating information and (ii) leads to a smoother adaptation of decision boundary.
The key challenge in CL is the consolidation of new information
3.3 FORMULATION
CLS-ER involves training working model fl: on data stream D sampled from non-iid distribution. Two additional EMA-weighted models are maintained as semantic memories: plastic model [(::0p and the stable model / (.; 0s). Finally, Reservoir sampling (Vitter, 1985) is employed to maintain a small episodic memory M.
At each training step, the working model receives the training batch X5 from the data stream and retrieves a random batch of exemplars Xm from the episodic memory. This is then followed by the retrieval of optimal semantic information, i.e. the structural knowledge encoded in the semantic memories which account for the consolidation of feature space and adaptation of the decision bound- aries of the previous tasks. The semantic memories are designed so that the plastic model has higher performance on recent tasks whereas the stable model prioritizes retaining information on the older tasks. Therefore, we would prefer to use the logits from the stable model Zs for older exemplars and the plastic model Zp for recent exemplars. As CLS-ER is general incremental learning method, instead of using a hard threshold or task information, we opt for a simple task-agnostic approach selection criterion that
of using the performance of the semantic memories on the exemplars empirically works well. For each exemplar, we select the replay logits Z based on which model has the highest softmax score for the ground-truth class (lines 5-6 in Algorithm 1).
The selected replay logits from the semantic memories are then used to enforce a consistency loss on the working model that it does not deviate from the already learned experiences. Hence, the working model is updated with a combination of the cross-entropy loss on the union of the data stream and episodic memory samples, X, and the consistency loss on the exemplars Xm.
f Lor(o( (Xx;(((X)) Y) + XL (X 0w),2)
(1)
4

---

Published as a conference paper at ICLR 2022
Algorithm 1 Complementary Learning -eserneExperience Replay Algorithm
Input: Data stream D, Learning rate 77, Consistency weight A, Update rates rp and rs, Decay parameters op and OsS
Initialize: fw 0p 0s
M < -
1: while Training
2:
Yb), DD and (Xm Ym)~ ~
3:
(X,Y) = ((x,, Yo), (Xm: Ym)}
4:
Zp, + f(Xmi ammi0e 0p): f(Xmi 0g)
Select optimal semantic memory
5:
Z Ep (Z) (Xm) 0 (Z)) else Zg
((m)))))
6:
[ = (fx 1), ) + ALMSE (f (xmi Bw) Z)
b Update working model
7:
Bw 8w n/Vew
8:
a, b ~ M(0,1) 1)
b Update semantic memories
9:
OP + ap0p (1 up 0r)) if Tp else 0p
10:
OS + asts + (1 as Bw i < rs else 0g
M + Reservoir (M, (Xb, Y.)
Update episodic memory (Algorithm 2)
return fw. 0p.0s 0s
where a is the softmax function, À the regularization parameter, and LMSE the mean squared error loss used as consistency term.
After updating the working model, we stochastically update the plastic and stable models with rates rp and rs (note that rp > rs so that the plastic model is updated more frequently). stochastic rather than deterministic approach is more biologically plausible (Maass, 2014; Arani et al., 2021) which reduces the overlap in the snapshots of the working model and leads to more diversity in semantic memories. The semantic memories are updated by taking exponential moving average of the working model's weights (Tarvainen & Valpola, 2017) with decay parameters ap and ag.
0 a,0, + (1 (1-a,)0w:
i E (P.S}
Note that ap I ag so that the plastic model mimics the rapid adaptation of information while the stable model mimics slow acquisition of structured knowledge. See Algorithm I for more details.
For inference, we use the stable model as it retains long-term memory across the tasks, consolidates structural knowledge, and learns efficient representations for generalization (Figure 1).
4 EXPERIMENTAL SETUP
To ensure fair comparison of different CL methods under uniform experimental settings, ex- tended the Mammoth framework (Buzzega et al., 2020a) and unless stated otherwise, we follow the same training scheme (learning rate, batch sizes of incoming data and memory buffer, and the number of training epochs) as them for each of the evaluation settings. To find the optimal hyper- parameters for CLS-ER,
grid search over X, ag, a,, Ts, and rp small validation set. Sections C.4 and E show that our method not highly sensitive to the particular choice of hyperpa- rameters and different settings can attain similar performance. Also, because of the complementary nature of the components, we can often fix set of parameters (e.g. X, a.s. ap and rs) and only finetune the remaining parameters (e.g. TP) which facilitates hyperparameter tuning significantly.
Following Buzzega et al. (2020a), we employ a fully connected network with two hidden layers, each with 100 ReLU units on all the variants of the MNIST dataset and ResNet-18 (He et al., 2015) without pretraining for the other datasets. In all the settings, we use the SGD optimizer. We use random horizontal flip and random crop on both the stream and buffer samples for S-CIFAR-10, S- -Tiin-Immageee and CCIL-CIFAR-100. The selected hyperparameters for each of the settings are provided in Table S4. Note that for the vast majority of datasets, we use uniform settings (Ir, epochs, batch size, memory batch size, and lambda) across different buffer sizes and only slight modifications in the other hyperparameters which shows that our method does not require extensive finetuning for different memory budgets. For each of our experiments, and report the average and one standard deviation of the mean test accuracy of all the tasks across 10 runs with different initializations. Section E provides further training and implementation details.
fix the order of the classes
5

---

Published as a conference paper at ICLR 2022
Class-IL
Domain-IL.
Buffer
Method
S-MNIST
S-CIFAR-IO
R-MNIST
P-MNIST
JOINT
95.37a024
92.20am.ss
59.99sa19
95. 95ma6
94.33am.7
SGD
19.60x004
19.62.0.05
7.92+026
67.66ia.s
40.70
ER
80.432189
44.7921.6
8.492016
85.01
72.3720.7
GEM
80. 1.554
25.54+0.76
80.80-115
66.93-1.25
iCaRL
70.5
49,02a3.20
7.53
FDR
79.43
30.9 22.74
8.70
85.22a3.35
74.77
200
GSS
38.92-2.40
39.07 +5.99
79.50=e.4
63.7240.70
DER++
85.61a1.40
64. 11
10.96a111
90.43
83.5820.59
CLS-ER
89.54.0.2
66.19.0.75
23.47
92.26.0.18
84.63
ER
12.12.1.8
27.74
9.99
88.91
80.60a0.8
GEM
85.99.1.35
26.20a1.26
81.15
16.88
iCaRL
70.10
47.55
9.38s1.53
FDR
85.87
28. 43.23
10.54
89.67
1
500
GSS
49.76x4.73
49. 19.73
81.58
76.00s0.87
DER++
91.00a1.49
72.70
19.38a141
92.77atas
88.21 a0:39
CLS-ER
92.05.0.2
75.2220.71
31.0320.5
94.06:0.07
88.30
ER
93.40s1.29
82.47
27 27.402031
93.45
89. 39.90am13
GEM
95.1 18087
25.26
88.57
87.42.0.95
iCaRL
70.60a1.05
55.07
14.08a192
FDR
87.47
19. 19.70
28.972041
94. 14.144
90.8720.1
5120
GSS
89.39x075
67.27.4.27 +4.27
85.24
82,22
DER+
95.30+1.2
85.24.0.49
39.02+097 46.742031
94.65-0.3
92.26.0.17
CLS-ER
95.7320.11
86.78
94.25
92.03anas
Table Comparison with prior works on Class-II. and Domain-Il. settings. The baseline results are from Buzzega et al. (2020a) (- indicates the experiments that the authors were unable to run).
EMPIRICAL EVALUATION
There are a plethora of evaluation protocols in the CL literature, each of which biases the evaluation towards a certain approach (Farquhar & Gal, 2018; Mi al., 2020; van de Ven & Tolias, 2019). is therefore of utmost importance to conduct an extensive and robust evaluation over different CL settings to gauge the versatility the method. Details of the datasets used in each CL. setting are provided in Section A. We compare our method with the state-of-ther -ff-the-artrr art rehearsal-basee approaches on various CL settings and memory budgets under uniform experimental settings. SGD refers to standard training and JOINT provides upper bound given by training all tasks jointly.
Class Incremental Learning (Class-IL): refers to the CL scenario where new classes are added with each subsequent task and the agent must learn to distinguish not only amongst the classes within the current task but also across previous tasks. Class-IL. measures how well the method can learn general representations, accumulate, consolidate, and transfer the acquired knowledge to learn efficient representations and decision boundaries for all the classes seen so far.
Table provides the comparison with six rehearsal-based approaches on Class-IL settings with varying datasets and task length complexities. CLS-ER provides the highest performance in all of these scenarios. In particular, as the dataset complexity and number of tasks increase from S-MNIST to S- S-Tiny- ImageNet, the performance gap between CLS-ER and DER++ increases considerably. Especially, with a smaller memory budget, CLS-ER is able retain more information than other methods. In the most challenging setting, S-Tiny- a percentage gain of 176% and 1 114% over the baseline ER and the current state-of-the-ar DER++ respectively. The results demonstrate the capability of CLS-ER to efficiently accumulate and retain knowledge over longer sequences under complex and memory restrictive scenarios.
with 200 buffer size, CL.SERR provides
We believe that the performance gains over DER++ highlight key component of an efficient CL agent: the ability to consolidate previously acquired knowledge. DER++ fails to account for the con- solidation of feature space and adaptation of the decision boundaries of the previous tasks. There- fore, constraining the model to match the sub-optimal logits might hamper the consolidation of knowledge. This becomes more prominent the number of classes in each task, the sequence

---

Published as a conference paper at ICLR 2022
JOINT
SGD
Buffer
ER
MER
GSS
DER++
CLS-ER
200
49 29.27
48.58
43.92a243
54. 16.302
66.37
82.98
19.09
500
65.04a1.5
6
54.45as.14 63.84
75.70 76.03x1.61
69.62a1.59
1000
18.18
10.9
79.54:034
Table 2: Comparison with prior works on MNIST-360 test set. The baseline results are from Buzzega et al. (2020a).
Distribution
Uniform
Longtail
JOINT
58.36s1.02
56.9421.56
SGD
12.67
22.88
Buffer
200
500
1000
200
500
1000
ER
16.40
28.2
31.98
19.27
20.30
34. 13a183
DER+ CLS-ER
18.84
32.92+0.4
38.95
26.94+1.27
25.82-081
33.6440.8
25.06-081
36.34.0.59
39.69m066
28.54.087
28.63-0
39.52+a.1
Table 3: Comparison with prior works on GCIL-CIFAR--000 dataset.
length, and the cross-task resemblance increase. For instance, for DER++ replaying a sample from Task-1 when training on S-Tiny-ImageNet Task-10, the reference logit values which are used to enforce the consistency are from a model representation state which has not considered how to dis- tinguish the 20 classes in Task-1 from 80 additional classes which are visually and semantically similar. stands to reason that the optimal representation space and subsequently the decision boundaries for the classes in Task- I would drift considerably when required to distinguish between 80 additional classes as well. Therefore, the local information provided by the sub-optimal saved logits in DER++ fails to provide the global context required for consolidating knowledge. CL-S-ER on the other hand, extracts logits from the semantic memories which consolidate knowledge across the tasks, and hence the working model receives more optimal feedback.
Domain Incremental Learning (Domain-IL): refers to the CL scenario where the classes remain the same in subsequent tasks but the input distribution changes. We consider R-MNIST where each task contains digits rotated by fixed angle and P P-MNIST which applies a fixed random permutation the pixels for each task. Table shows that CLS-ER provides generalization gains under both settings, particularly for lower memory budget, and performs on par with DER++ on 5120 buffer size. We attribute this to the consolidated soft targets from the semantic memories which provide relational information about the classes from a global context compared to the local information in DER++. This enables our method to maintain the similarity structure across sequences effectively,
General Incremental Learning (GIL): Class-IL and Domain-IL fail to assimilate the challenges the real-world setting where the task boundaries are blurry, and classes can reappear and have different distributions. The CL method has to consider the sample efficiency, challenge of imbal- anced data, and efficient knowledge transfer in addition to preventing catastrophic forgetting. We consider two GIL settings: MNIST-360 (Buzzega al., 2020a) exposes the model to both sharp (changes in class) and smooth (rotation of digits) distribution shifts. This requires the CL method to tackle the challenges of class-IL as well domain-IL. The Generalized Class Incremental Learning (GCIL; Mi et al. (2020)) is the closest to the real-world scenario as it utilizes probabilistic modeling sample the classes and data distributions in each task. The number of classes in each task is not fixed, the classes can overlap and the sample size for each class can vary.
Table 2 shows that CLS-ER provides considerable performance gains on the challenging MNIST- 360, particularly with low memory budget. Similarly, Table 3 demonstrates the effectiveness of CLS-ER on GCIL-CIFAR-100 under both uniform and imbalanced class samples. Both of these settings involve recurring classes in subsequent sequences which makes the transfer of knowledge from previous occurrences important. The performance gap between CLS-ER and DER++ in the recurring classes setting alludes to another shortcoming of saving logits from the previous state. Consider the case where class c appears in sequence (Seq)-1 with 20 samples, and then subsequently in S with 200 samples. In the following sequences, DER++ uses exemplars from class saved in Seq-1 with sub-optimal logits from the model state which was attained with only 20 samples and fails to take advantage of the better learned representations with additional data in Seq-5. CLS-ER,
7

---

Published as a conference paper at ICLR 2022
17.5
FR
BR DER+ CLS-ER
Tosk
15.0
DER CLS-ER
12.5
Train Accuracy
0.6
Task Probability
Trck
Train Loss
7.3
0.2
5.0
25-
0.1
20
30
20 30
CLS-ER
Perturbation (ax10
Perturbation (ax10
Figure 3: Model characteristics analyses of different methods trained on S-CIFAR-10 with 500 buffer size. The Left and middle figures show the training loss and aceuracy under varying Gaussian noise added to the weights of each layer of the model. CLS-ER is considerably less sensitive to per- turbations, suggesting convergence to flatter minima. The right figure shows task probabilities. CLS-ER effectively mitigates the bias to the recent tasks and provides more uniform probability of being predicted for the classes over the tasks even very early ones.
on the other hand, able take advantage of the additional samples and provide feedback from the improved learned representations. Moreover, the considerable performance improvement in the longtail setting shows that CLS-ER is more robust to class imbalance
Note that the MNIST-basee settings can be considered under the online CL setting (see Section A.4) as we only pass through the data once for each task and the performance of CLS-ER on these settings demonstrates its potential as an efficient method for online CL.
MODEL CHARACTERISTICS
analyze CLS-ER and provide some insights into characteristics of the proposed approach which enables it to learn effectively under challenging CL scenarios. In the subsequent analyses, we compare CLS-ER with the baseline ER and DER++
6.1 CONVERGENCE TO FLATTER MINIMA
Due to the non-convexity of the loss landscape, there can be multiple solutions to the optimization objective, however, the local geometry the convergence point can affect the generalization of the model. Solutions that reside in wide valleys instead of narrow crevices generalize better (Chaudhari et al., 2019; Hochreiter & Schmidhuber, 1997; Keskar et al., 2016) as predictions do not change drastically with small perturbations. A CL model which converges to flatter minima has more flex- ibility to explore the neighboring parameter space to optimize on the new task without drastically increasing the loss the previous tasks. Following the analysis in Zhang et al. (2018), 8), we add inde- pendent Gaussian noise of increasing strength to the parameters of the trained model and analyze the change in accuracy and loss across the training samples. Figure shows that CLS-ED ER is significantly less sensitive to perturbations compared to ER and DER++ CLS-ER ER also retains performance for longer period and its performance drops more smoothly. These results suggest that the fast and slow adaptation of information in CLSSER can guide the optimization to wider valleys.
6.2 TASK PROBABILITIES
Because of the sequential nature of CL, an implicit bias is induced towards the current task (Wu et al., 2019). number of CL methods employ explicit techniques to reduce this bias (Hou et al., 2019; Wu et al., 2019), however, they utilize the task boundaries which is counterproductive for general incremental learning. We believe that the efficient knowledge consolidation in CL.--RR through the semantic memories can implicitly mitigate the bias towards recent tasks. We follow the analysis performed in Buzzega et al. (2020b) to observe the probability of each task being predicted the end of the training. For each sample in the test dataset, we take the softmax output and then
8

---

Published as a conference paper at ICLR 2022
ER
DER+
CLS-ER
1.0
ECE=32.24
ECE=16.57
ECE=6.41
0.8
Accuracy
0.6
0.2
Gap
Outputs
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Confidence
Confidence
Confidence
Figure 4: Reliability plots for different methods on S-CIFAR-10 10 with 500 buffer size. CLS-ER results in considerably better-calibrated models and hence more reliable predictions. For other buffer sizes and S-TinylmageNet see Figures S5 and S6.
average the probabilities of the associated classes for each task across the dataset. We normalize the values and report the probability of each task being predicted. Figure 3 (right plot) shows that CLS-ER able to maintain more uniform prediction probability across all the tasks over a long sequence. Figures S3 and S4 shows similar results for other buffer sizes and S- S-Tiny TinylmageNet.
6.3 MODEL CALIBRATION
Model calibration refers to the accuracy with which the scores provided by the model reflect its predictive uncertainty. The class probabilities predicted by DNNs are uncalibrated, often tending towards over-confidence which is detrimental to the reliability of the model's prediction (Guo et al., 2017). This even more pronounced CL where the models tend be biassed towards recent tasks. Following Guo al. (2017), provide the reliability diagrams (model accuracy function of its prediction confidence) and the Expected Calibration Error (ECE; a weighted average over the absolute difference between accuracy and confidence). Figure 4 shows the remarkable ability of CLS-ER to provide well-calibrated models without the application of any calibration technique.
Note that these characteristics complementary in nature: convergence flatter minima allows our method to remain the vicinity of optimal parameters for previous tasks when adapting to the new task, this leads ities, and since the model is not too biased towards the current task, the model can provide reliable prediction across the tasks which improve the calibration. Additional characteristics analyses on different datasets and buffer sizes are provided in Appendix. We observe that our model's behavior consistent across varying datasets and buffer sizes.
uniform performance across tasks which improve the task probabil-
CONCLUSION
We proposed novel dual memory experience replay method based on the complementary learning systems theory in the brain. Our method maintains long-term and short-term semantic memories which are utilized to effectively replay the neural activities of the episodic memories and align the decision boundary of the working model for efficient knowledge consolidation. We demonstrated the effectiveness of our approach on benchmark datasets as well as more challenging general incre- mental learning scenarios and achieved the new state-of-the-art in the vast majority of the continual learning settings. We further showed that CLS-ER converges to flatter minima, mitigates the bias towards recent tasks, and provides well-calibrated heghhperformacc model. Our strong empirical results motivate further study into mimicking the complementary learning system the brain more faithfully enable optimal continual learning in DNNs.

---

Published as a conference paper at ICLR 2022
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances Neural Information Processing Systems, pp. 11816- 11825, 2019. 2, 14
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Noise resource for learning in knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3129-3135 2021.
Ari S Benjamin, David Rolnick, and Konrad Kording. Measuring and regularizing networks in function space. arXiv preprint arXiv:1805.08289, 2018. 2
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe-
rience for general continual learning: strong, simple baseline. arXiv preprint arXiv:2004.072/1, 2020a. 5, 6, 7, 13, 14, 17, 20
Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. Rethinking experience replay: a bag of tricks for continual learning. arXiv preprint arXiv:2010005595, 2020b.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12):
124018, 2019. 8
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.00420,, 2018. 1
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jía, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification 2019. 13
tasks. arXiv preprint arXiv:19099.0
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773
PMLR, 2020.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning, arXiv preprint 1005v03805.09733, 2018. 6, 13, 14
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999. 2
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, pp. 1321-1330. PMLR, 2017. 9
Demis Hassabis, Dharshan Kumaran, Christopher Summerficld, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, (5(2))245-258, 2017.
Tyler Hayes, Giri Krishnan, Maxim Bazhenov, Hava Siegelmann, Terrence Sejnowski, and Christopher Kanan. Replay in deep learning: Current approaches and missing biological elements. arXiv preprint wrXiv:2104.04132, 2021. 1,3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. corr Abs/1512..0338. (2015), 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
10

---

Published as a conference paper at ICLR 2022
Nitin Kamra, Umang Gupta, and Yan Liu. Deep generative dual memory network for continual learning. arXiv preprint arXiv:1710. 10368, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 1609.04366 2016. 8
J Kirkpatrick, R Pascanu, N Rabinowitz, J Veness, G Desjardins, AA Rusu, K Milan, J Quan, Ra- malho, A BaabsskaBBariiskk. et al. Overcoming catastrophic forgetting in neural ndeworks.(dec. arXiv preprint cs.LG/1612.007966 2016. 13
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A al. Overcom-
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences,
114(13)3351-33526 2017.
Giri Krishnan, Timothy Tadros, Ramyaa Ramyaa, and Maxim Bazhenov. Biologically inspired sleep algorithm for artificial neural networks. arXiv preprint arXiv: 1782001908.02240, 2019. 2, 3
Alex Krizhevsky et Learning multiple layers of features from tiny images. 2009. 13
Dharshan Kumaran, Demis Hassabis, and James McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20 (7):512-534, 2016.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-22 222222244 1998. 13
David Lopez-Paz. and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, pp. 6467-6476, 2017. 1, 2, 13
Wolfgang Maass. Noise as resource for computation and learning in networks of spiking neurons.
Proceedings of the IEEE, 102(5):860-880, 2014.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe. Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputiigg 469:28-51, 2022. 14
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. Generalized class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion Workshops, pp. 240-241, 2020. 6, 7, 13, 20
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual
lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019. 1,2
Hadi Pouransari and Saman Ghili. Tiny imagenet visual recognition challenge. CS231N course, Stanford Univ., Stanford, CA, USA, 2015. 13
Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint 1rXiv:1810.11911 2018. 2
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, pp- 2018.
11

---

Published as a conference paper at ICLR 2022
Anthony Robins. Catastrophic forgetting in neural networks: the role of rehearsal mechanisms. In Proceedings 1993 The First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems, pp. 65-68. IEEE, 1993. 2
Mohammad Rostami, Soheil Kolouri, and Praveen K Pilly. Complementary learning for overcoming catastrophic forgetting using experience replay, arXiv preprint arXiv:1903.0453 2019. 2
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint
arXiv:1606.04671 04677600000464,00 2016. 1
Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. Knowledge distillation beyond model compres- sion. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 6136-6143. IEEE, 2021.
Dongsub Shim. Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online cass-incremmenal continual learning with adversarial shapley value. arXiv e-prints, pp. arXiv-2009, 2020. 13
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.0177 2017. 3,5
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprint 2019. 6, 13
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 1 11(1:337-57, 1985.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:/708.01544 01547,, 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987, 2017. 13
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018. 8
12

---

Published as a conference paper at ICLR 2022
A CONTINUAL LEARNING SETTINGS
There are a plethora of evaluation protocols in the CL literature, each of which biases the evaluation towards a certain approach (Farquhar & Gal, 2018; Mi et al., 2020; Shim et al., 2020; van de Ven & Tolias, 2019). It is therefore of utmost importance to conduct an extensive and robust evaluation to gauge the versatility of the method. We believe that adhering to the key desiderata suggested in Farquhar& & Gal (2018) would help the CL community immensely in moving towards a robust evaluation of methods. An experimental protocol that trains the method on long sequence of tasks where the boundaries between the tasks are not distinct and the tasks themselves are not disjoint and the method does not make sure of task boundaries during training testing can be considered as adhering to all five desiderata. Our work focuses on the aforementioned setting which can be considered General Incremental Learning (GIL) setting. Here, we provide broad categorization of these evaluation protocols which test different aspects of CL.
A.1 CLASS INCREMENTAL LEARNING (CLASS-IL)
Class-IL refers to the CL scenario where new classes added with each subsequent task and the agent must learn to distinguish not only amongst the classes within the current task but also across previous tasks. Class-II, measures how well the method can learn general representations, accumu- late, consolidate, and transfer the acquired knowledge to learn efficient representations and decision boundaries for all the classes seen so far. Following Buzzega et al. (2020a); De Lange et al. (2019); Zenke et al. (2017), we consider the common benchmark datasets MNIST (LeCun et al., 1998) (S- MNIST), CIFAR-10 (Krizhevsky et al., 2009) (S-CIFAR-10) and Chili, 2015) (S- Im-aeey- which are split into 5, 5, and 10 tasks each including 2, 2, and 20 classes respectively. These represent Class-IL settings of increasing dataset complexity as well as longer sequences. While it is important and challenging benchmark, it assumes that each subse- quent task will have the same number of disjoint classes and have uniform samples for each class which is not representative of real-world scenarios. We do not consider the related Task Increment Learning (Task-IL.) setting as it assumes the availability of task labels at both training and inference which cannot truly be considered a CL task (Farquhar & Gal, 2018).
(Pouransari &
A.2 DOMAIN INCREMENTAL LEARNING (DOMAIN-IL)
Domain-IL refers to the CL scenario where the classes remain the same in each subsequent task but the input distribution changes. We consider Rotated-MNNST (Lopez-Paz & Ranzato, 2017) (R-MNIST) where each task contains digits rotated by a fixed angle between and 180 degrees and Permuted-MNIST (Kirkpatrick et al., 2016) (P-MNIST) which applies fixed random permutation to the pixels for each task. Though we provide the results for Permuted MNIST for completion, we share the opinion by Farquhar& & Gal (2018) that it should not be considered as a benchmark dataset as it violates the cross-task resemblance desiderata and deviates from the goal of continual learning.
A.3 GENERAL INCREMENTAL LEARNING (GIL)
The aforementioned CL scenarios fail to assimilate the challenges in the real world, setting where the task boundaries are blurry and the learning agent must rather learn from a continuous stream of data where classes can reappear and have different data distributions. The CL method must deal with the issues of sample efficiency, imbalanced classes, and efficient transfer of knowledge in addition to preventing catastrophic forgetting. To test the efficacy of our method in this challenging setting, we consider two GIL evaluation protocols. MNIST-360 (Buzzega et al., 2020a) models a stream of data which presents batches of two consecutive MNIST images with each sample rotated at an increasing angle and the sequence is repeated three times. This exposes the model to both a sharp distribution shift when the class changes and a smooth rotational distribution shift. However, the number of classes in each task and the samples are uniform. The Generalized Class Incremental Learning (GCIL) (Mi et al., 2020) utilizes probabilistic modeling to sample the classes and data distributions in each task. Hence, the number of classes in each task is not fixed, the classes can overlap and the sample size for each class can vary. Following Mi et al. (2020), we use GCIL on CIFAR- (Krizhevsky et al., 2009) dataset (GCHICCIFAR-100)) set the number of samples
13

---

Published as a conference paper at ICLR 2022
and maximum number of classes per task to 1000 and 50 respectively, number of tasks 20, and evaluate on both uniform and longtail (imbalanced) sample distribution.
A.4 ONLINE CONTINUAL LEARNING
Online continual learning refers to the challenging scenario where a stream of samples is only seen and is non-iid (Mai al., 2022; Aljundi al., 2019). The common approach in the literature is to use the single-epoch protocol where the network is trained on each task in the sequence for only one epoch and there are no additional passages over data. As we aim to position CLS-ER as general incremental learning method, we are also interested in the online continual learning setting. However, similar Buzzega et al. (2020a), we also believe that the dataset complexity needs be considered when setting the number of epochs to disentangle the effect of catastrophic forgetting from underfitting and share their suggestion that future CL works should strive for realism by de- signing experimental settings which are in line with the guidelines of General Continual Learning (Farquhar & Gal. 2018) which is the goal of our study rather than adopting the single-epoch proto- coL. For the MNIST-based settings, baseline to learn the single task well. And for the more complex settings, we increase the number of epochs: 50 epochs for Sequential CIFAR-10 and Sequential Imay--e and 100 epochs for GCIL-CIFAR-100,
only epoch per task as it is sufficient for the SGD
We would also like emphasize that the experiments on MNIST based settings (S-MNIST, R- MNIST, P-MNIST, and MNIST-360) can be considered as online continual learning settings as we only train the network for epoch, and thereby the model only sees the data for each task once. CLS- ER's 's performance in these settings demonstrates its potential for the challenging online continual learning setting.
B RESERVOIR SAMPLING
Here, we provide the algorithm for the Reservoir Sampling for maintaining a fixed-size memory buffer. Reservoir sampling takes in data stream of unknown length and assigns equal probability to each sample for being represented in the memory buffer (M) with fixed budget size (B). Sampling and replacement are done at random and no priority is assigned to the samples being added or replaced from the memory buffer.
Algorithm 2 Reservoir Sampling Algorithm
Input: Memory Buffer M, Memory Budget B, Number of seen examples N, Selected example
(a,, y)
1: if > N then
Memory is not full
2:
M[N] + (I.
3: else
> Select a sample to remove
4:
V/ = random wowoddomnnneeeermmiiinnn = 0, max N)
5: if < B then
6: Mly] + (z, v)
return M
C ADDITIONAL RESULTS
In this section, we provide additional experimental results and analysis of the behavior of the model.
C.I CLS-ER COMPONENTS PERFORMANCE
CLS-ER involves the interplay between the working model and the two semantic memories: the plastic and stable models. While we use the stable model for final inference, here we provide the performance of each of these individual components to provide further insights into the workings of our method. Table SI shows the corresponding performance of the working model and plastic model for each of our experimental settings. We can see that the stable model can effectively consolidate
14

---

Published as a conference paper at ICLR 2022
Dataset
Buffer
Stable Model
Working Model
Plastic Model
200
89.54s021
89.32
89.52aa.21
S-MNIST
500
92.05
91.61
92.04
5120
95.73+0.10
95.65
95.73+0.12
200
66,192075
50,09
62.68a194
S-CIFAR-10
500
75.22+a71
63.0941.12
71.32-n.89
5120
86.78+a.17
85.00
86.77-0.17
200
23.47
9.97=0118
17.19
T-iin-Immaee
500
31.03=156
15.35=034
27.16
5120
46.7 76.341
41,3920.39
47.10x0.2
200
92.26x0.18
89.37
89,9920.4
R-MNIST
500
94.06+a.7
93.24x0.14
93.52+0.0
5120
94.25+00s
94,28+aos
94.37-m.0
200
84.6 84.63=040
84.33=04s
84.54+0.1
P-MNIST
500
38.30
88. 12
88.25+0.14
5120
92.03
91.96x006
92.02+m.s
200
66.37
55,59
60.60a141
MNIST-360
500
75.702041
72.70
75.03
1000
79.54+0.3
x0.09
79.16:044
200
33.15
31.7412.72
32.7042.7
GCIL-CIFAR-100 100 (Uniform)
500
37.01+1.6
35,89
36.18 1
1000
41,09+1.5
40.4 10.44
40.70+1.6
200
29.57
28. 294190000
29.12+3.89
GCIL-CIFAR-IE 100 (Longtail)
500
33.26s3.6
32.22a3.9
32.95a3.0
1000
39.21
1
38.84.3.52
Table S1: CLS-ER components performance analysis for each of the experimental setting.
knowledge across the tasks and therefore provide the highest mean performance for the vast majority of the settings. Figures SI and S2 further shows how the task-wise performance (on test set) of each of the component varies subsequent tasks learned. The stable model retains the performance on previous tasks while the plastic model adapts better to the recent task. Both these models provide feedback to the working model which in turn improves the plastic and stable model.
C.2 TASK PROBABILITIES
To test the effectiveness of our method in mitigating the bias towards recent tasks, we provide the task probabilities of the models trained with different buffer sizes on S-CIFAR-I0 and S-Tiny- ImageNet. Figures S3 and S4 show that CLS-ER consistently achieves more uniform task prob- abilities compared to ER and DER++ and effectively mitigates the bias towards the last task.
C.3 MODEL CALIBRATION
To further test the consistency of CLS-ER in providing well-calibrated models and the impact of the buffer size, we evaluate calibration of models trained with different buffer sizes on S-CIFAR- 10 and Imminy- Figures S5 and S6 show that CLSSER consistently provides better calibrated models compared ER and DER++ Remarkably, for both the datasets, on lower buffer sizes, the difference in Expected Calibration Error (ECE) is considerable. This demonstrates the capability of CLS-ER to train high- pech-errffmmacc and reliable models under challenging conditions.
C.4 EFFECT OF HYPERPARAMETERS
The interaction between the three components of CLS-ER is complementary. Table S3 shows how the performance of each component is affected under different hyperparameter settings. We can draw the following conclusions from the results. The performance improvement in the plastic and stable model reflected the working model and the best performance is seen in cases where both the semantic memories performing well (albeit the focus tasks is different). This highlights
15

---

Published as a conference paper at ICLR 2022
Stable Model
Working Model
Plastic Model
Buttor
162
13.03
11.3
54
23:1
00:15
11.4
261
10.00
64
60
Abe 1
19.1
54
11:11
14:11
04
20:11
4.4
Buffor
Abe
211
ake
16.1
20/3
1121
24
1443
112
16.0
112
Abe 7ak
10.0
145
11.0
10.
14.33
Figure S1: Test set task wise performance for the individual models on S-CIFAR-10 10 with different buffer sizes. The task-wise performance (x-axis) is evaluated at the end of training of each task (y-axis) to evaluate how it is affected as training progresses.
the crucial role of both memories in enabling CLS-ER to learn efficiently. For a fixed Ts value, the final performance of the stable model affected considcrably by the performance of the plastic model. The method is not highly sensitive to the particular choice of hyperparameters as different settings can attain similar performance. Because of the complementary nature of the components, can often fix of parameters (e.g. X, as, aş and rs) and only finetune the remaining parameters (e.g. r'p) which facilitates hyperparameter tuning significantly.
D COMPARISON WITH SINGLE SEMANTIC MEMORY
CLS-ER employs two semantic memories aim to mimic the fast and slow learning mechanisms in the hippocampus and neocortex respectively. Here we compare our method with a single semantic memory (Mean-ER) and Table S2 shows that while still performs admirably compared to the other CL methods, the dual semantic memories in CLS-ER provides additional performance gains especially on the complex datasets under the challenging lower memory buffer settings and has much lower variance. We attribute this to the failure of Mean-ER in maintaining the performance on both the recent and earlier tasks together i.e there is an inherent trade-off as tuning the semantic memory to adapt to the recent changes comes at the cost of performance on earlier tasks and vice versa. CL.S-RR efficiently tackles this trade-off by maintaining two specialized long-term and short- term memories. The performance of Mcan-ER, however, provides further evidence for the benefits of using consolidated information for memory replay.
Note that for fair comparison, we use the same hyperparameter search space CLS-ER for find- ing the optimal parameters for Mean-ER and report the average and std of 10 runs with different initializations using the best parameters for each setting. Table S6 provides the chosen hyperparam- eters. For inference, similar to CLS-ER, we use the EMA-weighted model (semantic memory) for Mean-ER.
16

---

Published as a conference paper at ICLR 2022
Warking
Atte Task 224
223
60
60
60
60
11.3
24
23
Buffer 200
Atter Tank
452
4:4
503
223
60
60
66
60
66
24.3
23
33
03
03
03
356
22.
12.
200
63
XTe Tack
33.8
25.8
34.3
455
543
215
60
60
66
42
27
32
33
33
03
03
03
22.
6.2
225
C3
CO
30.5
323
223
233
25.7
45-7
55.
27.5
60
84
22
33
28
26
27
5.
03
03
5.5
4.8
1.7
6.5
5.5
K2
63
ATe THlk
324
50
242
357
8.0
133
25.2
363
100
28
35
24
28
1.5
14
33
34
5.5
244
4.3
12
34
4.2
52
11.
3
Att Tas4
533
26.6
Buffer 500
363
59.5
324
2X4
8.
40.
324
462
523
203
12.7
23A
25.0
40.0
35.8
4344
553
8.6
35.7
23.4
27-7
124
337
452
31.2
12.
25
22.0
25.0
19.
534
33.0
52.
375
222
262
233
939
25.0
13.7
12.
215
13.6
AeeTaak
603
60
6.0
26
48
4.8
6.8
6.8
6-8
Ate Task
624
624
60
6.0
8.0
8.
5.4
53.3
2.8
3.8
38
125
225
92.
6.8
Buffer 5120
Aiter
45.6
54.
65
623
21.2
600
6.0
86
54.2
44.6
53.5
55.5
12-
23
32
03
02
0.
$4.
$4.0
69.6
425
KTe Tack
57.6
425
552
5X3
555
515
36.5
0.0
66
8.6
45.5
25.2
45.2
47.5
47.2
44.0
03
03
0.
$5.7
45.6
554
935
512
40
6.3
6.
Whe Tack
43.0
43.6
433
542
53:
435
537
53.7
31-3
8.
40.7
3633
360
43.7
282
375
432
442
61
03
43.
401
432
535
404
401
966
50.
34
72m
Tond 7am
Raak
Figure S2: Test set task wise performance for the individual models on S-Ty Tiny-ImageNee with dif- ferent buffer sizes. The task-wise performance (x-axis) is evaluated at the end of training of each task (y-axis) evaluate how it is affected as training progresses.
Buffer 200
Buffer 500
Buffer 5120
Teal:
Teal
Task Probabilty
0:2
CLS-ER
BERiI
CLS-ER
CLS-ER
Figure S3: Task probabilities for different methods on S-CIFAR-10 with varying memory budget.
E TRAINING AND IMPLEMENTATION DETAILS
For a fair comparison, we aim to keep the experimental settings close to the current state-of-the- art DER+ (Buzzega al., 2020a) much possible to disassociate the effect of the training
17

---

Published as a conference paper at ICLR 2022
Buffer
14
Task Probability
Fuece
ER
DER+
CLS-ER
ER
DER++
CLS-ER
ER
DER++
CLS-ER
Figure S4: Task probabilities different methods on S-Tiny-ImageNet with varying memory bud-
get.
ER
DER++
CLS-ER
ECE=44.39
ECE=20.31
ECE=6.85
0.8
Buffer 200 Accuracy
0.4
0.2
Gap Outputs
0.0
1.0
ECE-32.24
ECE-16.57
ECE=6.41
0.8
Buffer 500 Accuracy
0.6
0.4
Gap
Outouts
0.0
1.0
113.37
0.8
Buffer 5120 Accuracy
0.6
0.2
Gap
Outputs:
0.0
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.0
0.2
0.4
0.6
0.8
1.0
Confidence
Confidence
Confidence
Figure 85: Reliability plots for the different methods S-CIFAR-10 with varying memory budget.
schedule. We use the same optimizer, the number of epochs, batch size, and memory batch size as we reduce the number of epochs to 50 from 100 used by DER++ method can learn efficiently with fewer epochs, and quickly acquiring new knowledge is preferred for CL. Similar to DER++, we finetune the memory batch size for S-MNIST and MNIST- 360. We select the hyperparameters for each of the experimental setting using small validation set,
DER++. For
18

---

Published as a conference paper at ICLR 2022
ER
DER++
CLS-ER
1.0
ECE=74.54
ECE=26.60
ECE=5,27
Buffer 200 Accuracy
0.6
0.4
0.2
Gap Outputs
0.0
1.0
ECE-69.54
ECE-14.31
ECE-8.23
0.8
Buffer 500 Accuracy
0.6
0.2
Gap Outputs
1.0
55..33.31
ECE-13.16
ECE-12.34
0.8
Buffer 5120
Accuracy
0.4
0.2
Gap Outputs
0.0
0.0
1.0
Confidence
Confidence
Confidence
Figure S6: Reliability plots for the different methods on S-Tiny-ImageNet with varying memory budget.
Class-IL
Domain-ll. IL
Buffer
Method
S-MNIST
S-CIFAR-10
S-Tiny-ImageNet
R-MNIST
P-MNIST
JOINT
95.3720.24
92.20aa.15
59.99aa19
95. 76.00
94.33aa.1
SGD
19.60-oa4
19.62-pas
7.92-a26
67.66.8.5
40.70
Mean-ER
88.32-o45
61.88.249
17.68+165
92.10+1.m
83.28
200
CLS-ER
89.54
66.191077
23.472080
92.26
84.6320.00
Mcan-ER
91.79
70.40ata1
24.974080
92. 92.7840.44
87.7320.99
500
CLS-ER
92.05=0.32
75.22-071
31.03
94.06
88.30-a.14
Mean-ER
95.57=a.15
84.84.20
45.69.058
94.25
91.90sam
5120
CLS-ER
95.73.0.11
86.78.0.17
46.74.031
94.25
92.03
Table S2: Comparison of CLS-ER with Mean-ER (single semantic memory) on Class-IL and Domain-IL, settings. We report the mean and std of 10 runs with different initializations.
0's, ap (0.99, 0.999), rs. E (0, X E (0, 2J. Table S4 provides the hyperparameters used for each the experimental settings. Note that for the vast majority datasets, we use uniform settings (Ir, epochs, batch size, memory batch size, and lambda) across the different buffer sizes and requires only slight modifications in the other hyperparameters which shows that our method does require extensive finetuning for different memory budgets.
19

---

Published as a conference paper at ICLR 2022
E. CILL-CIFR--10
To test our method under challenging GIL settings that better simulate the challenges of CL in the real world, we incorporate the GCIL setting from the code provided by Mi et. (2020) with the continual dataset template class in the mammoth framework. We set the number of phases (length of task sequences) 20, with the total number of samples each phase set to 1000 and the maximum number of classes in each phase set to 50. We evaluate on both uniform and longtail (imbalanced) data distributions. Since GCIL, involves the probabilistic sampling of the classes and their samples in each phase, the random seed determines the complexity of the GCIL setting. Therefore, for reproduciblility and to gauge the stability of the methods, we fix the dataset seed to 1993 and report the average and standard deviation of 10 differently initialized models trained on the same settings.
For each of our method, we use identical training scheme (Ir=0.1, epochs=100, batch size=32 and memory batch size=32). For DER++, per the authors suggestion, performed hyperparame- ter search over a E [0.2,0.] and beta E [0.5, 1.0] with step size of 0.1. Table S5 provides the parameters chosen for each of the method under the different settings.
E.2 PERTURBATION ANALYSIS
For the perturbation analysis, we used the code and checkpoints provided by Buzzega et al. (2020a) for DER++ and ER. We would like express our gratitude to the authors for their support and for making the mammoth framework available for the research community which provides a framework for fair comparison of different CL methods under uniform experimental conditions.
20

---

Published as a conference paper at ICLR 2022
rg
PP
Stable Model
Working Model
Plastic Model
0.2
73.53
62.80
71.11=221
0.3
22.44
63.53
70.97.171
0.4
73.05
+1.8
68.75=2m0
0.5
75. 16100
63.95s192
70.42=1.0
0.1
0.6
65.04
72.82.070
69.6
0.7
23.94
23.34
70.30
0.8
24.61
62.68.aor
70.74
0.9
73.74
62.692197
69.522079
1.0
75.732008
64.21
72.00
0.3
70.26-1.79
61 61.63-101
69.31-132
0.4
71.80+117
62.64 18
70.64-1.22
0.5
70.692213
61.762061
69.65 69.65
0.6
72.4520.88
63.8720s8
71.29
0.1
0.2
0.7
71.47
61.12
70,22a224
0.8
72. 12516
62.71
70.83
0.9
22.09
63.33
71.20
1.0
72.05+135
63.74
71.01-128
0.4
46-464148
60.96-162
68.31
0.5
70.05
63,06s1226
69, $9,90_2.57
0.6
69.57.1.7
61.25
69.3621.06
0.7
68. 68.99-234
61.61
68,81 22:272
0.3
0.8
63.08z082
70.99
0.9
71.26e1.4
62.33
71. 71.03a1.56
69.00ea41
+1.38
68.69s..31
0.2
70.19+1.7
61.39x206
69.8 19.81ss 33=11.0
0.3
73. 73.72.083
62.07 20.4
18.00
0.4
71.60a2.0
61.11
0.5
74. 18
63.32s098
71.08
0.1
0.6
74.90z040
62.35
71.58
0.7
74.52a1.10
62.59 22.59
70.90
0.8 75.27+121
62.00-1.0
71.27-164
0.9
74.61
63.47 =1.00
70.49=095
1.0
76.032061
63.63 03.63
71.42.1.1
0.3
72.59
61.8 21.00
72,02
0.4
71.30
63.1
70.92=288
0.5
69.89
60.60
68,87=256
0.6 72.34
62.18
71.15
0.15
0.2
0.7 72.70
62.50
71.49-121
0.8
72.42+1.50
11.85
71.04-164
0.9
71.18
1
70.09
73.52.005 1.0
64. 194086
72.56.0.52
0.4
70.32+139
62.399.200
70.13=138
0.5
71.60
62.67
71.40
0.6
70.36
62.28
12.13
0.7
69.79.1.1
61.13
69.65s1.2
0.3
0.8
69.85 85.0
60.69.1m1
69.78-0.0
0.9
11.32
61.79
71.032161
1.0
62.352as8
71.1
Table S3: The effect of different hyperparameter settings on the individual components of CLS-ER trained on S-CIFAR-10 with 500 buffer size. For all the experiments as and ap are fixed to 0.999 and the performance is averaged over runs with different initialization.
21

---

Published as a conference paper at ICLR 2022
Memory
Dataset
Buffer
Ir
Epochs
Batch Size
Batch Size
X
ap
200
0.03
10
128
2.0
0.99
0.99
0.9
1.0
S-MNIST
500
0.1
10
32
2.0
0.99
0.99
0.9
1.0
5120
0.1
10
32
2.0
0.99
0.99
0.8
1.0
200
0.1
50
32
0.15
0.999
0.999
0.3
S-CIFAR-I0
500
0.1
50
32
0.15
0.999
0.999
0.9
5120
0.1
32
0.15
0.999
0.999
1.0
200
0.05
50
32
0.999
0.999
0.04
0.08
Smanne-ImageNe
0.05
32
0,999
0.999
0.05
0.08
$120
32
0.999
0.07
0.08
0.2
128
0.75
0,999
0.99
R-MNIST
0.2
128
0.75
0.999
0.99
1.0
5120
0.2
128
0.75
0.999
0.99
1.0
0.2
128
128
1.0
0.99
0.99
0.8
1.0
P-MNIST
500
0.2
128
128
1.0
0.99
0.99
0.8
1.0
5120
0.2
128
128
1.0
0.99
0.99
0.9
1.0
200
0.2
16
0.75
0.999
0.99
1.0
1.0
MNIST-360
500
0.2
16
32
1.25
0.99
0.99
0.9
1.0
1000
0.2
128
0.75
0.99
0.99
0.9
1.0
200
0.1
100
32
32
0.1
0.999
0.999
0.6
0.7
GCIL-CIFAR-100
500
0.1
100
32
32
0.1
0.999
0.999
0.6
0.7
1000
0.1
100
32
32
0.1
0.999
0.999
0.6
0.8
Table S4: The hyperparameters used for each of the experimental settings for CLS-ER.
Memory Batch Size
Distribution
Buffer
Ir
Epochs
Batch Size
a
B
200
0.1
100
32
32
0.2
0.5
Uniform
500
0.1
100
32
32
0.2
0.6
1000
0.1
100
32
32
0.3
0.6
200
0.1
100
0.2
0.6
Longtail
500
0.1
100
32
32
0.2
0.8
1000
0.1
100
32
32
0.3
0.9
Table S5: The hyperparameters used for DER+ on GCIL-CIFAR-100 experiments. CLS-ER uses
the same hyperparameters for both Uniform and Longtail settings (Table S4).
Memory Batch Size
Dataset
Buffer
Ir
Epochs
Batch Size
X
200
0.03
10
128
2.0
0.99
1.0
S-MNIST
500
0.1
1
10
32
2.0
0.99
1.0
5120
0.1
1
10
32
2.0
0.99
1.0
200
0.1
50
32
32
0.15
0.999
0.2
S-CIFAR-I0 10
500
0.1
50
32
0.15
0.999
0.5
5120
0.1
50
32
32
0.15
0.999
0.8
200
0.05
50
32
32
0.1
0.999
0.06
S- ImageNe
500
0.05
50
32
0.1
0.999
0.08
5120
0.05
50
32
32
0.1
0.999
0.08
200
0.2
128
128
0.75
0.999
1.0
R-MNIST
500
0.2
128
128
0.75
0.999
1.0
5120
0.2
128
128
0.75
0.999
1.0
200
0.2
128
128
1.0
0.99
0.9
P-MNIST
500
0.2
128
128
1.0
0.99
1.0
5120
0.2
128
128
1.0
0.99
0.9
Table S6: The hyperparameters used for each of the experimental settings for Mean-ER.
22


Reviews:
[{"review_id": "zmbFw7xmLV", "reviewer": "Reviewer_MUyX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is very simple to understand and mimicking the CLS theory with the hippocampus and neocortex is interesting.\n\nCompared to existing methods, is it a fair comparison in terms of training cost such as using two models including plastic and stable models and using reply data to train the working model?  \n\nI think more analysis about plastic and stable models are needed because these model act like hippocampus and neocortex based on CLS theory. What is optimal semantic? Just select the logit with the highest softmax between the two models? ", "summary_of_the_paper": "They propose a dual memory experience based on complementary learning systems (CLS).\nThe working model is updated using consistency loss with the selected optimal semantic memory from plastic and stable models.\nPlastic and stable models are updated with an exponential moving average trick (EMA) and the working model.", "summary_of_the_review": "I am not an expert in this domain and have not much experience with related works. However, the idea is very simple and reasonable and the results outperform previous existing methods. However, more analysis is needed for the claim.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ohSnL1ocUt-", "reviewer": "Reviewer_8Zhu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n1, Semantic memory is the first leveraged in incremental learning, which is interesting in my opinion. The contribution of this paper is novel.\n2, The codes are released, which guarantees the reproduction of the proposed method.\n3, The experiment results prove the effectiveness of the proposed method, which obtains impressive performance.\n\nWeaknesses:\n1, Can this method be applied to online continual learning task?\n2, The experiments for domain-IL are only evaluated in one dataset. Experiment results on more datasets should be shown in this paper.\n3, For Class-IL, some experiments with different base classes should be shown in this paper.\n", "summary_of_the_paper": "This paper proposes a novel dual memory experience replay method to store the knowledge of previous tasks. In addition, long-term and short-term semantic memories are leveraged to replay the neural activities of the episodic memories and align the decision boundary. ", "summary_of_the_review": "In my opinion, the contribution of the proposed method is novel and interesting. In addition, the writing of this paper is clear and easy to understand. The experiments are sufficient to prove the effectiveness of the proposed methods. However, more experiments should be designed to prove the proposed method further.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EcHegK6X3sj", "reviewer": "Reviewer_M4Au", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The proposed method is well motivated and novel in the context of continual learning. Maintaining multiple models for different purposes is interesting and is shown to be helpful for continual learning.\n\n2. The writing and organization is clear. It is easy to understand the proposed algorithm and the details.\n\n3. Different continual learning settings are considered, the proposed approach shows better results across all the settings. The analysis of different models also make it easy to understand how the method works.\n\n\nWeaknesses:\n\n1. One of the main drawbacks of the proposed approach is that three models need to be maintained during training which incur a heavy cost.   One question is that if reducing the individual model size while keeping the whole model size the same, what's the performance in this case?\n\n2. The proposed method has a lot of hyperparameters to tune, which may be a problem if the method is applied to a different task. \n\n\nMore questions:\n\n1. Why update the plastic and stable models stochastically?\n\n2. In table 1, why CLS-ER is better than JOINT on S-MNIST? JOINT should be the upper bound performance?\n\n3. The authors make a connection to flat minima, if explicitly enforcing flat minima, would the proposed method can still do better?", "summary_of_the_paper": "In this paper, motivated from the CLS theory in neuroscience that efficient learning requires short-term adaptation and slow learning of structured information, the authors propose a novel dual memory experience replay method for continual learning. The idea is to build  long-term and short-term semantic memories by maintaining another two models: plastic model and stable model. The plastic model is used for fast learning of recent experiences and the stable model is used for slow learning of structural knowledge. The two models are updated with a Mean Teacher fashion during training with different frequencies. The proposed CLS-ER is  task-agnostic and can be applied in different continual learning settings. The experimental results show that the proposed CLS-ER outperforms several baselines for continual learning. ", "summary_of_the_review": "In this paper, a novel idea of continual learning is proposed. The authors clearly describe the proposed approach and also give a detailed analysis. Overall, it is a well-written paper with an interesting idea. The results also show the benefits of the proposed approach. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4xDO5MYo4vC", "reviewer": "Reviewer_azqK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this publication. I'm not very familiar with the latest CL methods but the authors address various related works and evaluate all models thoroughly. I think the setting without task boundaries is harder and more interesting since the real world also doesn't come with clear task boundaries. The paper is following recent work towards robust and extensive evaluation which is more difficult but will help the CL community to make real progress. \n\nThe empirical upper bound \"JOINT\" is very instructive and highlights that there is still room for progress. That said, the JOIN performance seems low given the excellent performance of other models on MNIST, CIFAR, Imagenet datasets. Is this because the models are very small? Does CLS-ER work also with larger models?\n\nThe flat mima discussion is less essential but is probably missing a reference to Flat Minima, Hochreiter et al 1997.\n\nInstead of the interesting section 6 I'd have preferred some ablation experiments (though the component analysis is appreciated). E.g. are two teacher networks necessary? How much worse is the model if one teacher network is used (with a tuned rate)? What if there were three networks (each with their own rate)? \n\nI did not understand how samples are removed from the buffer (the episodic memory). Are samples removed eventually? If yes, how and where is that explained? If not, what is the space complexity of algorithm 1? ", "summary_of_the_paper": "The paper presents CLS-ER, a dual memory mechanism for the continual learning setting. During training the model receives data from a non-i.i.d. source as well as random samples from a sample buffer (the episodic memory). The stable and plastic models act as teacher models. Their logits are the target of the student model. Which one is used as a target depends on the highest softmax score w.r.t. a ground trough class. The teacher networks are both exponential moving averages of the previous student models. Thus, the only difference between the teacher models \"plastic\" and \"stable\" is their EMA rates which in the case of the plastic model is larger such that it focuses on more recent weight updates.  The slower learning of the stable model is the mechanism which ought to mirror the consolidation of structural knowledge. It is used from inference Samples from the datastream are transferred to the sample buffer using reservoir sampling. \n\nThe authors experiment on image datasets (variants of MNIST, CIFAR10, Tiny-ImageNet, see the Mammoth framework) with data augmentation (random flips and crops). They compare with several prio works and the evaluation is thorough and convincing. ", "summary_of_the_review": "I think this work is good in general. The writing is great, the evaluation seems thorough, and comparisons look fair. The method is easy to understand and the empirical results are convincing to me. The authors discuss recent and related works and their method seems original but I lack the expertise to judge the novelty of the approach. It builds significantly on previous work (DER) so I'd argue that some aspects of the publication exist in previous work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript proposes an experience replay method that supports two time scales of memory, as in complementary learning systems from the cognitive sciences literature. The authors demonstrate that their method on a wide range of benchmarks and after the rebuttal demonstrate it on one additional benchmark.
The reviewers raised a number of questions and concerns around additional experiments (benchmarks and ablations), online training, the cost of training, number of hyperparameters, further analysis, clarifications, and citations. The authors address the majority of these concerns during the rebuttal period, and overall the reviewers were in favour of acceptance. Therefore, I recommend acceptance of this work.
Decision: Accept (Poster)