MENTARY LEARNING SYSTEM
Year: 2022

Abstract:
Humans excel at continually learning from an ever-changing environment whereas it remains a challenge for deep neural networks which exhibit catastrophic forgetting. The complementary learning system (CLS) theory suggests that the interplay between rapid instance-based learning and slow structured learning in the brain is crucial for accumulating and retaining knowledge. Here, we propose CLS-ER, a novel dual memory experience replay (ER) method which maintains short-term and long-term semantic memories that interact with the episodic memory. Our method employs an effective replay mechanism whereby new knowledge is acquired while aligning the decision boundaries with the semantic memories. CLSER does not utilize the task boundaries or make any assumption about the distribution of the data which makes it versatile and suited for “general continual learning”. Our approach achieves state-of-the-art performance on standard benchmarks as well as more realistic general continual learning settings.

heading: 1 INTRODUCTION
text: Continual learning (CL) refers to the ability of a learning agent to continuously interact with a dynamic environment and process a stream of information to acquire new knowledge while consolidating and retaining previously obtained knowledge (Parisi et al., 2019). This ability to continuously learn from a changing environment is a hallmark of intelligence and a critical missing component in our quest towards making our models truly intelligent. The major challenge towards enabling CL in deep neural networks (DNNs) is that the continual acquisition of incrementally available information from non-stationary data distributions generally leads to catastrophic forgetting whereby the performance of the model on previously learned tasks drops drastically as it learns new tasks (McCloskey & Cohen, 1989).
Several approaches have been proposed to address the issue of catastrophic forgetting in CL. These can be broadly categorized into regularization-based methods (Farajtabar et al., 2020; Kirkpatrick et al., 2017; Ritter et al., 2018; Zenke et al., 2017) which penalizes changes in the network weights, network expansion-based methods (Rusu et al., 2016; Yoon et al., 2017) which dedicate a distinct set of network parameters to distinct tasks, and rehearsal-based methods (Chaudhry et al., 2018; Lopez-Paz & Ranzato, 2017) which maintains a memory buffer and replays samples from previous tasks. Amongst these, rehearsal-based methods have proven to be more effective in challenging CL tasks (Farquhar & Gal, 2018). However, an optimal approach for replaying memory samples and constraining the model update to efficiently consolidate knowledge remains an open question.
In the brain, the ability to continually acquire, consolidate, and transfer knowledge over time is mediated by a rich set of neurophysiological processing principles (Parisi et al., 2019; Zenke et al., 2017) and multiple memory systems (Hassabis et al., 2017). In particular, the CLS theory (Kumaran et al., 2016) posits that efficient learning requires two complementary learning systems: the hippocampus exhibits short-term adaptation and rapid learning of episodic information which is then gradually consolidated to the neocortex for slow learning of structured information. Furthermore, a recent study by Hayes et al. (2021) identified the missing elements of biological reply in the replay
∗Contributed equally.
mechanisms employed in DNNs for CL. They highlight that many existing approaches only focus on modeling the prefrontal cortex directly and do not have a fast learning network which plays a critical role in enabling efficient CL in the brain. Inspired by these studies, we hypothesize that mimicking the slow and rapid adaptation of information and having an efficient mechanism for incorporating them into the working memory can enable better CL in DNNs.
To this end, we propose a novel dual memory experience replay method based on the complementary learning systems theory in the brain, dubbed as CLS-ER. In addition to a small episodic memory, our method builds long-term and short-term semantic memories which mimic the rapid and slow adaptation of information (Figure 1). As the network weights encode the learned representations of the tasks (Krishnan et al., 2019), the semantic memories are maintained by taking the exponential moving average of the working model’s weights to consolidate information across the tasks with varying time windows and frequencies. The semantic memories interact with the episodic memory to extract consolidated replay activation patterns and enforce a consistency loss on the update of the working model so that new knowledge is acquired while aligning the decision boundary of the working model with the decision boundaries of semantic memories. This maintains a balance between the plasticity and stability of the model for effective knowledge consolidation.
CLS-ER provides a general CL method that does not utilize the task boundaries or make any strong assumption regarding the distribution of the data and tasks. We demonstrate the versatility and effectiveness of our method on a wide range of CL benchmark tasks as well as more challenging scenarios which simulate the complexities of CL in the real world.

heading: 2 RELATED WORK
text: The base method for the rehearsal-based approach, Experience Replay (ER) (Riemer et al., 2018) combines the memory samples with the task samples into the training batch. Several techniques have since been employed on top of ER. Meta Experience Replay (MER) (Riemer et al., 2018) considers replay as a meta-learning problem for maximizing the transfer from previous tasks and minimizing the interference. iCARL (Rebuffi et al., 2017) uses the nearest average representation of past exemplars to classify in an incrementally learned representation space. Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) formulates optimization constraints on the exemplars in memory. Gradient-based Sample Selection (GSS) (Aljundi et al., 2019) aims for memory sample diversity in the gradient space and provides a greedy selection approach. Function Distance Regularization (FDR) (Benjamin et al., 2018) saves the network response at the task boundaries and adds a consistency loss on top of ER. Dark Experience Replay (DER++) relaxes this restriction and samples logits during the entire optimization trajectory.
CLS has been used as a source of inspiration for dual memory learning systems in earlier works (French, 1999; Robins, 1993) but they have not been shown to scale to current computer vision tasks (Parisi et al., 2019). Recently, Rostami et al. (2019) utilizes a generative model to couple sequential tasks in a latent embedding space. Kamra et al. (2017) utilizes two generative models
in a dual memory architecture. However, they utilize the task boundaries and generative replay has its own set of challenges as it is difficult to learn a faithful distribution and performs sub-par in comparison to instance-based replay methods on challenging CL settings. Generally, the inspiration from CLS theory in DNNs has been mostly limited to episodic memory and mimicking the rapid and slow learning mechanism is majorly ignored (Hayes et al., 2021) which we aim to address.

heading: 3 METHOD
text: We first provide an overview of the CLS theory for the brain and how we aim to mimic it for DNNs before introducing the main components of our method and the overall formulation.

heading: 3.1 COMPLEMENTARY LEARNING SYSTEM THEORY
text: The CLS theory posits that effective lifelong learning in the brain requires two complementary learning systems. The hippocampus rapidly encodes novel information as a short-term memory which is subsequently used to transfer and consolidate knowledge in the neocortex which gradually acquires structured knowledge representation as long-term memory through experience replay. The interplay between the functionality of the hippocampus and neocortex is crucial for concurrently learning efficient representations (for better generalization) and the specifics of instance-based episodic memory.

heading: 3.2 COMPLEMENTARY LEARNING SYSTEM BASED EXPERIENCED REPLAY
text: Inspired by the CLS theory, we propose a dual memory experience replay method, CLS-ER, which aims to mimic the interplay between fast learning and slow learning mechanisms for enabling effective CL in DNNs. Our method maintains short-term and long-term semantic memories of the encountered tasks which interact with the episodic memory for replaying the associated neural activities. The working model is updated so that it acquires new knowledge while aligning its decision boundary with the semantic memories to enable the consolidation of structured knowledge across the tasks. Figure 1 highlights the parallels between CLS theory and our method.
Semantic Memories: Central to our method is the maintenance of two semantic memories which accumulate and consolidate information over long-term and short-term periods. As the acquired knowledge of the learned tasks is encoded in the weights of DNNs (Krishnan et al., 2019), we aim to form our semantic memories by accumulating the knowledge encoded in the corresponding weights of the model as it sequentially learns different tasks.
An efficient method for aggregating the weights of a model is provided by Mean Teacher (Tarvainen & Valpola, 2017) which is a knowledge distillation approach that uses an exponential moving average (EMA) of the student’s weights during training as a teacher for semi-supervised learning. It can also be considered as forming a self-ensemble of the intermediate model states that leads to better
internal representations. We adapt the Mean Teacher approach to build our semantic memories as it provides a computational and memory-efficient method for accumulating knowledge over the tasks.
As CL involves learning tasks sequentially, the model weights at each training step can be considered as a student model specialized for a particular task. Therefore, averaging the weights during training can be considered as forming an ensemble of task-specific student models which effectively aggregates information across the tasks and leads to smoother decision boundaries. CLS-ER builds long-term (stable model) and short-term (plastic model) semantic memories by maintaining two EMA-weighted models over the working model’s weights. The stable model is updated less frequently with a larger window size so that it retains more information from the earlier tasks while the plastic model is updated more frequently with a smaller window size so that it adapts faster to information from new tasks (Figure 2). Section D further demonstrates the benefits of employing two semantic memories instead of a single semantic memory.
Episodic Memory: Replay of samples from the previous tasks stored in a small episodic memory is a common approach in CL that has proven to be effective in mitigating catastrophic forgetting. As we aim to position CLS-ER as a versatile general incremental learning method, we do not utilize the task boundaries or make any strong assumptions about the distribution of the tasks or samples. Therefore, to maintain a fixed episodic memory buffer, we employ reservoir sampling (Vitter, 1985) which assigns equal probability to each sample in the stream for being represented in the buffer and randomly replaces the existing memory samples (Algorithm 2). It is a global distribution matching strategy that ensures that at any given time the distribution of samples in the buffer will approximately match the distribution of all the samples seen so far (Isele & Cosgun, 2018).
Consolidation of Information: The key challenge in CL is the consolidation of new information with the previously acquired information. This requires an effective balance between the stability and plasticity of the model. Furthermore, the sharp change in decision boundary as a new task is learned makes the consolidation of information over tasks more challenging. CLS-ER tackles these challenges through a novel dual memory experience replay mechanism. The long-term and shortterm semantic memories interact with the episodic memory to extract the consolidated activations for the memory samples which are then utilized to constrain the update of the working model so that new knowledge is obtained whilst the decision boundary is aligned with the semantic memories. This prevents rapid changes in the parameter space as new tasks are learned. Furthermore, aligning the working model’s decision boundary with the semantic memories serves two goals: (i) helps in retaining and consolidating information and (ii) leads to a smoother adaptation of decision boundary.

heading: 3.3 FORMULATION
text: CLS-ER involves training a working model f(.; θw) on a data stream D sampled from a non-iid distribution. Two additional EMA-weighted models are maintained as semantic memories: plastic model f(.; θP ) and the stable model f(.; θS). Finally, reservoir sampling (Vitter, 1985) is employed to maintain a small episodic memoryM. At each training step, the working model receives the training batch Xb from the data stream and retrieves a random batch of exemplars Xm from the episodic memory. This is then followed by the retrieval of optimal semantic information, i.e. the structural knowledge encoded in the semantic memories which account for the consolidation of feature space and adaptation of the decision boundaries of the previous tasks. The semantic memories are designed so that the plastic model has higher performance on recent tasks whereas the stable model prioritizes retaining information on the older tasks. Therefore, we would prefer to use the logits from the stable model ZS for older exemplars and the plastic model ZP for recent exemplars. As CLS-ER is a general incremental learning method, instead of using a hard threshold or task information, we opt for a simple task-agnostic approach of using the performance of the semantic memories on the exemplars as a selection criterion that empirically works well. For each exemplar, we select the replay logits Z based on which model has the highest softmax score for the ground-truth class (line 5-6 in Algorithm 1).
The selected replay logits from the semantic memories are then used to enforce a consistency loss on the working model so that it does not deviate from the already learned experiences. Hence, the working model is updated with a combination of the cross-entropy loss on the union of the data stream and episodic memory samples, X , and the consistency loss on the exemplars Xm,
L = LCE(σ(f(X; θW )), Y ) + λLMSE(f(Xm; θW ), Z) (1)
Algorithm 1 Complementary Learning System-Experience Replay Algorithm Input: Data stream D, Learning rate η, Consistency weight λ, Update rates rP and rS , Decay parameters αP and αS Initialize: θW = θP = θS M←− {}
1: while Training do 2: (Xb, Yb) ∼ D 3: (Xm, Ym) ∼M 4: (X,Y ) = {(Xb, Yb), (Xm, Ym)} 5: ZP , ZS ←− f(Xm; θP ), f(Xm; θS) . Select optimal semantic memory 6: Z ←− ZP if σ(ZP )(Ym) > σ(ZS)(Ym) else ZS 7: L = LCE(σ(f(X; θW )), Y ) + λLMSE(f(Xm; θW ), Z) . Update working model 8: θW ←− θW − η∇θWL 9: a, b ∼ U(0, 1) . Update semantic memories 10: θP ←− αpθP + (1− αP )θW if a < rP else θP 11: θS ←− αSθS + (1− αS)θW if b < rS else θS 12: M←− Reservoir(M, (Xb, Yb)) . Update episodic memory (Algorithm 2)
return θW , θP , θS
where σ is the softmax function, λ the regularization parameter, and LMSE the mean squared error loss used as consistency term.
After updating the working model, we stochastically update the plastic and stable models with rates rP and rS (note that rP ≥ rS so that the plastic model is updated more frequently). Using a stochastic rather than a deterministic approach is more biologically plausible (Maass, 2014) which reduces the overlap in the snapshots of the working model and leads to more diversity in semantic memories. The semantic memories are updated by taking an exponential moving average of the working model’s weights (Tarvainen & Valpola, 2017) with decay parameters αP and αS , respectively.
θi = αiθi + (1− αi)θW , i ∈ {P, S} (2) Note that αP ≤ αS so that the plastic model mimics the rapid adaptation of information while the stable model mimics slow acquisition of structured knowledge.
For inference, we use the stable model as it retains long-term memory across the tasks, consolidates structural knowledge, and learns efficient representations for generalization (Figure 1). Details of our proposed method are provided in Algorithm 1.

heading: 4 EXPERIMENTAL SETUP
text: To ensure a fair comparison of different CL methods under uniform experimental settings, we extended the Mammoth framework (Buzzega et al., 2020a) and unless stated otherwise, we follow the same training scheme (learning rate, batch sizes of incoming data and memory buffer, and the number of training epochs) as them for each of the evaluation settings. To find the optimal hyperparameters for CLS-ER, we run a grid search over λ, αS , αP , rS , and rP on a small validation set. Sections C.4 and E show that our method is not highly sensitive to the particular choice of hyperparameters and different settings can attain similar performance. Also, because of the complementary nature of the components, we can often fix a set of parameters (e.g. λ, αS , αP and rS) and only finetune the remaining parameters (e.g. rP ) which facilitates hyperparameter tuning significantly.
Following Buzzega et al. (2020a), we employ a fully connected network with two hidden layers, each with 100 ReLU units on all the variants of the MNIST dataset and ResNet-18 (He et al., 2015) network without pretraining for the other datasets. In all of our settings, we use the Stochastic Gradient Descent optimizer. We use random horizontal flip and random crop on both the stream and buffer samples for S-CIFAR-10, S-Tiny-ImageNet, and GCIL-CIFAR-100. The selected hyperparameters for each of the settings are provided in Table S4. Note that for the vast majority of datasets, we use uniform settings (lr, epochs, batch size, memory batch size, and lambda) across the different buffer sizes and requires only slight modifications in the other hyperparameters which shows that our method does not require extensive finetuning for different memory budgets. For each of our

heading: Buffer Method Class-IL Domain-ILS-MNIST S-CIFAR-10 S-Tiny-ImageNet R-MNIST P-MNIST
text: – JOINT 95.57±0.24 92.20±0.15 59.99±0.19 95.76±0.04 94.33±0.17SGD 19.60±0.04 19.62±0.05 7.92±0.26 67.66±8.53 40.70±2.33
200
ER 80.43±1.89 44.79±1.86 8.49±0.16 85.01±1.90 72.37±0.87 GEM 80.11±1.54 25.54±0.76 - 80.80±1.15 66.93±1.25 iCaRL 70.51±0.53 49.02±3.20 7.53±0.79 - - FDR 79.43±3.26 30.91±2.74 8.70±0.19 85.22±3.35 74.77±0.83 GSS 38.92±2.49 39.07±5.59 - 79.50±0.41 63.72±0.70 DER++ 85.61±1.40 64.88±1.17 10.96±1.17 90.43±1.87 83.58±0.59 CLS-ER 89.54±0.21 66.19±0.75 21.95±0.26 92.26±0.18 84.63±0.40
500
ER 86.12±1.89 57.74±0.27 9.99±0.29 88.91±1.44 80.60±0.86 GEM 85.99±1.35 26.20±1.26 - 81.15±1.98 76.88±0.52 iCaRL 70.10±1.08 47.55±3.95 9.38±1.53 - - FDR 85.87±4.04 28.71±3.23 10.54±0.21 89.67±1.63 83.18±0.53 GSS 49.76±4.73 49.73±4.78 - 81.58±0.58 76.00±0.87 DER++ 91.00±1.49 72.70±1.36 19.38±1.41 92.77±1.05 88.21±0.39 CLS-ER 92.05±0.32 75.22±0.71 29.61±0.54 94.06±0.07 88.30±0.14
5120
ER 93.40±1.29 19.70±0.07 28.97±0.41 94.19±0.44 89.90±0.13 GEM 95.11±0.87 67.27±4.27 - 85.24±0.59 87.42±0.95 iCaRL 70.60±1.03 55.07±1.55 14.08±1.92 - - FDR 87.47±3.15 19.70±0.07 28.97±0.41 94.19±0.44 90.87±0.16 GSS 89.39±0.75 67.27±4.27 - 85.24±0.59 82.22±1.14 DER++ 95.30±1.20 85.24±0.49 39.02±0.97 94.65±0.33 92.26±0.17 CLS-ER 95.73±0.11 86.78±0.17 45.92±0.30 94.25±0.06 92.03±0.05
Table 1: Comparison with prior works on Class-IL and Domain-IL settings. The baseline results are from Buzzega et al. (2020a) (- indicates the experiments that the authors were unable to run).
experiments, we fix the order of the classes and report the average and one standard deviation of the mean test accuracy of all the tasks across 10 runs with different initializations. Section E provides further implementation details.

heading: 5 EMPIRICAL EVALUATION
text: There are a plethora of evaluation protocols in the CL literature, each of which biases the evaluation towards a certain approach (Farquhar & Gal, 2018; Mi et al., 2020; van de Ven & Tolias, 2019). It is therefore of utmost importance to conduct an extensive and robust evaluation over different CL settings to gauge the versatility of the method. Details of the datasets used in each CL setting are provided in Section A. We compare our method with the state-of-the-art rehearsal-based approaches on various CL settings and memory budgets under uniform experimental settings. SGD refers to standard training and JOINT provides an upper bound given by training all tasks jointly.
Class Incremental Learning (Class-IL): refers to the CL scenario where new classes are added with each subsequent task and the agent must learn to distinguish not only amongst the classes within the current task but also across previous tasks. Class-IL measures how well the method can learn general representations, accumulate, consolidate, and transfer the acquired knowledge to learn efficient representations and decision boundaries for all the classes seen so far.
Table 1 provides the comparison with six rehearsal-based approaches on Class-IL settings with varying datasets and task length complexities. CLS-ER provides the highest performance in all of these scenarios. In particular, as the dataset complexity and number of tasks increase from S-MNIST to S-Tiny-ImageNet, the performance gap between CLS-ER and DER++ increases considerably. Especially, with a smaller memory budget, CLS-ER is able to retain more information than other methods. In the most challenging setting, S-Tiny-ImageNet with 200 buffer size, CLS-ER provides a percentage gain of 163% and 104% over the baseline ER and the current state-of-the-art DER++, respectively. The results demonstrate the capability of CLS-ER to efficiently accumulate and retain knowledge over longer sequences under complex and memory restrictive scenarios.
We believe that the performance gains over DER++ highlight a key component of an efficient CL agent: the ability to consolidate previously acquired knowledge. DER++ fails to account for the consolidation of feature space and adaptation of the decision boundaries of the previous tasks. Therefore, constraining the model to match the sub-optimal logits might hamper the consolidation of knowledge. This becomes more prominent as the number of classes in each task, the sequence length, and the cross-task resemblance increase. For instance, for DER++, replaying a sample from Task-1 when training on S-Tiny-ImageNet Task-10, the reference logit values which are used to enforce the consistency are from a model representation state which has not considered how to distinguish the 20 classes in Task-1 from 80 additional classes which are visually and semantically similar. It stands to reason that the optimal representation space and subsequently the decision boundaries for the classes in Task-1 would drift considerably when required to distinguish between 80 additional classes as well. Therefore, the local information provided by the sub-optimal saved logits in DER++ fails to provide the global context required for consolidating knowledge. CLS-ER, on the other hand, extracts logits from the semantic memories which consolidate knowledge across the tasks, and hence the working model receives more optimal feedback.
Domain Incremental Learning (Domain-IL): refers to the CL scenario where the classes remain the same in subsequent tasks but the input distribution changes. We consider R-MNIST where each task contains digits rotated by a fixed angle and P-MNIST which applies a fixed random permutation to the pixels for each task. Table 1 shows that CLS-ER provides generalization gains under both settings, particularly for lower memory budget, and performs on par with DER++ on 5120 buffer size. We attribute this to the consolidated soft targets from the semantic memories which provide relational information about the classes from a global context compared to the local information in DER++. This enables our method to maintain the similarity structure across sequences effectively.
General Incremental Learning (GIL): Class-IL and Domain-IL fail to assimilate the challenges in the real-world setting where the task boundaries are blurry, and classes can reappear and have different distributions. The CL method has to consider the sample efficiency, challenge of imbalanced data, and efficient knowledge transfer in addition to preventing catastrophic forgetting. We consider two GIL settings: MNIST-360 (Buzzega et al., 2020a) exposes the model to both sharp (changes in class) and smooth (rotation of digits) distribution shifts. This requires the CL method to tackle the challenges of class-IL as well as domain-IL. The Generalized Class Incremental Learning (GCIL) (Mi et al., 2020) is the closest to the real-world scenario as it utilizes probabilistic modeling to sample the classes and data distributions in each task. The number of classes in each task is not fixed, the classes can overlap and the sample size for each class can vary.
Table 2 shows that CLS-ER provides considerable performance gains on the challenging MNIST360, particularly with a low memory budget. Similarly, Table 3 demonstrates the effectiveness of CLS-ER on GCIL-CIFAR-100 under both uniform and imbalanced class samples. Both of these settings involve recurring classes in subsequent sequences which makes the transfer of knowledge from previous occurrences important. The performance gap between CLS-ER and DER++ in the recurring classes setting alludes to another shortcoming of saving logits from the previous state.
Consider the case where class c appears in sequence (Seq)-1 with 20 samples, and then subsequently in Seq-5 with 200 samples. In the following sequences, DER++ uses exemplars from class c saved in Seq-1 with sub-optimal logits from the model state which was attained with only 20 samples and fails to take advantage of the better learned representations with additional data in Seq-5. CLS-ER, on the other hand, is able to take advantage of the additional samples and provide feedback from the improved learned representations.
Furthermore, note that the MNIST-based settings can be considered under the online CL setting (see section A.4) as we only pass through the data once for each task and the performance of CLS-ER on these settings demonstrates its potential as an efficient method for online CL.

heading: 6 MODEL CHARACTERISTICS
text: We analyze CLS-ER and provide some insights into the characteristics of the proposed approach which enables it to learn effectively under challenging CL scenarios. In our subsequent analyses, we compare CLS-ER with the baseline ER and DER++.

heading: 6.1 CONVERGENCE TO FLATTER MINIMA
text: Due to the non-convexity of the loss landscape, there can be multiple solutions to the optimization objective, however, the local geometry at the convergence point can affect the generalization of the model. Solutions that reside in wide valleys instead of narrow crevices generalize better (Chaudhari et al., 2019; Hochreiter & Schmidhuber, 1997; Keskar et al., 2016) as the predictions do not change drastically with small perturbations. A CL model which converges to flatter minima has more flexibility to explore the neighboring parameter space to optimize on the new task without drastically increasing the loss on the previous tasks. Following the analysis in Zhang et al. (2018), we add independent Gaussian noise of increasing strength to the parameters of the trained model and analyze the change in accuracy and loss across the training samples. Figure 3 shows that CLS-ER is significantly less sensitive to perturbations compared to ER and DER++. CLS-ER also retains performance for a longer period and its performance drops more smoothly. These results suggest that the fast and slow adaptation of information in CLS-ER can guide the optimization to wider valleys.

heading: 6.2 TASK PROBABILITIES
text: Because of the sequential nature of CL, an implicit bias is induced towards the current task (Wu et al., 2019). A number of CL methods employ explicit techniques to reduce this bias (Hou et al., 2019; Wu et al., 2019), however, they utilize the task boundaries which is counterproductive for general incremental learning. We believe that the efficient knowledge consolidation in CLS-ER
through the semantic memories can implicitly mitigate the bias towards recent tasks. We follow the analysis performed in Buzzega et al. (2020b) to observe the probability of each task being predicted at the end of the training. For each sample in the test dataset, we take the softmax output and then average the probabilities of the associated classes for each task across the dataset. We normalize the values and report the probability of each task being predicted. Figure 3 (right plot) shows that CLS-ER is able to maintain a more uniform prediction probability across all the tasks over a long sequence. Figures S3 and S4 shows similar results for other buffer sizes and S-TinyImageNet.

heading: 6.3 MODEL CALIBRATION
text: Model calibration refers to the accuracy with which the scores provided by the model reflect its predictive uncertainty. The class probabilities predicted by DNNs are uncalibrated, often tending towards over-confidence which is detrimental to the reliability of the model’s prediction (Guo et al., 2017). This is even more pronounced in CL where the models tend to be biassed towards recent tasks. Following Guo et al. (2017), we provide the reliability diagrams (model accuracy as a function of its prediction confidence) and the Expected Calibration Error (ECE; a weighted average over the absolute difference between accuracy and confidence). Figure 4 shows the remarkable ability of CLS-ER to provide well-calibrated models without the application of any calibration technique.
Note that these characteristics are complementary in nature: convergence to flatter minima allows our method to remain in the vicinity of optimal parameters for previous tasks when adapting to the new task, this leads to more uniform performance across tasks which can improve the task probabilities, and since the model is not too biased towards the current task, the model can provide reliable prediction across the tasks which improve the calibration. Additional characteristics analyses on different datasets and buffer sizes are provided in Appendix. We observe that our model’s behavior is consistent across varying datasets and buffer sizes.

heading: 7 CONCLUSION
text: We proposed a novel dual memory experience replay method based on the complementary learning systems theory in the brain. Our method maintains long-term and short-term semantic memories which are utilized to effectively replay the neural activities of the episodic memories and align the decision boundary of the working model for efficient knowledge consolidation. We demonstrated the effectiveness of our approach on benchmark datasets as well as more challenging general incremental learning scenarios and achieved the new state-of-the-art in the vast majority of the continual learning settings. We further showed that CLS-ER converges to flatter minima, mitigates the bias towards recent tasks, and provides a well-calibrated high-performance model. Our strong empirical results motivate further study into mimicking the complementary learning system in the brain more faithfully to enable optimal continual learning in DNNs.

heading: A CONTINUAL LEARNING SETTINGS
text: There are a plethora of evaluation protocols in the CL literature, each of which biases the evaluation towards a certain approach (Farquhar & Gal, 2018; Mi et al., 2020; Shim et al., 2020; van de Ven & Tolias, 2019). It is therefore of utmost importance to conduct an extensive and robust evaluation to gauge the versatility of the method. We believe that adhering to the key desiderata as suggested in Farquhar & Gal (2018) would help the CL community immensely in moving towards a robust evaluation of methods. An experimental protocol that trains the method on a long sequence of tasks where the boundaries between the tasks are not distinct and the tasks themselves are not disjoint and the method does not make sure of task boundaries during training or testing can be considered as adhering to all five desiderata. Our work focuses on the aforementioned setting which can be considered as General Incremental Learning (GIL) setting. Here, we provide a broad categorization of these evaluation protocols which test different aspects of CL.

heading: A.1 CLASS INCREMENTAL LEARNING (CLASS-IL)
text: Class-IL refers to the CL scenario where new classes are added with each subsequent task and the agent must learn to distinguish not only amongst the classes within the current task but also across previous tasks. Class-IL measures how well the method can learn general representations, accumulate, consolidate, and transfer the acquired knowledge to learn efficient representations and decision boundaries for all the classes seen so far. Following Buzzega et al. (2020a); De Lange et al. (2019); Zenke et al. (2017), we consider the common benchmark datasets MNIST (LeCun et al., 1998) (SMNIST), CIFAR-10 (Krizhevsky et al., 2009) (S-CIFAR-10) and Tiny-ImageNet (Pouransari & Ghili, 2015) (S-Tiny-ImageNet) which are split into 5, 5, and 10 tasks each including 2, 2, and 20 classes respectively. These represent Class-IL settings of increasing dataset complexity as well as longer sequences. While it is an important and challenging benchmark, it assumes that each subsequent task will have the same number of disjoint classes and have uniform samples for each class which is not representative of real-world scenarios. We do not consider the related Task Increment Learning (Task-IL) setting as it assumes the availability of task labels at both training and inference which cannot truly be considered as a CL task (Farquhar & Gal, 2018).

heading: A.2 DOMAIN INCREMENTAL LEARNING (DOMAIN-IL)
text: Domain-IL refers to the CL scenario where the classes remain the same in each subsequent task but the input distribution changes. We consider Rotated-MNIST (Lopez-Paz & Ranzato, 2017) (R-MNIST) where each task contains digits rotated by a fixed angle between 0 and 180 degrees and Permuted-MNIST (Kirkpatrick et al., 2016) (P-MNIST) which applies a fixed random permutation to the pixels for each task. Though we provide the results for Permuted MNIST for completion, we share the opinion by Farquhar & Gal (2018) that it should not be considered as a benchmark dataset as it violates the cross-task resemblance desiderata and deviates from the goal of continual learning.

heading: A.3 GENERAL INCREMENTAL LEARNING (GIL)
text: The aforementioned CL scenarios fail to assimilate the challenges in the real world, setting where the task boundaries are blurry and the learning agent must rather learn from a continuous stream of data where classes can reappear and have different data distributions. The CL method must deal with the issues of sample efficiency, imbalanced classes, and efficient transfer of knowledge in addition to preventing catastrophic forgetting. To test the efficacy of our method in this challenging setting, we consider two GIL evaluation protocols. MNIST-360 (Buzzega et al., 2020a) models a stream of data which presents batches of two consecutive MNIST images with each sample rotated at an increasing angle and the sequence is repeated three times. This exposes the model to both a sharp distribution shift when the class changes and a smooth rotational distribution shift. However, the number of classes in each task and the samples are uniform. The Generalized Class Incremental Learning (GCIL) (Mi et al., 2020) utilizes probabilistic modeling to sample the classes and data distributions in each task. Hence, the number of classes in each task is not fixed, the classes can overlap and the sample size for each class can vary. Following Mi et al. (2020), we use GCIL on CIFAR-100 (Krizhevsky et al., 2009) dataset (GCIL-CIFAR-100), set the number of samples
and maximum number of classes per task to 1000 and 50 respectively, number of tasks to 20, and evaluate on both uniform and longtail (imbalanced) sample distribution.

heading: A.4 ONLINE CONTINUAL LEARNING
text: Online continual learning refers to the challenging scenario where a stream of samples is only seen once and is non-iid (Mai et al., 2022; Aljundi et al., 2019). The common approach in the literature is to use the single-epoch protocol where the network is trained on each task in the sequence for only one epoch and there are no additional passages over data. As we aim to position CLS-ER as a general incremental learning method, we are also interested in the online continual learning setting. However, similar to Buzzega et al. (2020a), we also believe that the dataset complexity needs to be considered when setting the number of epochs to disentangle the effect of catastrophic forgetting from underfitting and share their suggestion that future CL works should strive for realism by designing experimental settings which are in line with the guidelines of General Continual Learning (Farquhar & Gal, 2018) which is the goal of our study rather than adopting the single-epoch protocol. For the MNIST-based settings, we use only one epoch per task as it is sufficient for the SGD baseline to learn the single task well. And for the more complex settings, we increase the number of epochs: 50 epochs for Sequential CIFAR-10 and Sequential Tiny-ImageNet and 100 epochs for GCIL-CIFAR-100.
We would also like to emphasize that the experiments on MNIST based settings (S-MNIST, RMNIST, P-MNIST, and MNIST-360) can be considered as online continual learning settings as we only train the network for 1 epoch, and thereby the model only sees the data for each task once. CLSER’s performance in these settings demonstrates its potential for the challenging online continual learning setting.

heading: B RESERVOIR SAMPLING
text: Here, we provide the algorithm for the Reservoir Sampling for maintaining a fixed-size memory buffer. Reservoir sampling takes in a data stream of unknown length and assigns equal probability to each sample for being represented in the memory buffer (M) with a fixed budget size (B). Sampling and replacement are done at random and no priority is assigned to the samples being added or replaced from the memory buffer.
Algorithm 2 Reservoir Sampling Algorithm Input: Memory BufferM, Memory Budget B, Number of seen examplesN , Selected example (x, y)
1: if B > N then . Memory is not full 2: M[N ]←− (x, y) 3: else . Select a sample to remove 4: ν = randomInteger(min = 0,max = N) 5: if ν < B then 6: M[ν]←− (x, y)
returnM

heading: C ADDITIONAL RESULTS
text: In this section, we provide additional experimental results and analysis of the behavior of the model.

heading: C.1 CLS-ER COMPONENTS PERFORMANCE
text: CLS-ER involves the interplay between the working model and the two semantic memories: the plastic and stable models. While we use the stable model for final inference, here we provide the performance of each of these individual components to provide further insights into the workings of our method. Table S1 shows the corresponding performance of the working model and plastic model for each of our experimental settings. We can see that the stable model can effectively consolidate

heading: Dataset Buffer Stable Model Working Model Plastic Model
text: S-MNIST 200 89.54±0.21 89.32±0.23 89.52±0.21 500 92.05±0.30 91.61±0.47 92.04±0.33 5120 95.73±0.10 95.65±0.15 95.73±0.12 S-CIFAR-10 200 66.19±0.75 50.09±1.48 62.68±1.94 500 75.22±0.71 63.09±1.12 71.32±0.89 5120 86.78±0.17 85.00±0.33 86.77±0.17 S-Tiny-ImageNet 200 21.95±0.26 9.72±0.41 16.72±1.78 500 29.61±0.54 14.30±0.49 25.47±0.92 5120 45.92±0.30 40.39±0.42 46.30±0.42 R-MNIST 200 92.26±0.18 89.37±0.47 89.99±0.43 500 94.06±0.07 93.24±0.14 93.52±0.09 5120 94.25±0.06 94.28±0.08 94.37±0.06 P-MNIST 200 84.63±0.40 84.33±0.45 84.54±0.41 500 88.30±0.14 88.12±0.16 88.25±0.14 5120 92.03±0.05 91.96±0.06 92.02±0.05 MNIST-360 200 66.37±0.83 55.59±1.74 60.60±1.41 500 75.70±0.41 72.70±0.80 75.03±0.37 1000 79.54±0.34 78.39±0.69 79.16±0.42 GCIL-CIFAR-100 (Uniform) 200 33.15±2.80 31.74±2.72 32.70±2.78 500 37.01±1.67 35.89±1.69 36.18±1.68 1000 41.09±1.58 40.44±1.80 40.70±1.66 GCIL-CIFAR-100 (Longtail) 200 29.57±3.80 28.19±3.90 29.12±3.89 500 33.26±3.66 32.22±3.79 32.95±3.70 1000 39.21±3.46 38.51±3.55 38.84±3.52
Table S1: CLS-ER components performance analysis for each of the experimental setting.
knowledge across the tasks and therefore provide the highest mean performance for the vast majority of the settings. Figures S1 and S2 further shows how the task-wise performance (on test set) of each of the component varies as subsequent tasks are learned. The stable model retains the performance on previous tasks while the plastic model adapts better to the recent task. Both these models provide feedback to the working model which in turn improves the plastic and stable model.

heading: C.2 TASK PROBABILITIES
text: To test the effectiveness of our method in mitigating the bias towards recent tasks, we provide the task probabilities of the models trained with different buffer sizes on S-CIFAR-10 and S-TinyImageNet. Figures S3 and S4 show that CLS-ER consistently achieves more uniform task probabilities compared to ER and DER++ and effectively mitigates the bias towards the last task.

heading: C.3 MODEL CALIBRATION
text: To further test the consistency of CLS-ER in providing well-calibrated models and the impact of the buffer size, we evaluate the calibration of models trained with different buffer sizes on S-CIFAR-10 and S-Tiny-ImageNet. Figures S5 and S6 show that CLS-ER consistently provides better calibrated models compared to ER and DER++. Remarkably, for both the datasets, on lower buffer sizes, the difference in Expected Calibration Error (ECE) is considerable. This demonstrates the capability of CLS-ER to train high-performance and reliable models under challenging conditions.

heading: C.4 EFFECT OF HYPERPARAMETERS
text: The interaction between the three components of CLS-ER is complementary. Table S3 shows how the performance of each component is affected under different hyperparameter settings. We can draw the following conclusions from the results. The performance improvement in the plastic and stable model is reflected in the working model and the best performance is seen in cases where both the semantic memories are performing well (albeit the focus on tasks is different). This highlights
After Task 1
After Task 2
After Task 3
After Task 4 After Task 5 Bu
ffe r 2
00
98.65 0.0 0.0 0.0 0.0
88.95 89.5 0.0 0.0 0.0
78.2 53.45 89.05 0.0 0.0
81.25 42.4 76.3 87.5 0.0
69.2 41.5 76.8 83.3 41.15
Stable Model
98.35 0.0 0.0 0.0 0.0
67.4 92.65 0.0 0.0 0.0
67.4 22.95 95.65 0.0 0.0
59.9 15.35 31.8 99.1 0.0
20.55 15.9 45.15 63.55 98.1
Working Model
98.85 0.0 0.0 0.0 0.0
75.8 94.15 0.0 0.0 0.0
71.0 29.3 95.55 0.0 0.0
72.2 27.95 54.9 97.05 0.0
59.9 34.05 73.05 80.6 74.75
Plastic Model
After Task 1
After Task 2
After Task 3
After Task 4
After Task 5
Bu ffe
r 5 00
98.7 0.0 0.0 0.0 0.0
93.05 88.55 0.0 0.0 0.0
87.5 74.6 78.3 0.0 0.0
88.8 73.4 74.25 46.05 0.0
77.5 55.7 76.1 84.3 78.55
98.6 0.0 0.0 0.0 0.0
75.55 90.95 0.0 0.0 0.0
80.45 31.5 96.05 0.0 0.0
79.8 37.4 48.35 98.25 0.0
51.1 40.05 58.05 72.65 97.75
98.85 0.0 0.0 0.0 0.0
84.25 92.55 0.0 0.0 0.0
84.1 55.25 92.65 0.0 0.0
86.05 66.9 67.9 82.9 0.0
68.3 50.7 70.25 75.8 94.2
Ta sk 1 Ta sk 2 Ta sk 3 Ta sk 4 Ta sk 5
After Task 1
After Task 2
After Task 3
After Task 4
After Task 5
Bu ffe
r 5 12
0
98.8 0.0 0.0 0.0 0.0
95.9 91.95 0.0 0.0 0.0
96.1 78.45 91.15 0.0 0.0
94.9 75.6 85.7 93.7 0.0
91.0 74.5 86.85 91.2 82.7
Ta sk 1 Ta sk 2 Ta sk 3 Ta sk 4 Ta sk 5
98.7 0.0 0.0 0.0 0.0
93.25 90.85 0.0 0.0 0.0
93.0 74.65 90.2 0.0 0.0
92.65 67.85 82.5 97.2 0.0
83.4 69.2 84.05 91.45 97.0
Ta sk 1 Ta sk 2 Ta sk 3 Ta sk 4 Ta sk 5
98.75 0.0 0.0 0.0 0.0
95.95 92.0 0.0 0.0 0.0
96.05 78.85 91.2 0.0 0.0
94.85 75.45 85.75 93.65 0.0
91.0 74.35 86.75 91.2 83.0
Figure S1: Test set task-wise performance for the individual models on S-CIFAR-10 with different buffer sizes. The task-wise performance (x-axis) is evaluated at the end of training of each task (y-axis) to evaluate how it is affected as training progresses.
the crucial role of both memories in enabling CLS-ER to learn efficiently. For a fixed rS value, the final performance of the stable model is affected considerably by the performance of the plastic model. The method is not highly sensitive to the particular choice of hyperparameters as different settings can attain similar performance. Because of the complementary nature of the components, we can often fix a set of parameters (e.g. λ, αS , αS and rS) and only finetune the remaining parameters (e.g. r′P ) which facilitates hyperparameter tuning significantly.

heading: D COMPARISON WITH A SINGLE SEMANTIC MEMORY
text: CLS-ER employs two semantic memories as we aim to mimic the fast and slow learning mechanisms in the hippocampus and neocortex respectively. Here we compare our method with a single semantic memory (Mean-ER) and Table S2 shows that while it still performs admirably compared to the other CL methods, the dual semantic memories in CLS-ER provides additional performance gains especially on the complex datasets under the challenging lower memory buffer settings and has a much lower variance. We attribute this to the failure of Mean-ER in maintaining the performance on both the recent and earlier tasks together i.e there is an inherent trade-off as tuning the semantic memory to adapt to the recent changes comes at the cost of performance on earlier tasks and vice versa. CLS-ER efficiently tackles this trade-off by maintaining two specialized long-term and shortterm memories. The performance of Mean-ER, however, provides further evidence for the benefits of using consolidated information for memory replay.
Note that for a fair comparison, we use the same hyperparameter search space as CLS-ER for finding the optimal parameters for Mean-ER and report the average and 1 std of 10 runs with different initializations using the best parameters for each setting. Table S6 provides the chosen hyperparameters. For inference, similar to CLS-ER, we use the EMA-weighted model (semantic memory) for Mean-ER.
After Task 1
After Task 2
After Task 3
After Task 4
After Task 5
After Task 6
After Task 7
After Task 8
After Task 9
After Task 10
Bu ffe
r 2 00
68.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
69.6 37.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
56.0 54.8 28.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0
46.2 41.6 57.7 19.9 0.0 0.0 0.0 0.0 0.0 0.0
39.0 33.6 49.9 54.7 23.9 0.0 0.0 0.0 0.0 0.0
30.0 26.8 34.8 45.5 54.3 27.5 0.0 0.0 0.0 0.0
23.6 18.5 29.6 31.2 35.3 55.8 16.0 0.0 0.0 0.0
19.5 12.9 21.1 20.8 21.7 41.7 55.1 27.6 0.0 0.0
16.8 9.6 18.4 18.3 11.8 22.4 39.5 51.4 20.8 0.0
12.0 9.0 14.1 15.7 8.0 13.9 26.2 34.9 46.9 19.0
Stable Model 76.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
17.9 70.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
11.6 10.4 76.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0
7.9 3.9 14.0 80.8 0.0 0.0 0.0 0.0 0.0 0.0
5.5 2.2 5.1 11.2 77.8 0.0 0.0 0.0 0.0 0.0
4.1 1.7 3.3 5.3 3.3 77.0 0.0 0.0 0.0 0.0
2.5 0.7 3.0 2.3 2.2 4.6 79.1 0.0 0.0 0.0
2.2 0.9 1.8 2.4 2.6 2.7 5.8 77.2 0.0 0.0
2.1 0.9 1.4 2.6 1.9 1.5 1.2 4.1 71.6 0.0
1.8 0.5 2.4 1.0 1.5 1.4 0.6 3.0 3.4 80.5
Working Model 72.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
49.7 64.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
35.2 41.9 69.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0
26.6 21.7 48.8 70.7 0.0 0.0 0.0 0.0 0.0 0.0
17.3 13.6 23.4 47.0 71.8 0.0 0.0 0.0 0.0 0.0
12.0 8.2 12.5 18.3 28.9 72.6 0.0 0.0 0.0 0.0
9.0 4.4 10.9 11.7 9.4 36.2 73.6 0.0 0.0 0.0
9.5 4.0 7.7 8.5 5.5 14.3 42.9 68.5 0.0 0.0
8.6 3.5 6.5 7.9 4.8 6.6 16.9 33.2 64.6 0.0
5.5 3.4 4.9 7.1 3.8 4.2 5.2 11.7 34.4 71.5
Plastic Model
After Task 1
After Task 2
After Task 3
After Task 4
After Task 5
After Task 6
After Task 7
After Task 8
After Task 9
After Task 10
Bu ffe
r 5 00
69.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
71.7 40.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
61.1 55.5 40.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0
53.3 44.4 61.6 34.5 0.0 0.0 0.0 0.0 0.0 0.0
46.9 37.4 54.7 59.4 23.4 0.0 0.0 0.0 0.0 0.0
40.9 31.5 46.1 51.7 55.4 27.0 0.0 0.0 0.0 0.0
36.5 25.0 33.5 40.4 45.1 55.5 25.4 0.0 0.0 0.0
30.3 21.4 27.5 30.7 33.9 46.1 56.2 31.2 0.0 0.0
29.3 18.4 23.7 25.7 25.1 33.0 44.8 56.5 28.9 0.0
24.2 17.5 19.8 21.1 18.6 23.9 34.3 48.1 52.8 28.2
75.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
35.7 71.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
26.5 20.6 77.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0
23.1 13.9 30.5 79.6 0.0 0.0 0.0 0.0 0.0 0.0
17.9 9.9 12.5 23.7 79.4 0.0 0.0 0.0 0.0 0.0
15.1 8.6 7.2 12.7 13.4 78.2 0.0 0.0 0.0 0.0
11.3 7.5 7.2 8.0 9.7 16.4 78.8 0.0 0.0 0.0
12.1 6.3 3.8 7.1 5.7 8.4 15.1 77.2 0.0 0.0
11.1 3.4 5.6 6.5 5.0 7.3 7.1 17.9 72.9 0.0
8.5 4.8 4.9 5.5 3.7 5.4 4.8 13.1 13.3 79.9
74.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
62.4 60.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
49.5 44.5 69.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0
42.4 34.1 59.0 66.1 0.0 0.0 0.0 0.0 0.0 0.0
35.4 24.7 40.9 53.5 67.1 0.0 0.0 0.0 0.0 0.0
28.4 20.2 25.0 35.2 49.0 68.5 0.0 0.0 0.0 0.0
25.7 16.7 18.4 24.9 29.7 51.0 67.1 0.0 0.0 0.0
22.8 15.1 15.8 19.3 19.5 31.6 53.7 66.8 0.0 0.0
20.0 11.6 12.9 15.1 12.5 19.5 31.0 52.6 63.0 0.0
16.8 10.7 12.7 14.3 10.8 14.6 21.5 36.2 51.6 68.8
Ta sk 1 Ta sk 2 Ta sk 3 Ta sk 4 Ta sk 5 Ta sk 6 Ta sk 7 Ta sk 8 Ta sk 9
Ta sk
10
After Task 1
After Task 2
After Task 3
After Task 4
After Task 5
After Task 6
After Task 7
After Task 8
After Task 9
After Task 10
Bu ffe
r 5 12
0
76.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
76.5 47.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
73.3 60.4 42.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0
69.7 57.5 63.3 41.1 0.0 0.0 0.0 0.0 0.0 0.0
65.6 54.5 64.9 62.0 31.8 0.0 0.0 0.0 0.0 0.0
61.4 52.0 60.6 62.1 54.4 28.8 0.0 0.0 0.0 0.0
57.6 47.1 56.2 59.3 56.5 51.5 34.6 0.0 0.0 0.0
52.8 44.3 52.7 55.4 53.1 51.6 56.4 38.8 0.0 0.0
49.0 40.6 48.8 54.2 50.1 48.6 53.7 53.7 31.8 0.0
48.5 36.9 44.6 48.9 47.9 44.7 50.7 51.7 45.8 32.5
Ta sk 1 Ta sk 2 Ta sk 3 Ta sk 4 Ta sk 5 Ta sk 6 Ta sk 7 Ta sk 8 Ta sk 9
Ta sk
10
78.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
67.3 68.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
62.4 51.0 73.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0
59.1 47.9 59.0 75.4 0.0 0.0 0.0 0.0 0.0 0.0
54.2 44.6 51.6 55.1 73.9 0.0 0.0 0.0 0.0 0.0
50.8 40.2 45.4 49.7 46.8 73.2 0.0 0.0 0.0 0.0
45.5 39.2 45.2 47.5 47.7 44.0 74.6 0.0 0.0 0.0
45.4 36.9 41.3 46.3 41.4 40.2 50.0 72.7 0.0 0.0
42.7 30.3 38.9 43.7 39.2 37.6 43.2 44.2 67.7 0.0
35.0 29.2 36.0 39.0 38.0 35.2 36.9 37.0 37.1 77.9
Ta sk 1 Ta sk 2 Ta sk 3 Ta sk 4 Ta sk 5 Ta sk 6 Ta sk 7 Ta sk 8 Ta sk 9
Ta sk
10
76.9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
76.3 47.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
72.5 59.5 49.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0
68.9 56.9 64.4 44.6 0.0 0.0 0.0 0.0 0.0 0.0
64.3 54.0 64.6 62.5 44.3 0.0 0.0 0.0 0.0 0.0
59.7 50.8 58.0 62.1 57.8 39.9 0.0 0.0 0.0 0.0
56.7 46.6 55.1 58.5 57.2 54.0 40.6 0.0 0.0 0.0
51.8 43.9 52.4 54.5 52.9 51.8 57.3 41.4 0.0 0.0
48.9 40.1 48.3 53.9 49.4 48.7 54.0 54.0 34.6 0.0
47.5 36.4 44.1 48.5 47.2 44.4 50.5 51.3 47.1 40.8
Figure S2: Test set task-wise performance for the individual models on S-Tiny-ImageNet with different buffer sizes. The task-wise performance (x-axis) is evaluated at the end of training of each task (y-axis) to evaluate how it is affected as training progresses.
ER DER++ CLS-ER 0.0
0.1
0.2
0.3
0.4
0.5
0.6
Ta sk
P ro
ba bi
lit y
Buffer 200
ER DER++ CLS-ER
Buffer 500
ER DER++ CLS-ER
Buffer 5120 Task 1 Task 2 Task 3 Task 4 Task 5
Figure S3: Task probabilities for different methods on S-CIFAR-10 with varying memory budget.

heading: E TRAINING AND IMPLEMENTATION DETAILS
text: For a fair comparison, we aim to keep the experimental settings close to the current state-of-theart DER++ (Buzzega et al., 2020a) as much as possible to disassociate the effect of the training
ER DER++ CLS-ER 0.0
0.2
0.4
0.6
0.8
1.0
Ta sk
P ro
ba bi
lit y
Buffer 200
ER DER++ CLS-ER
Buffer 500
ER DER++ CLS-ER
Buffer 5120 Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Task 10
Figure S4: Task probabilities for different methods on S-Tiny-ImageNet with varying memory budget.
0.0
0.2
0.4
0.6
0.8
1.0
Bu ffe
r 2 00
A
cc ur
ac y
ER
ECE=44.39
Gap Outputs
DER++
ECE=20.31
Gap Outputs
CLS-ER
ECE=6.85
Gap Outputs
0.0
0.2
0.4
0.6
0.8
1.0
Bu ffe
r 5 00
A
cc ur
ac y
ECE=32.24
Gap Outputs
ECE=16.57
Gap Outputs
ECE=6.41
Gap Outputs
0.0 0.2 0.4 0.6 0.8 1.0 Confidence
0.0
0.2
0.4
0.6
0.8
1.0
Bu ffe
r 5 12 0 A cc ur ac y
ECE=13.37
Gap Outputs
0.0 0.2 0.4 0.6 0.8 1.0 Confidence
ECE=8.49
Gap Outputs
0.0 0.2 0.4 0.6 0.8 1.0 Confidence
ECE=6.73
Gap Outputs
Figure S5: Reliability plots for the different methods on S-CIFAR-10 with varying memory budget.
schedule. We use the same optimizer, the number of epochs, batch size, and memory batch size as DER++. For S-Tiny-ImageNet, we reduce the number of epochs to 50 from 100 used by DER++ as our method can learn efficiently with fewer epochs, and quickly acquiring new knowledge is preferred for CL. Similar to DER++, we finetune the memory batch size for S-MNIST and MNIST360. We select the hyperparameters for each of the experimental setting using a small validation set, αS , αP ∈ (0.99, 0.999), rS , rP ∈ (0, 1], λ ∈ (0, 2]. Table S4 provides the hyperparameters used for each of the experimental settings. Note that for the vast majority of datasets, we use uniform settings (lr, epochs, batch size, memory batch size, and lambda) across the different buffer sizes and
0.0
0.2
0.4
0.6
0.8
1.0
Bu ffe
r 2 00
A
cc ur
ac y
ER
ECE=74.54
Gap Outputs
DER++
ECE=26.60
Gap Outputs
CLS-ER
ECE=5.27
Gap Outputs
0.0
0.2
0.4
0.6
0.8
1.0
Bu ffe
r 5 00
A
cc ur
ac y
ECE=69.54
Gap Outputs
ECE=14.31
Gap Outputs
ECE=8.23
Gap Outputs
0.0 0.2 0.4 0.6 0.8 1.0 Confidence
0.0
0.2
0.4
0.6
0.8
1.0
Bu ffe
r 5 12 0 A cc ur ac y
ECE=51.31
Gap Outputs
0.0 0.2 0.4 0.6 0.8 1.0 Confidence
ECE=13.16
Gap Outputs
0.0 0.2 0.4 0.6 0.8 1.0 Confidence
ECE=12.34
Gap Outputs
Figure S6: Reliability plots for the different methods on S-Tiny-ImageNet with varying memory budget.

heading: Buffer Method Class-IL Domain-ILS-MNIST S-CIFAR-10 S-Tiny-ImageNet R-MNIST P-MNIST
text: – JOINT 95.57±0.24 92.20±0.15 59.99±0.19 95.76±0.04 94.33±0.17SGD 19.60±0.04 19.62±0.05 7.92±0.26 67.66±8.53 40.70±2.33 200 Mean-ER 88.32±0.65 61.88±2.43 17.68±1.65 92.10±1.07 83.28±0.68CLS-ER 89.54±0.21 66.19±0.75 21.95±0.26 92.26±0.18 84.63±0.40 500 Mean-ER 91.79±0.23 70.40±1.21 24.97±0.80 92.78±0.44 87.73±0.39CLS-ER 92.05±0.32 75.22±0.71 29.61±0.54 94.06±0.07 88.30±0.14 5120 Mean-ER 95.57±0.18 84.84±2.0 45.69±0.58 94.25±0.51 91.90±0.11CLS-ER 95.73±0.11 86.78±0.17 45.92±0.30 94.25±0.06 92.03±0.05
Table S2: Comparison of CLS-ER with Mean-ER (single semantic memory) on Class-IL and Domain-IL settings. We report the mean and 1 std of 10 runs with different initializations.
requires only slight modifications in the other hyperparameters which shows that our method does not require extensive finetuning for different memory budgets.
E.1 GCIL-CIFAR-100
To test our method under challenging GIL settings that better simulate the challenges of CL in the real world, we incorporate the GCIL setting from the code provided by Mi et al. (2020) with the continual dataset template class in the mammoth framework. We set the number of phases (length of task sequences) to 20, with the total number of samples in each phase set to 1000 and the maximum
number of classes in each phase set to 50. We evaluate on both uniform and longtail (imbalanced) data distributions. Since GCIL involves the probabilistic sampling of the classes and their samples in each phase, the random seed determines the complexity of the GCIL setting. We use 10 different identical random seeds for our experiments so that each method is evaluated under identical settings and the difference between longtail and uniform is only the data distribution between the classes in each phase.
For each of our method, we use identical training scheme (lr=0.1, epochs=100, batch size=32 and memory batch size=32). For DER++, as per the authorts suggestion, we performed hyperparameter search over α ∈ [0.2, 0.3] and beta ∈ [0.5, 1.0] with step size of 0.1. Table S5 provides the parameters chosen for each of the method under the different settings.

heading: E.2 PERTURBATION ANALYSIS
text: For the perturbation analysis, we used the code and checkpoints provided by Buzzega et al. (2020a) for DER++ and ER. We would like to express our gratitude to the authors for their support and for making the mammoth framework available for the research community which provides a framework for a fair comparison of different CL methods under uniform experimental conditions.
λ rS rP Stable Model Working Model Plastic Model
0.1
0.1 0.2 73.53±1.07 62.80±0.63 71.11±2.21 0.3 72.44±1.37 63.53±1.98 70.97±1.71 0.4 73.05±0.93 61.81±1.92 68.75±2.20 0.5 75.16±1.09 63.95±1.92 70.42±1.09 0.6 75.04±0.66 62.82±0.70 69.61±0.31 0.7 73.94±0.48 63.34±0.46 70.30±1.68 0.8 74.61±1.10 62.68±0.65 70.74±0.39 0.9 73.74±2.14 62.69±1.97 69.52±0.79 1.0 75.73±0.68 64.21±1.11 72.00±0.56
0.2 0.3 70.26±1.79 61.63±1.03 69.31±1.82 0.4 71.80±1.17 62.64±0.18 70.64±1.22 0.5 70.69±2.13 61.76±0.64 69.65±1.92 0.6 72.45±0.68 63.87±0.85 71.29±0.72 0.7 71.47±1.98 61.12±1.90 70.22±2.24 0.8 72.16±0.56 62.71±0.57 70.83±0.64 0.9 72.09±0.59 63.33±1.01 71.20±0.87 1.0 72.05±1.35 63.74±1.75 71.01±1.28
0.3 0.4 68.46±1.48 60.96±1.62 68.31±1.40 0.5 70.05±2.54 63.06±1.26 69.90±2.57 0.6 69.57±1.07 61.25±1.96 69.36±1.06 0.7 68.99±2.34 61.61±2.17 68.81±2.27 0.8 71.21±0.48 63.08±0.82 70.99±0.57 0.9 71.26±1.47 62.33±0.64 71.03±1.56 1 69.00±0.41 61.38±0.92 68.69±0.31
0.15
0.1 0.2 70.19±1.97 61.39±2.06 69.81±1.60 0.3 73.72±0.83 62.07±0.84 70.18±0.09 0.4 71.60±2.30 61.11±2.00 69.15±1.08 0.5 74.18±0.37 63.32±0.98 71.08±2.04 0.6 74.90±0.40 62.35±2.31 71.58±0.79 0.7 74.52±1.10 62.59±2.64 70.90±2.30 0.8 75.27±1.21 62.00±1.98 71.27±1.64 0.9 74.61±0.91 63.47±1.60 70.49±0.95 1.0 76.03±0.64 63.63±1.01 71.42±1.11
0.2 0.3 72.59±1.44 61.81±1.03 72.02±1.30 0.4 71.30±3.42 63.15±0.51 70.92±2.83 0.5 69.89±1.95 60.60±0.95 68.87±2.56 0.6 72.34±0.89 62.18±1.31 71.15±0.94 0.7 72.70±1.11 62.50±1.18 71.49±1.21 0.8 72.42±1.50 61.85±0.83 71.04±1.68 0.9 71.18±0.71 61.81±1.29 70.09±0.54 1.0 73.52±0.65 64.19±0.86 72.56±0.52
0.3 0.4 70.32±1.39 62.39±2.00 70.13±1.33 0.5 71.60±1.53 62.67±2.08 71.40±1.54 0.6 70.36±1.82 62.28±2.28 70.13±2.03 0.7 69.79±1.93 61.13±1.37 69.65±1.82 0.8 69.85±0.95 60.69±1.63 69.78±0.60 0.9 71.32±1.68 61.79±1.41 71.03±1.61 1.0 71.39±0.49 62.35±0.88 71.11±0.55
Table S3: The effect of different hyperparameter settings on the individual components of CLS-ER trained on S-CIFAR-10 with 500 buffer size. For all the experiments αS and αP are fixed to 0.999 and the performance is averaged over 3 runs with different initialization.
Dataset Buffer lr Epochs Batch Size MemoryBatch Size λ αS αP rS rP
S-MNIST 200 0.03 1 10 128 2.0 0.99 0.99 0.9 1.0 500 0.1 1 10 32 2.0 0.99 0.99 0.9 1.0 5120 0.1 1 10 32 2.0 0.99 0.99 0.8 1.0 S-CIFAR-10 200 0.1 50 32 32 0.15 0.999 0.999 0.1 0.3 500 0.1 50 32 32 0.15 0.999 0.999 0.1 0.9 5120 0.1 50 32 32 0.15 0.999 0.999 0.8 1.0 S-Tiny-ImageNet 200 0.1 50 32 32 0.1 0.999 0.999 0.04 0.08 500 0.1 50 32 32 0.1 0.999 0.999 0.05 0.08 5120 0.1 50 32 32 0.1 0.999 0.999 0.07 0.08 R-MNIST 200 0.2 1 128 128 0.75 0.999 0.99 1.0 1.0 500 0.2 1 128 128 0.75 0.999 0.99 1.0 1.0 5120 0.2 1 128 128 0.75 0.999 0.99 1.0 1.0 P-MNIST 200 0.2 1 128 128 1.0 0.99 0.99 0.8 1.0 500 0.2 1 128 128 1.0 0.99 0.99 0.8 1.0 5120 0.2 1 128 128 1.0 0.99 0.99 0.9 1.0 MNIST-360 200 0.2 1 16 16 0.75 0.999 0.99 1.0 1.0 500 0.2 1 16 32 1.25 0.99 0.99 0.9 1.0 1000 0.2 1 16 128 0.75 0.99 0.99 0.9 1.0
Table S4: The hyperparameters used for each of the experimental settings for CLS-ER.
Distribution Method Buffer Parameters
Uniform
CLS-ER 200 λ = 0.15, αS = 0.999, αP = 0.999, rS = 0.6, rP = 0.7 500 λ = 0.15, αS = 0.999, αP = 0.999, rS = 0.6, rP = 0.9 1000 λ = 0.15, αS = 0.999, αP = 0.999, rS = 0.6, rP = 0.8 DER++ 200 α = 0.2, β = 0.5 500 α = 0.2, β = 0.6 1000 α = 0.3, β = 0.6
Longtail
CLS-ER 200 λ = 0.1, αS = 0.999, αP = 0.999, rS = 0.6, rP = 0.7 500 λ = 0.1, αS = 0.999, αP = 0.999, rS = 0.6, rP = 0.7 1000 λ = 0.1, αS = 0.999, αP = 0.999, rS = 0.6, rP = 0.8 DER++ 200 α = 0.2, β = 0.6 500 α = 0.2, β = 0.8 1000 α = 0.3, β = 0.9
Table S5: The hyperparameters used for GCIL-CIFAR-100 experiments. For each of the experiment, we used lr=0.1, epochs=100, betch size=32 and buffer memory batch=32.
Dataset Buffer lr Epochs Batch Size MemoryBatch Size λ α r
S-MNIST 200 0.03 1 10 128 2.0 0.99 1.0 500 0.1 1 10 32 2.0 0.99 1.0 5120 0.1 1 10 32 2.0 0.99 1.0 S-CIFAR-10 200 0.1 50 32 32 0.15 0.999 0.2 500 0.1 50 32 32 0.15 0.999 0.5 5120 0.1 50 32 32 0.15 0.999 0.8 S-Tiny-ImageNet 200 0.1 50 32 32 0.1 0.999 0.06 500 0.1 50 32 32 0.1 0.999 0.08 5120 0.1 50 32 32 0.1 0.999 0.08 R-MNIST 200 0.2 1 128 128 0.75 0.999 1.0 500 0.2 1 128 128 0.75 0.999 1.0 5120 0.2 1 128 128 0.75 0.999 1.0 P-MNIST 200 0.2 1 128 128 1.0 0.99 0.9 500 0.2 1 128 128 1.0 0.99 1.0 5120 0.2 1 128 128 1.0 0.99 0.9
Table S6: The hyperparameters used for each of the experimental settings for Mean-ER.

Review summaries:
The paper presents CLS-ER, a dual memory mechanism for the continual learning setting. During training the model receives data from a non-i.i.d. source as well as random samples from a sample buffer (the episodic memory). The stable and plastic models act as teacher models. Their logits are the target of the student model. Which one is used as a target depends on the highest softmax score w.r.t. a ground trough class. The teacher networks are both exponential moving averages of the previous student models. Thus, the only difference between the teacher models "plastic" and "stable" is their EMA rates which in the case of the plastic model is larger such that it focuses on more recent weight updates.  The slower learning of the stable model is the mechanism which ought to mirror the consolidation of structural knowledge. It is used from inference Samples from the datastream are transferred to the sample buffer using reservoir sampling.   The authors experiment on image datasets (variants of MNIST, CIFAR10, Tiny-ImageNet, see the Mammoth framework) with data augmentation (random flips and crops). They compare with several prio works and the evaluation is thorough and convincing. 

They propose a dual memory experience based on complementary learning systems (CLS). The working model is updated using consistency loss with the selected optimal semantic memory from plastic and stable models. Plastic and stable models are updated with an exponential moving average trick (EMA) and the working model.

In this paper, motivated from the CLS theory in neuroscience that efficient learning requires short-term adaptation and slow learning of structured information, the authors propose a novel dual memory experience replay method for continual learning. The idea is to build  long-term and short-term semantic memories by maintaining another two models: plastic model and stable model. The plastic model is used for fast learning of recent experiences and the stable model is used for slow learning of structural knowledge. The two models are updated with a Mean Teacher fashion during training with different frequencies. The proposed CLS-ER is  task-agnostic and can be applied in different continual learning settings. The experimental results show that the proposed CLS-ER outperforms several baselines for continual learning.