Error Sensitivity Modulation based Experience Replay: Mitigating Abrupt Representation Drift in Continual Learning
Authors: Fahad Sarfraz
Elahe Arani
Bahram Zonooz
Venue: ICLR 2023 poster (2023)
Keywords: Continual Learning
Catastrophic Forgetting
Multi memory System
Experience Replay
Error-Sensitivity modulation
Brain inspired Algorithm
Representation Drift

Abstract: Humans excel at lifelong learning, as the brain has evolved to be robust to distribution shifts and noise in our ever-changing environment. Deep neural networks (DNNs), however, exhibit catastrophic forgetting and the learned representations drift drastically as they encounter a new task. This alludes to a different error-based learning mechanism in the brain. Unlike DNNs, where learning scales linearly with the magnitude of the error, the sensitivity to errors in the brain decreases as a function of their magnitude. To this end, we propose "ESMER" which employs a principled mechanism to modulate error sensitivity in a dual-memory rehearsal-based system. Concretely, it maintains a memory of past errors and uses it to modify the learning dynamics so that the model learns more from small consistent errors compared to large sudden errors. We also propose "Error-Sensitive Reservoir Sampling" to maintain episodic memory, which leverages the error history to pre-select low-loss samples as candidates for the buffer, which are better suited for retaining information. Empirical results show that ESMER effectively reduces forgetting and abrupt drift in representations at the task boundary by gradually adapting to the new task while consolidating knowledge. Remarkably, it also enables the model to learn under high levels of label noise, which is ubiquitous in real-world data streams.

Published as a conference paper at ICLR 2023
ERROR SENSITIVITY MODULATION BASED EXPERI- ENCE REPLAY: MITIGATING ABRUPT REPRESENTA- TION DRIFT IN CONTINUAL LEARNING
Fahad Sarfraz"), Elahe Arani*2 & Bahram Zonooz!.
Advanced Research Lab, NavInfo Europe, Netherlands
Dep. of Mathematics and Computer Science, Eindhoven University of Technology, Netherlands fahad. sarfraz@navinfo.cet {e.arani, bahram. zonooz ) @ggaill com.
ABSTRACT
Humans excel at lifelong learning, as the brain has evolved to be robust to distri- bution shifts and noise our ever-changing environment. Deep neural networks (DNNs), however, exhibit catastrophic forgetting and the learned representations drift drastically they encounter a new task. This alludes based learning mechanism in the brain. Unlike DNNs, where learning scales lin- early with the magnitude of the error, the sensitivity to errors in the brain decreases function of their magnitude. To this end, we propose ESMER which employs a principled mechanism to modulate error sensitivity in dual-memory rehearsal- based system. Concretely, it maintains memory of past emrors and uses it modify the learning dynamics so that the model learns more from small consistent errors compared large sudden errors. We also propose Error-Sensitive Reser- voir Sampling to maintain episodic memory, which leverages the error history to pre-select low-loss samples candidates for the buffer, which are better suited for retaining information. Empirical results show that ESMER effectively reduces forgetting and abrupt drift in representations at the task boundary by gradually adapting to the new task while consolidating knowledge. Remarkably, it also en- ables the model to learn under high levels of label noise, which ubiquitous real-world data streams. Code: https: : //g /ggbtbb.b.. cwm.cooo///////o /NurrA A f.ab /SMMR
different error-
INTRODUCTION
The human brain has evolved to engage with and learn from an ever-changing and noisy environ- ment, enabling humans excel lifelong learning. This requires it to be robust to varying degrees of distribution shifts and noise to acquire, consolidate, and transfer knowledge under uncertainty. DNNs, on the other hand, inherently designed for batch learning from a static data distribu- tion and therefore exhibit catastrophic forgetting (McCloskey & Cohen, 1989) of previous tasks when learning tasks sequentially from a continuous stream of data. The significant gap between the lifelong learning capabilities of humans and DNNs suggests that the brain relies on fundamentally different error-based learning mechanisms.
Among the different approaches to enabling continual learning (CL) in DNNs (Parisi et al., 2019), methods inspired by replay of past activations in the brain have shown promise in reducing forgetting in challenging and more realistic scenarios (Hayes et al., 2021; Farquhar & Gal, 2018; van de Ven & Tolias, 2019). They, however, struggle to approximate the joint distribution of tasks with a small buffer, and the model may undergo drastic drift in representations when there is considerable distribution shift, leading to forgetting. In particular, when a new set of classes is introduced, the new samples poorly dispersed in the representation space, and initial model updates significantly perturb the representations of previously learned classes (Caccia et al., 2021). This even more pronounced in the lower buffer regime, where it increasingly challenging for the model to recover from the initial disruption. Therefore, it is critical for a CL agent to mitigate the abrupt drift in representations and gradually adapt to the new task.
"Contributed equally.

---

Published as a conference paper at ICLR 2023
Lee
Nemory et
10m6
Toes
Dual Memories
Epi sodic Memory
Working Model
Momentum
Stable Model
Uodate
Dm
D,
Figure 1: ESMER employs a principled mechanism for modulating the enror sensitivity in a dual- memory rehearsal- based system. includes a stable model which accumulates the structural knowl- edge in the working model and episodic memory. Additionally, memory of eπrors is maintained which informs the contribution of each sample in the incoming batch towards learning such that the working model learns structure of the learned classes. Finally, we employ error sensitive reservoir sampling which uses the error memory to prioritize the representation of low-loss samples in the buffer.
from low errors. The stable model is utilized retain the relational
To this end, we look deeper into the dynamics error- based learning in the brain. Evidence suggests that different characteristics of the error, including its size, affect how the learning process occurs in the brain Ccrisimmagaa-Hemmiie Hemminger et al., 2010). In particular, sensitivity to errors decreases as function of their magnitude, causing the brain learn more from small errors compared to large errors (Marko et al., 2012; Castro et al., 2014). This sensitivity is modulated through principled mechanism that takes into account the history of past errors (Herzfeld al., 2014) which suggests that the brain maintains an additional memory of errors. The robustness of the brain to high degrees of distribution shifts and its proficiency in learning under uncertainty and noise may be attributed to the principled modulation of error sensitivity and the consequent learning from low errors.
DNNs, on the other hand, lack any mechanism to modulate the error sensitivity and learning scales linearly with the error size. This is particularly troublesome for CL. where abrupt distribution shifts initially cause considerable spike errors. These significantly larger errors associated with the samples from unobserved classes dominate the gradient updates and cause disruption of previously learned representations, especially the absence of sufficient memory samples. this end, we propose an Error Sensitiviry Modulation based Experience Replay (ESMER) method that employs a principled mechanism to modulate sensitivity to errors based on its consistency with memory of past (Figure 1). Concretely, trajectory and utilizes it to adjust the contribution of each incoming sample to learning based on how far they are from the mean statistics of error memory. This allows the model learn more from small consistent errors compared to large sudden errors, thus gradually adapting to the new task and mitigating the abrupt drift in representations the task boundary. To keep the error memory stable, task boundary information is utilized to prevent sudden changes. Additionally, we propose an Error- Sensitive Reservoir Sampling approach for maintaining the buffer, which utilizes the error memory to pre-select low-loss samples from the current batch as candidates for being represented in the buffer. It ensures that only the incoming samples that have been well learned are added to the buffer that are better suited to retain information and do not cause representation drift when replaying them. The proposed sampling approach also ensures higher-quality representative samples in memory by filtering outliers and noisy labels, which can degrade performance.
method maintains memory of errors along the training
Another salient component of the learning machinery the brain is the efficient use of multiple memory systems that operate different timescales (Hassabis et al., 2017; Kumaran et al., 2016). Furthermore, replay of previous neural activation patterns is considered to facilitate memory forma- tion and consolidation (Walker & Stickgold, 2004). These components may play role in facilitating
2

---

Published as a conference paper at ICLR 2023
lifelong learning in the brain and addressing the challenges of distribution shifts. Therefore, ESMER also employs experience replay in a dual-memory system. In addition to episodic memory, we main- tain a semantic memory, which gradually aggregates the knowledge encoded in the weights of the working model and builds consolidated representations, which can generalize across the tasks. The two memories interact to enforce consistency in the functional space and encourage the model to adapt its representations and decision boundary to the new classes while preserving the relational structure of previous classes. The gradual aggregation of knowledge in semantic memory and the enforced relational consistency further aid in mitigating the abrupt change in representations.
We extensively evaluate our method against state-of of-te-of-the-rr rehearsal -aased approaches under vari- ous CL settings, which simulate different complexities in the real world including distribution shift, recurring classes, data imbalance, and varying degrees of noisy labels. ESMER provides consid- erable performance improvement in all scenarios, especially with lower memory budget, demon- strating the versatility of our approach. On the challenging Seq- CEQ-CIFRRIO with 200 buffer size and 50% label noise, ESMER provides more than 116% gain compared to the baseline ER method. Fur- thermore, we show that mitigates abrupt representation drift, is more robust to label noise, and reduces the task recency bias, leading to uniform and stable performance across tasks.
RELATED WORK
Different approaches have been proposed to address the problem of catastrophic forgetting in DNNs (De Lange et al., 2021). Among them, rehearsal-based methods (Hayes et al., 2021) that use episodic memory for continual rehearsal of samples from previous tasks have shown promise in challenging CL scenarios (Farquhar & Gal, 2018). The base Experience Replay (ER; Riemer et al. (2018)) performs interleaved training of the new task and the memory sample to approximate the joint distribution of tasks. Dark Experience Replay (DER++ Buzzega et al. (2020)) applies an additional distillation loss by storing the output logits together with the samples and enforcing consistency in the output space. Although promising, these methods struggle to approximate the joint distribution of tasks. Caccia et al. (2021) show that the task transition causes drastic drift in the learned representations of the previous classes seen and propose an asymmetric cross-entropy loss (ER-ACE) on the incoming samples that only considers logits of the new classes. However, an optimal approach to replaying and constraining the model update to retain knowledge and mitigate representation drift remains open question.
Another promising direction is the utilization of multiple memory systems (Wang et al., 2022; Pham et al., 2021). Notably, CLS-ER (Arani et al., 2021) mimics the interplay between fast and slow learning systems by maintaining two semantic memories that aggregate the weights of the model at different rates using an exponential moving average. It is still unclear, however, what the optimal approach is to build and utilize multiple memory systems. Our method focuses on experience replay in
architecture and demonstrates the benefits of mimicking error sensitivity modu- lation in the brain. We also advocate for a more comprehensive evaluation that covers the different challenges involved in lifelong learning in the real world. In particular, we also evaluate on noisy CL setting in which the agent has to learn continuously from a noisy data stream.
METHODOLOGY
We first motivate our study with an overview of the issue of abrupt representation drift in DNNs in the CL setting, followed by inspiration from the learning dynamics in the brain. Finally, we present the details of the different components of our method.
3.1 ABRUPT REPRESENTATION DRIFT IN DNNS
The sequential learning of tasks in CL potentially exposes the model to varying degrees of distri- bution shifts in the input and output space. In particular, the introduction of previously unobserved classes at the task boundary may lead to an abrupt representation drift, leading to forgetting. Caccia et al. (2021) show that the samples of the new classes are poorly dispersed and lie near and within the representation space of the previous classes. Therefore, the initial parameter updates cause a significant perturbation of the representations of the previous classes. This is exacerbated by the
3

---

Published as a conference paper at ICLR 2023
inherent imbalance between the samples of the new classes in the current task and the stored sam- ples of previous classes in memory. especially in the lower buffer size regime. Caccia ct al. (2021) further show that while the model is able to recover somewhat from the disruptive parameter updates at the task transition with larger buffer sizes, the model fails to recover when the buffer size is small. Moreover, learning scales linearly with the size of the error in standard training. Hence, the con- siderably higher cross-entropy loss from the unobserved classes dominates learning and biases the gradients towards the new classes. These insights suggest that there a need for different learning paradigm tailored for CL that is more robust to abrupt distribution shifts.
3.2 INSPIRATION FROM THE BRAIN
The human brain, on the other hand, has evolved to be robust to distribution shifts and to continually learn and interact with dynamic environment. This is enabled by complex set of mechanisms and interactions of multiple memory systems. In particular, we probe the dynamics of error-based learn- ing in the brain to draw some insights. Unlike DNNs where learning scales linearly with emror size, evidence suggests that the brain modulates its sensitivity to error as a function of error magnitude, so it learns more from small consistent errors compared to large errors (Marko et al., 2012; Castro et al., 2014; Smith & Shadmehr, 2004). To enable such a mechanism, the brain appears to maintain an additional memory of errors that informs error sensitivity modulation (Herzfeld et al., 2014).
Another salient element of the learning machinery of the brain is the efficient use of multiple memory systems that operate on different timescales (Hassabis al., 2017: Kumaran et al., 2016). More- over, replay of previous neural activation patterns is considered to facilitate memory formation and consolidation (Walker & Stickgold, 2004; McClelland et al., 1995). These elements could facilitate in enabling lifelong learning in the brain and addressing the challenges of distribution shifts.
3.3 PROPOSED APPROACH
We hypothesize that error sensitivity modulation and the interactions of multiple memory systems that operate on different timescales may be crucial for continual learning agent. To this end, our method aims to incorporate a principled mechanism to modulate error sensitivity based on the history of errors in a dual memory experience replay mechanism. It involves training a working model Ow sequence of tasks, while maintaining two additional memories: an instance- fixed-size episodic memory M, which stores input samples from previous tasks, and a parametric stable model 6,, which gradually aggregates knowledge and builds structured representations.
The CL setting considered here involves learning a sequence of T i.i.d. tasks from a non non-stationary data stream D, E (D1, DT) where the goal is to approximate the joint distribution of all the tasks and distinguish between the observed classes without access to the task identity at inference. In each training step, the model has access random batch of labeled incoming samples drawn from the current task (x,, Vr) ~ D, and memory samples drawn from episodic memory (Imm Um) ~ M.
3.3.1 LEARNING FROM LOw-Loss
To mimic the principled mechanism of error sensitivity modulation in the brain, we maintain a memory of errors along the training trajectory and use it to determine how much to learn from each sample. To this end, we maintain error memory using the momentum update of the mean cross- entropy loss be the batch of samples from the current task. As the stable model is intended to consolidate knowledge and generalize well across tasks, we use it to assess how well the new samples are placed in the consolidated decision boundary and representation space. This also avoids the confirmation bias that can arise from the use of the same learning model.
For each sample i in the current task batch, the cross-entropy loss Lee is evaluated using stable model:
= (J 0.), vi
Subsequently, the weight assigned to each sample for the supervised loss is determined by the dis- tance between the sample loss and the average running estimate p4g
[1
if S 3 pe otherwise
x
(2)
4

---

Published as a conference paper at ICLR 2023
where 3 controls the margin for sample to be considered low-loss. This weighting strategy essentially reduces weight X as sample loss moves away from the mean estimate, and consequently reducing learning from high-loss samples that the model learns more from samples with low-loss. The modulated error-sensitive loss for the working model is then given by the weighted sum of all sample losses in the current task batch: Se -1 [aal X l..
This simple approach to reducing the contributions of large errors can effectively reduce the abrupt drift in representations the task boundary and allow the model to gradually adapt the new task. For instance, when previously unseen classes are observed, the loss for their samples would be much higher than the running estimate; thus, by reducing the weights of these samples, the model can gradually adapt to the new classes without disrupting the learned representations of previously seen classes. This implicitly accounts for the inherent imbalance between new task samples and past tasks sample in memory by giving higher weight to samples in memory, which will most likely have low-loss they have been learned.
The memory of errors is preserved using a momentum update of loss on the task batch on stable model. To prevent sudden changes when the task switches, we have task warm-up period during which the running estimate is not updated. This necessitates the task boundary information during training. To further keep the error memory stable and make it robust to outliers in the batch, we evaluate the batch statistics on sample losses which lie within one std of batch loss mean:
1 = (T' E (1...., [aal]: 1 S μ4s + o.}
(3)
where I/s and as are the mean and std of task losses in stable model, 1,. The mean of the filtered sample losses T,, is then used to update the memory of errors:
He - ai jl + (1 - (1-oz)
(4)
where oy is the decay parameter that controls how quickly the momentum estimate adapts to current values. This provides with an efficient approach to preserve a memory of errors that informs the modulation of error sensitivity.
3.3.2 DUAL MEMORY SYSTEM
To mimic the interplay of multiple memory systems in the brain and employ different learning timescales, our method maintains a parametric semantic memory and a fixed-size episodic memory.
Episodic Memory enables the consolidation of knowledge through interleaved learning of samples from previous tasks. We propose ErroorSenssitvv Reservoir Sampling that utilizes the memory of errors inform Reservoir Sampling (Vitter, 1985). Importantly, we perform a pre-selection of candidates for the memory buffer, whereby only the low-loss samples in the current batch of task samples are passed to the buffer for selection:
(5) This ensures that only the learned samples from the current batch are added to the buffer, thereby preserving the information and reducing the abrupt drift in the learned representations. It can also facilitate the approximation of the distribution of the data stream by filtering out outliers and noise.
(xe, Uc) = ((1,, Mb) (az, Us) E Dr, 12 S B- }
Semantic Memory is maintained using a parametric model (stable model) that slowly aggregates and consolidates the knowledge encoded in the weights (Krishnan et al., 2019) of the working model. Following Arani al. (2021), we initialize the stable model with the weights of the working memory, which thereafter keeps a momentum update of that:
a0,- + (1- aw) rf> 10
where a is the decay parameter that controls how quickly the semantic model adapts to the working model and is the update rate to enable stochastic update throughout the training trajectory.
Stable model interacts with episodic memory to implement dual memory replay mechanism to facilitate knowledge consolidation. extracts semantic information from buffer samples and uses the relational knowledge encoded in the output logits to enforce consistency in the functional space. This semantic consistency loss encourages the working model to adapt the representation and deci- sion boundary while preserving the relational structure of previous classes. The loss the memory samples is thus given by the combination of cross-cntrops loss and semantic consistency loss: L
Lae (X Ow)) (.)) + Le (ʃ (xm; Ow), , (xm, 0.))
5

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of CL methods in different settings with varying complexities and memory constraints. We report the mean and one std of runs. For complction, we provide results under Task- IL setting by using the task label to select from the subset of output logits from the single head.
Seq- CIFACI0
Beq-CIFARIOG 100
GCIL
Buffer
Method
Class-IL
Task-IL
Class-IL
Task-IL
Uniform
Longtail
JOINT
92.20-0
98.31
70.62+064
58.59+1.0
58.42-1.3
SGD
19.62
61.024333
17.5810.04
40.462099
10.38
9.61
ER
41.10-1.10
89.34+1.14
19.99
54.88
14.43an36
13.22x0.0
DER++
55.66.0.7
89.27,2.28
25. 22.22
16.21
21. 21.168
20.29
100
ER-ACE
52.55
88.96s1.21
31.28
59.23s1.39
23.76st.61
23.05
CLS-ER
62.30 .988
93.10 10
38.87
70.29.0.45
33.42:0..0
33,92:0.9
ESMER
65.37
93.32an.15
42.89
73.00
35.77
34.26:0.26
ER
44.79-1
91.1940.4
21.91 +0.36
61.34to.1
16.52+o.10
16.20+a.00
DER+
64.88
91.9240.60
30.68
62.7 20.68
27.732093
26,48s201
200
ER-ACE
62.08
92.20ta.57
17.17
63.09
27 27.4420.64
25.2941.89
CLS-ER
66. 10=075
93.90sm.00
43.80+1.99
73.49+1.04
35.88en.1
35.67=0.7
ESMER
69.16-054
94.10+a.3s
48.77
75.86=096
39.00
37.98+0.5x
where the mean squared loss is used the semantic consistency loss Lae and controls the strength of consistency loss. The overall loss of the working model is the sum the loss on the new task samples and the buffer samples, that is, L. L
C
Slow aggregation of knowledge in stable model further helps mitigate the abrupt change in represen- tations. Even after potentially disruptive initial updates the task boundary, stable model retains the representations of the previous classes, and the enforced consistency loss encourages the working model to retain the relational structure among the previous classes and not deviate too much from them. Therefore, the two components complement each other, and the modulation of error sensi- tivity coupled with the slow acquisition of semantic knowledge reduces the drift parameters and hence forgetting. For inference, we use stable model, as it builds consolidated representations that generalize well across tasks (Figure S1 and Table S3). See Algorithm for more details.
CONTINUAL LEARNING SETTINGS
To faithfully gauge progress the field, it is imperative to evaluate methods on representative ex- perimental settings that measure how well they address the key challenges of CL in the real world. To this end, we evaluate scenarios that adhere to the proposed desideratas (Farquhar & Gal, 2018) and present a different set of challenges. Class Incremental Learning (Class-IL) presents the most challenging scenario (van de Ven & Tolias, 2019) where each task presents a new set of disjoint classes and the agent is required to distinguish between all observed classes without access to the task identity. However, the Class- IL setting assumes that the number of classes per task remains constant, classes do reappear, and the training samples are well-balanced across classes. Gen- eralized Class-IL (GCIL) (Mi et al., 2020) alleviates these limitations and offers more realistic setting where the number of classes, the appearing classes, and their sample sizes for each task are sampled from probabilistic distributions. We further emphasize that noise is ubiquitous in the real world; therefore, the learning agent required not only to avoid forgetting, but also learn under noisy labels. We consider the Moisy-Class-Il setting, which adds varying degrees of symmetric noise in the Class-IL setting. Details of the datasets used each setting are provided Section A.3.
EMPIRICAL EVALUATION
We compare our method with strong rehearsal based methods with single memory (ER, DER++, ER-ACE) and multiple memories (CLS-ER) under different CL settings and uniform experimental settings (details in Section A. in Appendix). Table provides the results for the Class-IL IL and GCIL settings with different data complexity and memory constraints.

---

Published as a conference paper at ICLR 2023
Table 2:
Performance under extremely low buffer sizes Seq-CIFAR10 and different number of tasks on Seq-CIFARI00 with 200 buffer size. We report the mean and one std of 3 runs.
Seq- (Buffer)
C 100 (#Tasks)
Method
200
100
50
10
5
10
20
ER
44.7921.0
41.10a1.1
12.
22.20a136
21.91
14.17
9.97=0.88
DER++
64.88
55.66:078
49.36s2.12
29.04.265
30.68
25.50a3.0
20.50a144
ER-ACE
62.08
52.55a300
43.92a3.74
26.26x2x8
35.1721.17
25.75a1.56
18.68
CLS-ER
66. 10
62.30.1.58
53.80s3.4
33.87a471
43.80+1.8
35.42a0:47
25. 15.98
ESMER
69.1620.54
65.37=m0N
59.94
36.83.1.53
48.77
36.372046
27.26
Table 3: Effect of varying degrees of label noise on CIq-CIFARIO and Beq-CIFARI00 datasets. All experiments use a buffer size of 200. We report the mean and one std of runs.
Seq-CIFAR10
Seq-CIFAR10n 100
Method
Clean
10%
25%
50%
Clean
10%
25%
50%
ER
44. 79.189
36.972180
25.54:214
19.72.1.s
21.91
17.12s036
13,00sa4
8.01
DER+
64. 18
53.
43.17
32.95.2.43
30.68a1.s
25.13
18,99
11.04.o.z
ER-ACE
62. 08..4
50,39a129
37.16.1m
30.14
35.1 1
23,77
13,59
5.71
CLS-ER
66. 10.75
54,86=186
34,49s223
23.97+1.99
43 13.80
35.19
24.59
12.1 16
ESMER
69.16
63.24-1.1
56.65.1m
42.72
48.77
41.80
33.49.1.9
20.68
Class-IL requires the method to transfer and consolidate knowledge while also tackling the abrupt representation drift at the task boundary when new classes are observed. The consistent performance improvement in both considered datasets demonstrates the proficiency of ESMER in consolidating knowledge with low memory budget. Generally, we observe that multiple-memory systems per- form considerably better in these challenging settings. Note that Seq-CIFAR 100 is a considerably more complex setting terms of both the recognition task among semantically similar classes and the potential representation drift in the split setting, which introduces 20 unobserved classes with high semantic similarity previous classes. The improvement of ESMER over the well-performing CLS-ER in this challenging setting suggests that our error-senstiivtty modulation method success- fully mitigates representation drift and reduces interference among similar tasks. The performance of our method under extremely small buffer sizes in Table 2, which exacerbates abrupt representa- tion drift, further supports the claim. This gain is promising, given that CLS-ER uses two semantic memories with different update frequencies, while our method uses only one.
Table also provides the results for the GCIL setting, which further requires the model to tackle the challenges of class imbalance, sample efficiency, and learn from multiple occurrences of an abject. ESMER provides a consistent improvement in both uniform and longtail settings. The error-sensitiiity modulation and consequent gradual adaptation to unobserved classes allow the model to deal with the class imbalance and learn over multiple occurrences well. Considering a new task with mix of unobserved and recurring classes, the loss of samples from the recurring classes would most likely be low, as they have been learned earlier, while the loss on unobserved classes would be significantly higher. Consequently, the contribution of unobserved classes would be down- weighted, allowing the model to focus and adapt the representations of the recurring class with reduced interference and representation drift from the unobserved classes. Learning more from low errors also provides more consistent learning and can improve sample efficiency by avoiding potentially suboptimal gradient directions caused by outliers.
Finally, Table 3 provides the results under the Clasy-C setting where the agent is required to concurrently tackle the challenges of learning from noisy labels. Our strong empirical results demonstrate the superiority of ESMER in tackling the unique challenges of this setting. Learning from low errors implicitly makes our model more robust to label noise and reduces memorization since noisy labels would correspond to higher errors. Furthermore, the error-sensitive reservoir sampling approach ensures a higher purity of the memory buffer, which plays a critical role in noisy CL. as buffer samples replayed multiple times, further aggravating their harmful effect.
7

---

Published as a conference paper at ICLR 2023
Table 4: Ablation Study;
Effect of systematically removing different components of ESMER on performance on Seq-CIFARI0 with different buffer sizes and label noise (for 200 buffer size).
Error Sensitivity
Stable
Error Sensitive
Buffer
Label Noise
Modulation Model
Reservoir Sampling
50
100
200
25%
50%
59.94
65.37
69.162254
56.61=1.4
42.43
57.53
64.444am
69.07.4xx7
51.16-156
34.42-2m
37.1
21.59
63.44a144
38. 38.84
22.10
X
36.03
42,56
54,58s2.
38
24.70
X
X
32.5
41.10zau
44.79a1.56
25.54:214
19.72a153
(a) Representations Drift
(b) Recency Bias
100
0.6
Taok
Task Performance (%)
Task
80
Task Probability
0.5
Task
Task
Tank
60
0.4
0.3
0.2
20-
CR
ESMER (SM)
ESHER (Wh)
250
500
750
1000
1250
1500
0.0
Eh
DER+
CLS-ER
ESMER
Training Task
Method
Figure 2: (a) Effecct Task performance the models are trained on Task 2. SM and WM denote stable model and working model (b) The average probability of predicting each task at the end of training Seq- CIFARIO with 200 buffer size. Figure S2 provides recency bias for other settings.
WHY DOES ESMER WORK?
Here, we attempt to provide insights into what enables the gains with ESMER. We perform our analysis on Beq-CIFARIDE 10 with 200 buffer size and compare it with ER, DER++ and CLSER.
We systematically remove the salient components of our method and evaluate their effect on performance in Table 4. Error-sensitivity modulation provides considerable gains in both low-buffer and noisy label settings. Furthermore, the gradual accumulation of knowledge in stable model proves to be highly proficient approach to consolidating knowledge. Finally, the proposed error-sensitive reservoir sampling technique provides further generalization gains, particularly the Noisy-Class-II setting, where maintaining clean buffer is critical. Importantly, these components complement each other so that the combination provides the maximum gains.
Ablation Study:
Drift in Representations: To assess ESMER in mitigating the abrupt drift in representations the task boundary, we monitor the performance of the model on Task as learns Task 2. Figure 2 (a) shows that ESMER successfully maintains its performance, while ER suffers sudden performance drop and fails to recover from earlier disruptive updates. Furthermore, while the working model ESMER undergoes reduced representation drift after the task warmup period, the stable model retains the performance and provides a stable reference to constrain the update of the working model, helping it recover from the disruption. This suggests that error-sensitivity modulation, coupled with gradual adaptation in stable model, is a promising approach to mitigate abrupt representation drift.
Robustness to Noisy Labels: We probe into what enables ESMER be more robust to label noise. One of the key challenges in learning under noisy labels is the tendency of DNNs to memorize in- correct labels (Arpit et al., 2017) that adversely affect their performance (Sukhbaatar et al., 2014). To evaluate the degree of memorization, we track the performance of the model on noisy and clean samples during training. Figure 3 (a) shows that ESMER significantly reduces memorization, lead- ing to better performance. Furthermore, the quality of the samples in the memory buffer play critical role the performance of the model, as the memory samples are replayed multiple times. Figure (b) compares the percentage of noisy labels in the buffer with proposed emoorsensitive reservoir sampling and the typical reservoir sampling approach. Our approach considerably reduces the probability of adding noisy sample in episodic memory and maintains higher quality of the
8

---

Published as a conference paper at ICLR 2023
(a)
(b)
ER
ESMER
Quality of Buffer
Aym
Computed Labes
Arcuncy
Arcumcy
13
C.can
Clean
10-
Nrnsy
12-
Nraay
ESMER
200
Epoch
Epoch
Epoch
Figure 3: (a) Accuracy of the models clean and noisy samples as training progresses Seq- CIFARIO with 50% label noise. lower value better as it indicates less memorization, See
Figure S3 for the working model. (b) Percentage of noisy labels in the memory with reservoir sampling in ER and error-sensitive reservoir sampling in ESMER.
ER
+
Task 98.8
98.2
98.7
98.1
Training
67.0
89.5
67.7
Task
54.0
16.9
95.8
82.0
50.0
90.0
78.2
53.5
89.0
94.2
61.4
63.7
Task 55.9
13.1
22.9
98.7
79.3
33.4
63.0
94.8
81.2
42.4
87.5
55.1
70.1
65.7
36.0
81.1
83.3
59.4
68.7
Tork
Task
S
Tekk
Rask
Teck
Taak
Taak
Taak
Taok
Figure 4: The performance of the models (x-axis) after training on each task in Ceq-CIFARIOO with 200 buffer size. See Figure S4 for other datasets and buffer sizes.
buffer. Note that both features byproduct of mimicking error-sensitivity modulation in the brain rather than being explicitly designed for the setting.
The sequential learning of tasks with standard SGD in CL induces recency bias in the model, making much more likely to predict newly learned classes (Hou et al., 2019). To evaluate the task recency bias of the different methods, we calculate the average probability of predicting each task at the end of training by averaging the softmax outputs of the classes associated with each task the test set. Figure (b) shows that ESMER provides across tasks. Generally, methods that employ multiple memories are better at reducing the recency bias, and the dynamics of low-loss learning in our method further reduce the bias. Figure 4 further shows that ESMER maintains better balance between the plasticity and the stability of the model.
Recency Bias:
uniform predictions
7 CONCLUSION
Inspired by the dynamics of error-based learning and multiple memory systems in the brain, we proposed ESMER, a novel approach for error sensitivity modulation based experience replay in a dual memory system. The method maintains a memory of errors along the training trajectory. which used modulate the contribution of incoming samples towards learning such that the model learns more from low eπrоrs. Additionally, semantic memory is maintained that gradually aggregates the knowledge encoded in the weights of the working model using momentum update. We also proposed an error-sensitive reservoir sampling approach that prioritizes the representation of low-loss samples in the memory buffer, which are better suited for retaining knowledge. Our empirical evaluation demonstrates the effectiveness of our approach in challenging and realistic CL settings. ESMER mitigates abrupt drift in representations, provides higher robustness to noisy uniform performance across tasks. Our findings suggest that principled mechanism for learning from low errors in dual-memory system provides a promising approach to mitigating abrupt representation drift in CL. that warrants further study.
labels, and provides

---

Published as a conference paper at ICLR 2023
REFERENCES
Wickliffee C Abraham. Metaplasticity; tuning synapses and networks for plasticity. Nature Reviews Neuroscience, 95):38773377, 2008. 12
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: general contin- ual learning method based on complementary learning system. In International Conference on Learning Representations, 2021. 3, 12, 13, 14
Devansh Arpit, Stanishaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pp- 233-242. PMLR, 2017. 8
Craig Bailey, Maurizio Giustetto. Yan- You Huang, Robert D Hawkins, and Eric Kandel. het- erosynaptic modulation essential for stabilizing hebbian plasiticity and memory. Nature Reviews Neuroscience, (1)::11200, 2000. 12
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe- rience for general continual learning: strong, simple bascline. arXiv preprint arXhv:2004.07211, 2020. 3, 12
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In Interna- tional Conference on Learning Representations, 2021. 1, 3,
Luis Nicolas Gonzalez Castro, Alkis M Hadjiosif, Matthew A Hemphill, and Maurice A Smith. Environmental consistency determines the rate of motor adaptation. Current Biology, 24(10): 1050-1061, 2014. 2,
Sarah E Criscimagna-
Triscimagna-Hemminger Hemminger, Amy J Bastian, and Reza Shadmehr. Size of error affects cere- bellar contributions to motor learning. Journal of neurophysiology. 103(4):275--2284, 2010.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3385, 2021.
Kenji Doya. Metalearning and neuromodulation. Neural networks, 15(4-6):495-55) 2002. 12
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint
1rX5::1805.09733. 2018. 1,
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 95(2):245-258, 2017. 2,
Tyler L. Hayes, Giri P Krishnan, Maxim Bazhenov, Hava T Siegelmann, Terrence J Sejnowski, and Christopher Kanan. Replay in deep learning: Current approaches and missing biological elements. arXiv preprint TrXiv:2104.04132 2021. 1,
David J Herzfeld, Pavan A Vaswani, Mollie K. Marko, and Reza Shadmehr. A memory of errors in sensorimotor learning. Science, 345(620)) 1349-1353, 2014. 2,
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- -arnskaaaaaiikkaaa..aa al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):32113356 2017. 12
Giri Krishnan, Timothy Tadros, Ramyaa Ramyaa, and Maxim Bazhenov. Biologically inspired sleep algorithm for artificial neural networks. arXiv preprint arXiv: 028.002240 2019.
10

---

Published as a conference paper at ICLR 2023
Dhireesha Kudithipudi, Mario Sgui,, Jonathan Babb, Maxim Bazhenov, Douglas Black- iston, Josh Bongard, Andrew P Brna, Suraj Chakravarthi Raja, Nick Cheney, Jeff Clune, et al. Biological underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196- 210, 2022. 12
Dharshan Kumaran, Demis Hassabis, and James L McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20 (7):512-534, 2016. 2,
Sandeep Madireddy, Angel Yanguas-Gii Gil, and Prasanna Balaprakash. Neuromodulated neural archi- tectures with local error signals for memory-constrained online continual learning. arXiv preprint arXiv:2007.08/59 2020. 12
Mollie K Marko, Adrian M Haith, Michelle D Harran, and Reza Shadmehr. Sensitivity prediction error in reach adaptation. Journal of neurophysiology, 108(6): 2012. 2,
James L McClelland, Bruce L. McNaughton, and Randall C O' O'Reilyy. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Fei Mi, Lingjing Kong, Tao Lin, Kaicheng Yu, and Boi Faltings. Generalized class incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion Workshops, pp. 240 241, 2020. 6, 13
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role of training regimes in continual learning. Advances in Neural Information Pro- cessing Systems, 33, 2020. 12
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:5477,, 2019.
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34:16131-16144, 2021. 3
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arxiv:1810.11910, 2018.
Maurice A Smith and Reza Shadmehr. Modulation of the rate of crror-dependent learning by the statistical properties of the task. Advances in Computational Motor Control, 3:1392-1404, 2004. 4
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv preprint urXiv:1406.200 2014.
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprint 19Xv:1994.007344, 2019. 1,
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.
Matthew P Walker and Robert Stickgold. Sleep-dependent learning and memory consolidation. Neuron, 44(1):121-13, 2004. 2,4
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su. Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning, arXiv preprint arXiv:2204.04799 04799,, 2022. 3
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. Proceedings of machine learning research, 70:3987, 2017. 12

---

Published as a conference paper at ICLR 2023
A APPENDIX
Algorithm Error Sensitivity Modulation based Experience Replay (ESMER) Algorithm
Input and Params: eπror memory params and B; stable model params and semantic consistency weight y data stream D; learning rate Initialize: 44 = 0, M - (}. 8₂ 6m
1: while Training do
2: Sample batch from task stream, (Ir: y/) D,, and episodic memory, (Iim; ymn) M
3: Evaluate loss of task samples on stable model, 12 (Eq. 1)
4: Get the weights of task samples, X (Eq. 2)
5: Evaluate the sensitivity-modulaeee task loss, Sy2 = 1
6:
Evaluate the cross-entropy and consistency loss memory samples, D (Eq. Combine overall loss: L = Ly + L
7:
8:
Update parameters of working model: Ou <- Ow "V
9:
Update the stable model: 0₂ - at, + (1 - a) ow: if r am UO0,1) (Eq. 6)
Filter candidates for episodic memory, (Ies yo) (Eq.
11:
Update episodic memory: .M + Reservoir (M.. (Ees Uc))
Remove outliers from task batch loss: i/ (Eq.
13:
Update error memory if epoch task warm-up period: μe (Eq. 4)
return 0,
A.1 EXPERIMENTAL DETAILS
We conducted all experiments under common settings disentangle the performance improve- ment the different CL. methods from the training settings (Mirzadeh et al., 2020). Following (Arani et al., 2021; Buzzega al., 2020), For all our experiments, we train for 50 epochs with a batch size of 32 for both incoming samples and memory and apply random crop and horizontal flip data augmentations. Note that we store non-augmented images in the buffer and apply the data augmentations to the memory samples for replay. We report the mean and std of runs.
ResNet-18 18 network and train with SGD optimizer.
A.1.1 HYPERPARAMETER TUNING
We use small validation set to select the hyperparameters for all methods. For ESMER, ag 0.99, ag = 0.999, and tune y E {0.15, 2.0}. 3 E [0.8, 3.0] and r E [0. 1]. Note that did not conduct exhaustive parameter search for each experiment, and it is possible that better hyperparameter search can further improve performance. The hyperparameters used for each experiment provided in Table $1.
For the baselines, we use the results from (Arani et al., 2021) similar settings are provided; oth- erwise, conduct grid search over the hyperparameter values used in the original paper. For ER and ER-ACE, we search v) E (0.1, 0.01, 0.001 J. For DER+ we search over E [0.1, 0.01, 0.001). E (0.1. 0.2, 0.5, 1.0} and e {0.5, 1.0). Finally, for CLS-ER, we search over a 0.999, ap = 0.999, {0.15, 2.0). rp E [0, 1] and r. [0, 1]. The parameters selected for the basclines in each setting are provided in Table $2.
A.2 COMPARISON WITH NEUROMODULATION
Metaplasticity (Abraham, 2008) and Neuromodulation (Doya, 2002) refers to different mechanisms to modulate the plasticity rate of individual synapses (strengths / weights of the incoming and out- going connections in DNNs) (Kudithipudi et al., 2022; Bailey et al., 2000). These form the biolog- approaches where the goal is to identify synapses that important for previous tasks and penalize changes them while learning task (Zenke et al., 2017; Kirkpatrick et al., 2017). Neuromodulation has also been explored con- junction with local learning (Madireddy et al., 2020) which shows promise.
ical underpinning behind the popular regularization-base
The error sensitivity modulation in ESMER differs substantially from these methods, as instead of modulating the plasticity rate of individual synapses, we modulate the contribution of each sample
12

---

Published as a conference paper at ICLR 2023
Table S1: Hyperparameters for ESMER. For all experiments, the models are trained for 50 epochs with a batch size of 32 for incoming and memory samples.
Dataset
Buffer
Label noise
1/
ar
8
ax
r
100
0.03
0.99
1.2
0.15
0.999
0.07
C0q-CIFARI00
200
0.03
0.99
1.0
0.15
0.999
0.07
100
0.05
0.99
2.5
0.2
0.999
0.2
GCIL Unif'
200
0.05
0.99
2.5
0.2
0.999
0.2
100
0
0.05
0.99
3.0
0.2
0.999
0.2
GCIL Longtail
200
0.05
0.99
2.5
0.2
0.999
0.2
10
0.03
0.99
0.9
0.15
0.999
0.1
50
0.03
0.99
0.9
0.15
0.999
0.1
1e0-CIFAR10
100
0.03
0.99
0.9
0.15
0.999
0.1
200
0.03
0.99
1.2
0.15
0.999
0.1
200
10%
0.03
0.99
0.9
0.15
0.999
0.1
200
25%
0.03
0.99
0.9
0.15
0.999
0.1
Seq-CIFA--IOOO
200
50%
0.03
0.99
1.0
0.15
0.999
0.1
the batch based on its error magnitude. Our work aims to bring about a salient characteristic the learning dynamics of the brain: it learns more from small consistent errors compared to large errors. This is not the case for standard ANNs, where learning scales linearly with the error size. Therefore, the error sensitivity modulation in ESMER specifically focuses utilizing memory of errors to modulate the contribution of samples to learning based on their error magnitude, so that the model learns more from small errors. Please note that our approach is complementary to metaplasticity and neuromodulation-buu based approaches, and the interaction of these biologically plausible mechanisms an interesting avenue of research.
A.3 DATASETS
Here we provide details of the datasets used in each of the considered CL setting.
A.3.1 CLASS-IL
For Class-IL,, the common sequential versions of CIFARIO (Seq-CIFARIO) and CIFAR100 (Seq-CIFAR 100) datasets whereby the classes are split into five tasks each containing a disjoint set of and 20 classes, respectively. We do vary the order classes, and so, for instance, Task in Seq-CIFARI 10 always contains the first two classes.
Note that method does utilize separate classification heads or subnets. For completion, also evaluate the performance of the same model trained for Class-IL under the Task-IL setting where the model assumes availability of task labels at inference. We use the task label to narrow down predictions to the subset of output logits belonging to classes belonging to the task.
A.3.2 GCIL
Following (Mi al., 2020; Arani et al., 2021), we apply the GCIL setting on CIFAR100 dataset. The setting involves training 20 tasks sequentially, and for each task, the number and appearing classes are sampled from probability distribution but capped at maximum samples drawn from either uniform (Unif) or longtail sample distributions. For further details, see (Mi et al., 2020).
and subsequently 1000
Importantly, found that even for fixed dataset seed, the corresponding tasks in the Longtail and Unif settings differ not only in the sample distribution (as expected) but also significantly in the number of classes drawn and the appearing classes. For instance, with the dataset seed set to 1993, the last task in the Longtail setting contains classes, whereas the corresponding task the Unif setting contains only 17 classes. This makes the former task significantly easier.
13

---

Published as a conference paper at ICLR 2023
Table S2: Selected hyperparameters for baselines. All methods trained for 50 epochs with batch size of 32 for incoming and memory samples. For CLS-ER, a, ap =0.999
Dataset
Buffer
Method
Hyperparameters
ER
0=0.1
ER-ACE
0000
100
DER+
00033, a=0.1, 9=0.1
CLS-ER
00 A=0.15, r,=0.111 r,0005
Seq-CIFAR100
ER
7001
ER-ACE
700.1
200
DER+
1/003 a=0.2. 3=0.1
CLS-ER
001. A=0.15, 7p r,=00,,
ER
001
ER-ACE
700.1
100
DER+
70003, a=0.5 3=0.1
CLS-ER
7,
GCIL-Unif
ER
00
ER-ACE
100.1
200
DER+
00003 o=0.2, 9=0.1
CLS-ER
001 A=0.1, 7,
ER
00.
ER-ACE
70011
100
DER+
10003, o=0.2, 9=0.1
CLS-ER
7=01 X=0.1, ,,07.,,,-
GCIL-Longtail
ER
700.
ER-ACE
700.1
200
DER+
70003 a=0.5, 3=0.1
CLS-ER
0.1 A=0.1, rp 0p=0.7, =,=
ER
0.1
ER-ACE
70001
10
DER+
70003, a=0.5, 3=0.5
CLS-ER
0.11 A=0.15, -03. r,0044
ER
00.
ER-ACE
70001
Seq-CIFAR10
50
DER+
00.03, o=0.2, 3=0.5
CLS-ER
1=0.15, r,-03,, r,,000
ER
7=0.1
ER-ACE
0001
100
DER++
0.03, a=0.2, 5=0.5
CLS-ER
0.1 A= r, r.00.
ER
0-0.1
ER-ACE
70001
Noisy-Seq- Masyy-Se--CIFA Seq-CIFAR10
200
DER+
00003, a=0.1, 3=0.5
CLS-ER
001 A= rp=0.3, 0.1
We fix this in our implementation by drawing 20 random numbers (after setting the random seed to 1993) when the dataset is initialized and setting the same unique seed before sampling each task. This ensures that the same number of classes and the appearing classes (if not all the samples for particular class are seen already) are drawn for the two settings. This provides a better evaluation of the effect of the difference in data distributions between the two variants.
This leads to a significant difference baseline performance compared to those reported in (Arani et al., 2021). ). We also perform the hyperparameter for all the considered methods and find that, for the modified setting, our choice parameters performs considerably better than the ones used in (Arani et al., 2021). We report the results optimal hyperparameters provided in Table $2.
14

---

Published as a conference paper at ICLR 2023
Table S3: Working model and stable model performance all the considered CL scenarios, Stable model consistently provides better performance.
Dataset
Buffer
Label Noise
Stable Model
Working Model
100
42.89
27.562126
Seq-CIFARI00
200
48.77
37.87+129
100
35. 35.77a0.32
23.362081
GCIL Unif
200
39.00a1.14
29.55a124
100
34.2610.26
22.944028
GCIL Longtail
200
37.98.0.8
28.592094
36.83
26.35
50
59.94 39.94.1.67
44.98a3.18
Seq-CIFAR10
100
65.37
55.80s244
200
69.1640.54
65,
200
10%
63. 63.24.1.1
56.01+1.44
200
25%
56.6521.01
48. 48.76.1557
Noisy-Seq- Sen--I-A-CIOOOO
200
50%
42.72,270
34.24.2.6
A.3.3 NOISY-CLASS-IL
Noisy Class-IL aims evaluate the robustness of learning agent label noise. Since noise is ubiquitous in real-world streams, idcally, a CL method should be able to tackle the challenges of learning under noisy labels and CL concurrently. This requires them avoid memorization and forgetting. We consider varying degrees of symmetric label noise on Seq-CIFAR 100 dataset, Noisy- Seq-CIFAR10. For each task, we sample random labels from the classes in the task and replace fraction of the original labels with the random ones. Note that, under this setting, the random label may correspond to the original label.
A.4 WORKING MODEL PERFORMANCE
To assess how well stable model aggregates knowledge across the task, we compare its performance with the performance of the working model across all datasets and memory constraints in Table S3, Stable model considerably provides higher performance. To better understand, we look the task- wise performance on the test set at the end of each task in Figure S1. Stable model retains more knowledge of previous tasks and consolidates information to build representations that generalize across tasks.
A.5 EFFECT OF HYPERPARAMETER
Tuning ESMER involves setting the hyperparameters for two complementary components: error sensitivity modulation (3, oz) and stable model (r, a, 7). To understand the interaction of these components and the sensitivity of our method to hyperparameters, we look at the performance of the model with different sets of values for the most pertinent parameters (8, r). Table S4 shows that the loss margin, 8, and the update frequency, complement each other and, generally, a higher requires lower 3. Furthermore, different set of values can provide similar performance, which significantly facilitates hyperparameter tuning. As can be seen in Table S1, the majority of parame- ters (a,, ai and 7) can be fixed, and the optimal parameters do not vary much for different memory budgets.
15

---

Published as a conference paper at ICLR 2023
Seq-CIFAR10
Seq-CIFAR100
Working Model
Stable Model
Working Model
Stable Model
After T1
$6.2
95.9
$1.0
75.5
100
After 12
76.7
88.7
93.0
66.6
27.9
86.3
68.0
03.0
Buffer =
25.6
50.8
89.7
14.5
29.9
46.6
61.7
12.4
52.9
95.2
74.0
7.2
22-1
22.4
42.0
Buffer 200
After
70.3
After 13
83.9
45.8
88.8
94.2
61.4
63.7
34.4
43.2
$1.5
611.8
65.8
50.1
After 14
74.5
40.9
53.5
94.8
30.6
55.1
70.1
65.7
29.3
30.0
38.1
81.7
50.6
54.9
53.8
56.0
Afte
61.4
21.3
Figure S1: Comparison between the task-wise performance (x-axis) of working model and stable model after training on each task. Stable model provides higher stability and provides more uniform performance across the tasks.
Table S4: Performance of the models trained on Seq-CIFARIO with different hyperparameters, We use 1/00.33, ay=0.99, a=0.999 and 20.15 for all the experiments.
Buffer
Stable Model
Working Model
0.8
78
53.76=204
0.08
1.0
60.67.2.0
54.8
1.2
63.6
54.26a1
0.8
61.2
52.86a2.
100
0.10
1.0
65.37
55.80
1.2
62.73a242
51,432390
0.8
59.63+1.7
55.96+144
0.20
59. 59.29+1.17
55.20
1.2
57.49
52.98
0.8
59.06+298
62.15
0.08
1.0
62.47+075
64.34-o87
1.2
63.6410.55
64.0020s3
64. 42..5
63.82s081
200
0.1
66.7941.4
63.7821166
1.2
69. 1620.44
65.18
0.8
64.81+1.50
61.70+1.55
0.2
-7.37-148
64.40+174
1.2
66.36+a.1
62.93-a50
16

---

Published as a conference paper at ICLR 2023
Seq-CIFAR10
Seq-CIFAR100
Taok
Taak
Tosk
3.5
Toskk
Taskk
Thisk Probability
Buffer 100
0.8-
82
0.0
1.0
0.8
Buffer 200
Task Probablitt
0.4
02-
ER
DER+
ER-ACE
CLS-ER
ESMER
ER
DER
FRAÁCE
CLS-ER
ESMER
Figure S2: Average probability of predicting each task at the end of training for different datasets and buffer sizes. ESMER provides more uniform predictions.
ER
ESMER (WM)
ESMER (5M)
Nolky
Accurnty 1900
200
Epoch
Epoch
Figure S3: Accuracy of the models on clean and noisy samples training progresses. We compare ER with the working model (WM) and stable model (SM) in ESMER on Seq-Cifarl0 with 50% symmetric noise and 200 buffer size. The working model initially has slightly higher memorization than SM but then recovers with the aid of the consistency loss from the stable model.
17

---

Published as a conference paper at ICLR 2023
Seq- leq-CIFAR10
Seq-CIFAR100
Buffer 100
Buffer 200
Buffer 100
Buffer 200
After T2
45.5
91.7
67.0
92.1
7.0
84.8
15.2
83.8
After T3
35.6
10.5
95.8
54.0
16.9
95.8
1.2
8.8
83.3
4.7
13.3
62.5
After
14.5
After
52.7
9.7
After T2
84.1
89.4
89.2
87.3
18.9
85.6
26.9
85.2
DER++
After
73.6
54.4
79.7
82.00
50.0
90.0
7.5
17.6
84.3
20.9
40.4
61.5
27.6
95.7
After T2
86.7
81.5
91.6
82.8
36.3
73.3
43.5
74.8
ER-ACE
After
81.4
51.4
70.3
85..3
64.2
70.0
20.0
42.7
71.2
27.9
44.0
71.8
After
46.5
66.0
43.9
62.
33.6
39.8
66.2
After
79.3
87.8
89.0
89.5
65.2
60.7
73.4
49.4
CLS-ER
After T3
76.2
81.5
40.8
78.2
53.5
89.0
40.8
63.8
61.6
-56.0
66.5
52.0
After
66.5
71.8
20.1
8:.22
42.4
16.3
87.5
45.4
61.0
54.1
39.1
71.5
37.6
52.1
50.1
48.7
After
93.0
66.6
94.3
67.7
68.0
65.0
70.3
63.4
ESMER
After
89.7
52.7
65.8
94.2
61.4
63.7
46.6
61.7
65.2
61.8
65.8
50.1
After 88.2
44.2
68.6
70.7
90.6
55.1
70.1
65.7
30.4
47.6
57.2
61.3
50.6
54.9
53.8
56.0
68.7
11
13
74
T1
13
74
11
T3
74
11
T3
74
Figure S4: Taskwise performance of the models (x-axis) after training each task different datasets with varying memory budget. ESMER provide a good tradeoff between the stability and plasticity of the model.
18


Reviews:
[{"review_id": "wGgRt1YA0Ak", "reviewer": "Reviewer_pRxq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "= Update after author response =\n\nI thank the authors for their clarifications. I maintain  my recommendation for acceptance.\n\n= Original review = \n\nThe paper proposes a new approach to replay-based continual learning. The purpose of this approach is to prevent catastrophic representation drift caused by the large errors that occur when new tasks are introduced.\n\nThe approach maintains a slow-changing version  of the main learning network, clips sample losses if they are too far above a running average of previous losses (as estimated by the slow-changing network), penalizes disagreement with the slow-changing version for replay loss, and actually uses the slow-changing network at inference. \n\nThe method is compared to different variants of the so-called ER method  (another replay-based method) and found to improve performance.", "summary_of_the_review": "This new method for improving stability of continual learning seems interesting and promising. If more knowledgeable reviewers are satisfied that the set of baseline methods used for performance comparison is representative, I have no objection to acceptance (pending clarifications as requested above).", "strength_and_weaknesses": "- Strengths:\n\nThe method seems novel and interesting, and the results are promising.\n\n- Weaknesses:\n\nOne possible concern is that all baselines used for comparison are variants of the ER method. Since this is a bit beyond my  field I can't really assess whether this is an adequate representation of existing replay-based methods. If this is acceptable to other reviewers, then I can't see anything much wrong with the substance of paper.\n\nAs a minor point, the presentation is at times a bit confusing and could be improved, see below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "a9NYmRyMM7e", "reviewer": "Reviewer_CEB8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work builds upon the notion of a complementary learning system that consists of multiple memories: episodic memory and the semantic memory in this case. Although this notion is utilized in many works, the main contribution of this work is in designing the episodic memory to contain samples from the current batch that are pre-selected based on their distance from the mean of an errors of all samples along the training trajectory. These errors are also used to modulate the learning. The results on Seq-CIFAR-10, Seq-CIFAR-100 and GCIL show good accuracies in low memory buffer scenarios as well as in the presence of label corruptions. ", "summary_of_the_review": "•\tThe related works and consequently the approaches compared needs to be updated.\n\n•\tThe effect of increase in the memory for dual memory approaches need to be considered when comparing with other approaches.\n\n•\tAlternative continual learning metrics and task sequence need to be considered.\n\n\n\n", "strength_and_weaknesses": "Strengths: \nThe idea of modulating the learning based on the error is well motivated and using it to create an episodic memory buffer seems improve the continual learning accuracy in the clean as well as label corruption settings\n\nWeaknesses: \n\n•\tThere is a significant overhead introduced by the dual memory mechanism so it will be fair to compare with other single memory approaches by maintaining a fixed overall memory. \n\n•\tIn general, the related works need to be more comprehensive and explicit in explaining how the proposed work is different from the literature and compare if needed [6] : The idea of error sensitivity-based modulation seems to be explored before in the local learning-based approaches [1,2,3]. Why only compare with replay-based approached when comparing the performance. Other non-replay-based approaches have shown superior performance and have the advantage of not requiring the memory buffer [2,4].\n\n•\tNot clear if the improved results hold when a larger memory buffer (eg., 5k) is used.\n\n•\tOnly a single metric (accuracy) is used for comparison. Other metrics such as average forgetting [5] shall be used to\n\n•\tEffect of task sequence not considered! do the results hold when the task sequence is changed?\n\n•\tWhy does ESMER have lower accuracy (on test data) for the task it is trained on? As seen in figure 3 diagonal elements.\n\n•\tThe ablation does not show the effect of just keeping the error sensitivity modulation without the semantic memory or reservoir sampling.\n\n•\tDo the results on label corruption (fig2) hold for Cifar-100 data as well?\n\n•\tDoes this approach work in the online continual learning setting?\n\n[1] Dellaferrera, G., & Kreiman, G. (2022). Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass. arXiv preprint arXiv:2201.11665.\n\n[2] Madireddy, S., Yanguas-Gil, A., & Balaprakash, P. (2020). Neuromodulated neural architectures with local error signals for memory-constrained online continual learning. arXiv preprint arXiv:2007.08159.\n\n[3] Kudithipudi, Dhireesha, et al. \"Biological underpinnings for lifelong learning machines.\" Nature Machine Intelligence 4.3 (2022): 196-210.\n\n[4] Li, S., Du, Y., van de Ven, G. M., & Mordatch, I. (2020). Energy-based models for continual learning. arXiv preprint arXiv:2011.12216.\n\n[5] Mai, Zheda, et al. \"Online continual learning in image classification: An empirical survey.\" Neurocomputing 469 (2022): 28-51.\n\n[6] Pham, Q., Liu, C., & Hoi, S. (2021). Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34, 16131-16144.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MkAmHyCWF-1", "reviewer": "Reviewer_ZgEC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new experience replay method using error sensitivity as a modulation method in continual learning. The proposed method keeps track of the classification loss during continual learning and when the newly-received data incurs a high classification loss, it gets downweighed to reduce its effect. In this way, the learned representations are more stable and suffer less from abrupt drift during task transition. The paper also adopts a semantic memory (a momentum-updated model) to further stabilize the continual learning process, following a prior work, CLS-ER. Experiments show that the proposed method achieves better performance in both standard continual learning benchmarks and settings with label noise. ", "summary_of_the_review": "This is a great paper presenting a novel method and strong results. It would be better if more experiments can be performed on some more complex datasets. ", "strength_and_weaknesses": "Strength:\n1. The proposed method considers the large errors incurred by data from new tasks as a potential reason for abrupt presentation drift in continual learning. Based on this insight, the paper proposes to downweigh the data points with large errors to reduce the drift and the experiments show that it is very effective. \n2. The experiments show that the performance is strong, surpassing CLS-ER even with one fewer model to keep track of. (Only the stable model instead of both plastic and stable models in CLS-ER)\n3. The experiments on continual learning with noisy labels are interesting. \n4. The paper is well-written and easy to follow. \n\nWeakness:\n1. The proposed method is only tested on small and relatively simple datasets like CIFAR10 and CIFAR100. I think the results are more complete and convincing if results on tiny/Mini/Full ImageNet are also reported and compared. \n2. The proposed method has high complexity and many hyperparameters. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster