New Insights on Reducing Abrupt Representation Change in Online Continual Learning
Authors: Lucas Caccia
Rahaf Aljundi
Nader Asadi
Tinne Tuytelaars
Joelle Pineau
Eugene Belilovsky
Venue: ICLR 2022 Poster (2022)
Keywords: continual learning

Abstract: In the online continual learning paradigm, agents must learn from a changing distribution while respecting memory and compute constraints. Experience Replay (ER), where a small subset of past data is stored and replayed alongside new data, has emerged as a simple and effective learning strategy. In this work, we focus on the change in representations of observed data that arises when previously unobserved classes appear in the incoming data stream, and new classes must be distinguished from previous ones. We shed new light on this question by showing that applying ER causes the newly added classes’ representations to overlap significantly with the previous classes, leading to highly disruptive parameter updates.  Based on this empirical analysis, we propose a new method which mitigates this issue by shielding the learned representations from drastic adaptation to accommodate new classes. We show that using an asymmetric update rule pushes new classes to adapt to the older ones (rather than the reverse), which is more effective especially at task boundaries, where much of the forgetting typically occurs. Empirical results show significant gains over strong baselines on standard continual learning benchmarks.

Published as a conference paper at ICLR 2022
NEW INSIGHTS ON REDUCING ABRUPT REPRESENTA- TION CHANGE IN ONLINE CONTINUAL LEARNING
Lucas Caccia" McGill University, Mila Facebook AI Research
Rahaf Aljundi Toyota Motor Europe
Nader Asadi Concordia University, Mila
Tinne Tuytelaars KU Leuven
Joelle Pineau
Eugene Belilovsky Concordia University, Mila
McGill University, Mila Facebook AI Research
ABSTRACT
In the online continual learning paradigm, agents must learn from a changing distri- bution while respecting memory and compute constraints. Experience Replay (ER), where small subset of past data is stored and replayed alongside new data, has emerged simple and effective learning strategy. In this work, focus the change in representations classes appear in the incoming data stream, and new classes must distinguished from previous ones. We shed new light on this question by showing that applying ER causes the newly added classes' representations to overlap significantly with the previous classes, leading to highly disruptive parameter updates. Based on this empirical analysis, we propose a new method which mitigates this issue by shielding the learned representations from drastic adaptation to accommodate new classes. We show that using an asymmetric update rule pushes new classes to adapt to the older ones (rather than the reverse), which is more effective especially at task boundaries, where much the forgetting typically occurs. Empirical results show significant gains strong baselines on standard continual learning benchmarks
observed data that arises when previously unobserved
INTRODUCTION
Continual learning is concerned with building models that can learn and accumulate knowledge and skills over time. A continual learner receives training data sequentially, from a potentially changing distribution, over the course of its learning process. The distribution change might be either a shift in the input domain or new categories being learned. The main challenge is to design models that can learn how to use the data and acquire new knowledge, while preserving improving the performance on previously learned data. While different settings have been investigated of how new data being received and learned, we focus the challenging scenario of learning from an online stream of data with new classes being introduced at unknown points in time and where memory and compute constraints are applied on the learner. Additionally, we assume shared output layer among all the learned classes (Aljundi et al. 2019b). This setting is different and harder than the conventional multi-head setting (Farquhar Gal 2018) where each new group of classes is considered as new task with a dedicated head (classification layer), requiring a task oracle at test time activate the correct head. The axes of our setting (online learning, no task boundary, no test time oracle, constant memory, and bounded compute) align with the main desiderata of continual learning described in De Lange et al. 2019).
Catastrophic forgetting (McCloskey & Cohen 1989), where previous knowledge is overwritten as new concepts are learned, remains a key challenge in the online continual learning setting. To prevent forgetting, methods usually rely storing small buffer of previous training data and replaying samples from as new data is learned. This can partially counteract catastrophic forgetting, but
"Corresponding Author lccas, page-cacci cacci a@mai wmg.lllll.. 11.ca
'Code reproduce experiments available wwww..g. thub. com/pa hemmpplucasse uuaasl 4/AMI.

---

Published as a conference paper at ICLR 2022
TSNE plot of the penultimate layer at the first task boundary
Task 1 Accuracy during 2nd Task
100
Class (Taak 1)
Class (Task 1)
80
Class (Task 2)
Task Accuracy
88
Class (Task 2)
Task 1 gradient
60
Task 2 gradient
40
ACE, buffer size 200
ER, buffer size 200
ER. buffer size 1000
200
Number of training steps the 2nd task Figure I: (left) Analysis of representations with the first task's class prototypes at a task boundary. Under ER when Task 2 begins, class 1 & 2 prototypes experience a large gradient and subsequent displacement caused by the close location of the unobserved sample representations, this leads to a significant drop in performance (right). Our proposed method (ACE) mitigates the representation drift issue and observes no performance decrease on a task switch.
still tends to lead to large disruptions in accuracy, particularly at the initial task boundary or shift in distribution. Various works focus on studying which samples to store (Borsos et al.) 2020 Aljundi et al., 2019b) or which samples to replay when receiving new data (Aljundi et al. 2019a.. In this work, we direct our attention to the representations being learned and investigate how the features of previously learned classes change and drift over time.
Consider the time point in a stream when a new class is introduced after previous classes have been well learned. If we consider the representation being learned, incoming samples from new classes are likely to be dispersed, potentially near and between representations of previous classes, while the representations of previous classes will typically cluster according to their class. Indeed, one might expect minimal changes to the learned representation of the previous classes, while the new classes samples are pushed away from the clusters of old class data. However, with standard Experience Replay (ER) algorithm (Chaudhry et al. 2019), we observe that it is the representations of older classes that is heavily perturbed after just a few update steps when training on the new class samples. We hypothesize that the fundamental issue arises from the combination of: new class samples representations lying close older classes and the loss structure of the standard cross entropy applied on mix of seen and unseen classes. We illustrate the observed effect in Fig. (left).
This behavior is exacerbated especially in the regime of low buffer size. With larger replay buffers, the learner can recover knowledge about the prior classes over time, while with smaller buffers the initial disruptive changes in representations are challenging to correct. Indeed we illustrate this effect in Fig. (right), we see that ER only recovers from the initial displacement given a much larger buffer size.
In standard continual learning with replay (Aljundi et al. 2019a Chaudhry et al. 2019) the same loss function is usually employed on both the newly received samples and the replayed samples. In contrast, we propose simple and efficient solution to mitigate this representation drift by using separate losses on the incoming stream and buffered data. The key idea is to allow the representations of samples from new classes to be learned in isolation of the older ones first, by excluding the previously learned classes from the incoming data loss. The discrimination between the new classes and the older ones is learned through the replayed batches, but only after incoming data has been learned, added to the buffer, and made available for replay. To allow more direct control of the structure in representations we first consider a metric learning based loss for the incoming data, proposed in Khosla et al. (2020), where we propose to exclude samples of previously learned classes from the negative samples. We show that this type of negative selection is critical, and in contrast issues arise when negative examples are sampled uniformly from the buffer. These issues mimic those seen with standard losses in experience replay (ER) (Aljundi et al. 2019a). On the other hand we use a different loss on replay buffer data that is allowed to consider new and old classes, thereby consolidating knowledge across current and previous tasks. We call this overall approach ER with asymmetric metric learning (ER-AML).
Since cross entropy losses can be more efficient in training for classification than metric learning and contrastive losses (avoiding positive and negative selection) and it is widely used in incremental
2

---

Published as a conference paper at ICLR 2022
and continual learning, we also propose alternative cross entropy solution that similarly applies an asymmetric loss between incoming and replay data. Notably, the cross entropy applied to the incoming data only considers logits of classes of the incoming data. This variant, named ER with asymmetric cross-entropy (ER-ACE), along with ER-AMI. show strong performance, with little disruption at task boundaries Fig. [ (right). We achieve state of the art results in existing benchmarks while beating different existing methods including the traditional ER solution with an average relative gain of 36% in accuracy. Our improvements are especially high in the small buffer regime. We also show that the mitigation of the old representation drift does not hinder the ability to learn and discriminate the new classes from the old ones. This property emerges from only learning the incoming data in isolation; as we will see, also isolating the rehearsal step (as in Ahn et al. (2020) leads to poor knowledge acquisition on the current task. Furthermore we show our ER-ACE objective can be combined with existing methods, leading to additional gains. Finally, we take a closer look at the computation cost of various existing methods. We show that some methods, while obtaining good performance under standard evaluation protocols, fail to meet the computational constraints required in online CL. We provide an extensive evaluation of computational and memory costs across several baselines and metrics.
To summarize, our contributions are as follows. We first highlight the problem of representation drift in the online continual learning setting. We identify root cause of this issue through an extensive empirical analysis (Sec. [.2]). Second, we propose a new family of methods addressing this issue by treating incoming and past data asymmetrically (Sec. 4.1 4.3) Finally, we show strong gains over replay baselines in a new evaluation framework designed to monitor real world constraints (Sec. (5) To the best or our knowledge, we are the first to report the computation costs of different methods in our setting, revealing new insights.
2 RELATED WORK
Research on continual learning can be divided based on the sequential setting being targeted (sce
Zeno et al. 2018); van de Ven & Tolias (019);; Normandin al. (202I); Lesort et al. (2021) for categorizations of the settings and De Lange et al. (2019) for a broad survey on continual learning).
Earlier works consider the relaxed setting of task incremental learning (Aljundi et al., 2017] Serrá
cet al. 2018 Li & Hoiem| (016)) where the data stream is divided into chunks of tasks and each task is learned offline with multiple iterations over the data of this task. While this setting is easier to handle
as one task can be learned entirely, it limits the applicability of the solution.
In this work, we consider the challenging setting of an online stream of non-i.idd data where changes can anytime occur in the input domain or in the output space. This more realistic setting has attracted increasing interest lately (Lopez-Paz et al. 2017 Aljundi et al. 2019a). Specifically, we study the single-head (or shared head) setting, where when queried, the learner is not told which task the sample belongs to (as opposed the multi-heat setting). The single-head assumption is further studied in task-agnostic continual learning settings (He et al. 2019: Caccia et al. 2020 Ostapenko jet al. 202T Von Oswald et al. 202T) which the task-boundary assumption, amongst others, is also relaxed. Many of the solutions to the online continual learning problem rely on the use of a buffer formed of previous memories which are replayed alongside new data during the learning process. Several works (Borsos al., 2020 Chaudhry et al. 2019 Aljundi et al. (2019b) propose solutions to select which samples should be stored, or retrieved for replay (Aljundi et al. 2019a)), or both (Shim al./ 2021). Lopez- et al.] (2017): Chaudhry et al.) use replay to perform constrained optimization, limiting interference with previous tasks as new ones are learned. Our work, on the other hand, focuses on the appropriate loss function in this context. Tang & Matteson (2020) propose a traph-based approach that capture pairwise similarities between samples. Dark Experience Replay (DER) (Buzzega et al., 2020) suggests an alternative replay loss. Samples are stored along with their predicted logits and once replayed the current model is asked to keep its output close to the previously recorded logits. While the method is simple and effective it is worth noting that it relies heavily on data augmentation. Our work is orthogonal and can be combined with DER Finally, concurrent work (Mai et al. 2021) also use a contrastive loss for online continual learning, but not in an asymmetric fashion.
show in Sec. D
In our work we also investigate the underlying causes for performance degradation in replay- based methods. Related to this study are works in the class incremental setting, where similar to our case shared output layer is used, but classes are learned offline. Works in this area address the implicit class imbalance issue occurring when new classes are learned alongside replayed data. Zhao et al.
3

---

Published as a conference paper at ICLR 2022
(2019) proposes to correct last layer weights after group of classes is learned via adjusting the weights norm. Wu et al. (2019) suggests to deploy extra additional parameters in order to linearly correct the "bias" in the shared output layer. Those parameters learned at the end of each training phase. Hou et al. (2019) considers addressing this imbalance through applying cosine similarity based loss opposed to the typical cross entropy loss along with distillation loss and margin based loss with negatives mining to preserve the feature of previous classes, Recently, Ahn et al. (2020) propose to learn the incoming tasks and the previous tasks separately. They use a masked softmax loss for the incoming and rehearsal data, to counter the class imbalance. All the methods highlighted above operate in the offline setting, where data from the current task can be revisited as needed making the disruptive issues emphasized at the task boundary less critical. In this paper, we focus on the online setting, with potentially overlapping tasks. As we will see, work by Ahhn et al. (2020) developed to counter class imbalance, can inhibit learning of the current task in the online setting (see Appendix B) Lastly, Zeno et al. (2018) uses logit masking related to our method but their context is based on the multi-head setting, and does not consider replay based methods, where learning across tasks occurs. Their goal is to activate only the head of which the samples within the new batch belong to. However, our approach is more general and it applies to the single head setting (where we have single output layer for all classes, and no task oracle.)
3 LEARNING SETTING AND NOTATION
We consider the setting where a learner is faced with a possibly never-ending stream of data. At every time step, labelled set of examples (X"", Y drawn from distribution D, is received. However, the distribution Dr itself is sampled at each timestep and can suddenly change to Dr+1, when a task switch occurs. The learner is not explicitly told when task switch happens, nor can it leverage task identifier during training or evaluation. We note that this definition generalizes task-incremental learning, where each task is seen one after the other. In this scenario, given T tasks to learn, D, changes T - times over the full steam, yielding T locally i.i.d learning phases. We also explore in this paper more general setting without the notion of clearly delineated tasks (Aljundi et al. 2018 Chen et al. 2020a), where the data distribution gradually changes over time.
Given model (x) representing a neural network architecture with parameters 6, want to minimize the classification loss C on the newly arriving data batch while not negatively interfering with the previously learned classes (i.e. increasing the classification loss). simple and efficient approach to achieve this is to replay stored samples from fixed size memory, M, in conjunction with the incoming data (Chaudhry et al.] [201 Rolnick et al., 2018)). The core of our approach is that instead of treating the replayed batch and the incoming one similarly and naively minimizing the same loss, we opt for a specific loss structure on the incoming batch that would limit the interference with the previously well learned classes. We approach this by allowing the features of the newly received classes in the incoming data to be initially learned in isolation of the older classes. We first present our idea based on metric learning loss and then generalize to the widely deployed cross-entropy loss.
4 METHODS
4.1 A DISTANCE METRIC LEARNING APPROACH FOR REDUCING DRIFT (ER-AML)
In order to allow fine- fine-grained grained control of which samples will be pushed away from other samples given an incoming batch, we propose to apply, on the incoming data, metric learning based loss from Khosla et al. (2020). Related loss functions have recently popularized in the self supervised learning literature (Chen et al., 2020b). We combine this in a holistic way with a cross-entropy type loss the replay data. This allows us to control the representation drift of old classes while maintaining strong classification performance. Note that if a metric learning loss is used alone we needs to perform predictions using Nearest Class Means Rebuffi et al. (2017) approach, which show is computationally expensive in the online setting.
Given input data point v, consider the function fe(x)) mapping to its hidden representation before the final linear projection. We denote the incoming N datapoints by X and data replayed from the buffer by x/. We use the following loss, denoted SupCon (Khosla et al. 2020), on the
4

---

Published as a conference paper at ICLR 2022
incoming data X
s sim sim(So(Xp)) fo(x:))
Li(Xim) =
log
(I)
P(xx)
ExU(x)(( sim sim(f((xx),x((x))
XiEXin
X2 xpp)())))))
where sim(a.b) =exp( b
Tla ||b|| com-
Algorithm 1: ER-AMI
putes the exponential cosine similarity be-
tween two vectors, with scaling factor 7 Qi al., 2018 He al.) 2020). Here we denote the incoming data
Input: Learning rate a Initialize: Memory M; Model Params 0 do Receive X
X We
//Receive from stream X pos: Xnog FETCHPOSNEG(X* ,M)
use the P and N to denote the set of pos- itive and negatives with respect to x, and the positive examples Tp are selected from the examples in tM UMM, which are from the same classes as x; In the sequel we will consider Xn selected from X U M in two distinct ways: (a) from mix of cur- rent and previous classes and (b) only from
X ~ SAMPLE(M)
1/Sample buffer
C = 7hi (xin, Xpos; Xmeg) +
(x
SGD 0,
//Param Update
RESERVOIRUPDATE(M, )")
//Save
while
The stream hus not ended
classes of the X Note that this implicitly learns distance metric where samples of the same class lie close by. For the rehearsal step, we apply a modified cross-entropy objective as per Qi et al. (018)
which allows to link the similarity metric from above the logits.
sim(wa(z) (We(z) fo(x))
sim
L Lx(xx)) = Σ log
(2)
zEXsf Ecec.u sim We fx())
where Caul the set of all classes observed, and c(x) denotes the label of x. The above formulation allows us to interpret the rows of the final projection {weteEC.n as class prototypes and inference to be performed without need for nearest neighbor search. We combine the loss functions on the incoming and replay data
C(X UX*) 7Li + C2(Xx))
(3)
We refer to this approach as Experience Replay with Asymmetric Metric Learning (ER-AML). We describe the full rehearsal procedure with ER-AML in Algo Note the buffer may contain samples with the same classes as the incoming data stream. The subroutine F FetchP ooNeg is used to find one positive and negative sample per incoming datapoint in Xtn, which can reside in either the buffer memory M or in X
4.2 NEGATIVE SELECTION AFFECTS REPRESENTATION DRIFT
The selection of negatives for the proposed loss L can heavily influence the representation of previously learned classes and analogous to the key issues faced in the regular replay meth- ods where cross entropy loss is applied to both incoming and replay data. A typical approach in this loss for classification may be to select the negatives from any other class (Hoffer & Ailon 2015). However this becomes problematic in the continual learning setting as the old samples will be too heavily influenced by the poorly embed- ded new samples that lie close the old sample representations. To illustrate what is going on in the feature space, consider the case of ER-AML's C₁ term, which explicitly controls distances between sample representations. L1 considers the incoming batch samples (containing new classes) as anchors. As the representations from these classes haven't been learned, anchors may end up placed near or in-between points from previous classes (analogous to the illustration in Figure []. Since the previous classes samples will be clustered together, we use them as negatives for the incoming sample anchors, the gradients magnitude of the positive term will be out- weighte by the negative terms coming from the new class samples, similar to what is observed in Figure case there is sharp change in gradients norms of the loss w.r.t. the features of previous classes, as we illustrate in Appendix [E] which leads to a large change in the representation at the task boundary (and subsequently poor performance). On the other hand if we use only incoming batch examples as
Drift of buffered data on consecutive updates
0.06
ER-AML ER- negs.
0.05
0.04-
0.03
0.02
0.01
0.00
101
501
601
801
901
Step
Figure 2: Buffer displacement in a 5 task stream. Background shading denotes different tasks.
this
5

---

Published as a conference paper at ICLR 2022
negatives can avoid this excessive representation drift. We illustrate this in Figure 2 the representations drift at the task boundaries for ER-AML when using negative samples from all classes and when using only classes in the incoming batch. In the context of the model under consideration we measure the one iteration representation drift of a sample æ as fo (x) - f (x)]l- the output of the network being normalized. We observe that naively applying the proposed loss results in large changes of the learned representation. On the other hand when allowing only negatives from classes in the incoming batch, we see reduction in this representation drift. In the Appendix 7 we further demonstrate that the accuracy of models trained using ER-AML with only incoming batch negatives can improve the continual learning system performance by large margin. We emphasize the that ER-AML with all negatives and the regular ER method used for online continual learning suffer from a similar issue and thus lead to similar poor performances, with appropriate negative selection resolving the problem. This is further emphasized in Appendix H where we observe similar poor drift behavior for ER.
by showing
4.3 CROSS-ENTROPY BASED ALTERNATIVE (ER-ACE)
Having demonstrated the effect of controlling the incoming batch loss in avoiding drastic represen- tation drift, we now extend it to be applicable to the standard cross-entropy loss typically studied in ER (Aljundi et al., 2019a; Chaudhry al., 2019). Given an incoming data batch, consider Cotd the set of previously learned classes and Ceuer the set of classes observed in the current incoming mini-batch. Denoting C the set of classes included in the cross-entropy loss, we define the Lee (X, C)
Ezex log Exeo simi [e)] where C C Call denotes the classes used to compute the denominator. We note that restricting the classes used in the denominator has an analogous effect to restricting the negatives in the contrastive loss. Consider the gradient for a single datapoint= x, (G.,(..C) = W - 37) @ LEE)). Here ş denotes the softmax output of the 0fo
cross-entropy loss as: CX,
sim("cte)_fo(())
network, a one-hot target, 1 sEC a binary vector masking out classes not in C, and W the matrix with all class prototypes (oeJoEC.u. When the loss is applicd in the batch setting, it follows that only prototypes whose labels are in C' will serve roles analogous to positives and negatives in the contrastive loss. We can then achieve a similar control as the metric learning approach on the learned representations.
Now, our loss applied at each step would be:
Lace (f U x = (x Cold U Ccurr) + Lee (X .eaur)
where Courr denotes the set of the classes represented in the incoming batch and Cotd denotes previously seen classes that are not presented in the incoming batch, those that we want to preserve their representation. Note this is a straightforward procedure and induces no additional computational overhead. We refer to it as Experience Replay with Asymmetric Cross Cross-Entropy (ER-ACE).
EXPERIMENTS
We have highlighted the issue of abrupt representation change when new classes are introduced, and propose two methods that address this issue. We now demonstrate that mitigating drift directly leads to better performance
standard online continual learning benchmarks. As in Lopez- Paz ctt al. (017);; ; Aljundi et al. (2019a); Chaudhry et al. (2019) we use a reduced Resnet- 18 for our experiments, and leave the batch size and the rehearsal batch size fixed 10. This allows to fairly compare different approaches, as these parameters have a direct impact on the computational cost of processing given stream.
5.1 DATASETS
All benchmarks are evaluated in the single-head setting, i.e. task descriptors are not provided to the model at test time, hence the model performs N N-way classification where is the total amount of classes seen.
Split CIFAR-10 partitions the dataset into 5 disjoint tasks containing two classes each (as in Aljundi 2 Shim et al. (2020))
al.
Split CIFAR- 100 comprises 20 tasks, each containing a disjoint set of 5 labels. We follow the split in Chaudhry al. (2019). All CIFAR experiments process 32 32 images.
Split Minilmagenet splits the Minilmagenet dataset into 20 disjoint tasks of labels each. Images
84 x 84.

---

Published as a conference paper at ICLR 2022
5.2 BASELINES
We focus our evaluation on replay-based methods, as they have been shown to outperform other approaches in the online continual learning setting Chaudhhy et al. (2019); Aljundi et al. (2019a); Ji et al. (2020). We keep buffer management constant across methods: all samples are kept or discarded according to Reservoir Sampling Vitter (1985). We consider the following state-of-the-an baselines:
ER: Experience Replay with a buffer of a fixed size. Unlike Aljundi et al. (2019a), we do not leverage the task identifier during training to ensure that rehearsal samples belong to previous classes.
Rebufi et al. (2017) A distillation loss alongside binary cross-entropy is used during training. Samples are classified based on closest class prototypes, obtained from recomputing and averaging buffered data representations.
MIR Aljundi et al. (2019a) selects for replay samples interfering the most with the incoming data batch.
DER++ Buzzega et al. (020) uses a distillation loss on the logits to ensure consistency over time.
SS-IL Ahn al. 2020) learns both the current task loss and the replay loss in isolation of each other.
An additional task-specific distillation is used on the rehearsal data.
GDUMB Prabhu al. (2020) performs offline training on the buffer with unlimited computation and unrestricted use of data augmentation at the end of the task sequence.
iid: The learner is trained with a single pass on the data, in a single task containing all the classes. We also consider version of this baseline using similar compute budget as replay methods (iid++)
We note additional baselines such as Lopez-Paz et al. 2017); Chaudhry et al. were shown to perform poorly in this setting by prior work Buzzega et al. (2220) and are thus left out for clarity.
5.3 EVALUATION METRICS AND CONSIDERATIONS
Our evaluation includes the metrics and experimental settings used in previous works on online continual learning with single-head (Aljundi et al.) [019a] Ji et al. (2020; Shim et al.) 2020). We provide extra emphasis on anytime evaluation and comparisons of the computation time per incoming batch. We also consider several additional settings in terms of computation and use of image priors.
Anytime evaluation A critical component online learning is the ability to use the learner at any point De Lange et al. (2019). Although most works in the online (one-pass through the data) setting report results throughout the stream Lopez-Paz et al. (017): Chaudhry et al.) Aljjundi et al. (2019b), several prior works have reported the final accuracy as a proxy Aljundi et al. (2019a); Shim ct al. (202)). However a lack of anytime evaluation opens the possibility to exploit the metrics by proposing offline learning baselines that are inherently incompatible with anytime evaluation Prabhu
et al. (2020).
In order make sure that learners are indeed online learners, we evaluate them throughout the stream. We define the Anytime Accuracy at time l (AA²) as the average accuracy on the test sets of all distributions seen up time k. If the learning experience lasts T steps, then AAr is equivalent the final accuracy. Finally, we report the Averaged Anytime Accuracy (AAA) (Caccia et al. 2020), which measures how well the model performed over the learning experience
T
AAA
T (AA)..
(4)
Computation and Memory Constraints While memory constraints are well documented in previ- ous work, careful monitoring of computation is often overlooked; some methods can indeed hide considerable overhead which can make the comparison across methods unfair. On the other hand this is critical to the use cases of online continual learning. To remedy this, we report for each method the total number of FLOPs used for training. While we cannot fix this quantity as we can for memory (since different methods require different computations). this will shed some light on how different methods compare. Note that we also include in this total any inference overhead required by the models; Nearest Class Mean (NCM) classifiers must compute class prototypes before inference for example. We add this cost every time the model is queried to measure its Anytime Accuracy, Let
T
T
Mem Σ t=1
10.1 + [Mel Comp Σ [(m((;00))
(5)
(=1
where (m{ (:m(=;6) 6,)) denotes the number of FLOPs used at time t. Since the same backbone and buffer used for all methods in this paper, we will focus our constraint analysis on computation
7

---

Published as a conference paper at ICLR 2022
Method
Data Aug.
M =
M
20
M
100
Train TFLOPs
Mem.
AAA
Ace
AAA
Acc
AAA
Ace
(Mb)
iid ud+
62.7407
62.7+00
62.7=0o
8
4
72.9407
72.9407
72.9=07
16
DER++
50.7
31.8109
$5.6-12
39.3110
60.1
1.
24
(4, 7)
40.0108
19.7-103
45.2-13
26.72:10
55.4
38.7=008
ER
45.6+111
28
55.9-1.2
40.3+06
60.3413
49.4+
17
(4,
iCaRI.
47.0108
49.1
30.6108
33,4110
54.4=0.7
55.1=007
41.74.06
39.2108
59.3406
56.9107
45.1
42.3:0.8
(21, 47)
(8. 11)
MIRI
39.3+10
44.9
29.8.08
19.7
44.7=111
49.7=1.8
41.81066
29.74.0
54.6
53.8+17
43.3=1.0
49.3
(4,
SS-IL
42.6117
29.610.4
44.8-1.8
35.1100
48.1:22
41.1204
(8, 11)
41.1=16
31,6225
47.0-12
38.3104
48.1
47.5207
ER-ACE (ours)
53.1+10
52.6109
35.1
35.6+1.0
56.4-10
58.0-07
43.41116
42.6107
61.9-09
61.7.109
53.7-1.1
52.2=07
(4,
ER-AMI. (ours)
49.4110
30.9408
$7.0-10
39.2:110
63,3+100
52.2-111
17
(4,
50.4+13
36,4114
56.8-10
47.7+00
62.0
55.7=113
GDUMB
01000
35.0.00
0.0.0
45.81009
0400
61.3a17
(43, 853)
(11, 14)
Table split CIFAR-10 results. i indicates the method leveraging a task identifier at training time. For methods whose compute depend on the buffer size, we report min and max values. We evaluate the models every 10 updates. Results within error margin of the best result are bolded.
Train TFLOPs
Mem. (Mb.)
Train TFLOPs
Mem. (Mb.)
Method
AAA
Acc.
AAA
Acc.
iid
19.8+a3
16.7
59
idd+
2803
17
25.0+as
118
DER+
23.3
15.1 +04
25
36
21.7+06
12.9 +0.3
176
217
ER
24.2206
19.8+04
17
35
26.2+ax
18.2+00.
118
216
iCRRL
26.32a0
17.3
294
39
24.4
17.1+-0.1
2097
220
MIR
23.6.108
20.6:105
27.2
20.2
294
216
SS-IL
31.5
25.0103
19
39
29.7
23.5
137
220
ER-ACE (ours)
32.7
25.8 +04
17
35
30.2
22.7
118
216
ER-AML (ours)
30.2+06
24.3+00
28
27.0+07
19.3
200
216
Table 2:
Split CIFAR-100 100 (left) and Mini- Iini-emmeeeet (right) results with M - 100. For each method, we report the best result between using (or not) data augmentations.
Data Augmentation In the settings of Aljundi et al. (2019a) Lopez-Paz et al. (017) Ji et al, (2020)
Shim et al. (2020); Chaudhry al. (2019) data augmentation not used. However, this a standard practice for improving the performance on small datasets and can thus naturally complement most methods utilizing replay buffers. Notably, Prabhu al. (2020), the offline learning method, utilizes data augmentation when comparing to the above online learners. To avoid unfair comparisons, in our experiments we indicate when a method uses augmentation. When not specified, we treat it as hyperparameter and report the best performance.
Hyperparameter selection For all datasets considered, we withhold 5 % of the training data for validation. For each method, optimal hyperparameters were selected via grid search performed on validation set. The selection process was done per dataset basis, that is we picked the configuration which maximized the accuracy averaged over different memory settings. We found that for both ER-AML and ER-ACE, the same hyperparameter configuration worked across all settings and datasets. All necessary details to reproduce our experiments can be found in the Appendix.
5.4 STANDARD ONLINE CONTINUAL LEARNING SETTINGS
We evaluate Split CIFAR-10, 10, Split 100 and Split Minilmagenet using the protocol and constraints from Aljundi et al. (2019)): Ji et al. (022)); Shim et al. (2020) We note all results each method is 10 times, and we report the mean and standard error. We first discuss dataset specific results, before analysing the computation cost of each method.
CIFAR-10 results found in Table I using variety of buffer sizes. In this setting, both the methods we propose, ER-AML and ER-ACE consistently outperform other methods by significant margin. This result holds in both settings where data augmentation is (or not) used,
that
8

---

Published as a conference paper at ICLR 2022
outperforming previous state-of-ther art methods MIR and DER++. Shifting our attention to SS-IL, its underperformance w.r.t to ER-ACE highlights the importance of having a rehearsal objective that considers the new classes. In Appendix B we observe that when applying SS-IL in the online setting: (1) the method performs poorly on the current task, as is it unable to consolidate old and new knowledge, (2) yet mitigates representation drift even on a perfectly balanced stream. The latter is surprising, as the method was designed specifically to address stream imbalance. Finally, we note the offline training baseline G-DUMB cannot satisfy the anytime evaluation criteria.
Longer Task Sequence results are shown in 2 CIFAR-I00 on the left and Minilmagenet on the right. On both datasets similar findings are observed, our proposed methods match or outperform strong existing baselines. SS-IL performs similarly to our method on mini-imagenet hile having a higher computational and memory cost. As mentioned above, the method struggles to learn the current task, however here the "weight" of the current task is small in the final acc of the 20-task regime, We see that average anytime accuracy is higher for ER-ACE and indeed the anytime curves in Appendix [ further illustrate this. Finally, ER-ACE shows relative gains of 35% in accuracy over ER, without any additional computation cost. For Mini- Mini-Imagenet, ER-ACE outperforms the single-pass iid baseline, and nearly reaches the performance of the equal-compute iid baseline.
Computation Budget To provide another view of the computational advantages of our proposal we report the accuracy given compute budget over the length of the se- quence in Fig 3 When monitoring the com- putation performed by each baseline, we
Split Cifar- 10 with M=100
50
Accuracy
40
DER++
notice that several methods do not compete on equal footing. First, the use of Near- est Class Mean (NCM) classifiers leads to a significant compute cost, as shown for iCaRL. For experiments, evaluate the model after 10 mini-batches (100 to- tal samples), where NCM classifier must forward the whole buffer to get class pro- totypes. We argue that such approach has disadvantages in the online setting due to poor computational trade-off's. Second, MIR AAljundi et al. (2019)) has an expen- sive sample retrieval cost. remains to show this step can be approximated more efficiently. Finally, we note that our method, ER-AMI. has varying compute: for streams with small number of classes per task (CIFAR10), it can compute the incoming loss leveraging only the incoming data. other datasets, where an incoming batch may not have least two samples of each class, additional cost to forward buffered point is incurred.
ER
ICaRL
30
MIR
55-IL
20
ER-ACE
ERI
10
6x 10
10
Cumulative TeraFlops spent
Figure 3: Total Accuracy function of TeraFLOPSs spent. Here the models are evaluated on all 10 classes, to ensure consistency across timesteps.
Evaluation with augmentation The of augmentations also permits extra benefits of replay methods particularly in settings where buffer overfitting is more present, c.g. in the small buffer regime. From the results in Table 1 large set of methods. It is therefore crucial to compare methods on equal footing, where they can all leverage (or not) data augmentation. For example, gains reported in Prablu et al. (2020) over ER completely vanish when ER is given the same access to augmented data. We note that for Mini- Imagenet, augmentations did not help. We hypothesize that since this is the hardest task the risk of overfitting on the buffer is less severe.
that augmentations provides significant gains for
5.5 BLURRY TASK BOUNDARIES
Table 3: CIFAR-10 10 Blurry Task Boundary Experiments
Next, we explore a setting where the distribution is continuously evolving, rather than clearly delineated by task boundaries (sim- ilar to settings considered in (Aljundi et al. (2019b)). To do this,
Method M = 20
M 100
we linearly interpolate between tasks over time, resulting in new classes being slowly mixed into the data stream. This experiment
ER 32.
42.7
DER++
31.0214
41.7#14
is done on Split- 10, and the interpolation such that at every timestep, the incoming data batch has average 2 unique labels (as in the original experiment). We only evaluate task- free
ER-AML
45.6
55.2+1.1
ER-ACE
44.5tbos
50.2
methods in this setting: methods like MIR and SS-IL cannot be used in such setting. Results in table

---

Published as a conference paper at ICLR 2022
report the final accuracy, averaged over 5 runs, report the standard error. We observe ER-AML and ER-ACE methods perofrm well in this setting. More details provided in Appendix A.2
6 CONCLUSION
We have illustrated how in the online continual learning setting the standard loss applies excessive pressure on old class representations. We proposed two modifications of the loss function, both based on treating the incoming and replay data in an asymmetric fashion. Our proposed method does not require knowledge of the current task and is shown to be suitable for long task sequences achieving strong performance with minimal or no additional cost. We also raise the standard for high quality evaluation in online continual learning by considering wide number of baselines and metrics.
REPRODUCIBILITY STATEMENT
We have made several efforts ensure that the results provided in the paper are fully reproducible. We first provide a detailed codebase from which all the results in this paper are generated. In this codebase, one can find the results of our grid search, as well as optimal hyperparameters for each method and setting. We have provided Readme file to help guide used to reproduce results. Details of all hyperparameters are also clearly described in the main paper and particularly in the appendix.
ACKNOWLEDGEMENTS
Lucas Caccia is funded by Borealis AL. EB and NA supported by NSERC Discovery Grant RGPIN-2021-041004 We acknowledge resources provided by Compute Canada and Calcul Quebec.
REFERENCES
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. arXiv preprini arXiv:2003. 1100003..9444 13947, 2020.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. arXiv preprint arXiv:1711.0960) 2017.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In CVPR 2019, 2018.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In Advances in Neural Information Processing (NeurIPS), 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. arXiv preprint arXiv:1903.08671. 2019b.
Zalán Borsos, Mojmír Mutný, and Andreas Krause. Coresets via bilevel optimization for continual learning and streaming. arXiv preprint arXiv:2006.03875, 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe- rience for general continual learning: a strong, simple baseline. arXiv preprint 11X1v:2004.07211, 2020.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, et al. Online fast adaptation and knowledge accumulation: a new approach to continual learning. arXiv preprint 0rX.v:2003.05856 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR 2019.
Arslan Chaudhry, Mareus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint arXiv: 1902. 10486, 2019.
10

---

Published as a conference paper at ICLR 2022
Hung-Jen Chen, An-Chieh Cheng, Da-Cheng Juan, Wei Wei, and Min Sun. Mitigating forgetting in online continual learning via instance-awee aware parameterization. Advances in Neural Information Processing Systems, 33, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020b.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909 08383, 2019.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning. arXiv preprint arXiv: 087333933 2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 9729-9738, 2020.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning, ArXiv, abs/1906.05201, 2011,, URL
atttps://arxiie httpp////a.aiivoog//aaassa./// / 1906. 05201
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International workshop on similarity- based pattern recognition, pp. 84-9 92. Springer, 2015.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Xu Ji, Joao Henriques, Tinne Tuytelaars, and Andrea Vedaldi. Automatic recall machines: Internal replay, continual learning and the brain. arXiv preprint arXiv:2006.1232: 2020.
Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sama,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18661-18673 Curran Asso- ciates, Inc., 2020. URL htttps://pooeeedinnn nwwrips cc/ /ttpp//s...//..e/////////e/ Papee. pdf
Timothée Lesort, Massimo Caccia, and Irina Rish. Understanding continual learning settings with data distribution drift analysis. arXiv preprint arXtv:2104.01678 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In European Conference Computer Vision, pp. 614-629. Springer, 2016.
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner: Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3589-3599, 2021.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation, 24:109-165, 1989.
Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko. Matthew Riemer, Pau Rodriguez, Julio Hurtado, Khimya Khetarpal, Timothée Lesort, Laurent Charlin, Irina Rish, Massimo Caccia. Sequoia towards a systematic organization of continual learning research, http:::::htttsststtttuttt
htmm/eebricc/Seequiii.. 2021. URL wtt.t:tp:///t http://ggthhbbbb / /ttpp///p/////aaaaaaoaa////////a
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. In Fiith Conference on Neural Information Process- 2021. URL
htw.:/ceeeeeiinn...nei..r.....ee cc/ http:/ppps.pppppppp////////aa// fe55:776600bbaed2222444444438883

---

Published as a conference paper at ICLR 2022
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision, pp. 120. Springer, 2020.
Hang Qi, Matthew Brown, and David Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5822-5830, 2018.
AllvestreAAlvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. Proc. CVPR, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. Experience replay for continual learning. arXiv preprini arXiv: 1811:111111 1682, 2018.
Joan Serrà, Didac Surís. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. arXiv preprint 1rXiv:1801.01444 2018.
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. arXiv e-prints, pp. arXiv-2009, 2020.
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. Online class-incremental continual learning with adversarial shapley value. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9630-9638, 2021.
Binh Tang and David S Matteson. Graph-based continual learning. arXiv preprint arXiv:2007.04813, 2020.
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.0773: 2019.
1904. 07734
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.
Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Cac- cia, Nicolas Zucchet, and João Sacramento. Learning where to learn: Gradient spar- sity meta and continual learning. Advances Neural Information Processing Syy- tems, 34, 2021. URL lattps://proceedinng/ neurips ct/ cc/papper/022/hhhhh .tml
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv: 1803. 10123, 2018.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shutao Xia. Maintaining discrimination and fairness in class incremental learning. arXiv preprint 1rXiv:1911.07053,000 2019.
A EXPERIMENTAL SETUP
In this section we provide additional experiments regarding the basclines and hyperparameters. In all experiments, we leave the batch size and the rehearsal batch size fixed at 10, following Aljundi et al. (ChuudyChh et al.) This allows to fairly compare different approaches, as these parameters have a direct impact on the computational cost a given run. The model architecture (0 in Alg. [ is also kept constant, which is reduced ResNet-18 18 used in Lopez-Paz et al. (2017); Chaudhry al.) Aljundi et al. (2019 b). where the dimensions of the last linear layer change depending on the input height and width. The model has 1.09M params for the CIFAR experiments and 1.15M params for Minilmagenet. For all datasets considered, we keep the original ordering of the classes, meaning that the first task will always contain the first & classes.
12

---

Published as a conference paper at ICLR 2022
A.1 HYPERPARAMETERS
All results in the paper have heen (re)implemented by us, with the expection GDUMB Prabhu et al. (2020), where results were run from the author's public codebase. For each method grid search was ran on the possible hparams, which we detail below. We will also described method specific details.
DER++ Buzzega et al. 2020)
LR: [0.1, 0.01, 0.001]
[0.25, 0.5, 0.75]
3 [0.5, 0.75, 1]
We also tried to implement the DER (not DER++) algorithm described in Buzzega et al. (2020). We found that it did not lead to improvements w.r.t to ER the single epoch setting. Moreover, the setting in the original paper uses a wider Resnet-18. 18. We found that both these differences account for the drop in performance when comparing to the numbers in Buzzega et al. (2020).
Finally, we highlight that in general, methods using distillation (iCaRI Ahn et al. (2020), and DER Buzzega et al. (2000) typically perform better in the onlne setting without it.
al. (017). SS-IL
ER Chaudhry et al.
* .LR: [0.1, 0.01, 0.001]
Note that unlike the ER implementation in Aljundi al. (2019a), use "task-free" implementation. This leads to two differences. First, rehearsal begins as soon as the buffer is not empty. Second, when fetching points in the buffer, we do exclude classes from the current task, as done in MIRAljundi et al. (2019a).
iCaRL Rebuffi al. (2017)
LR: [0.1, 0.01, 0.001]
For all "task-based methods (iCaRL, MIR, SS-IL) we fully leverage the task identified and do not start rehearsal until the second tasks. This typically leads to better performance, especially in the small buffer setting, as it reduces the work of overfitting to the buffer.
MIR Aljundi al. 2019a)
LRR : 10.1, 0.01, 0.001]
Note that unlike in the original paper, the final results in the paper are the full training set. In other words, once the hyperparameter cross-validation is done, we train on the validation set. This changes the results slightly from the original paper. Finally, kept the number of items subsampled from the buffer for the sampling step (Ne) equal to 50 as the original codebase.
Ahn et al. 2020)
LR [0.1, 0.01, 0.001]
should distill. [Yes, No]. When turned on, this method also uses the distillation loss as prescribed in Ahn et al. (2020)
As we will see in using the
ER-ACE
LR: 10.1, 0.01, 0.001]
To implement the masking loss, we simply use logits maskedfilll (mask, -1e9) to filter classes which should not receive gradient. Using small constant in this step equivalent to removing the masked classes from the softmax denominator.
ER-AML
.R [0.1, 0.01, 0.001]
SupCon Temperature [0.1, 0.2)
13

---

Published as a conference paper at ICLR 2022
A.2 BLURRY TASK BOUNDARIES EXPERIMENT
Here we provide additional details on the experiment described in Section 5.4 In the original (task based) benchmark, each task comprises 10K samples (or 1K minibatches of 10 samples), so a total of 5K minibatches streamed. For the smooth alternative, at each timestep & E (1, 5000) the unnormalized probability of seeing class is given by
pe(() N No
with N₂ denotes the number of samples of class c, and pec (2c-1)N, At every timestep we normalize this probability for each class and sample according to Categorical distribution with these probabilities. The parameters for the mean and variance are chosen that on average, the model receives unique labels per minibatch of 10 items (as in the original task-based experiment).
In such setting where there is no notion of the current task, or rather set of current labels, one cannot use SS-IL, as it needs to leverage a task identifier during training. Through this experiment we show that our method can overcome this limitation, despite sharing some similarities with SS-IL.
B AN IN-DEPTH ANALYSIS OF SS-IL IN THE ONLINE SETTING
SS-IL. is a related method. In this section, we highlight several key observations when deploying SS-IL in the online setting which are the other hand not issues for ER-AMI and ER-ACE. We then provide several additional experiments, shedding some light on the inner workings of the method.
SS-IL fails to learn the current task As stated earlier, the key difference between SS-IL without distillation and ER-ACE is that in the latter, the rehearsal loss in wnmasked. In this section, highlight the problems that occur when using a masked rehearsal loss alongside a masked incoming loss as in SS-IL. We show that since both losses are masked, the model never learns to classify classses across tasks. Specifically, there is no objective in which the model learns to distinguish classes in the current task from classes in the previous tasks. As we show Figure 4 SS-IL is unable to classify samples from the current task in single-head setting. The method actually performs worse than random chance on samples form the current task. On the other extreme we see that ER does very well on the current task (shifting abrupty the previous representations to accomodate the new task). Finally, we see that ER-ACE strikes a good tradeoff between the two, reaching a reasonable accuracy on the current task without disrupting the learned representations of previous tasks. We note that the same conclusion is reached when using the original SS-IL method with the distillation loss.
SS-IL does more than correcting for class imbalance SS-IL is motivated as a method which addresses the class imbalance issue arising in replay methods. Specifically, when drawing fixed number of rehearsal points at every epoch, it follows that as more and more tasks are seen, previous classes are underrepresented in the training stream when compared to points from the current tasks.
In this section, we test whether or not the behavior of SS-IL differs from standard Experience Replay when no class imbalance is present. In this experiment, we increase the number of rehearsal points sampled at every task such that when combining incoming and rehearsal data, we obtain perfectly balanced training data on average. This is experiment is done on the Split-IIFAR10 benchmark with 2 classes per task, with a minibatch of 10 incoming datapoints. Therefore, we sample 0, 10, 20, 30, 40 rehearsal points per incoming databatch for the first, second, third, fourth and fifth task.
What we observe is that SS-IL still outperforms regular Experience Replay, suggesting that the method does more than simply addressing class imbalance in the data stream. We report final accuracy in T SS-IL's performance gap with ER is bigger with small buffer. This is consistent with what we observe for representation drift: methods with larger buffer can better correct for abrupt representation change, making the gap between ER vs ER-ACE and ER-AML smaller. From this we give new insights on the inner workings of SS-IL, namely that it works well because addresses representation drift rather than class imbalance.
Method
M 20
M 50
M 100
ER
21.0 1.2
25.7
37.8 31.6
1.1
SS-II.
30.3 + 1.0
+
0.8
39.1 + 0.6
Table 4: Final Accuracy on split CIFAR-10 with class balanced stream.
14

---

Published as a conference paper at ICLR 2022
Accuracy on the current task
Task Acc.
50
100
200
300
400
500
600
700
800
900
Number of training updates on task
Task 3 Acc.
100
200
300
400
500
600
700
800
900
Number of training updates on task 3
Task 2 Acc.
50
100
200
300
400
500
600
700
800
900
Number of training updates on task 2
Task Acc.
50
100
200
300
400
500
600
700
800
900
Number of training updates on task
Task 0 Acc.
ER
50
ER-ACE
SS-IL
100
200
300
400
500
600
700
800
900
Number of training updates on task 0
Figure For Split- Split-CIFAR-10, 10, we monitor the performance on the current task observed in the stream for SS-IL, ER, and ER-ACE. ER fits too abruptly current task; ER-ACE incorporates this knowledge slowly; SS-II. barely on the other hand is unable learn new tasks when they are first observed in the stream
15

---

Published as a conference paper at ICLR 2022
C
OVERFITTING ON BUFFERED SAMPLES
We study the extent to which our proposed method reduces over-fitting to samples stored in the buffer.
Cosine Similanity between held-out and buffered data tasks
Cosine Similarity
ER-ACE
0.8
0.7
0.5
Task
Task 2
Task 3
Task 4
Task 5
Figure 5: Alignment between buffer and holdout representations. ER-ACE has constantly larger alignment between seen and unseen samples compared to ER especially for older tasks.
A good model fit should yield learned representation where same class datapoints aligned, whether or not they were seen during training. To evaluate this potential mismatch, we first train a model and compare the representations of a) samples in the buffer M after training and b) held-out samples from the validation set V. That is, for each datapoint Im E .M we find the point Ty V with c(zm) = c(x.)) which maximizes the cosine similarity between fr( and fe(æ.) This allows to compare alignment across models, irrespective of their internal scaling. We report the results in Figure where similarity values are averaged over points from the same task. We find that our proposed method, ER-ACE designed to reduce representation drift also reduces the extent to which the model overfits on the buffer. We observe that for earlier tasks, ER-ACE still retains strong alignment between rehearsal and held-out data, which is not the case for ER.
D COMBINING ER-ACE WITH DER++
In this section, we apply our method on top of the strong DER+ BBuzega et al. (2020) baseline. For this experiment, we use the same setting in the DER paper. Specifically, port our implementation to their public codebase https: / /ttpthb.....gg/ggg com/ wamaaaeealaa.. als m We keep the default settings for CIFAR-10 10, using single pass through the data. We find that combining ER-ACE with DER++ yields additional advantages. Not only do we observe small gains in accuracy. we notice significant gains in forgetting. Results shown in Figure 6 Forgetting defined as in Chaudhry et al.
CIFAR-10 Accuracy
CIFAR-10 Forgetting
70
40
DER++
DER++ACE
60
30
Final Acc.
ER-ACE
Forgetting
50
20
40
10
30
M=20
M=50
M=100
M=20
M=50
M=100
Buffer Size
Buffer Size
Figure 6: Comparison to Dark Experience Replay (DER). We obtain improved performance and we can enhance the DER method using the ER-ACE approach
16

---

Published as a conference paper at ICLR 2022
(3.2 1 1.8) 10
Arrp-Tipppll ER- AL-L Triplet w. All Negs
(3.0 0.6)
x 10 -
ER-AML Triplet Incoming Negs
(2.5 1 0.6)
10
Table 5: Average Drift (avg distance feature space) of buffered representations for CIFAR-10 during learning of the second task. We observe similar behavior to ER-AML with SupCon
M Accuracy f M
20
=
M 100
iid online
60.8 1: 1.0
60.8 1.0
1.0
60.8 1 1.0
iid++ online
72.0 0.1
72.0
0.1
0.1
12.00..1
iid offline
79.2 0.4
79.2
7.42
79.2 0.4
fine-tuning
0.3
1.34
18.4
0.3
18.4
ER
19.0 : 0.1
2.3
36.1
0.6
41.5
ER-AML Triplet
33.0 - 0.3
40.1
46.0
0.5
49.8 1 0.5
ER-AML SupCon
33.0 = 0.2 4.1
48.3 - 0.2
51.9 0.3
Table 6: Ablation comparing ER-AML with triplet loss to ER-AML with SupCon. We observe both
improve over ER but SupCon has better performance in larger buffer sizes
E GRADIENT NORM
Figure 7 shows the gradients norms of the features of previous classes in a stream of two tasks. Note how for normal ER, the task switch the gradients the previous classes features are suddenly very high leading potentially to large drift on these features.
gradient norm hidden rep. 1st Task
0.010
method
Asym. Loss w. All Negs
0.008
Asym. Loss w. Incoming Negs
Gradient L1 Norm
0.006
0.004
0.002
0.000
250
500
750
1,000
1,250
1,500
1,750
Step
Figure 7: Gradient's norm for first task features in a two task learning scenario. We observe a sharp increase when all negatives are used and decrease using only incoming negatives.
F ER-AML WITH TRIPLET LOSS
We observe similar behavior for ER-AML implemented with the triplet loss in terms of the importance of negative selection on drift as illustrated Table We also ablate ER-AML based on SupCon and Triplet in Table finding the former outperforms settings with higher buffer sizes, but that both outperform ER.
G ABLATIONS NEGATIVE SELECTION
As discussed the main paper, the selection of negatives is a critical aspect of ER-AML and motivates ER-ACE. To further illustrate this ablate the performance of ER-AML when all possible negatives are used versus the prescribed negative selection strategy (using only classes the incoming batch). The results are shown in Table ? We observe that performance of ER-AML with all negatives similar to but slightly better than ER, while use well-selected negatives greatly improves performance.
17

---

Published as a conference paper at ICLR 2022
Accuracy (1 is better)
Forgetting (4 is better)
M 50
M 20
M
26.7 +
36.1 0.6
47.1
0.8
37.6 +
Eall negatives)
28.5 + 0.3
11.40.4
56.7
+ 0.6
35.0 + 0.4
(incoonnn negatives)
41.9 0.1 48.3 4 0.2
33.6
4 0.2 25.8 + 0.3
Table 7: Ablation of ER-AML with all negative selection versus negatives selected from incoming classes. We use the CIFAR-I0 dataset. We observe that performance of ER-AML with all negatives is similar to but slightly better than ER, while use of well-selected negatives greatly improves performance.
H ADDITIONAL DRIFT RESULTS
We showed in Figure 2 that the selection of negatives has a significant impact on the amount of representation change. Here we show that a similar behavior is observed with ER ER-ACE.
Drift of buffered data on consecutive updates
0.06
ER
0.05
ER-ACE
0.04
MSE
0.03
0.02
0.01
0.00
101
201
301
401
501
601
701
801
901
Step
I ANALYSIS OF THE REPRESENTATIONS DURING THE SECOND TASK
In this section we take a closer look at the model's internal representation during the learning of the second task for different methods. This experiment replicates the setup illustrated in Figure [ (split-CIFAR-10 with M = 20). For each method, the figures for all iterations were projected together ensure that the figures are comparable across timesteps. All methods were initialized starting from the same base model trained on the first task. The dotted representations shown for each class come from held-out samples.
We start by looking at the representations obtained at the begining of the second task. We see that for all three methods, (i) the prototypes of the classes from the first task (Class and Class 1) are well placed, while the other prototypes are placed at random since they are not trained.
-10
20
60
Class Class
Clase
Closs 3
-00
-20
20
Clocc 0
-40
Cless
-60
-75
-50
-20
25
75
-50
-40
-20
20
40
-20
Figure 8: 'Training Iteration on the Second Task
After 100 training iterations, we see that for ER, the prototypes of the old classes have been signifi- cantly displaced and are far from the points of similar class. This is not the case for the latter two
18

---

Published as a conference paper at ICLR 2022
methods; for ER-ACE and ER-AML, the model is beginning separate de classes from another, and the class prototypes are near their respective classes.
40
20
-20
-2
-10
Class
+40
Gase
Clsss
-20
Class
Casss
600
Class
50
25
25
50
75
-00
25
50
75
75
50
25
50
Figure 9: 100 Training Iterations on the Second Task
After 400 training iterations. ER still struggles align the class prototypes with the respective classes. ER-ACE has already well clustered the respective classes. ER-AML, continues to cluster the classes together, however does not as fast as ER-ACE.
ACE
ER.AML
75
60
Class
Cress 2
Class
Crsss 3
Clses
20
-20
Crear
Class
-40
-00
Case
80
75
Cree
-40
-00
-25
25
50
75
-75
-00
-25
25
-75
-00
-25
25
Figure 10: 400 Training Iterations on the Second Task
At the end of the second task, ER-ACE and ER-AML have successfully clustered the classes and aligned their respective prototypes with the clusters. As for ER, while the data clustered, the prototypes not properly aligned with class clusters. Moreover, still strong overlap between prototypes of Class and 3.
50
ER ACE
DR-MML
Cass 0
Class
Cloce
Cloxs
Class 2
Class
40
20
20
-20
Cree
-20
Class
-40
Figure 11: End of the Second Task
J ADDITIONAL BLURRY TASK BOUNDARIES EXPERIMENTS
Here provide blurry task results for varying levels of task overlap. To give an idea of how much the tasks overlap, we report the average number unique classes per incoming minibatch (MB): small number means that the tasks well separated. A high number means that there is strong overlap. the fully i.d setting, this number would be maximized. On the other hand, when this equals 1, each data class is streamed
after the other.
Experiments are performed again on CIFAR-10 10 with M = 20. We use augmentations to fairly compare with DER++. Results averaged over 5 runs.
19

---

Published as a conference paper at ICLR 2022
Method Avg. unique classes per MB
4
5
23.1
25.7
26.3
31.1
34.4
DER+ 20.3
31.1
31.4
37.3
34.4
ER-ACE
32.8
36.2
36.8
41.7
44.5
ER-AML 34.0
40.4
46.0
47.6
47.9
We see that through wide range of different blurriness levels, methods show strong improvement over other task-free baselines
K EXPERIMENTS WITH LIMITED TRAINING DATA AVAILABLE
Next, we evaluate the methods above using varying percentages of the training data from the second task onwards (we all the data for the first task the model has converged reasonable solution before the first distribution shift). Moreover, we augment the rehearsal batch size for ER, ER-ACE and ER-AML to 20, so that their compute cost equals DER++. This again on CIFAR- 10, M 20. Results averaged over 5 runs.
Method
% of Data Used
5%
10%
25% 50%
ER
17.3
22.5
28.0
33.2
DER++
17.4
19.9
24.8
32.8
SS-IL
15.2
21.7
28.6
31.9
ER-ACE
20.5
25,4
3.2
36.1
ER-AMI.
18.1
24.3
31.0
38.7
Again, that the proposed methods outperforms the baselines suggested above.
20

---

Published as a conference paper at ICLR 2022
CIFAR-10 M=5 Anytime Evaluation Acc
ER-ACE
ER-AML
30
ER
SS-IL
Accuracy
iCaRL
20
MIR
10
0
100
600
1,100
1,600
2,100
2.600
3,100
3,600
4,100
Iteration
CIFAR-10 M=20 Anytime Evaluation Acc
ER-ACE
40
ER-AML
ER
30
SS-IL
Accuracy
iCaRL
MIR
20
10
100
600
1,100
1,600
2,100
2,600
3.100
3,600
4,100
Iteration
L ADDITIONAL RESULTS
In this section we provide full results (shown in the figures below) for various memory sizes on all three datasets considered, i.c. Split CIFAR-IO, Split CIFAR-100 and Split Minilmagenet, with and without data augmentation. The results largely align with those presented but also illustrate the anytime performance.
L. ANYTIME EVALUATION WITHOUT DATA AUGMENTATION
L.2 ANYTIME EVALUATION WITH DATA AUGMENTATION
21

---

Published as a conference paper at ICLR 2022
CIFAR-10 M=50 Anytime Evaluation Acc
50
ER-ACE
ER-AML
40
ER
SS-IL
Accuracy
30
iCaRL
MIR
20
10
0
100
600
1,100
1,600
2.100
2.600
3.100
3,600
4,100
Iteration
CIFAR-10 1=100 Anytime Evaluation Acc
ER-ACE
50
ER-AML
ER
40
SS-IL
Accuracy
iCaRL
30
MIR
20
10
0
100
600
1,100
1,600
2.100
2,600
3.100
3,600
4,100
Iteration
22

---

Published as a conference paper at ICLR 2022
CIFAR-100 M=20 Anytime Evaluation Acc
ER-ACE
ER-AML
15
ER
SS-IL
Accuracy
iCaRL
10
MIR
5
0
100
350
600
850
1,100
1,350
1,600
1.850
Iteration
CIFAR-100 M=50 Anytime Evaluation Acc
ER-ACE
20
ER-AML
ER
SS-IL
Accuracy
15
iCaRL
MIR
10
5
0
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
CIFAR-100 M= 100 Anytime Evaluation Acc
ER-ACE
25
ER-AML
ER
20
SS-IL
Accuracy
iCaRL
15
MIR
10
5
0
100
350
600
850
1,100
1,350
1,600
1.850
Iteration
23

---

Published as a conference paper at ICLR 2022
Miniimagenet M=20 Anytime Evaluation Acc
17.5
ER-ACE
ER-AML
15.0
ER
12.5
SS-IL
Accuracy
iCaRL
10.0
MIR
7.5
5.0
2.5
0.0
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
Miniimagenet M=50 Anytime Evaluation Acc
ER-ACE
ER-AML
20
SS-IL
Accuracy
15
iCaRL
MIR
10
0
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
Miniimagenet M=100 Anytime Evaluation Acc
25
ER-ACE
ER-AML
20
ER
SS-IL
Accuracy
15
iCaRL
MIR
10
5
0
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
24

---

Published as a conference paper at ICLR 2022
CIFAR-10 5=5 Anytime Evaluation Acc with Augmentations
40
ER-ACE
ER-AML
30
ER
SS-IL
Accuracy
DER++
20
iCaRL
MIR
10
0
100
600
1,100
1,600
2,100
2,600
3,100
3,600
4,100
Iteration
CIFAR-10 M=20 Anytime Evaluation Acc with Augmentations
50
ER-ACE
ER-AML
ER
SS-IL
Accuracy
30
DER+ +
iCaRL
MIR
20
10
100
600
1,100
1,600
2.100
2,600
3,100
3,600
4,100
Iteration
CIFAR-10 M=50 Anytime Evaluation Acc with Augmentations
ER-ACE
ER-AML
ER
SS-IL
Accuracy
+ +
iCaRL
MIR
100
600
1,100
1,600
2.100
2.500
3,100
3,600
4,100
Iteration
CIFAR-10 M=100 Anytime Evaluation Acc with Augmentations
60
ER-ACE
ER-AML
ER
SS-IL
Accuracy
DER+
30
iCaRL
MIR
10
0
25
100
600
1,100
1,600
2,100
2,600
3.100
3,600
4,100
Iteration

---

Published as a conference paper at ICLR 2022
CIFAR-100 M-20 Anytime Evaluation Acc with Augmentations
20
ER-ACE
ER-AML
ER
15
SS-IL
Accuracy
DER+ +
iCaRL
10
MIR
5
0
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
CIFAR-100 M=50 Anytime Evaluation Acc with Augmentations
ER-ACE
20
ER-AML
ER
15
SS-IL
Accuracy
DER+
iCaRL
10
MIR
5
0
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
CIFAR-100 M=100 Anytime Evaluation Acc with Augmentations
ER-ACE
20
ER-AML
ER
SS-IL
15
Accuracy
DER+
iCaRL
10
MIR
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
26

---

Published as a conference paper at ICLR 2022
Miniimagenet M=20 Anytime Evaluation Acc with Augmentations
ER-ACE
ER-AML
15
ER
55-IL
Accuracy
DER++
10
iCaRL
MIR
5-
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
Miniimagenet M=50 Anytime Evaluation Acc with Augmentations
20
ER-ACE
ER-AML
ER
15
SS-IL
Accuracy
+
iCaRL
MIR
5
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
Miniimagenet M=100 Anytime Evaluation Acc with Augmentations
20
ER-ACE
ER-AML
15
ER
SS-IL
Accuracy
DER++
10
iCaRL
MIR
5-
100
350
600
850
1,100
1,350
1,600
1,850
Iteration
27


Reviews:
[{"review_id": "wclzbbN8AjZ", "reviewer": "Reviewer_aUDf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper does good work explaining the phenomenon in the introduction section, which the reviewer found interesting. The paper also states the metric learning idea, which sounds plausible. Unfortunately, the writing somehow doesn’t connect this idea to the proposed method in section 4. The paper assumes the readers are familiar with the contrastive learning literature and doesn’t explain the loss functions quite well. It doesn’t explain why minimizing the loss function will lead to the effects mentioned in section 1. Since the reviewer is not from the area of representation learning, I couldn’t justify the correctness of the method. Besides this main issue, some other comments are as follows:\n* The reviewer doesn’t find any experiments demonstrating the correctness of the method, i.e., whether the proposed method fixes the representation drift problem or not.\n* The page format, i.e., margins between sections, were adjusted significantly and affects the reading.\n* Add necessary parentheses over citations in Sections 3 and 4.\n* Use a larger arrow in Fig. 1 (left) to represent gradients. It is hard to distinguish between gradients for tasks 1 and 2. The authors may want to explain how the gradient for task 1 is computed since the learner is in the task 2 phase.\n\n", "summary_of_the_paper": "The paper proposed a new explanation of a performance drop phenomenon in a memory-based continual learning method, Experience Replay. The performance initially has a sharp drop when the task switch happens. As the training goes on, the performance drop can be improved but not as much as satisfactory. The paper analyzed this phenomenon from a representation learning perspective and applied a contrastive learning approach to reconciling the performance drop. The authors conducted experiments to illustrate their methods have a superior performance over baselines.", "summary_of_the_review": "The paper analyzed an interesting representation drift problem of a popular continual learning method. However, the paper doesn’t explain the loss functions in detail. The reviewer couldn’t justify the correctness of the applied contrastive learning method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "upEq3RwbAXM", "reviewer": "Reviewer_MwZR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well motivated.  Figure 1 is good to illustrate motivation.  It could be better to show the feature distribution with and without Asymmetric loss. Will the old class prototypes be pushed further away from the new ones? \n\nThe online continual learning scheme is valid and practical in real use cases. In an offline setting, after abundant iterations to replay old data, the performance can be recovered to some extent, but at the beginning stage, the accuracy will be drastically decreased. \n\nRegarding the formulation of AML, the loss for buffer data is still in cross-entropy form, wondering if there is any special consideration not to design it in ML form by sampling positive and negative data. \nAML is designed to be slightly more complex (sampling of positive and negative samples) yet less effective than ACE. It uses features from live training examples instead of the class weights. Could elaborate more on their respective resultant prototype distributions with AML and ACE. \n\nFrom the experiment results, ER-AML leads to unstable performance while using data augmentation. The author can elaborate more on this.  \n\nMore choices for the contractive loss could be explored. Also, the components of the proposed methods are not new, although it may be first used in this continual learning task. \n\nMinor adjustment- N is used twice: \"we denote the incoming N datapoints...\" and \"We use the P and N to denote the set of positive and negatives...\"", "summary_of_the_paper": "This paper studied one problem of continual learning which causes the forgetting of old classes, i.e new class representations will often overlap significantly with the previous classes, leading to highly disruptive parameter updates during the incremental training phase. To tackle this problem, this paper presented an Experience Replay (ER) based approach with asymmetric parameter update for old and new  data.  Two versions of asymmetric losses are proposed and compared: metric learning and cross-entropy based  losses. Experiments are conducted over multiple datasets to verify the effectiveness of the proposed method. ", "summary_of_the_review": "The overall idea of this paper is simple but effective. The motivation is well illustrated and the results are convincing. It can be improve with more insightful discussion and analysis on the outcome of the feature distribution (similar to Fig1) to show the proposed method can indeed separate the representation of old and new data along with the training. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P24mIHdaB1Q", "reviewer": "Reviewer_qLmu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is clearly written and the main ideas are very well presented\n- The motivation is clear and well illustrated by the experiments run in Figure 1\n- Solid benchmarking effort. Many recent baselines were compared against, for a range of buffer sizes, and results reported with confidence intervals.\n\nWeaknesses:\n- It appears that the advantage of this method may start to break down when the data stream is a combination of new classes and old classes, since the old class representations would also start to be updated; is this the case? There are some preliminary experiments in Section 5.5, but I would have liked to see a much more thorough analysis - what happens when the number of examples per new class in the incoming data batch is very low? What about at a range of \"blurriness\" levels?\n", "summary_of_the_paper": "This paper proposes a new update rule for the continual learning setting to boost performance when novel classes are added to the data stream.", "summary_of_the_review": "Overall, the paper presents a strong analysis of a potential failure mode in continual learning, proposes a simple method as a fix, and compares thoroughly against baselines to validate their results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FXy_T6olkEs", "reviewer": "Reviewer_mCT3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ The explanation of catastrophic forgetting from the perspective of abrupt representation drift is somewhat novel. The observation that the knowledge of old classes is recovered by stored samples after disrupted is interesting.\n+ Some new metrics are proposed, such as Averaged Anytime Accuracy, Total FLOPs, which make the comparison more rigorous.\n\nCons:\n+ Some important related methods of online continual learning are not included in comparison, e.g. [1, 2, 3]. In particular, [3] also uses a SupCon loss just as ER-AML.\n+ The paper claims old tasks suffer from a significant drop in performance when learning new tasks (Fig 1 right). However, this statement is not quite convincing, since the acc drop is not entirely related to representation drift. Some previous work [4, 5] confirm that forgetting is more likely to occur at the layers near the output, especially the last fc layer. Moreover, [6] also finds that adopting metric learning on representations results in less forgetting. To make it clear that forgetting happens at the last fc layer or the whole network, I suggest using NCM classifier to calculate acc here, or another better metric.\n+ The motivation of using metric loss is not clear. Compared to [3, 6], which also use metric loss for CL, the paper doesn’t provide enough special motivations or insights to use metric loss for reducing abrupt representation drift. What’s more, the idea to calculate cross entropy loss with replay batch only has already been used in [7]. The proposed losses need more theoretical explanation for why/how they work.\n+ Compared to SS-IL, the improvement is not significant.\n[1] Online Class-Incremental Continual Learning with Adversarial Shapley Value. AAAI 2021\n[2] Graph-Based Continual Learning. ICLR 2021\n[3] Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learning. CVPR Workshops 2021:\n[4] Large Scale Incremental Learning. CVPR 2019\n[5] Learning a Unified Classifier Incrementally via Rebalancing. CVPR 2019\n[6] Semantic Drift Compensation for Class-Incremental Learning. CVPR 2020\n[7] Discriminative Representation Loss (DRL): Connecting Deep Metric Learning to Continual Learning. arxiv.org/abs/2006.11234\n", "summary_of_the_paper": "In the online continual learning setting, representation of previous classes will change over time, especially at the task boundary. The authors present a new explanation for the abrupt representation change when applying experience replay (ER) algorithm. They claim that without sufficient training, the representation of new samples is initially mixed with that of previous classes, which causes a sharp increase of gradient resulting shift. To reduce the negative impact of negative new samples, they modify the ER by leveraging asymmetric losses on current data and replay data separately.", "summary_of_the_review": "Overall, this paper proposes an interesting phenomenon in online continual learning. However, the relation between the solution and the insights are not strong. Besides, the empirical improvements are marginal. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript develops new insights into how catastrophic forgetting takes place in the context of continual learning. The authors develop a new method based on this insight and demonstrate that it performs better than or as well as previously developed baselines, as well as showing that it is more widely applicable than close competitors (e.g. to cases where task boundary are unknown).  
The manuscript starts by pointing to evidence that catastrophic forgetting at task boundaries is due at least in large part to abrupt representation drift (e.g. in the penultimate layer of a network) caused by gradients coming from new class examples. Most reviewers found this novel and interesting. Reviewers also tended to be happy with the writing, motivation, and experimental results supporting the conclusions.
One of the reviewers recommends against publishing (3 - Reject): mCT3 cites positives in the novel explanation of forgetting and the development of new metrics (Averaged Anytime Accuracy) and Total FLOPs, which they say help make the analysis more rigorous.
However, fundamentally, they believe that the relationship between the insights and the proposed methods are not strong enough and that the methods do not provide more than marginal improvements empirically. They point to work such as SS-IL as a baseline which, in their opinion, is not improved upon significantly enough to adjust their recommendation.
The authors provide multiple effective rebuttals to the concerns, as well as detailed experimental analysis of SS-IL: 0. The new method is shown to be as good or better than SS-IL, 1. That their method are more computationally and memory efficient than SS-IL, 2. SS-IL requires task-ids, whereas they do not, 3. Detail analysis (Appendix B) shows that SS-IL mostly fails to learn the current task in the online setting in the miniImageNet case that the reviewer worries about, a fact that is obscured by the simple Acc metric. Reviewer mCT3 does not respond to the rebuttals in any substantive or compelling fashion, and leaves their score at 3/Reject.
While I believe that the reviewers concerns should be thoroughly addressed in a final version of the manuscript, I am in agreement with the 3 of 4 reviewers who recommend publication.
Decision: Accept (Poster)