Centroids Matching: an efficient Continual Learning approach operating in the embedding space
Authors: Jary Pomponi
Simone Scardapane
Aurelio Uncini
Venue: Accepted by TMLR ()

Abstract: Catastrophic forgetting (CF) occurs when a neural network loses the information previously learned while training on a set of samples from a different distribution, i.e., a new task. Existing approaches have achieved remarkable results in mitigating CF, especially in a scenario called task incremental learning. However, this scenario is not realistic, and limited work has been done to achieve good results on more realistic scenarios. In this paper, we propose a novel regularization method called Centroids Matching, that, inspired by meta-learning approaches, fights CF by operating in the feature space produced by the neural network, achieving good results while requiring a small memory footprint. Specifically, the approach classifies the samples directly using the feature vectors produced by the neural network, by matching those vectors with the centroids representing the classes from the current task, or all the tasks up to that point. Centroids Matching is faster than competing baselines, and it can be exploited to efficiently mitigate CF, by preserving the distances between the embedding space produced by the model when past tasks were over, and the one currently produced, leading to a method that achieves high accuracy on all the tasks, without using an external memory when operating on easy scenarios, or using a small one for more realistic ones. Extensive experiments demonstrate that Centroids Matching achieves accuracy gains on multiple datasets and scenarios.

Published in Transactions on Machine Learning Research (10/2022)
Centroids Matching: an efficient Continual Learning approach operating in the embedding space
Jary Pomponi
jary. pon.yoooppn@@uiinoo..ttt
Deportment of Information Engineering, Sapienza University of Rome, Italy
Simone Scardapane
Department of Information Engineering,
Sopienza University of Rome, Italy
Aurelio Uncini
aurelio. wtt.://wiciii@@iiooa......
Department of Infor Information Engineering,
Sapienza University of Rome, Italy
Reviewed on OpenReview: // net/ forum? id- 7g2Ql tQuur
Abstract
Catastrophic forgetting (CF) occurs when neural network loses the information previously learned while training on a set of samples from different distribution, i.e., a new task. Existing approaches have achieved remarkable results in mitigating CF, especially in a scenario called task incremental learning. However, this scenario unrealistic, and limited work has been done to achieve good results more realistic scenarios. In this paper, we propose novel regularization method called Centroids Matching, that, inspired by meta- learning approaches, fights CF by operating in the feature space produced by the neural network, achieving good results while requiring a small memory footprint. Specifically, the approach classifies the samples directly using the feature vectors produced by the neural network, by matching those vectors with the centroids representing the classes from the current task, or all the tasks up to that point. Centroids Matching is faster than competing baselines, and it can be exploited to efficiently mitigate CF, by preserving the distances between the embedding space produced by the model when past tasks were over, and the one currently produced, leading to method that achieves high accuracy on all the tasks, without using external memory when operating easy scenarios, using small one for more realistic ones. Extensive experiments demonstrate that Centroids Matching achieves accuracy gains on multiple datasets and scenarios.
1 Introduction
An agent which operates in the real world must be able continuously learn from the environment. Learning from stream of samples, usually in the form of static datasets, also called tasks, referred to as Lifelong Learning Continual Learning (CL). A continual learning scenario comes often with phenomenon called Catastrophic Forgetting (CF) McCloskey and Cohen 1989), that arises when agent loses the knowledge lcarned from past samples while extracting information from newer ones. This phenomenon inhibits the correct working of agents that operate in such scemarios, but it can be mitigated, or removed, using methods built for that purpose. A key point is that such methods must present a contained memory footprint, because we can't save all past samples encountered during the training Parisi et al. (2019), and the agents cannot. grow indefinitely, consuming all the memory. Thus, the external memory, intended as the collection of all the things saved on the hardware that must. be preserved across all the tasks, must be contained.

---

Published in Transactions on Machine Learning Research (10/2022)
Over the last years, large amount of research has been done about methods to alleviate the CF phenomenon. Usually, CL techniques are grouped into three categories (regularization-bese
methods, rehearsal methods, and architectural methods), and a method can belong to one, or more, categories at the same time (Parisi cct al.) 2019). The methods in the first set are designed in such a way that important paramcters of the model, with respect to past lasks, are preserved during the training of newer tasks using any sort of regularization technique, by directly operating over the paramcters of the model, or by adding a regularization term to the training loss (Li and Hoiem 2017 Kirkpatrick et al. 2017 Zenke et al., 2017 Serra al. 2018 Saha et al.) Rehearsal-based methods save portions of past tasks and use the information contained in the memory to mitigate CF while training on new tasks (Rebuffi et al. 2017 Chaudhry et al.
2020 Chaudhry et al. 2021).
2019 van de Ven et al. 2020
Chaudhry et al. 2021 Rosasco et al. 2021); the samples associated to past tasks can also be generated using a generative model, and in that case the methods are called pseudo-rehearsall Finally, architectural-based methods freeze important parameters or dynamically adjust and expand the neural network's structure, to preserve the knowledge associated to past tasks, while the model learns how to
solve the current (ask (Rusu et al. 2016 Aljundi et al., 2017 Yoon et al. 2017 Veniat et al. 2020 Pomponi
et al. 2021
Aside from developing techniques to solve CF, another issue is formalizing scenarios describing how tasks are created, how they arrive, and what information is provided to the model itself (e.g., the task identifie). Usually, a method is designed to solve a subset of all possible CL scenarios (Van de Ven and Tolias 2019h. In this paper, we operate on scenarios in which the identity of a task is always known during the training, but it may not be known during inference phase, and the classes are disjoint (the same class appears in only one task). If we can use the task identity during the inference phase, we have scenario called task imcremental, otherwise, the scenario is called class incremental; the latter is much harder to deal with, being closer to real-world scenario Van de Ven and Tolias (2019). The task incremental scenarios have been studied exhaustively, due to the simplicity of the problem, while fewer methods have been proposed to efficiently solve the latter. In this paper, we propose a method that achieves better results on both task and class incremental scenarios.
Our proposal, Centroids Matching, is a pure regularization-basee method when it comes to fight CF in task incremental problems (which are casier), while it uses an external memory, containing samples from past tasks, when we require to solve class incremental scenarios; when fighting the CF phenomenon in task incremental scenario, our approach requires no memory, since we force the model to keep its extraction ability by using the current samples and the current model, without the need to store the past model when the learning of a task is over, nor an external memory containing past samples. Our approach differs from the existing literature because it does not train the neural network using a slandard training procedure, in which a cross-entropy error is minimized, but it operates directly on the embeddings vectors outputted by the model, by producing an embedding for each point. These vectors are moved to match the centroids of the associated classes, which are calculated using a sub-set of training samples (support set). When fighting the CF phenomenon in task incremental scenario, our approach requires no memory, since we force the model to keep its extraction ability by using the current samples and the current model, without the need to store the past model when the learning of a task is over, nor an external memory containing past. samples.
2 Related Works
An agent that has the Continual Learning (CL) property is capable of learning from a sequence of tasks (De lange et al. 2021] without forgetting past learned knowledge. When past learned knowledge is lost, and with it also the ability to solve tasks already solved in the past, we have a phenomenon called Catastrophic Forgetting (CF) (French) 1999. McCloskey and Cohen 1989), which occurs because the information saved in the parameters of the model is overwritten when learning how to solve new tasks, leading to partial or total forgetting of the information. A CL method should be able of alleviating, or removing, CF while efficiently learning how to solve current lasks. Initial CL works focused on fighting the CF phenomenon on the easiest supervised CL scenario, called task incremental learning, but, recently, we have seen a shift toward class incremental scenario, being closer, and more suitable, to real-world applications; nevertheless, a limited number of proposed approaches focus on that specific scenario Masaana et al.) [020 Belouadah et al. )021). The main difference between the two is that in the first scenario we can classify only samples in
2

---

Published in Transactions on Machine Learning Research (10/2022)
Head Task
P2
Backbone
Head Task
Task Incremental Scenario
Class Incremental Scenario
Figure 1: The proposed approach applied on both Task (Ieft) and Class (right) Incremental learning scenarios. The bigger circles are the centroids the classes, while the smaller ones are samples from the same class of the corresponding centroid. We see that a CIL scenarios is solved by merging the embedding spaces together, into a new space that contains all the classes so far; the merging process is explained in Section 4.2
the context of a lask, thus the lask identity must be known a priori, while in the latter one the model must discriminate at inference time between all classes seen during the training procedure, without having the task identity. Depending on how a CL method achieves this goal, we can group, following Parisi et al.) (2019), the CL methods into three categories: regularization methods, rehearsal methods, and architectural methods. Our approach belongs to the first. set when we dealing with task incremental scenarios, and both the first. and the second one when the scenario is class incremental one.
Our approach regularizes the model by constraining the embeddings, but other approaches, based on the same principle, have been proposed over the years. One of the first was proposed in Hou et al. (2019), and it does not work directly on the embeddings of the model, but on the logits produced by it, and uses them to regularize the training by reducing the distance between the current model and the past one, while also correcting biases that arise when training a model solve a CL scenario (e.g. class imbalances). In Pomponi al. (0020), the authors proposed a regularizatiio-rceearss approach that works directly the embeddings space produced by the model. Given sample and the task from which the sample comes, the proposed method uses a regularization term that forces the model to reduce the distance between the current embeddings vector and the one obtained when the training on the source task was over; moreover, the approach requires a very small memory to work, but it can only regularize models that operate on task incremental sconarios because it requires lasks spaces to be separated. In Han and Guo (2021) a regularization-relee rehearsal met hod which uses the embeddings vectors to regularize the model is proposed. The vectors are used to calculate multiple contrastive losses used as regularization factors; also, mechanism that overwrites portion of the embeddings is used, enabling selective forgetting; unfortunately, the approach requires big external memory in order to achieve competitive results.
More CL scenarios exist, such as stream--spperised scenario, often called Online Incremental Learning, in which the model sees a sample only once, and the idea of using the embeddings to regularise the model has also been exploited in these scenarios. Starting from the same ground idea which inspired our approach, De Lange and Tuytelaars) (2021), the authors proposed a CL approach that operates over stream of samples, by continuously updating the prototypes extracted using the same stream, by using novel loss which synchronizes the latent space with the continually evolving prototypes. When batch is retrieved, it is used to update the model using the proposed loss function and the centroids, using combination of the current centroids and the embedding vectors associated with each image in the batch. When this process is over, the samples in the batch are used to populate a class-wise memory; the base idea, classifying using prototypes, is similar to our approach, with the difference that the authors used a different loss function that works both on positive and negative samples, and the prototypes are calculated mobile average while samples are retrieved from the stream, and, in the end, the concept of task is missing in their proposal. Similarly, the
3

---

Published in Transactions on Machine Learning Research (10/2022)
authors of Taufique et al. (0222) proposed an approach that works in the context of unsupervised domain adaptation, in which a buffer containing prototypes is used to calculate a contrastive loss against. the current. batch of samples. In the approach proposed in Kurniawan et al. (2021), which aims to solve online continual learning scenarios, the authors used many loss functions to train the model, and one of them is based on the similarity calculated in the embedding space, which pulls closer the samples belonging to the same class.
Other CL methods, that do not use the embeddings regularize the training, have been proposed over the years. The approaches belonging to the regularization-basee set fight CF by forcing the model's parameters, any output of the agent, which are relevant for past tasks, to be as close as possible to the optimal parameters obtained when these past tasks were over. One of the first methods that use a regularization approach to fight CF is Elastic Weight Consolidation (EWC), proposed in Kirkpatrick et al. (2017), that assigns an importance scalar to each parameter of the model, slowing the change of the parameters that are considered more important to preserve the ability to solve past tasks; in some cases this constraining approach could be too strong, leading to an incapacity of the model to learn how to solve new tasks. Other methods, such as the ones proposed in Saha et al. (2020) and [arajtabar et. al. (2020), regularize the model by foreing the gradients to go toward space of the parameters where the CF is minimized for all the tasks, while the current one is being solved as efficiently as possible, by moving the weights in the space that satisfics all the constraints. Memory- methods save small number of samples from each solved task, or generate synthetic samples using generative models, to be used jointly with the current training samples, in order to preserve past learned knowledge, These methods are often called rehearsal methods, or pseudo-rehearsal when a generative model is involved. The most representative method in this set, which is a hybrid regularzioo approach, is proposed in Lopez-Paz and Ranzato (2017), which uses the samples from the external memory to estimate the gradients associated to past tasks, which are used to modify the gradients associated with the current training samples, to solve, jointly, the current and the past lasks; moreover, this was one of the methods, along with plain rehearsal approach, that can improve the scores obtained on past tasks, supposing that the memory dimension is big enough to be fully representative of past tasks. A more straightforward approach, yet very effective, is to use the external memory to augment the current batch by concatenating a random batch extracted from the memory and the current batch, as proposed, for instance, in Chaudhry et al. (2019); Riemer et al. (2018); Yoon et al.] (2021). Being a straightforward approach, many other similar approaches, as well as theoretical studies to understand what CF really is and how to fight it, have been proposed over
the years (Rebulli el al. 2017 Rolnick el al. 2019 Ostapenko el al. 2022
3 Continual Learning Setup
We define a supervised CL. scenario as a set of N tasks T = (Ti)i=I..N, in which task can be retrieved only when the training on the current one is over; when a new task is collected, the past ones cannot be retrieved anymore (except for testing purpose). Each task T; is a set of tuples (x, v), where x is a sample, and y E Y2 is the label associated to it, where Y; is the set of classes contained in the task T, Also, the tasks are disjoint, meaning that: N Y; = Ø (a class cannot belong to two different tasks). The goal of a CL method is to help the model to generally perform well on all learned tasks so far, by adapting to new tasks while preserving the previously learned knowledge. A method that solves task at the expense of another is not desirable, and thus a trade-off must be achieved, by looking at the overall performance.
Assuming that the tasks' boundaries are always known and well defined during the whole training procedure, i.e., we always know when a task is over or retrieved, we follow Van de Ven and Tolias (0019 to define two different scenarios, based on how the inference procedure is carried out:
Task Incremental Learning (TIL): in which the task's identity of a sample is given during the inference phase.
Class Incremental Learning (CIL): in which the task's identity of a sample is not given during the inference phase.
The difference is minimal, but yet crucial. In fact, we can consider the first one as a more simple and theoretical scenario, which is also the most studied one. Its main limitation is that, to correctly classify
4

---

Published in Transactions on Machine Learning Research (10/2022)
sample, must know the task from which the sample comes, and usually, this is not the case. In fact, such scenarios are more suitable to develop and test novel methods, before adapting them to an agent that operates on more realistic scenarios. The second scenario is more difficult and the agents suffer CF drastically, because not only the model must be regularized, but the space of the prodiction must be extended to include also the upcoming classes, leading to faster forgetting of past lasks.
It must be noted that the scenario definition is untied from the architecture of the neural network involved, which can have any topology, as long as the scenario's rules are followed. Nevertheless, usually, multi-head strategy is adopted to operate in TIL scenarios, in which backbone is shared, and, for each task, smaller neural network, usually called head, is used to classify the samples belonging to that task; each head takes as input the output of the backbone, making the prediction spaces of the tasks well separated. The shared backbone is also used when operating in CIL scenarios, but the classification head is usually just one, whose output neurons are expanded (the newer classes are added during the training) when new task collected.
4 Centroids Matching (CM) framework
Our approach is inspired by the Prototypical Networks proposed in Snell al. (0017), following the idea that there exists an embedding space, also called feature space, in which vectors cluster around the most probable centroid, representing class, and called prototype. Following the approach used in Prototypical Networks, our model does not follow a standard classification approach in which cross-entropy loss is minimized, but it uses the model to ext ract features vector from an inpul sample, and then it forces the vector to be as close as possible to the correct centroid, representing the class in the embedding space of the task.
In the following section, we explain how this approach can be used to easily mitigate CF in multiple CL scenarios.
4.1 TIL scenario
Following Section 3 suppose the model consists of two separate components: feature extractor, also called backbone, v/ R RD, where I is the size of an input sample and D is the dimension of the features vector produced by the backbone, and a classifier head fi RD + RE, one for each task, where E is the dinension of the task specific feature vector. The backbone operates as a generic feature extractor, while each head is a specific model that, given the generic vector of features extracted by the backbone, transforms the vector into a vector of features for that specific task. Given an input sample x and the task from which the sample comes, the final vector, used for training and testing, carried out by combining the functions: (x) = f. o (x). The backbone remains unique during the whole training, while a new head added each time a new task is encountered. Both the backbone and all the heads are updated during the whole training process.
When a new task Tj is available, we extract and remove a subset of the training set, named support set and identified as S;, containing labelled training samples that won't be used to train the model. This support set is used to calculate the centroids, one for each class in the task. A centroids, for a given class k, in the space of the task i, is the average of the feature vectors extracted from the samples in the support set, calculated using the corresponding head:
of
Σ cx(x) (x,))SS)
(1)
|
where Sk is the subset of St that contains only samples having label k. During the training, these centroids are calculated at each iteration, in order to keep them up to date. Then, given the Euclidean distance function d RM, R + R,, and a sample x, our approach produces distribution over the classes based on the softmax distance between the features produced using (x() and the centroids associated to the current task:
5

---

Published in Transactions on Machine Learning Research (10/2022)
exp(-d(ck, (x())
ply k|x, 1) = Evex,
(2)
(x))))
We note that, in this scenario, it makes no sense to calculate the distances between different tasks' heads, since each head produces centroids placed in their own embedding space (see the left side Fig. D without interfering with the others. The loss associated to sample is then the logarithm of the aforementioned probability function:
L(x, k) -logp(v k)
(3)
If the current task is not the first one, in order to preserve past learned knowledge, we need to regularize the model. When a new task is collected, we clone the current model, both the backbone and all the heads created so far, which we indicate as E7.), for each lask Then, while training on the current lask, we augment the loss using the distance between the features extracted using the cloned model and the one extracted by the current one, both calculated using the current set of training samples, without the support of external memory containing past samples. The regularization term is the following:
R(x, = Σ 7 K<
((((()),
(4)
The proposed regularization term works because it relies on the fact that Prototypical Network can be reinterpreted as linear model Mensink al. (2013). Using this simple regularization term, force the model to preserve the ability to extract the same information that it was able to extract when the previous task was over. Moreover, since the regularization term is calculated only using samples from the current lask, no external memory is needed to regularize the model. The overall regularization approach works because the past heads are trained at the same time as the new ones, while leaving the weights of the model unconstrained, as long as the output distance minimized. Then, the final loss for a task which is not the first one is:
ku(x,k,t) log = k|x,t) t) AR((,
where À is a scalar used to balance the two terms. When task over, the final centroids for the classes in the task are calculated and saved. Thus, the external memory contains, when all the tasks over, only the controids for the classes seen during the training, and thus the required memory is negligible.
To classify sample x from a task I, we use the same approach used during the training process, based on the distance between the centroids of that task and the features extracted from the sample x:
argmax p④y k|xxt)
(6)
KEY,
This is possible because we always know from which task the sample comes. In the next section, will show how this can be achieved when the task identity is not known.
4.2 CIL scenario
The class incremental scenario is more difficult if compared the TIL scenario, because the identity of the fask is available only during the training process but not during the inference phase, requiring different.

---

Published in Transactions on Machine Learning Research (10/2022)
approach. Most of the methods fails to overcome CF in this scenario, mainly because a single head classifier is used to classify all the classes, leading to faster CF, because the capacity of single head is limited. Instead, in our approach, we keep the heads separated as in the TIL scenario, and we train by projecting the embeddings vectors into a shared embedding space, containing all the classes so far, leading to an casier classification phase.
As before, we train on the first task using the loss 5 If the task not the first one, we also add projection loss to the training loss, which is used to project all the embedding spaces into single one. First of all, we use an external memory M, which can contain fixed number of samples for each task, or fixed number of samples during the whole training, removing portion of past images when new ones need to be stored. In our experiment, we use a fixed sized memory, which is resized each time a new task must be saved in the memory.
If the current task is not the first, we augment the current dataset with the samples from the memory. Since the memory is smaller than the training set, we use an oversampling technique when sampling samples from the memory, in way that batch always contains samples from past tasks. We define the projected loss, which is a modified version of the equation 2 as:
p = k|xx = exp(-d(pi(c)), 1 EjsiPi(x)))
Ewer, ((ps-d ) (i
where = - Yy contains all the classes up to the current task, and the function Pt(+) is a projection function, one for each task, that projects the embeddings, and the centroids, from the task wise embedding space to the shared embedding space. For this reason, the labels y must be scaled accordingly using simple scalar offset. For generic lask i, the projecting function y: is defined as:
p. (x(x) (x) Sigmoid((s(e((z))) + tx(e(()))
in which the functions 8i, 4 E E are, respectively, the scaling and the translating function, implemented using two small neural networks, trained along with the backbone and the heads. The final loss for this scenario is defined as:
Lx, t) - log = k|x,t)
(9)
At inference time, we know the task associated to a sample, we can perform the inference step as in the TIL scenario, otherwise, we classify directly the samples, without inferring the corresponding task:
y = argmax ply = k| KEY
(10)
where Y is the set of all the classes seen during the training, and N is the total number of tasks. In this way, all the tasks are projected into the same embedding space, thus the classification of sample does not require the task identity.
5
Experiments
5.1 Experimental setting
Dataset: We conduct extensive experiments multiple established benchmarks in the continual learning literature, by exploring both TIL as well as the harder CIL. The datasets we use to create the scenarios are: CIFAR10, CIFAR100 [Krizhevsky] 2009), and TinyImageNet (a subset of ImageNet (Deng et al. 2009] that contains 200 classes and smaller images). To create a scenario we follow the established approach, where the classes from a dataset are grouped into N disjoint sets, with N the number of tasks to be created. We use CIFAR10 to build a scenario composed of 5 tasks, each one with 2 classes; using CIFAR100 we create 10 tasks with 10 classes in each one; in the end, the classes in TinylmageNet are grouped into 10 tasks, each one containing 20 classes.
7

---

Published in Transactions on Machine Learning Research (10/2022)
Baselines: we test approach against many continual learning methods: Gradient Episodic Memory
(GEM) [oppe-Paz and Ranzato 2017). Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017
Online EWC (OEWC) (Schwarz et al. 2018), Experience Replay (ER) Chaudhry et al. (2019), SS-IL Ahn
ct al. (021), CoPE De Lange and Tuytehars (2021), despite being introduced for solving stream domain scenarios, under both multi epochs approach (ME) and stream approach (S), and, in the end, Embedding Regularization (EmR) (Pomponi et al. 2020); regarding the latter, it only works on TIL scenarios, and the
other results are omitted. We also use two baseline approaches: Naive Training, in which the model is trained without fighting the CF, and Cumulative Training, in which the current training task is created by merging all past tasks as well as the current one.
Hyper-parametees:: for each method, we searched for the best hyper-parameterr following the results presented in respective papers. For EWC, we used 100 as regularization strength weight for all the scenarios. For GEM we used a memory for each task, composed of 500 samples for CTFAR10 and 1000 for the other experiments. In the EmR memory, we saved 200 samples from each task, while for CoPE we have a memory for each class, and each one can contain up to 300 samples. Lastly, for ER and SS-IL,, we used fixed memory size of 500 for CIFAR100 scenarios and 1000 otherwise. Regarding our approach, the support set contains 100 images from the training set of cach task, and we set the penalty weight À to 0.1 for CIFAR10, 0.75 for CIFAR100 and TinyImageNet; regarding the CIL scenarios, we used a fixed size memory of 500 for each scenario. For both ER and CM, the memory is shared across all the tasks, and it is resized, by discarding images from past tasks, each time a new task must be saved.
Models and Training: for each dataset we use ResNet20 (He et al. 2016) architecture, trained using SGD with learning rate set to 0.01 and momentum to 0.9. For CIFARIO-100, we trained the model for 10 epochs each task, while for Tinylmagenet we used 30 epochs; the only exception is SSIL, for which we used 100 epochs for each scenario. For each classification head, we used two linear layers, with the first layer that takes as input the output of the backbone, followed by a ReLU activation function, and then an output layer whose output size depends on the number of classes in the current task. Regarding our proposal, each head is composed of two linear layers, and it projects the output of the backbone into a vector of 128 features (the output of the ResNet model has 64 values). We repeat each experiment 5 times; each time the seed of the experiment is changed in incremental way (starting from 0). Regarding our proposal, since it is not rehearsal method when operating in the TIL scenario, after each task we save the statistics of the batch norm layers, which are retrieved when a sample from the corresponding task must be classified. Also, we used the following augmentation schema for the proposed datasets: the images are standardized, randomly flipped with probability 50%, and then a random portion of the image is cropped and resized to match the original size.
A scenario, usually, is built by grouping the classes in an incremental way (the first n classes will form the first task, and on). We use this approach for the first experiment, instead, when the experiment is not the first, each of the N tasks is created using randomly selected subset of classes. Using this approach, different scenario is built for each experiment, and more challenging scenarios could be created since the correlation wetween the classes disappears.
Metrics: to evaluate the efficiency of CL method, we use two different metrics from Diaz Modziguz et al. (2018), both calculated on the results obtained on the test set of cach task. The first one, called Accuracy, shows the final accuracy obtained across all the lest splits of the tasks, while the second one, called Backward Transfer (BWT), measures how much of that past accuracy is lost during the training on upcoming tasks. To calculate the metrics we use matrix R E NNN, in which an entry R,j is the test accuracy obtained on the test split of the task j when the training on the task i is over. Using the matrix R we calculate the metrics as:
N RN
BWT = Ei Σj (Rej Ría)
Accuracy-
NNN 1)
Both metrics are important to evaluate a CL method, since a low BWT does not imply that the model performs well, especially if we also have low Accuracy score because, in that case, it means that the
8

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: Mean and standard deviation (in percentage), calculated over 5 experiments, of achieved Accuracy and BWT for each combination of scenario and method; some results are missing because the corresponding method does not work on that specific scenario. The best results for each combination of dataset-scenario are highlighted in bold.
Method
CHPARCI
CIPATUIO
Fine Iaaggeeee
CIFARIE
CIPANTI
Flay Eangghee
DWT
Arcunnyy
BWT
Accurscy
BWT
Accurncy
DWT
Accuracy
DWT
Accuracy
BWT
Acetmmty
-1139
1800
Osmmlative
-2 22788a
00.8
2.00
1300gmmm
63.06
-284g1
56.9210.12
-3.28g00
30.8620.0
-3,88.911
20 2auee
IWC
77.00
-5.11-9
58.02.5m
3.88
27.4
-225249
17,07a0
6.13gnm
40.58.11 tn
OWC
15.0733.00
70.07
24.57a19:0
-500llani2
15.7041.2
-G1 Bagn
597anm
1123 45
ER
-2 2.
90 90.50
-8
70.55
-17.15200
63.31,023
-9019
52.00g1.0
-07 21
25.004m:0
-3330
GEXT
90-15 10
71.85-our
2280,141
17.09
FanRt
-2 2s
01.30e1 m1
-2
66.8652011
55-16
-32,911.30
56.0111.7
-3001.201
2113
-2212
0.50
CoPE (S)
49:14
3.25:1.4
1.61.010
CoPE (MB)
1.34,0
CM
92. 92.72.1 mm
74.6.1
134602
47.90
11877amm
64.64_11744
-22
27.91-0
-22.30-mm
12,04.032
approach regularizes too much the training approach, leaving no space for learning new tasks. In the end, the combination of the metrics is what we need to evaluate. In addition to these metrics, we also take into account the memory used by each method. The memory is calculated as the number of additional scalars, without counting the ones used to store the neural network, that must be kept in memory after the training of task while waiting for the new one be collected; the memory used during the training process is not counted additional, since it can be discarded once the training is over.
The formulas used to calculate the approximated required memory arc:
EWC: this approach saves a snapshot of the model after each task. The required memory is: N x P.
OEWC: similar the EWC, but it saves only one set of paramcters, which updated after the training on each task. The final memory is P.
Rehearsal approaches (CIL): these methods need an external memory in which subset from each task is saved, and then used to regularize the training. The required memory depends on the number of images saved, and is calculated as IxMxN.
EmR: this approach requires not only the images but also the features vector associated with each image (the output of the backbone). The required memory size is: (D + I)xMxN.
CM (TIL): requires only to save, after each task, the centroids of the classes in the tasks; thus, the memory size is x T.
where N the number of tasks, P is the number of parameters in the neural network, I the dimension of the input images, D is the dimension of the feature vector extracted by the model, and E is the dimension of the output related to our proposal.
To perform all the experiments, we used the Avalanche framework, which implements the logic to create tasks and evaluate the CL approaches. Regarding the methods, we implemented in Avalanche EMR and Centroids Matching, while the others are already present in the framework. The code containing all the files necessary to replicate the experiments is available here
Implementation:
5.2 Results
Classification results: Table summarizes all the results obtained across the experiments, in terms of Accuracy and BWT. We can see that CM significantly improves the results in all the scenarios. The improvements on TIL scenarios are significant, by achieving an accuracy that is close to the upper bound set by the cumulative stratogy. Morcover, the results are better than all the other methods when comes CIL scenarios. Surprisingly, the ER approach achieves also good results in all the scenarios, but not as good the ones achieved by our proposal. The results clearly show the difficulty discrepancy between TIL and
9

---

Published in Transactions on Machine Learning Research (10/2022)
GEN
ER
CM CaFE
Required memory
Accuracy
10a
40
13*
(T.)
100
500
1000
1500
2000
Samples saved the memory
Number of tasks
(a) How the accuracy score, obtained on CI- FAR10 CIL scenario, changes when the number of the samples saved in the memory changes.
(b) How the required memory grows when the number of tasks increases. The images have size 3 x 32 x 32.
The images show the required memory for each method, as well as how the accuracy changes when the rehearsal memory grows (only for methods that require an external memory containing past samples).
Figure 2:
Table The time, in terms of soconds, roquired to train the models on CIFAR10. The standard deviation well the scenario in which the times are obtained are shown.
Methud
Scenario
Task 1
Task 2
Task
Task
Task
Naive
TIL-CIL
9.74.0.26
10.84
10. 10.40.0..2
11.06:0.16
10.50
Cummlative
TIL-CIL
11.0540.20
20.54g
20.90
47.85
55.98
EWC
TIL-CIL
13.61 10.6020.020 10.020
17. 17.76o.n0
22.32.0024
28.001.0
28.89
OEWC
TIL-CIL,
11.87
16. 13..11
16. 16.42=0.67
16. 72..6...1
17.29
ER
TIL-CIL
12.40
34.00
61. 61.17=11.2
88, 20.28
113.78.2.62
GEM
TIL-CIL
11.7940.44
26. 26.9640.96
55.58 40.07=1.06
70.47
EmR
TIL
9.88
18.97
43.08 43.08_20.0
72.49+7.40
102. 102.204u.an
SS-II.
CIL
13.0g0.42
47.23 47.23.22.02 38.60
61.97
641.33
CM
TIL
24.07
28.6210.17
35.27 on
41.7620007
48.90
CM
CIL
20.0710.49
30.6210.17
45.2710.12
54.7610.24
68.0810.27
CIL, because approaches that work well on the first one, drastically fail to overcome the CF in the second one; also, it seems that only methods that use an external memory are capable of achieving good results on CIL, and the sole parameters regularization not enough to fight CF.
Memory comparison: the memory required by CL methods crucial aspect, and in this section we study how the accuracy score is correlated to this aspect. Figure 2 shows all the aspects related to the memory size. All the results are obtained using the same training configuration used for the main experiments. The image 2a shows the memory usage, correlated with the achieved accuracy score, required by each method when solving CIFARIO CIL scenario. Firstly, we see that GEM is not capable of achieving competitive results even when a large subset of samples is saved, while the others achieve good results even with smaller memory dimension, and this probably because large number of samples is required to correctly estimates the gradients which are used to regularize the training. Regarding the other approaches, we see that our proposal achieves better results even when using smaller memory. Not all the methods require containing samples from past tasks, and shoowss how the memory required by all the memory changes when the number of tasks grows. We clearly see that, when it comes to solving TIL problems, our proposal requires a smaller memory than all the others. When looking at the results in Table | for CIL problems, and combining them with the curves in Image 2b we can conclude that, despite a large amount memory requested by some methods, few of them are capable of achieving good results; on the other hand, when solving CIL scenarios our approach becomes a rehearsal one, and the required memory almost the same if compared to other rehearsal approaches.
external memory
10

---

Published in Transactions on Machine Learning Research (10/2022)
(a) The embedding space when the first lask is over.
(b) The embedding space when the third task is over.
(c) The embedding space when the last task is over
Figure 3: The images show how the embeddings spaces, associated with the first task from CIFAR10 TIL scenario, change while training on new tasks. We can see that, despite the small changes in the shape of the samples, the overall space is preserved during the whole training. The points are projected into a bi-dimensional space using PCA (Hotelling 1936). Better viewed in colors.
(a) The emnbeddin space when the second task is over.
(b) The embedding space when the third task
(c) The embedding space
(d) The embedding space when the last task is
when the fourth task is
Figure 4: The images show how the merged embeddings space obtained on CIFAR10 CIL changes during the training on all the tasks. The images clearly show that new classes are added without interfering with the ones already present in the space. To visualize clearly the clustering space, used Voronoi diagrams over the 2D projections of the centroids, obtained using PCA (Hotelling 1936). The samples omitted for clarity. Better viewed in colors.
Training time comparison: the time required to train a model depends on the Cl approach used to fight the CF. Table 2 contains the time, in seconds, required to train an epoch each of the 5 lasks of TIL CIFAR10 scenario exposed before. We can see that, among the most performing approaches, our proposal is the one that requires the lowest training time: it requires more time than the others during the first task since the extraction of the prototypes from the support set is involved, but then the overall time grows slower than the other approaches. The main difference is the use of the memory in fact when our method is used to solve CIL scenarios, the time required is closer to the other rehearsal approaches.
Analysis of the embedding spaces produced by Centroids Matching: this section we analyze how the regularization approach proposed influences the shape of the embedding space produced by model. In Figure 3 we see how the embedding space, extracted from model trained on CIFAR10 TIL scenario, changes while new tasks are learned: the regularization term is capable of keeping the embedding space almost unchanged, and well separable, during the whole training process. Is also interesting to how proposal merges the embedding spaces during the training on CIL seenario, and this aspect is shown in Figure 4 We can see that the classes remain highly separable even in the late stages of the training procedure. The merged space achieved in an incremental way, by inserting new classes into the existing embedding space, without moving alrcady present centroids. For example, we see that, when passing from the first space the second one, new classes are added on the left of the existing embedding space, without interfering with the existing centroids. This is possible because the distance regularization works well, and also because
11

---

Published in Transactions on Machine Learning Research (10/2022)
Table The results, in terms of average and standard deviation calculated over 2 runs, obtained on CIFARIO CIL scenario when varying the merging strategy used, are shown. The results are both terms of Accuracy and BWT (in the brackets), and both are calculated when training on the last task over.
Task
Task
Task 3
Task
Task
Accuracy
Translae
58.90 ( 33.99]
34.35
(-50.75)
58.60 26. 26)
75.05
(-.0.(-20.30
93.60
64.10 -33.99)
Linear
43.15 (-3477)
51.00 (-35 35.40)
46.55 (-4.880)
63.35 (-2.95)
89.35
58.08 (-40.21)
Offset 47.25 (-50.90)
46.30 42.05)
44.35 (-6..1)
61,25 288.55)
87.85
57.40 (-41.1)
None 43.45(-53.900
41.701 46.95)
42.00( 50.06)
(-2.74)
87.80
36.01(-41..0
the approach is capable of adapting the model to the embedding space, by correctly projecting the centroids and the samples of newer lasks.
5.3 Ablation Study
Comparing merging procedures for CIL scenario. As exposed in Section 4.2 the merging function used to merge the embedding spaces uses a scale plus transaction function. Here, we study how the choice of the merging function pe(-) affects the results. To this end, we implemented different functions:
Trane-Trasslate: the merging function proposed in Section 4.2
Linear: a simple linear layer is used to project the embeddings vector into the new space.
Offset: offset is calculated using a linear layer on the embedding, and it is used to shift the embeddings of given task.
None: the merging step is performed directly on the embeddings outputted by the model.
For each approach, the weights of the merging networks are shared between the centroids and the embeddings of the same task, to avoid adding many parameters. Tn Table 3 the results are shown. We see that the Scale-translate approach achieves better results, on average, than all the other approaches, probably due to its inherent capacity to transform the embeddings. The only exception is the second task, in which the approach mentioned above loses more accuracy. Also, as expected, the approach None achieves the worst results but, surprisingly, it is capable of achieving decent average accuracy.
How the dimension of the support set affects the training procedure? Being the support set crucial to our proposal, we expect that its dimension affects the overall training procedure. On the other hand, we also expect that, once an upper bound on the number of support samples is stepped over, the results are not affected anymore, since the same centroids could be calculated using fewer samples. T the results obtained while changing the dimension of the support set. We can clearly see that, under a certain threshold, the results are very close. When the threshold is exceeded, we see a decrease in the achieved accuracy score. This could happen because more images are removed from the training set in order to create the support set, and this negatively affects the results, since some patterns could be missing from the training dataset.
Different Merging approaches for embeddings and centroids. In the experiments so far we took into account only the merging approach in which the same merging strategy, and the same weights to perform the merging, are used for both embeddings and centroids. In this section, we also explore the approach which different strategics are applied scparately or when the same strategy uses different weights for centroids and embeddings, in order to understand better which approach is the best one when isolated. The results of this study are exposed in Table 5 We can see that the best results, overall, are achieved when the Scale- Translate merging approach is applied to the embeddings. By combining the results with the ones in the Table 3 we can conclude that the best models are achieved when both the centroids and the embeddings are projected using the same Scale-Trane Transsare layer.
How does affect forgetting? In this section we analyze how the parameter X, used to balance the regularization term in [ affects the obtained results. The results are shown in Table 0 and we can see that setting X too high leads the training process to fail when the scenario is CIL one, and inhibits the training
12

---

Published in Transactions on Machine Learning Research (10/2022)
Table 4: The table shows the accuracy, averaged over runs, obtained while changing the number of samples in the support set. The results are calculated using CIFAR10 scenarios; the hyperparameters the same used in the main experimental al section.
Table 5: The table shows how combining the merging strategies, used by Centroids Matching, affects the final accuracy obtained on CIFAR10 CIL.
Embeddings
S-T
MLP
Offiset
None
Support set size
Centroids
S-T
62.40
56.92
60.91
60.67
10
50
100
200
500
MLP
62.13
56.55
59.25
62.05
TIL
90.86
90.86
91.70
89.79
90.70
Offset
62.84
60.94
58.37
61.82
CIL
59.63
63.97
63.55
63.16
61.04
None
61.69
62.05
60.67
59.05
Table 6: The results, in terms of average and standard deviation calculated over 2 runs, obtained CIFAR10 CIL scenario when varying the merging strategy used, are shown. The results are both in terms of Accuracy and BWT (in the brackets), and both are calculated when training on the last task is over.
0.01
0.1
10
100
C10 TIL
89.25 (-.258
91.39 (-..94)
8.94)
79.39 ( 10.94)
50.00 (2.03
50.00
(0.00(112.01 12.01)
C10 CIL
59.75 42.75)
62.32 ( 39.51)
49.57 38.82)
42.19 (- 17.70)
15.95 15.95)
when it comes to TIL scenarios (achieving small forgetting but small overall accuracy). Moreover, the more the regularization term grows, the more the results degenerate; the same is true also when A is too small, leading to a model that is not able to remember correctly past tasks. In the end, we can conclude that the best results are obtained when the weight parameter is close and smaller than 0, leading to a model with balanced trade-off between remembering past tasks and training on the current one.
6 Conclusions
In this paper, we proposed approach to overcome CF in multiple CL scenarios. Operating the embedding space produced by the models, our approach is capable of elfectively regularising the model, leading to a lower CF, requiring memory when it comes to solving easy CL scenarios. The approach reveals that operating on a lower level, the embedding space, can lead to better CL approaches while having the possibility to analyze the embedding space to understand how the tasks, and classes within, interact.
References
H. Ahn, J. Kwak, S. Lim, H. Bang, H. Kim, and T. Moon. Ss-il: Separated softmax for incremental learning. In Proceedings of the LEEE/CVE International Conference on Computer Vision, pages 844 853, 2021.
R. Aljundi, P. Chakravarty, and T. Tuytelaars. Expert gate: Lifelong learning with a network of experts. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366-3375, 2017.
F. Belouadah, A. Popescu, and Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 135:38-54, 2021.
Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, episodic memories in continual learning. arXiv preprint arxiv: 1902.100.1111 190.. 2019.
Dokania, P. H. Torr, and M. Ranzato. On tiny
Chaudhry, A. Gordo, P. Dokania, P. Torr, and D. Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 6993 7001, 2021.
M. De Lange and T. Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9250-8259, 2021.
13

---

Published in Transactions on Machine Learning Research (10/2022)
M. De lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. TEEE Transactions on Pallern Analysis and Machine Intelligence, 2021.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
N. Diaz-Rodriguez, V. Lomonaco, D. Filliat, and D. Maltoni. Don't forget, there more than forgetting: new metrics lor continual learning. Xiv preprint arXiv: 1.10:1810.11160 2018.
M. Farajtabar, N. Azizan, A. Mott, and A. Li. Orthogonal gradient descent for continual learning. In International Conference on Arlificial Intelligence and Statistics, pages 3762 3773. PMTR, 2020.
R. M. French. ropast lorgetting in connectionist networks. Trends in Cognilive Sciences, 3(4):128 135, 1999.
X. Han and Y. Guo. Contrastive continual learning with feature propagation. arXiv preprint arXiv:2112.01715 2021.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognilion, pages 770 778, 2016.
Hotelling. Relations between two sets of variates. Biometrika, 28(3 377, 1936. ISSN 00063444.
S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin. Learning unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognition (CVPR), June 2019.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Graasska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 3526, 2017.
Krizhevsky. Learning multiple layers features from tiny images. Technical report, 2009.
M. R. Kurniawan, X. Wei, and Y. Gong. Online continual learning via multiple deep metric learning and uncerlainty-guiied episodic memory replay 3rd place solution for icev 2021 workshop sslad track 3a continual object classification. arXiv preprint arXiv:2111.02755 2021.
Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935- 294477997777 2017.
D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. Advances in newral information processing systems, 30, 2017.
M. Masana, X. Liu, B. Twardowski, M. Menta, A. D. Bagdanov, and J. van de Weijer. Dlass-incremental learning: survey and performance evaluation image classification. ar preprint arXiv:2010. 11207..5777.. 2020.
M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pages 109-165. Academic Press, 1989.
T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Distance-based image classification: Generalizing to new classes at near-zero cost. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(11): 2624-2637, 2013. doi: N0.1109/TPAMI.
O. Ostapenko, T. Lesort, P. Rodríguez, M. R. Arefin, A. Douillard, I. Rish, and L. Charlin. Foundational models for continual learning: An empirical study of latent replay. arXio preprint arXiv:2205.00 00329, 2022.
G. Parisi, R. Kemker, L. Part, C. Kanan, and S. Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71 71, 2019.
14

---

Published in Transactions on Machine Learning Research (10/2022)
J. Pomponi, S. Scardapane, V. Lomonaco, and A. Uncini. Efficient continual learning in neural networks with embedding regularization. Neurocompuling, 397:139-18 148, 2020.
J. Pomponi, S. Scardapane, and A. Uncini. Structured ensembles: An approach reduce the memory footprint of ensemble methods. Neural Nebworks, 144:407-418, 2021. ISSN 0893-6080. doi: https: //doi.org/ //doi.org/10.001////
S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2010, 2017.
M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv:1810.111100, 2018
D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
A. Rosasco, A. Carta, A. Cossu, V. Lomonaco, and D. Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. arXiv preprint arXiv:2103.15851 2021.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv:1606. 16666666,,,7,, 2016.
G. Saha, I. Garg, and K. Roy. Gradient projection memory for continual learning. In International Conference on Learning Represenlations, 2020.
J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Tch, R. Pascamu, and R. Hadsell. Progress& & compress: A scalable framework for continual learning. In International Conference on Machine Learning, pages 4128-4537. PMLR, 2018.
J. Serra, D. Suris, M. Miron, and A. Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In International Conference Machine Learning, pages 4548-4557. PMLR, 2018.
J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. Advances in newral information processing systems, 30, 2017.
A. M. N. Taufique, C. S. Jahan, and A. Savakis. Unsupervised continual learning for gradually varying domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 3740-3300 37500, June 2022.
G. M. Van de Ven and A. S. Tolias. Three scenarios for continual learning. arXiv preprint 1977941904.07734. 2019.
G. M. van de Ven, H. T. Siegelmann, and A. Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature commanicaatinns, 11(1):1 14, 2020.
T. Veniat, L. Denoyer, and M. Ranzato. Efficient continual learning with modular networks and task-driven priors. arXiv preprint ar Xiv:20112 12631, 2020.
J. Yoon, E. Yang, J. Lee, and S. J. Hwang. Lifelong learning with dynamically expandable networks. ar preprint arXio:17088 01547, 2017.
J. Yoon, D. Madaan, E. Yang, and S. J. Hwang. Online coreset selection for rehearsal-based continual learning. aixiv preprinl Xr::22106. 01 01085, 2021.
F. Zenke, B. Poolc, and S. Ganguli. Continual learning through synaptic intelligence. In D. Precup and Y. W. Teh, editors, Proceedings of the 94th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3987 3995. PMLR, 06 11 Aug 2017.
15


Reviews:
[{"review_id": "C2MD3PIHB9", "reviewer": "Reviewer_uPg7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Writing, Problem Statement, Motivation, Related Work\n### Strengths \nThe paper does a thorough job of presenting the problem statement in a concise manner. The introduction explains the continual learning problem and issues of catastrophic forgetting. This work extends beyond the standard \"Task Incremental Learning\" version of continual learning (known task boundaries and identities) to the \"Class Incremental Learning\" formulation (unknown task identities for samples). While the text proposes two somewhat different approaches for the two settings, the text clearly presents the distinction while limiting any confusion. \n\n### Weaknesses\n- Portions of the text are redundant and repetitive. For example, the distinction between task incremental and class incremental learning appears in Sec 1, Sec 2, Sec 3 and Sec 4.2. \n- While most relevant literature has been cited, I think the discussion on the distinctions between existing work and the proposed work can be elaborated further. For example, Prototypical Networks serves as an important backbone for this work and has been just cited in passing. There also seem to be a lot of similarities to  De Lange and Tuytelaars (2021). This work also has been very briefly mentioned. \n\n\n\n\n# Approach\n\n### Strengths, Novelty\nThe presented approach is very intuitive to follow. The key component is a novel regularization used in conjunction with Prototypical Networks trained in a continual learning setting. The regularization ensures that the previously learned prototypes (or centroids) are unperturbed. This overall idea is simple, novel and might be of general interest. \n\nThe proposed approach is also fairly general and can be applied in both the continual learning problem formulations - Task Incremental Learning and Class Incremental Learning. Due to the difference in the two settings the paper proposes slightly modified implementations of the regularization. I believe these ideas provide interesting and useful guidelines for implementing general regularization methods in the continual learning framework.\n\nUnlike some previous work on continual learning, the proposed approach also makes efficient use of memory during training.  \n\n### Weaknesses, Concerns, Questions\n- The objective in Eq 4 relies on images from novel tasks for ensuring that the embedding of previously seen images is unperturbed. However, this might not be very effective if the images are from significantly different distributions. \n- Since the amount of regularization increases with number of additional tasks, it might be the case that the model performs poorly on the later tasks due to over-regularization. This might also mean that the training is sensitive to the order of tasks seen. \n\n\n\n# Results\n\n### Strengths\n\nThe proposed work outperform the accuracy of reported existing works on numerous continual learning benchmarks (CIFAR10, CIFAR100, TinyImageNet). Furthermore, the reported accuracy is higher for both formulations of the problem (TIL, CIL).\n\nThe paper presents a thorough analysis of the different features of the proposed work. The analysis of memory consumption shows that the performance of the proposed work scales better compared to existing work. The memory consumption of the proposed work also increases slower with increasing number of tasks. \n\nThe results present very interesting visualizations of the learned feature embeddings. The visualizations show that the centroids are indeed unperturbed when training on novel tasks. It also demonstrates that the model learns to cleverly assign novel centroids to minimize conflicts with previously learned centroids. \n\n### Weaknesses\n- The BWT metric seems to be a better metric for measuring the amount of forgetting. However, the proposed work seems to underperform on this metric while being better on overall accuracy. \n- To the best of my understanding, De Lange and Tuytelaars (2021) is the closest related work. However, it looks like experimental comparisons to this work has been omitted. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Problem**:\nThis work addresses the problem of continual learning *i.e.* learning a sequence of tasks without forgetting/deteriorating performance on the previous. The work focuses on the standard setting of a sequence of image classification tasks. \n\n**Solution**: The paper proposes a solution called \"centroid matching\" which involves learning and memorizing the centroids of the classes for each task. In addition to this, the training involves novel regularization methods which help in alleviating issues of forgetting. The regularization ensures that previously learned centroids are not perturbed while training on novel tasks. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Some suggestions for improving the paper:\n#### From weaknesses in Approach above\n- Analysis of sensitivity to the order of tasks might be interesting to add. For example, if the subsets of cifar100 were permuted, does the final overall accuracy stay the same?\n- Analysis of sensitivity to more significant changes in distribution might be interesting to compare with existing works. For example, training on a subset of cifar10 and then a subset on tinyimagenet and then a subset of cifar100. \n\n#### From weaknesses in Results above\n- Could you add an explanation or experimental analysis for the worse BWT?\n- Is there a reason that De Lange and Tuytelaars (2021)  cannot be included in the comparisons? \n\n", "broader_impact_concerns": "The paper does not include a broader impact section. However, I do not see any obvious ethical concerns in this work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CGD9kja7uh", "reviewer": "Reviewer_7Yto", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\n\n\n\n## Strengths\n\n1. The paper does a great job of explaining the method very clearly.  \n\n2. The proposed method performs well under task and class incremental setups.  \n\n\n\n## Weaknesses\n\n1. **Baselines**: My first concern is about the baselines. I believe there have been several methods proposed in the past two years that perform significantly better than the classic CL methods, such as EWC and GEM. In addition, while CoPE is motivated by streaming settings, I believe a comparison would be nice since the fundamental assumptions in both works are similar.\n\n2. **Computation**: From the paper, it seems like to calculate the regularization term (Eq. (4)), we need both the previous backbone and heads. So I think the runtime is twice as the naive model. However, there are no runtime comparisons between the baselines, and I think it is necessary to compare the runtimes/computation requirements.\n\n3. **Memory Size**: In Sec. 5.1, the authors mention they use different memory sizes for different methods. For instance, for the CIFAR-100 benchmark, GEM uses 1000 examples, ER uses 500 examples, and EmR uses 2000 examples. I think this makes the evaluation confusing. Could the authors explain the rationale behind this choice? \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies task incremental and class incremental continual learning where the authors propose to learn and use class centroids for classification. The Centroid Matching (CM) framework uses a held-out set to calculate the centroids during the training and then uses(distribution of) the euclidean distance of the inputs and centroids as the objective. In addition, in the task incremental setting, a regularization term will be used that penalizes new task data for being close to previous tasks' centroids. In contrast, in the class incremental setting, since the classification head is shared, CM uses an episodic memory and projects embeddings into a shared space. The authors then validate the effectiveness of the CM approach in both task and class incremental setup, in addition to some ablation studies.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\nI believe the following changes are crucial (see above for details):  \n1- Including more recent/related baselines.  \n2- Providing runtime comparison between different methods.  \n3- Explain the different memory sizes or use a similar memory size for all methods.  ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iPxJmDuckt", "reviewer": "Reviewer_NcrE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work proposes a straightforward and simple approach for continual learning. I found it to be well-written and easy to follow, for which I congratulate the authors. However, I believe that several additional steps need to be taken for the proposal to be clearly outlined and fairly evaluated and for related works to be more accurately represented. I list them in the following:\n\n## Proposal Design\n- **Distinction w.r.t. CoPE**: the presented approach is quite similar to CoPE, proposed in (De Lange and Tuytelaars, 2021), as also mentioned in the text. Both approaches follow the same idea and basically optimise the same loss, but I reckon there are some differences (i.e.: CoPE proposing a specific heuristic for centroid update vs. CM introducing an architectural solution by having a separate $f_i$ for each task). I strongly recommend expanding the discussion of (De Lange and Tuytelaars, 2021) to let the reader better understand how these two works differ. This needs to be complemented with a thorough experimental analysis, thus also providing a quantitative counterpart to the analysis (more on that below).\n- **Usage of replay memory**: I find it somewhat convoluted that the authors propose two separate methods for the TIL and CIL settings. In my understanding, the only rationale for this dichotomy is that TIL is significantly easier than CIL, thus not requiring the introduction of a separate memory buffer. Nonetheless, I expect the memory-equipped version of CM to outperform memoryless CM in TIL too. Instead of proposing two practically identical methods, I believe that a clearer exposition could be made by only proposing memory-equipped CM and then investigating the possibility of removing the memory in TIL in a separate ablation study.\n\n## Fair Evaluation\n- **Better competitors are needed**: the selection of evaluated competitors in section 5.1 is not reflective of the current state of the art in continual learning classification. The only competitor which is not designed for the TIL setting is ER (still, its effectiveness varies considerably depending on implementation details [1, 2]). In a modern paper targeting the CIL setting, I expect the inclusion of several of the following strong recent baseline methods: iCaRL [3], LUCIR [4], GDumb [5], LSIL [6], GSS [7], DER++ [8], SS-IL [9], ER-ACE [10]. As discussed above, I believe that the experimental inclusion of CoPE (De Lange and Tuytelaars, 2021) is also of the utmost importance.\n- **Number of epochs**: I could not find any information about the number of training epochs, which is a pretty important factor conditioning the results that are obtained in CL classification. This information should be featured clearly in the text.\n- **Uneven memory in comparisons**: from my understanding, the authors write that the following memory buffers are used: \n\t- CIFAR10: GEM uses 500 examples per task (so, 2500 in total?), EmR uses 200 (1000 in total?), ER uses 1000 (in total? per task? this is not clear, CM uses 500 (also, unclear whether this is the total or per-task number).\n\t- CIFAR100: GEM uses 1000 examples per task (10000 in total, 1/6 of the overall dataset?), EmR uses 200 (1000 in total?), ER uses 500 (in total? per task?), CM uses 500 (in total? Per task?).\n\t- TinyImageNet: GEM uses 1000 examples per task (10000 in total, 1/6 of the overall dataset?), EmR uses 200 (1000 in total?), ER uses 1000 (in total? per task?), CM uses 500 (in total? Per task?).  \n\nI think that this pieces of information should be provided more clearly; from my understanding, I see a remarkable disparity in the allowance of replay buffer across these methods, which makes the proposed experimental comparison hard to understand. I would recommend using at least the same memory size for all competitors.\n- **Memory usage**: in line with and beyond the previous point, I agree with what is stated in Sec. 5.1, namely that different competitors employ widely varying working memory for their training and operation. I believe that it would be very useful for the reader if a memory occupation/performance analysis graph were provided for a couple of experimental CIL settings. Such a comparison clearly allows for a direct comparison across all methods. It would also be interesting to highlight the support set memory footpring (which is not needed by the competitors) in this comparison.\n- **Batch normalisation trick**: is the batch norm trick described in the first paragraph of page 8 also employed for other competitors in TIL benchmarks? If not, I would be very interested in seeing how it affects the performance of e.g. ER, given that the batch norm drift is factor known to affect CL results in a major fashion [11].\n\n## Exposition of related works\n- **Improving the characterisation of TIL vs CIL**: the authors suggest that mostly of the experimental evaluation in CL happens in the TIL scenario and \"fewer methods have been proposed to efficiently solve [CIL]\". I believe that this is no longer reflective of the current state of literature, where CIL is widely understood - as also stated by the authors - to be much more interesting than TIL [12, 7, 13, 14] and a consistent number of baselines targets this setting directly [3, 4, 5, 6, 7, 8, 10]. I would argue that in the last two/three years it is much easier find papers referring to CIL in major conferences than TIL. I would suggest accounting for this fact in the main text and de-emphasising the role of TIL. \n- **Improving the characterisation of regularisation and rehearsal methods**: in the related works section, I do not agree with some proposed characterisation of regularisation and rehearsal methods. For the former, it is written that they \"fight CF by forcing the model's parameters [...] to be as close as possible to the [previous optimum]\". This description is not necessarily fitting to methods that regularise the output instead of parameters, such as LwF. Similarly, I do not at all agree with the statement that \"the most representative method in [rehearsal methods] is [GEM]\". In fact, I would rather characterise it as a somewhat hybrid solution that uses replay examples to compute a regularisation/projection term instead of simply replaying stored data. Furthermore, I do not understand the statement that \"[GEM] was the first method that can improve the scores obtained on past tasks\": this is not true, as ER (which was originally proposed in the 90s) also fits this description, if one performs few epochs on a given task, then the model is likely not to perfectly fit the data and thus to improve its performance on it upon replay. I recommend adjusting these sentences to better characterise regularisation and rehearsal methods.\n- **Introduction could use more citations**: as a minor suggestion, I believe that the introduction presents multiple claims which are not supported through citations in literature and thus end up appearing as quite unjustified. I suggest introducing further citations so as to make it possible for the reader to go in depth on specific aspects of CL by themselves. Examples of sections that would need it are \"we can't save all past samples encountered during training\", \"class-incremental [...] is much harder to deal with, being closer to a real-world scenario\".\n\n## Misc Minor Points\n- **Unclear claim in sec. 5.3**: I do not entirely understand the claim made in the first paragraph of sec. 5.3, which states that there exists a critical number of samples beyond which results don't improve since the same centroids are computed with fewer samples. I am not sure that this is immediately evident, could it be experimentally verified that one obtains the same centroids for different sizes of the support set?\n- **Reordering ablation tables**: I would suggest presenting the Tables 2-5 in the same order they are cited in text.\n- **Typos**: while reading, I found the following typos:\n\t- the spelling of Matthias De Lange's name is inconsistent in citations;\n\t- sec. 4.1, two lines above eq. 1: centroids -> centroid\n\t- sec. 4.1, one line below eq. 5: the final centroids, for the classes in the task, are calculated -> the final centroids for the classes in the task are calculated\n\t- sec. 4.2, in eq.7 $Y_i$ seems to have a different meaning w.r.t. $Y_i$ in eq.2, I would recommend choosing a different symbol\n\t- sec. 4.2, in eq.7 it seems that the second operand of d is $p_i(x)$, should it not be $p_i(e_i(x))$? Is it operating on an embedding or on the input?\n\t- sec. 4.2, one line above eq.10: corresponding class -> corresponding task\n\t- why is there no variance in the results of EWC and OWC on TinyImageNet?\n\n[1] Chaudhry, Arslan, et al. \"On tiny episodic memories in continual learning.\" ICML workshop 2019.  \n[2] Buzzega, Pietro, et al. \"Rethinking experience replay: a bag of tricks for continual learning.\" 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.  \n[3] Rebuffi, Sylvestre-Alvise, et al. \"icarl: Incremental classifier and representation learning.\" Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017.  \n[4] Hou, Saihui, et al. \"Learning a unified classifier incrementally via rebalancing.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.  \n[5] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. \"Gdumb: A simple approach that questions our progress in continual learning.\" European conference on computer vision. Springer, Cham, 2020.  \n[6] Wu, Yue, et al. \"Large scale incremental learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.  \n[7] Aljundi, Rahaf, et al. \"Gradient based sample selection for online continual learning.\" Advances in neural information processing systems 32 (2019).  \n[8] Buzzega, Pietro, et al. \"Dark experience for general continual learning: a strong, simple baseline.\" Advances in neural information processing systems 33 (2020): 15920-15930.  \n[9] Ahn, Hongjoon, et al. \"Ss-il: Separated softmax for incremental learning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.  \n[10] Caccia, Lucas, et al. \"New insights on reducing abrupt representation change in online continual learning.\" ICLR 2022.  \n[11] Pham, Quang, Chenghao Liu, and Steven Hoi. \"Continual normalization: Rethinking batch normalization for online continual learning.\" ICLR 2022.  \n[12] Farquhar, Sebastian, and Yarin Gal. \"Towards robust evaluations of continual learning.\" ICML workshop 2018.  \n[13] Masana, Marc, et al. \"Class-incremental learning: survey and performance evaluation on image classification.\" arXiv preprint arXiv:2010.15277 (2020).  \n[14] Belouadah, Eden, Adrian Popescu, and Ioannis Kanellos. \"A comprehensive study of class incremental learning algorithms for visual tasks.\" Neural Networks 135 (2021): 38-54.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a continual learning method that operates in the Class- and Task-Incremental scenarios by learning through centroid matching and inferring via a nearest neighbour scheme. The authors evaluate their proposal on three commonplace CL classification datasets and carry out several ablation studies to better characterise their design decisions and the behaviour of the proposed approach.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Summing up from my previous points, I recommend the following revisions (listed by descending importance)\n- Expanding and enriching the theoretical comparison with CoPE (De Lange and Tuytelaars, 2021);\n- Introducing CoPE as a full-fledged experimental competitor in this paper;\n- Introducing better CIL competitors to the experimental section (I recommend choosing among iCaRL, LUCIR, GDumb, LSIL, GSS, DER++, SS-IL, ER-ACE);\n- Providing the results for memory-equipped CM in TIL and exploring the trade-off of TIL accuracy/memory occupation linked to removing its memory;\n- Mentioning the secondary role of TIL for current literature and highlighting that a relevant portion of contemporary works refer to CIL instead;\n- Revising the statements in the related works section concerning regularisation methods and GEM;\n- Introducing more citations in support of the claims of the introduction;\n- Further explaining/adding experimental evidence in support of the claim in sec. 5.3;\n- fixing the typos listed above;\n- Reordering tables 2-5.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for mitigate catastrophic forgetting in task-incremental and class-incremental learning settings. The approach builds upon Prototypical Networks, which performs classification using a softmax over distances to prototypes formed by taking the centroid of the embeddings of examples belonging to each class. In the task-incremental learning setting, each task gets a new head, and the model is regularized so that the embeddings of the new examples using the old heads match the embeddings obtained using a frozen clone of the model made before initiating training on the new task. In the class-incremental learning setting, all tasks are additionally projected to a common embedding space where all classes can be discriminated, and a small set of examples from previous tasks are used to ensure that it is possible to separate the classes in the common embedding space.

Reviewers noted that, although the proposed approach is similar to CoPE (De Lange and Tuytelaars, 2021), it is novel and performs well, but requested additional baselines. The authors have added these baselines, and all reviewers now recommend acceptance.

I am satisfied with the revisions the authors have made over the course of the review process and recommend acceptance. There are a few changes that I believe could further strengthen the paper:

- Given the discussion between the authors and Reviewer NcrE of what constitutes a fair CoPE baseline, I feel that the paper should describe the CoPE baseline setup in greater detail than it currently does, and include both the single-epoch and multi-epoch CoPE results that they provided over the course of the response period.
- I believe there is a missing log in Eq. 9. Furthermore, the first paragraph of Section 4.2 implies that the loss on the shared embedding space in the CIL setting is applied in addition to the other losses applied in the TIL setting, but these losses do not appear in Eq. 9. It would be helpful to clarify these points.
- As Reviewer NcrE privately in their recommendation, there is a red CoPE label in the legend of Figure 2a, but no red bars in the graph.
Decision: Accept (poster)