FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning
Authors: Dipam Goswami
Yuyang Liu
Bartłomiej Twardowski
Joost van de Weijer
Venue: NeurIPS 2023 poster (2023)
Keywords: Continual Learning
Class-Incremental Learning

Abstract: Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks and thus suffers from catastrophic forgetting. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. In this paper, we explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes. In an analysis of the feature distributions of classes, we show that classification based on Euclidean metrics is successful for jointly trained features. However, when learning from non-stationary data, we observe that the Euclidean metric is suboptimal and that feature distributions are heterogeneous. To address this challenge, we revisit the anisotropic Mahalanobis distance for CIL. In addition, we empirically show that modeling the feature covariance relations is better than previous attempts at sampling features from normal distributions and training a linear classifier. Unlike existing methods, our approach generalizes to both many- and few-shot CIL settings, as well as to domain-incremental settings. Interestingly, without updating the backbone network, our method obtains state-of-the-art results on several standard continual learning benchmarks. Code is available at https://github.com/dipamgoswami/FeCAM.

this paper, FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning
Dipam Goswami Yuyang Liu
Bartlomiej Twardowski 1,266 Joost van de Weijer Depaarment of Computer Science, Universitat Autònoma de Barcelona
CComputer Vision Center, Barcelona 'University of Chinese Academy of Sciences
$ State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences 5 Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences 'DEEAS-NBRR {dgoswami, btwardonski, joost) www./ecc...b....... sunshinell
Abstract
Exemplar-free intremmaatee learning (CIL.) poses several challenges since it prohibits the rehearsal of data from previous tasks and thus suffers from catastrophic forgetting. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. we explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes. In an analysis of the feature distributions of classes, we show that classification based on Euclidean metrics is successful for jointly trained features. However, when learning from non-stationary data, we observe that the Euclidean metric suboptimal and that feature distributions are heterogeneous, To address this challenge, we revisit the anisotropic Mahalanobis distance for CII.. In addition, we empirically show that modeling the feature covariance relations better than previous attempts at sampling features from normal distributions and training a linear classifier. Unlike existing methods, our approach generalizes both many- and few-shot CIL settings, well as domain-incrementt settings. Interestingly, without updating the backbone network, our method obtains state-of: of- the-art results several standard continual learning benchmarks. Code is available at /uttps://gitee com/ dipangosvami/FeCAN
Introduction
In Continual Learning (CL), the learner is expected accumulate knowledge from the ever-changing stream new tasks data. As a result, the model only has access to the data from the current task,
making it susceptible to catastrophic forgetting of previously learned knowledge [46] (3]. This
phenomenon has been extensively studied in the context of Class Incremental Learning (CIL) [3 61
9 74], where the objective is to incrementally learn new classes and achieve the highest accuracy for all classes encountered so far in a task-agnostic way without knowing from which tasks the evaluated samples are [56). While one of the simplest approaches to mitigate forgetting storing exemplars of each class, it has limitations due to storage and privacy concerns, e.g., in medical images. Hence, the
focus has shifted towards more challenging exemplar-free CIL methods [0] 63
35 2]
In exemplar- free CIL methods, the challenge to discriminate between old and new classes without access to old data. While some methods (79 [1 78 80) trained the model new classes favoring plasticity and used knowledge distillation 20 to preserve old class representations, other methods [43]
3 41 35 froze the feature extractor after the first task, thus favoring stability and incrementally
The corresponding author Yuyang Liu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

OO Old Classes
O A New Classes
Decision
Boundary
(a) Joint Training
(b) CIL High Stability
(c) CIL High Plasticity
(d) CIL FeCAM
Figure 1: Illustration of feature representations in CIL. settings. In Joint Training (a), deep neural networks learn good isotropic spherical representations [17] and thus the Euclidean metric can be used effectively. However, it is challenging to learn isotropic representations of both old and new classes in CIL settings. When the model is too stable in (b), it is unable to learn good spherical representations of new classes and when it is too plastic in (c), it learns spherical representations of new classes but loses the spherical representations of old classes. Thus, it is suboptimal to use the isotropic euclidean distance. We propose FeCAM in (d) which models the feature covariance relations using Mahalanobis metric and learns better non- non-linear decision boundaries for new classes.
learned the classifier. One of the drawbacks the inability learn new representations with a frozen feature extractor. Inspired by transfer learning [49]. the objective of these classffer-incrementtal methods is to make best use of the learned representations from the pretrained model and continually adapt the classifier to learn new tasks. Recently, pretrained feature extractors have been used in exemplar-fiee CIL by prompt-based methods [63], using linear discriminant analysis [1] and with a simple nearest class mean (NCM) classifier [22] These methods use transformer model pretrained on large-scale datasets like ImageNet-211
[45 and solely focus on classifier inneimeennnneeettall learning.
This paper investigates methods to enhance the representation of class prototypes in CII,, aiming to improve plasticity within the saaility--avoorinn
plassffirr-nncrmmenn setting. A standard practice in 70 722 is to obtain the feature embeddings of new class samples and average them to generate class-wiss prototypes. The test image features are then classified by computing the Euclidean distance to the mean prototypes. The Euclidean distance is used in the NCM classifier, following [7]. which claims that the highly non-linear nature learned representations eliminates the need to learn the Mahalanobis (65 I1 metric previously used (38). Our analysis shows that this holds true for classes that are considered during training, however, for new classes, the Euclidean distance is suboptimal. To address this problem, propose use the anisotropic Mahalanobis distance. In Fig. [ we explain how the feature representations vary in CIL settings. Here, the high- stability case in CTL is explored, where the model does achieve spherical representations for new classes in the feature space, unlike joint training. Thus, it intuitive to take into account the feature covariances while computing the distance. The covariance relations between the feature dimensions better captures the more complex class structure in the high-dimensional feature space. Additionally, in Fig. 3 we analyze singular values for old and new class features observe the changes variances in their feature distributions, suggesting a shift towards more anisotropic representations.
few-shot CIL [1 42
76,
While previous methods (88 proposed learning Mahalanobis metries, we propose using optimal Bayes classifier by modeling the covariance relations of the features and employing class prototypes. We term this approach Feature Covariance- Aware Metric (FeCAM). We compute the covariance matrix for each class from the feature embeddings corresponding to training samples and perform correlation normalization to ensure similar variances across all class representations, which is crucial for distance comparisons. We investigate various ways of using covariance relations in continual settings. We posit that utilizing Bayes classifier enables better learning of optimal decision boundaries compared to previous attempts (67] involving feature sampling from Gaussian distributions and training linear classifiers. The proposed approach is simple to implement and requires no training since we employ a Bayes classifier. The Bayes classifier FeCAM can be used for both mhoy CIL and few-shot CIL, unlike existing methods. Additionally, we achieve superior performance with pretrained models on both class- insremennt and domain-incremenn incremental benchmarks.
2
Related Work
Many-shot class-ineremental learning (MSCIL) the conventional setting, where sufficient training data is available for all classes. A critical aspect in msot CIL methods is the semantic drift in

---

the feature representations [68] while training on new tasks. While recent methods use knowledge
distillation [20] or regularization strategies [2]
68 to maintain the representations of old classes.
these methods are dependent on storing images 30
7 21
14
16
6
5
24 representations [23
or instances [*] from old tasks and becomes ineffective in practical cases where data privacy proposed freezing the feature extractor after the first task and learning only the classifier on new classes. We follow the same setting and do not
required. Another set of methods 43 3 13
41 [35
violate privacy concerns by storing exemplars.
On the other hand, Few-Shot clase-nncrementll Learning (FSCIL) considers that very few (1 or 5) samples per class are available for training [3] 53 58 Generally, these settings assume big first task and freezes the network after training on the initial classes. To address the challenges of FSCIL, various techniques have been proposed. One approach is to use meta-learning to learn how to learn new tasks from few examples 39
54]. Another approach to incorporate variational inference to learn a distribution over models that can adapt to new tasks 44)] Most FSCIL methods g
42
76 70 72 obtains feature embeddings of a small number of examples
and averages them to get the class- wise prototypes. These methods uses the Euclidean distance to classify the features assuming equally spread classes in the feature space. We explore using the class prototypes in both MSCIL and FSCIL settings.
Since the emergence of deep neural networks, Euclidean distance is used in NCM classifier follow- ing [17], instead of Mahalanobis distance (38]. Mahalanobis distance has been recently explored for out-of-distribution detection with generative classifiers (29] and an ensemble using Mahalanobis 22 and also within the context of cross-domain continual learning (50..
3 Proposed Approach
3.1 Motivation
For the classification of hand-crafted features, Mensink el al. (NCM) classifier using (squared) Mahalanobis distance DM instead of Euclidean distance to assign an image to the class with the closest mean (see also illustration in Fig. 2
38 proposed the nearest class mean
argmin D DM (I, Dar(z, My) = (x myy) M(x My)
(1)
y=l....
where Y is the number of classes, z, My E RD, class mean My = TX,1 Exex, x. and M a positive definite matrix. They learned low-rank matrix M WTW where W E Rm D. with m S D.
However, with the shift towards deep feature representations, Guerriero el al. [1 assert that the learned representations with deep convolutional network X' RDD eliminate the need of learning the Mahalanobis metric M and the isotropic Euclidean distance D. can be used as follows:
y" argmin De (b)()), De(o(s), = (ø(x) - ( (Φ(x) py) y=
where (xx) My E R. My = T Exex, d(x) is the class prototype or the average feature vector of all samples of class y. Here, (xx) is the feature vector corresponding to image and could be the output of penultimate layer of the network. In Euclidean space, M I, where is an identity matrix.
The success of the NCM classifier for deep learned rep- resentation (as observed by (17) has also been adopted by the incremental learning community. The NCM clas- sifier with Euclidean distance is now commonly used in incremental learning [44] 68 [0 79 72 42 70, (6).. However, in incremental learning, we do not jointly learn the features of all data, but are learning on non- static data stream. As a result, the underlying learning dynamics which result in representations on which Eu- clidean distances perform excellently might no longer be valid. Therefore, we perform a simple compari- son with the Fuclidean and Mahalanobis distance (see Fig. 4) where we use network trained 50% of
Fealture Space
Originel samples
P Prototype
Contour lines
(a) Euclidean Distance
(b) Mahalanobis Distance
Figure 2: Illustration of distances (contour lines indicate points at equal distance from prototype).
3

---

1.2
3.0
Old Classes New
53
0.0
2.5
0.0
100
200
300
(a) Singular values
(b) Old classes
(c) New classes
Figure 3: (a) Singular values comparison for old and new classes, (b-c) Visualization features for
old classes and new classes by t-SNE, where the colors of points indicate the corresponding classes.
classes of CIFARI00 (identified as old classes). Interestingly, indeed the old classes, on which the feature representation is learned, are very well classified with Euclidean distance, however, for the new classes this no longer holds, and the Mahalanobis distance obtains far superior results.
As second experiment, we compare singular values of old and new classes (see Fig. 3a). We observe that the singular values of new class features vary more and are in general larger than those of old classes. This points out that new class distributions are more heterogeneous (and are more widely spread) compared the old classes and hence the importance (like Mahalanobis). This is also confirmed by t-SNE plot of both old and new classes (see Fig. 3b.c) showing that the new classes, which were not considered while training the backbone are badly modeled by a spherical distribution assumption, as is underlying the Euclidean distance.
heterogeneous distance measure
Based on these results, two extreme cases of CIL are illustrated in Fig. D considering maximum stability where the backbone frozen, and maximum plasticity where the training is done with fine-tuning without pre- venting forgetting. In this paper, we revisit the nearest class mean classifier based on heterogeneous distance measure. We perform this for classifier incremental learning, where the backbone is frozen after the first task. However, we think that conclusions based on the heterogeneous nature of class distributions also have consequences for continual deep learning where the representations are continually updated.
FaDAM (common)
FcCAM (per
Assureccy
60
518
522
40
20
20
Old
New
All
Figure 4: Accuracy Comparison of NCM (Euclidean) and FeCAM (Mahalanobis) us- ing common covariance matrix and matrix per class on CIFAR 100 50-50 (2 task) se- quence, for Old, New, and All classes at the end of the learning sequence.
3.2 Bayesian FeCAM Classifier
When modeling the feature distribution of classes with multivariate normal feature distribution N(p Ey), the probability of sample feature x belonging class y can be expressed as,
-1
P(a|C = y) = exp 2
(x (0-1)) (0-Ma)'' Ey)' -ryd))
(3)
is straightforward to see that this is the optimal Bayesian classifier, since:
argmax P(Y = argmax P(X
P(X) = argmax P(X())))) = argmax P(X (4)
y
where optimal boundary occurs at those points where each class is equally probable P(y) = P(u,).
Since logarithm is a concave function and thus argmax P(XIV) = argmax log P(X
"
argmax Pox Y) = Argmax(-(e2 Myy) ", (x y p.)} argmin DM (I,HH)
(5)
y
where the squared mahalanobis distance D (1. (by) (x My) T (,,))))
In the following, we elaborate on several techniques that can be applied to improve and stabilize Mahalanobis-baset distance classification. We apply these, resulting in our FeCAM classifier (an ablation study in section 4.2.4 confirms the importance of these overall performance).

---

Covariance Matrix Approximation. We obtain the covariance matrix from the feature vectors ((x) corresponding to the samples at any task. The covariance matrix can be obtained in different ways. A common covariance matrix Di can be incrementally updated as the mean covariance matrix for all seen classes till task as follows:
E E1 |
|
| Y
(6)
Y
where E' refers the common covariance matrix obtained from the features of samples E X from all classes seen in task and |.| denotes the number of classes seen till that task. This common covariance matrix will represent the feature distribution for all seen classes and requires storing only the common covariance matrix from the last task.
The other alternative is to use covariance matrix Σy for y 1,..Y to represent the feature distribution of each class separately. Here, My is the covariance matrix obtained from the feature vectors of all the samples E Xy. This will involve storing separate matrix for all seen classes.
Normalization of Covariance Matrices. The covariance matrix Σ, obtained for each class will have different levels of scaling and variances along different dimensions. Particularly, due to the notable shift in feature distributions between the old and new classes, the variances are much higher for the new classes. As result, the Mahalanobis distance of features from different classes will have different scaling factors, and the distances will not be comparable. So, in order to be able to use covariance matrix per class, we perform a correlation matrix normalization on all the covariance matrices. In order to make the multiple covariance matrices comparable, we make their diagonal elements equal to 1. A normalized covariance matrix can be obtained as:
(1. (1))
(1(1)= v (i, o,(i) = Eg(i.)
(7)
where (1) and Ø (ن) refers to the standard deviations along the dimensions i and respectively.
We identify the difficulties of obtaining an invertible covariance matrix in cases when the number of samples are less than the number of dimensions. So, we use a covariance shrinkage method to get a full-rank matrix. We also show that a simple gaussianization of the features using tukey's normalization (55] is also helpful.
Covariance Shrinkage. When the number of samples available for a class is less than the number of feature dimensions, the covariance matrix Σ is not invertible, and thus it is not possible to use E to get the Mahalanobis distance. This is very serious problem since most of the deep learning networks have large number of feature dimensions [69]. Similar to (27 57), we perform a covariance shrinkage to obtain full-rank matrix as follows:
Σ, = Σ + 71 V 12
(8)
where V, is the average diagonal variance, V2 is the average off-diagonal covariance of Σ and / is an identity matrix.
Tukey's Ladder of Powers Transformation. Tukey's Ladder of Powers transformation aims to reduce the skewness of distributions and make them more Gaussian-liz like. We transform the feature vectors ((x) using Tukey's 's Ladder of Powers transformation [55]. It can be formulated as:
((x) = {
if X + 0 if =
log()(x))
where À is hyperparameter to decide the degree of transformation of the distribution. In our experiments, we use 0.5 following (67].
We obtain the normalized features using Tukey's transformation and then use the transformed features to obtain the covariance matrices. When using multiple matrices, we perform correlation normalization to make them comparable. The final prediction and the squared Mahalanobis distance the different class prototypes using one covariance matrix per class can be obtained as:
y² argmin (Mr(((x) py), (or((()), p.) (0(x) ) ($n)² (6(x) Bny y
(10)
where (xx) and j.y refers to the tukey transformed features and prototypes respectively and (Σy) denotes the inverse of the covariance matrices which first undergoes shrinkage followed by normaliza- tion. Note that the covariance matrices are computed using the tukey transformed features. Similarly,
the common covariance matrix Il: can be used in Eq. (10).
5

---

3.3 On the Suboptimality of Learning Linear Classifier
Previous methods like 67 27 in few-shot learning as- sumed Gaussian distributions of classes in feature space and proposed to transfer statistics of old classes to obtain calibrated distributions of new classes and then sample examples from the calibrated distribution to train a linear logistic regression classifier. In our setting, we consider a similar baselinn for comparison which assumes Gaus- sian distributions for features of old classes (by storing the mean and the covariance matrix from the features of the old classes) and samples features from these distributions to learn a linear classifier. We state that this is not an ideal solution since the optimal decision boundaries need not be linear. The optimal decision boundaries are linear when the covariances of all classes are equal like in equal, the optimal decision boundaries
Classifier
Bayesian Classifier
70,9
Avg A (%)
70
68
67.07
67.42
67.8
67.9
66
500
1000
5000
20000
Sampled features per class
Figure 5: Avg Ace comparison of bayesian and linear classifier on CI- FARI00 (T=5) setting.
the Euclidean space. When the covariances of classes are non-linear and forms a quadratic surface in high-dimensional feature space. We show in Fig. that using the optimal Bayesian classifier obtains much better performance compared to sampling features and training linear classifier, even when sampling many examples per class.
4 Experiments
4.1 Experimental Setup
We evaluated FeCAM with strong baselines on multiple datasets and different scenarios.
MSCIL datasets and setup. We conduct experiments on three publicly available datasets: 1) CIFARION 100 [26]- consisting of 100 classes, x2 pixel images with 500 and 100 images per class for training and testing, respectively; 2) TinylmageNet (28 - a subset ImageNet with 200 classes, 64x64 pixel images, and 500 and 50 images per class for training and testing, respectively; 3) ImageNet- Subbsett [12 a subset of the ImageNet LSVRC dataset [47] consisting of 100 classes with 1300 and 50 images per class for training and testing, respectively. We divide these datasets into incremental settings, where the number of initial classes in the first task is larger and the remaining classes are evenly distributed among the incremental tasks. We experiment with three different incremental settings for CIFAR 100 and ImageNet-Subsee
1) 50 initial classes and 5 incremental learning (IL) tasks of 10 classes; 2) 50 initial classes and 10 IL tasks of 5 classes; 3) 40 initial classes and 20 IL tasks of 3 classes. For Tiny ImageNet, we use 100 initial classes and distribute the remaining classes into three incremental settings: 1) IL tasks of 20 classes; 2) 10 IL tasks of 10 classes; 3) 20 IL tasks of 5 classes. At test time, task IDs are not available.
ESCIL datasets and setup. We conduct experiments on three publicly available datasets: I) CIFARIO0 (described above); 2) minilmageNet (58)- consisting of 100 classes, 84 84 pixel images with 500 and 100 images per class for training and testing, respectively; 3) Caltech-UCSD Birds-200- 2011 (CUB200) 59 consisting of 200 classes, 224x224 pixel images with 5994 and 5794 images for training and testing, respectively. For CIFAR 100 and minilmageNet we divide the 100 classes into 60 base classes and 40 new classes. The new classes are formulated into 8-step 5-way 5-shot incremental tasks. For CUB200, we divide the 200 classes into 100 base classes and 100 new classes. The new classes are formulated into 10-step 10-way 5-shot incremental tasks.
Compared methods. We compare with several exemplar-free CIL. methods in the many-shot 32
setting (25 44 21
68 79 78 80 43 and in the few-shot setting [58] 53 70 76 8 72 42)
Results of compared methods marked with are reproduced. For the upper bound of CIL, a joint training on all data is presented as a reference.
Implementation details. We use PyCIL 73 framework for our experiments. For both MSCIL and FSCIL settings, the main network architecture is ResNet-18 [8] trained on the first task using SGD with an initial learning rate of 0.1 and weight decay of 0.0001 for 200 epochs. For the shrinkage, we use 71
and ?22 = for many-shot CIL and higher values 71 = 100 and 72 = 100 for few-shot CIL. in our experiments. Following most methods, we store all the class prototypes. Similar to 788. we also store the covariance matrices for all classes seen until the current task. In the experiments with visual transformers, we use ViT-B/16 [15) architecture pretrained on ImageNet-

---

Table 1: Average top-1 incremental accuracy in exemplar-free maoy CIL with different numbers of incremental tasks. Best results in bold, second best - underlined.
CIL Method
CIFAR
TinyInageNet
ImageNet Subset
7=5
T=10
7=20
T=5
T=10
7=20
7=5
7=10
T=20
EWC
21.2
18.8
12.4
20.4
LwF-MC
45
45.9
27.4
20.1
29.1
23.1
17.4
31.2
DecSIL 8
60.0
50.6
38.1
49.8
43.9
34.1
67.9
60.1
50.5
30.2
26.6
35.1
SDC 68
56.8
$7.0
58.9
61.2
PASS 79
63.5
61.8
58.1
49.6
47.3
42.1
64.4
61.8
51.3
IL2A 78
66.0
60.3
57.9
47.3
44.7
40.0
SSRE
FeTriL.
43
67.6
66.6
63.5
55,4
54.3
53.0
71.9
69.1
Eucl-NCM
72.2
68.4
FeCAM (ours) 1
55.7
73.5
FeCAM (ours) Eg
70.9
70.8
69.4
59.6
59.4
59.3
78.3
78.2
75.1
Upper Bound
79.2
79.2
79.2
66.1
66.1
66.1
84.7
84.7
84.7
The extracted features are 512 dimensional when using Resnet- 18 and 768 dimensional when using pretrained ViT. More implementation details for all hyperparameters are provided in the supplementary material.
21k [52].
4.2
Experimental Results
4.2.1 Many-shot CIL Results
The results for an exemplar-free MSCIL setup are presented in Table We present the results for set of different MSCIL methods, joint training of classifier, and simple NCM classifier (Eucl-NCM) on a frozen backbone. FeCAM outperformed all others by large margin in all settings. FeCAM version with Ih storing a single covariance matrix representing all classes, already gave significantly better results than the current state-of of-the-art method FeTrrIL. However, FeCAM with covariance matrix approximation E.g pushes the average incremental accuracy even higher and present excellent results. It is worth noticing that Eucl-NCM outperforms many existing CII. methods. Only Fe'TrIL performs better than Eucl-NCM on all datasets. In Fig. 6 we present accuracy curves after each task for ten task scenarios. SSRE has lower starting point due to different network architecture. The rest of the methods, despite having the same starting point, end up with very different accuracies at the last task. Eucl-NCM still presents more competitive results than SSRE. FeTrIL presents better performance but is still far from FeCAM with a common covariance matrix. The FeCAM with covariance matrix per class outperforms all other methods starting from the first incremental task. Here, in comparison to common covariance matrix, we pay the price in memory and need to store covariance matrix per class. Despite storing matrix per class, have less memory overhead compared exemplar- based methods and do not violate privacy concerns by storing images.
Additionally, we compare our method against popular exemplar-bane based CIL methods in Table 3 where the memory buffer is set to 2K exemplars. Our method outperforms others that not expand the model significantly (see #P column for the number of parameters after the last task). Only Dynamically Expandable Representation (DER), which grows the model almost six times, can outperform our method.
4.2.2 Experiments with pre-trained models
In Table different settings for exemplar-fre free MSCIL are presented where follow experimental settings of Learning-to-Pront (L2P) method 63). Here, all methods use a ViT encoder pre- trained
on ImageNet-2IKK L2P 663 is strong baseline that does not train the encoder and learns an
additional 46K prompt parameters. However, as Janson et al. [22] presented, a simple NCM classifier perform better for some datasets in CIL, e.g., Split-ImageNet-R [19] Split-CIFAR100 (66 and for inntaiinnncremmntall learning on CoRe50 [3].
We use the widely-used benchmark in continual learning, Split-CIFAR-100 which splits the original CIFAR-IOM 100 (26) into 10 tasks with classes each task unlike the other settings Table 1, which have different task splits. Based ImageNet-R [19]. Split-ImageNet-R was recently proposed
7

---

PASs
OURS comuaso con)
OURSS pes clo
80
75
Accuraoys9
Accumeytta
Accumcyr55)
70
5.46
65
4.27
60
8
55
15
incremental Session
3 4
5
10
incremental
5
10
5
9
10
(s) CIEAR 100
(b) TiyyInggNNe
(c) EnsigeNet-Subact
Figure 6: Accuracy of each incremental task for: (a) CIFARIOO, (b) TinylmageNet, and (c) ImageNet- Subset and multiple MSCIL methods. We annotate the Avg Acc. of all sessions between FeCAM and the runner-up method at the end of each curve. Here, prototype refers to NCM with euclidean distance.
Table 2: Avg Ace or Test Ace at the end of the last task class-incremennal Split- C0 (26]. Split-ImageNet-R [19] and domain-incrementas CoRe50 (4 benchmarks.
Table 3: Comparison of our method with recent exemplar-based methods which store 2000 exem- plars. For fair comparison, we show the number parameters (in millions) by #P. Results on Imagenet-Subset excerpted from [74]
All methods are initialized with pretrained weights from ViT-B/16 [15) for fair comparison.
CIL Method
CIFAR-100 (T
SuapeNettSubset (T
Spli--Cifr100
Spiit-lmgeNat-R Aug Ase
AP
Ex.
Arg. Ace Last Auc
Avg. Ace
Last Ace
Fl-inosen
CIL Method
Arg Ace
17.7
39,5
Test Aec
ICRRL
PODNet
LA
11:17
11.17
V
V
65.4
67.8
57.6
56.3
73.8
62.6
62.9
53.7
FT
28.9
1,2
LwF 100
BWC (25)
[3
83.8
60.7
47.0
35.0
61.6
38,5
78.3
75.5
74,8
WA
BiC
TOSTER
00
11:37
11:17
11:17
67.9
66.1
69.9
60.2
61.5
55.3
66.4
63.1
56.6
40.9
FrCAM (ours)
85.7
63.7
MEMO
75
53.14
36.7
70.2
Irint
79.1
FeCAM)our)
X
by [62] for continual learning which contains 200 classes randomly divided into 10 tasks of 20 classes
each. It contains data with different styles like cartoon, graffiti and origami, as well as hard examples from ImageNet with a high intra-class diversity making more challenging for CIL experiments. settings where the domain of the same class of objects changing in new tasks. It consists of 50 different types of objects from 11 domains. The first domains used for learning and the other 3 domains are used for testing. Since it has single Results of
We use CoRe50 (34) for domain-incremente
test task, we report the test accuracy after lcarning on all 8 domains similar to 63 compared methods excerpted from [22]
We use the proposed FeCAM method with the pre-trained VIT using covariance matrix per class on CIL settings. In the across domains and update the matrix in every new domain by averaging domain and from the current one. FeCAM outperformed both L2P and NCM, in all the settings. Notably, FeCAM outperforms 1.2P by 11.55% and NCM by 4.45% on the CoRe50 dataset.
setting, we maintain single covariance matrix per class
matrix from the previous
4.2.3 Few-shot CIL Results
In many-shot settings, is possible to obtain a very informative covariance matrix from the large number of available samples, while can be difficult get good matrix in FSCIL from just samples per class. To stabilize the covariance estimation, we use higher values of 71 and 72. In Fig. 7 we present accuracy curves after each task for FSCIL on minilmageNet, CIFARI00 and CUB200 datasets. While TOPIC [53] does better than the finetuning methods, it does not perform well in comparison to the recent methods which has significantly improved the performance. ALICE 42] recently proposed to obtain compact and well-clustered features in the base task which helps in generalizing to new classes. We follow the experimental settings from ALICE [42] We take strong base model after the first task from ALICE and use the proposed FeCAM classifier (with a covariance matrix per class) instead of NCM classifier the incremental tasks. We outperform ALICE significantly on all the FSCIL benchmarks.
8

---

-6-X60 Dataebre Sepby DOIRR Angdcr- -0UR
Arploe-Bauurlp
70
04.50
Accumcyyy
50
Accumey
60
50
90
39
Session
Incremental CURR20
(N) CIPAREM
Figure 7: FSCIL methods accuracy of each incremental task for (a) minilmageNet, (b) CIFAR100 and (c) CUB200. We annotate the performance gap after the last session and Avg Acc. of all sessions between OURS and the runner-up method at the end of each curve. Refer supplementary material for detailed values.
Table 4: Ablation of the performance indicating the contribution from the different components of our proposed method FeCAM for MSCIL with five tasks on CIFAR-100 and ImageNet-Subset datasets.
Note that here we use variance normalization (different from Eq. when using diagonal matrix.
CIFAR-100 (T~5)
ImageNet (T~5)
Distance
Cov. Matri
Takey Eq.
Shrinkage Eq.
Norm. Eq.
Last Acc
Avg Aue
Last Ave
Arg Ace
Euclidean
51.6
64.8
60.0
72.2
Euclidean
54,4
66.6
66.2
73.6
Mahalanobis
Full
14.6
29.7
33.5
45.1
Mabalanobis
Foll
20.6
36.2
$4.0
65.6
Mahalanohis
Fall
/
X
44.6
59.3
39.9
56.9
X
Mahalanobis
Full
52.1
62.8
36.5
67.3
Mabalanobis
Diagonal
55.2
66.9
64.0
74.1
Mahalanobis
58.1
Mahalanobis
Fall
62.1
70.9
70.9
78.3
4.2.4 Ablation Studies
FeCAM propose to use multiple different components to counteract the CIL effect on the classifier performance. In Table 4 the ablation study for MSCIL is presented, where contribution of each component exposed
meaning of average incremental and last task accuracy. Here, we consider the settings using covariance matrix per class. We show that Tukeys transformation significantly reduces the skewness of the distributions and improves the accuracy using both Euclidean and Mahalanobis distances. The effect of the covariance shrinkage is
significant for CIFAR 100 which has 500 images per class (less than 512 dimensions feature space) while also improves the performance Iuasee.t-Subset. Here, also show the usage of the diagonal matrix where only the diagonal (variance) values from the covariance matrices. We divide the diagonal matrices by the norm of the diagonal to normalize the variances. When using the diagonal matrix, the storage space is reduced from 2 to D. Finally, we show that using the correlation normalization from Eq. (7) gives the best accuracy by tackling the variance shift and better using the feature covariances.
Time complexity. While previous methods train the classifier 43 or the model 77 for several epochs in the new tasks, we do not perform any such training in new tasks. Among the existing methods, FeTrIL [43] claims to be the fastest. We compare the time taken for the incremental tasks on ImageNet-Subset (T=5) for FeTrIL and the proposed FeCAM method. Using Nvidia RTX 6000 GPU, FeTTIL takes minutes to complete all the new tasks while FeCAM takes only 6 minutes.
Feature transformations. We analyze the t-SNE plot for the feature distributions of old and new classes from CIFAR100 50-50 tasks) setting in different scenarios in Fig. X When the model is trained jointly on all classes, the features of all classes are well clustered and separated from each other. In CIL settings with frozen backbone when the model trained on only the first 50 classes, features are well-clustered for the old classes while the features for new classes are scattered and not sepaaaep....ed. When we make the feature distributions more gaussian using Tukeys transformation, observe that the new class features are comparatively better clustered.

---

(a) Loint-Training
and PID2002-143257NBB-100 (b) CL-Frozzen backbone
(c) CL-Frozen backbone with Tukeys transformation
Figure 8: The t-SNE plot for the features of new and old classes after Joint- Joint-Training (a) and after learning only the first 50 classes (b,c). In thatnTrainnn,, the features are well clustered for all classes, however when the feature extractor is trained only on the first 50 classes, the new class representations are spread out. On applying Tukeys transformation, the new class embeddings are better clustered.
Table 5: Analysis of performance when the first task has 20 classes only and 20 new classes are added in incremental tasks, on CIFAR-100 and ImageNet-Subset datasets.
CIFAR-100
ImageNet-Suue Subbeet
Method
Avg Ace
Last Ace
Avg Acc
Fuclidean-NCM
30.6
50.0
35.0
54.5
FeTriL
46.2
61.3
48.4
63.1
FeCAM (ours)
48.1
62.3
52.3
66.4
CIL settings with a small first task. Usually one method sticks to one setting. Exemplar-free methods use 50% of data in the first task as equally splitting is much more challenging setting which is usually tackled by storing exemplars or by expanding the network in new tasks.
When half of the total classes is present in the first task, the feature extractor is better. When start with fewer classes (20 classes in the first step) and add 20 new classes at every task, we can observe the same behavior in Table FeCAM still works and outperforms other methods. However, the average incremental accuracy not very high this challenging setting because the representation learned in the first task not as good in the big first task setting.
5 Conclusions
In this paper, revisit the anisotropic Mahalanobis distance for exemplar-free CIL and propose using it to effectively model covariance relations for classification based on prototypes. We analyze the heterogeneity in the feature distributions of the classes that are not Icarned by the feature extractor. To address the feature distribution shift, we propose our Bayes classifier method FeCAM, which uses Mahalanobis distance formulation and additionally normalization, covariance shrinkage, and Tukey's 's transformation to estimate better covariance matri- for continual classifier learning. We validate FeCAM both many- and few-shot incremental settings and outperform the state-of-the-an methods by significant margins without the need for any training steps. Additionally, FeCAM evaluated in the class- and pretrained vision transformers yields state-of-the-arr results. FeCAM does not store any exemplars and performs better than most exemplar-based methods many-shot CII. settings. As future work, FeCAM can be adapted to CII. settings where the feature representations are continually learned.
some techniques like correlation
benchmarks with
The proposed approach needs strong feature extractor large amount of data in the first task to learn good representations, as we do not learn new features but reuse the ones learned on the first task (or from pretrained network). Therefore, the method is not apt when training from scratch, starting with small tasks. We would then need to extend the theory to feature distributions which undergo feature drift during training: next prototype drift [68 also covariance changes should be modeled.
Limitations.
Acknowledgement, We acknowledge projects TED2021-13251338 financed by MCIN/AEE/11. (tt///..1339990 13039/501 10001 1033 and FSE+ and the Generalitat de Catalunya CERCA Program. Bartlomiej Twardowski acknowledges the grant RYC2021-032766.
10

---

References
[1] Afra Feyza Akyünek, Ekin Akyürek. Deny Wijaya, and Jacob Andreas. Subspace regularizers for few-shot class incremental learning. In International Conference on Learning Representations (ICLR), 2022.
Nader Asadi, MohammadReza Davari, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Prototype- sample relation distillation: towards replay-free continual learning. In International Conference on Machine Learning (ICML), 2023.
[3] Eden Belouadah and Adrian Popescu. Deesil: Deep-shallow incremental learning. In European Conference on Computer Vision (ECCV) Workshops, 2018.
[4] Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations (ICIR), 2019.
[5] Prashant Shivaram Bhat, Bahram Zonooz, and Elahe Arani. Consistency is the key to further mitigating catastrophic forgetting in continual learning. In Conference on Lifelong Learning Agents (CoLLAs), 2022.
[6] Prashant Shivaram Bhat, Bahram Zonooz, and Elahe Arani. Task-aware information routing from common representation space in lifelong lcarning. In International Conference on Learning Representations (ICLR), 2023.
[7] Francisco M Castro, Manuel J Marin-Jiméned Nicolás Guil, Cordelia Schmid. and Karteek Alahari. End-to-end incremental learning. In European Conference on Computer Vision (ECCV), 2018.
[8] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: meta-learning ap- proach for few-shot class incremental learning. In Conference on Computer Vision and Pottern Recognition (CVPR), 2022.
[9] Matthias De Lange, Rahaf Aljundi, Mare Masana. Sarah Parisot, Xu Jia. Aleš Leonardis, Gregory Slabaugh. and Tinne Tuytelaars. continual learning survey: Defying forgetting in classification tasks. Transactions on Puttern Analysis and Machine Intelligence (T-PAMI), 2021.
[10] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. In International Conference on Computer Vision (ICCV), 2021.
[11] Roy De Maesschalck, Delphine Riiun-,, and Désiré L Massart. The mahalanobis distance. Chemometrics and intelligent laboratory systems, 2000.
[12] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei- Imagenet: A large-scale hierarchical image datahase. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[13] Akshay Raj Dhamija, Touqeer Ahmad, Jonathan Schwan, Mohsen Jafarzadeh, Chunchun Li, and Terrance Boult. Sulf-supe features improve spen-world learning. arXiv preprint arXiv:2102.0788 2021.
[14] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Conference on Compuder Vision and Pattern Recognition (CVPR), 2019.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[16] Arthur Douillard, Matthiuu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Computer Vision (ECCV), 2020.
[17] Samantha Guerriero, Barbara Caputo, and Thomas Mensink. Deepnem: Deep nearest class mean classifiers. International Conference on Learning Representations Workshop (ICLR-W (W), 2018
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[19] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In International Conference Computer Vision (ICCV), 2021.
[20] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint 503.511,, 2015.

---

[21] Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang, and Dahua Lin. Learning unified classifier incrementally via rebalancing. In Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[22] Paul Janson, Wenxuan Zhang, Rahaf Aljundi, and Mohamed Elhoseiny. A simple baseline that questions the use of pretrained-models in continual learning. In NeurlPS 2022 Workshop Distribution Shifts: Connecting Methods and Applications, 2022.
[23] Kishaan Jeeveswaran, Prashant Bhat, Bahram Zonooz, and Elahe Arani. Birt: Bio-inspired replay in vision transformers for continual learning. In International Conference Machine Learning (ICML), 2023.
[24] Gyuhak Kim, Bing Liu, and Zixuan Ke. A multi-head model for continual learning via out-of-distribution replay. In Conference on Lifelong Learning Agents (CoLLAs)) 2022.
[25] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Baraansaaa et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2017.
[26] Alex Krizhevsky, Geoffre Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Shakti Kumar and Hussain Zaidi. Gde-generalized distribution calibration for fow-shot learning. arXiv preprint ari:ir2244.00000 05230,, 2022.
[28] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
[29] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems (NeurlPS), 2018.
[30] Zhizhong Li and Derek Hoiem. Learning without forgetting, Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2017.
[31] Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanbao Yu, Jun Chen, and Jin Tang. Few-shot class- incremental learning via entropy-regularin data-free replay. In Luropean Conference on Computer Vision (ECCV), 2022.
[32] Yu Liu, Sarah Parisot, Gregory Slabaugh, Xu Jia, Ales Leonardis, and Tinne Tuytelaars. More classifiers, less forgetting: A generic multi-llassifier paradigm for incremental learning. In European Conference on Computer Vision (ECCV), 2020.
[33] Yuyang Liu, Yang Cong. Dipam Goswami, Xialei Liu, and Joost van de Weijer. Augmented box replay: Overcoming foreground shift for incremental object detection. In International Conference Computer Vision (ICCV), 2023.
[34] Vincenzo Lomonaco and Davide Maltoni. Core50: recognition. In Conference on Robot Learning (CoRI.), 2017.
new dataset and benchmark for continuous object
[35] Chunwei Ma, Zhanghexuan Ji, Ziyun Huang, Yan Shen, Mingehen Gao, and Jinhui Xu. Progressive voronoi diagram subdivision enables accurate data-free dass-incremennal learning. In International Conference on Learning Representations (ICLR), 2023.
[36] Mare Masana. Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: survey and performance evaluation. Transactions Puttern Analysis and Machine Intelligence (T-PAMI), 2022.
[37] Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation. Elsevier, 1989.
[38] Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabricla Csurka Distance-basse image classifi- cation: Generalizing to new classes at near-zero cost. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2013.
[39] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Muchine Learning (ICML), 2017.
[40] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations (ICLR), 2018.
[41] Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E. Tumer. First session adaptation: A strong replay-free free baseline for Computer Vision (ICCV). 2023.
learning. In International Conference on
12

---

[42] Can Peng, Kun Zhao, Tianren Wang, Meng Li, and Brian C Lovell. Few-shot class-incremental learning from open-set perspective. In European Conference on Computer Vision (ECCV), 2022.
[43] Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar free class daas-incremennal learning. In Winter Conference on Applications of Computer Vision (WACV), 2023.
[44] Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph HI Lampert. icarl: Incremental classifier and representation learning. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[45] Tal Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Zelnik-Manor Imagenet-211 pretraining for the masses. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[46]
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 1995.
[47] Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015.
[48] Andrei A Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations (ICLR), 2019.
[49] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Can features off-the-shelf: an astounding baseline for recognition. In Conference on Computer Vision and Pattern Recognition (CVPR) workshops, 2014.
[50] Christian Simon, Masoud Faraki, Yi-Hsuan Tsai, Xiang Yu, Samuel Schulter, Yumin Suh, Mehrtash Harandi, and Manmohan Chandraker. On generalizing beyond domains in cross-domain continual learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[51] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-incremental learning. In International Conference on Computer Vision (ICCV), 2021.
[52] Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakoh Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions Machine Learning Research, 2022.
[53] Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class- inssemenreltttll learning. Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[54] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datascts for learning to learn from few examples. In International Conference on Learning Representations (ICLR), 2020.
[55] John W. Tukey. Exploratory data analysis. Addison-Wesley series in behavioral science: quantitative methods. Addison- Wesley, 1977.
[56] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv: T.:::900 1904007744 2019.
[57] John Van Ness. On the dominance of non-parametric Bayes rule discriminant algorithms in high dimensions. Pattern Recognition, 1980.
[58] Oriol Vinyals. Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems (NewrIPS), 2016.
[59] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[60] Yu-Yun Wang, Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: Feature bousting and compression for class-incremental learning. In European Conference on Computer Vision (ECCV). 2022.
[61] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00453 2023.
13

---

[62] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi. Ruoxi Sun. Han Zhang, Chen-Yo Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision (ECCV), 2022.
[63] Zifeng Wang. Zizhao Zhang. Chen- Yu Lee, Han Zhang. Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. Learning to prompt for continual learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[64] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[65] Shiming Xiang, Feiping Nie, and Changshui Zhang. Learning a mahalanobis distance metric for data clustering and classification. Puttern recognition, 2008
[66] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Conference on Computer Vision and Puttern Recognition (CVPR). 2021.
[67] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. International Conference on Learning Representations (ICLR), 2021.
[68] Yu, Bartlomiej Twardowski. Xialei Liu, Luis Herranz. Kai Wang. Yongmei Cheng, Shangling Jui. and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[69] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference (BMVC), 2016.
[70] Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
[71] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Sau-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Conference Computer Vision and Pattern Recognition (CVPR). 2020.
[72] Da- Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible sho- dass-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[73] Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, and De-Chuan Zhan. Pycil: python toolbox for class- incremental learning, SCIENCE CHINA Information Sciences, 2023.
[74] Da-Wei Zhou, Qi- Wang. Zhi-Hong Qi. Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class- incremental learning: survey, arXiv preprint 33iv:23022.06645 2023.
[75] Da-Wei Zhou, Qi- Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient class-incremental learning. In International Conference on Learning Representations (ICLR), 2022.
[76] Da-Wei Zhou, Han-Jia Ye, Liang Ma, Di Xie, Shiliang Pu, and De-Chuan Zhan. Few-shot class- incremental learning by sampling multi-phase tasks, IEEE Transactions on Puttern Anayssis and Machine Intelligence (TPAMI), 2022.
[77] Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Co-transport for class-incrementar learning. In ACM International Conference on Multimedia, 2021.
[78] Fei Zhu, Zhen Cheng, Xu-Yao Zhang, and Cheng-lin Liu, Class-incremental learning via dual augmentation. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[79] Fei Zhu, Xu-Yau Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self- supervision for incremental learning. In Conference Computer Vision and Pattern Recognition (CVPR). 2021.
[80] Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self sustainnt representation expansion for non-exemplar inssemnncementll learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
14


Reviews:
[{"review_id": "MANP15IZUg", "reviewer": "Reviewer_MMgd", "summary": "The paper presents a method for class-incremental learning when a strong base classifier is available and incrementally classes are added to the classifier. The base classifier is kept frozen and another classifier is used on top of the features from this classifier. Instead of using deep networks, authors propose using a bayes classifier and modeling the distance between classes using mahalanobis distance instead of the popular euclidean distance. Authors argue and show that the features of the incoming new classes are heterogenous and are better suited to be measured with mahalanobis distance. The authors also propose some solutions to the problems with the covariance matrix in mahalanobis distance calculation and to improve the performance further. The method is evaluated on many and few-shot settings and evaluated on many standard datasets where the method seems to beat many existing methods.", "questions": "1. How is the order of the classes which are added decided?\n2. Which classes are selected as part of MiniImagenet and ImagenetSubset, the classes and few-shot images should also be provided for full reproducibility\n", "limitations": "Authors have sufficiently addressed the limitations of the method. The limitations make the method only applicable in certain cases but the authors acknowledge it and the performance of the method seems pretty good in that setting.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well-written and easy to follow, supplementary material also provides good background and completes the picture,\n2. Authors do a good job in motivating and validating their motivation and methodology, with plots, figures and qualitative numbers. The use of mahalanobis distance and corresponding proposals to the covariance matrix seem like simple but powerful changes.\n3. The method works without additional training of classifiers so it would be quicker to incorporate new classes in an incremental setting\n4. It does not use significant additional storage space and the quantitatively the methods does well compared to existing methods\n", "weaknesses": "1. There are no error bars added for the results which would provide more confidence in the results, it is a convention in incremental learning research to only have a single plot per method without any error bars, earlier since it used to be computationally expensive but no reason we should start doing it now,\n\n2. The limitations already address this, but a follow-up on changing features would exacerbate the drift problem and would love to see the theory handle that case as well, the authors already list as future work so no action is needed for this submission,\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for all the feedback and suggestions.\n\n__W1:__ FeCAM is independent of random initialization. Since we do not train any deep neural network after the first step, the method is deterministic and thus we do not see any variation in the results on multiple runs.\n\n__W2:__ As a consequence of changing features and the backbone feature drift, the distribution of previously seen classes change and would need to be adapted. Semantic Drift Compensation (SDC) [A] proposes a method to estimate the drift of a single point in the latent space, based on the observed drift of the current data. In [A] this method is used to update the mean of the distribution (it is then combined with the Euclidean distance). We would also need to update the covariance matrix of the distribution. A potential approach could sample a set of points from the previous distribution, apply SDC to each of them, and then compute the covariance matrix on these points. We can then use FeCAM with the updated covariance matrix and the updated prototypes.\n\n__Q1:__ Following iCaRL [B], PyCIL [C] and several other CIL papers, we follow the same order of classes for fair comparison (iCaRL uses seed 1993). We included the sequence of class indices in supplementary material.\n\n__Q2:__ For ImageNet-Subset in many-shot settings, we use the same classes as in PyCIL [C], which is openly available on Kaggle platform under the name: `arjunashok33/imagenet-subset-for-inc-learn`. For miniImageNet in few-shot setting, we use the same set of images in every task as done previously in [D, E, F].\n\nWe will release the source code of our method in GitHub for reproduction (code for one setting is already provided in the supplementary materials). In the source code, the dataset details are more clear and can be easily used for reproducing our results.\n\n[A] Lu Yu, et al. Semantic drift compensation for class-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.\n\n[B] Sylvestre-Alvise Rebuffi, et al. icarl: Incremental classifier and representation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[C] Da-Wei Zhou, et al. Pycil: a python toolbox for class-incremental learning. SCIENCE CHINA Information Sciences, 2023.\n\n[D] Chi Zhang, et al. Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.\n\n[E] Can Peng, et al. Few-shot class-incremental learning from an open-set perspective. In European Conference on Computer Vision (ECCV), 2022.\n\n[F] Da-Wei Zhou, et al. Forward compatible few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022."}, {"review_id": "CixQrdxVRL", "reviewer": "Reviewer_nDen", "summary": "The authors study classifiers in the incremental learning scenario with a fixed strong pre-trained feature extractor. They point out the limitations of the Euclidean distance-based nearest class mean classifier in continual learning and demonstrate the benefits of Mahalanobis distance-based classifier. Several methods (shrinkage, Tukey’s transformation, covariance normalization) are proposed to estimate the covariance matrices and their effectiveness is examined through ablation studies. The authors evaluate their method in various settings, including many-shot CIL, few-shot CIL, and domain incremental learning. Although the proposed method does not require saving previous samples, it outperforms the existing CL methods including those based on replay, in terms of performance.", "questions": "1. How does the performance change when using models pre-trained using classes dissimilar to the classes used in CL?\n2. Can the authors provide any suggestions on how to apply their method for a trainable feature extractor?", "limitations": "Refer to Weaknesses and Questions.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is simply, yet outperforms the existing methods\n2. The proposed method does not require saving previous samples\n3. The paper is well-written and easy to follow", "weaknesses": "1. My main concern regarding this method is that it relies on a fixed feature extractor after the initial training, with only the classifier being adapted. As [1] shows, fixed models underperform when the pre-training data and the continual learning data are dissimilar because they cannot acquire new knowledge. It’d make the paper stronger if the authors show how the model performs with a feature extractor pre-trained with dissimilar classes from CL. For instance, [2] uses a model pre-trained with ImageNet after removing the classes similar to CIFAR and Tiny-ImageNet.\n2. Saving the covariance matrix for each class in the feature-level seems to be expensive for memory, especially when there are a large number of classes.\n\n[1] Ostapenko et al., Continual learning with foundation models: An empirical study of latent replay. CoLLAs, 2022\n\n[2] Kim et al. A multi-head model for continual learning via out-of-distribution replay. CoLLAs, 2022\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the inputs and suggestions.\n\n__W1, Q1:__ Similar to [2], we perform experiments using the DeiT-S/16 vision transformer pretrained on the ImageNet data with different pre-training data splits and then evaluate the performance of NCM (with euclidean distance) and the proposed FeCAM method on Split-CIFAR100 (10 tasks with 10 classes in each task). In order to make sure that the pretrained classes are not similar to the classes of CIFAR100, [2] manually removed 389 classes from the 1000 classes in ImageNet. We take the publicly available DeiT-S/16 weights pre-trained on remaining 611 classes of ImageNet by [2] and evaluate NCM and FeCAM. As expected, the performance of both methods drops a bit when the pre-training is not done on the similar classes. Still FeCAM outperforms NCM by about 10% on the final accuracy. Thus, this experiment further validates the effectiveness of modeling the covariance relations using our FeCAM method in settings where images from the initial task are dissimilar to new task images.\n\n| Split-CIFAR100 | DeiT-S/16 pre-trained on 1k classes | DeiT-S/16 pre-trained on 611 classes [2] |\n|  :-------: | :-------: | :-----: |\n|    |  Avg Inc Acc (Final Acc) | Avg Inc Acc (Final Acc) |\n| NCM (euclidean) | 71.4 (60.5) | 69.2 (58.5) |\n| FeCAM (ours) | __78.5 (70.2)__ | __76.9 (68.6)__ |\n\n__W2:__ We want to point out that we describe this in the supplementary materials and we discuss it in comparison to exemplars-based methods where we show that FeCAM requires much less storage space as compared to exemplar-based methods.\n\nFurthermore, due to the symmetric nature of covariance matrices, we can store half (lower or upper triangular) of the covariance matrices and reduce the storage to half. The analysis of storage requirements after every task for FeCAM and the exemplar-based methods (storing 2000 exempars) for the ImageNet-Subset are as follows:\n\n| Method | Task 0 | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 |\n| :----: | :----: | :----: | :----: | :----: | :----: | :----: |\n| Exemplar-based | 312 MB | 312 MB | 312 MB | 312 MB | 312 MB | 312 MB |\n| FeCAM (ours) | 53 MB | 63 MB | 74 MB | 84 MB | 95 MB | 105 MB |\n\nAs future work, using covariance matrix factorization to reduce the storage requirements can be explored. \n\nSVD(covariance matrix) =  U\\*S\\*V \n\nHere, the singular values S are in descending order. So, one can consider taking the first k singular values from S, and similarly truncating the U and V matrices (by taking the first k columns from U and the first k rows from V). This direction can be explored since it reduces the matrix dimensions from 512\\*512 to (512\\*k + k + k\\*512) where k should at least be less than 256.\n\n__Q2__: As a consequence of backbone feature drift, the distribution of previously seen classes change and would need to be adapted. Semantic Drift Compensation (SDC) [A] proposes a method to estimate the drift of a single point in the latent space, based on the observed drift of the current data. In [A] this method is used to update the mean of the distribution (it is then combined with the Euclidean distance). We would also need to update the covariance matrix of the distribution. A potential approach could sample a set of points from the previous distribution, apply SDC to each of them, and then compute the covariance matrix on these points. We can then use FeCAM with the updated covariance matrix and the updated prototypes.\n\n[A] Lu Yu, et al. Semantic drift compensation for class-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.\n"}, {"review_id": "JWaM4MeIUB", "reviewer": "Reviewer_YVv6", "summary": "This manuscript introduces a Bayes classifier method FeCAM to address the feature distribution shift within the realm of Continual Incremental Learning (CIL). This method uses Mahalanobis distance formulation and additionally uses some techniques like correlation normalization, covariance shrinkage, and Tukey’s transformation to estimate better covariance matrices for continual classifier learning, thereby enhancing the efficacy of the model in addressing CIL tasks.", "questions": "1.\tPage 1, ‘Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention.’ I think it’s necessary to explain the weakness of the method after this sentence.\n2.\tIt can be observed that the proposed method FeCAM outperforms the existing state-of-the-art (SOTA) results on all datasets under the few-shot settings. But there should further analysis to explain the advantage of the proposed FeCAM.\n", "limitations": "The proposed method relies on a pretrained network to learn good representations because the method does not learn new features but reuse the ones learned on the first task. Therefore, when training from scratch, starting with small tasks, this method may not work well.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The manuscript presents clear and logical theoretical reasoning, making it easily understandable for readers. Moreover, the method is firmly grounded in a well-defined theoretical framework.", "weaknesses": "1.\tThere exist some mistakes, for example, on line 224, it states “20 initial classes and 10 IL steps of 5 classes”. It should be corrected to “20 initial classes and 10 IL steps of 8 classes”.\n2.\tFrom table 1, it can be observed that the results of the FeTrIL are recomputed, whereas the other results are reproduced using the original configurations of the methods. This deviation contradicts the manuscript 's description. In addition, please explain how the experimental setup in this manuscript differs from that of FeTrIL.\n3.\tOn Page 7, the datasets “Split-ImageNet-R”, “Split-CIFAR100”, “CoRe50” and the “domain-incremental learning” need to be further explained. In particular, it is important to explain the differences between Split-CIFAR100 and the previous CIFAR100 dataset.\n4.\tFrom ablation Studies, it can be observed that the experimental results in row 7 of Table 4 outperform those in row 6. Therefore, when the diagonal matrix is combined with \"Tukey\", \"Shrinkage\" and \"Norm\", whether the performance will be better.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the inputs and the suggestions.\n\n__W1:__ We clarify that in the main text to “50 initial classes and 10 IL steps of 5 classes”.\n\n__W2:__ We train FeTrIL with more epochs, which results in improved accuracy in Table 1 compared to the original paper. We recompute FeTrIL as this is the state-of-the-art method with code available. The results of the other methods are taken from the FeTrIL paper which were reproduced from their original papers. We now corrected the text in the paper and clarified this in more detail.\n\n__W3:__ We added the following details of these datasets and splits in the supplementary materials.\n\nWe use the widely-used benchmark in continual learning, Split-CIFAR-100 which splits the original CIFAR-100 [A] into 10 tasks with 10 classes in each task unlike the other settings in Table 1 which have different task splits. Based on ImageNet-R [B], Split-ImageNet-R was recently proposed by [C] for continual learning which contains 200 classes randomly divided into 10 tasks of 20 classes each. It contains data with different styles like cartoon, graffiti and origami, as well as hard examples from ImageNet with a high intra-class diversity making it more challenging for CIL experiments. We use CoRe50 [D] for domain-incremental settings where the domain of the same class of objects is changing in new tasks. It consists of 50 different types of objects from 11 domains. The first 8 domains are used for learning and the other 3 domains are used for testing. Since it has a single test task, we report the test accuracy after learning on all 8 domains.\n\n__W4:__ As mentioned in lines 307 to 309, the diagonal matrix is normalized (differently from Eq. 7) by dividing with the norm of the diagonal. Eq. 7 does not make sense with the diagonal matrix since it does not have the covariance values and making the diagonal values one, will lose all the information. So, we perform normalization of only the variance values. The results in row 7 of Table 4 uses ‘Tukey’, ‘Covariance Shrinkage’ and ‘Normalization’ but not the normalization from Eq. 7. We now clarified this in Table 4.\n\n__Q1:__ We add the following statement in line 33: One of the drawbacks is the inability to learn new representations with a frozen feature extractor.\n\n__Q2:__ We added more analysis to explain the advantage of FeCAM, particularly in the few-shot settings:\n\nFeCAM can easily be adapted to available few-shot methods in CIL since most methods obtain class prototypes from few-shot data of new classes and then use the euclidean distance for classification. We show in our paper that starting from the base task model from ALICE [E] and simply using the FeCAM metric for classification significantly improves the performance across all tasks for the standard few-shot CIL benchmarks. \n\nFor further analysis to demonstrate the applicability of FeCAM, we take the base task model from FACT [F] and use FeCAM in the incremental tasks on the CUB200 dataset. FeCAM improves the performance on all tasks when applied to FACT as shown in the table below.\n\n| Method | Task 0 | Task 1 | Task 2 | Task 3 | Task 4 | Task 5 | Task 6 | Task 7 | Task 8 | Task 9 | Task 10 | Avg |\n| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| FACT | __77.9__ | 74.9 | 71.6 | 66.3 | 65.9 | 62.5 | 61.2 | 59.8 | 57.9 | 57.6 | 56.4 | 64.7 |\n| FACT+FeCAM | __77.9__ | __75.3__ | __72.2__ | __67.6__ | __67.0__ | __63.5__| __62.4__ | __61.3__ | __59.8__ | __59.1__ | __57.9__ | __65.8__ |\n\nOne of the main drawbacks of the many-shot continual learning methods is overfitting on few-shot data from new classes and hence these methods are not suited for few-shot settings. FeCAM is a single solution for both many-shot and few-shot settings and thus can be applied in both type of continual learning settings.\n\n[A] Alex Krizhevsky, et al. Learning multiple layers of features from tiny images. 2009.\n\n[B] Dan Hendrycks, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.\n\n[C] Wang Z, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, 2022.\n\n[D] Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference on Robot Learning, pages 17–26. PMLR, 2017.\n\n[E] Can Peng, et al. Few-shot class-incremental learning from an open-set perspective. In European Conference on Computer Vision (ECCV), 2022.\n\n[F] Da-Wei Zhou, et al. Forward compatible few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022."}, {"review_id": "EbqpbVyxtn", "reviewer": "Reviewer_9pER", "summary": "The authors claim that feature space of a learnt network with old classes is more scattered in newly learned classes and this prevents Euclidean distance to represent the distribution properly. Therefore, they have adopted Mahalanobis distance instead to handle the heterogenety of the newly coming class distribution. To boost up the performance, covariance normalization, covariance shrinkage, covariance matrix approximation have been adopted.", "questions": "- I agree that a bayesian classifier is well suited with incremental learning. However, when it comes to applying this incremental learning procedure to other kinds of tasks such as segmentation or generative models, will it be still effective to use it? This is not a critical comment and I just want to know the opinion of the authors.", "limitations": "No comments required here.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The performance presented in this paper is state of the art. Even though bayesian classifier is usually only valid for classification problem, it is still impressive to see the advance in performance.\n- The settings and amount of experiments in this paper are rich enough to support the hypothesis claimed in this paper.\n- Especially, thorough ablation over the components of this method is very interesting.", "weaknesses": "- many existing techniques comprising the proposed method make this paper somewhat short of novelty but I still think the whole idea throughout this paper is reasonable.\n- In Eq (6), I dont understand why the norm of Y is used all over the equation. Y should be the target of all samples and the norm of this does not seem to be important. I think more explanation about how this equation is derived is required here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and suggestions.\n\n__W2:__ This is not the norm. We now clarified this in the paper. We use |A| to indicate the cardinality of a set A. Here, we used | | to signify the number of classes seen till that task.\n\n__Q1:__ Yes. Whenever the classification can benefit from having richer representation by using covariance matrix, FeCAM can help with getting better classifier accuracy. However, maybe the covariance matrices should be adapted differently for different tasks. \n\nWe think that FeCAM can be extended to other tasks in incremental settings which apply prototype-based classification (based on Euclidean distance). Current incremental semantic segmentation methods are mostly not prototype-based, but could be adapted to this setting. In principle, we believe that FeCAM can then be improved to obtain performance gains. \n\nWe think it is harder to extend theory to incremental generative models. These models typically are not using any explicit Euclidean distance (which could be replaced by the proposed metric). In the case of pseudo-replay [A] where a generative model is used for image replay of old classes, the resulting feature distribution can form realistic anisotropic distributions. However, it should be noted that these methods require an additional generative model, and are typically reported to struggle when applied to complex image datasets (like ImageNet).\n\n[A] Shin, Hanul, et al. \"Continual learning with deep generative replay.\" Advances in neural information processing systems 30 (2017).\n"}, {"review_id": "Ywwfn3si2z", "reviewer": "Reviewer_QHUR", "summary": "This paper addresses the problem of continual learning, which holds great importance in the machine learning field. The authors argue that using the Mahalanobis distance is more optimal than the Euclidean metric for handling new classes. The proposed method is evaluated on various continual learning settings and compared against state-of-the-art approaches.", "questions": "Remarks\n1. In Figure 5, clarification is needed regarding how the authors managed to sample 20,000 instances per class for CIFAR100, as the maximum number of instances per class is 500 for CIFAR.\n2. What is “classifier incremental learning” in Line 137?\n3. It would be beneficial to provide an implementation guideline with pseudo code.\n\n\n\nIn summary, this paper addresses an interesting phenomenon in the field of continual learning and proposes a simple baseline method to address it. The proposed approach is evaluated on various protocols, including many-shot CIL, few-shot CIL, and pre-trained CIL, demonstrating superior performance compared to state-of-the-art methods. Although there are some evident drawbacks, the novel findings presented in this paper contribute to the continual learning field. Therefore, my initial rating is a borderline accept.\n", "limitations": "NA", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Continual learning is of great importance in the machine learning field, and the availability of code is appreciated.\n2. The proposed method is evaluated on multiple continual learning settings and compared against state-of-the-art approaches.\n3. The figures effectively illustrate the distribution of old and new classes, making the proposed approach reasonable.\n", "weaknesses": "1. The largest concern about this paper is the assumption of pre-trained model (or say, relying on large number of classes in the first task). There are two typical settings of class-incremental learning, training from scratch and training from half. The former equally assigns classes into each incremental task, while the latter treats half of the total classes in the first task. According to a recent work [1], these different settings concentrate on different aspects of the continual learning algorithms. In this paper, the experiments (as well as empirical observations) are conducted on the training from half setting, making it less convincing on the other setting. Would the proposed method work for the other setting? Will the observations of covariance be the same on the former setting? More empirical evaluations and experimental comparisons are essential.\n2. In figure 1, why does the training process lead to the gaussian-like distribution of the seen classes and non-gaussian-like distribution of unseen classes? Perhaps some discussions about theoretical insights behind such phenomena should be addressed.\n3. The current proposed method is a combination of several mature tricks, e.g., Mahalanobis distance, Covariance Shrinkage, and Tukey’s transformation. I understand that the most contributions lie in the motivations in the preliminary part, while the combination still makes the novelty/contribution to share with former works. \n\n[1] Online Hyperparameter Optimization for Class-Incremental Learning. AAAI 2023\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and suggestions.\n\n__W1:__ We agree that usually one method sticks to one setting. Exemplar-free methods use 50% of data in the first task as equally splitting is a much more challenging setting which is usually tackled by storing exemplars or by expanding the network in new tasks.\n\nWhen half of the total classes is present in the first task, the feature extractor is better. When we start with fewer classes (20 classes in the first step) and add 20 new classes at every task, we can observe the same behavior in the table below. FeCAM still works and outperforms other methods. However, the average incremental accuracy is not very high in this challenging setting because the representation learned in the first task is not as good as in the big first task setting.\n\n| Method |        CIFAR100         | ImageNet-Subset    |\n|   :----:    |              :----:             |          :----:                |\n|              |  Avg inc. acc (Final acc)  | Avg inc. acc (Final acc)  | \n| NCM (euclidean) |  50.0 (30.6) | 54.5 (35.0) |\n| FeTrIL | 61.3 (46.2) | 63.1 (48.4) |\n| FeCAM (ours) | __62.3 (48.1)__ | __66.4 (52.3)__ |\n\n__W2:__ As previously stated by Guerriero et al. [A], the highly non-linear nature of deep neural networks leads to isotropic spherical representations of the learned classes and hence using the euclidean distance is effective for NCM classification. However, in the class-incremental learning setting with fixed feature extractor, the unseen or new classes are not explicitly learned using the non-linear deep networks and hence the representations of these classes are anisotropic and not spherical as it used to be the case before the emergence of deep neural networks [B]. We also analyze this phenomenon in Fig. 3 (b,c) of the paper.\n\n__Q1:__ This is now clarified in the paper that we sample features for the linear classifier from the gaussian distributions of the old classes (using the prototypes and the covariance matrices for the old classes). We also updated Fig. 5 to clarify this.\n\n__Q2:__ We defined the classifier incremental learning in Lines 30-35 of introduction. It refers to the settings where the feature extractor is not updated after the first task and only the classifier is learned in new tasks.\n\n__Q3:__ We now added the pseudo code in the supplementary materials. Additionally, we will release the source code of our method in GitHub for reproduction.\n\n[A] Samantha Guerriero, et al. Deepncm: Deep nearest class mean classifiers. International Conference on Learning Representations Workshop (ICLR-W), 2018.\n\n[B] Thomas Mensink, et al. Distance-based image classification: Generalizing to new classes at near-zero cost. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2013."}]

Author rebuttal:
We thank the reviewers for their insightful comments and sincerely appreciate their efforts in providing valuable feedback. The reviewers agree that the paper is very well-written and is easy to follow (MMgd, nDen), has clear and logical theoretical reasoning (YVv6), along with a good motivation with plots and figures (MMgd, QHUR) and also appreciated the availability of code (QHUR). The reviewers appreciated the multiple experimental settings (9pER, QHUR, MMgd) and the thorough ablation experiments (9pER). 

We address all the weaknesses and questions raised by all the reviewers in the respective rebuttals. We believe that most of these inputs are valid and greatly improve our paper. 

We summarize the major changes we made to the paper in the rebuttal process:

1. In order to make sure that the pretrained classes are not similar to the incremental classes, we perform experiments with the base model pre-trained with only the dissimilar classes and observe that FeCAM still performs good and improves over euclidean-NCM significantly.
2. We clarified the experimental settings for many-shot methods in more detail. We also added the details of the Split-CIFAR100, Split-ImageNet-R and CoRe50 datasets.
3. We performed more experiments to show that FeCAM can be easily adapted to multiple few-shot continual learning methods with improved performance.
4. We clarified the notations in equation 6.
5. We perform experiments in settings starting with only 20 classes in the first step and show that FeCAM still outperforms the most competitive exemplar-free method FeTrIL and NCM with euclidean distance.
6. We updated Figure 5 for more clarity. We also added a pseudo code of FeCAM.

We hope that our responses to the reviewer's questions and the additional experiments will help in the review process.

Regards,

Authors

Meta-review: The authors propose an algorithm to improve the performance of exemplar-free class-incremental learning (CIL). In doing so, they studied the commonly used Euclidean metric in exemplar-free CIL and showed its shortcomings in heterogeneous cases. Based on this observation, the authors opt for an adaptive metric (based on the Mahalanobis distance) and achieve improvements over prior arts consistently. This paper was quite well received by reviewers, with scores of 6, 4, 7, 6, and 7. Reviewers felt the paper was well-written,  clear, and expressed the value of the core idea. The AC agrees with the reviewers and recommends the acceptance of the work; congratulations. Please revise your work based on comments and incorporate the new results provided during the rebuttal phase.
Decision: Accept (poster)