Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning
Authors: Grzegorz Rypeść
Sebastian Cygert
Tomasz Trzcinski
Bartłomiej Twardowski
Venue: NeurIPS 2024 poster (2024)
Keywords: continual learning
exemplar free
exemplar free class incremental learning
class incremental learning
exemplar-free
Primary area: online_learning
TL;DR: Exemplar-free class incremental learning, we explain task-recency bias in modern methods and develop a method that adapts means and covariances of classes from task to task

Abstract: Exemplar-Free Class Incremental Learning (EFCIL) tackles the problem of training a model on a sequence of tasks without access to past data. Existing state-of-the-art methods represent classes as Gaussian distributions in the feature extractor's latent space, enabling Bayes classification or training the classifier by replaying pseudo features. However, we identify two critical issues that compromise their efficacy when the feature extractor is updated on incremental tasks. First, they do not consider that classes' covariance matrices change and must be adapted after each task. Second, they are susceptible to a task-recency bias caused by dimensionality collapse occurring during training. In this work, we propose AdaGauss - a novel method that adapts covariance matrices from task to task and mitigates the task-recency bias owing to the additional anti-collapse loss function. AdaGauss yields state-of-the-art results on popular EFCIL benchmarks and datasets when training from scratch or starting from a pre-trained backbone.

Task-recency bias strikes back: Adapting covariances in Exemplar-Free Class Incremental Learning
Grzegorz Rypešč"
Sebastian Cygert
IDEAS NCBR
IDEAS NCBR
Warsaw University Technology
Gdańsk University of Technology
sebastian.e cygert@ideas cesaanrreeertiiesnncnbba -ocbrnncbrp... .pl
Tomasz Trzciński
Bartlomiej Twardowski
IDEAS NCBR
IDEAS NCBR
Warsaw University of Technology Tooploox
Autonomous University of Barcelona Computer Vision Center
Abstract
Exemplar-Free Class Incremental Learning (EFCIL) tackles the problem of training model sequence of tasks without access to past data. Existing oftt-of-theeart methods represent classes as Gaussian distributions in the feature extractor's 's latent space, enabling Bayes classification training the classifier by replaying pseudo features. However, we identify two critical issues that compromise their efficacy when the feature extractor is updated incremental tasks. First, they do not consider that classes' covariance matrices change and must be adapted after each task. Second, they susceptible a task-recency bias caused by dimensionality collapse occurring during training. In this work, we propose AdaGauss a novel method that adapts covariance matrices from task task and mitigates the task- recency bias owing to the additional anti-collapse loss function. AdaGauss yields state-of-the-arr results on popular EFCIL benchmarks and datasets when training from scratch or starting from pre- rreitri backbone.
Introduction
Continual learning (CL), essential of machine learning, focuses developing algorithms that can learn progressively from a continuous stream of data and adapt to new tasks while retaining previously acquired knowledge. This paradigm is paramount for creating systems capable of lifelong learning, much like humans, and robust in dynamic environments where data distribution evolves over time. A significant challenge within CL exemplar-free class incremental learning (EFCIL) 41
26). which requires the model to incorporate new classes without storing previous data samples (exemplars). This approach is especially relevant in scenarios with privacy constraints or limited storage capacity, as it compels the model to retain knowledge and prevent catastrophic forgetting [9] 27 solely through internal mechanisms, such knowledge distillation regularization
24 parameter
17 2 6), expanding neural architecture 44 53 32
or generative replay 40 14
28
Recent state-of-the-an methods designed for EFCIL often represent classes Gaussian distributions the latent space of the feature extractor. That enables an inference using Bayes classifier [10] 35 or training linear classifier using prottoypsss sampled from these distributions [24 31 51 37
However, we present this work that these methods have multiple shortcomings and can be improved. First, they assume that covariance matrices of past classes constant across incremental training.
"Code: http::://ghhhub..ooo//rree/////p
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Real data points of past classes
Memorized distribution of the pest class
DEL= = 0.0, Acc 36.2%
DRLL = 228.3, Acc = 22.6%
DKL - 65.7, Acc 48.0%
DEL = 16.7, Acc 51.1%
Frazen feature extractor
Unfrozen, adaptation
Unfrazen, adaptation
covariance adaptation
Figure Latent space visualization, average accuracy after the last task, and symmetrical KL divergence between memorized and ground truth distributions for ResNet18 trained sequentially on ImagenetSubset dataset split into ten tasks. Freezing the feature extractor prevents changes in data distribution but results in inseparable classes. When the network is trained on incremental tasks (unfrozen). the ground truth distributions change and do not match the memorized ones. A suitable CL method should adapt the mean and covariance of distributions to retain valid decision boundaries.
However, as presented in Fig. when the feature extractor is updated on incremental tasks (it is unfrozen), distributions of previous classes change and no longer match the memorized ones. Suitable methods must adapt both means and covariances. EFC (4 predicts drift (change) only of the distribution mean and points out that adapting covariances is an open question. Second, the methods suffer from dimensionality collapse (30 [6], which is more significant in early tasks. That makes old classes' covariances to be of lower rank than those from recent tasks, which introduces errors while inverting the matrices for the classification, leading increased task-recency bias. We explain this in detail in Sec. 3.2
This work focuses on the challenging problems of adapting classes' covariances and overcoming dimensional collapse in EFCIL. We are the first to introduce a method that adapts the mean and covariance of memorized distributions, significantly reducing the error between memorized and ground truth distributions. We also overcome the dimensionality collapse of feature representations by introducing novel anti-collapse loss, which alleviates the problem of task-recency bias. We dub the resulting method AdaGauss Adapting Gaussians. Our contributions are as follows:
We analyze dimensionality collapse in EFCIL settings and explain that it leads to task- recency bias. We introduce novel anti-collapse loss to prevent it.
We show that knowledge distillation techniques in EFCIL provide different representation strengths of the feature extractor. We are the first to utilize knowledge distillation through a learnable projector network EFCIL.
Based on these findings, we propose AdaGauss, a novel method to adapt both means and covariances of memorized class distributions, which results in state-of-the-arr results when the model is trained from scratch or starting from a pre-trained weights.
2 Related works
Semantic drift. We investigate offline, EFCIL setting (26 focusing on keeping the network size constant, where no task information available test time. Regularization-bur based approaches penalize changes to important neural network parameters 6,77 or use distillation techniques to regularize neuron activations 21 45
5T 24]. However, even with knowledge distillation, the
features from old classes will change, causing catastrophic forgetting g 27 Therefore, few works
36). However, those strategies' limitations are that they adapt only prototypes, ignoring changes in covariance matrices, which we experimentally show suboptimal. As predicting the drift is challenging, many methods focus scenarios where the backbone is frozen after the first task 4
tried predict these changes by approximating their semantic drift [45] 15]
31
2
29 10. However, this
prevents the feature extractor to adapt to new tasks (2 We show that possible to change the feature extractor and adapt the covariance matrices of classes.

---

Task-recency bias. Another challenge in CL is a task-recency bias, where the model is biased towards classifying classes from new tasks [33 26 50). While some works approached this problem using the problem is amplified in an exemplar-free setting. Some works considered
exemplars 1 43
13 48
38, (33. To improve this strategy, PASS [52] included prototype augmentation, and EFC 24 updates their prototypes after each task. In this work, we point out that the cause for task-recency bias in the EFCIL scenario is the dimensionality collapse of the feature extractor, leading to numerical instabilities when inverting covariance matrices.
prototype replay, which maintains the decision boundary between classes [3 51
Dimensionality collapse. Recent works revealed that supervised learning exhibits signs of neural collapse [30 [6], where large fraction of features' variance is described only by a small fraction of their dimensions. Since then, several studies [6] [6] showed that utilizing additional MLP
projector is a crucial component to alleviate the collapse of the representations and improve their transferability. Another implication of the neural collapse in CL is that it becomes challenging to
invert covariance matrices. Existing methods add constant value to the diagonal 35 24 [1 of
the covariance matrices or utilize shrinking (10] to prevent that. On the contrary, we propose an
anti-collapse loss, which is more elegant and does not artificially alter covariance matrices.
3 Method
3.1 Exemplar-Free Class-Incremental Learning (EFCIL)
Mass-Incremental Learning (CIL) scenario considers dataset split into T tasks, each corresponding to the non-overiapping set of classes C1 U C2 U...u U CT = C' such that C2 n nCC = 0 for t & s. In Exemplar-Free CIL (EFCIL), during a training step t, have only access current task data Dj {(2, v)ly E J.} and we cannot store any exemplars from the previous steps. The objective to train model that discriminates between old (< f) and new classes combined. We assume a task-agnostic evaluation [1 26 where the method does not know the task id during the inference.
3.2 The three observations that motivate towards AdaGauss
In this section, we provide insight into problems with current EFCIL methods. We train the standard ResNet18 [T] network on the ImagenetSubset dataset divided into ten equal tasks. We point out that: 1. covariance of class distributions during CL sessions change and must be adapted; 2. the task recency bias comes from the differences in representational strength of the model; 3. when training from scratch in EFCIL, the models are susceptible to dimensionality collapse.
Observation 1. As illustrated in Fig. [ training the feature extractor on incremental tasks makes memorized distribution not match the ground truth (GT) ones. More specifically, the mean and covariance of GT change, and to keep valid decision boundaries, both memorized means and covariances must be adapted. That decreases symmetrical KL divergence between memorized and GT distributions, thus increasing average accuracy after the last task. However, existing state-of-the-art art methods 36 24 51 52 do not adapt covariance matrices, while others [I] 10 55] [4] freeze the
feature extractor after the initial task, which does not guarantee separability of classes from new tasks (first image in Fig. [
Observation When training the feature extractor with different knowledge distillation methods (feature 53 52 45), logit [21 33 projected (18) ),, representational strength of the feature extractor increases with each task, as presented in Fig. 2 That makes memorized covariance matrices of late tasks have a higher rank than those from early tasks, presented in Fig- 3 When these matrices are inverted, the opposite happens due to numerical instabilitics, norms of inverted covariance matrices of carly tasks will be greater. That causes task-recency bias presented in Fig. 4 In the case of Bayes classification [35 [0], the Mahalanobis distance is much higher for early tasks, whereas in the case of sampling gecudo-proooyyee
[24] [1]] the logits for recent tasks are higher, what skews classification towards recent tasks. This bias differs from already well-studied linear head bias [3] 43 48) as
occurs at the level of the representations, where no linear head and exemplars are utilized.
Observation 3. Fig. also presents that feature extractor suffers from dimensionality collapse (30 16 as ranks of covariance matrices are much lower than the latent space size (512 for ResNer18) That makes classes covariance matrices non-invertible. That, turn, disallows the calculation of Mabalanobis distance, likelihood, and sampling from such collapsed distribution. In order
3

---

8000
none
Coveriance matrices rank
Irwented covvel malt mattiiies
distance
7000
30
feature
projected
1403
E000
Flgm NePOON wegeente
1003
8
8000
5
800
2000
12
Log
400
2000
10
Task 10
10
Task
Figure 2: The representational
Figure 3: Average rank of mem- Figure 4: Average Mahalanobis orized covariance matrices of distance between memorized classes after each task (black) distributions and joint dataset ImagenetSubset for logit distil- per each task after the last task lation. Norm of these matrices (black) and average logit value
strength of ResNet18 trained
on 10 tasks of ImagenetSubset dataset split into tasks for different knowledge distillation methods. After each task, we when inverted (green). Lower on linear head trained by sam- measure how many cigenvalues rank leads to larger values in pling prototypes from memo- sum to 95% variance of all fea- inverses of covariance matrices rized distributions. There is a tures provided.
due to numerical instabilities. visible task-recency bias.
to overcome this issue, the existing methods utilize shrinking (10) or add constant value to the diagonal 335 24 51] of the covariance matrices to prevent that. However, these techniques artificially alter classes' distributions, introducing additional hyperparameters and source of errors accumulating during long CIL sessions. A more elegant solution would directly prevent the dimensionality collapse of the feature extractor during training while preserving the class separability provided by cross-entropy.
3.3 AdaGauss method
Motivated by these three observations, we made the following decisions about AdaGauss. Based on the first observation, after training the feature extractor F on incremental task, we train an auxiliary network (adapter), which we utilize adapt the means and covariances of old classes to the latent space of the new feature extractor. To perform knowledge distillation and improve the representation strength the feature extractor (second observation), we utilize feature distillation through learnable projector. In order to overcome the dimensionality collapse and task-recency bias, showcased by the second and third observations, we utilize a novel anti-collapse loss that regularizes the features' covariance matrix and prevents dimensional collapse. AdaGauss memorizes each class as mean and covariance and performs Bayes classification as in [10 35]. We provide a pseudo-code of our method in Alg. Below, explain the motivation and details of the method.
3.3.1 Feature distillation through learnable projector
Inspired by appresenaaiional-laanii [18], we utilize feature distillation through learnable projector to mitigate forgetting, which we refer to as projected distillation. As presented in Fig. 2 this distillation technique provides representations with better eigenvalues distribution, thus decreasing problem task-recency bias compared to standard logit [21] distillation techniques. As the projector, utilize 2-layer MLP network d R
[3] and feature 45
53
R
with hidden size times bigger than the latent space S. Following existing continual learning works 21 45 33), when training F; minibatch B, we freeze Fi-1 trained the previous task. Finally, we calculate our knowledge distillation loss as follows:
LPKD
Σ E 1F (x:)) - Fr- (I:) TEB
3.3.2 Overcoming dimensionality collapse
As described in Sec. 3.2 existing methods for EFCIL that represent classes as Gaussian distributions suffer from dimensionality collapse, which leads to task-recency bias caused by the fact that ranks of covariance matrices different amongst the tasks. To overcome the collapse, encourage the

---

feature extractor to produce features whose dimensions are linearly independent. Therefore, in each task, we directly optimize the covariance matrices of features produced by F, to be positive-defintive by the diagonal of the Cholesky decomposition of covariance of each training minibatch to be positive. More precisely, S the size of the feature vectors and a, be i-th element of the diagonal of Cholesky decomposition of minibatch's s covariance matrix. We formulate the anti-collapse loss AA in the form:
Σ min(a;, LAC = S
(2)
This loss forces Cholesky's decomposition covariance of each minibatch to have diagonal entries greater than 1. Therefore, they are positive, and the covariance matrix is positive-definite due to the property of Cholesky decomposition. More on the definition Lac in Appendix, Sec. A.2)
3.3.3 Training the feature extractor
In each task 1, we train all parameters of the feature extractor Fr together with additional projector
used for knowledge distillation. Following most works [T0] 35 51] 31 21], we utilize popular
cross-entropy loss Log to discriminate between classes. The final loss function is:
L = LCE + LAc + ALPKD,
where À E R a plasticity-stabiliy trade-off hyperparameter, similar to 21
After training the feature extractor, we represent classes Ct as multivariate Gaussian distributions in the latent space. More precisely, we represent any class c E C2 as N ((He: (He:Ee).
3.3.4 Adapting Gaussian distributions
(changed) and no After training of F; is completed, representations of old classes drifted 45 longer match memorized Gaussians. Therefore, we update memorized Gaussians representing past classes to recover ground truth representations. To do that, we train an auxiliary adaptation network wit-1-t TR. -> R (called adapter), which maps features from the old latent space the new one. We use only the current data from task for that. Training loss is:
Ly Σ (- BEB
(Fi-1(z.)) - F((2))11 + LAC.
LAC is the same anti-collapse loss as used during the training the feature extractor. After training the adapter, for each old class c, we sample from ((te: Ex) set of N points: nl, R2,.-.. NN where N >> |S| and transform them through & obtaining new set: (,))),, (n2),.... ((NN) }. We calculate adopted mean pa new and covariance Nnew using new of data and update the old distribution as follows: (Me, Ex) (urew, Enew) A pseudocode of the full AdaGauss method is presented in Alg. 0
4 Experiments
Datasets and metries. We evaluate our method on several well-established benchmark datasets.
CIFAR 100 [9] consists 50k training and 10k testing images in resolution 32x32. TinyIma- geNet 200. subset of ImageNet [8). has 100k training and 10k testing images in 64x64 resolution. ImagenetSubset contains 100 classes from ImageNet (ILSVRC 2012) [34]. We split these datasets into 10 and 20 equal tasks. Thus, each task contains the same number of classes, standard practice 35 24]. We also evaluate our method on fine-grained datasets: CUB200 42
in EFCIL 21 45|
represents 11, 788 images of bird species, and FGVCAircraft (25 dataset consists of 10, 200 images of planes. We split fine-grained datasets into 5, 10, and 20 tasks. As the evaluation metric, we utilize commonly used average accuracy Atast, which is the accuracy after the last task, and average incremental accuracy Aines which is the average of accuracies after each task [26] [0).
Baselines and hyperparameters. We compare our method to multiple EFCIL baselines. Well- established ones, like EWC (17), LwF 21), PASS 52] 11.2A (5I, SSRE (33) and the most recent
and strong EFCIL basclines: FeTrIL 31. FeCAM [10] DS-AL 54 and EFC 24 For the bascline
5

---

Algorithm 1 AdaGauss: Adapting Gaussians in EFCIL
1: Initialize: Training data (D1, D.,...,,DDD........ Dr), Pi (feature extractor), X, N
2: Train F1 on D1 using LCE + LAC
3: for c C1 do
4: Obtain set of features: = ( D1}
5: Store P. = mean(O) and Σ = covariance(O)
6: end for
7:
for t ===2, 2. 4. do
8: Initialize o² 0' (distiller), 1 (adapter)
9: Train F, on D. using L = LCF LAC + ALPKD
10:
for E C2 do
11:
Obtain set of features: 0 = (F((z)) I, )
12:
Store He = mean( (O) and De = covariance(O)
13:
end for
14:
Train adapter 1 on Dr using Lo + LLC
15:
for C EEU do
16:
Sample n.,, R .2,.... NN from (H Σ.)
17:
Calculate ueeu Snew of set (w- (((--+((m))) (m1), (ft (02-)-)((2). "..),......1. 10-1~11(000 (nx))
18:
[ec. = pe neew Ex Dnew
19:
end for
end for
results on CIFAR100. TinylmageNet, and ImagenetSubset, we take the results reported in 24], while for FeCAM, we run its original implementation. For fine- fine-graine grained datasets (CUB200, FGVCAircrafts) we run implementations provided in FACIL 266 and PyCIL 449 frameworks (if provided) or from the authors' repositories. We set default hyperparameters proposed in the original works. We utilize random crops and horizontal flips as data augmentation.
Implementation details and reproducibility. We utilize standard ResNet18 11 as a feature extractor FF for all methods. We train it from scratch on CIFARIO0, IinylmaaenetSusstl, and IuagenetSubset, while for experiments on fine-grained datasets, we utilize weights pre-trained on ImageNet. We implement our method in FACIL[26] benchmark We set À 10, N 10000, d = 32 and add single linear bottleneck layer at the end of the F with S output dimensions, which define the latent space. When training from scratch, we set == 64, while for fine-graing graigre datasets, we decrease it to 32, as there are fewer examples per class. We use an SGD optimizer running for 200 epochs with weight decay equal to 0.0005. When training from scratch, we utilize a starting learning rate (Ir) of 0.1, decreased by ten times after 60, 120, and 180 epochs. We train the adapter using an SGD optimizer with weight decay of 0.0005, running for 100 epochs with a starting Ir of 0.01; we decrease it ten times after 45 and 90 epochs.
We utilize a single machine with an NVIDIA RTX4080 graphics card to run experiments. The time for execution of a single experiment varied depending on the dataset type, but was at most ten hours. We attach details of utilized hyperparameters scripts the code repository. We report all results as the mean and variance of five runs.
4.1 Results
Training from scratch. We present the baseline results and AdaGauss method when training from scratch in Tab. [ We consider T = 10 and T 20 equal tasks. We can see an improvement over the most recent state-of-the-art method EFC 24 We improve its results by 3.7% and 6.8% points in terms of average accuracy on ImagenetSubset split into 10 and 20 tasks, respectively. This improvement is also consistent terms of average incremental accuracy 5.1 1% and 7.5% points and on the other datasets. This increase can be attributed to the fact that EFC does not adapt covariance matrices from task task (just means), which, as we showed in Sec. 3.2 is required to improve the results. Older method- IL2A [5I), which does not adapt their classes representations (means and covariance matrices) method at all, achieves much lower results than our approach 23.4% and 25.1% % points lower average accuracy on ImagenetSubset.
"The code is provided in the Supplementary Materials and will be published upon acceptance.

---

Table I: Average incremental and last accuracy in EFCIL when training the feature extractor from scratch. The mean of 5 runs is reported. Full results are in Tab.
We denote the best results bold.
CIFAR- 100
TinylmageNet
ImagenetSubset
Method
T=10
T=20
7=10
7=20
T=10
T=20
Atast
Ame
Alast
Aine
Alast
Aine
Alast
Ame
Atast
Ainc
Atast
Aine
EWC 17
31.2
49.1
17.4
31.0
17.6
32.6
11.3
26.8
24.6
39.4
12.8
27.0
LWF 21
32.8
53.9
17.4
38.4
26.1
45.1
15.0
32.9
37.7
56.4
18.6
40.2
PASS 52
30.5
47.9
17.4
32.9
24.1
39.3
18.7
32.0
26.4
45.7
14.4
31.7
IL2A [5
31.7
48.4
23.0
40.2
25.3
42.0
19.8
35.5
27.7
48.4
17.5
34.9
SSRE
30.4
47.3
17.5
32.5
22.9
17.3
30.6
25.4
43.8
16.3
31.2
FeTHIL
31
34.9
51.2
23.3
38.5
31.0
25.7
39.5
36.2
$2.6
26.6
42.4
FeCAM
10
32.4
48.3
20.6
34.1
30.8
44.5
25.2
38.3
38.7
54.8
29.0
44.6
DS-AL
54
40.8
54,9
31.7
43.2
33.6
47.2
26.5
41.6
46.8
58.6
36.7
48.5
EFC
24
43.6
58.6
32.2
47.3
34.1
48.0
28.7
42.1
47.4
59.9
35.8
49.9
AdaCiauss
46.1
60.2
37.8
52.4
36,5
50.6
31.3
45.1
51.1
65.0
42,6
57.4
Table 2: Average incremental and last accuracy in EFCIL fine-grained scenarios when utilizing pre-trained feature extractor. We report the mean runs, while variances are reported Tab,
CUB200
FGVCAircraft
Method
T=5
T=10
T=20
T=5
T=10
T=20
Ajast
Aine
Alast
Ainc
Alast
Ainc
Alast
Ainc
Alast
Ainc
Alast
Ainc
EWC 17
21.6
38.2
15.8
32.6
12.3
27.2
24.3
44.0
14.3
34.5
10.9
27.9
LwF 21
44.3
57.7
30.4
46.1
19.4
34.7
39.0
55.2
28.0
46.5
14.7
30.5
PASS [22
34.5
48.6
27.0
42.3
18.1
36.9
33.3
48.9
26.4
41.0
13.9
28.1
IL2A [51]
36,9
51.3
29.4
45.5
20.8
35.1
39.4
49.1
27.3
45.1
14.2
28.7
FeTril 3T
41.9
53.2
36.9
48.2
34.6
45.3
46.0
58.5
40.5
53.4
32.5
43.3
FeCAM
10
43.5
56.0
40.2
54.9
36.2
48.9
45.3
58.0
41.4
55.2
34.0
46.0
DS-AL
54
49.4
61.9
45.8
59.1
41.4
53.8
50.6
62.7
42.6
56.4
34.2
46.7
EFC 24
58.3
68.9
51.0
63.3
46.1
59.3
50.1
63.2
43.1
57.6
28.1
48.2
AdaGauss
60.4
69.2
55,8
66.2
47.4
60.6
53.3
64.0
47.5
58.5
34.8
48.6
overcome the problem of distribution drift by freezing the feature extractor on the first task. However, it cannot adapt well to the new incremental tasks, resulting in poor plasticity and worse results than AdaGauss and EFC (24). FeTriL achieves 14.9% and 16.0% points lower average accuracy on ImagenetSubset, while FeCAM 12.4% and 13.6%.
Methods such as FeTrIL [1 FeCAM (10) and DS-AL
34
Training from pre-trained model. We provide the baseline results and our method when training from ImageNet pre-trained model in Tab. [ Despite having a strong feature extractor from the very beginning. still needs be adapted discriminate better between fine-grained classes. We report results for 5, 10, and 20 equal tasks. AdaGauss achieves state-of-the-a results. It improves the average accuracy of the second-best method EFC (24) by 4.8% and 4.4% points on CUB200 and StanfordCars for T' 10, respectively. The results are consistent for other number of tasks.
Ablation study. We perform ablation of our method CIFARI00 and ImagenetSubset datasets split into ten equal tasks in Table 3 First, we test our method with the nearest mean classifier (NMC) instead of the Bayes classifier to verify whether considering covariance improves the results. Without covariance matrices and with NMC [33] row), we get worse results: 9.7% and 9.6% points lower average accuracy CIFAR 100 and ImagenetSubset, respectively. Memorizing covariances and sampling pseudo-prototypes to adapt means (2nd row) improves NMC results only slightly. Next, we utilize the Bayes classifier instead of NMC but assume that class distributions have diagonal covariance matrices (3rd row). That decreases the average accuracy of our method by 5.0% and 3.9%, respectively, proving that ground truth test distributions have non-zero off-diagonal. Then, test our method without adapting means (5th row) like in IL2A [5I] method. That severely hurts the performance average accuracy decreases by 21.5% and 27.2 %. On the contrary, if we adapt means but not covariances like in EFC (24). we lose far less, 3.2% and 3.1%, respectively. Lastly, we check the performance of our method without the LAC component. To allow covariance matrices to be invertible, we add shrink value of 0.5, similarly to [10]. This results in average accuracy drop 5.9% and 4.0%. The results are also consistent with the average incremental accuracies. This ablation proves our design choices and that all components are necessary to get the best results.

---

Table 3: Ablation of AdaGauss indicating the contribution from the different components. signifies that we utilized covariance matrix shrinking with the value of 0.5 (chosen on the validation set) instead of anti-collapse loss to overcome the covariance matrix singularity problem.
CIFAR-IO 100 (T=10)
ImugenetSubset (T=10)
Classifier
Cov. Matrix
Adapt mean
Adapt covariance
LAC
Ataat
Aine
Alaxt
Auue
NMC
None
36.4
54.0
41.5
57.9
NMC
Full
37.6
54.8
42.3
58.7
Bayes
Diagonal
41.1
56.3
45.5
61.3
Bayes
Full
x
X
x
22.9
42.8
22.5
43.4
Bayes
Full
x
24.6
44.7
23.9
44.9
Bayes
Full
X
42,9
57.7
48.0
62.7
Bayes
Full
V
40,2
56.2
46.7
57.1
Bayes
Full
46.1
60.2
51.1
65.0
4.2
Adaptation results
We verify how our adaptation method improves the quality of memorized class distributions on ImagenetSubset split into ten equal tasks. For this purpose, we measure the average distances between memorized and real classes after each task. Moor precisely. we measure the L2 distance between means and covariances as well as symmetrical Kulbach-Leibles divergence (Dki.) between memorized and real distributions. We utilize projected distillation (A = 10) and compare our method to baseline that does not adapt distributions like in [51 52] (No adapt) and the prototype drift
compensation introduced in EFC [24] that adapts only means. We provide results in Fig. 5 We can see that our approach allows us to better approximate ground truth distributions. More precisely, compared to EFC, it decreases the distance to real-mean by =29%, real-covariance by =399 and ky distance by 772%. We can also see that the EFC approach does not improve distance to revl-covariance compared to adaptation, which is drawback of this method.
distance distrieution
400
Distance real-mean
Distance to real-covariance
350
No adapt
EFC
McCaus
10
800
250
200
6
150
4
100
Task
Task
Task
Figure 5: Distances from memorized distributions to the real ones in terms of distributions' mean, covariance and KL divergence across 10 tasks on ImagenetSubset dataset. AdaGauss greatly reduces enrors and allows for better adaptation than prototype drift compensation (EFC).
4.3 Analysis of anti-collapse loss
We analyze the impact of anti-collapse LaC regularization term ImagenetSubset split to 10 equal tasks. After the last task, we verify how much LAC improves the distribution of classes' covariance eigenvalues. We report results in Fig. Without utilizing TAACs the largest eigenvalue is ~ 1.2 10 times greater than the lowest, showcasing the dimensionality collapse. However, with LAC, this difference equals s88,, proving that more eigenvectors contribute towards representations, and the collapse is greatly diminished.
Next, we measure the average rank of covariance matrices memorized in each task for different knowledge distillation methods and projected distillation with LAC- Here, we S 64. In Fig can see that without LAC, all distillation methods present in existing methods struggle to achieve class covariance equal to latent size S, which according to Sec. 3.2 results in task-recency bias. Interestingly, when combining projected distillation with LAC, the rank of covariance matrices equals 64 for each task, proving that TAAG a promising approach for combating dimensionality collapse when training from scratch.
8

---

An alternative method for overcoming singularity in covariance matrices is shrinking [0]. In Fig. B we present results for our method with different values of shrink performed when calculating covariance matrices on CIFARIO0 100. Intuitively, increasing the shrink value decreases the method's s efficacy, as it artificially alters the covariance to be different from the ground truth representation. Without using Ac and without shrink, it is impossible to invert the matrices, resulting in the crash of the method. Nevertheless, the results are the highest when utilizing LAc without shrink (60.2%)
With Ezc
Without
Class covariance rank
100
Distilation
projected
Avarage Inc. acturacy 56
57.5
62.0
Lac
Without Lx
feature
projected Lac
80
ligit
55.0
Elgenvalue
10'
60
52.5
10
500
40
10-4
47.5
20
45.0
10"4
42.5
3.0
16
2
3
4 5
8
9
10
Shrink
Index of eigenvector
Task
Figure 6: Distribution of eigen- Figure 7: Ranks of classes' Figure Average incremen- values of class representations covariance matrices with dif- tal accuracy for AdaGauss for for our method with and without ferent distillation methods and
different values of covariance LAC (anti-collapse loss) term. projected distillation with anti- LAC greatly reduces the differ- collapse term (red) for latent ence between the most and least space size S = 64. Lac makes
shrinking, with and without anti- collapse regularization. Results without LaC and shrink were not included due to the inabil- ity invert covariance matrices after the first task.
significant eigenvalues, thus pre- covariance ranks to equal to venting dimensional collapse. S in every task.
4.4 Different distillation techniques
We test the performance of the projected distillation against other distillation techniques in AdaGauss. We train from scratch on CIFARI00 ImagenetSubset and utilize the pre-trained model on CUB200. We split datasets into ten equal tasks and use hyperparameters from experiments in Tab. L and Tab. [] We present the results in Fig. Projected distillation achieves better average accuracy than logit distillation by 1.4%. 0.9%. and 4.0% points on CIFAR 100, ImagenetSubset, and CUB200, respectively. Interestingly, the gap between projected distillation and not using knowledge distillation is much lower on CUB200, which we contribute to using strong pre-trained model.
4.5 Memory requirements
Our method does not increase the number of feature extractor's parameters. In addition, the adapter and distiller are discarded after the training, thus not increasing memory during the long CIL sessions and evaluations. AdaGauss requires S S(8-1)
Avcroso sccursc (9))
parameters to memorize the mean and covariance of a class, where S is the latent space size. Therefore, the method requires the same number of parameters as FeCAM [10] and fewer weights than EFC [24 as we do not expand the linear classifier. Additionally, S can be decreased using linear bottleneck layer before the latent space.
CIFAR100
ImageNetSurose
CUB200
Figure 9: Average last task acc. of our method for different knowledge distillation techniques.
4.6 Time complexity of AdaGauss
We measure the training and inference time of popular EFCIL methods using their original implemen- tations single machine with NVIDIA GeForce RTX 4060 and AMD Ryzen 5 5600X CPU. We repeat each experiment times, train all methods for 200 epochs, use four workers, and have a batch size equal to 128. We test vanilla AdaGauss and AdaGauss, where the Bayes classifier is replaced with a trained linear head, where the classifier is trained on samples from class distributions (mean and cov. matrix). We utilize the FeTriL version with linear classification head.
We present results in Tab. The inference of our method takes similar amount of time as in FeCAM, as the feature extraction step is followed by performing Bayes classification. The inference

---

time of AdaGauss is slightly higher than that of methods with linear classification head (LwF, FeTrIL, AdaGauss with linear head) because Bayes classification requires an additional matrix multiplication when calculating the Mahalanobis distance.
The training time of AdaGauss is longer than for LwF, EFC, FeCAM, and FeTriL we do not freeze the backbone after the initial task and additionally train the auxiliary adaptation network. Still, AdaGauss takes less time train than its main competitor EFC, and much faster than SSRE. Our method does not increase the number of networks' parameters because the distiller and the adapter are disposed after training steps.
Table 4: Time complexity of EFCIL methods measured on CIFAR100 split into 10 tasks.
LwF
FeTriL
FeCAM
SSRE
EFC
AdaGauss
AdaGauss (lin, head)
Inference time (sec)
66.3
68.343.4
74.244.0
274.7=12.3
67.43311
72,843.2
65. 65,3422,
Training time (min)
2.9
5.310.3
0.4
548.1117,4
92.1
86,3132
103,314.2
5 Conclusions and limitations
In this work, we analyze the impact of dimensionality collapse in EFCIL. We explain that it leads to differences across tasks in ranks classes' covariance matrices, which in turn causes task-recency bias. We also present that due to distribution drift, means and covariance of classes change, and they should be adapted from task to task. Based on these findings, we propose the first EFCIL method to adapt both means and covariances, dubbed AdaGauss. It utilizes feature distillation through learnable projector and a novel anti-collapse regularization term during training that prevents having degenerated, non-invertible features covariance matrices class representations. That, in turn, alleviates the task-recency bias of the classifier in continual learning. With the series of experiments, show that AdaGauss achieves state-of- offtthe-rt art results in common EFCIL scenarios, both when trained from scratch and when initialized from a pre-trained model.
The limitation of our method is that the cross- cros--enropyy separates classes only from the current task. However, when training the feature extractor, old classes can begin overlapping with each other and with new classes int he latent space causing forgetting. This problem is an open question in EFCIL. We speculate it can alleviated wit contrastive loss. Another problem arises when there very little data representing single class, making high-dimensional covariance matrix impossible to calculate. We tackle it by introducing a bottleneck layer at the very end of the feature extractor. However, it can limit its representational strength.
Acknowledgments
The research of Grzegorz Rypeśé was supported by the National Science Centre (Poland), PRE- LUDIUM 23 grant no. 2024/53/N/ST6/0001 This research was partially funded by National Science Centre, Poland, grant no: 2020/39/B/ST6/015/ 2022/45///ST6/0228 17, and 1/23S//////6/002846 Bartiomicj Twardowski acknowledges the grant RYC2021--032765-1 This paper has been supported under the project "ELIAS: European Lighthouse of AI for Sustainability". GA no. 101 120237. We gratefully acknowledge Polish high- performnrre computing infrastructure PLGrid (HPC Center: ACK Cyfronet AGH) for providing computer facilities and support within computational grant no. PL.G/2223/174331.
by the Horizon Europe Programme CKORZZOX-CLL-2002-HUAAAA
References
[1] Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyconsu Bang, Hyojun Kim, and Tacsup Moon. Ss-il: Separated softmax for incremental learning. In ICCV, 2021.
[2] Rahaf Aljundi, Francesca Babiloui, Mohamed Elhoseiny, Mareus Rohrbach, and Tinne Tuytclaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pages 139- 154,-1544220 2018
[3] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3366-3375. 2017.
10

---

[4] Eden Belouadah and Adrian Popescu. Deesil: Deep-shallow incremental learning, In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[5] Florian Bordes, Randall Balestriero, Quentin Garrido, Adrien Bardes, and Pascal Vincent. Guillotine regularization: Why removing layers needed to improve generalization in self-sup learning. Transactions on Machine Learning Research, 2023.
[6] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingan Ajanthan, and Philip HS Torr. Riemannian walk incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pages 532-547, 2018.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 245-255.
leee, 2009.
[9] Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1(4):128-135, 1999.
[10] Dipam Goswami, Yuyang Liu, Bartlomiej Twardowski, and Joost van de Weijer. Fecam: Exploiting the heterogeneity of class distributions in exemplar- free continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint Ar0.i:5503.00233 023.3333,,,, 2015.
[13] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831-839, 2019.
[14] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao Tao, Dongyan Zhao, Jinwen Ma, and Rui Yan. Overcoming catastrophic forgetting for continual learning via model adaptation. In International conference learning representations, 2019.
[15] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through feature adaptation. In Computer Vicion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 699-715. Springer, 2020.
[16] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning, arXiv preprint arXiv:21 10.00310000344,, 2021.
[17] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan. Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2017.
[18] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pages 3519-3529, PMLR, 2019.
[19] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[20] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)3, 2015.
[21] Zhizhong Li and Derek Hoiem. Learning without forgetting. Transactions Pattern Analysis and Machine Intelligence (T-PAMI), 2017.
[22] Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bag- danov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 2262-2268. IEEE, 2018
[23] Chunwei Ma. Zhanghexuan Ji, Ziyun Huang, Yan Shen, Mingchen Gao, and Jinhui Xu. Progressive voronoi diagram subdivision enables accurate data-free class-incremental learning. In In The Eleventh International Conference on Learning Representations, 2023.
[24] Simone Magistri, Tomaso Trinci, Albin Soutif- Cortil-Cormerais, Joost van de Weijer, and Andrew D Bag- danov. Elastic feature consolidation for cold start exemplar- free incremental learning. arXiv preprint arXiv:2402.00 03910, 2024.
[25] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-graine visual classification of aireraft. arXiv preprint 13055:1300.5151, 2013.
11

---

[26] Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-incremental learning: Survey and performance evaluation image classification. IEEE Transactions on Puttern Analysis and Machine Intelligence, pages 1-20, 2022.
[27] Michacl McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[28] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11321-11329. 2019.
[29] Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E Turner. First session adaptation: strong replay-free baseline for class-incremental learning. In Proceedings of the IEEE/CVF International Conference on Computer Vixion, pages 18820-18830, 2023.
[30] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 2020.
[31] Grégoire Petit, Adrian Popescu, Hugo Schindler. David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar-frce class-incremental learning. In Winter Conference on Applications of Computer Vision (WACV), 2023.
[32] Quang Pham. Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Neural Information Processing Systems, 34:16131-16144 44, 2021.
[33] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[34] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy. Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-25 252, 2015.
[35] Grzegorz Rypeśé, Sebastian Cygert, Valeriya Khan, Tomasz Trzeinski, Bartosz Michat Zielińskii and Barttomicj Twardowski. Divide and not forget: Ensemble of seleetively trained experts in continual learning. In The Twelfth International Conference on Learning Representations, 2023.
[36] Grzegorz Rypešć, Daniel Marezak, Sebastian Cygert, Tomasz Trzciński, and Bartlomiej Twardowski.
Category adaptation meets projected distillation in generalized continual category discovery. In European Conference on Computer Vision (ECCV), 2024.
[37] Wuxuan Shi and Mang Ye. Prototype reminiscence and augmented asymmetric knowledge aggregation for non-exemplar class-ineremental learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1772-1781 1781, 2023.
[38] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class-incremental learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9374-9384, 9384, 2021.
[39] Marco Toldo and Mete Ozay. Bring evanescent representations to life in lifelong class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16732 16741, 2022.
[40] Gido M Van de Ven, Hava T Siegelmann, and Andreas S Tolias. irain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):4069, 2020.
[41] Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904 1904.077444..4 2019.
(42] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd hirds-200-2011 1 dataset, 2011.
[43] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEFF/VV conference on computer vision and pattern recognition, pages 374-382, 2019.
[44] Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023, 2021.
[45] Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class-incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6982-6991, 2020.
[46] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self Self-supervieed learning via redundancy reduction. International Conference Machine Learning (ICML), 2021.
12

---

[47] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning, pages 3987-3995. PMLR, 2017.
[48] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF conference on compiter vision and pattern recognition, pages 13208 13217, 2020.
[49] Da-Wei Zhou, Fu-Yun Yun Wang. Han-Jia Ye, and De-Chuan Zhan. Pycil: A python toolbox for class- incremental learning, 2021.
[50] Da- Wei Zhou, Qi- Wi- Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu, Deep class- incremental learning: survey, arXiv preprint arXiv:2302.0360 2023.
[51] Fei Zhu, Zhen Cheng, Xu- Yao Zhang, and Cheng lin Liu. Class-incremental learning via dual augmentation. Advances in Neural Information Processing Systems (NeurlPS), 2021.
[52] Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self- supervision for incremental learning. In Conference Computer Vision and Pottern Recognition (CVPR), 2021.
[53] Kai Zhu, Wei Zhai, Yang Cao, Jieho Luo, and Zheng-Jun Zha. Self-sustaining representation expansion for non-exemplar class-incremental learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[54] Huiping Zhuang, Run He, Kai Tong. Ziqian Zeng, Cen Chen, and Zhiping Lin. Ds-al: dual-stream analytic learning for exemplar-free class-incremental learning, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17237-17244, 2024.
[55] Huiping Zhuang. Zhenyu Weng, Hongxin Wei, Renchunzi Xic, Kar-Ann Toh, and Zhiping Lin. Acil: Analytic class incremenalt learning with absolute memorization and privacy protection. Advances in Neural Information Processing Systems, 35:1 15:11600 1602-11614, 2022.
13

---

A Additional experiments
A.1 Adaptation results when starting from a pretrained model
We evaluate how our adaptation method improves the quality of memorized class distributions on CUB200 42] split into ten equal tasks when starting from a model pre-trained on ImageNet. For this purpose, we measure the average distances between memorized and real classes after each task. More precisely, we measure the L. distance between means and covariances well symmetrical Kulbach-Leibler divergence (Dxr. between memorized and real distributions. We utilize projected distillation (A 10) and compare our method to a baseline that does not adapt distributions like in 52 (No adapt) and to the prototype drift compensation introduced in EFC [24] that adapts means only (EFC). We provide results in Fig, I0 Results are consistent with Fig. 10 AdaGauss improves memorized distributions after every task.
20.0
17.5
Nio acept
DFC
AdaGaues
Distance to real- coval-ovaaanncee
No adap:
EFC
AdaGouss
distribution
200
Distance real-mean
275
:50
12.5
10.0
100
75
5.0
Dxc
25
Task
5
6 Task
9
10
Task
8
9
10
distances from memorized distributions to the real ones in terms of distributions" mean, covariance and KL divergence across 10 tasks on CUB-200 dataset. The feature extractor was pre-trained on ImageNet.
Figure 10:
A.2 Impact of anti-collapse loss on optimization
Training the feature extractor AdaGauss combines three loss functions: cross-entropy LCE. knowledge distillation through learnable projector LPKD. and anti-collapse term to prevent features from dimensional collapse LAC. We analyze average values of these losses during training of our method ImagenetSubset (we set hyperparameters as in Tab. [ Additionally, modify Eq. 2 to incorporate strength covariance regularization (3 hyperparameter).
1 -
LAC
min(a;, B)
(5)
S
In all of our experiments, we utilized B = as this was sufficient to prevent dimensional collapse. Here, we test AdaGauss for 3 {0.1, 1, 100).
We present results in Fig. We can see that for 3 1, all losses are stable and consistently decrease. LAC decreases to -1.0, value for which it is clipped. However, when increasing 3, LCE and LKD become bigger, underfitting our approach. This results much lower average and incremental accuracies. On the other hand, decreasing / to 0.1 is not sufficient prevent task-recency bias resulting in decreased accuracies.
131
Se C3
Araw
Figure 11: Value of Leg. LPKD, LAC losses for different 3 parameters, last task average accuracy and average incremental accuracy.
14

---

A.3 Tab. and 2ab.2withh variance
We report the mean and variance results reported in Tab. when training from scratch in Tab. 5 Also, we report results from Tab. when training from a pre-trained model in Tab. 6 Although we utilize additional anti-collapse loss compared other methods, variance accuracies achieved by AdaGauss is simillar to EFC.
Table 5: Average incremental and last accuracy in EFCIL scenarios when training the feature extractor from scratch. We report means and variances of runs. We denote the best results in bold.
CIPAR-100
TaylmageNet
ImagenctSubeet
Methed
F-10
7-10
T-10
T-20
Aust
Asuet
EWC 17
212222
49.1+113
174422
310+112
17.6415
32.6+122
11:3+112
26.+111
24.6-4.1
504-3.1
12.8-2.0
27.001.
150000.7
32.9.105
5641100
18.6-177
30.51:10
32:0::1
24.1:005
187::14
32.0+18
26.411.3
45.7:0.2
144-112
31:2:04
198118
3251111
17.311.1
43.1111
16.3-1.1
38.5+11
25.7=06
5266000
26.6-1.5
RCAM
34.1111
445813
25.2:06
38.7810
54820.
24
43.6+0.7
586+09
12.2+13
473+14
341108
4800006
28.7-04
4.1++10
47.4+144
50.9144
358-1.7
40.9+2.1
AdCCus
J6.1105
60.300
378115
524114
365109
50.610.8
31.3110
45.111.2
51,131.2
65.011
42.6.16
$7.4199
Table 6: Average incremental and last accuracy in EFCIL fine-grained scenarios when utilizing pre-trained feature extractor on ImageNet. The means and variance of 5 runs are reported with the best results in bold.
CUB200
StanforoCars
Method
7-101
7-20
T-10
Arsa
Abse
Absrt
Asec
Ams
Aune
Ausi
Ams
Alsst
Abse
Abue
Abe
EWC 1H
21.610.4
38.21..3
15.8.0077
32.6.0.5
12.3108
27.21.0.6
24.3.0.6
44.0.066
14.310.88
34.510.77
10.9.112
17
Lwt
44.320.7
57.7997
30.4:1.1
26.1=100
19.411
34.7=1.8
3902
55.22006
28.021.0
463210
14.7=1.5
305114
PASS
34.5-0.5
486+04
27.0-0.9
42.3-09
181+12
369--11
33.1+10
48.9+09
264+1.1
4100+08
13.0-16
281+1.2
1122 M
169-1.2
51.3+14
204++1.1
45:5-1.7
308+20
35.1+233
304+1.
40.1+1.
27.5+1.6
45.1+17
142-2.1
287++28
TeTtil [3]
41.9105
53.21066
36.910.7
4822.066
346.10
45.3.0.9
16.1.0.7
58.51100
40.510.8
$3.1109
32.5.111
43.311.3
RCAM [0]
45.5205
56020077
40.2:08
$49:10
362811
48921.
45.32115
5800206
414208
56.2509
34.0=111
46D21.
EFC
24
58.3+044
68.106
$1.0+06
63 63.3-0.
46+10
59.3+1.3
901+00
63,2407
43.1408
$7.6+0.7
38.1-188
483+-.
692100
662.08
474405
333.00
640110
473300
348
356113
A.4 Different architecture pretrained backbone
We test AdaGauss with different feautre extractors, namely ViT small and ConvNext. The results, presented in Tab. [ are for EFCIL setting with 10 and 20 equal tasks and weights pretrained on ImageNet (as in Tab. 2). Using more modern feature extractors architectures further improves the results of AdaGauss.
Table AdaGauss results with different backbone architectures. We report last accuracy average accuracy.
CUB200
FGVCAircraft
T=10
T=20
T=10
T=20
Resnet18
55.8
66.2
47.4
60.6
47.5
58.5
34.8
48.6
ConvNext (small)
63.4
73.1
47.9
64.1
49.3
62.9
37.3
51.4
VIT (small)
68.2
77.5
48.9
67.5
48.0
60.6
35.7
50.2
A.5
Half' dataset results
Learning from scratch more challenging than half-dataset setting as it requires incrementally train feature extractor, not just the classifier. However, using the pre-trained model (or learning from half) can be considered more practical and real-life setting. Thus, we evaluated our method with pre-trained model in Table However, we have additionally compared our method to the mentioned baselines a half-dataset setting using augmentations as AdaGauss. Please note that we did not have enough time to perform hyperparameter search for our method- we utilized these from the equal task setting, whereas the results for other methods were optimized by their authors. We provide results in the Tab.
original implementations under the same data
15

---

AdaGauss performs better than PASS, SSRE, and FeTrIL (5 tasks) in the half-dataset setting. However, is slightly worse than most recent baselines when using default hyperparameters, FeCAM. ACIL, and DS-AL. freeze the feature extractor after the initial task, which can explain their good results in the half-dataset training.
Table 8: Half dataset in the first task results.
We report last accuracy average accuracy. ImageNetSubset
CIFARIOU
100
=
T= 10
==
T= 10
PASS
54.5
61.8
53.8
60.9
57.9
64.4
58.2
61.8
SSRE
55.7
63.9
54.9
63.2
58.3
65.2
61.4
67.7
FeTrIL
58.3
65.1
56.2
64.6
65.6
72.8
65.3
72.1
FeCAM
60.2
67.2
59.9
66.9
67.3
75.3
67.6
74.9
ACIL
57.8
66.3
57.7
66.0
67.0
74.8
67.2
74.6
DS-AL
61.4
68.4
61.4
68.4
68.0
75.2
67.7
75.1
AdaGauss
58.9
65.7
55.4
63.7
66.8
74.1
62.8
68.0
A.6 Predicted semantic shift for classes
Here, we verify whether our adaptation network can predict different shift for different classes in experiments from T We test on CIFAR for T = 10 and the answer is positive, as shown in Fig. 12
14
Class
12
0
1
10
11
Adapted distance
10
8
6
4
2
0
2
4
5
6 Task
7
8
10
Figure 12: Predicted shift for different classes on CIFAR 100 (10 equal tasks) by AdaGauss. The Euclidean distance measured between old and new position in each task.
A.7 Batch norm influence on AdaGauss
We measure the accuracy achieved by AdaGauss in the EFCIL scenario from scratch (CIFAR 100, ImageNetSubset) and pretrained (CUB200). We train for tasks without batch norm layers and with frozen batch norm layers. Results provided in Tab, We can that possesing batch-norm layers in R8snet18 is beneficial.
Table 9: AdaGauss results without or with frozen batch-norm. We report last accuracy and average accuracy separated by
CIFAR 100
ImageNetSubset
CUB200
Resnet18 (no BN)
44.6.10.7
58.1
49.110.8
61.7+00.
66.1 54.2+10.5
Resnet18 (freezed BN)
45.3.F0.7
58.710.9
49.410.8
62.0=1.0
55.2+10.4
65.74.0.6
Resnet18
46.140.88
60.2-10.9
51.1+1.0
65.0
55.8+0.5
66.2++0.8
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, main claims accurately reflect contributions and scope of our paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we discuss limitations is Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We discuss reproducibility Sec. We enclose code and scripts to reproduce results. We will also publish them upon acceptance of the manuscript.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We enclose code and scripts to reproduce results. We will also publish them on Github upon acceptance of the manuseript. We utilize open source datasets and libraries.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cmm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //nips.cc/pil wwwppbblccgguiicuceccccc
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We discuss it in Sec. and enclose scripts in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars on our plots. We repeat experiments 5 times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification: Yes, we have read and applied NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work considers fundamental research in EFCIL, we do not see its connec- tion to societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data nor models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, cite creators original owners of assets utilize in work.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, provide documentation of the code submit.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not perform crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "k3Gnl6A54u", "reviewer": "Reviewer_xJqZ", "summary": "This paper analyzes the impact of dimensionality collapse in EFCIL and examines the distribution shift of the mean and covariance matrix. Based on these findings, the paper proposes the AdaGauss method, which adapts the covariance and mean, and designs a loss term to prevent the dimensionality collapse of the feature extractor.", "questions": "1. Please address the questions in the weakness.\n2. Can the proposed method be applied to other network structures such as 4-layer convolutional networks or ViT?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The analysis of the dimensionality collapse of the feature extractor is interesting.\n- The paper provides a comprehensive experimental analysis, which clearly demonstrates the effectiveness of different components within the proposed model.", "weaknesses": "- The writing in the current version needs improvement. There are many parts that I had to reread several times to understand. For example, the three observations in Lines 89-118, I strongly recommend to summarize them in one sentence initially to highlight the core findings. In Section 3.3.2, which is one of the most important parts and contributions of the paper, the explanation is insufficient even though I refer to the appendix. What is the meaning of $a_i$ ? There is no clear definition.\n- Regarding Figure 1, is it merely an illustrative figure, or is it a visualization figure? If it is a visualization figure, what are the details used to depict this figure?\n- In Section 3.3.4, the learned adaptation network is trained on the t-th task. Why can it be used for previous μ and σ? The underlying assumption is that all previous tasks share the same shift pattern. Is this a reasonable assumption?\n- For Algorithm 1, I am not sure how it addresses the Batch Norm statistics, which are crucial to continual learning and the covariance shift this paper focuses on [1,2].\n- The time complexity of the proposed method compared to other methods should be discussed.\n\n\n[1] Continual Normalization: Rethinking Batch Normalization for Online Continual Learning, ICLR 2022.\n[2] Diagnosing batch normalization in class incremental learning, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for the insightful comments and providing relevant works. We begin by responding to the weaknesses:  \n\n**W1: Writing improvements**  \nWe have revisited the whole Section 3 and applied the Reviewer's suggestions in the new version of the manuscript.  \n\n**W1: Meaning of $a_{i}$**  \nThe definition of $a_i$ is provided in line 146: *More precisely, let S be the size of the feature vectors and $a_i$ be the i-th element of the diagonal of a Cholesky decomposition of minibatch’s covariance matrix.* We have made it clearer in the new version.  \n\n**W2: Regarding Fig. 1**  \nFig. 1 is an illustration that provides intuition into the problem of mean and covariance adaptation; it is not a visualization. However, the provided accuracy after the last task and KL divergence (between memorized and real class distributions) were measured in experiments on the ImageNetSubset dataset split into ten tasks. Such illustrations are utilized in a variety of works, e.g. [1] (Fig. 1), [2] (Fig. 1), [3] (Fig. 3), [4] (Fig. 1c).  \n\n**W3: Assumption that all tasks share the same shift pattern**  \nIn EFCIL, we do not have access to old data. Thus, we learn the non-linear adaptation network with current task data and use its generalization for the old classes' distribution (μ and $\\Sigma$). We do not assume that the shift pattern for all the previous tasks is the same. We train an additional **non-linear** network to learn these patterns, assuming that we have only access to the current data. That is the main assumption. The learned adapter network estimates the shift **differently** for each task and its classes, as we visualize in Fig. 1 of the rebuttal file (classes 0, 1 are from 1st task, 10 and 11 - 2nd). On the contrary, in the recent EFC [4], only the mean is adapted, and there is an assumption that the covariance does not shift (Fig. 5).  \n\n**W4: Regarding batch norm**  \nWe agree that BN is an important aspect of CL. However, it is more severe for methods that do not perform shift compensation, which is reflected in the provided works [5, 6]. The fact that BN changes the shift of class representations is fine in *AdaGauss*, as we utilize the MLP adaptor to predict and compensate for this shift. Therefore, our method is agnostic to BN, and BN does not negatively impact *AdaGauss*. To prove it, we provide (Tab.1 in rebuttal pdf) results ($A_{last}$ | $A_{inc}$) with frozen after the first task or removed BN layers from Resnet18. We can see that removing or freezing BN layers slightly decreases the performance of the method.  \nAdditionally, our method is architecture agnostic and can work with networks in which the BN is absent, i.e., ViT, which uses Layer Norm. We provide results with different backbones below in the response to Question 2. We have added a discussion about BN and mentioned RW [5, 6] to our manuscript.  \n\n**W5: Time complexity**  \nWe have gathered cumulative inference and training time of our experiments on CIFAR100 split into ten tasks. We utilize the original implementation for each method and run experiments on a single machine with NVIDIA GeForce RTX 4060 and AMD Ryzen 5 5600X CPU. We repeat each experiment 5 times, train all methods for 200 epochs, use 4 workers and batch size equal to 128. We test vanilla *AdaGauss*, and *AdaGauss* where the Bayes classifier is replaced with a trained linear head, where the classifier is trained on samples from class distributions (mean and cov. matrix). We utilize FeTrIL version with linear classification head.  \n\nWe present results in Tab.2 of the rebuttal pdf file. The inference of our method takes a similar amount of time as in FeCAM, as the feature extraction step is followed by performing Bayes classification. The inference time of *AdaGauss* is slightly higher than that of methods with linear classification head (LwF, FeTrIL, *AdaGauss* with linear head) because Bayes classification requires an additional matrix multiplication when calculating the Mahalanobis distance.  \n\nThe training time of *AdaGauss* is longer than for LwF, EFC, FeCAM, and FeTriL as we do not freeze the backbone after the initial task and additionally train the auxiliary adaptation network. Still, *AdaGauss* takes less time to train than its main competitor - EFC, and is much faster than SSRE. Our method does not increase the number of networks' parameters because the distiller and the adapter are disposed after training steps.  \n\n**Q2: Can the proposed method be applied to other network structures?**  \nYes, the method is architecture-agnostic and can be applied to different network structures and different training regimes. We do not assume any requirements towards backbone architecture. We will make it clearer in the paper, and we will include AdaGauss results for different backbones in the appendix, namely ViT small and ConvNext. These results ($A_{last}$ | $A_{inc}$), presented in Tab.3 of the pdf file, are for EFCIL setting with 10 and 20 equal tasks and weights pretrained on ImageNet (as in Tab. 2). Using more modern feature extractors architectures further improves the results of AdaGauss.  \n\n--- \nWe hope that our explanation has addressed the Reviewer's concerns. Should there be any additional queries, we are willing to provide further details. If no further clarification is needed, we kindly ask the Reviewer to increase the final score.  \n\n---\n[1] Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NeurIPS 2024.  \n[2] Fetril: Feature translation for exemplar-free class-incremental learning. WACV 2023.  \n[3] Semantic drift compensation for class-incremental learning. CVPR 2020.  \n[4] Elastic Feature Consolidation For Cold Start Exemplar-Free Incremental Learning. ICLR 2024  \n[5] Continual Normalization: Rethinking Batch Normalization for Online Continual Learning, ICLR 2022.  \n[6] Diagnosing batch normalization in class incremental learning, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqlyBztETi", "reviewer": "Reviewer_5j6M", "summary": "The paper addresses the problem of Exemplar-Free Class Incremental Learning (EFCIL), which involves training a model on sequential tasks without access to past data. Current methods represent classes as Gaussian distributions in the feature space, enabling Bayes classification or pseudo-feature replay for classifier training. However, these methods face issues such as the need to adapt covariance matrices after each task and susceptibility to task-recency bias due to dimensionality collapse. The authors propose AdaGauss, a novel method that adapts covariance matrices between tasks and incorporates an anti-collapse loss function to mitigate task-recency bias. AdaGauss achieves state-of-the-art performance on popular EFCIL benchmarks and datasets, whether training from scratch or using a pre-trained backbone.", "questions": "I have some questions mainly about the experiments:\n\n1. The experiments in this paper use the \"learn from scratch\" setting. Another common setting used in CIL is the \"learn from half\", which is also a dominant setting in EFCIL. It simply means that half of the classes in the dataset are learned as an initial task, and then the remaining classes are evenly divided into subsequent incremental learning tasks. This experimental setting is used in most of the compared works, such as PASS, SSRE, FeTrIL and FeCAM. How does AdaGauss perform in this setting?\n2. Although the authors provide a detailed ablation analysis in Table 3, I still have some concerns. Why is $L_{AC}$  replaced by \"utilized covariance matrix shrinking with the value of 0.5\" in the sixth row of table 3, instead of just removing it? Are there other components of the proposed method that cannot be run in the absence of $L_{AC}$?\n3. Does the adaptive tuning for Gaussian distributions fail if there is no $L_{AC}$ to resist dimensionality collapse?\n4. Additionally, it would be beneficial if the authors could include a row comparing the performance of AdaGauss without both \"Adapt mean\", \"Adapt covariance\" and $L_{AC}$. This additional row would contribute to a clearer comparison and enhance the transparency of the evaluation.", "limitations": "It is hard to foresee any potential negative societal impact of this theoretical work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall this is a good paper with the following strengths:\n* This paper provides a clear experimental analysis and elaboration of the motivation for the proposed method.\n* For EFCIL, it is important and interesting to track changes in the distribution of past classes.\n* The proposed method achieves good results in multiple settings.", "weaknesses": "* The method of knowledge distillation through projectors proposed in this paper has been widely studied and discussed [1, 2]. Different from the general field of knowledge distillation, what are the technical innovations of the proposed method? Does it have unique insights for continual learning tasks?\n* The proposed method introduces several additional structures and requires a large number of samples from the simulated distribution to be trained, so the number of model parameters and the required training time should be discussed.\n\n[1] Chen Y, Wang S, Liu J, et al. Improved feature distillation via projector ensemble. NeurIPS 2022.\n\n[2] Miles R, Mikolajczyk K. Understanding the role of the projector in knowledge distillation. AAAI 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the Reviewer for the feedback and insightful remarks. We shall begin by addressing the specific weaknesses pointed out:   \n\n**W1: Innovation of knowledge distillation through a projector**  \n\nIn this work, we focus on knowledge distillation in EFCIL, which is different than [1, 2]. KD with the projector for EFCIL is only one part of the proposed solution. Together with adaptation step and anti-collapse loss the method receives the best results (Tab. 3). \n\nAdditionally, our unique insight about KD is explaining how it impacts the task-recency bias in EFCIL. As presented in Section 3.2, KD through a learnable projector is less susceptible to bias because it provides stronger representations. That, combined with $L_{AC}$, allows us to overcome the bias and achieve good results. We have made that clearer in the new version of the manuscript.  \n\n**W2: Number of parameters and time complexity**  \n\nThe number of model parameters is discussed in Section 4.2: *Memory requirements*. After the training, the projectors for KD and adaptation are removed and not stored. Therefore, the number of network parameters is not increased.  \n\nWe aggregated logs from our experiments on CIFAR100 split into 10 equal tasks to compare the training and inference time. We utilized original implementations of baseline methods and utilized Resnet18 trained for 200 epochs with batch size of 128 on a single machine. We used NVIDIA GeForce RTX 4060 and AMD Ryzen 5 5600X CPU for the experiments; we repeated them 5 times. We use a linear head classifier for FeTrIL and also test a version of *AdaGauss* with a linear head classifier instead of a Bayes classifier - we sample from memorized distributions to train it. We present results in Tab.2 of the pdf file.  \n\nMethods that freeze the backbone after the first task (FeTrIL and FeCAM) have lower training time than others. AdaGauss takes less time to train than EFC, as it does not require training the linear head.  \n\nThe inference time of methods that utilize the Bayes classifier (FeCAM, AdaGauss) is higher than methods that utilize linear classification heads (LwF, FeTrIL, EFC). Replacing the Bayes classifier with a linear head in *AdaGauss* boosts its inference time by 7.5 seconds.  \n\nWe have added these results and discussion to the Appendix.  \n\n**Q1: Learn from half dataset**  \n\nLearning from scratch is more challenging than half-dataset setting as it requires to incrementally train feature extractor [3], not just the classifier. However, using the pre-trained model (or learning from half) can be considered a more practical and real-life setting. Thus, we evaluated our method with a pre-trained model in **Table 2**. However, we have additionally compared our method to the mentioned baselines in a half-dataset setting using the original implementations under the same data augmentations as *AdaGauss*. Please note that we did not have enough time to perform hyperparameter search for our method - we utilized these from the equal task setting, whereas the results for other methods were optimized by their authors. We provide results in the form ($A_{last}$ | $A_{inc}$) in the Tab. 4 of the rebuttal pdf file. \n\n*AdaGauss* performs better than PASS, SSRE, and FeTrIL (5 tasks) in the half-dataset setting. However, it is slightly worse than most recent baselines when using default hyperparameters. FeCAM, ACIL, and DS-AL freeze the feature extractor after the initial task, which can explain their good results in the half-dataset training. We have added these results to the Appendix.  \n\n**Q2.1: Why is $L_{AC}$ replaced by shrinking in Tab. 3?**  \n\nSuppose we just remove the $L_{AC}$. In that case, covariance matrices of classes become singular, and it is mathematically impossible to invert them, leading to the inability to perform Bayes classification or to sample from these distributions (required to adapt distributions). Therefore, we utilized the lowest possible shrink value to invert covariance matrices (in this case 0.5; we performed a grid search to obtain it from values 0.01, 0.05, 0.1, 0.5, ...). Shrinkage is a standard technique used in FeCAM[4] and EFC[3] to alleviate the problem of singularity and matrix inversion.  \n\n**Q2.2: Are there other components of the method that cannot be run in the absence of $L_{AC}$**  \n\nWithout $L_{AC}$ we cannot perform classification and adaptaing Gaussians. Both of these require the inverse of class covariance matrices, and they cannot be calculated when covariance matrices are singular. $L_{AC}$ prevents that. We have improved the writing to emphasize this important aspect of the AdaGauss method and made it more clear in the ablation study.  \n\n**Q3: Does the adaptive tuning for Gaussian distributions fail if there is no $L_{AC}$**  \nYes, without $L_{AC}$, class covariance matrices are singular, and we cannot invert them. Therefore, we cannot sample from class distributions (step 16. in Alg. 1).  \n\n\n**Q4: Providing additional row to the ablation study**  \nWe have added this row and the discussion to the new version of the Appendix. We also present the results ($A_{last}$ | $A_{inc}$) in Tab.5 of the pdf file. With all of these components turned off, *AdaGauss* achieves poor results compared to the baseline.  \n\n---- \n\nIf the Reviewer's concerns have been sufficiently addressed in our responses, we humbly seek their support for the paper and improving the score.  \n\n ----\n \n[1] Chen Y, Wang S, Liu J, et al. Improved feature distillation via projector ensemble. NeurIPS 2022.  \n[2] Miles R, Mikolajczyk K. Understanding the role of the projector in knowledge distillation. AAAI 2024.  \n[3] Elastic Feature Consolidation For Cold Start Exemplar-Free Incremental Learning. ICLR 2024  \n[4] Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. NeurIPS 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sLreWfQPnA", "reviewer": "Reviewer_6bmT", "summary": "Existing methods use Gaussian distributions to represent classes in the feature extractor's latent space, but face unchanged covariance matrices and task-recency bias. This paper introduces AdaGauss, an approach that adapts covariance matrices and mitigates the bias through an anti-collapse loss function. AdaGauss achieves top performance on EFCIL benchmarks and datasets, whether training from scratch or using a pre-trained backbone.", "questions": "see weakness.", "limitations": "n.a", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents innovative approaches to tackle the EFCIL problem. \n2. The writing is quite good, and easy to follow overall. \n3. he paper includes comprehensive experimental findings to support its claims.", "weaknesses": "1. Literature is incomplete. The paper concentrates on the EFCIL, yet quite a number of important EFCIL techniques [1-3] are not presented, and I want to see experimental comparisons with [1,3] if possible.\n\n[1]Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, Zhiping Lin”ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection”, Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS) 2022.\n\n[2] Ma, C.; Ji, Z.; Huang, Z.; Shen, Y.; Gao,M.; and Xu, J. 2023. Progressive Voronoi Diagram Subdivision Enables Accurate Data-free Class-Incremental Learning. In The Eleventh International Conference on Learning Representations.\n\n[3] Zhuang H, He R, Tong K, et al. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 17237-17244.\n\n2. The claim \"dimensionality collapse\" concept is very important, but the authors didn't explain anything related.\n\n3. An adaptor is needed for implementing the proposed algorithm. Could you provide a comparison among the proposed methods regarding the auxillary networks imposed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for providing important references, constructive feedback, and insightful comments. Below we respond to the weaknesses mentioned.  \n\n**W1: Incomplete literature and comparison to ACIL and DS-AL**  \n\nThank you for providing relevant literature - we have added it to the Related Works of the paper. Thanks to the good quality of the code in [1, 3] we have compared our method to ACIL and DS-AL. Please note that [1, 2, 3] methods utilize half-dataset warm-start settings in the original papers. Firstly, we ran [1, 3] in the **equal** task setting using original implementations. The results ($A_{last}$ | $A_{inc}$) are below:  \n\n \n|  | CIFAR100  | CIFAR100 | ImageNetSubset | ImageNetSubset |\n| -------- | :--------: | :--------: | :--------: | :--------: |\n|  | T = 10 | T = 20 | T = 10 | T = 20 |\n| ACIL[1]     | 38.8 \\| 53.2    | 30.8 \\| 42.7     | 44.2 \\| 54.8    | 35.3 \\| 47.6 |\n| DS-AL[3]     | 40.8 \\| 54.9 | 31.7 \\| 43.2     | 46.8 \\| 58.6   | 36.7 \\| 48.5 |\n| AdaGauss     | **46.1** \\| **60.2** | **37.8** \\| **52.4** | **51.1** \\| **65.0** | **42.6** \\| **57.4** |\n\nNext, we have run AdaGauss with the half dataset in the first task setting:  \n\n|  | CIFAR100  | CIFAR100 | ImageNetSubset | ImageNetSubset |\n| -------- | :--------: | :--------: | :--------: | :--------: |\n|  | T = 5 | T = 10 | T = 5 | T = 10 |\n| ACIL[1]     | 57.8 \\| 66.3     | 57.7 \\| 66.0   | 67.0 \\| 74.8    | 67.2 \\| 74.6 |\n| DS-AL[3]     | **61.4** \\| **68.4**  | **61.4** \\| **68.4**     | **68.0** \\| **75.2**   | **67.7** \\| **75.1** |\n| AdaGauss     | 58.9 \\| 65.7     | 55.4 \\| 63.7     | 66.8 \\| 74.1    | 62.8 \\| 68.0|\n\nWe can see that *AdaGauss* performs better in the equal task scenario. However, in the half-dataset scenario, ACIL and DS-AL are better. In our opinion, the half-dataset setting prefers methods with a frozen feature extractor (FE) after the first task [1, 2, 3]. However, in this paper, we focus on explaining the task recency bias when training the FE in incremental steps. We also did not have enough time to perform a hyperparameter search. We have added these results and the discussion to the Appendix.  \n\n**W2: Explaining the dimensionality collapse**  \nWe tried to explain this concept in line 73: *a large fraction of features' variance is described only by a small fraction of their dimensions.* This is also reflected by the analysis done in Section 3.2, line 109, where we presented that the rank of the class covariance matrix is far lower than its dimensionality. That is the result of training the feature extractor with cross-entropy that forces the representational collapse of the latent representation to the number of classes that need to be linearly separable [5]. However, we agree that this can still be improved. We have done that in the introduction in our work and included more related work.  \n\n**W3: Different adaptor architectures**  \nWe have included results ($A_{last}$ | $A_{inc}$) for different types of adaptors in the appendix and below. $d$ denotes number of times the hidden layer is bigger than the input and output layers. We train for 10 equal tasks.   \n\n|  | CIFAR100 | ImageNetSubset |\n| :--------: | :--------: | :--------: |\n| SDC[4]     | 43.7 $\\pm$ 0.6     | 46.7 $\\pm$ 0.8     |\n| Linear     | 42.3 $\\pm$ 0.6    | 45.5 $\\pm$ 0.7    |\n| MLP (2-layers, d=4)     | 45.7 $\\pm$ 0.8     | 50.4 $\\pm$ 0.8     |\n| MLP (2-layers, d=16)     | **46.1 $\\pm$ 0.8**     | **51.1 $\\pm$ 1.1**     |\n| MLP (3-layers, d=4)     | 44.6 $\\pm$ 0.7     | 49.9 $\\pm$ 1.0     |\n| MLP (3-layers, d=16)     | 45.1 $\\pm$ 0.8     | 50.1 $\\pm$ 0.8     |\n\nWe can see that utilizing non-linear MLP networks yields better results than linear network and SDC methods. 2-layer MLP networks are also preferred over the 3-layer ones.  \n\n---\nIf we have adequately addressed the Reviewer's concerns, we kindly ask for your support and slightly improving the score. If you have any further concerns or additional points to raise, we are eager to address them. Your insights are valuable in enhancing the quality and impact of our research.  \n\n---\n[1] Huiping Zhuang, Zhenyu Weng, Hongxin Wei, Renchunzi Xie, Kar-Ann Toh, Zhiping Lin ”ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection”, Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS) 2022.  \n[2] Ma, C.; Ji, Z.; Huang, Z.; Shen, Y.; Gao,M.; and Xu, J. 2023. Progressive Voronoi Diagram Subdivision Enables Accurate Data-free Class-Incremental Learning. In The Eleventh International Conference on Learning Representations.  \n[3] Zhuang H, He R, Tong K, et al. DS-AL: A dual-stream analytic learning for exemplar-free class-incremental learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(15): 17237-17244.  \n[4] Yu, Lu, et al. \"Semantic drift compensation for class-incremental learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.  \n[5] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "26JeGQJkKt", "reviewer": "Reviewer_VMJn", "summary": "This paper addresses the Exemplar-Free Class Incremental Learning (EFCIL) challenge. It identifies two critical issues that undermine the effectiveness of existing methodologies and proposes a novel approach, AdaGauss, which adapts covariance matrices from task to task and mitigates task-recency bias.", "questions": "See the weakness.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written and well motivated. It clearly demonstrates that the changes of covariance matrices also matters in Exemplar-Free Class Incremental Learning.\n2. The proposed method demonstrates pretty good experimental results on all five datasets whether training from scratch or starting from a pre-trained backbone. \n3. The proposed method is straightforward and easy to understand.", "weaknesses": "1. I appreciate the efforts the authors have devoted to detailing the three observations encountered in Exemplar-Free Class Incremental Learning. However, observations 2 and 3 appear quite similar to me, as both seem to represent a simplification of the representation for previously seen classes. Additionally, the analysis of observations 2 and 3 does not significantly depart from existing explanations for why classification results tend to skew towards recent tasks, a phenomenon (task-recency bias / representation forgetting) already well-documented in the broader field of continual learning.\n2. The proposed idea that both the mean and covariance should be adapted during training shares similarities with test-time adaptation methods. Therefore, some comparisons are necessary to delineate these relationships further.\n3. Although it is quite straightforward that encouraging the feature extractor to produce features with linearly independent dimensions can mitigate dimensionality collapse, this approach does not guarantee the production of meaningful features. Additionally, it remains unclear whether simply optimizing the covariance matrices of features from one mini-batch can ensure linear independence. Some theoretical analysis would better explain the effectiveness of the proposed loss term $L_{AC}$ in mitigating dimensionality collapse (which I think there do exist some possible results to derive).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback provided by the Reviewer. We will now address the specific weaknesses indicated:  \n\n**W1: Novelty of explanation of the task-recency bias**  \n\nWe agree that task recency bias was well explored in CIL literature. However, in most works [1-4], the focus is on the bias in the linear classifier, which CIL methods try to unskew. On the contrary, in our work, we explain the bias in the embedding space - before the linear classifier and a novel method to prevent it ($L_{AC}$). Observations 2 and 3 show that ranks of covariance matrices increase in the later tasks, leading inverses of covariance matrices to have higher norms. That, in fact, generates task-recency bias in methods that utilize Bayes classifier or sample from class distribution in the embedding space. To the best of our knowledge, we are first to explain this and to provide a simple solution in the form of $L_{AC}$ loss. Less elegant solutions involve techniques like shrinking covariance matrices (used in FeCAM and EFC) as they introduce more hyperparameters.   \n\n\n**W2: Comparison to TTA methods**  \n\nSome TTA approaches, e.g., CAFA [5] or TTAC [6] align the means and covariances at test-time, with those computed during pretraining. However, there exist notable differences compared to the EFCIL setting considered in our work. In TTA, data of all the classes are available during the adaptation phase (it is only the features that change, like in domain incremental learning), whereas in EFCIL, the model is learning new classes in new tasks, and therefore, a class-level feature alignment cannot be applied (as there is no access to previous task classes). However, it is true that there are some similarities (e.g., the feature drift that occurs in TTA due to changes in domains), and we will add references and discussion to TTA works in our work.\n\n**W3.1: Impact of $L_{AC}$ on meaningfullness of features**  \n\nOur loss function consists of three components, which are of a different optimization goal, and a sweet spot between them must be found. As shown in Eq. 5 and Fig.11 in the Appendix, we analyzed the $\\beta$ hyperparameter, which sets the strength for the covariance regularization. As the Reviewer noticed, increasing the $\\beta$ hyper-parameter increases the value of the cross-entropy loss, meaning that features produced by the network are less meaningful (from the classification perspective). Therefore, a good $\\beta$ must be found during hyperparameter search. In all of our experiments, we utilized $\\beta=1$. We have made the tradeoff between loss functions clearer in the new revision.  \n\n**W3.2: Explaining linear independence of features**  \n\nWe verified the linear dependency problem experimentally. The value of  $L_{AC}$ loss remains at -1 after each epoch in our experiments, as presented in the Appendix, Fig.11 ($\\beta$=1). That means that for each minibatch the values of the diagonal of its Cholesky decomposition are greater than 1, and thus are positive. The latter implies that the covariance matrix of each minibatch is full rank [7]. Thus, the features are linearly independent. We agree that optimizing $L_{AC}$ at the mini-batch level is a stochastic problem like SGD, and it also requires that each mini-batch should be representative of the train dataset distribution. However, it results in the overall full-rank covariance matrices (what is presented in Figure 6 and 7 in the main paper). In our work, we use a mini-batch size that is four times bigger than the latent dimension to perform Cholesky decomposition efficiently. \n\n----\nWe hope our response alleviates any concerns the Reviewer may have. However, if there are any remaining uncertainties, kindly specify references and additional questions, and we can further discuss them. Otherwise, we would appreciate if the Reviewer reconsidered improving the final score of our submission.  \n\n----\n[1] Wu, Yue, et al. \"Large scale incremental learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.  \n[2] Zhou, Da-Wei, et al. \"Deep class-incremental learning: A survey.\" arXiv preprint arXiv:2302.03648 (2023).  \n[3] Castro, Francisco M., et al. \"End-to-end incremental learning.\" Proceedings of the European conference on computer vision (ECCV). 2018.  \n[4] Hou, Saihui, et al. \"Learning a unified classifier incrementally via rebalancing.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.  \n[5] Su, Yongyi, Xun Xu, and Kui Jia. \"Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering.\" NeurIPS 2022.  \n[6] Jung, Sanghun, et al. \"Cafa: Class-aware feature alignment for test-time adaptation.\" ICCV 2023.  \n[7] Horn, Roger A.; Johnson, Charles R. (1985). Matrix Analysis. Cambridge University Press. ISBN 0-521-38632-2", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to express our gratitude to all the Reviewers and Chairs for their dedication and effort. The majority of Reviewers consider accepting the work, all agree on comprehensive experimental analysis, and three Reviewers highlight the motivation behind the method (5j6M, VMJn, 6bmT). We have thoroughly reviewed the feedback and addressed all the raised concerns, what significantly enhanced the quality of our submission.  

We have now prepared a revised version of our work and are ready to engage in further discussions. Based on the comments received, we have performed additional experiments, which are presented in the rebuttal pdf file, and made the following changes to the revised version:  

* Improved introduction and method section to better explain the problem of task recency bias caused by dimensionality collapse in EFCIL, and why it is different than popular recency bias in the classification head (xJqZ, 6bmT, VMJn).  
* Added analysis of training and inference time complexity  of *AdaGauss* compared to other methods (xJqZ, 5j6M).  
* Tested our method in a half-dataset setting and added results to the Appendix (5j6M, 6bmT).  
* Added discussion of relevant works proposed by reviewers (6bmT, xJqZ).  
* To confirm that *AdaGauss* works with different backbone architectures, we have run experiments with ConvNext and ViT and added them to the Appendix (xJqZ).  
* Experimentally proved that the predicted distribution shift is different for different classes (xJqZ).
* To understand the impact of batch norm (BN) layers on our method we have performed experiments with frozen and removed BN layers. We have added a discussion to the Appendix (xJqZ).  
* Provided *AdaGauss* results for different feature adaptation networks (6bmT).  

We thank the Reviewers once again and look forward to discussing any other aspects of the paper that require further clarification.  
-- authors

Meta-review: After discussion, all reviewers lean towards acceptance for this work with two borderline scores. The reviewers recognized that the problem is significant, the proposed method to be straightforward and effective with strong experimental results. However, the reviewers still have some remaining concerns, including no new insights for the community and lacking some baselines that do not freeze the feature extractor. In my view, I tend to recommend acceptance for this work and the authors should add those baselines in the final version.
Decision: Accept (poster)