Elastic Feature Consolidation For Cold Start Exemplar-Free Incremental Learning
Authors: Simone Magistri
Tomaso Trinci
Albin Soutif
Joost van de Weijer
Andrew D. Bagdanov
Venue: ICLR 2024 poster (2024)
Keywords: Computer vision
continual learning
class-incremental learning
exemplar free
lifelong learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We define an Empirical Feature Matrix used as a pseudo-metric we use to both regularize feature drift and update class prototypes during exemplar-free class-incremental learning.

Abstract: Exemplar-Free Class Incremental Learning (EFCIL) aims to learn from a sequence of tasks without having access to previous task data. In this paper, we consider the challenging Cold Start scenario in which insufficient data is available in the first task to learn a high-quality backbone. This is especially challenging for EFCIL since it requires high plasticity, which results in feature drift which is difficult to compensate for in the exemplar-free setting.  To address this problem, we propose a simple and effective approach that consolidates feature representations by regularizing drift in directions highly relevant to previous tasks and employs prototypes to reduce task-recency bias. Our method, called Elastic Feature Consolidation (EFC), exploits a tractable second-order approximation of feature drift based on an Empirical Feature Matrix (EFM). The EFM induces a pseudo-metric in feature space which we use to regularize feature drift in important directions and to update Gaussian prototypes used in a novel asymmetric cross entropy loss which effectively balances prototype rehearsal with data from new tasks. Experimental results on CIFAR-100, Tiny-ImageNet, ImageNet-Subset and ImageNet-1K demonstrate that Elastic Feature Consolidation is better able to learn new tasks by maintaining model plasticity and significantly outperform the state-of-the-art.

Published as a conference paper at ICLR 2024
ELASTIC FEATURE CONSOLIDATION FOR COLD START EXEMPLAR-FRES INCREMENTAL LEARNING
Simone M Tomaso Trinci', Albin Soutif-Cormerais", Joost van de Weijer", Andrew D. Bagdanov'
Department of Information Engineering, University of Florence, Italy'
Computer Vision Center, Universitat Autònoma de Barcelona, Spain (simone magistri, tomaso. trincii, andrew www.::baddnooojunn.. @un wwwnfiiiiii...... (albin, joost levc. uab. cat
ABSTRACT
Exemplar-Free Class Incremental Learning (EFCIL) aims to learn from sequence of tasks without having access to previous task data. In this paper, we consider the challenging Cold Start scenario in which insufficient data is available in the first task to learn a high-quality backbone. This is especially challenging for EFCIL since it requires high plasticity, which results in feature drift which is difficult to compensate for in the exemplar-fre free setting. To address this problem, we propose a simple and effective approach that consolidates feature representations by regular- izing drift in directions highly relevant to previous tasks and employs prototypes reduce task-recency bias. Our method, called Elastic Feature Consolidation (EFC), exploits tractable second-order approximation of feature drift based on Empirical Feature Matrix (EFM). The EFM induces a pseudo-metric in feature space which we use to regularize feature drift in important directions and to update Gaussian prototypes used in novel asymmetric cross entropy loss which effec- tively balances prototype rehearsal with data from new tasks. Experimental results and ImageNet- demonstrate
on CIFAR-100, Tiny-ImageNet, ImageNett that Elastic Feature Consolidation is better able to learn new tasks by maintaining model plasticity and significantly outperform the state-of-thc-art
INTRODUCTION
Deep neural networks achieve tftte-of-theann performance on broad range of visual recognition problems. However, the traditional supervised learning paradigm is limited in that it presumes all training data for tasks is available in single training session. The goal of Class-Incrementts Learning (CIL.) is to enable incremental integration of new classification tasks into already-trained
models as they become available (Masana et al. 2022). Class-ineremental learning entails balancing model plasticity (to allow learning of new tasks) against model stability (to avoid catastrophic forgetting of previous tasks) (McCloskey & Cohen 1989). Exemplar-bar based approaches retain a small set of samples from previous tasks which replayed to avoid forgetting, while exemplar-fee methods retain no samples from previous tasks. This latter category is of particular interest when retaining exemplars is problematic due storage or privacy requirements.
Exemplar-free Class Incremental Learning (EFCIL) must mitigate catastrophic forgetting without recourse to stored samples from previous tasks. Approaches can be loosely grouped into those based on weight regularization and those based on functional regularization. Elastic Weight Consolidation (EWC) is elegant approach based on Laplace approximation of the previous-task posterior (Kirk) patrick et al. 2017). When training on a new task, an approximate Fisher Information Matrix is used to regularize parameter drift in directions of significant importance to previous tasks. This mitigates forgetting while maintaining more plasticity for learning the new task. EWC second order approach, and such needs aggressive Fisher Information Matrix approximations (Liu et al. 2018)
*Coresponding author
!Code to reproduce experiments is available https: http:thccmmmmmmmmm///lealaaa Feature_conace idaticnnnniiiiiiinnnooooo idationn

---

Published as a conference paper at ICLR 2024
Mastic Feature Consolidation
Empitiaal Feature Matrix reference frame B
Prototype updatte via Empinical Feature Matrit
Asymmetric Prototype Replay Loss
Ri
N
PIa-1
ligg bettsaiion
Wi-1
Claxsiiction
P
N
Wr
C.R-ACE
X
B 1
We-18
Pie-1
C CPB-ACE
(X)
(9)
Figure I: Elastic Feature Consolidation. (a) Architecture overview; (b) The Empirical Feature Matrix (EFM) measures how outputs vary with features and identifies important directions to mitigate forget- (c) The EFM induces pseudo- metric in feature space used to estimate prototype drift (Section 4.3 and (d) The Asymmetric Prototype Replay loss adapts previous task classifiers to
ting (Section 3.3
the changing backbone by balancing new-task data and Gaussian prototypes (Section 4.2
Functional regularization, especially through feature distillation, is a central component in recent state- of-the-art EFCIL approaches (Zhu et al. 2021b 2022 Yu et al. 2020 Toldo & Ozay 2022). Instead of regularizing weight drift, feature distillation regulanzes drift in feature space to mitigate forgetting. Feature distillation is often combined with class prototypes, either learned (Toldo & Ozay 2022)
or based on class means (Zhu et al. 2021b), to perform pseudo-rehearsal of features from previous tasks which reduces the task-recency bias common in EFCIL (Masana al., 2022). Prototypes differently than exemplars - offer a privacy-presers
way to mitigate forgetting, Feature distillation and prototype rehearsal reduce feature drift across tasks, but at the cost of plasticity.
Existing EFCIL methods are predominantly evaluated in Warm Start scenarios in which the first task contains larger number of classes than the rest, typically 50% or 40% of the entire dataset. For Warm Start scenarios] stability is more important than plasticity, since a good backbone can be already learned on the large first task. Current state-of ootte--f--he-rrt approaches use strong backbone regularization (Zhu et al. 20216) or indeed even freeze backbone after the large first task and focus on incrementally training the classifier (Petit al. 2023).
In this paper we consider EFCIL in the more challenging Cold Start scenario in which the first task is insufficiently large to learn a high-quality backbone and methods must be plastic and adapt their backbone to new tasks. EFCIL with Cold Starts faces two main challenges: an alternative feature distillation is required, since the backbone must adapt to new data, and an exemplar-free mechanism needed to adapt previous-task classifiers to the changing backbone.
We propose a novel EFCIL approach, which we call Elastic Feature Consolidation (EFC), that regularizes changes in directions in feature space most relevant for previously- tasks and allows more plasticity in other directions. A main contribution of this work is the derivation of a pseudo-metric in feature space that is induced by matrix we call the Empirical Feature Matrix (EFM). In contrast to the Fisher Information Matrix, the EFM can be easily stored and calculated since does not depend on the number of model parameters, but only on feature space dimensionality. To address drift of the more plastic backbone, we additionally propose an Asymmetric Prototype Replay loss (PR-ACE) that balances between new-task data and Gaussian prototypes during EFCIL. Finally, we propose an improved method to update the class prototypes which exploits the already-computed EFM. A visual overview of EFC and its components is given in Figure]] [
2 RELATED WORK
We focus on offline, class-incremental learning problems. Class-incremental problems are distin- guished from task-incremental problems in that no task information is available at test time. Offline incremental learning allows multiple passes on task datasets during training (Rebuffi et al. 2017]
Wu et al. 2019; Masana et al. 2022), while online incremental learning considers continuous data
'We define Warm Start, differently than Ash & Adams 2020 as continual learning with large initial task, and Cold Start, consistent with Hu al. [021] for continual learning beginning with a random network initialization.
2

---

Published as a conference paper at ICLR 2024
streams in which each sample is accessed just once during training (Lopez- Paz & Ranzato 2017 Mai et al. 2022 Souttif-Crmeena et al. (023).
Exemplar-based approaches rely on a memory of previous class samples that are replayed while learning new tasks. Hou et al.) (2019); Douillard et al. (2020); Kang et al. (2022) combine cosine normalization and distillation losses with exemplars to mitigate forgetting. Another category of exemplar-based approaches focuses on expanding sub-network structures across the incremental steps. Liu et al. (22I)) adds two residual blocks to mask layers and balance stability and plasticity. Yan et al. (2021) train a single backbone for each incremental task, and more recently Zhou al. (2022) share generalized residual blocks and extend specialized blocks for new tasks improve performance and efficiency. The main drawbacks of exemplar-ban based approaches are the lack of privacy preservation and high computational and storage costs, particularly when growing memory buffer is used.
Exemplar-free approaches are less common in class-incremental learning. Early works computed importance scores for all weights and used them to perform weight regularization (Kirkpatrick et al. 2017 [iu al. 2018 Ritter et al. 2018). Other works like Li & Hoiem (2017); (2017);Uung Jung al. (2016) use functional regularization, such as knowledge or feature distillation, constraining network activations to match those of previously-train trained network. Most functional regularization approaches use feature distillation (Zhu et al.) 2021a| Yu et al.] 2020), or even freeze the feature extractor after the first task (Petit et al. 2023 Belouadah Popescu 2018 8 Panos et al. 2023), greatly reducing the plasticity of the network. For this reason, they consider Warm Start scenarios in which the first task is much larger that the others, so that reducing plasticity after the first task has less impact on the performance.
On top of this decrease in plasticity, only using functional regularization is not enough to learn new boundaries between classes from previous tasks and classes from new ones. For this reason regularization is often combined with prototype rehearsal Zhu et al. (2021b) 2022);; Smith et al. (202T). Prototypes are feature space statistics (typically class means) used reinforce the boundaries of previous-taane classes without the need to preserve exemplars. Zhu et al. 2202 a use prototype augmentation and self-supervised learning learn transferable features for future tasks. [oldo & Ozay (2022) learns and updates prototype representations during incremental learning. Zhu et al. (2022) combine prototypes with a strategy to re-organize network structure to transfer invariant knowledge across the tasks. |Petit et al. (2023) fixes the feature extractor after the training the large first task and generates pecudo-samples to train a linear model discriminating all seen classes.
We propose an EFCIL approach that uses functional regularization and prototype rehearsal, but in contrast with most tfateoof-hhearr methods, we also report results in Cold Start incremental learning scenarios that do not start with large first task.
EFCIL REGULARIZATION VIA THE EMPIRICAL FEATURE MATRIX
In this section we introduce the general EFCIL framework and then derive a novel pseudo-metric in feature space, induced by a positive semi-definite matrix we call the Empirical Feature Matrix (EFM), used as regularizer control feature drift during glass-incremental learning (see Figure Ip).
3.1 EXEMPLAR-FREE CLASS-INCREMENTI LEARNING (EFCIL)
During class-incremental learning model M is sequentially trained on K tasks, each characterized by disjoint set of classes {C, 1 jt=1. where each C, is the set of labels associated to the task t. We denote the incremental dataset as = (X, Vr ) where X; and Vi are, respectively, the set of samples and the set of labels for task t. The incremental model M, at task consists of feature extraction backbone fe(:; 0c) shared across all tasks and whose parameters 0, are updated during training, and classifier W2 " |c,| which grows with each new task. The model output at task / the composition of feature extractor and classifier:
Me 0g W.) p(y 0g W.) = wW. fx(x;))))
(1)
Directly training M, (i.e. fine-tuning) with a cross-entropy loss at each task t results in backbone drift because at task t we see no examples from previous-tas, classes. This progressively invalidates the rrvikus-task classifiers and results in forgetting. In addition to the cross-entropy loss, many state-of-the-as EFCIL approaches use a regularization loss C to reduce backbone drift, and a prototype loss L,xh to adapt previous-task classifiers new features.
3

---

Published as a conference paper at ICLR 2024
3.2 WEIGHT REGULARIZATION AND FEATURE DISTILLATION
Kirkpatrick et al. (2117) showed that using l regularization to constrain weight drift reduces Torgetting but does not leave enough plasticity for learning of new tasks. Hence, they proposed Elastic Weight Consolidation (EWC), which relaxes the &2 constraint with a quadratic constraint based on diagonal upproximation of the Empirical Fisher Information Matrix (E-FIM):
= Eew Bymppeelu:07) {
(alogg 00;
a llgg 30;
where 02 is the model trained after task 4. The E-FIM induces pseudo-metric in parameter space (Amari 1998) encouraging parameters to stay low-error region for previous-task models:
CE-FIM (O. 1T-
1
The main limitation of approaches of this type is need for approximations of the E-FIM (e.g. diagonal assumption). This makes the computation of the E-FIM tractable, but in practice unrealistic since does not consider off- off-diagonal entries that represent influences of interactions between weights on the log- og-likelihood. To overcome these limitations, more recent EFCIL approaches (Zhu et al. rely on feature distillation (FD), which scale better in the number of parameters:
2021b 2022)
Σ [1f((z) - fc-1(x)) SEN
The isotropic regularizer using the &2 distance too harsh a constraint for learning new tasks. We propose to use a pseudo-metric in feature space, induced by matrix which we call the Empirical Feature Matrix (EFM), constraining directions in feature space most important for previous tasks, while allowing more plasticity in other directions when learning new tasks.
3.3 THE EMPIRICAL FEATURE MATRIX
Our goal is to regularize feature drift, but we do not want to do so isotropically as in feature distillation. Instead, we draw inspiration from how the E-FIM identifies important directions in parameter space and take a similar approach in feature space.
The local and empirical feature matrices. Let fe(x) be the feature vector extracted from input Xg from task and p(y) = (x);; W.) the discrete probability distribution over all the classes seen so far. We define the local feature matrix as:
( (ologr(u) 0f((z) Es(()) = Eyrptw
0f((z) (
(
a log p(y)))) (logp())) )
(5)
The Empirical Feature Matrix (EFM) associated with task is obtained by taking the expected value of Eq. olver the entire dataset at task t
E. = E=
(6)
Both E and Eg are symmetric, as they are weighted sums of symmetric matrices, and positive semi-definite. Naively computing (f(()) requires complete forward pass and backward pass up to the feature embedding, however we exploit an analytic formulation that avoids computing gradients:
Ef((a) = Ey-p(u) [Wi(Lm - (W, 1,
where m = Ei= [C,l, Im the identity matrix of dimension m x m, and P is a matrix built from the row of softmax outputs associated with fz( replicated m times (see Appendix B for a derivation).
To gain additional insight into what this matrix measures, we highlight some (information) geometrical aspects of E,. The Empirical Fisher Information Matrix provides information about the information geometry of parameter space. In particular, it can be obtained as the second derivative of the KL- divergence for small weight perturbations (Martens 2020; Grementieri & Fioresi 2022). Similarly, the Empirical Feature Matrix provides information about the information geometry of feature space. Since E' fz() is positive semi-definite, we can interpret as a pseudo-metric in feature space and use it to understand which perturbations of features most affect the predicted probability:
(L fe (2; 0.) + 5; Wt) || p(u [x( 0z); W.)) 8 F O(16119).
(8)
4

---

Published as a conference paper at ICLR 2024
0=1
d=
Per dass Prob. Variaton
0.20
0.20
0.15
0.10
X1
0.05
0.05
0.3
12
0.00
0.00
0.2
Class ID
Class
Figure 2: The regularizing effects of F, on Cold Start CIFAR-100 10 step (see Section 5.1 for more details on the dataset settings). Left: Perturbing features principal directions of E results in significant changes in classifier outputs (in blue), while perturbations in non-principal directions leave the outputs unchanged (in red). Right: If we continue incremental learning up through task and perturb features from all three tasks in the principal (solid lines) and non-principal (dashed lines) directions of Eg, we see that Ey captures all important directions in feature space up through task 3.
The Empirical Feature Matrix loss. We now have all the ingredients for regularization loss:
CEM
- -f 2E
E
Ee + (Nrr) - (x)()))
where fe(z) and (z) E IR are the features of sample extracted from the current model Me and the model M trained on the previous task, respectively. E
[we employ a damping term 7) R to constrain features to stay in region where the quadratic second-order approximation the log-likelihood in Eq, S remains valid (Martens Sutskever, 2012). must ensure
of the KL-divergence
However, to ensure that regularizer does not degenerate into feature distillation, that ΛgPM/4 > where μ4 are the non-zero eigenvalues of the EFM. Additional analysis these constraints and the spectrum of the EFM are given in Appendix B.2 In the next section we give empirical evidence of the effect of this elastic regularization.
Regularizing effect of the EFM. By definition, the directions of the eigenvectors of the EFM associated with strictly positive eigenvalues are those in which perturbations of the features have the most significant impact on predictions. To empirically verify this behavior, we perturb features in these principal directions and measure the resulting variation in probability outputs. Let fr = fx) IR" denote feature vector extracted from and E, the EFM computed after training on task t. We define the perturbation vector Ei.A E I such that each of the first entries is sampled from a Gaussian distribution (N00 or. where k is the number of strictly positive eigenvalues of the spectral decomposition of E,. The remaining n-k entries are set to zero. Then, letting be the matrix whose columns are the eigenvectors of E,, we compute the perturbed features vector fe as follows:
j. = + U, E1sk-
(10)
In Figure 2 (left) we show that perturbing the feature space along the principal directions in this way, after training on the first task, results in a substantial variation in the probabilities for each class. Conversely, applying the complementary perturbation 6kun i.e., setting zero on the first directions and applying Gaussian noise the remaining ni-k directions, has impact on the output. In Figure (right) we see that, after training three tasks, perturbing in the principal directions Es degrades performance across all tasks, while perturbations in non- non-principal directions continue to have no affect on the accuracy. This shows that Eg captures the important feature directions for all previous tasks and that regularizing drift in these directions should mitigate forgetting.
PROTOTYPE REHEARSAL FOR ELASTIC FEATURE CONSOLIDATION
In this section first describe prototype rehearsal and then describe our proposed Asymmetric Prototype Rehearsal strategy (see Figure It). We also show how the Empirical Feature Matrix Eg defined above can be used to guide classifier drift compensation (see Figure Ic).
4.1 PROTOTYPE REHEARSAL FOR EXEMPLAR-
C.A.S--ICCREMENTAL LEARNING
EFCIL suffers from the fact that the final classifier is never jointly trained the all seen classes. At each task t, the classifier W is extended to include [C, new outputs. Thus, if classifier W, is only trained samples from the current classes C,, it becomes incapable of discriminating classes
5

---

Published as a conference paper at ICLR 2024
c + 4
EPP +
40
10
10
10
10
Step
Step
Stop
Figure 3: Accuracy after each incremental step on the Cold Start CIFAR-100 10-step scenario. Feature distillation with symmetric prototype loss (left) reduces forgetring at the cost of plasticity and new tasks are not learned. EFM regularization with symmetric loss (middle) increases plasticity at the cost of stability and previous tasks are forgotten. EFM regularization with asymmetric PR-ACE loss (right), balances current task data with prototypes and achieves better plasticity/stability trade-off.
from previous tasks. leading inter-task confusion. Additionally, the classifier is biased towards the newest classes, resulting in task rask-re bias (Masana et al. 2022).
A common exemplar-free solution is to replay only prototypes of previous classes (Zhu et al. 2022 2021b Petit et al. 2023 Toldo Ozay 2022). At the end of task 1, the prototypes pi-1 are computed as the class means of deep features and the set of prototypes Pi:-1 is stored.
During training of the next task the prototypes are perturbed and fed to the classification heads. Let (p. ya) ~ Pi:t-1 be a batch of perturbed prototypes and (æt, Ut) ~ (X,
batch of current task data. The final classification loss, which we call the symmetric loss to distinguish it from the asymmetric loss we propose below, consists of two cross entropy losses evaluated on the all classification heads:
sym (xt, ve)) C... + Apr L. (p. Up)lc,
(11)
where (x,, Ut) Ct. represents the cross-entropy loss computed over all encountered classes CJ.t- In our work, we use Gaussian prototypes to enhance the feature class means, as suggested by Zhu al. (02 To be more specific, we create augmented prototypes by sampling from a Gaussian distribution N M(pf_, 29). Here, E represents the class covariance matrix of feature representations.
4.2 ASYMMETRIC PROTOTYPE REHEARSAL
C.ym from Eq. 11 above is able to balance old task classifiers with new ones only when feature space drift is small, for instance when using feature distillation (see Figure But with the greater plasticity introduced by the EFM, the symmetric loss fails to effectively adapt old task classifiers to the changing backbone (see Figure 3 (middle)). To address this, propose an Asymmetric Prototype Replay loss (PR-ACE) to balance current task data and prototypes:
(IL, CPR-ACE (I (I Wlle, + C. yp) U (2,, 9:) Ie.).
(12)
where (Ie, yt), (fr,, Ur) ~ (1) and p- Pi:--1 with yp the associated label. (r, y)l is the cross entropy loss restricted to classes in C². The first term relies on the current task data, processed through the backbone and the current task classification head. It aligns the initially random head with the feature representation learned during previous tasks and aids learning of task-specific features. The second term calibrates the classification heads. It combines a batch of current task data with batch prototypes. Prototypes and current task data are uniformly sampled from all classes and used to train all task classifiers; consequently larger proportion of prototypes is used compared to the current task data. In Figure 3 (right) we see that the addition of the PR-ACE loss achieves just this balance between stability and plasticity.
PR-ACE takes inspiration from the asymmetric loss proposed by Caccia et al. (0221) for online, ecemplar-based class incremental learning. In their work, the goal was to learn inter-task features with loss term based on both exemplars and current task data passed through the current backbone. In contrast, for prototype rehearsal only current task data may be passed through the backbone.
4.3 PROTOTYPE DRIFT COMPENSATION VIA EFM
Fixed prototypes can become inaccurate over time due to drift from previous class representations Yu et al. (2020) showed that the drift in embedding networks can be estimated using current task data.

---

Published as a conference paper at ICLR 2024
They suggest estimating the drift of class means from previous-tasks by looking feature drift in closely related classes of current task. After each task, they update the prior normalized class means via weighted average of the current task feature representation drift. These updated class means then used for nearest class-mean classification at test time.
across the entire training phase. As proposed by SDC, after each task we update the prototypes as follows:
Our aim is to update prototypes pi-1 used in for rehearsal in PR-ACE (Eq. 12
Pi- -i-1 Enixx, w;
8/-1= fe(xx) (
(13)
Exiox, W/
where of-1 is the feature drift of samples z; between task t 1 and t.
We define the weights w; using the EFM (Figure Db). In Section 3.3 we showed that the EFM induces a pseudo-metric in feature space, providing a second-order approximation of the KL-divergence due to feature drift. We extend the idea of feature drift estimation to softmax- based classifiers by weighting the overall drift of our class prototypes. Writing p" = p and fi fi-1 (2+) to simplify notation, our weights defined as:
(fir pd) (E) ) exp
KL(p(y)ff Wan) plypp :(ol)=:: Wi.)))
wy= =exp exp
(14)
20
202
where E111 is the EFM after training task t 1. Eq. 14 assigns higher weights to the samples more closely aligned to prototypes in terms of probability distribution. Specifically, higher weights are assigned to samples whose softmax prediction matches that of the prototypes, indicating strong similarity for the classifier. See Appendix C for more details.
4.4 ELASTIC FEATURE CONSOLIDATION WITH ASYMEETIIC PROTOTYPE REHEARSAL Figure summarizes our approach, which a combination of the EFM and PR-ACE losses:
EEFC CEPM
C.R-ACE
(15)
where L.R-ACE is the asymmetric cross entropy loss (Eq. 12) and LEPM is the Empirical Feature Matrix loss from (Eq. (9) After each task, our prototypes are updated using (Fq. [3) equipped with the EFM (Eq. [4] and new prototypes with the corresponding class covariances are computed. Finally, we compute the EFM E, using the current task data for use when learning subsequent incremental tasks. In Appendix
D we provide the pseudocode of the overall training procedure.
Note that it is essential to combine _ with CPR-ACE, E--1 selectively inhibits drift important feature directions, in contrast to Feature Distillation which inhibits drift in all directions (Figure 3 (left)). The resulting plasticity increases the risk of prototype drift, which in turn adapts previous-tas classifiers drifted prototypes (Figure 3 (middle)). Our use of C.R-ACE to balance prototypes and new task samples, and our prototype drift compensation, counters this effect (Figure 3 (right)).
5 EXPERIMENTAL RESULTS
In this section we compare Elastic Feature Consolidation with the state-of-the-arr in EFCIL.
5.1 DATASETS, METRICS, AND HYPERPARAMETERS
We consider three standard datasets: CIFAR-100 Krizhevsky et al.) 2009). 20n9).Tiny-aae....
(Wu]
ctt al., 2017) and ImageNet-Subset (Deng et al. 2009). Each is evaluated in two settings. The first the Warm Start (WS) scenario commonly considered in EFCIL Zhu et al. (2021b) 2022);
Petit et al. (2023): Toldo & Ozay (2022) which uses a larger first task consisting of 50 classes for CIFAR-100 and Seuueeett in the 10-step scenario, and 40 classes in the 20-step scenario. For Tiny-ImageNer, both the 10-step and 20-step scenarios use large first task consisting of 100 classes. Regardless of 10-step or 20-step, the remaining classes after the first task are uniformly distributed among the subsequent tasks. The second setting, referred to as Cold Start (CS), uniformly distributes all classes among all tasks. We are mostly interested in the Cold Start scenario since it requires backbone plasticity and is very challenging for EFCI.. In Appendix GG we report the performance on full ImageNet-1 1K dataset. See Appendix for more dataset details. We use as metries the per-step incremental accuracy A and average incremental accuracy AKK
Σ 1 Σx|
K Σ i=1
AK sep
AK
Aleep
(16)
[C,
K
7

---

Published as a conference paper at ICLR 2024
Warm Start
Cold Start
A
step
Method
Stop
20 Step
10 Step
20
Step
10 Step
Step
10 Step
Step
EWC
21.08 + 1.00
13.53
+ 1.11
41.00
+ 1.11
31.79
2.77
31.17 + 2.94
17
2.43
49.14
1,28
31.02
1.15
CIFAR- 100
LwF
20.73
1.47
11.78
0.00
1.30
1.62
32
3.08
53.91
166
38.39
1.05
PASS
53.42
0.48
47.51
0.37
63.12
0.69
59.55
0.97
30.15 1 1.01
17:41
= 0.09
47.86
d 1.93
32.86
1.03
Fusion
56.86
51.75
65.10
61.60
FelriL
56.79
0.40
52.61
0.81
0:86
1.0
34.94
0:46
23.28
1:24
51:20
1:13
38.
1.07
SSRE
37.48:0.55
52.98
1: 0.63
63.78
1 0.59
03.11
0.81
30.10
2: 0.71
17.52
= 0.80
17.20
1: 1.91
32.15
1.07
EFC
69.87
+: 0.39
55.78
68.23
0:68
65,90
0.97
43.62
0.70
32.15
1.35
58.58
4: 0.91
47.36
1.37
EWC
6.73 0.44
5.96 + 1:17
18.18
0.60
13.71
0.52
8.00 + 0:27
5.16
0.54
21.01
0.51
15.70
0.35
Tiny- Mor-AmggeNee
LWF
24.00 4: 1.44
7.58
4: 0.37
43.15 1 1.02
0.64 22.89
26.09 4: 1:29
15.02
0.67
45.14
4: 0.58
32.94
1: 0.54
0.61
35.01
+: 0.39
0.31
16.65
21.11
18.73
1.43
39.25
32.01 1.08
Fusion
46.92
44.61
PeTvil.
45.71 + 0.30
44.63
+ 0.49
13.90 + 0.42
02.96
0.45
30.97
0.90
25.
0.61
45.00
1.67
31.14
+ 1.11
0.45
14.68
0.36
63.27
0:43
0.42 62.94
222
0.95
1.06
1.99 38.82
30,62
1.96
EFC
$0.40
1:0.5
48.68
1: 0.65
$7.52 #: 0:40
56.52
:: 0.53
34,10
0.77
28.69
0.40
47.95
0.61
42.07
0.96
2.48
10.
1.74
23
2:01
06.05:1
1:10
24.69
4:13
12.78
1.95
39.40
3.06
26.95
1.02
LWF
21.89 1: 0.52
19.24
+: 1.61
37.15
1: 2:47
25.96
= 01.95
97.71
4: 2.53
18.64
1.67
56.41
1: 1.03
40.23
0.43
PASS
52.04 +: 1,06
44.01
+ 2.19
65.14
+ 0.36
58.88
+ 2.15
26.40
+: 1.30
1.22 14.38
45.74
4: 0.18
31.65
0.42
Fusion
60:20
61
63:70
FeTiLL
63 63.56
+: 0.59
57.62
+: 1.13
71.87
+: 1:46
08.01
= 1.60
36.17 +: 1.18
26.63
= 1.45
52.63 4: 0.56
12.43
2.05
SSRE
G1 11.8450
+ 0.93
56.
+ 0.97
70.08
+ 1.37
06,
73.73-22
1.61
25, + 1.17
16.25
1.00
43. 76
# 1.07
31.15
1.53
EFC
0.58
62.17
0.68 75.40
0:92
71.63
1.13
47.38
1.43
35.75
1.74
59.94
1.35 49.92
2.05
Table 1: Comparison with the sfate-of-the-ard CIFAR-100, Tiny-ImageNet, and ImageNet-Subsett Fusion results are those reported in (Toldo & Ozay 2022) as no code is provided to reproduce them.
where ax represents the accuracy of task after training task K.
Hyperparameter settings. We use the standard Re (He et al. 2015) trained from scratch for all experiments. We train the first task of each state-of-ther fi--the-art method using the same optimizer, number of epochs and data augmentation. In particular, we train all the first task using self-rotation as performed by Zhu et al. (2021b); Toldo & Ozay (2022) to align all the performance. In Appendix we provide optimization settings for the first tasks.
For the incremental steps of EFC we used Adam with weight decay of 2 and fixed learning rate of 1f for Tmay-ImageNet and CIFAR- 100, while for ImageNet-SSubset we use a learning rate of 1 for the backbone and I for the heads. We fixed the total number of epochs to 100 and use a batch size of 64. We set λEFM 10 and 1) 0.1 in Eq for all the experiments. We ran all approaches using five random seeds and shuffling the classes in order to reduce the bias induced by the choice of class ordering (Rebuffi et al.] [017]] Masana et al. 22022.. In Appendix E we provide the optimization settings for each ofate-of-the method we evaluated.
5.2 COMPARISON WITH THE STATE-OF-THE-ART
In Table 1 we compare EFC with baselines and state-of-the-art EFCIL approaches. Specifically we consider EWC (Kirkpatrick al., 2017), LwF (Li & Hoiem| 2017). PASS (Zhu et al. (021b), Fusion (Toldo & Ozay] 2022), FeTrIL Petit et al. 2023) and SSRE (Zuu et al. 2022). Our evaluation considers two key scenarios: Warm Start, commonly found in the EFCIL literature, and the more difficult Cold Start scenario. EFC significantly outperforms the previous sfate-of-the-ard in both metrics across all scenarios of the considered datasets.
Notably, on the ImageNer- Warm Start scenario EFC exhibits a substantial improvement of about 5% over FeTriL in both per-step and average incremental accuracy. For SSRE, our incremental accuracy results are significantly higher compared to the results reported in the original paper. This is attributable the self-rotation we used as initial task to align all results with PASS and Fusion (which both use self-rotation.. This alignment is crucial when dealing with Warm Start scenarios, as performance on the large first task heavily biases the metries (Eq. 16) as discussed by Zhou et al.) (2023); Castro et al. (2018). In Appendix J and K we provide per-step performance plots in all the analyzed scenarios, showing that all the evaluated methods begin from the same starting point.
In Cold Start scenarios we see that methods relying on feature distillation, such as FeTrIL, SSRE and PASS, experience significant decrease in accuracy compared to EFC. This drop in accuracy can be attributed to the fact that the first task does not provide a sufficiently strong starting point for the entire inas-incrementll process. Moreover, most of these approaches exhibit weaker performance even when compared to LwF, which does not use prototypes to balance the classifiers. On the contrary, the plasticity offered by the EFM allows Elastic Feature Consolidation achieve excellent performance
8

---

Published as a conference paper at ICLR 2024
Warm Start
Cold Start
K
K
K
Variant
Step
20
Step
10 Step
20 Step
10 Step
20 Step
10 Step
20 Step
Each Task 49.00 48
47.26
18
$6,31
10.51
55.06 10.39
34,81
+0. 29.39
10.52
47.84
0.33
42.53 10.17
Last Task 10.43
45.64
1051
55.85
1.0.4
54.12 10.37
34.36
10.14
28.57
10.32
47.50
0.24
42.11 10.11
EFC
50.40 +0.25 48.68
=0.65
$7.52
0.43 56.52 +0.53
34.10
#0.77
28.69 +0.40
47.95
=0.61 42.07 +0.96
Table 2: Mitigating storage costs (on Tiny-ImageNet), Each Task stores a single covariance per task, while Last Task stores a single covariance from the most recent task (see Appendix !
Table A5).
even in this setting, surpassing all other approaches by a significant margin. In Appendix G we provide results comparing EFC with FeTrIL on ImageNet-IK.
5.3 ABLATION STUDY AND STORAGE COSTS
In this section we report the ablation study and the storage costs of EFC on CIFAR-100. A more detailed analysis on other dataset is reported Appendices H and
Ablation on PR-ACE and Feature Distillation. We evaluated the performance of our regularizer in two settings: when combined with the proposed PR-ACE Eq. 12 and when combined with the standard symmetric loss Eq. [1] Figure A clearly demonstrates that the symmetric loss works well in conjunction with feature distillation. However, when used alongside the EFM, it fails to effectively control the adaptation of the old task classifier, resulting in an overall drop in performance. Additionally. it is evident from the results that feature distillation essentially degenerates into a method similar to Fe'TrLL, which freezes the backbone after the first task. These findings suggest that using feature distillation as a regularizer is equivalent to freezing the backbone after the first task and employing suitable prototype rehearsal approach to balance the task classifier, pseudo-samples for FeTril., and Gaussian prototypes in our specific scenario.
CIFAR-100 (CS) - 20 Step
Ablation on prototype update. In Figure 4p we evalu- ate the effect of prototype updates (Eq. 13 and Eq. 14] These results indicate that prototype updates enhance per- formance in both the Warm and the Cold Start experiments. The performance boost is more pronounced in the Cold Start scenario, where we achieve and 5 points of im- provement for the 10 and 20 steps respectively.
Gu + Ci
Per-step Accuracy
2 Cu PITTT
60
10-
Storage costs. Storing class covariance matrices results in a linear increase in storage as the number of classes grows. This storage cost is easily mitigated, however, without sac- rificing state-of-the-ee accuracy. In Table we consider the use of task-specific covariance matrix which comes with linear memory increase in the number of tasks, and using single covariance from the most recent task for prototype generation which results in constant memory overhead. In Appendix []] we provide results an analysis of using low-rank approximations of all covariance matrices which does not sacrifice EFCII. performance.
20-
1
5
7
11
13
15
17
19
Step
(a) Ablation regularization and prototype losses.
Update Proto (WS)
Update Proto (CS)
Step
X
10
58.24
60.87
39.28
43.62
20
52.60
55.78
27.28
32.15
CONCLUSIONS AND LIMITATIONS
(b) Ablation on prototype update.
Cold Start EFCIL requires careful balance between model stability and plasticity. In this paper we proposed Elastic Feature Consolidation which regularizes feature drift in directions in feature space important to previous-task classifiers. We derive an Empirical Feature Matrix that induces pseudo-metric used to control feature drift and improves plasticity with respect to feature distillation. We further use this Empirical Feature Matrix to constrain prototype updates in an Asymmetric Prototype Replay loss. Our results show that EFC significantly outperforms the state-of-the-art on both Cold Start and Warm Start EFCIL. EFC updates class means based on current class sample drift, but stored means could be distant from real class means, potentially leading to forgetting in long task streams. Estimating covariance drift is still open question. EFC needs storage that grows linearly with classes number. This growth can be mitigated using approximations or proxies for class covariance matrices (see Appendix [].
Figure 4: Ablations (a) losses and (b)
prototype update for CIFAR-100.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
We ran all experiments five times with different seeds and report the average and standard deviations in order to guarantee that reported results are reproducible and not due random fluctuations in performance. We report all hyperparameters used to reproduce our results in Section 5.1 and Appendix E2.2 for Elastic Feature Consolidation and all methods we compare to. Source code to reproduce our experimental results is public released.
ACKNOWLEDGEMENTS
This work was supported by funding by the European Commission Horizon 2020 grant #951911 (A14Media), project and 1322021-132213-111
and PID2022-1 (DD2022-14225711 100---14277NB--000 funded by MCIN/AEI/ 10. 10039/500 10001 1033, by the Italian national Cluster Project CTN01 00034 23154 "Social Museum Smart Tourism", by the European Union project NextGenerationEE//PRRR, and by the Spanish Development Fund FEDER.
REFERENCES
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 102):251-276. 1998.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884 3894, 2020.
Eden Belouadah and Adrian Popescu. Deesil: Deep-shallow incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky.
New insights on reducing abrupt representation change in online continual learning, In International Conference on Learning Representations, 2021.
Francisco M Castro, Manuel Marín-liménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pp. 86-102. Springer, 2020.
Luca Grementieri and Rita Fioresi. Model-centric data manifold: the data through the eyes of the model. SIAM Journal on Imaging Sciences, 140--1166 2022.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 831-839, 2019.
Huiyi Hu, Ang Li, Daniele Calandriello, and Dilan Gorur. One pass imagenet. NeurIPS 2021 Workshop ImageNet: Past, Present, and Future, 2021.
Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-fogge learning in deep neural networks. arXiv preprint 1r7tv.1607.00122. 2016.
Minsoo Kang, Jacyoo Park, and Bohyung Han. Class-increnn incremental learning by knowledge distillation with adaptive feature consolidation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16071-16080. 2022.
10

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barkkkkaaaaiikaa,,aaa et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2447 2017.
Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting, In 2018 24th International Conference on Puttern Recognition (ICPR). pp. 2262-2268. IEEE, 2018.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. Adaptive aggregation networks for class-incremental learning. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pp. 2544 2553,, 2021.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28-51, 2022.
James Martens. New insights and perspectives the natural gradient method. The Journal of Machine Learning Research, 21(1))5776-558 58511, 2020.
James Martens and Ilya Sutskever. Training deep and recurrent networks with hessian-free op- timization. In Neural Networks: Tricks of the Trade; Second Edition, pp. 479-535. Springer, 2012.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E. Turner. First session adaptation: A strong replay-frre bascline for class- faas-inerrmmnntl learning. In International Conference on Computer Vision (ICCV), 2023.
Grégoire Petit, Adrian Popescu, Hugo Schindler, David Picard, and Bertrand Delezoide. Fetril: Feature translation for exemplar-free class-incremental learning. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision pp. 391 1-3900,, 2023.
Sylvestre-Allise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. Advances in Neural Information Processing Systems, 31, 2018.
James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: new approach for data-free class-incremen incremental learning. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 9374-9384 2021.

---

Published as a conference paper at ICLR 2024
Albin Coutif-Cormerais Antonio Carta, Andrea Cossu, Julio Hurtado, Vincenzo Lomonaco, Joost Van de Weijer, and Hamed Hemati. A comprehensive empirical evaluation on online continual learning. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 3518-3528, 2023.
Marco Toldo and Mete Ozay. Bring evanescent representations to life in lifelong class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16732-16741, 2022.
Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny imagenet challenge. Technical report, 2017.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 374 382, 2019.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014-3023, 2021.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. Semantic drift compensation for class caass-incrementll learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6982- 6991, 2020.
Wei Zhou, Qi- Wang, Han-Jia Ye, and De-Chuan Zhan. A model 603 exemplars: To- wards eeemory-efficiee inceemmaatttt learning. In The Eleventh International Conference on Learning Representations, 2022.
Da- Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep inasemaacrmental learning: A survey. arXiv preprint arXiv:2302.03648, 2023.
Fei Zhu, Zhen Cheng, Xu-yao Zhang, and Cheng- lin Liu. Clase-nneremental learning via dual augmentation. Advances Neural Information Processing Systems, 34: 143306114388 2021a.
Fei Zhu, Xu-Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-supervision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5871-588188 58800 2021b.
Kai Zhu, Wei Zhai, Yang Cao, Jiebo Luo, and Zheng-Jun Zha. Self- Self-sustaining representation expan- sion for non-exemplar class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9296-9305, 2022.
12

---

Published as a conference paper at ICLR 2024
Appendices
A SCOPE AND SUMMARY OF NOTATION
In these appendices we provide additional information, experimental results, and analyses that complement the main paper. For convenience, we summarize here the notation used the paper:
M, The incremental model at task i.
K: The total number of tasks.
C₁ The set of classes associated with task t.
D: The incremental dataset containing samples X; and labels V for each task t.
fa(:; 0,): The feature vector computed by the backbone at task with parameters O,.
n: The dimensionality of the feature space.
Wy: The classifier task with dimensions 1 E= IC
A hyperparameter associated to the Empirical Feature Matrix, Eq. ii the paper.
7y: damping hyperparameter associated to the Identity Matrix, Eq. in the paper.
we derive an analytic form for computing the Empirical Feature Matrix and analyze how its rank evolves during incremental learning. In Appendix C we provide additional analysis on using the EFM to update prototypes. In Appendix[D [D] we present the training loop employed in our method. In Appendix we provide additional information about the dataset and the implementation of the approaches method on a short task sequence. Appendix G. we compare the results in both Warm and Cold Start scenarios of EFC against FeTrIL on ImageNet-IK. In Appendix H extend the ablation study presented in Section S.3 on the main paper. In Appendix we analyze the storage and computation costs of EFC and propose techniques to mitigate them. Finally, in Appendix ! and Appendix K we give per-step plots of accuracy on all datasets and scenarios.
In Appendix B
tested. In Appendix [F) we provide comprehensive evaluation of our proposed
B
THE EMPIRICAL FEATURE MATRIX
B. ANALYTIC FORMULATION FOR COMPUTING THE EMPIRICAL FEATURE MATRIX
In this section we derive an analytic formulation of the local features matrix defined in the main paper Section 3.3 simplify the notation, we recall its definition without explicitly specifying the task, as it is unnecessary for the derivation:
(
i @ p(u)
(Dlgg(u))
(
a (logp(u)
T
]
Ef(z) = Ey-p(v)
(17)
af(x)
0f(x)
We begin by computing the Jacobian matrix with respect to the feature space of the output function g 1 m which is the log-likelihood of the model on logits z:
((2)
log log(softmax(e)) (2) (og((r(2)) log (m T
where
a1(2) = Em
E== ezi
The partial derivatives of y with respect to each input 2; are:
3 Olog(o
1 (z(2)
ifi= j.
(18)
Ozj
- 0z(2) otherwise.
13

---

Published as a conference paper at ICLR 2024
Thus, the Jacobian matrix of g is:
1101((2)
-02(2)
(2))
-01
1 02(2)
[(z)
(19)
- (z)
1 - 0m (z)
[1)()) T
mm 1m
Im P.
(20)
(m)
where In E IR is the identity matrix and 1m represent the vector with all entries equal to 1.
Recalling Eq. 17 we are interested in computing the derivatives of the log- bog-probability vector with respect to the feature vector:
i logg .... Um If: W)
f(W ((lo((softmax2))
(21)
Of
Of
Oz
J(z).
(22)
where: = W for W the classifier weight matrix mapping features f to logits 2. Combining this with Eq. 20 we have:
( (logp(e)
(Dlogp(e) ) (
F Eyrp(e) (W
E Byspis)
(23)
0f
-a
0f
where p(y) (x)) W), P is the matrix containing the probability vector associated with ʃ in each row, and (Im p)y is the column vector containing the YIA row of the Jacobian matrix. Computing Ee using Eq. requiress only single forward pass of all data through the network. whereas a naive implementation requires an additional backward pass up to the feature embedding layer.
B.2 SPECTRAL ANALYSIS OF THE EMPIRICAL FEATURE MATRIX
In this section we conduct spectral analysis on our Empirical Feature Matrix (EFM). Since Elastic Feature Consolidation (EFC) uses E, to selectively regularize drift in feature space, it is natural to question how selective its regularization is. We can gain insight into this by analyzing how the rank of E, evolves with increasing incremental tasks.
Let us consider a model incrementally trained on CIFAR- 100 using EFC method up to the sixth and final task and compute the spectrum of the E, for each f [L,...,6 .6. We chose shorter task sequence for simplicity, but the same empirical conclusions hold regardless the number tasks. Recalling that E, Rax" is symmetric, thus there exist U, and A, R such that:
E, = U; A.U, U.T A.U,,
(24)
where U, an orthogonal matrix and A, = diag(A1, An) ) is diagonal matrix whose entries are the eigenvalues of F.. Since F, is positive semi-definite, X; 2 O for each i E [1,..., nn. For our convenience we can rearrange the matrices A, and U. of the decomposition such that the eigenvalues on the diagonal are ordered by the greatest to the lowest, with the first being positive. In Figure Al we plot the spectrum of E, at each ł E ,,....,6, 6). The plot shows that the number of classes on which the matrix is estimated corresponds to elbow in the curve beyond which the spectrum of the matrix vanishes. This implies that the rank of the matrices is exactly equal to the number of observed classes. This empirical observation is in line with the results shown in Grementieri & Fioresi (022)) where the authors introduce the concept of a local data matrix similar to our local feature matrix, with the distinction that they take derivatives with respect to inputs instead of features. In the paper they prove, for a simple classification task, that the local data matrix has rank equal to the number of classes minus one, which is exactly what we observe in the evolution of E, during incremental learning.
Finally, we highlight that the results reported for EFC in the various tables of the paper were obtained using parameter values of λEFM = 10 and 0.1 in the regularization loss (Eq, 9. In Figure Al the horizontal line indicates that the plasticity constraints AgEM/ > (described in Section 3.3 of the paper) are satisfied for nearly every 14 0. This observation clearly demonstrates that our regularizer effectively constrains the features in a non-isotropic manner, distinguishing it from feature distillation.
14

---

Published as a conference paper at ICLR 2024
Incremental Spectrum, CIFAR-100 (WS) 5 Step
0.14
E1
E2
Ea
E
0.12
Rank 49
Rank 59
Rank 69
Rank 79
Rank 89
Rank 99
0.10
Magnitude
0.08
0.06
0.04
0.02
λEFC
0.00
20
40
60
100
120
Eigenvalue
Figure A1: The spectrum of the Empirical Feature Matrix across incremental learning steps. For better visualization of the spectrum in the analysis considered Warm Start 5-step scenario CIFAR-100. The r-axis is truncated at the 120th eigenvalue.
CIFAR-100 (WS) 10 Step
a= 2
Penclass Prob, Variation
0.3
0.3
0.7
0.5
0.1
0.1
0.4
Xg
42
0.0
0.0
20
40
20
40
0.5
1.0
1.5
2.0
Class
ID
Class ID
Figure A2: The regularizing effects of Eg in the Warm Start Scenario. Principal and non-principal direction are represented by solid lines and dashed lines, respectively.
B.3 REGULARIZING EFFECTS OF THE EMPIRICAL FEATURE MATRIX
In this section we extend the analysis provided Section 3.3 and Figure on CIFAR-100 (CS) 10 steps. Our aim is to show that the regularizing effect of the Empirical Feature Matrix remains consistent regardless of the number of classes considered Start scenario. Figure A2) shows that applying perturbation in principal directions of the EFM leads significant changes in classifier outputs, resulting in degradation of performance. Conversely, when the perturbation is applied in the non-principal directions, neither the classifier probabilities performance change. The results are in line with those observed in the same analysis but on the Cold Start scenario.
the initial task, namely in the Warm
the
Finally, we demonstrate the robustness of this analysis with respect to the order of classes within the tasks in both scenarios. To achieve this, we consider five different random network initializations and class orderings. In Figure A3 we give the average accuracy drop when adding Gaussian perturbations along principal and non-principal directions (separately). The plot reveals that perturbations along non-principal directions have no impact on accuracy (indicated by three dashed, overlapped lines on the x-axis), while perturbations along the relevant direction consistently degrade the performance of the model on all the considered tasks.
C PROTOTYPE UPDATE IMPLEMENTATION DETAILS
Here we provide some additional details about the prototype update using the Empirical Feature Matrix described in the main paper in Section 4.3 We weight the drift to update the prototypes
15

---

Published as a conference paper at ICLR 2024
CIFAR-100 (WS) Stop
CIFAR-100 (CS) 10 Step
Accuracy Drop
%g
Noerrey Droe
0.4
%
0.3
0.0
0.00 0.25 0.50
0.75
1.00
1.25
1.50
1.75
2.00
0.00
0.25
0.50
0.75
1000
1.25
1.50
1.75
2.00
Average accuracy drop in the Warm Start (left) and in the Cold Start (right) scenarios after 3 incremental steps when perturbing along principal (solid lines) or non-principal (dashed lines) directions of Eg. The values are averaged over 5 seeds, and the shaded region around the lines represents the standard deviation.
Figure A3:
using:
(fir p¹) F exp (- 20
KL(p(ylffi Wo1) plp(p Weal)
Wy exp
(25)
20
Let df = (f) (2rr D² be the distance according the EFM between current task feature fa extracted from the model trained task / - 1 and the prototype yf. However, since dē is not bounded (similar to the KL-Divergence), the exponential function becomes unstable across incremental learning steps. To address this problem, we normalize all the distances dţ be fall in the interval [0. 1] before applying the Gaussian kernel. In all our experiments, we use fixed 0.2 for the Gaussian kernel.
D PSEUDO-CODE FOR ELASTIC FEATURE CONSOLIDATION
The training procedure for the proposed method is described in Algorithm I The algorithm assumes that the first training has already been completed, as it does not require particular treatment.
Algorithm 1: Elastic Feature Consolidation
Data: M1. E,, P1 for t .....T do
/ * Iterate over epochs for each optimization step do
sample Nr, 2n X;
sample gaussian proto p P1a-1
compute CPR-ACE (Eg. 12
compute CFFM (Eq. 9
LEFC + CUR-ACE + L. CFFM
Update 0, + 02 - a 06,
end
P111 f- P1: + A(E(1)
(Eqs. 13 14
Pit + PIa UP, E, EFM(X;, (Eq.
end
E DATASET, IMPLEMENTATION AND HYPERPARAMETER SETTINGS
In this section we expand the content Section 5.1 and Section 5.2 providing additional information about the dataset and the implementation of the approaches we tested.
E. DATASETS
We perform our experimental evaluation on three standard datasets. CIFAR-100 (Krihhessy et al.) 2009) consists of 60,000 images divided into 100 classes, with 600 images per class (500 for training
16

---

Published as a conference paper at ICLR 2024
and 100 for testing). ImageNet- IK is the original ImageNet dataset Deng et al.. 2009 consisting (Wu et al. (i.]20177 7 consists
of about 1.3 million images divided into 1,000 classes. of 100,000 images divided into 200 classes, which are taken from ImageNet and resized to 64 x 64 pixels. ImageNet-Subset a subset of the original ImageNet dataset that consists of 100 classes. The images resized to 224 x 224 pixels.
E.2 IMPLEMENTATION AND HYPERPARAMETERS
For all the methods, we use the standard ResNet-18 (He et al. 2015) backbone trained from scratch. First Task Optimization Details. We use the same optimization settings for both the Warm Start and Cold Start scenarios. We train the models on CIFAR-100 and on Tiny-ImageNet for 100 epochs with Adam Kingma & Ba 2014) using an initial learning rate of Le and fixed weight decay of 2e The learning rate is reduced by factor of 0. after 45 and 90 epochs (as done in Zhu et al. (022b)) Toldo we followed the implementation of PASS et al. 2b22 fixing the number of epochs at 160, and used Stochastic Gradient Descent with an initial learning rate of 0.1, momentum of 0.9, and weight decay of 5e The learning rate was reduced by factor of 0.1 after 80, 120, and 150 epochs. We applied the same label and data augmentation (random crops and flips) for all the evaluated datasets. For the first task of each dataset, we use self-rotation as performed by Zhu et al. (2021b): Toldo & Ozay (2022).
& Ozay (2022)) For ImageNet-Subsee
Incremental Steps. Below provide the hyperparameters and the optimization settings we used for the incremental steps of each state-of-the- of-tth-artt method we tested.
EWC (Kirkpatrick et al., 2017): We used the implementation of Masana et al. (022) Specifically, we configured the coefficient associated to the regularizer as XE-FIM 5000 and the fusion of the old and new importance weights is done with = 0.5. For the incremental steps we fix the total number of epochs to 100 and we use Adam optimizer with an initial learning rate of 1e and fixed weight decay of 2e The learning rate is reduced by factor of 0.1 after 45 and 90 epochs.
LwF (Li & Hoiem 2017): We used the implementation of Masana et al. (2022). In particular, set the temperature parameter T = 2 as proposed in the original work and the parameter associated to the regularizer ÅLWF to 10. For the incremental steps we fix the total number of epochs to 100 and we use Adam optimizer with initial learning rate of 1 and fixed weight decay of 24, The learning rate is reduced by a factor of 0.1 after 45 and 90 epochs.
PASS (Zhu et al. 2021b): We follow the implementation provided by the authors. is an approach relying upon feature distillation and prototypes generation. Following the original paper we set ÄFD = 10 and Apr 10. In the original code, we find a temperature parameter, denoted T. applied to the classification loss, which we set to T = 1. As provided in the original paper. for the incremental steps we fix the total number of epochs to 100 and we use Adam optimizer with initial learning rate of Le and fixed weight decay of 2e The learning rate is reduced by a factor of 0.1 after 45 and 90 epochs
SSRE (Zhu et al. 2022): ; We follow the implementation provided by the authors. It is an approach relying upon feature distillation and prototypes generation. Following the original paper, we set AND = 10 and Apr = 10. In the original code, we find a temperature parameter, denoted as T, applied to the classification loss, which we set to original code, for the incremental steps we fixed the total number of epochs to 60 and used Adam Optimizer with an initial learning rate of 21 and fixed weight decay of 51 The
1. Following the
learning rate is reduced by factor of 0.1 after 45 epochs.
FeTrIL (Petit et al., 2023): We follow the official code provided by the authors. During the incremental steps, it uses Linear SVM classifier working on the pseudo-features extracted from the frozen backbone after the first task. We set the SVM regularization C and the tolerance to 0.0001 as provided by the authors.
Symmetric loss parametrr We fix the value of Apr 10 in Eq. 11 as reported in the literature (Zhu et al. 2022 2021b Toldo & Ozay 2022).
Feature Distillation loss hy sayeer-paaann parameeerr Feature distillation can be easily obtained setting 7) = 10 and XEFM in our training loss (see Eq. 9.. In our preliminary tests, we verified that feature
17

---

Published as a conference paper at ICLR 2024
Warm Start
1
Sttpp
CIL Method
5 Step
Step
EWC (Kirkpatrick al.
2017
30.47 + 3.38
50.08
1.13
CIFAR- 1-0
LwF-MC [L] & Horem
2017
43.57 1: 1.93
59.68
1 0.97
PASS Zhu ct al. 20216
36.73 0.44
65.94
0.68
Fusion
[Tldd Ozay
2022
58.72
66.80
FeTrIL [Peli 2023
37.52
SSRE 2hu
2022
37.79 0.52
66.53
0.60
EFC
62.57 0.33
69.51
0.64
FWC KKrrkattrik et al.
2017
10.64 + 0.59
27.51 + 0.61
Tiny- In--emegeee
LwF-MC &L Homem
2017
39.93 1 0.48
52.41
PASS ZZu D
44.80 0.35
53.98
0.48
Fusion Toldo Ozay 2022
48.57
FeTrII. (Pen 2023
46.35 + 0.28
54.96 t: 0.39
SSRE (hu et al. 2022
44.87 + 0.45
54.23 = 0.31
EFC
51.19 + 0.49
58.12
0.53
Table A1: Performance in the 5-step Warm Start scenario on CIFAR-10 110 and Tiny-Imagenet.
distillation is robust with respect to this hyper-paramette parameter, especially in Warm Start Scenario, due to little plasticity introduced the incremental steps.
F CIFAR-100 AND TINY-IMAGENET PERFORMANCE ON 5-STEP WARM START
Many methods are evaluated using short task sequences. To facilitare broad comparison with the state-of-the-art, in Table AT we report performance of all implemented methods in the 5-Step Warm Start scenario on CIFAR- 100 and Tiny-ImageNet.
G EXPERIMENTAL RESULTS ON IMAGENET-IK
To comprehensively evaluate the performance EFC, we conducted experiments across all analyzed scenarios on ImageNet- IK (see Table [22). In the Cold Start scenario, evenly split the classes, with 100 for the 10-step scenario and 50 for the 20-step scenario. In the Warm Start scenario, we consider large initial task with 500 classes for the 10-step scenario and 400 classes for the 20-step scenario. We compare our results with FeTrILL, which exhibited performance close to ours on all other datasets. For these experiments, in the ImageNet-Subset scenario in Appendix 2.2 but we fix the batch size to 256 and deactivate self-rotation for the first task. Furthermore, we fix the class ordering to align with that reported by] Yan et al. (202T).
the same hyperparameters and training protocol as
Even on the much larger ImageNet-IK IK dataset, our results demonstrate that in both Warm and Cold Start scenarios EFC remains highly effective at maintaining plasticity throughout training. Specifically, in the Warm Start scenario EFC exhibits substantial improvement of approximately 4% over FeTrIL both per-step and average incremental accuracy. is worth noting that the results reported in Petit al. (2023) are slightly better in the Warm Start scenario in terms of average incremental accuracy with respect to those reported in Table A2 This gap can probably be attributed to the performance obtained the initial task, which has a strong impact terms of average incremental accuracy. In Cold Start scenarios, we observe once again that freezing the feature extractor does not provide enough plasticity for FeTrIL learn the new tasks effectively, resulting in significant decrease (more than 8%) in accuracy compared to EFC.
Warm Start
Cold Start
Method
20 Step
Step
20
10 Step
Step
10
20 Step
FeTHL
55.26 1 0.16
51.10 + 0.22
62 1 DJIS
60.93
1 0.12
31.28 :: 0.20
26.64 64
0.25
48.93 = 0.15
40.02 0.15
EFC
59.49 0.07
55.94 1. 0.09
66.64
0.10
64.89
0.10
42.62
36.32
0.35
56.52
0.10
49,50
0.20
Table A2: Performance in the 10-step and 20-step in Warm and Cold Start scenarios on ImageNet- 1K.
18

---

Published as a conference paper at ICLR 2024
C
Cime
CIFAR-100 100 (CS)
Tan-lmageheet (CS) Stop
ImageNies Subset (CS)
Per-trep Accuracr
60-
6
Pre-stop
20-
20
23-
2
11
10
11 Sup
23
17
19
17
Step
Step
Figure A4: Ablation regularization and prototype losses.
Update Proto (WS)
Update Proto (CS)
Dataset
Step
10
58 24
+ 0.62
60.87
1: 0.39
39.28 1 0.97
43.62 1 0.70
CIFAR-100
20
52.60
= 0.73
55.78
0.42
27.28 + 1448
32.15
1.33
10
48. 16 1 0.53
50.40
1 0.25
31.23 1223000 0.88
34.10 1 0.77
TmageemageNet
20
45.70
+ 0.74
48.68
0.65
25.12 4 0.23
28.69
+ 0.40
10
66.92
1 0.43
68.85
0.58
39.99
2.20
47.38
1 1.43
ImageNNt-Subset
20
59.90
+: 1.09
62.17
29.62 2.60
35.75 + 1.74
Table A3: Ablation on prototype update.
H ADDITIONAL ABLATION STUDIES
In this section we extend the ablation study in 5.3 by providing the same analysis on the Tiny- ImageNet and ImageNet-Subset datasets.
Considering first the the ablation PR-ACE and Feature Distillation, Figure A4 confirms that asymmetric loss is a pivotal component in controlling the plasticity introduced by the EFM loss, surpassing feature distillation.
For the ablation on prototype updates, Table A3 clearly shows that updating prototypes enhances performance across all datasets and scenarios. As discussed in the main paper, the performance improvement more significant in the Cold Start scenario, where feature drift is more pronounced compared to the Warm Start scenario.
I ANALYSIS OF COMPUTATION AND STORAGE COST
In this section we consider the computational and storage efficiency of EFC, which computes the EFM after each task, regularizes using the EFM pseudo-metric, and stores class covariance matrices for Gaussian prototype sampling. In terms of storage efficiency, spectral analysis of the covariance matrices reveals that for all classes in all datasets and all task sequences the spectra highly concentrated in the first 30-50 eigenvalues and are thus -ell-appro by low-rank reconstructions. To verify this we ran experiments with low-rank reconstructions and give results for CIFAR-100 and Image in Table A4 On average, using only 30 eigenvectors sufficient to preserve 99% of the total variance and maintain the same EFCII. accuracy.
In Table A5 we consider two more aggressive strategies for mitigating storage costs. The Each Task approaches uses single covariance matrix per task for prototype generation and requires storage scaling linearly in number of tasks, while the Last Task stores only the covariance matrix from the last task learned and thus requires constant storage. Although the EFCIL accuracy suffers from using these proxies, the EFC results are still comparable with the state-of-the-art. Specifically, in the Warm Start scenario storing one covariance per class has a significant impact since the covariance matrices slightly change across the incremental learning steps. With respect to the Cold start scenario in this specific setting, feature drift is mitigated by the fact that we learn half of all available classes during the first session and thus the initial representation are already effective for new tasks. On the
19

---

Published as a conference paper at ICLR 2024
CIFAR-100
Tiny-ImageNet Avg
Preserved Variance
Arg # Components
Preserved Variance
A5
Components
0.90
11
57.35
0.90
15 3
19:02
Step
0.99
0.95
39 1:
61.65
0.99
0.95
48 6
50.85
49.72
1.00
512
62.57
1.00
0.90
11+22
55.79
0.90
15 +
47.79
Step
0.95
18 4:3
58. 28
0.95
24 1: 4
48.77
0.99
1,000
512
611.87
1.00
512
50:40
0.90
1:
50.67
20 Step
0.95
18 1:
32.95
0.95
23 4:
47.49
0..99
38 18.4100 6
54.91
0.99
48
6
48.39
1.00
512
35.78
48.68
Table A4: Warm Start EFC performance with low-rank covariance approximations for Gaussian prototype sampling. On both datasets in all task sequences using fewer than 10% of the principal directions average preserves 99% of the total variance and maintains the same accuracy as using full covariance matrices (full covariance with preserved variance = 1.00 given for reference).
Start
Map
A Step
sap
Variant
10 Step
20 Step
10 Step
10 Step
20 Step
10 Step
Step
Chiak 100
Each Task
59.37
0.18
51.97 + 0.86
07 0.38
65.12
0.92
45.26 + 0.62
33,73 + 2338
60.11 + 0.85
48.96
1.90
Task
0.80 68.22
62.70
0.10
06.70
0.86 63.80
43.72 0.38
31.43 + 2.48
10:46 0.09
47.27
1.53
EFC
60.87
0.30
58.78
+ 0.42
68.23 + 0.08
65.90
0.97
43.02
+ 0.70
32.15 + 1.33
58.58 + 0.91
47.36 1.37
Tng Thylnghe
Each Task
19.00 0.48
47.26
0.18
56.31
+: 0.51
55.06
4 0.39
34.81
0.56
29.39
1 0.52
47.81 0.39
4
Last Task
48.00
1. 0.43
45.64
1: 0.51
56.85
1: 0.54
54.12
1. 0.37
34.36
1. 0.14
28.57
1. 0.32
47.50 1. 0.24
42.11
1.111
EFC
50.40
1.0.25
48.68
0.65
0.53
34.10
0.77
28.69
42.07
0.96
Each Task Task
66.53
0.48
60.55
1: 0.73
74.10
0.62
70.36
1.24
48.49
2.18
36.31
1: 3.35
61.19 1: 2.46
50.52
2.22
65.46
0.60
50.20
0.69
73.42
69.59
1.40
47.21
1.57
35.43
3.00
60.50
50.22
EFC
68.85
0.58
62.17 4: 0.69
75.40 +: 0.92 71.63
1.13
47.38
1.43
35.75 4: 1.74
59.94 -: 1.38 49.92 2.05
Table A5: Mitigating storage costs with proxy covariance matrices. We can save only one covariance matrix per task (Each Task) use for prototype generation, which results in storage that grows linearly in number of tasks. Alternatively, we can store only the covariance matrix from the Last Task, which results in constant storage cost. Both proxies sacrifice some accuracy the Warm Start scenario, but results are still comparable with the state-of-the-ee (compare with results in Table IT.
contrary, in Cold Start scenario the representations are more prone to changes, and having per-class fixed covariances is less crucial. Consequently, the differences in Cold Start Scenario between the variants of our method are practically negligible. Theoretically, both scenarios, but especially Cold Start, could be improved if we design method to update these matrices during incremental learning. However, as we highlight in Section of the paper, this remains an open point.
In terms of computational efficiency we derived a method (Eq. 7 in the main paper and Appendix B. for computing the EFM without backward operations. Finally, we conducted wall-clock time analysis to assess the computational costs of EFC. The results of this analysis are presented in Figure A5 We see that EFC does not add significant overhead, but rather it is the addition of prototypes or exemplars that yields the most significant increase in computational time. LwF is the fastest methods since it does not rely upon prototype rehearsal. Among the prototype- based methods, EFC is comparable to SSRE in terms of computational time and it is much faster than PASS which is more time-consuming since applies self-rotation across the incremental steps. In the plot we do not report FeTrlL, relying upon SVM for the training. Currently, FelTll. is the fastest offte-of-the-art EFCIL approach since it freezes the backbone and updates only the classifier.
20

---

Published as a conference paper at ICLR 2024
CIFAR-100 Stop)
3.50
Wall clock timels)
2.
2.25
2.00
Step
Figure A5: Wall clock timing comparison. We compute the per-epoch time averaged over 100 epochs.
J WARM START PER-STEP ACCURACY PLOTS
we compare the per-step incremental accuracy of EFC and recent approaches on Warm Start scenarios in all three datasets for various task sequences. All methods begin from the same starting point after training on the large first task, but EFC maintains more plasticity thanks to the EFM regularizer and is thus better learning new tasks.
In Figure A6
K COLD START PER-STEP ACCURACY PLOTS
we compare the per-step incremental accuracy of EFC and recent approaches on Cold Start scenarios in all three datasets for various task sequences. All methods begin from the same starting point after training on the small first task, EFC both maintains plasticity thanks to EFM regularizer and is able to better update previous task classifiers to learn new tasks and mitigate forgetting.
In Figure A7
21

---

Published as a conference paper at ICLR 2024
EFC
EWC
FeTriL
LwF
PA55
SSRE
CIFAR-100 (W5) Step
CIFAR-IO0 100 (W5)- 10 Step
CIFAR-108 (W5) 20 Step
80
80
80
70
Accursyy
60
63
60-
50
40
30-
10
Step
Stop
Tiny-Imagelbe (WS)- Stop
Tiny-ImagehWet (WS) 10 Stop
Tiny-ImageNet (WS) 20 Stop
70
54
Acursy
50
stop Accuraa
80
40
30
30
20
20-
20-
10-
Step
ImageNet-Subset (WS) 10 Step
ImageNeet Subset (WS) Step
70
70-
Per Accursy
50
20
Figure A6: Warm Start (WS) per-step accuracy during the incremental learning. The plots compare
recent EFCIL methods against EFC three different dataset for different incremental step sequences.
22

---

Published as a conference paper at ICLR 2024
FeTAL
SSRE
(CS) Step
(CS) Step
90
70
Perstep Accuarc
60
50
50
30
10
10
20
Tiny-lm ageNet Step
Tiny-ImageNet. (CS) Step
80
70
70
Accurayy
stop Accurscy
60
50
40
30
20
10
10
10
16
18
20
Step
Step
ImageNet-Subee Subset (CS) 10 Step
ImageNet Subset (CS) Step
Persto Acoursc
50
40-
10
10
12
14
16
18
20
Stop
Stop
Figure A7: Cold Start (CS) per-step accuracy plots during incremental learning. The plots compare
recent EFCIL methods with EFC on three different datasets for different incremental step sequences.
23


Reviews:
[{"review_id": "m6A32bMlSa", "reviewer": "Reviewer_4Tis", "summary": "This article introduces a new EFCIL approach, called Elastic Feature Consolidation (EFC), which regulates changes in feature space directions most pertinent to previously-learned tasks while permitting greater adaptability in other directions. An essential contribution of this research is the establishment of a pseudo-metric in feature space generated by a matrix termed the Empirical Feature Matrix (EFM). Unlike the Fisher Information Matrix, the EFM can be easily stored and computed as it is independent of the number of model parameters, relying solely on the dimensionality of the feature space. To tackle the drift of the more flexible backbone, it also proposes an Asymmetric Prototype Replay loss (PR-ACE) that strikes a balance between new-task data and Gaussian prototypes during EFCIL. Lastly, it presents an enhanced method for updating class prototypes that leverages the already-computed EFM.", "questions": "Please address the questions raised in the weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- This paper presents a compelling and complex issue in incremental learning. Many current methods depend on storing exemplars from the previous task and require thorough exploration of highly informative exemplars to combat forgetting effectively. In contrast to existing approaches, the method proposed in this paper loosens the constraints regarding the availability of examples from the previous task. Consequently, it addresses a more practical and formidable problem.\n\n2- The proposed method underwent experiments on three datasets, demonstrating a noteworthy enhancement in comparison to existing approaches. To substantiate the assertions and efficacy of the model components, a comprehensive ablation analysis was conducted and included in the paper.", "weaknesses": "1- In this method, each task is divided into disjoint classes, and the class boundaries are known for each task. However, can this method be extended to cases where the task boundaries are unknown for each task?\n\n2- This method calculates the Empirical Feature Matrix (EFM), which determines the significant feature direction from a prior task, the mechanism of which remains unclear. Could you please provide a comprehensive explanation for better comprehension?\n3- In Table 2, the results for the cold start issue demonstrate undesirable outcomes. Could you please provide an explanation in the main paper to justify this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2wK4CjKEPV", "reviewer": "Reviewer_sY5k", "summary": "This paper is on the topic of continual learning. Specifically, it addresses the popular scenario of Class Incremental Learning (CIL). Within CIL, it assumes two of the most challenges constraints: (1) no exemplar/rehearsal memory (in which samples from past tasks are stored in memory and used to supplement training of new tasks) is permitted (EFCIL) and (2) the size of the first task is not large - the authors call this cold-start.  The paper takes careful note of both main challenges in continual learning: forgetting, and inter-task confusion, and weaknesses in past method, EWC. The new approach proposed aims to address both challenges, and the weakness of EWC: the strategy is to define a loss function that mitigates forgetting (by regularization of features rather than  weights) but still allows for plasticity, to ensure the new task can be learned.  The net result is the loss stated in Equation (15).  From ablation studies on split CIFAR-100, this new loss is sufficient to outperform comparison methods on the cold start scenario by large margins. But the papers adds an addition trick to perform even better: the use of prototype updates.  The new losses are supported by careful theoretical derivations.", "questions": "As per above, do ablations on all data sets show the same trend, i.e. that the EFM+PR-ACE loss is enough to surpass comparison methods by itself, and adding prototypes rehearsal to this creates a further boost?\n\nI found it very difficult to read the tiny font in Figure 2 and Figure 4(a), so please can you find a way to make those more easily readable?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: the paper makes two novel contributions to EFCIL - the combination new loss (EFM+PR-ACE), and supplementation of this with their prototype rehearsal method. \n\nQuality: the chain of reasoning motivating the approach, and the supporting theory are carefully argued and demonstrate a deep understanding of the challenges in EFCIL, resulting in new SOTA performance in this subset of CIL. The appendices are well written, comprehensive and support the paper well. I looked through the code supplied as SM and found it ran out of the box and reproduced one of the results.\n\nClarity: the paper is very well expressed and presented. I liked the sequence of material, and found it enjoyable to read.\n\nSignificance: this paper has clear importance for EFCIL and is likely to be an often compared-with benchmark for future competing methods. I think there's scope for the approaches here to impact on work that does assume exemplars, and potentially on EFCIl methods that utilise pre-trained models.", "weaknesses": "I only have one weakness within the material presented: I would prefer to see ablations for more than just one dataset. In particular, in Fig 4, only CIFAR-100 is seen, in Table 2, only TinyImageNet. Figure 4(b) tells me that the EFM+PR-ACE loss is enough by itself (i.e. without prototype rehearsal) for better results than comparison methods on cold startup. But is this also true for the other datasets?\n\nSince I only have one weakness to list on the contributions of the paper, I'll take the opportunity to make a mild critique of two weaknesses (in my opinion) in the assumptions that are made in the problem setup in this paper, and in much related work in continual learning.\n\n1. Recently some of the community have started to question the frequent assumption that we need CL methods because storage of data is expensive, or unavailable due to privacy. See for example https://arxiv.org/abs/2305.09253 \n    - I understand there's very little space to discuss the arguments for and against the importance of \"exemplar free\" but it would be better to at least acknowledge that there's some debate on this.\n\n2. This paper, and many others like it, seek a CL method that trains a network from a random initial state. However, in practical industry settings, it's nearly always better to consider transfer learning, which for an image classifier has for a long time meant starting with a model trained on ImageNet, but nowadays is also meaning self-supervised pre-training. It's puzzling why the CL community ignore this for the most part. This might be changing now, with a bunch of papers in 2023 now doing CIL with pretrained ViT models (and also ResNets - e.g. your cited ref Panos et al). It could be a simple experiment to try this and assess the benefits of using your method in this different assumption. I am raising to highlight a potential opportunity, and to recommend you at least mention your assumption of the need to train a model from scratch rather than use pre-trained weights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r8tLlOQpE4", "reviewer": "Reviewer_Hr2j", "summary": "This paper investigates the cold start problem in exemplar-free class incremental learning (EFCIL) scenario. In this setup, the first task contains limited classes and the backbone should be plastic enough to learn other classes. The Elastic Feature Consolidation (EFC) is proposed to balance the plasticity and stability in EFCIL. Experimental results demonstrate the superiority of EFC.", "questions": "No more questions", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Although it is commonly considered in exemplar-based class incremental learning, the cold start scenario is ignored in exemplar-free class incremental learning. EFCIL with cold start brings new challenges, and this paper is well-motivated to address them.\n\nInspired by the well-known Elastic Weight Consolidation (EFC), the proposed Elastic Feature Consolidation (EFC) is novel and effective.\n\nI like the illustration in Figure 2, which demonstrates the motivation of EFC. Besides, the spectral analysis in B.2 is interesting.", "weaknesses": "Lack of experiments on Imagenet-1k dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRFAsVpG66", "reviewer": "Reviewer_RQMN", "summary": "This paper studies the examplar-free class incremental learning (EFCIL) problem. The author is motivated by the cold-start scenario, i.e., the first task is insufficiently large to learn a high-quality backbone and thus demands the method to be more plastic to adapt their backbone to learn new tasks. Moreover, the author thinks existing methods in EFCIL pose too strong a regularization to restrict the model to be plastic, like the EWC in parameter space and the L2 regularization in feature space. To give the model more plasticity in learning new tasks, the author proposes a new method called the empirical feature matrix (EFM), which is a variant of the empirical fisher matrix in that the author calculates the square of the derivative over the feature extractor instead of the whole network and uses this as a regularization term to penalize the change in feature space. Moreover, the author combines both the prototype rehearsal and prototype drift compensation from existing works via the proposed EFM to create an asymmetric prototype rehearsal loss to provide elastic feature consolidation. Extensive experiments are conducted on the standard continual learning benchmarks and the author also compares with state-of-the-art EFCIL methods and provides further analysis.", "questions": "Please refer to the Weaknesses section for more details.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall, the paper is well-written. The idea of the empirical feature matrix is an intuitive extension of the empirical fisher matrix, which seems interesting. The combination of the existing methods of prototype rehearsal and prototype drift compensation makes the proposed methods more sound given that the effectiveness of these components has been empirically verified previously.\n2. The proposed method can achieve new SOTA performance on either warm or cold start scenarios, which demonstrates the effectiveness and generality of the proposed method.", "weaknesses": "1. The reviewer is unconvinced with the so-called definition of the \"cold start\" scenario in the present paper. In the definition provided by the author in the Introduction:\n\n\"In this paper we consider EFCIL in the more challenging Cold Start scenario in which the first task is insufficiently large to learn a high-quality backbone and methods must be plastic and adapt their backbone to new tasks.\"\n\nFirst, how to quantify the term \"insufficiently large\"? The reviewer sees that the author uniformly distributes all classes among all tasks in the Experiments, which means like 10 or 5 classes learned in step 1 for CIFAR-100. However, such experimental protocols are widely used in existing CL literature (e.g., see [1] and its follow-up works), which should be a *standard* experimental protocol for benchmarking CL. Moreover, how can the author confirm that 10 classes can not be a \"sufficiently large\" task? What criteria that the author explicitly measure whether a task is \"sufficient\" or \"insufficiently\" large? \n\nSecond, how can the author confirm that the backbone is not of \"high quality\"? What quality does the author refer to? The quality of learning the current tasks or the quality of learning the new tasks? Why does the model need to have more plasticity for learning the new task if the model is not of \"high quality\"? One can also say that the model has **exactly high plasticity** for learning new tasks given that the model has **only** learned for 5-10 classes given the overparameterized property of the modern CNN/DNN. \n\nOverall, the definition of the so-called \"cold-start\" scenario and the corresponding motivations that lead the author to propose the current method are not solid and sound. This paper may still stand out even if we do not mention the so-called \"cold-start\" scenario at first and instead only talk about the strong regularization effect made by existing methods and thus demand more flexible regularization methods. Then what is the necessity of this \"cold-start\" scenario? \n\nLast but not least, from the convention in machine learning, cold-start scenarios often refer to scenarios where we do not have any data to start with, e.g., for the recommendation system. With 5-10 classes (which means thousands of images) for training the model can hardly be thought of as a \"cold\" start.\n\n2. The details of Figure 2 are missing. What dataset and experimental protocols have the author used? Will the conclusion be influenced by the number of classes? Will the number of training data in the initial task have an impact on the conclusion? The author should provide the mean and std and repeat experiments to support this empirical observation.\n\nMinor:\n1. Typo in the first sentence of Section 5. \"with with\" -> \"with\" \n\n\n[1] DER: Dynamically Expandable Representation for Class Incremental Learning (CVPR 2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)