Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting
Authors: Timm Hess
Eli Verwimp
Gido M van de Ven
Tinne Tuytelaars
Venue: Accepted by TMLR ()

Abstract: Continual learning research has shown that neural networks suffer from catastrophic forgetting "at the output level", but it is debated whether this is also the case at the level of learned representations. Multiple recent studies ascribe representations a certain level of innate robustness against forgetting - that they only forget minimally in comparison with forgetting at the output level. We revisit and expand upon the experiments that revealed this difference in forgetting and illustrate the coexistence of two phenomena that affect the quality of continually learned representations: knowledge accumulation and feature forgetting. Taking both aspects into account, we show that, even though forgetting in the representation (i.e. feature forgetting) can be small in absolute terms, when measuring relative to how much was learned during a task, forgetting in the representation tends to be just as catastrophic as forgetting at the output level. Next we show that this feature forgetting is problematic as it substantially slows down the incremental learning of good general representations (i.e. knowledge accumulation). Finally, we study how feature forgetting and knowledge accumulation are affected by different types of continual learning methods.

Published in Transactions on Machine Learning Research (May/2024)
Knowledge Accumulation in Continually Learned Representations and the Issue of Feature Forgetting
Timm Hess*
timmfelithhess@kuluunnn..
KU Lewven, Belgium
Eli Verwimp" KU Lewven, Belgium
eli. wwb.verrinmm@kkuueeee...u
Gido M. van de Ven KU Leuven, Belgium
gido. www.vvadeeee@@ulueee...kk
Tinne Tuytelaars KU Leuven, Belgium
chone.tugeebaars@@uuuvvee..
Reviewed on OpenReview: https: //
// openreview. net/ forun? id- aHt2uZfHef
Abstract
Continual learning research has shown that neural networks suffer from catastrophic for- getting "at the output level", but it is debated whether this is also the case at the level of learned representations. Multiple recent studies ascribe representations a certain level of innate robustness against forgetting that they only forget minimally in comparison with forgetting at the output level. We revisit and expand upon the experiments that re- vealed this difference in forgetting and illustrate the cocxistence of two phenomena that affect the quality of continually learned representations: knowledge accumulation and fea- ture forgetting. Taking both aspects into account, we show that, even though forgetting in the representation (i.e. feature forgetting) can be small in absolute terms, when mcasuring relative to how much was learned during task, forgetting in the representation tends to be just as catastrophic as forgetting at the output level. Next we show that this feature forget- ting is problematic as it substantially slows down the incremental learning of good general representations (f.e. knowledge accumulation). Finally, we study how feature forgetting and knowledge accumulation are affected by different. types of continual learning methods.
1 Introduction
Machine learning models typically trained static datasets, and once they are deployed, they usually not updated anymore. However, sometimes models make mistakes. Sometimes they do not work in domain that was not trained on. Sometimes they do not recognize certain classes or corner cases. To overcome such malfunctions, the default choice industry is to gather new data and to retrain a model from the beginning with both the new and old data (Huyen 2022). Retraining a full model is costly and time-consuming, especially in deep learning. The goal of continual learning is to enable models to train continually, learn from new data when they become available. This has proven to be hard challenge (De Lange et al,, 2022 van de Ven al.) 2022), deep learning models that are continually trained exhibit catastrophic forgetting (MeCloskey & Cohen 1989 Rateliff 1990). Without precautionary measures, new information is learned at the expense of forgetting earlier acquired knowledge.
The data to train machine learning models rarely come in a format that is adapted to the problem one intends to solve. Taking the example of visual data, it is near impossible to infer high-level properties directly from
* Authors contributed equally.

---

Published in Transactions on Machine Learning Research (May/202) /204)
image's raw pixel values. Hence, first step is usually to transform the data into a representation that makes solving the problem at hand easier. Often deep neural networks are used for this (Bengio et al. 2013). These networks learn semantically meaningful representations while optimizing their parameters to learn an input-output mapping. Sometimes the goal is the representation itself, yet often it is a final layer, or head, that uses the learned representation to assign an output (e.g. a class label) to an inpul. Ofien the backbone network used to calculate a representation and the head that uses this representation are trained in unison, but it can be useful to think of them as two separate entities working together.
In continual learning, there are at least two good reasons to care about representations. First, a strong reprosentation makes it casier to learn new information (Kornblith et al. (019).. When model already has a good representation, it may require less changes to adapt to new data, which lowers the risk of forgetting (Cha let al. 2021). Second, contimually learning a good representation by progressively accumulating knowledge from individual tasks a goal on its own. Effective continual learning should be able to use new information to its benefit and build a stronger representation over time (Chen & Liu 2018). which can finally be used to solve variety of lasks (Bengio et al. 2013)
It is with these motivations that recent works have been studying how representations are learned in con- tinual learning, and how they forget. Multiple studies observe an apparent robustness to forgetting for opprssntatiins (Davari et al. 2022 Zhang et al.) 2022 Hu et al. 2022; Wang et al. 2023). For example, Davari et al. (2022) write: "[...] in many commonly studied cases of catastrophic forgetfing, the representa- tions under naive fine-tuning approaches, undergo minimal forgetting, without losing critical task informa- tion" (p. 16713), and Zhang et al. (2022) comment: "there seems to be no catastrophic forgetting in terms of representations" (p. 4) and "common techniques for måtigating catastrophic forgetting [.-.] have little effect on improving frepresentations!" (p. 4). These studies further suggest that in the case that some forgetting of learned features does happen, this does not hinder the learning of good general representations. For ex- ample, when measuring the representation quality for a task that was not included in the training sequence, Zhang et al. (202)) conclude that "representation learning and catastrophic forgetting are largely separate issues" " (p. 9). If this is true, forgetting would only be a problem if one cares about the performance on the trained tasks, but not if one cares about learning a good general representation.
In this work, carcfully revisit the dynamics of learning and forgetting at the representation level, and we illustrate where its apparent innate robustness to forgetting originates. First, we highlight two key phenomena affecting the quality of the continually learned representation: feature forgetting and knowledge accumulation. Then, by considering both jointly, we re-investigate continual learning at the representation level. In short, we try to answer two questions:
Question 1: Do continually trained representations forget catastrophicallyy
With extensive experiments show that, also at the level of representations, newly learned information tends to be rapidly and drastically forgotten during continued training on other tasks. The first tasks socmingly forget less, yet we show that the non-forgotten information is information that is shared between tasks. This leads us to the follow-up question:
Question 2: What are the consequences of forgetting these representations?
To determine the impact of fcature forgetting on the quality of the continually learned representation for downstream tasks, we compare the representation of a continually trained model against, a representation that is ensembled from copies of the model obtained after finishing training each task. This ensemble bascline and the continually trained model learn in the exact same way, but differ because the former does not forget. We find that the ensemble baseline yields a substantially better general representation than the continually trained model, which indicates that feature forgetting slows down knowledge accumulation (Le the improvement of the learned representation over time). This means that preventing feature forgetting is not only important for the performance on tasks that a model was trained on, but also to learn strong representations in general.
Most experiments in this paper study the continual learning and forgetting of representations in supervised learning, though in Appendix C we show evidence that suggests that our main conclusions also hold for
2

---

Published in Transactions on Machine Learning Research (May/200 (024)
self-supervised learning. We conclude the paper by evaluating examples of important families of continual learning methods and reporting how they influence the learning and forgetting of representations. In summary, our contributions include
We illustrate the characteristics of feature forgetting and knowledge accumulation evaluated at the representation level (Section 3
We show that newly learned information is forgotten just as catastrophically at the representation level as it is at the output level, except when the information can also be learned from other tasks (Sections
4.1 and 4.2
We show that forgetting in the representation hinders knowledge accumulation (Section 4.3
We compare the level of feature forgetting and knowledge accumulation for different types of con- tinual learning methods (Section 5).
2 Preliminaries
We follow the common definition of a continual learning setting by assuming a sequence of m disjoint tasks T {Ti, ",,, Tm Each task T consists of training data with inputs Xj and targets Yj, as well respective test data X,, Yz. While training on task Tj, only the training data {X,, Yj} of that task are usod. Exceptions are replay memorics, which can store small subsots of data from past tasks; we only use such replay memories when studying experience replay in Section 5 Finally, we use Ta to refer to a downstream lask, i.e. one that is not part of the training sequence T. To study the representation learned by model the sequence T, distinguish between the backbone parameters 0B and the task-specific heads HH = :(),........ ..... Pn. J. For task Tj, the output of a model is calculated as g 0g); 0h,) and its representation as f(;0n)) In this paper, each head On, specifies a linear layer and we use, with slight abuse of notation, f(x; 0n, 0g) to refer to the combination of backbone and head. Alternatively, we use f, to denote the entire model directly after training on task T., and we use fo for a model with random weights.
Our main focus is on classification tasks. We use rf as general symbol to refer to a metric that quantifies the performance quality of model fa for task Tj- We drop super- and subscript when they are clear from context. To differentiate between the quality of the backbone and the entire model, we define two such performance metrics: output accuracy OUT, and linear probe accuracy LPi, (or LP accuracy). OUT, refers to the standard test accuracy of model f(; 0B) immediately after training task T,. When i < j. output accuracy is equal to random performance, í.e. the performance of randomly initialized head. To measure the quality of a representation, one option would be to use the optimal head 07, for a particular backbone 8 and task Ty- To approximate this optimal head, we train a new head with parameters 6,, using the training data of task T3, while keeping the backbone parameters 0g frozen, as is common in representation learning [Bengio et al. 2013). Using this new head, LPi, is equal to the test accuracy of f(x; (1,; 03) on task Tj, whereby On are the backbone parameters immediately after training on task Tü. Alternative ways to evaluate representations reach similar results for the experiments reported in this paper, see Appendix HH]
In the main paper, the reported results are on Split MiniImageNet, a 20 task (five classes cach) split of MinilmageNet (Vinyals et al. 2016). We use 19 tasks as the training sequence T, while the remaining task is never seen during training and used as downstream task Ta. To reduce the influence of the difficulty of a task, use five task splits and report mean and standard error (SE) on all results. The splits are randomly generated but consistent across experiments. When evaluating the performance on a task, the model only needs to choose between the classes of that task. We do not include all classes seen so far (i.e., as would be done in a class-incremmnnal learning evaluation), because this would make the evaluation 'task' more difficult after each trained task, thereby confounding the analysis. Further details of our experimental protocol are we report experiments using Split CIFAR-100 KKizhhesky et al. 2009).
in Appendix A In Appendix G
"ode available at: https://gttti //gtt///////bbb.//// com//Tinmieess/KAafF
3

---

Published in Transactions on Machine Learning Research (May/2024)
Evaluation on T
Evaluation downstream task (Ta)
1.00
1.00
Knowledge
accumulation
0.75
0.75
Feature forgetting
1ie
0.50
0.50
0.25
0.25
0.00
0.00
T
T
Tis Trained task
Ts
T.
T.
Ta Trained task
LP aecuracy
Output accuracy
Figure 1: Illustaaionn of feature forgetting and knowledge accumulation, for model continually finctuned Split MinilmageNeet. On the left, the output accuracy and LP accuracy Tg are shown. The difference between LP accuracy directly after the task was trained and the LP accuracy after training on later tasks is what we call feature forgetting. On the right, the same continually trained model is evaluated with the same metrics but on a downstream task (Ta). We refer to the improvement in LP accuracy over that of a randomly initialized model (Jo) as knowledge accumulation. (Mean + SE, 5 runs)
3 Knowledge accumulation and feature forgetting
Catastrophic forgetting is observed when measuring accuracy at the output level of continually finetuned neural network. However, recent studics exemplify how evaluating performance of the same model at repre- sentation level instead leads to a different result. In the following, we highlight two evaluation perspectives that have been used in recent. literature to illustrate this difference between the output and representation level. With these evaluation perspectives, two phenomena can be identified that affect the quality of contin- ually learned reprosentations: feature forgetting and knowledge accumulation. The evaluation perspectives differ in what data they evaluate on. On the one hand, a task that was included in the training sequence (e.g. T&) can be used, on the other hand, a downstream task (Ta) that is excluded from the training sequence can be used. Each perspective can show one of the two phenomena.
Evaluating on a task included in the training sequence (e.g. Te), it can be observed that linear probing, Le. retraining the head, can recover much of the performance that is lost in the output accuracy (Figure left), which could be interpreted as representations forgetting less than at the output (Davari et al. 2022 The performance degradation, or forgetting, that persists beyond the recovered performance of the linear probe is what we call feature foryetting. When features are forgotten, the representations of past classes become less separable. This contributes to the total amount of forgetting in continual lcarning, to which e.g. representation drift and misaligned heads contribute as well (Caccia et al. (021), see also Section 7
The second evaluation perspective inspects the performance of a continually trained model on downstream task. Using the same setting above, we find that the representation for Ta progressively improves with each learned task, without significant declines (Figure 1 right). We call this improvement of a representation for unseen tasks knowledge accumulation. A model that has accumulated more knowledge by this definition, is a more general model and is better suited to solve new tasks, although it may have forgotten features specific to a past task at the same time. This definition is similar to the one used by [Jin et al. (2021). who equate knowledge accumulation with the capability of generalizing to unseen tasks. [Lesort et al. (2023) and Caccia et al. (2200) use the term knowledge accumulation slightly differently, loosely refer to performance on both old and new tasks. Knowledge accumulation is related to forward transfer, which is one of the main desiderata in mask-incremental learning (van de Ven et al. 2022 De Lange et al. 2022). Yet mask-nncrementa learning results themselves are not well suited to study the reprosentations directly, as representation drift (Caccia et al. 2021) can result in misaligned heads, which could suggest that there is no knowledge accumulation even when there is. Finally, in continual pretraining (Hu et al. 2022 Lee et al., 2023 Cossu et al. 2022) the performance on downstream tasks what matters, yet the influence of forgetting pretrained features has not been studied in these works. In what follows, we use both concepts, feature forgetting and knowledge accumulation, to study the dynamics of continual learning at the representation level.
4

---

Published in Transactions on Machine Learning Research (May/202) (224)
4 Forgetting in continually learned representations
In this section, we analyze feature forgetting by taking knowledge accumulation into account. Section 4.1 shows that newly gained performance tends to be lost similarly at the output and the representation. Features learned during the first tasks in the training sequence are an exception to this, which we investigate further in points out that feature forgetting hinders effective knowledge accumulation
Section 4.2 Finally, Soction 4.3
for downstream tasks.
4.1 Relative forgetting
McCloskey Cohen (989)] and Ratcliff (9900) are often credited for discovering the phenomenon of catas- trophic forgetting. They describe it respectively as: "fthraining on a new set of items may drastically disrupt performance on previously learned items" (p. 110), and "well-learned information is forgotten rapidly as new information is learned" (p. 285). In recent literature, following Lopez-Paz & Ranzato (2017), forgetting is commonly quantified as the difference between the performance immediately after task T3 was trained on and the performance after training on n new tasks: rj Tytn.. Both output accuracy and linear probing accuracy have been used in this way (e.g. Davari et al.
2022 Forgetting, defined as such, is thus equal to the absolute drop in performance. In contrast, the amount of forgetting can also be considered relative to how much information was learned during training on the task. We propose to define the relative forgetting of task T3 after new tasks as:
Tji Tjtnjj FFn.
(1)
T3 1--
Or, in words: relative forgetting measures how much of the performance that was gained during training on a task forgotten afterwards. Figure 2 illustrates how absolute forgetting and relative forgetting can lead to different conclusions. When inspecting the absolute forgetting the output and the representation (middle panel), one could conclude that the forgetting at the output. is worse than at the representation level. However, the representation quality also improved less during training of task T' (left panel), partly because of previously accumulated knowledge. Relative to the gained performance, forgetting is roughly equal at the representation and at the output (right. panel); both lose nearly all gained performance. Considering the gained performance is thus crucial when comparing forgetting in settings where the performance before training a task differs across the compared settings, because it changes the denominator of Equation [ This happens when considering representations and LP accuracy, but in most cases not for output accuracy. Their difference is comparable to the difference between the answers to the following two questions before seeing any task data: Which test samples belong to the wnknown category x? and Given that x looks like this, which other lest samples are of calegory x?. While the first answer will be random, the second one depends on how good the description, e.g. the representation, of x is. A model that has accumulated more knowledge and has a bettor representation can have improved performance on the second question, but not on the first.
Figure 3 shows the relative forgetting of the representation and of the output for lasks until 9 (the one of To is thus copy of the right panel in Figure D). For early tasks (especially 7i), forgetting as measured by lincar probing stabilizes after one or two new tasks, while for the output rF,., contimues to increase. For the later tasks, the representation forgets at least much as at the output (see Appendix F for task T1o and more). This observation noticeably leads to a different conclusion than the one drawn by the papers in the introduction: except for the first tasks, almost all performance that was gained while training a task, is lost when training on new ones, both at the output and the representation level.
4.2 Exclusion baseline
The previous section showed that nearly all performance gained during a task lends to be quickly lost both for the representation and at the output, except for the first tasks in the training sequence. For these tasks, there a significant difference between the relative forgetting at the output and the representation, see Figure 3 After training a few new tasks, the output has rF 1, while the representation has significantly lower rF. In this subsection, we show that this difference can be explained by knowledge accumulation from
5

---

Published in Transactions on Machine Learning Research (May/2024)
Performance T
Absolute forgetting of
Relative forgetting of T
1.00
1.00
0.75
0.75
0.75
0.30
0.50
0.25
0.25
T;
T,
T3 Trained task
Tix
0.00
T:
T₁
0.00
1,
Tie
Ti
LP accuracy
Output accuracy
Trained task
Figure 2: Performance, absolute forgetting and relative forgetting of Te for model continually finetuned Split MinilmageNet. Comparing output accuracy and representation quality (LP accuracy) can have different interpretations when considered in absolute or relative terms. Absolute forgetting (middle) suggests that output forgetting is worse than at the representation level. Such absolute forgettting does not account for performance already accumulated before learning To (left). When expressing forgetting relative to newly gained performance (right), forgetting trends are similar the output and representation level. (Mean SE, runs)
1.0
Relative Torgotting
0.5
T2
T.
Tg
Tg
T2
Tg
To
accursicy
Outppt accuracy
Figure 3: Relative forgetting at the output level and at the level of representations, as per Eq. 1 for a model continually fine-tuned on the Split MinilmageNett sequence. Relative to the knowledge gained during training on the task, forgetting in the representation and at the output are similar, except for the first few tasks. (Mean SE, runs)
other tasks. When learning one task improves the representation for another, those tasks have something in common. Even if model forgets everything it had learned, the common information can be re-learned from the later tasks. After training on all tasks, the final model would still perform better than a model with random weights, which results in rF scores lower than one. To further unravel how knowledge accumulation influences forgetting,
train new model f-J the same task sequence in the previous section, but withoul task Tj, see Figure 4a. Model f cannot forget specific information task T3 as it was never trained on it, but it can accumulate common knowledge from other tasks. We define EXC,,; as the difference between both in models in LP accuracy for task T, after training task Ty:
EXC,, = LP - -LP
If this difference is larger than zero, model f remembers some specific information that was not learned from other tasks. Figure and Table g show that for all tasks, the difference reduces to almost zero. The information that was not forgotten from the earlier tasks, could thus also be learned from other tasks and is not specific to that task only. Table D suggests that there might be a subtle difference the end of the training sequence between f and f f-, but the errors are too large to draw definite conclusions.
Learning one task does, in most cases, not improve the output accuracy of another. See e.g. the left panel of Figure D while the LP accuracy for Tg improves even before Tç is trained, the output accuracy remains at chance level. Forgolten information thus cannol be re-learned from other lasks, which suggests why the relative forgetting at the output does approach one. Similar results are to be expected if the learned representation for one task does not improve the other. This observation implics an important nuance: even when the output accuracies of two models are both chance level, one can have a better representation than the other for a particular task.

---

Published in Transactions on Machine Learning Research 2024)
Training sequence
Eval
E
EXC,2
EXC1,
0.0
Tg evaalaation
EXC,,
EXC,s
E
0.9
011
0.7
EXC,T
EXC,s
EXC;3
0.5
T,
T2
T2
0.0
Trained task
(a)
(b)
Figure 4: Illustration of the task exclusion difference (EXC, Eq. 2) for continual finetuning Split Minilm- ageNet. Panel (a) depicts the adjusted training sequence for the "exclusion model" f and a comparison of the resulting LP accuracy for contimually finetuned model ʃ and ʃ-6 on task Te- Panel (b) shows that EXC;,; for the first 9 tasks, always follows the same trend. It averages around zero for tasks < j, peaks for j, and quickly reduces to almost zero again for i> j (Mean + SE, runs).
Table 1: Quantitative results of Figure 4b The final difference (EXC19,)) and maximal difference (EXC,J) between model f and its exclusion counterpart f-1. (Mean + SE, 5 runs)
T
T;
T2
T,
To
T:
T,
Tg
EXCLoJ
0.001 + 0.004
0.002
0.011
0.008 + 0.004
0.018 + 0.013
0.023 + 0.012
0.010 + 0.010
0.012 + 0.015
0.006
0.007
0.012 + 0.012
EXC,
0.234
0.024
0.150
0.015
0.151
0.025
0.165
0.015
0.119
0.010
0.132 1 0,012
0.140
0.014
0.130
0.021
0.153 0.008
4.3 Feature forgetting reduces knowledge accumulation
Previous sections showed that feature forgetting severely impacts newly learned features in neural networks and that what is not forgotten can also be learned from other tasks. This leads us the second question of the introduction: what are the consequences of fcature forgetting? Is the forgotten information important for that specific task only, or can it also contribute to more general knowledge accumulation? Phrased differently, how much knowledge accumulation would there be if there was no forgetting? To answer this, we consider a baseline that learns continually but has no forgetting, inspired by Vogelstein et al. (2020) and Yan et al. 2021). This ensemble bascline stores a model copy after every task. During evaluation, their representations ions are concatenated and a linear probe is trained on top of this concalenation ion. This ensemble learns exactly the same way as continually finetuned model, but it cannot forget, as the task's original reprosentation remains intact and can always be recovered by a head. See Figure [5a] for schematic of the approach, and Figure 5b for the results. Finetuming and the ensemble are exactly the same after training the first task. Knowledge accumulation from the first task aside, our ensemble baseline accumulates almost three times as much knowledge, signified by a 27.4% improvement on a downstream task versus only 9.3% for finetuning. It is interesting that finetuning also accumulates some knowledge during later tasks, albeit far inferior to what is possible when forgetting nothing.
Models trained on all tasks concurrently (jointly) rather than sequentially can also lead improved represen- tations (e.g. Zhang et al.) 2022 Cha et al.) 2022). This does not unmbigguuusy lead to the same conclusion experiment. Besides having no forgetting, concurrent training can improve the representation for each task itself since it can contrast single class with a wider variety of other classes. With our ensemble strategy, training remains task-per-task and the difference in performance with finetuning is only a conse- quence of forgetting nothing. Evaluation of the ensemble requires linearly more compute with every task, so we use as handy tool to examine what would happen if a method does not forget but do not propose it as a new method. The larger dimension of the ensemble representation can be potentially confounding. In A Appendix B we control for this using PCA dimension reduction and reach the same conclusion.
7

---

Published in Transactions on Machine Learning Research (May/2024)
Training
LP evaluation
Finetune
Knowledge Accumulation
Ensemble
Frozen Trainable
0.8
Exc
0.6
Pai
hackhone gurlon 1dd
head 1E
backhone
head
(fgoodie R2)) RR -
T
T
Tp
T
Tp
(a)
(b)
Figure 5: Schematic and result of the ensemble baseline. Panel (a) shows that training the ensemble baseline is the same as continual finetuning, but after every training phase a model copy is stored. During evaluation each stored copies produces a separate representation. Their representations are then concatenated and a linear classifier is Irained lop of the concatenation. Panel (b) compares the LP performance of ensemble and finetuning on downstream task. (Mean + SE: 5 runs)
One task improving the performance of another, effect of knowledge accumulation, is often named desideratum of continual learning (Chen & Liu 2018 yet it persists a difficult problem. Better representations should also lead to better continual learners. If one representation is better than another for data of a new task, it likely needs to change less to reach the same performance after training, thereby roducing the risk forgetting. With few samples (e.g. a replay memory), strong reprosentation can also be used to quickly recover past information, sometimes referred lo as 'fast remembering' (Hadsell el. al. 2020 Davari al. 2022). However, in Appendix D we re-emphasize that even a strong pretrained representation does not always imply mitigation of forgetting. An important step towards accomplishing those goals is preventing feature forgetting much possible without preventing models to learn.
The last sections illustrated and explained how addressing (feature) forgetting without considering knowledge accumulation and vice versa, can make representations appear to be robust against forgetting. Combining both offers a more nuanced story. Representations forget newly learned information as much at the repre- sentation level as the output level, except for information that shared across tasks. Preventing feature forgetting, as happens in our ensemble, presents an opportunity increase knowledge accumulation. The next section will look at how some established continual learning methods seize this opportunity.
5 Can feature forgetting be prevented?
Over the last years, many methods to alleviate forgetting have been proposed. In this section, we review examples of some of the main familics of methods and evaluate how they deal with feature forgetting and knowledge accumulation. The choice of algorithms is not driven by finding the best possible method, but we try to cover the most. central ideas, in their simplest form. We test replay with simple experience replay algorithr with 20 samples per class (ER), parameter regularization using Memory Aware Synapses (MAS) (Aljundi et al. 2018), functional regularization with Learning without Forgetting (LwF) (Li & Hoiem 2017)
and PackNet (Mallya Lazebnik] 2018) as an architectural method. More variations on the standard replay algorithm are reported in the Appendix.
Figure 6 shows the relative forgetting of the tested methods and Figure 7 the results of the exclusion baseline for task Te- Additionally, Table 2 reports the average learning accuracy (the accuracy immediately after training on that task) and the knowledge accumulation methods, relative forgetting in the representation is at least as bad as at the oulput level, except. for the first task. Relative forgetting in the representation for ER and MAS lower than for finetuning, yet their knowledge accumulation similar. Adding more samples accumulation and reduces overall forgetting, see Appendix. One explanation for this their lower learning accuracy; they learn less in the first place. LwF has even lower relative forgetting, and in contrast to E.R.
task downstream task. For all
the memory in ER does improve knowledge

---

Published in Transactions on Machine Learning Research (May/2024)
1.0
1.0
# 0.5
Madi
mmm
to 0.5
T2
(a) Replay
ER (20 samples / class)
(b) Parameter regularization MAS
1.0
1.0
#
0.5
0.0
T
Tg
T;
(c) Functional regularization
LwF
(d) Architectural PackNet
accuracy
Output accuracy
Figure 6: Relative Forgetting at the output level and the level of representation as in Eq. for Ti to T3 of the tested methods on the Mini-ImageNet sequence. (Mean + SE, 5 runs)
ER
MAS
LwF
0.1
PackNet
8-
Figure 7: Task exclusion results of the tested methods on T3 of the Mini-ImagetN sequence. Results with 3 > 0 show that some task-specific information is retained. (Mean + SE, runs)
and MAS, the exclusion baseline shows that remembers at least some information specific to the trained task. Its knowledge accumulation is between the ensemble and finetuning, indicating that more task-specific information is retained than with the other methods, but not everything. PackNet is an entirely different story. It does not forget, but what it learns is a lot less useful for other tasks. This is likely because by having masks specific to a single task, the representation for one task nothing like the representation for another, hindering effective learning and knowledge accumulation.
Table 2: Learning accuracy averaged over all lasks (LPi,) and knowledge accumulation on a downstteea task (I.P19.4 - LP,,a), reported for the two baselines in Section 4.3 and the met of Section 5 SE, runs)
(Mean +
Finetune
Ensemble
ER
MAS
LwF
Packnet
LP., 0.860 0.002
0.864
0.002
0.815 + 0.002
0.792
0.001
0.852 + 0.001
0.792 0.003
I.P LPa..' 0.249
0.023
0.365 0.020
0.247 + 0.018
0.259
0.013
0.321 + 0.019
0.195 0.022
6 Discussion and future work
This paper sheds light on the issues feature forgetting and knowledge accumulation and can guide future research developing new and better methods to learn representations continually. Representations of one task can help other tasks, so it might not be necessary to train later tasks as if was the first task. Part of the information will already be learned by the model, so the model should not need relearn this. Similarly, not every sample might be equally important to be part of replay memory to maintain high performance. Some of the samples can also be learned from other tasks and thus may not need to be replayed. To improve knowledge accumulation, functional regularization seems a promising candidate, and is thus recommended to be explored further. Understanding why it works relatively well and how it can be improved upon are both interesting directions for future work.
It is possible think of two tasks that have nothing in common and thus learning one task does not improve the representation for the other. Yet, tasks composed of natural images, like those in the benchmarks
9

---

Published in Transactions on Machine Learning Research MMy/2224)
we used, do share information. An open question is how much of the shared information is at the image level (e.g. detecting edges and shapes) and how much at the semantic level (e.g. being able to recognize a specific feature, like eyes or car tyres). Further unraveling these two might lead to important insights in how neural networks operate. Likewise, in this paper we always used n-way classification in both training and testing, with n constant throughout each experiment. The influence of changing problems during training or evaluation (e.g. a different amount of classes per task during evaluation than during training) on the representation quality is something that should be considered in future work.
7 Background
In this section we provide more background information related to the results in this paper. They are not necessary to understand our paper, but they may be good starting points to explore the subject further.
Representation learning. Data rarely come in a format that is adapted to the task we want to per- form (Bengio et al. 2013). Except for very simple problems, it is near impossible to directly classify images in their raw pixel representation. For a long time, rescarchers have been scarching for a represcntation of images that makes it convenient to solve semantic tasks. Handcrafting features was the standard (e.g. Beeg.[surka) iet al., 2004), but this requires expert knowledge engineering and may not result in optimal features. Since the rise of deep learning, features are more commonly learned by neural networks, directly from the raw data. Both Bengio et al. (2013) and Goodfellow et al. (2016) define good representations as ones that make it easier to solve tasks of interest, a definition we adopt. They see deep neural networks as inevitable representation learners, even when this is not explicitly the goal. Neural networks trained to predict image-label pairs indiroctly learn a representation where semantically different images are lincarly scparable in the output of the penultimate layer. Yet representations can also be learned directly, which can improve robustness, boost generalization, or reduce the need for labeled data (Jing & Tian 2020
Head vs. representation. The paper proposing iCaRL (Rebuffi et al. 2017) is one of the first continual learning works to explicilly disentangle the representation and head. The head of a model can be relatively well learned with small subsets of data, e.g. in the case of classification as a linear layer or with non-parametric approaches like k-nearest neighbors (Wang et al., 2020 Taunk et al.) 2019). On the other hand, heads often do not transfer well, and can quickly become disconnected from the representation when the representation changes while the head is static (Caccia et al.) 2021 J. In the context of continual learning this property has been identified to impact performance severcly, and methods updating the last layer only on small memories with balanced data, have shown successes in overcoming much of the observed forgetling (Wu et al. 2019
Zhao et al. 2020
Recently some continual learning methods explicitly try to foster transfer of knowledge by taking inspiration from advances in representation learning (Jing & Tian 2020). Some approaches apply contrastive losses (Cha et al.] 2021 Mai et al. 20021) and self-supervised learning (Marsocci & Scardapane) 2023 Hu et all, 2022 Fini et al. 2022, Rao et al.] 2019) to improve continual learning performance, other works take ideas
from meta-learning (Javed & White 2019 Caccia et al. 2020 to learn representations that can easily adapt to new tasks. Lastly, Pham et al. (2021) take inspiration from neuroscience and combine fast and slow learners, f.e. supervised and self-supervised modules, in one system.
Evaluating representation quality. Effectively leveraging generalization and transfer properties of deep representations is one thing, evaluating their quality is another. As pointed out throughout this work, measuring forgetting at the output (the head) of neural network does not tell us everything about the internal state of a network. Studies that retrain the last layer (Xiong et al. 2019), or a set of deeper layers (Murata et al. 2020), with the earlier layers frozen, hint that representations of lower layers are still useful for soemingly forgotten tasks. However, rather than these layers remembering something specific to the observed lasks, her works interpret this as better generalizability of the lower layers (Ramasesh el al., 2021 Yosinski et al.) 2014 Zeiler & Fergus 2014). Early layers may not seem to forget as much, because their representations are so general that they are almost fully reusable for future tasks, while deeper layers successively encode information more specific to the observed data, that is prone to being overwritten by information of new task's 's data (Ramasesh et al. 2021
10

---

Published in Transactions on Machine Learning Research (May/2024)
A number of recent works use linear probing (Alain & Bengio 2016 to analyze contimually learned repre-
sentations at the penultimate layer (Hu et al. 2022: Davari et al. 2022; Zhang et al. 2022 Chen et al. 2023 Kim & Han, 2023). Considering the amount of forgetting in the continually learning representations, Davari ct al. (2022) conclude that this forgetting is less catastrophic than at the output and suggest that no task-critical information is lost. Hu el al. (2022) and Zhhang et al. (2022) evaluate the representation 's
quality with respect downstream tasks and arrive at similar findings as Davari et al. Yoon et al. (2023)
consider a continual pre-training setting that allows fine-tuning of all parameters in the model during prob- ing. They report incrementally improving performance transferability of the features learned continually. In contrast to these studies just listed, Kim & Han (0023)
downstream task, and ascribe it to increasing
show severe forgetting in the representation, but start from models pre-trained on half the respective dataset rather than the more even split considered in Davari et al. (0022). Chen et. al. (2023) show that continual learners that have less forgetting at the representation level are better few-shot learners, hinting at the same relation we show with our ensemble bascline.
8 Conclusion
In this work we studied how deep neural networks learn and forget representations when continually trained sequence of image classification tasks. If forgetting is expressed as the proportion of newly gained performance that is forgotten, representations forget about as much at the output level, except for the first tasks. During the first task, information that is shared across tasks is learned for the first time. Because this shared information is repeated to the model in each lask, it is not forgolten. The information that is forgotten at the representation level reduces how much knowledge a model accumulates, as exemplified by the ensemble bascline. Finally, we compared the fcature forgetting and knowledge accumulation of different types of continual learning methods, whereby we found that functional regularization can prevent a large portion of representation forgetting. We hope that the insights provided by our work inspire the development of continual learning methods with less feature forgetting and more knowledge accumulation.
Broader Impact Statement
Successful continual learning can reduce the need for the ever larger growing compute required, without the need for retraining from scratch, reducing the environmental costs. It would allow fixing mistakes that machine learning models make in more efficient way and hence contribute machine learning safety, As with all machine learning applications, there always is a dual-use risk. However, given the exploratory nature of this work, we believe the risk of this particular work to be limited.
Acknowledgments
This paper is part of a project that has reccived funding from the Europcan Union under the Horizon 2020 research and innovation program (ERC project KeepOnLearning, grant agreement No. 101021347) and under Horizon Europe (Marie Sklodowska-Curie fellowship, grant agreement. No. 101067759).
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In- ternational Conference on Learning Representations (ICLR) 2017 workshop, 2016.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV), pp. 139 154, 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798- 1828, 2013.
Pictro Buzzega, Mattco Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advonces in neural information processing systems, 2020.
11

---

Published in Transactions on Machine Learning Research (May/200 (024)
Lucas Caccia, Rahaf Aljundi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. Reducing represen- tation drift in online continual learning. International Conference on Learning Represenlations (TCLR) 2022, 1(3), 2021.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccial Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, al. Online fast adaptation and knowledge accumulation (osaka): new approach to continual learning. Advances in Newral Information Processing Systems, 33:16532 1-116522-16545,, 2020.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference computer vision, pp. 9516 1516-9525, 2021.
Sungmin Cha, Dongsub Shim, Hyunwoo Kim, Moontae Lee, Honglak Lee, and Taesup Moon. Is continual learning truly learning representations continually? arXiv preprint orXiv:220608810 101101 2022.
Jiefeng Chen, Timothy Nguyen, Dilan Gorur, and Arslan Chaudhry. Is forgetting less a good inductive bias for forward transfer? In The Eleventh International Conference on Learning Representations, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Gcoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Zhiyuan Chen and Bing Liu. Lifelong machine learning, volume Springer, 2018.
Andrea Cossu, Tinne Tuytelaars, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, and Davide Baccin. Continual pre-training mitigates forgetting in language and vision. arXiv preprint arXiv:2205.090 093577 2022.
Gabriella Csurka, Christopher Dance, Lixin Fan, Jutta Willamowski, and Cédric Bray. Visual categorization with bags of keypoints. In Workshop on statistical learning in computer vision, ECCV, volume 1, pp. 1-2.
Prague, 2004.
Roea Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing repre- sentation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognation, pp. 16712-16722 16721,, 2022.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-338) 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largo-scale hicrarchical image database. 2009 IEEE conference on compuler vision and pullern recognition, pp. 248 255. Ieee, 2009.
Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. Self-supervised representation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine, 39(3): (::3):42-00 62, 2022.
Ruili Feng, Kecheng Zheng, Yukun Huang, Deli Zhao, Michael Jordan, and Zheng-Jun Zha. Rank diminishing in deep neural networks. Advances in Neural Information Processing Systems, 35:33074-33065, 2022.
Enrico Fini, Victor G Turrisi Da Costa, Xavier Gameda--Pindda Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9621-9630, 2022.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Thends in cognitive sciences, 24(12):1028-1444 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770 770-778, 2016.
12

---

Published in Transactions on Machine Learning Research MMy/2222)
Dapeng Hu, Shipeng Yan, Qizhengqiu Lu, HONG Lanqing, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng. How well does self-supervised pre-training perform with streaming data? In International Conference on Learning Representations, 2022.
Chip Huyen. Real-time machine learning: challenges and solutions, 2022. URL https:
and -od-talttisiooonnoo html# towards- continual- (Online; accessed 22-Jamuary-2024,
Khurram Javed and Martha White. Meta-learning representations for continual learning. Advances in newral information processing systems, 32, 2019.
Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, and Xiang Ren. Learn continually, generalize rapidly: Lifelong knowledge accumulation for few-shot learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 714-729, 2021.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(11):4037-4058, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661 18673, 2020.
Dongwan Kim and Bohyung Han. On the stability-plasticii dilemma of class-incremente learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 20196 20204, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL (ttt:t////aattiii wrgg www./b//////eeoooooo..oo..e
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEBE/CVF conference compuler vision and pollern recognition, pp. 2661-2677 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Kuan- Kuan-Ying Lee, Yuanyi Zhong, and Yu-Xiong Wang. Do pre-trained models benefit equally in continual learning? In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pp. 6485-6493, 2023.
Timothée Lesort, Oleksiy Ostapenko, Pau Rodríguez, Diganta Misra, Md Rifat Arefin, Laurent Charlin, and Irina Rish. Challenging common assumptions about catastrophic forgetting and knowledge accumulation. Tn Conference Lifelong Learning Agents, pp. 43-65, PMLR, 2023.
Zhizhong and Derek Hoiem. Learning without forgetting. IEEE transactions on pottern analysis and machine intelligence, 40(12))2935-290) 2017.
David Lopez-Pat Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference Learning Representations (ICLR) 2019, 2017.
Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online dass-incremental continual learning. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognilion, pp. 3589 3599, 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEF.E conference on Computer Vision and Pattern Recognition, pp. 7765 17757777, 2018.
13

---

Published in Transactions on Machine Learning Research (May/202) 2024)
Valerio Marsocci and Simone Scardapane. Continual barlow twins: continual self-supervised learning for remote sensing semantic segmentation. TEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Molivalion, volume 24, pp. 109 165. Elsevier, 1989.
Kengo Murala, Tetsuya Toyola, and Kouzou Ohara. What is happening inside a continual learning model? representation- based evaluation of representational forgetting. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Workshops, pp. 234-235, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Newral Information Processing Systems Curran Associates, Inc., 2019. URL attp://papers http://pppprrs..uurpp..... cc/ppprr/ wwwhhhheggooeeeeeeeeeeeeeeeeeeee oerenceeece--oo perrorraace--eep--lee learning library. pddf
32, pp.
9015- ho11--yyoorrdda----omtttteevvee
Quang Pham, Chenghao Liu, and Steven Hoi. Dualnet: Continual learning, fast and slow. Advances in Newral Information Processing Systems, 34:16131-1614 2021.
Vinay V Ramasesh, Ethan Dyer, and Maithra Raghu.
Anatomy of catastrophic forgetting: Hidden rep- resentations and task semantios. International Conference on Learning Representations (ICLR) 2020, 2021.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascamu, Yee Whye Tch, and Raia Hadsell. Continual unsupervised representation learning. Advances in Newral Information Processing Systems, 32, 2019.
Roger RatelifT. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological Review, 97(2):285 2885-308,, 1990.
Aolvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Kashvi Taunk, Sanjukia De, Srishti Verma, and Aleena Swetapadma. A brief review of nearest neighbor algorithm for learning and classification. In 2019 International Conference on Intelligent Computing and Control Systems (ICCS), pp. 1255-1260. IEEE, 2019.
Atharva Tendle and Mohammad Rashedul Hasan. A study of the generalizability of self-supervised repre- sentations. Machine Learning with Applications, 6:100124, 2021.
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, 4(12):1185-1199) 2022.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advonces in neural information processing systems, 29, 2016.
Joshua T Vogelstcin, Jayanta Dey, Hayden S Helm, Will LeVine, Ronak D Mchta, Tyler M Tomita, Haoyin Xu, Ali Geisa, Qingyang Wang, Gido M van de Ven, et al. Representation ensembling for synergistic lifelong learning with quasilincar complexity. arXiv preprint arXiv:2004. 12908, 2020.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory, method and application. arXiv preprint or arXiv:2302 00487, 2023.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM compuling surveys (esur), 53(3):1 34, 2020.
14

---

Published in Transactions on Machine Learning Research (May/200 (024)
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEFE//VV CVF Conference on Computer Vision and Pattern Recognition, pp. 374 382, 2019.
Yuwen Xiong, Mengye Ren, and Raquel Urtasun. Learning to remember from a multi-task teacher. arviv preprint arX:t:1910.0444 04660,, 2019.
Shipeng Yan, Jiangwei Xie, and Xuming He. Der: Dynamically expandable representation for class incremen- tal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014-3023, 2021.
Jaehong Yoon, Sung Ju Hwang, and Yue Cao. Continual learners are incremental model generalizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (cds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 40129 40146. PMLR, 23 29 Jul 2023. URL https://
proceedings mlr, press/v202/yoon2333 html.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Aduances neural information processing systems, 27, 2014.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310 12320. PMLR, 2021.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer EEsV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 1 13, pp. 818 833. Springer, 2014.
Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning on pre-trained model. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023.
Xiao Zhang, Dejing Dou, and Wu. Feature forgetting in continual representation learning. arXie preprint arXiv:2205. 13359v1, 2022.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness class incremental learning. In Proceedings of the IEEE/CVP conference on compuler vision and pattern recognition, pp. 13208-13214 2020.
15

---

Published in Transactions on Machine Learning Research (May/202))
Appendix
The Appendix contains additional information for the content presented in the main body. The first two sections provide further detail of our experimentation settings and the ensemble baseline. Thereafter, we report on additional experiments: First, regarding continual representation learning with supervised con- trastive and a self-supervised training approaches, and second, starting from a higher quality representation obtained pre-trained weights, rather than training from-scratch. Further we provide the full plots for rel- alive forgetting that were not used in the main body due to spatial constraints, re-evaluation of all our experimentation the CIFAR-100 dataset, well as consideration of an alternative probing mechanism to quantify representation quality, namely k-NN. Finally, we list detailed task sequence information to improve reproducability of our results. In summary, the the supplementary material is ordered as follows:
A
Experimentation details
Eusemble: further details
Supervised contrastive and self-supervised Barlow Twins
Pre-trained representations
Relative forgetting: more results with replay
P Relative lorgetting: extra results
Results on CIFAR-100
H
Evaluation with k-NN
Detailed task sequence information
A Experimentation details
This section details the training and evaluation of all experiments in the main paper and supplemental material, unless explicitly stated to deviate.
Data MinilmageNet consists of 50, 000 train and 10, 000 test RGB-images of resolution 84 x 84 equally divided over 100 classes. We split this dataset into 20 disjoint tasks such that each task contains five classes. The second benchmark is Split CIFAR-100, which is based on the CIFAR-100 dataset (Krizhevsky et al. 2009) with the same amount of RGB-images and classes as MinilmageNNet, but with reduced resolution of x 32. We split this dataset into ten disjoint tasks with ten classes each. All experiments are run with five different seeds that also shuffle the class splits over the tasks. See Table 4 and Table for the exact sequences.
Architecture and optimization Throughout this work ResNet-18 He et al. 2016 is the base architec- ture for all models. For MinilmageNet we adopt the implementation as default in the pytorch-torchvisson (Paszke et al.] 2019) library. For CIFAR-100 we employed the slim version of the model as proposed by Lopez-Paz & Ranzato (2017). All networks are trained from scratch, not explicitly specified otherwise. The optimization schodules are adjusted with respoct the training critcrion. For supervised training with the cross-entropy loss we use AdamW (Loshchilov & Hutter 2017) optimizer with static learning rate of 0.001, weight decay 0.0005, and beta-values 0.9 and 0.999. Each task is trained for 50 epochs with mini-batches of size 128.
For the SupCon (Khosla et al. 2020) and Barlow Twins Zbonlar el al. 2021 optimization criteria, we stuck to optimization schedules proposed in literature for their application to continual learning. line with obsorvations by Cha al. (2021), the SupCon training regime uses an SGD optimizer with momentum 0.9. The learning rate is scheduled in the same way for every task warming up from 0.0005 0.1 in the first ten epochs, then annealing by cosine schedule back to its starting value. The first task is trained for
16

---

Published in Transactions on Machine Learning Research (May/2024) (024)
500 epochs, all subsequent tasks for 100 epochs, with batch size of 256. The projection network necessary for this objective consists of an MLP with (single) hidden dimension of 512, projecting to 128 dimensional space. Barlow-Twins optimization is aligned to (Marsocci & Scardapane 2023; Fini et al. 2022). We use an Adam optimizer (Kingma & Ba 2015) with learning rate 0.0001 and weight decay 0.0005. We train 500 epochs for each lask with batch size of 256. Again, the projection head is an MLP but with two hidden layers, and hidden and final projection dimension of 2048. All methods use the same augmentations, see below.
Probe optimization To quantify the quality of the representation we apply probes based on linear clas- sifiers and k-nearest neighbors (ENN). Linear classifiers consist of single lincar layer and are optimizod with access to all training data, in way analog to Cha et al. (2021). Keeping a batch-size of 128, we use SGD with momentum of 0.9 and no weight decay for 100 epochs. The learning rate of 0.1 is decaying epochs 60, 75, and 90 by factor of 0.2. Similarly, kNN uses all training data to evaluate the reprosentations, with k- = 20.
Continual learning mechanisms LwF and MAS are using value of A = 1.0 advocated by its original authors. Replay uses random selection of 20 exemplars per class. The weight of the loss on replayed samples is increased proportionally to the number of previously observed tasks, to prevent favoring the current task in the optimization. PackNet prunes 75% of the model's parameters in each layer, followed by a post-pruning phase of 25 epochs (1.5x training epochs), similar to the settings reported in its original work. An upper bound is reported by jointly training the model on all observed data. For our lower-bound we want document the impact the singled out tasks have. This we achieve by re-initializing the model before training new task, but allowing the new task to train for as many iterations as continual model would have, e.g. 50 epochs for the first lask, then 100 for the second, and on. By design this model has zero transfer of knowledge, and we will refer to it as 'Single task' baseline.
Augmentations In all experiments use the data augmentation pipcline from SimCLR CChn et al. 2020). The augmenlations pipeline consists of random crops and horizontal flips, color-jitter (brightness=O0) contrast=0.4, saturation=0.2, hue=0.1), random grayscaling (p=20%) and Gaussian blur using a kernel of size 9 and sigma range 0.1 to 0.2. In PyTorch, the augmentations are defined as follows:
fron torchvision.co transforns Import
Sandoxllesieeddcoo(lle---- 52), ecale=(0.2, 1.0)), RandonApply(
[ColorJitter (brghtteerr,,,,,,,ee.......,, co saturation=0.2, 2, hue=O.1)], pm0.0),
fandortrayscaala(spD... RandonApply((
Gauasian (awrnallaazeeiiiiaaaeee sizzerinuut_azzen (ttpppput_sttaa0///022///// [0] sigma=(0.1, 2.0)]], p=0.6)
B Ensemble: further details
The ensemble method stores a model copy after every task's training. Each of these models output a representation fi with dimension & for the input data. During training, only the model of the current, i-th, task is used and the others are frozen. During training, the model learns exactly in the same way as the default continually fine-tuned model (c.f. Section 2) For example, when training on the n-th task, the n-th sub-model of the ensemble baseline ( the sub-model being trained on the n-th task), is precisely the same as the fine-tuned model when it is being trained the n-th task. The difference is in the evaluation. During evaluation, not only the latest model's representation ft is used, but all of the representations are concatenated to form one large representation f = /2 ... ft]. ... same, all "intermediate models" (t..f) do not contain any information that was not at some point also Icarned by the finc-tuned model. Hence, the ensemble model does not use any additional information; is keeping track of information that is forgotten by the fine-tuned model. On top of this large representation f. linear layer with input dimension lk is trained, instead of just k for the finetuned model.
Note that because learning was exactly the
17

---

Published in Transactions on Machine Learning Research May/2024)
To mitigate the influence of the higher dimension, for which it might be easier to find linearly separable features, we add dimension reduction to lower the dimension back to k. We do this by projecting the features of the ensemble on the top-k most significant PCA dimensions. The results are shown in Figure S The reduced ensemble performs slightly worse than the full ensemble, yet still significantly better than fineluning.
0.8
La
0.6
Finetune
Pinetune ensemble Finecune ensemble red.
T
T2
T
T
T
Figure 8: L.Paacuraciess of finetuning, the ensemble baseline and its reduced version, explain in Section
C Supervised contrastive and supervupd Barlow Twins
In this section, we extent investigation of the apparent innate robustness to forgetting for contimually learned representations from supervised fine-tuning with the cross-entropy (CE) loss (c.f. Section 4 to the alternative training approaches of contrastive learning and self-supervised learning. The recent success (Ericsson et al. 2022) of these learning mechanisms sparked interest for their application in continual learning
as well (Fini et al. 2022 Cha ct al.) 2021 Marsocci & Scardapanc, 2023 Wang et al., 2023), and they have been found to be less prone to lorgetting Wang et al. (2023). which gol linked to the improved generality of the learned representation (Tendle & Hasan 2021). From the plethora of potential methods, we chose SupCon (Khosla et al., 2020) and Barlow Twins (Zbontar et al. 2021 as representatives of contrastive and self-supervised losses. Both have already been used in continual learning settings (Marsocci & Scardapane,
Cha et al. 2021 Davari et al.] 2022 Cha et al. 2022). In the following, we repeat. to quantify the quality of a learned representation by its linear probing (LP) accuracy on the tasks of the continual training sequence T, as well as a downstream task Ta that is never seen during training. First, we provide an intuition to the continual learning performance of SupCon and Barlow Twins in contrast to supervised CE fine-tuning. Then, we analyze forgetting analog to the main text by relative forgetting (c.f. Section 4.1 and task exclusion difference (c.f. Section 1.2p. Finally, we again link the effect of feature forgetting to reduced knowledge accumulation efficiency using the ensemble baseline (c.f. Section 4.3). before summarizing the results.
2023
Learning and forgetting SupCon and Barlow Twins representations To provide a frame-of- reference first relate the linear probing (LP) accuracies of continual SupCon, Barlow Twins, and fine- tuming with supervised CE loss in Figure 9 This qualitative comparison provides intuition their evolving representation quality on the training sequence and downstream task. Continual SupCon almost resembles the continual supervised CE fine-tuning baseline. It displays prominent peaks and decay after, as well as some knowledge accumulation for the downstream task. However, in our experiments contimual SupCon falls short of the performance obtained from CE fine-tuning. With continual Barlow Twins, the increase performance from training and evaluating on the same task (LPx,.) is comparatively limited. Yet, the ob- tained representations are more general regarding other tasks in the sequence and show increased knowledge accumulation.
We introduced relative forgetting in Section 4.1 of the main body as an evaluation perspective that reveals how forgetting at the output level and representation level is equally catastrophic once we consider the knowledge already contained in our continual learning model. For SupCon and Barlow Twins a direct comparison is not possible, since there no output performance to be mcasured. Both require second (head-) training stage, to apply the learned representation a task by annotated data. Still, the relative forgetting plots in figure TO offer a second perspective to theevolution of representation quality in figure
18

---

Published in Transactions on Machine Learning Research (May/2024)
Evaluation on T
Evaluation downstream task
0.9
0.9
LP.
0.7
Supervized CE
0.8
Barlow
0.6
Supcun
0.5
T,
T.
Figure 9: Illustration of feature forgetting and knowledge accumulation for continually finetuned Su and Barlow Twins on Split MiniImageNet, in comparison continual supervised finetuning using the Cross- Entropy loss (i.e. finetuning in the main paper).
1.5
1.5
1.0
1.0
#
-0.5
-0.5
T,
(a) Self-supervised learning Barlow
(b) Contrastive learning SupCon
Figure 10: Forgetting for the first 5 tasks using sule-super and contrastive losses, instead of the default cross-entropy. Output accuracy is not properly defined for these methods, they always have to train a final linear layer. (Mean + standard error)
SupCon rapidly forgets newly learned information from each task in continued training, but still achieves a certain degree knowledge accumulation. Barlow Twins demonstrates more unstable relative forgetting, but ultimately shows the same trend. The information gain for any task is not as pronounced as when training with additional labels provided for the data. Despite their increased transferability, features can become quickly forgotten to the extent that the accumulating knowledge surpasses early training performance for initial tasks.
Also the task exclusion difference (EXC1,), c.f. Eq. 2p, depicted in figure 11 confirms the previous observa- tions. The task exclusion measure allows to study the information gain with respect to a particular task, while accounting for the knowledge accumulation with respect to all other tasks in the training sequence. For both, SupCon and Barlow Twins, the information gain from any individual tasks is marginal. At the end of continual training the gap on average reduced to almost zero.
Feature forgetting reduces knowledge accumulation Lastly, to study the impact of the observed feature forgetting to knowledge accumulation, we re-employ the ensemble baseline introduced in Section 4.3 It allows to quantify the knowledge accunulation without the impact of forgetting throughout continual training. From the comparison between contimually fine-tuning and its ensemble counterpart in Figure I2
we can find an ample increase in the quality of the representation for SupCon and Barlow Twins when using
EXCLB
EXC,.1
EXC;
EXC₁nı
0.1
0.1
0.1
0.1
(a) Self-supervised learning Barlow
(b) Contrastive learning SupCon
Figure 11: Task exclusion difference for continually fine-tuned SupCon and Barlow Twins on Split MiniTm- ageNet for tasks To and Tu
19

---

Published in Transactions on Machine Learning Research (May/2024)
0.9
0.8
0.8
0.7
Barlow
Supcon
0.6
Barlow ensemble
0.6
Supcon ensemble
Barlow ensemble red.
Supcon ensemble red.
T To
T
T.
T
T2
T₂
Figure 12: Ensemble baseline comparison for SupCon and Barlow Twins. Ensemble referes the full ensemble, 'ensemble red. is the ensemble that has its dimension reduced again by PCA.
the ensemble. This clearly shows the impeding effect of feature forgetting the representation learning also for supervised contrastive and self-supervised learning, We note that Barlow Twins does suffer a lot more from correcting for the ensemble's dimensionality increased through PCA, compared to the supervised approaches. An explanation to this might. be in the definition of the Barlow Twins' loss (Zbontar et al. 2021). Because it explicitly optimizes towards a high rank representation, each task's representation should spread a high number of significant components. Assuming considorable quantity of information is not shared between tasks, as result, reducing the concatenated representations by keeping its most significant components does not maintain all tasks' information. Supervised approaches suffer a lot less in this regard since they typically employ few significant components (Feng et al. 2022) in the representation space. This leaves enough effective encoding space for information from all tasks.
Do they forget catastrophically? In summary, confirm similar trends for relative forgetting and task exclusion difference with SupCon and Barlow Twins as we detailed for continual supervised CE fine-tuning in our main results. Also, using the ensemble baseline, we demonstrate that feature forgetting hinders efficient continual representation learning for these methods as well. However, we choose not to claim a definite conclusion on whether forgetting should be considered calastrophic or not. This is for the reason that we cannot link the common expression of catastrophic forgetting of supervised CE settings to SupCon or Barlow Btwins directly, as did in the main body, due to the there being no meaningful output accuracy measure. We emphasize that the results listed in this section suggest that, forgetLing is indifferent to the training loss and supervision. What does change is the "kind" of features extracted from the data by the specific learning mechanism approaches such as Barlow Twins may render features to be more similar across tasks. We exemplified how this is giving the impression less forgetting, but can similarly be explained by the repeated re-exposure of the model the specific feature(s).
D Pre-trained representations
Similarly to representations being attributed an innate robustness to forgetting when lcarning continually, pre-trained backbones receive this reputation as well (Lee el al.) 2023 Zhang et al. 2023). In light of our study, here we also consider the extent to which representations considered "high quality", such as obtained from largo-scale pre-training (Kornblith al. 2019), impact forgetting at representation level. For this experiment we simply exchange the random initialization that we have been using through the main body with weights from backbone pre-trained ImageNet1k (Deng et al. 2009) An innate robustness to forgetting of the representation should manifest overall improved performance due to the increased generality and/or quality of the pre-trained features. Note that we exemplarily investigate an extreme case where pre-training (ImageNet1k) coveres an almost. identical data distribution as the continual training sequence (Split Mini-Imagenet).
Little surprisingly, the results in Figure show that, for our case of Split of Mini-Imagenan the representa- tion of the pre-trained backbone (To) is already a good foundation for the task sequence. Nevertheless, with soquential training the initial generality is quickly lost and again observe the training dynamics governed
2 weightt publicly available in the Py
library
20

---

Published in Transactions on Machine Learning Research (May/202))
0.9
0.8
0.7
Random Pretrain
0.5
Tw
Tn
Figure 13: Comparison of continual learning linear probing (LP) when training from-scratch (random weight
initialization) or pre-trained weights (ImageNet-Ik).
by feature forgetting and knowledge accumulation We hepoth that, even though the features obtained from ImageNet pre-training are well suited for the sequential training task, they cannot be similarly obtained from any of the tasks in the continual training sequence. As a results, they are forgotten despile their generality. Our finding aligns with the work of Kim & Han 2023), who show that continual learning approaches making effective use pre-trained representa- tions either drastically reduce their learner's plasticity or otherwise extend the initial representation without altering it, to circumvent forgetting.
describe them in the main body (c.f. Section 4
E Relative forgetting: more replay results
In Figure wc show relative forgetting on the Mini-ImageNet sequence when storing 20, 50 or 100 samples per class, and when adding distillation using the output logits (DER (Buzzega al. 2020)): Figure 15 shows the exclusion results of Tg and Table 3 the amount of knowledge accumulation. Having more samples in the memory reduces both forgetting in the representation and the head and substantially improves knowledge accumulation. Interestingly, the final exclusion difference remains small. This supports our theory. When adding more replay samples the model can learn more from the combination of replay and new task data, but what cannot be learned from this combination data still lost. DER is mostly helpful to further roduce forgetting in the head. When storing the output logits for cach sample, more information is stored, and is possible that this reduces the risk of overfitting of the head on the few samples that are in the memory.
1.0
#
Ma
mMm
mam
Cance
my mu
m
T.
T.
T
(a) Replay- ER (20 samples / class)
(b) Replay
DER Buzzega et al. (20 samples class)
1.0
ww
AAAA
Aumey
O0
T,
T
T,
T,
(c) Replay- ER (50 samples/
/ class)
(d) Replay
ER (100 samples / class)
accuracy
Output accuracy
Figure 14: Relative Forgetting at the output level and the level of representation as in Eq. L for Ti to Tb of more replay variants on the Mini-ImageNet sequence. The inffuence of storing more samples and adding dist illation with soft logits (DER (Buzzega al. 2020)) evaluated. (Mean + SE, runs)
21

---

Published in Transactions on Machine Learning Research (May/2024)
(n- 0.1
50)
100)
(n 20)
BXCa
0.1
0.1
0.0
0.0
0.0
0.0
Figure 15: Task exclusion results of more replay variants the Mini Image-m sequence. The influence
of storing more samples and adding distillation with soft logits (DER (Buzzega et al. 2020 is evaluated.
(Mean + SE, 5 runs)
Table 3: Learning accuracy averaged over all tasks (LP;,) and knowledge accumulation on a downstream task (LP19.4 LPo,)), reported when adding more replay samples and with DER [uzzzga et al. 2020 (Mean + SE, runs)
Finelune
Ensemble
ER (ix 20)
ER (n = 50)
ER (x 100)
DER (n = 20)
0.860 0.002
0.864 + 0.002
0.815 + 0.002
0.814
0.003
0.821
0.003
0.826 0.003
LP LPo.4
0.249
0.023
0.365 = 0.020
0.247 + 0.018
0.298
0.014
0.309
0.015
0.271 + 0.013
F
Relative forgetting: extra results
Figure only shows the relative forgetting in the Mini- Imai-ImaeNeet task sequence. For completeness, we report here the relative forgetting of all tasks in Figure 16
1.0
T,
T,
1.0
t 0.5
Tp Tp
T
Tis
Tp
Tp
LP acouracy
Ourgut accuracy
Figure 16: Representation and observed lorgetting using linear probes for all tasks in Mini-ImageNee using finetuning (except first and last, for which we cannot calculate relative forgetting)
22

---

Published in Transactions on Machine Learning Research (May/2024)
G Results on CIFAR-100
To reduce the dependency only having experiments single dataset, report main results here also on CIFAR-100. The results CIFAR-100 100 follow the same general trends as those Mini- Miai--tageNet in the main paper. The largest difference is that the effects are sometimes smaller, due to the shorter task sequence.
1.00
accuracy
Output accuracy
r-accuracy
0.75
0.50
0.25
0.00
Figure 17: Linear probe and output accuracy of T4 during the entire CIFAR-100 sequence.
1.0
0.0
acearacy
Output accaracy
Figure 18: Representation and observed forgetting using linear probes for all tasks in CIFAR-100 using finetuning (except first and last task, for which cannot calculate relative forgetting)
0.8
0.8
LP.
0.6
0.6
Finetune
Ensemble
0.4
T,
T
T
T
Figure 19: Finetune, exclusion, single task and multi task with CIFAR-100.
Figure 20: Comparing the ensemble and finetuning on CIFAR-100.
LP
0.6
Finetune
MAS
Replay
T.
T.
Figure 21: LP-accuracies downstream task of CIFAR-100.
23

---

Published in Transactions on Machine Learning Research (May/2024)
H Evaluation with kNN
In this section we report the most important results of the main paper using KNN instead of using linear probes. This has the benefit. that it. there are no hyper-parameters to tune and does not depend on the optimization used. We report it here for completeness, and keep the linear probes in the main paper as this is how previous papers reported their results Davari et al. (2022); Cha et al. (2022) Zhang et. al. (002)). In general, the results in Figure 22
23 24
and 25 follow the same trends observed in the main paper, with
the main difference that the absolute values are lower, likely due to the suboptimalitiy of ANN compared to linear probe optimization.
1.00
0.75
0.50
0.25
0.00
T
T
T
T 15
T
10
19
KNN accuracy
Output accuracy
Figure 22: KNN and output accuracy of T7 during the entire Mini-ImageNee sequence.
T.
T.
T.
T.
T.
T
T2
1.0
t 0.5
0.0
T3
Tg
To
T,
Tp:
T24
T
KNN accuracy
Output accuracy
Figure 23: Representation and observed forgetting using ANN for all tasks in Mini Mini-Imagee ImageNet (except last and first, for which cannot calculate relative forgetting)
0.8
0.8
Ensemble
1,
kNN,a
0.6
0.6
0.4
0.4
T
T2
T
T
T
T.
T
T
Tg
T
Figure 24: Finetune, exclusion, single task and multi task using KNN.
Figure 25:
Comparing the ensemble and finetuning using ANN,
24

---

Published in Transactions on Machine Learning Research (May/2024)
Detailed task sequence information
In Table f and Table 5 we report the exact task sequences used in the experiments in the main paper. These are chosen at random, but consistent in all experiments. The randomness of the tasks also means that there difficult varies quite a bit, which explais some of the higher standard errors in the experiments reported.
Synset
Synset name
Synses
Synset name
idx
Synset
Synset name
x015522529
house funch rubin
33
ad33-002221
frying pan
66
mU12981792
calamaran
x01558993
34
n034776684
hair slide
67
u039980874
poncho miniskirt
n01 101704323
triceratops
35
n03527444
holster lipetick
68
n03770439
x917499839
green mamba
36
4833666183
69
a0/2091244
Ibizan hound white wolf
x01770081
harvetman
37
n03835899
oboe
70
u02114558
n0 843388
38
n03554065
organ
71
n02174001
rhinoerros. beetle
x019100747
jellyfsh
39
a133888605
paralled bars
72
n033417042
garbage truck
n02074337
dugong
40
x03908618
pencill box
73
n02971336
carton
n02089867
Walker hound
41
m03924679
photocopier
74
n03584254
iPod
1020011831
Saluki
42
x039998194
prayer rug reel slot
75
a02138444
meeraat
10
002000006
Gordon setter
43
n04067472
76
=03773504
missile
11
n021 n02105505
komondor
44
n04243546
77
n02950826
camnon
12
0922108089
buxer
45
nd4251144
suorkel
78
n01855672
guuse
13
x021188551
Tibetan mastiff
46
n04258138
solar dish
79
n09256479
coral roef
14
n02108915
French French_bulldogg
47
n04275548
spider_ weh
80
n021 10311
dulmatian memalode
15
u0221 10277
Newfoundland
48
n04296562
stage
81
s01930112
16
13712
miniature poodle
49
n04389033
tank
82
n02219446
17
n021 20079
Aretic fox Iudybug
50
n04435653
tile roof
83
n02443484
black-footed forret
18
65:456
51
n044433257
tobacco shop
84
u01981276
king crab liom
19
n02457408
threp-taed sloth
52
n04509417
unieyele
85
n021 29165
20
n02606052
rock beauty
53
n04515003
upright wuk
86
n04522168
vase
21
n02287112
aireratt carrier
54
n04596742
87
a02093601
gulden retriever
22
n02747177
ashean
55
n04604644
worm frmoe
88
n03775546
mixing__ooo bowl
23
n02795169
barrel
56
n04612504
yau!
89
n021
malamute
24
n0/2523428
bottle
57
n06794110
street sign
90
n02116738
African humning dog
25
n02966193
carousel
58
n07584110
consomme
91
n03146219
euirass
26
n03017168
chime
59
n07697537
hotdog
92
n02871525
bookshop
27
n03044660
clog
60
n07747607
orange
93
n03127925
crate
28
n03062245
cocktail shaker
n00246464
cliff
94
n03544143
hourglass
29
n03207743
dishrag
62
n130544500
bolete
95
n03272010
electric guitar
30
n03220553
dome
63
n13133661
96
n07613180
trifle
31
nO3337140
File
64
nO3533780
horizontal bar
97
n04146614
32
n033-17037
fire sereen
65
n03076370
combination_ Jock
98
n04418357
theater currain
Table 4: The classes included in Split Minilmagenet, with their index, (which is not general, but used in the task splits), their synsets and their name.
Seed
Seed
Seed 62
72
T1
83
53
70
d5
44
82
8
44
19
2
76
48
62
80
29
76
82
43
16
84
72 33
38
2
55
T2
39
22
80
10
75
97
89
18
90-
60
80
30
68
95
22
84
54
75
28
40
92
24
14
25
T4
76
77
12
31
95
68
40
46
54
92
18
63
14
24
23
91
15
55
88
26
42
69
40
72
56
36
51
6- 82
77
27
63
23
76
9-
00
27
64
68
6
92
18
T6
15
40
96
72
36
13
17
28
27
29
94
26- 94
38
85
77-
98-
96
18
5-
86
66
86
16
58
62
87
02-
95
G1 43
93-
15-
28
58 30
97
68 88
14
50-
T9
40
34
7-
70 12
36
42
76
31
56 98
86-
24 61
36-
T11
3-
38
17 96
9-
20- 90
50
84-
42
0-
17-
T12
78
64
96
89
98 - 68
70 -0-077-00
64-
44
70- 18
50
400-40-000
66
94 37
5- 26
T15
56
99
31 15
75
25-22-
95
41
68
73
37
8
60 20
23
T15
79
41
20
77
78
71
16
52
71
72
17
54
15
67
18
51
7
82
83
T16
57
75
32
94
59
22
30
63
76
85
57
12
13
66
31
52
61
34-
19 68
9
9-
52
T17
84
37
29
78
21
91
83
28
16
69
87
37
119
91-
74
86
20
35
98
69
99
45
30
17
58
88
29
7-
26
79
623
78
Dowastream task
60
71
14
92
81
86
23
18
28
11
81
24
18
94
74
46
19
88
77
27
99
86
Table 5: Task splits used in the results with Split Minilmagenet. The indices correspond to the classes listed in Table 4 Results reported on Split Minilmagenet average over these 5, randomly determined, task
sequences.
25


Reviews:
[{"review_id": "jWYXL7KT4K", "reviewer": "Reviewer_VqjT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper is well-written and quite easy to follow.\n\nI like the figure 2, which makes the point of this \"relative forgetting\" metric quite clear. I think it is interesting to ponder forgetting by the relative performance gain that the task provides, which might be not significant for LP.\n\nThe figures are well explainable of what the authors aim to show.\n\nThe experiments are well conducted and are convincing to make the point of the authors.\n\n**Weaknesses:**\n\ngeneral weaknesses:\n- the related work offers a list of related literature however it is not used as context for the reader to understand what the proposed paper is about. There is no link made between the literature and the proposed contribution, which misses the point of related work. Also as a paper about knowledge accumulation (as the title suggests currently) related work section on knowledge accumulation would make sense.\nFor example, the paper cites,  [Caccia et al., 2022] paper which is about representation drift, it sounds like it is a closely related paper that the reader should be aware of as context for reading the whole paper.\n\n\n- it is not always clear how each section serves the purpose of the paper, I missed it for section 4.2 for example. Some small introductions/conclusions for subsections would help to keep the reader on board with how each section is articulated to serve a specific purpose.\n\nWriting weaknesses:\nSome formulations are a bit vague and should be made clearer in particular when it is in the abstract.\n\n- [abstract] > Multiple recent studies ascribe representations a certain level of innate robustness against forgetting - that they only forget minimally and no critical information.\n-> The whole sentence and in particular \"no critical information\" sounds like an overclaim about what is written in the literature here, I believe only one of the cited paper really claim such a bold thing while the others just claim that forgetting is quite limited with respect to usual catastrophic forgetting.\n\n- [abstract] >  \"Even though it is true that feature forget-\nting can be small in absolute terms, newly learned information tends to be forgotten just\nas catastrophically at the level of the representation as it is at the output level\"\n-> unclear, does the author want to make a distinction between initially learned features and the one that are learned later? What is the relationship between information, features and representation here?\n\n-> One claimed contribution is \"We show that forgetting of the representation hinders knowledge accumulation\". I do not understand, in such a formulation, how it could be otherwise. How forgetting could not be a knowledge accumulation issue? Can you comment on that, please?\n\n- Also, in section 3, you are citing [Zang et al 2022] to say \"had observed this upward trend and concluded that feature forgetting does not pose an issue for learning representations continually\".\n-> One of the listed contributions of [Zang et al 2022] is \"We show interpretable dynamics of feature learning and that feature forgetting can significantly impair performance\". Can you comment on that? In my understanding, [Zang et al 2022] claims that \"representation learning and catastrophic forgetting are largely separate issues\", they do not say representation forgetting is not an issue.\n\n- [4.3] \"A better representation can make learning easier\"\n-> I think a better representation that makes learning harder would not be better anymore, what is the point of this claim?\n\n\n- [Discussion] \"It is possible to think of two tasks that have nothing in common and thus have no knowledge accumulation\"\n->Learning a new task that brings no improvement to the former is not proof that the model does not have more knowledge than before.\nIf your concept of knowledge accumulation assumes that indeed such a case is a situation of 'no knowledge accumulation' this concept should be renamed because it is misleading.\n\n\n- [4.3] \"Previous sections showed that there is severe feature forgetting\"\n-> I think saying that \"feature forgetting severely impacts newly learned features\" would be more in line with what was shown in the previous section.\n\n\n\n\nPositioning weaknesses:\n\nMy biggest concern is about the positioning of the paper in the literature.\nBeyond the fact that the related work is in the end and is not used to put the paper in the context of the literature.\n\n- Despite the few papers cited in the introduction by the authors (Davari et al., 2022; Zhang et al., 2022; Hu et al., 2022; Wang et al., 2023),\nthe whole literature on task incremental learning shows the issue of feature forgetting.\nTask incremental learning is only about feature forgetting.\nHow does the author experiment in section 5 show anything different than what task incremental learning would show?\n\n\n- [related work] The paper title starts with \"Knowledge Accumulation in Continually Learned Representation\" A section on knowledge accumulation in related work should appear. At least, to compare definitions of knowledge accumulation and the difference of approaches, for example in \"Challenging Common Assumptions about Catastrophic Forgetting and Knowledge Accumulation\" Lesort et al, CoLLas 2023, or in [Caccia et al 2020] that you cite in section 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of this paper is to assess that when looking at performance increase and decrease when learning a task and then forgetting is linear probed performance forgetting is similar to output forgetting.\nThey propose to use the \"relative forgetting\" metric to assess their claim.\nAnd they show that this is indeed the case for four approaches published before 2019.\n\nThis contribution put a grain of salt into recent publications that could lead to the idea that feature forgetting was less problematic than it could appear.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While I believe this paper could be an interesting reading for anyone willing to understand the relationship between feature forgetting and catastrophic forgetting, the current state of the paper is misleading toward the positioning of the paper in the existing literature.\n\nHence, the positioning of the paper should be reworked because it currently does not reflect where the paper stands in the space of continual learning literature. \nFirst, the literature opinion should not be assumed from a couple of papers.\nWhile I think it is important to have new articles adding a grain of salt to recent publications, the link between such articles and other literature has to be made more in-depth and more clearly.\nIn my understanding, the paper aims at clarifying and quantifying the distinction between feature forgetting and catastrophic forgetting, which is important.\nHowever, it does not discover that feature forgetting is a thing that prevents knowledge accumulation, because that is what literature on task incremental learning is about.\nConsequently, my change request is essentially to make more clear that this paper is about understanding better claims about low feature forgetting and existing literature.\n\nAlso, a slight rework on introducing/concluding each part would be necessary to follow easily how each section relates to each other.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zF7JaBPwiT", "reviewer": "Reviewer_xXFR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In my opinion, this is an excellent, carefully performed study of representation forgetting \n\nStrengths:\n- The paper studies an important problem, namely the forgetting of representations in CL.\n- The main observations made in this paper are quite interesting and could be useful for further studies on continual learning.\n- The experiments are thorough and, as far as I can tell, carefully done. I particularly appreciate the ablation of the ensemble experiment provided in Appendix B and additional experiments on Cifar-100 in Appendix F.\n- To the best of my knowledge, the related work is properly discussed.\n- The paper is clearly written and the authors properly explain their chain of thought.\n\nWeaknesses:\n- This is more of a “nice-to-have” than an actual weakness, but I would like to see more experiments on the experience replay in Section 5:\n    - ER with 20 samples per class is not that many. I would be curious to see what happens as we increase the number of samples -- in the limit, we should achieve something similar to multi-task learning.\n    - I think using ER + some kind of distillation could be also interesting (e.g. Dark Experience Replay) to see whether it impacts the representation forgetting. LwF already is a distillation method, but I think a more thorough investigation here could lead to further insights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the question of representation forgetting in continual learning. The authors reevaluate the methodology used in previous papers of how to use linear probes to measure forgetting. In particular, they notice that the baseline linear probing performance is high even before starting training on a given task. As such, the absolute drop of linear probing accuracy is not high, even though the relative drop is quite significant. The authors then compare the performance of fine-tuning with an ensemble-like approach to demonstrate that there is in fact a significant gap in terms of knowledge accumulation. Finally, the authors check how different CL methods impact the dynamics of knowledge accumulation and forgetting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggested some additional experiments in the weaknesses section above, but it's more of a \"nice-to-have\". I think it is a solid study as it is.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YQHqa5JEtj", "reviewer": "Reviewer_DdoQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. The paper presents a timely and nuanced discussion on CL forgetting, which offers potential insights to the community for developing new CL algorithms.\n\nWeakness\n1. The setting considered in this work is limited to task-based CL, a restrictive setting requiring task ID to be present at test time. Class-incremental CL is a more general setting and the focus of more recent works (e.g., [1, 2]).\n2. The paper does not offer concrete proposals for mitigating forgetting based on the findings. While I agree with the general finding on forgetting at representational level, it is unclear how this issue can be tackled effectively. The paper in this sense appears incomplete and offers limited contribution.\n3. Technically, $r_{j-1, j}$  for the relative forgetting metric appears problematic. For $r_{i, j}=OUT_{i, j}$, $i$ should be be equal or larger than $j$, since the classification head $\\theta_{h_j}$ does not exist at $f_{j-1}$, which has not seen the task j. Thus it is unclear what the appropriate interpretation for OUT_{j-1, j} is. In contrast, $r_{j-1, j} makes sense only for linear probing since the classifier head is computed on-the-fly for each task.\n\n\n[1] https://arxiv.org/abs/2210.05561\n\n[2] https://arxiv.org/abs/2112.08654", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an empirical investigation of forgetting in task-based continual learning (CL). It highlights and distinguishes between forgetting in representation level and output level. The paper shows that forgetting occurs at representational level and affects generalization performance to future tasks. The paper introduces several new metrics for measuring forgetting to support its analysis. Lastly, it demonstrated that functional regularization appears to more effectively mitigate forgetting.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The current paper requires significant revision:\n1. Consider the more general setting of class-incremental CL.\n2. Propose concrete solutions to tackle the forgetting issues identified in the paper.\n3. Revise the relative forgetting metric for output level (see discussion above in Strength and Weakness). In my view, linear probing is sufficient to demonstrate representational forgetting. Its comparison with forgetting at output level is unnecessary and unclear.", "broader_impact_concerns": "N.A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the three reviewers originally recommended that the paper be accepted, recognizing the quality of this study. We discussed this further with the third reviewer, who seems to agree that the work meets the TMLR criteria. 

Overall, the main points of discussion with (two of) the reviewers were whether the findings were previously known to the community, given that current works in CL do not (explicitly) distinguish between "representation" vs. "output" forgetting. In other words, how important is this distinction? Of course, as the authors point out, recent works discard the importance of representation forgetting, which is partly what this paper clarifies. There was also a suggestion by reviewers to explain how the findings add to the literature on class-incremental learning (with functional regularization), which seems to deal with representation forgetting more directly. 

I recommend this paper be accepted. I propose minor corrections since the authors might want to address the above issue further. I don't make it a requirement since these points have already been discussed in the paper and its revision. However, the fact that two (expert) reviewers still had comments after the discussion might be indicative that the positioning of the contribution with respect to the current literature could be improved. One suggestion would be to consider further clarifying/emphasizing using additional references, but again, this is at the authors' discretion. 

Congratulations!
Decision: Accept (poster)