Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Authors: Vinay Venkatesh Ramasesh
Ethan Dyer
Maithra Raghu
Venue: ICLR 2021 Poster (2021)
Keywords: Catastrophic forgetting
continual learning
representation analysis
representation learning

Abstract: Catastrophic forgetting is a recurring challenge to developing versatile deep learning models. Despite its ubiquity, there is limited understanding of its connections to neural network (hidden) representations and task semantics. In this paper, we address this important knowledge gap. Through quantitative analysis of neural representations, we find that deeper layers are disproportionately responsible for forgetting, with sequential training resulting in an erasure of earlier task representational subspaces. Methods to mitigate forgetting stabilize these deeper layers, but show diversity on precise effects, with some increasing feature reuse while others store task representations orthogonally, preventing interference. These insights also enable the development of an analytic argument and empirical picture relating forgetting to task semantic similarity, where we find that maximal forgetting occurs for task sequences with intermediate similarity.

Published as a conference paper at ICLR 2021
ANATOMY OF CATASTROPHIC FORGETTING: HIDDEN REPRESENTATIONS AND TASK SEMANTICS
Vinay V. Ramasesh Blueshift, Alphabet Mountain View, CA
Ethan Dyer Blueshift, Alphabet Mountain View, CA
Maithra Raghu
Google Brain
Mountain View, CA
ABSTRACT
Catastrophic forgetting is a recurring challenge to developing versatile deep learn- ing models. Despite its ubiquity, there is limited understanding of its connections to neural network (hidden) representations and task semantics. In this paper, we address this important knowledge gap. Through quantitative analysis of neural representations, find that deeper layers are disproportionately responsible for forgetting, with sequential training resulting in an erasure of earlier task representa- tional subspaces. Methods to mitigate forgetting stabilize these deeper layers, but show diversity on precise effects, with some increasing feature reuse while others store task representations orthogonally, preventing interference. These insights also enable the development of an analytic argument and empirical picture relating forgetting to task semantic similarity, where we find that maximal forgetting occurs for task sequences with intermediate similarity.
INTRODUCTION
While the past few years have seen the development of increasingly versatile machine learning systems capable of learning complex tasks (Stokes et al. 2020 Raghu & Schmidt [020 Wu et al. 2019b), catastrophic forgetting remains a core capability challenge. Catastrophic forgetting the ubiquitous phenomena where machine learning models trained on non-stationary data distributions suffer performance losses on older data instances. More specifically, if our machine learning model is trained on a sequence of tasks, accuracy on earlier tasks drops significantly. The catastrophic forgetting problem manifests many sub-domains of machine learning including continual learning
(Kirkpatrick et al., 2017), multi-task learning (Kudugunta et al.) 2019). standard supervised learning
through input distribution shift (Toneva et al. 2019 Snoek et al. 2019: Rabanser et al. 2019
Recht
al. 2019) and data augmentation (Gontijo-Lopes et al. 2020
Mitigating catastrophic forgetting has been an important research focus (Goodiellow et al. 2013
Kirkpatrick et al. 2017]] Lee et al. 2017 Li et al. 2019; Serrà et al., 2018 Ritter et al. [201 8 Rolnick ct 2019)), but many methods are only effective in specific settings (Kemker et al.] 2018). and progress is hindered by limited understanding of catastrophic forgetting's 's fundamental properties. How does catastrophic forgetting affect the hidden representations of neural networks? Are earlier tasks forgotten equally across all parameters? Are there underlying principles common across methods to mitigate forgetting? How is catastrophic forgetting affected by (semantic) similarities between sequential tasks? This paper takes steps to answering these questions, specifically:
1. With experiments on split CIFAR-10, a novel distribution-shift CIFAR-100 variant, CelebA and ImageNet we analyze neural network layer representations, finding that higher layers are disproportionatlly responsible for catastrophic forgetting, the sequential training process erasing earlier task subspaces.
2. We investigate different methods for mitigating forgetting, finding that while all stabilize higher layer representations, some methods encourage greater feature reuse in higher layers, while others store task representations as orthogonal subspaces, preventing interference.
3. We study the connection between forgetting and task semantics, finding that semantic similarity between subsequent tasks consistently controls the degree of forgetting.
4. Informed by the representation results, we construct an analytic model that relates task similarity to representation interference and forgetting. This provides quantitative empirical measure of

---

Published as a conference paper at ICLR 2021
VGG on
ResNet
DonseNet CIFAR100
2.2
0.3
0.4
0.6
AccuTeey
6-1
0.1
Thale2
Teac
0.2
T
Task1
Thcc2
62
Thdk 2
Tranning
Examples of catastrophic forgetting across different architectures and datasets. We plot accu- racy of Task I (purple) and Task 2 (green), both the split CIFARIO task, and the CIFARI00 distribution-shie task across multiple architectures. Catastrophic forgetting seen as the significant drop Task accuracy when Task training begins.
Figure 1:
task similarity, and together these show that forgetting is most severe for tasks with intermediate similarity.
RELATED WORK
Mitigation strategies Developing mitigation strategies for catastophic forgetting is an active area of research (Kemker et al. 2018). Popular approaches based on structural regularization include Elastic Weight Consolidation (Kirkpatrick et al. 2017) and Synaptic Intelligence (Zenke et al.) (017). An alternative, functional regularization approach is based on storing and replaying earlier data in a replay buffer (Schaul et al. (2155) Robins 1995 Rolnick et al. (209)). We study how these mitigation
methods affect the internal network representations to better understand their role in preventing catastrophic forgetting.
Understanding catastrophic forgetting Our work similar in spirit to existing empirical studies aimed better understanding the catastrophic forgetting phenomenon (Goodfellow et al., 2013 Toneva et al. 2019). We focus specifically on understanding how layerwise representations change, and on the relation between forgetting and task semantics, which have not previously been explored. This semantic aspect is related to recent work by Nguyen et al. (2019) examining the influence of task sequences on forgetting. It also builds on work showing that learning both biological and artificial neural networks is affected by semantic properties of the training data (Saxe et al. 2019 Mandler McDonough T993).
Fine-tuning and transfer learning While not studied in the context of catastrophic forgetting, layerwise learning dynamics have been investigated in settings other than continual learning. For example. Raghu et al. (2017) showed that layerwise network representations roughly converge bottom- up: (from input to output). Neyshabur et al. (200)) observed similar phenomena in a transfer-leariing setting with images, as have others recently transformers for NLP (Wu al.) 2020 Merchant et al., 2020). Furthermore, the observation that early layers in networks learn general features like edge detectors, while latter layers learn task specific features is well-known result in computer vision (Erhan et al. 2209), and here we quantitatively study its ramifications for sequential training and catastrophic forgetting.
Gaahpe.,Raghu
SETUP
Tasks: We conduct this study over many different tasks and datasets: (i) Split CIFAR-10, where the
ten class dataset split into two tasks of 5 classes each (ii) input distribution shift CIFAR-100, where each task is to distinguish between the CIFAR- 100 superclasses, but input data for each task is different subset of the constituent classes of the superclass (see Appendix A.3) (iii) CelebA attribute the two tasks have input data either open (iv) ImageNet superclass prediction, similar to CIFAR 100.
prediction:
women, and we predict either smile mouth
We perform experiments with three common neural network architectures used in image classification - VGG (Simonyan & Zisserman (2014), ResNet (He et al. 2013) and DenseNet (Huang et al. 2016). Examples of catastrophic forgetting in these models are shown in Figure
Models:
2

---

Published as a conference paper at ICLR 2021
VGG on CIFARIC
Reahlet on CIFARDD
Reshiet CIFAR100
Denseliet CIFAR100
0.050
0.023
Aoc (100222
4.75
0.775
0.25
6.00
6,49
6,325
0.750
5,533
K.Praan
Figure 2: Freezing lower layer representations after Task training has little impact Task 2 accuracy. We freeze the parameters a contiguous block layers (starting from the lowest layer) after Task training, and only train the remainder on Task 2, finding that freezing the lowest layers has little impact on Task 2 accuracy.
CATASTROPHIC FORGETTING AND HIDDEN LAYER REPRESENTATIONS
We begin by investigating how catastrophic forgetting manifests in the hidden representations of the neural network. Do all parameters (and layers) forget equally? Or are specific components of the network particularly responsible for the drop in accuracy in sequential training? Through combination of layer freezing experiments, and representational analysis, we find that higher layers (layers closest to output) disproportionately responsible for catastrophic forgetting, with lower layers (layers closest to input) remaining representationally stable through sequential training. Further analysis of the representational subspaces of Task 1, Task (post Task training) and Task 2, provide insights on both feature reuse and interference.
4.1 FREEZING LAYER REPRESENTATIONS
To study the effect of individual layers forgetting, measure the effect of freezing layer representations on Task 2 accuracy (Figure Specifically, we freeze a contiguous block of layers (starting from the lowest layer) after training on Task 1, and only train the remaining layers Task 2. Across different architectures and tasks, we observe that lower layers can be reliably frozen with very little impact on Task 2 accuracy. This suggests the possibility of lower layer features being reused between both tasks, with higher layers being the main contributor to catastrophic forgetting.
4.2 REPRESENTATIONAL SIMILARITY THROUGH SEQUENTIAL TRAINING
The results of Figure 2 illustrate that lower layer representations Task can be reused for good performance on Task 2. To determine this is what actually occurs during the training process, we turn Centered Kernel Alignment (CKA) (Kornblith et al., 2019), a neural network representation similarity measure. CKA and other related algorithms (Raghu et al. 2017 Morcos al. 2018) provide scalar score (between and 1) determining how similar a pair (hidden) layer
representations are, and have been used to study many properties of deep neural networks (Gotmare
al. 2018 Kudugunta et al. 2019 Wu et al. 2019a).
Specifically, letting X E R and Y E Raxp be (centered) layer activation matrices of (the same) n datapoints and neurons, CKA computes
HSIC(XXT, YYT)
CKA(X, Y)
/HSIC(XXT, XXT) HSSCCYYY, YYT)
for HSIC Hilbert-Schmidt Independence Criterion (Gretton al. 2005). We use kernel CKA.
In Figure 3 we plot the results of computing CKA on Task layer representations before and after Task 2 training. Across architectures and tasks, observe that the lower layers have high representation similarity, suggesting that lower layer Task features are reused Task 2. The higher layers however show significant decreases in representation similarity, suggesting they disproportioonatel contribute to catastrophic forgetting. These conclusions are further supported by additional layer reset experiments in Appendix Figure 12, which shows that rewinding higher layer parameters from Post-Task22 2 training to their pre- Task 2 training values significantly improves Task performance.
4.3 FEATURE REUSE AND SUBSPACE ERASURE
Further insights on how the representations of lower and higher layers evolve during sequential training is given through a subspace similarity analysis. Letting X e m=*p be the (centered) layer activation matrix of n examples by p neurons, we compute the PCA decomposition of X, i.e. the
3

---

Published as a conference paper at ICLR 2021
VGG CIFAR1O
Rediet on CIFARIC
Reslitt on CIFAR100
Dencebui CIFARL
1.0
1.0
2.05
0.8
0.90
0.85
0.8
0.00
0.73
0.2
0.79
Stags
Stage
Siage
Stagt
ColobA Resnet CXA
013
Figure 3: CKA analysis of layer representations before/after Task 2 training shows lower layer feature reuse and higher layer representations changing significantly on CIFARIO, CIFARIOO, ImageNet and CelebA. We compute CKA of the Task 1 layer representations before and after Task 2 training. We observe that later layers change more than early layers through second task training.
CIFAR1O Resnet: Subspace Simiaritt Between and
CIFAR1O vSc: Sub ace Similanty fetween training
CI ARPA0 Subspace Simllarity Betweer Task
3.0
10:10
Subspace Smilutty
0.2
0.6
6.4
6-
Tase2)
(Tape), Tisk2t
[laski, Tass21
04
(Tua1. Taacl Tesk2)
(Taak) Taki Poub Tusk2
Taski Taxc Post Tuak Teak2 Taxck PaakTakk2
(Tea2 Teac Tesk)aak2)
(Taak), Taki Poub Task21
6.1
Stage"
Stage
Stage
Figure 4: Subspace similarity analysis shows feature reuse in lower layers and Task subspace erasure in higher layers. For Resnet on CIFARIO (left), VGG on CIFARIO (middle), Resnet on CIFAR100 (right), we compute subspace similarity between: (i) (Task 1, Task 2) (purple line) (ii) (Task I, Task post-Task 2 training) (green line) (ii) (Task 2, Task 1 past-Task2 2 training) (blue line). All comparisons show high similarity in lower layers, indicative of feature reuse. Most striking is the comparison between (Task 1, Task post-Task training) and (Task 2, Task post-Task 2 training), which show that after Task 2 training, higher layer representations for Task are more similar to Task 2 than to Task 1! Specifically, the Task 1 subspace in higher layers is erased through Task 2 training.
eigenvectors (01, ...) and eigenvalues AA1, of X Letting Vv be the matrix formed from the top h principal directions, vi...,, Uk as columns, and Ug the corresponding matrix for a different activation matrix Y, we compute
SubspaceSim, (X,Y) 1 K LIV
This measures the overlap in the subspaces spanned by (V1...., Uk) and (uI...., ug). Concretely, if X and Y correspond to layer activation matrices for two different tasks, SubspaceSim, (X, Y) measures how similarly the top representations for those tasks are stored in the network.
Figure 4 shows the result of computing SubspaceSimg (X, Y) for X, Y being the layer activation matrices in (i) (Task 1, Task 2) (purple line) (ii) (Task 1, Task post- 2 training) (blue line) (iii) (Task 2, Task post-Taask 2 training) (blue line). Supporting the CKA analysis, lower layers have high subspace similarity between Task 1 and Task 2 (feature reuse), but higher layers have low similarity (representations change significantly during Task 2 training.) Furthermore, (Task I, Task post-Task 2 training) also shows low subspace similarity in the higher layers, indicating that the representational change in later layers is not restricted to Task 2 data, but also alters the networks representation of the initial task. Strikingly, (Task 2, Task post-Task Task 2 training) shows high similarity in all layers, despite being different tasks.
In summary, these results illustrate that during sequential training, effective feature reuse happens in the lower layers, but in the higher layers, after Task 2 training, Task 1 representations are mapped
4

---

Published as a conference paper at ICLR 2021
Resnet on CFAR10: Replay
Restee CIPARDD EECC
Resnet on CIARLO0 Replay
Reand CHAR100
OKO Smd kmy
CG ontt
IKC
Jo02
Jed
--120-2
A-G2
Stage number
Stage number
Stgenamier
Stage number
Figure 5: CKA analysis of mitigation methods demonstrates that mitigation methods stabilize the higher layer representations. We compute CKA between layer representations of Task I before and after Task 2 training, with varying amounts and types of mitigation. We find that across all mitigation methods, even a small amount of mitigation works to stabilize the higher layer representations. In Figure 6 we investigate whether this stabilization results in greater feature reuse or through finding independent subspaces.
CIFAR10 Rushaat repky Subspace Smiirrity
OIFA11 Recnet EWC Subispace Smilarity
CIFAR100 Recnet replay Subspace Similariy
CIFAR100 Resnet EWC Subbspace Similarity
Texd. Taa:)
Tard:
Masti Tand Drakk Seit Ferd)
Tall hob-Teakd Thall: Tack Pual Trakk
FapTard
[tead Teal Feet
(Teak, Tuak Pwo Tral))
Tone Teald
CIFARLD Sennet repley: Subepaccdim
OFAR10 Rwane EWC, Sutspacadim iTase 2 Task 1 post Task 2 trailigg)
CIPARIO Blsnet Suteoacedim Task 2. Task p Tack tratting)
CIFAR1DO Iteentt replay: Subapecedi in (Task 2. Task 1 Task trating)
iTask 2, Task 1 post Task training
Task
Neee
Seee
123.0
Rome
670
0.79
070
Steee
Figure 6: Subspace similarity analysis reveals differences between different mitigation methods, with replay storing Task 1 and Task representations in orthogonal subspaces, while EWC and SI promote feature reuse in the higher layers. Top row: We plot subspace similarities like in Figure 4 for different miligation methods. We observe that (Task 2. Task post-Task 2 training) is much lower in replay compared to EWC/SI, as is (Task1, Task2) similarity. The bottom row plots (Task 2, Task I post-Task amounts of mitigation. Again see that replay has much lower (Task 2, Task post- TastTTak& training) similarity than no mitigation, while EWC/SI maintain the similar subspaces of Task I and Task 2 post Task 2 training.
Task 2 training) for different
into the same subspace as Task 2. Specifically, Task 2 training causes subspace erasure of Task 1 in the higher layers. (Additional details and results are in Appendix Section A.5)
5 FORGETTING MITIGATION METHODS AND REPRESENTATIONS
Informed by this understanding of how catastrophic forgetting manifests in the hidden layers (feature reuse in lower layers and subspace erasure in higher layers), we next study methods to mitigate forgetting. Do popularly used mitigation methods act to stabilize higher layers - given possible alternate approaches such as weight orthogonalization (Appendix ?
We investigate these questions across different successful mitigation strategies (i) EWC (elastic weight consolidation) (Kirkpatrick et al.) (017)). a regularization- based approach (ii) replay buffer (Schaul et al. 2015; Ratclifi 1990, Rolnick et al. 2009). a rehearsal-styled approach (iii) SI (Synaptic Intelligence) Zenke et al. (017). another popular regularization-bant approach.
In Figure 5] we plot the results of computing CKA on layer representations of Task before/after training on Task 2, with different amounts of mitigation applied, and across different mitigation methods, tasks and architectures (full results in Appendix [B7 Across all of these settings, we observe that even a small amount of mitigation results in a significant increase in similarity (stabilization) in the higher layers. However, there remains central open question is this increase in similarity due to feature reuse in the higher layers between Task I and Task 2, or does the network store Task and Task 2 representations in inteerterf, orthogonal subspaces?
5

---

Published as a conference paper at ICLR 2021
CelesA Recnet: Supspace 2.0
Celebd Respel Subspace Smii affty
CelebA Roonet ruplay: Subaparesie
Taysk Trsk post-Task 1m nag!
3.55
3:30
Task? Teark Teaks Peak-Takk) Thack Teats Posk
[Tank Teati Por:Teace (Tabh Teati Purc
035
0.
Stage
Stegs
Figure 7: Subspace similarity analysis of CelebA, with and without replay mitigation. Left pane computes subspace similarities in CelebA, like in Figure showing subspace erasure at top layers. Middle and right panes showt the effect of adding replay, where subspaces in lower/middle layers become more similar but higher layers become more orthogonal, like in Figure
To answer this, we compute the Subspace Similarity measure introduced in Section 4.3 with results in Figure [6] (and additional experiments in Appendix Section B.5). These results reveal key differences between the mitigation methods: replay buffer results in orthogonal network subspaces for higher layer representations of Task and Task 2, while EWC and SI promote feature reuse even in the higher layers. In more detail:
Comparing the top row of Figure 6 to Figure 4 (the same plot without mitigation), we see that mitigation methods increase the subspace similarity of (Task Task 1 Post-Task 2) and (Task 1,
Task 2) representations in carlier/middling layers (more feature reuse).
Most interestingly, comparing the different mitigation methods across top row Figure 6 we see that while EWC and SI maintain high subspace similarity of (Task 2, Task Post-Task 2) even in the highest layers, replay significantly decreases this value the highest layers. This suggests that replay performs mitigation through use of orthogonal subspaces, while EWC/SI encourage feature
reuse even in the highest layers.
In the bottom of the different mitigation methods. We observe a clear decrease in subspace similarity in higher layers even when little replay is used (again supporting orthogonal subspaces), while EWC/SI maintain or increase this (feature reuse even in higher layers.)
show subspace similarity of (Task 2, Task Post-Tase 2) for varying strengths
6 SEMANTICS
With the insights on how catastrophic forgetting relates to the hidden representations, and the effects on network internlls of different mitigation methods, we turn to investigating how the semantic similarity between the sequential tasks affects forgetting. Specifically the prior results on lower layer feature reuse suggest key question: does similarity between tasks result in less forgetting? Surprisingly, the answer is quite nuanced, depending on how precisely the different tasks represented in the network.
6.1 FORGETTING, TASK SEMANTICS AND SIMILARITY PUZZLE
Our study of the relationship between task semantic similarity and forgetting reveals surprising puzzle. In Figure and [ we observe that depending on the task setup, a semantically similar task may be forgotten less or more than less similar task. This behavior is consistent across different settings and datasets. Specifically, in Figure Sa and Sd Task I and Task consist of binary classifications between animals and objects from CIFAR-10, 10, where we find that the similar categories forget less. But in Figure 8b and Se four category classification between animals and objects followed by binary classification task results in the similar categories being forgotten more. Table I shows analogous result on ImageNet. This effect of similar categories being forgotten more can also be observed in the CIFAR- 100 distribution shift setting, where input distribution shift to the CIFAR- 100 superclasses results in more similar categories being forgotten more, e.g. input shifts to natural superclasses (Figure Sc) or artifical superclasses (Figure 8f Further details and supporting experiments can be found in Appendix B.9
In the subsequent sections, we resolve this puzzle through (i) the development of an analytic model that formalizes the subtle connection between task similarity and forgetting (ii) ensuing measure of

---

Published as a conference paper at ICLR 2021
Similar tasks forget less ship, truck
Similar tasks forget more
Similar tasks forget Altered: med. small mammals
Trus pozibee rata
0.6
0.4
cat. home
0.1
ship
0.2
devices
carricores
())
plane, car
(b)
dog
truck
(c)
storage
vehicles
100
150
Task cat,
2,
1.0
0.8
0.8
Tru paltiee
0.8
0.6
0.7
Task 2 dasses
Task dasses
Unaltered dassess
deer,
[d)
ship, truck
e)
dog
flowers
50
1.00
150
1.00
50
100
150
Spach
Epoch
Epoch
Figure 8: Forgetting severity consistently varies along semantic divisions. (a.d) For sequential binary classification tasks models trained either objects animals forgets less second object task of the same semantic category. (b.e) Models trained initially four way classification two object and two animal classes see more forgetting in semantically similar classes when trained on a second two class task of either objects or animals. (c,f) Models trained on sequential six- six-way classification with four unaltered categories (shown) and altered categories show increased forgetting in the unaltered categories semantically most similar to altered categories, For all panes, performance shown for ResNet, with additional results in Appendix B.9
Task Description
Performance Drop Task Animals
Artifacts
Task1: animals Task2: animals
0.0885 + 0.0259 0.1832 0.0484
Task1: animals Task2: artifacts
Task1: artifacts Task2: artifacts
0.2884 + 0.0187
TaskI: artifacts Task2: animals
0.3720 0.0288
Task1: animals and artifacts Task2: animals
0.1752 0.0193
0.1532 + 0.0231
Task animals and artifacts Task2: artifacts
0.2057
0.0312
0.2330 0.0292
Table 1: Task Semantics and forgetting in ImageNet Here a ResNet50 is first trained (task 1) classify either 10 animals, 10 artifacts 10 animals and artifacts (chosen randomly from ImageNet). The network is then trained (task 2) to either classify 10 different animals 10 different artifacts. The table shows performance drop on Task after Task 2 training. Similar to Figure when Task either animals or artifacts (top part of table). similar tasks forget less, while when Task 1 is both animals and artifacts, similar tasks show greater forgetting (bottom part of table).
7

---

Published as a conference paper at ICLR 2021
task similarity, trace overlap, empirically quantifies task similarity (iii) showing maximal forgetting happens with intermediate task similarity.
6.2 AN ANALYTIC MODEL OF FORGETTING SEMANTICS
We first formalize model with single classification head (like in distribution shift CIFAR-100), with the corresponding multi-head setup in Appendix C We write the network output as f(z) = 2 4x (w;z)), æ), where & are the last layer weights and g are features. We consider training sequentially two tasks. For the first task, train normally. For the second task. inspired by empirical observation above that forgetting is driven by deeper layers, we freeze the the earlier layer features, 9u (w; 9μ(W; (w; when training the second task (here in represent the weights after training on the first task). If we train with loss function L(f,y) and learning rate the network output SGD update at timestep is
@ x)) OL((('),8)) 0f
(Л(1) = -1
Σ
x'.g'E mmm
Here (22)) is the second-task training data and O(z, = = 9u (w;; x x) (w;; æ²) is the overlap (inner product) between the model last-layer representations on data point x and x'. If the overlap (x, x') is small, then the change in the logits is small and forgetting is minimal, specifically:
Lemma
Let fo be the logits of a neural network trained via the frozen feature model. Let x be element of the initial task test set. Let further denote the vector of the feature overlap applied to the second task training set as Ō(x) == {0(x, x') x' E X ) and the loss vector as == (L(f(z'),) v/) x'.u' of E Di?) With this, the change in the original task logits is bounded
|6(0|
al as
(aЛ(a)
Thus representational similarity is necessary for forgetting to occur, and in this model, sufficiently similar and sufficiently dissimilar tasks have minimal forgetting, while intermediate similar tasks have maximal forgetting.
6.3 CATASTROPIICC FORGETTING ACROSS VARYING TASK SIMILARITIES
The analytical result enables to make precise measure of task similarity, through using the feature overlap matrix ( æ). Concretely, letting 11 be the overlap matrix on Task I, 222 on Task 2, and 1 on Task 1 and Task 2, we define:
Tr (0120T))
Trace overlap
(4)
V(Tr (fn Tr (0220%)
Trace overlap is directly analogous to CKA, where instead of comparing the similarity of different features on the same data, we are comparing the similarity of common features on different data.
Increasing task dissimilarity can help reduce forgetting In Lemma] 1 we see that very dissimilar tasks, with small overlap matrix, exhibit minimal forgetting. In Figure 9 we test this in the context of sequential binary classification. We introduce other category the classification problem (of images from classes not included in the training tasks), which encourages the model to represent Task I and Task 2 dissimilarly (lower trace overlap in Figure Pp), and reduces forgetting (Figure 9h).
Maximal forgetting at intermediate similarity To empirically study the conclusion from the model that intermediate-simee data augmentation technique, to interpolate between the two tasks. Specifically, we first train a model on binary classification (with an other category encourage dissimilarity) and for X E [o, 1]. generate -
tasks have maximal forgetting,
mixup (Zhang et al. 2017). standard
(A) an mixup dataset train {(xx(2) + (1 (1-Xi ()) Xy + +11-+(11)))) -)) We compare model behavior across range of mixup fractions, finding that the task similarity measured through the trace
8

---

Published as a conference paper at ICLR 2021
CIFAR1O: calleg cry helps
Representation Similarity forgetting
Representation Similartty forgetting
1.0
1.0
There ouee
0.9
Thace
0.8
0.5
0.7
Other
0.7
0.7
(8)
Nane
(b)
(c)
0.6
0.5
Epoch
Stage number
Stago number
Figure Adding an other category reduces forgetting and increases orthogonalization. (a) Models trained initially on classifying two objects: airplanc/automobile, and subsequently on two animals: decr/dog exhibit diminished forgetting when we add an other category to task training. The other category contains randomly sampled images from all classes not present in the initial final tasks. (b,c) The presence of the other category reduces the similarity between the model's representations of each task. Additional plots are in Appendix B.8
Interpolating task
Overlap similarity
Categorical similarity
1,00
0.9
0.95
0.8
0.90
0.7
0.85
0.80
Trace
0.6
2u2 nnjects
0.75
Fre forgatting forgstting
0.7
0.5
4v2 animals
0,70
(a)
(b)
(c)
0.4
fraction
Figure 10: Task similarity affects performance. (a) We measure initial task performance for sequential binary classification as we dial the mixup fraction between 0.0, pure task 2 data, and 1.0, pure task data. Maximum forgetting (minimal accuracy) occurs at an intermediate value of the mixup fraction, ~ 0.1 - 0.2. (b) vary the mixup fraction, supporting the intuition
We measure the similarity between task I and task 2 that interpolating mixup fraction interpolates task similarity. (c) Task performance after task training plotted against task versus task representational similarity for animal and ohject classes in the sequential binary classification tasks (solid) and four-way followed by binary classification tasks (hollow) of Figures Sa and Figures 8b) In the case of sequential binary tasks both object and animal second tasks are represented extremely similarly the initial object task and the slightly less similar, object task, forgets more. For the four way classification task, animals are represented dissimilarly to the second object task and cause less forgetting.
overlap between task1 and task2 varies monotonically with mixup fraction (Figure 10bj, quantifying that varying the mixing parameter, X, tunes the degree of task similarity. Figure 10a shows that the most extreme forgetting happens for intermediate similarity (i.e. mixing fraction ~ 0.1 - 0.2).
Semantic similarity revisited Armed with these quantitative measures, we revisit the task semantics puzzle in Figure where (i) for sequential binary classification, semantically more similar categories were forgotten less, while in (ii) four class classification followed by binary classification and input distribution shift, similar classes were forgotten more. Figure T0c (solid dots) shows that in (i) the net didn't represent objects and animals differently, and the slightly more similar task is forgotten less. But in (ii) Figure 10c (hollow dots), the model encodes animals and objects differently. minimizing forgetting on the dissimilar tasks, consistent with maximal forgetting for intermediate similarity.
7 CONCLUSION
In this paper, we have studied some the fundamental properties of catastrophic forgetting, answering important open questions on how it arises and interacts with hidden representations, mitigation strategies and task semantics. By using representational analysis techniques, we demonstrated that forgetting is not evenly distributed throughout the deep learning model but concentrated at the higher layers, which change significantly and erase earlier task subspaces through sequential training. Consistent with this, find mitigation methods all stabilize higher layer representations, but vary on whether they enforce more feature reuse, or store tasks in orthogonal subspaces. Informed by these insights, we formulate analytic model to investigate connections between forgetting and task

---

Published as a conference paper at ICLR 2021
semantics. This gives quantitative measure of task similarity and shows that intermediate similarity in sequential tasks leads to maximal forgetting. In summary, these results help provide a foundation for deeper understanding of catastrophic forgetting and suggest new approaches for developing and measuring mitigation methods.
10

---

Published as a conference paper at ICLR 2021
REFERENCES
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. Massively multilingual neural machine translation in the wild: Findings and challenges. arXiv preprint arXiv:1907.05019 2019.
Dumitru Erhan, Y. Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. Technical Report, Univeristé de Montréal, 01 2009.
Dar Gilboa and Guy Gur-Ari. Wider networks learn better features, 2019.
Raphael Gontijo-Lopes, Sylvia J Smullin, Ekin D Cubuk, and Ethan Dyer. Affinity and diversity: Quantifying mechanisms of data augmentation. arXiv preprint arXiv:2002.088) 2020.
lan J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting gradient-based neural networks. arXiv preprint arKiv: 1312.6211, 2013.
Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprini arXiv:1810.13243, 2018.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. International conference on algorithmic learning theory, pp. 63-77. Springer, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
Gao Huang. Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In NeurlPS, 2018
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Mea- suring catastrophic forgetting in neural networks. In Thirty-second AAAI conference on artificial intelligence, 2018
James Kirkpatriek, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barn Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. arXiv preprint arXiv:1905.00414, 2019.
Sneha Reddy Kudugunta, Ankur Bapna, Isaac Caswell, Naveen Arivazhagan, and Orhan Firat. Investigating multilingual nmt representations at scale. arXiv preprini 1909:09999001999 2019.
Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Woo Ha, and Byoung-Take Zhang. Overcoming catastrophic forgetting by incremental moment matching. In Advances in neural information processing systems, pp. 4652-4662, 2017.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong, Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. arXiv preprint N9Xv:/994.0003111 2019.
Jean M. Mandler and Laraine McDonough. Concept formation in infancy. Cogni- tive Development, 8(3):291 - 318, 1993. ISSN 0885-2014. doi: http:///oi.org// 10. URL http: wwwp//wwww.. www.weeceeced.... rect wwt.eccc/sccccccc
1016/908855-20144338000033 atticcle/s11/008821443333
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to bert embeddings during fine-tuning?, 2020.
11

---

Published as a conference paper at ICLR 2021
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems, pp. 5727-5736, 2018.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning?, 2020.
Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Stefano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXiv: 00091, 2019.
Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: an empirical study of methods for detecting dataset shift. In Advances in Neural Information Processing Systems, pp. 1394-1406, 2019.
Maithra Raghu and Eric Schmidt. A survey of deep learning for scientific discovery. arXiv preprint arXiv:2003. 2020.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems, pp. 6076-6085, 2017.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97 2:285-308, 1990.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? arXiv preprint /9X. 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber, Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pp. 3738-3748, 2018.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neurul Information Processing Systems, pp. 348-358, 2019.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116 doi: 10. 111.07//paa.11222200..
(23):11537-11546, 2019. ISSN 0027-8424.
URL https
7/ //www, pnws.ppas.coom www.maaa..rrrceteeeeeeeeeee httpg///cooeeee///////555
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint 1XXiv::5/1.009// 2015.
Joan Serrà, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. arXiv preprint arXiv: 1801.0123,, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2014.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D Sculley, Joshua Dillon, Jie Ren, and Zachary Nado. Can you trust your model's ' uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 1396,-13980, 2019.
Jonathan M Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz, Nina M Donghia, Craig R MacNair, Shawn French, Lindsey A Carfrae, Zohar Bloom-Ackerman, et al. A deep learning approach to antibiotic discovery. Cell, 180(4))688-702 2020.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning. ArXiv, abs/1812.05111 2019.
12

---

Published as a conference paper at ICLR 2021
John M. Wu, Yonatan Belinkov, Hassan Sajjad. Nadir Durrani, Fahim Dalvi, and James Glass. Similarity analysis of contextual word representation models, 2020.
Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross- lingual structure in pretrained language models. arXiv preprint arXiv:191..0146; 2019a.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick, Detectron2. 2019b.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3987- 3995. JMLR. org, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17X191171..09412, 2017.
13

---

Published as a conference paper at ICLR 2021
A METHODS
A.1
MODELS
Here we detail the model architectures used in this study. We use standard versions of the popular VGG (Simonyan & Zisserman| 2014), ResNet (He et al. 2013), and DenseNet (Huang et al. 2016)) architectures. These architectures have varying structural properties (e-g., presence or absence of skip connections), meaning observed behaviors common to all of them are likely to be robust and generalizable.
VGG: The VGG model we use consists of five stages. Each stage comprises a convolutional layer, a ReLU nonlinearity, another convolutional layer, another ReLU nonlinearity, and finally a MaxPool layer. Each convolution uses a 3-by-3 kernel with unit stride and padding. The MaxPool operation uses 2-by-2 kernel with stride 2. The number of channels by stage is 16, 32, 64, 128, 128. We do not use batch normalization in the VGG.
We test two different version of the VGG: one with just the stages described above, and another with two fully connected layers, of width 1024, after the convolutional layers.
ResNet: Our ResNet consists of initial 3-by-3 convolutional layer with 32 channels, followed by 2d batch norm operation. This initial pair is followed by four stages, each consisting of two residual blocks per stage. Each block consists of two Ben-BB-ReLU sequences, with shortcut layer directly routing the input to add to the final ReLU preactivations. All convolutions are 3-by-3, with unit padding. In all but the first block, the first convolutional layer downsamples. with a stride of 2; the second convolutional layer maintains stride of 1. The number of channels doubles each block, starting from a base level of 32 channels in the first block.
Like ResNet, our DenseNet consists of an initial 3-by-3 convolution, followed by four dense blocks. In between each pair of dense blocks is a transition block. Following the final dense block is batch-normalization. ReLU, and average pooling operation to generate the final features. Our DenseNet is characterized by a growth rate 12 and compression rate of 0.5. The first stage of the DenseNet features 6 blocks, with the number of blocks doubling in each subsequent stage.
NenseNet:
A.2 TRAINING
All networks are trained using cross-entropy loss with SGD with momentum (3 =0=00 0.9), using a batch size 128. We do not use learning-rate schedules here, leaving that investigation to future work. To better correspond with practical situations, however, we choose a learning rate and total number of epochs such that interpolation (of the training set) occurs. For the split CIFAR-10 10 task, this typically corresponds to training for 30 epochs per task with a learning rate of 0.01 (VGG), 0.03 (ResNet), and 0.03 (DenseNet). We use weight decay with strength le-4, and do not apply any data augmentation. For the split CIFAR- 100 task, we usually train for 60 or 90 epochs per task, with other the other hyperparameters identical to the CIFAR- case.
For multi- head networks (used in split CIFAR-10), each head is initialized with weights drawn from a normal distribution with variance 1/n,, where n is the number of features; biases are initialized to zero. We do not copy the parameters from the old head to the new when switching tasks.
For the split CIFAR- 10 setup, the plots shown in the main text refer to experiments in which the initial task was classifying between the five categories airplane, automobile, bird, cat, and deer; and the second task comprising dog,free frog, horse, ship, and truck. To verify that our results were not unique to the particular split of CIFAR-10 we chose, we used three other splits: airplane, bird, deer, frog, ship (task 1) and automobile, cat, dog, horse, truck (task 2); automobile, cat, dog, horse, truck (task 1) and airplane, bird, deer, frog, ship (task 2); and dog, frog, horse, ship, truck (task 1) and airplane, automobile, bird, cat, deer (task 2).
A.3 CIFAR- 100 DISTRIBUTION SHIFT TASK
Here provide the concrete example of the CIFAR- distribution-shife task shown in Figure I This is meant to capture the scenario where catastrophic forgetting arises through shift the input data to the neural network undergoes distribution shift, causing the neural network
input distribution
14

---

Published as a conference paper at ICLR 2021
to perform poorly earlier data distributions (Arivazhagan et al. 2019 Snoek et al. 2019 Rabanser et al. 2019). As mentioned in the main text, in this task, the model must identify, for CIFAR-100 images, which of the 20 superclasses the image belongs to. The difference in tasks comes from the difference in subclasses which make up the superclass. As an example, we take the five superclasses aquatic mammals, fruity and vegetables, household electrical devices, trees, and vehicles- O with the corresponding task subclasses (1) dolphin, (2) apple. (3) lamp, (4) maple tree, and (5) bicycle and task 2 subclasses (1) whale, (2) orange. (3) television, (4) willow, and (5) motoreycle. A key feature of this setting is that task mask-specific components (including multiple heads) are precluded since the model does not know the task identity either inference or time. While we do not explore here, this setup also allows for continuously varying the data distribution, another situation likely to occur practice.
A.4 REPRESENTATIONAL SIMILARITY AND CENTERED KERNEL ALIGNMENT
To understand properties of neural network hidden representations, we turn to representational similarity algorithms. A key challenge in analyzing neural network hidden representations is the lack of alignment - no natural correspondence exists between hidden neurons across different neural networks. Representational similarity algorithms propose different ways to overcome this one of the first such algorithms, SVCCA (Raghu et al., [017]] Morcos et al.) 2018), uses a Canonical Correlation Analysis (CCA) step align neurons (enable invariance) through linear transformations.
We use centered kernel alignment (CKA), proposed by Kornblith et al. 2019 to measure the similar- ity between two representations of the same dataset which is invariant orthogonal transformation and isotropic scaling (but not arbitrary linear transformations). Given a dataset of m examples, we compare two representations X and Y of that dataset (say, from two different neural networks or from the same neural network with different parameters), with n, and ny features respectively; that is, X E Rmx*= and Y E Rmxx, Then the Linear-kerne CKA similarity between the two representations is given by
I 2 XX
CKA(X,Y)
(5)
A.5 SUBSPACE SIMILARITY
For the subspace similarity computations we pick a fixed threshold & for all layers so that the first k principal components capture 80% of variance in the final layer. This gives k 3 for Resnet on CIFAR-10, 10, k- = 4 for Resnet on CIFAR- 100, k = 4 for VGG on CIFAR- 10 and k = 7 for VGG on CIFAR-10M 100 and & for DenseNet on CIFAR- and le 5 for DenseNet on CIFAR-100.
CIFAR10 Densenet: Subspace Similarity Between CIFAR100 Densenat: Subspace Similarity Between
CIFAR10O VGG: Subspace Similarity Between Task 1, Task post Task training and Task
Task 1, Task past Task training and Task
Task 1, Task post: Task 2 training and Task 2
1.0
1.0
1.0
0.9
0.9
0.9
Subspacee Similarty
Sulospate Similarit
Subspace Similerty
0.8
0.8
0.7
0.7
0.7
[Task1, TasiZ)
(Taskl, Taski
0.6
0.6-
0.6
(Task2, Taskd Posk-Task2)
0.5
0.5
(Taskl. Task2)
(Task1, Task2)
0.4
(Taskl, Taski Post Task2)
0.4
(Task2, Taskl Task:
0.4
(Task2, Teskl Post Tesk2)
(Task2, Task1 Post Task2)
0.3
0.3
Stage
Stage
Stage
Figure 11: Subspace similarity analysis shows feature reuse in lower layers and Task 1 subspace crasure higher layers.
'The CIFAR-100 dataset features two vehicle superclasses, denoted vehicles-1 and vehicles-2
15

---

Published as a conference paper at ICLR 2021
A.6 MITIGATION METHODS
A.6.1 ELASTIC WEIGHT CONSOLIDATION
Developed by Kirkpatrick et al. in 2017, elastic weight consolidation (EWC) adds a term to the loss function of the new task get the regularized loss function C(6):
( = LB(0) + Σ P. (0. - x7,1)
(6)
Here, 0 are the model parameters; (0) is the unregularized loss function of the new task; À parameter which controls the strength of the regularization; F; is the diagonal of the Fisher information matrix with respect to the parameters; and 07 are the model parameters after having been trained on task A. We compute the Fisher information via using the squared gradients of the log-likelihood, averaged over subset of the trainset. We use the empirical Fisher as an approximation.
Following standard practice, in experiments estimate the Fisher information using subset of the full training data; for all our models on CIFAR-10, 200 samples proved sufficient to be reasonably confident of converged estimate. We do include a delay between the start of second- task training and the application of the EWC penalty. Naturally, we do not apply the EWC penalty to the parameters of the head the multi-head setting.
A.6.2 SYNAPTIC INTELLIGENCE
Developed by Zenke et al. 2017). synaptic intelligence, like EWC, is structural regularization method which adds an elastic penalty to the second task's 's loss function:
[(0) (())) + 2 Σ C. (0. 0a,)²²
The difference from EWC is in how the elastic penalties are calculated. In synaptic intelligence, for each parameter, contribution to the loss is tracked via the product of the gradient of the loss and stepwise change the parameter value. The coefficient then constructed such that changing the parameter by amount equal to its change during task-1 I training will incur a loss penalty equal to the change in loss from the initial value. Our implementation does not differ from the standard implementation.
B SUPPORTING EXPERIMENTS
Here we present experiments omitted from the main text for space, which help flesh out some of the conclusions. This section is roughly organized with layerwise experiments first, followed by semantic experiments.
B. LAYER RESET EXPERIMENTS
As described in the main text, we perform an additional experiment to check that deeper repre- sentations get corrupted more than shallower representations. This experiment, corresponding to Figure [2] shows that when layers are reset to their pre-forgetting value, performance improves more dramatically when deeper layers are reset first.
B.2 TASK-SPECIFIC STAGES
Our observations that forgetting is driven primarily by the higher layers suggests that in a setting with known task identities (i.e. in the split CIFAR-10 setting described the main text but not the CIFAR-100 100 distribution-shiftt setting), making the deeper stages of the network task-specific would likely mitigate large portion of forgetting on the old tasks while still allowing for full performance on the tasks. In particular, since the changes in deeper represntations their shallower counterparts, only a few task-specific stages should be necessary to get significant performance increases the old task. For each of the three network architectures considered
much greater than
16

---

Published as a conference paper at ICLR 2021
ResNet CIFAR10
VGG CIFAR10
DenseNet CIFAR10
0.9
Acc (T3pkc1
0.8
Acc: (Tack 11
Acc (Task 31
0.8
0.7
0.7
0.7
0.6
0.6
0.6
Top-down Bettorm-M..
Top-down Bottom-up
Top-down
0.5
Bottom-up
0.5
0
stages reset
N stages reset
stages reset
Figure 12: Layer reset experiments. After training on Task we reset contiguous blocks of layers to their values before training and record the resulting accuracy on Task (bottom row). We see a significant increase in accuracy when resetting the highest N layers (blue line) compared to resetting the N lowest layers (gray line). Together, these results demonstrate that higher layers disproportionately responsible for forgetting.
(VGG, ResNet, and DenseNet), we investigated the performance of these task-specific networks on the split CIFAR task. We intend these results be further evidence of the role of deeper layers in forgetting, not a proposed mitigation scheme.
VGG
ResNet
DenseNet
4.5
accuracy
Epech
Epech
Epoch
Forgetting in networks with task-specific stages on CIFAR-10. Using the same settings as Figure I of the main text, we run networks with task-specific layers on split CIFAR-10. We make deeper layers task-specific before shallower layers. Performance increases with number of task-specifie layers, denoted by the line color (see legend). In all architectures, task-specific stages is sufficient recover significant performance.
Figure 13:
Results, shown in figure 13 show that for all architectures, performance dramatically rises with the number of task-specific stages. For all architectures, having the deepest two stages be task-specific recovers a significant fraction of the forgotten task performance.
B.3 LAYER RESET AND RETRAIN EXPERIMENTS
As an additional probe of the degree to which catastrophic forgetting impacts the representations of various network layers, we perform reset-and-retrain experiment on the split CIFAR-I0 task, using the same training setup as in Figure 3 of the main text. The reset-and-retrain experiment is designed to probe the degree to which network representations get corrupted and can no longer be used by a network of the same architecture to productively classify images. To investigate this, after having trained the network on Task we freeze parameters stages through to their post-task-2 values, reset the parameters of stages N + onwards to their post-task-1 values, and then retrain those parameters on the Task loss function. This is natural, operational measure of the degree to which layer 1 through N representations have been corrupted.
The results of this experiment, shown in Figure 14 tell a similar story to that of Figure ?? the main text. In particular, recovers nearly the pre-forgetting performance of the network.
retraining the network with all but the last couple of layers frozen
Some of the performances in Figure 14 even outperform the fre-forgetiig network performance, suggesting that network representations of intermediate layers may actually be improved by training on Task 2. Further suggestive results shown in Figure 15 in which retrain the full network
17

---

Published as a conference paper at ICLR 2021
again Task I after the training Task 2. For ResNet and, to degree, VGG, training on the second task clearly improves performance on the first.
DenseNet
0.85
0.9
0.80
0.75
0.8
0.70
0.65
0.7
0.7
0.60
0.8
0.6
0.55
0.50
0.3
0.3
stages reset
N stages reset
N stages reset
Figure 14: Layer reset aud retrain experiments. Consistent with the observations in the freeze- training.i layer reset. and CKA experiments, resetting and retraining only few stages from the top is enough recover performance before forgetting.
VGG
ResNet
DenseNet
0.85
0.9
0.80
0.8
0.7
0.70
0.7
0.65
0.8
0.6
0.5
0.55
0.4
16. Fe.
f1.
0.50
50
100
150
200
250
50
100
150
200
250
50
100
150
200
250
Epoch
Epoch
Epoch
Figure 15: Training on different can improve performance on the original task. Networks are trained on two five-class splits of CIFAR-10. The gray curve shows the test accuracy on task during training on solely that task for 270 epochs; the teal curve shows the task-O test accuracy when training task 0 for 90 epochs, then task for 90 epochs, and task 0 again for 90 epochs. While for duration of training on task 1, the performance is quickly recovered once we begin training on and fact surpasses the performance of the model only exposed to task-O data. This effect is consistent across different splits of the original dataset. Of the architectures we studied, this effect is most pronounced for ResNet, but also exists some degree for VGG and DenseNet models.
sharp performance drop
B.4 REPRESENTATION CHANGES IN THE CIFAR- DISTRIBUTION SHIFT EXPERIMENTS
Here show the change in representations due training distribution-shifee CIFAR- 100 for the VGG (Figure 16) and DenseNet (Figure 17 networks, to accompany the ResNet results in the main text.
Densenet on CIFAR100 Distribution-shiff task
Similarity
Acc Acc(11ss) (1ask 2)
km0
8:0
Task
(a)
(b)
(c)
680
10
25
33
45
10
15
Epoch
Stage
Num frozen stages
Figure 16: Change in Densenet representations under CIFAR-100 distribution shift.
18

---

Published as a conference paper at ICLR 2021
VGG fc CIFAR100 Distribution-shift task
Similarity
Acc (Task 2)
Accuracy
1.4
0.3
OKA
Task Task 2
02
(a)
(b)
(c)
Epoch
Stage
Num frozen stages
Figure 17: Change in VGG representations under CIFAR- distribution shift.
B.5 ADDITIONAL EXPERIMENTS FOR SUBSPACE SIMILARITY OF MITIGATION METHODS
We include additional experiments on computing subspace similarities for different mitigation methods across different architectures. The results support the orthogonal storage of Task 1, Task representations by replay, while EWC and SI enforce greater feature reuse in the higher layers.
CIFAR10 Densenet replay Subspace Smilarity
CIFARIO Dengenet EWC Subipace Smilarity
GFARIO Densenet SI Subspace Simlartt
CIFAR100 Densenet replay Subspace Similerity
A7
0.
0A
Thaa), Tek2)
Thxl): Takk Zhril Taal had Trsky
Diak: Souts (riak) Soat Tawd)
(rand Tesla Teal2) Thai2 Teak Fed Trii!
sage
stape
kaye
CIFAR100 Donsenet SutspaceSim
CIFAR100 EWC Subspace5im (Twak Teak 1 Task 2 themiig)
CIFAR100 Densanet SubcpaciSim (Task 2. Task 1 poss tratning)
QFAR10 VGS repbyy; Subspacesim (Tank 2, Tast posl-Task Task trainin))
(Tank 2, Task 1 post-Tede trianing
3.55
6.86
365
Sibguare Sirakrdy
415
15
20
25
25
25
29
0.79
65
10
15
20
25
Figure 18: Subspace similarity analysis reveals differences between different mitigation methods, with replay storing Task 1 and Task representations orthogonal subspaces, while EWC and SI promote feature reuse in the higher layers. Additional results, compare to Figure Bli in the main text.
B.6 LINEAR REGRESSION ON FORGOTTEN NETWORK ACTIVATIONS
To further probe the manner in which catastrophic forgetting affects the internal representation of networks, we perform the following experiment (in the split CIFAR-10 setting). We train (multi- head) model on the first task for some time, resulting in model Mi, and then train on the second task for some time, resulting in model M2. Due forgetting, the model M2 of course performs significantly worse than does Mj on the first task. But, to probe how much information has been lost internally, we train a linear model classify between the first-task categories using only the internal activations of M2 on the first-task images. We measure both the amount of performance which we can recover using this linear regression, and the weight placed by the linear model on various stages of the network.
Model performances are given in Table For all three architectures, the performance of the model before forgetting is naturally the highest, with performance drop of up to forty percent due to forgetting. Remarkably, however, training a linear model on the internal post-forgetting activations recovers a substantial fraction of the performance loss: for ResNet and VGG, the linear model on post-forgetting activations is only a percent less accurate than the model before forgetting. As a control, also train linear model the activations of networks at initialization (denoted the
19

---

Published as a conference paper at ICLR 2021
DenseNet
ResNet
VGG
rre-forgetting accuracy
0.869
0.7714
0.7970
forgetttg accuracy
0.4754
0.5924
0.5877
Linear regression on activations
0.802
0.7592
0.7896
Linear regression on random lift
0.447
0.4198
0.543
Table 2: Performance of linear model trained internal activations.
random lift in the table). Because the number of activations is much higher than the number of pixels in CIFAR-10 image, this control is necessary to ensure that the boost in performance we see not simply due to performing high-dimensional lift which renders the data linearly separable. As the table shows, however, this random lift performance is quite poor.
Figure 19 shows the weights placed by the linear model the various internal activations of the network. We use the activations both before and after forgetting, Before forgetting, linear model trained on the activations places most importance forgetting, the model learns to use information stored in earlier-layer representations of the network. The reduced weighting of the final layers is consistent with the experiments we describe in the main text showing that deeper representations suffer the most from forgetting.
the final layers, expected; however, after
VGG: Before forgetting
ResNet: Before forgetting
DenseNet: Before forgetting
weight histogram
0.8
0.6
0.2
6.2
0.0
02
-0.6
-0.0
-0.6
-0.0
VGG: After forgetting
ResNet: After forgetting
DenseNet: After forgetting
Linear weight histogram
0.8
0.6
0.2
6.2
6.
-022
-0.
0.6
-0.6
-0.0
Layer
Layer
Layer
Figure 19: Weights of linear models trained on internal activations.
B.7 CKA MEASUREMENTS MITIGATION METHODS
This section gives CKA similarity measurements to accompany those in the main text presented in figure 5) The main text figure showed results for ResNet; here we show corresponding results for all architectures both split CIFARIO 10 and split CIFAR 100 tasks, in figures 20 and 21] repsectively. These plots show that the results discussed in the main text apply broadly across architectures and datasets.
B.8 OTHER CATEGORY RESNET AND DENSENET
In Figure 9 of the main text, showed for the VGG architecture that adding an 'other' category (by sampling from all images in the dataset which are not in either the main tasks) can mitigate forgetting by reducing the similarity between the model's representations of the task data versus
20

---

Published as a conference paper at ICLR 2021
DenseNet on CIFAR10 EWC
Densenet on CIFAR10: Replay
Densenet on CIFAR10 S1
Similarity
CKA Simil arity
CKA Strilarity
0.
2.4
A-0.0
1-0.1
A-0.0
Am30.0
r.0.2
A-0.025
1-80.0
[-0.3
A-0.05
1-120.2
622
f-0.4
A-0.075
Stage number
Stage number
Stage number
ResNet on CIFAR10 EWC
Resnet on CIFAR1O: Replay
Resnet on CIFAR10 -
1.
Similarity
CKA Similarity
CKA Similarity
8.8
83
2.6
A-0.0
1-00
644
A-0.025
0.5
2-60.0
m-0.3
r-0.2
A-0.075
4-0.05
120.0
6:2
1-0.4
1=0.1
Stage number
Stage number
Stage number
VGG on CIFARIO EWC
on CIFARID Replay
VGG on CIFAR10 SI
3.0
CKA Similanity
3.-
CKA Similarity
CKA Sinilarity
0.2
100.3
A-0.0
A-0.0
3-30.0
A-0 025
A-00.0
r-0.3
A-0.05
1-120.0
1-0.4
A-0.0/5
Stage number
Stage number
Figure 20: CKA Analysis of mitigation methods the split CIFAR1O task
21

---

Published as a conference paper at ICLR 2021
Densenet on CIFAR100: Replay
Densenet on CIFAR100
CKA Similanty
255
CKA Similarity
Similaity
r=0.]
A-0.5
1-0.2
A-0.000
r-0.3
A-0.00 0.000
f-0.4
A-0.075
Stage number
Stage number
Stage number
ResNet on CIFAR1O EWC
Resnet on CIFAR100: Replay
Resnet on CIFAR100
2.0
8.
CKA Similarity
CKA Smillariy
CKA Similarity
6.-
0:3
A-0.0
A-0.0
A=0.001 0.005
20.0
ro0.2
A-0.05
1-60.0 0
[-0.3
A=0.075
1-120.2
1-0.4
A-0.1
Stage number
Stage number
Stage
VGG CIFAR100 EWC
CIFAR100:
CKA Similaaity
3.55
Simtiartty
0.
CKA Similarity
3:52
0.6
1-50
J-25
1-0.2
A-0
2-1.
1+0.3
A-0
2w1.5
1=0.4
A-0.075
Stage
Stage number
Figure 21: CKA Analysis of mitigation methods the split CIFAR100 task
22

---

Published as a conference paper at ICLR 2021
Densenet CIFAR10:
Representation Similarity Before forgetting
Representation Similarity
Other category helps
After forgetting
1.00
1.00
Tack Accuracy
Trace overlap
0.80
0.85
0.75
None
(b)
(c)
0.80
0.80
0
10
20
40
50
60
Epoch
Stage number
Stage number
Figure 22: Adding an 'other' category during initial training helps performance through orthogonaliza- tion: Densenet. Model is trained initially classifying two objects: airplane/automobile. and subsequently on animals: deer/dog. When add the 'other' category. it comprises random sampling of images from all other classes. This reduces the similarity between the model's representations each tasks' datapoints, leading to improved continual learning.
Resnet CIFAR
Representation Similarity
Representation Similarity After forgetting
Other category helps
B Before forgetting
0.95
Tack Acaracy
0.95
0.90
0.90
Trace overlap
0.85
0.90
0.80
(a)
None
(b)
(c)
0.80
0
10
20
30
40
50
60
Epoch
Stage number
Stage number
Figure 23: Adding an "other' category during initial training helps performance through orthogonaliza- tion: Resnet. Model is trained initially classifying two objects: airplane/automobile, and subsequently on animals: deer/dog. When add the 'other' category, it comprises random sampling of images from all other classes. This reduces the similarity between the model's representations each tasks' datapoints, leading to improved continual learning.
the task 2 data. our analytic model suggested, increasing this dissimilarity should help forgetting.
Figures 22 and 23 show corresponding plots for the Resnet and Densenet architectures.
B.9 ADDITIONAL SEMANTIC, EXPERIMENTS
In this section, comprising figures experiments presented in Section 6 Figure In particular present results for sequential binary classification tasks, four-class classification followed by two-class classification, and the CIFAR- 100 distribution shift task. We find consistent results across different choices of objects and animals and across VGG, ResNet, and DenseNet architectures. In particular, we find that when models are encouraged to distinguish objects and animals, dissimilar categories lead less forgetting. In contrast when models are trained with only a single semantic category, there is no pressure to distinguish objects from animals and similar categories lead to less forgetting. We observe the intuitive semantic groupings of categories. For VGG in Setup (Figure 25) the truck category behaves similarly animal classes in that the performance suffers more when the second task is animal classification then when it is object classification.
and 26 we present additional realizations of the semantic
exception to
C FROZEN-FEATURE MODEL: FURTHER DETAILS
Here we consider extensions and applications of the analytic model introduced in Section 6.2 Both the single and multi-head models relate the change in predictions during training second task to
23

---

Published as a conference paper at ICLR 2021
VGG
ResNet
DenseNet
Accuracy
0.8
0.8
Task
0.6
cat, horse
0.6
plane, car
50
100
150
50
100
150
50
100
150
Task 1 ship, truck, Task 2 cat, horse or plane, car
1.0
Accuracy
Task deer, dog
0.6
0.6
ship. truck
Task plane, car, Task 2 deer, dog or ship, truck
0.9
Accuracy
0.8
0.8
0.7
0.7
Task
deer, dog
0.6
0.6
0.5
ship, truck
0.5
0.5
50
100
150
50
100
150
50
100
150
Task cat, horse, Task 2 deer, dog ship, truck
0.9
0.9
0.8
0.8
Accuracy
0.8
Task cat, horse
0.7
0.6
0.6
plane,
0.4
0.5
50
100
150
50
100
150
50
100
150
Epochs
Epochs
Epochs
Task 1 deer, dog, Task 2 cat, horse or plane, car
Figure 24: Sequential binary classification tasks show consistent semantic structure. train on sequen- tial binary subsets of CIFAR-10, distinguishing between two animals or two objects. We find that less forgetting occurs when the initial task is more similar to the second task.
the overlap between features, O(z,z') En 9u (x)), evaluated the initial and second task data
Multi-head model In the main text we focused for simplicity an analytic model of single head forgetting.
Here we construct a solvable model for the multi-head setup. We consider model
"This feature overlap matrix shows frequently when studying linear models, where it is sometimes known kernelJacot
reproducing kernel, or wide neural networks, where it is called the neural tangent kernel Jacot al. 2018 and governs SGD evolution.
24

---

Published as a conference paper at ICLR 2021
VGG
ResNet
DenseNet
1.0
1.0
True positive rate
0.5
0.5
Task classes
plane
deer
car
dog
0.0
50
100
150
50
100
150
50
100
150
Task 1 plane, car, deer, dog, Task 2 cat, horse
1.0
True positive
0.5
0.5
0.5
Task classes
deor dog
0.0
car
0.0
Task plane, car, deer, dog, Task ship, truck
1.0
True positive
0.5
0.5
0.5
Task 1 classes
0.0
dog
truck
0.0
0.0
50
100
150
50
100
150
50
100
150
Task deer, dog, ship, truck, Task plane,
1.0
1.0
1.0
True posiive rate
Task classes
deer
ship
0.0
truck
0.0
0.0
50
100
150
50
100
150
50
100
150
Epochs
Epochs
Epochs
Task 1- deer, dog, ship, truck, Task 2 cat, horse
Figure 25: Sequential four class and binary tasks show decreased forgetting for dissimilar tasks.
train on sequential subsets of CIFAR-10, initially distinguishing between four classes (two animals and two objects) and then distinguishing between either two animals or two objects. We find that less forgetting occurs for categories that dissimilar those used in the second task.
consisting of non-linear features g(w; w) a linear layer with weights, 0, and read out head (9))
= Σ 1. (w;)).
Here runs the feature dimensions. while over the output dimensions of our additional linear layer. For the initial task we use the head (1),, while for the second task, we swap the head and use (2)
Again we consider model trained without restrictions on Task using head O For Task first swap the head and then perform head only training head (2),, After training the head,
25

---

Published as a conference paper at ICLR 2021
VGG semantics on CIFAR100
DenseNet semantics on CIFAR100
0.9
1.0
accuracy
Class accuracy
0.6
0.7
0.5-
0.6
0.4-
0.5
Electrical Devices
Cariioors
Electriccl Devices
Camiieres
Food Containers
Vehicles 2
Food Conta ieer
Vehicles 2
0.1
0.2
20
40
60
80
100
120
20
60
80
200
120
Epoch
Epoch
Figure 26: Semantic structure of forgetting on the CIFAR-100 distribution-shfi task for VGG and NenseNet models
VGG
ResNet
DenseNet
No headfirst training
0.147 = 0.051
0.183 = 0.001
0.162 + 0.008 0.700 0.012
5 epochs headfirst training
0.550 0.029
0.673 = 0.010
Table 3: Headfirst CKA similarity. CKA similarity between the readout layer after headfirst training and after final training on the second task.
freeze the head, (2), and the features g(w; xz) = ( æ). where i is the value of the feature weights after training on the initial task. This model is again inspired by our observation that forgetting is driven by changes in the latter network layers. The multi head model is additionally inspired by our observing that after head first training, the CKA similarity between the second task head before and after second task training is relatively high (see Table 3
After training the second task head, we continue training the weights 0. The model output evolves as (L( v)
Σ (zz,z') (L(f(((x))
A R (x)=
0f A (0)
affe/ a'.'EE (2)
Again we find that if the representation overlap matrix ((2, En In (w; x r) ( (w;; x²) is small between the features evaluated on the initial task and second task data then the predictions do not change significantly. If this overlap is zero, then the predictions are constant. In the multi-head setup we are considering here, we see that the change in predictions is further proportional to the similarity between the model heads, h (6)
Finally, we note that we have considered final linear layer before the readout head for simplicity. One can instead include a ReLU non- lineariiy without changing the essential point, that the change in model output is governed by the overlap matrix, O.
Rotating representations in the analytic model. We can use our analytic model (Section 6.2 to investigate how forgetting depends on representation similarity. We again consider sequential binary tasks (car vs plane) followed by (cat vs horse). We then tune the overlap of our initial task and final task features by explicitly rotating the frozen features for the second task to produce new features g' (0; w)
g((0; (; ((0))((;;))) x).
(10)
Here, if we have P features, R(0) E PxP is a one- oee-ooneeeeeee ane-parameter family of rotation matrices designed such that R(0) = 1 and gf z)gg ạ /2; a. for x(2) and x' E x(2) is small. Explicitly, we take
[P/2]
R(0) VT II
(0VV
(II)
s=1
26

---

Published as a conference paper at ICLR 2021
Effect of rotating representation
Interpolating task similarity
0.8
Accuracy
0.7
0.7
0.6
0.6
Rotation angle
forgetting Post-forgetting
0.5
m/
10
20
m//8
n/4 3n/8
n/2
Epech
Representation Angle
Figure 27: Rotating representations shows diminished forgetting for dissimilar tasks. Here consider the performance our frozen feature model on sequential binary CIFAR-10 tasks (Task plane, Task 2: cat horse). We use model built from two hidden-layer fully connected network with ReLU activations. increase decrease representational similarity by explicitly rotating the second task features and find predicted that this leads to diminished forgetting.
L2 Weight Change
Cosine Weight Change
1.0
0.8
0.4
Forgetting No torgetting
Forgetting No forgetting
0.2
0
10
20
30
40
50
0
10
20
30
40
50
Epochs
Epochs
Figure 28: Stabilizing final layer weights is not necessary stop forgetting. Here look the change in final layer weights during second task training for model with severe forgetting (gray) and forgetting mitigated by rotating the second task representations (teal). We see that final layer weights become less and less similar in both cases. The setup is identical to Figure 27
where rıj (0) is the two dimensional rotation matrix between axes and j. and the orthogonal matrix appearing in the singular value decomposition of the feature by data matrix the initial task, g(x()) = UDV. We find that explicitly enforeing task dissimilarity via large rotation minimizes the effect of forgetting (Figure 27
Stopping forgetting without stabilizing weights We saw in Section that both EWC and replay buffers stabilize deeper network representations. As mentioned above, this need not be the case priori. As an illustrative example, construct setup where sequential training leads to no forgetting
despite a significant change to final layer weights. We consider the single head model of Section 6.2
where the Task and Task 2 representations
completely orthogonal, O(z,x') 0 for r E X
and x' E (2)
gain this case the Task predictions remain constant throughout all of Task training. despite significant changes the final layer weights and predictions the second task. In Figure 28 we present an example of this setup. The representations are again made orthogonal by explicitly rotating the second task features by the rotation matrix, R(x/2) defined in Equation equation
27

---

Published as a conference paper at ICLR 2021
D ADDITIONAL EXPERIMENTS
This section presents two additional experiments which are slightly unrelated to the main thrust of the paper. We examine how the degree to which network forgets is influenced by its width, and also how training the head independently of the rest of the network, a multi-head setting, can mitigate forgetting.
D.1 FORGETTING VERSUS NETWORK WIDTH
VGG
ResNet
DenseNet
Performance loss (%)
0:00
c.0
0:4-
Base channels
Base channels
Growth rate
1:00
232
112
344
116
12 24
Parameeters (1e5)
Parameters (1e5)
Parameters (1e5)
Figure 29: Forgetting as function of network width. (a) VGG, (b) ResNet, and (c) DenseNet networks of varying widths were trained on two subsets of CIFAR-10 classes. We trained each of these (multi-head) networks for 30 epochs on each task with constant 17, enough achieve perfect training accuracy. Further training details can be found in the text. From the plots, which show the percent drop accuracy (on task 1) due to training on task the effect width on the performance drop seems to be minimal, even sweeping across roughly a factor of ten in the number of model parameters.
Inspired by recent work of Gilboa & Gur-Ari (019) showing that wider networks can learn better features than their narrower counterparts, we sought to determine whether wider networks also exhibit less catastrophic forgetting than narrower ones. We tested forgetting versus width for VGG, DenseNet, and ResNet networks on the split CIFAR-10 task in the multi-head setting. For VGG and ResNet, the width was varied by changing the number of channels in the initial convolutional layer and scaling the rest the network layers concurrently while maintaining the network shape; for DenseNet, the width was varied by by changing the growth rate.
networks on each task improved as they grew wider, the amount of forgetting in each network, measured by the percent drop in accuracy (on task 1) due to training on task, exhibited a minimal dependence on network width. These results are shown in Figure 29
Surprisingly, though we found that the performances
D.2 HEADFIRST TRAINING
When training multi-head networks on sequences of tasks, we found that the networks suffered smaller performance drop on the original task, i.e. forgot less, if only the new task head was trained for a few epochs prior to training the full network. In Figure 30 this effect is shown for all three network architectures on the split CIFAR- 10 task. Training only the head for up to five epochs seemed to improve the original-task performance without sacrificing performance suggests that coadaptation between the freshly-initialize head and the rest the model contributes to forgetting when there is headfirst training.
the new task. This
28

---

Published as a conference paper at ICLR 2021
VGG
epochs epochs epochs
Task 1. Accuracy
Accuracy task
Accuracy. task
0.79
541
Task Epochs
Epochs
Accuracy
ResNet
Task Accuracy
Accuracy, ta.kk
Accuracy,
0.33
0.79
0:08
Epochs
Epachs DenseNet
Accuracy
3.4
0.42
3.1
0.80
Task 1 Accuracy
441
Task Epochs
Accuracy
Figure 30: Effect of training readout layer separately between task switches. (a) VGG, (b) ResNet, and (c) DenseNet models were trained on two subsets of CIFAR-10 classes: task O (dog. frog, horse, ship, and truck), and task 1 (airplane, automobile, bird, cat, deer). When switching tasks, we first trained only the final classification layer (known as the readout layer or head) for a specified number of epochs (here 0, 1, and from a random Gaussian initialization. During this period, the rest of the model parameters are held fixed; after the head-oni -old training, the entire model trained as usual. Across all architectures, training only the head consistently improves the forgetting performance of the model (its accuracy on task 0) while having a negligible impact on the performance on the next task (task 1).
E FIGURES FOR ANONREVIEWER2
F IMAGENET RESULTS FOR TASK SEMANTICS
Figures corresponding results tables in the main text.
29

---

Published as a conference paper at ICLR 2021
Weight change through training
Weight change through training
10a2
ee Wo
[IM W W
0.95
0.90
10'1
co0.00
0.80
10a4
0.75
Move Waan
0.70
Weam Winn
Stage
Stage
Figure 31: Stage-wing biieewwiseaa weight change between initial weights and weights after Task (blue) and between weights after Task I and Task2 (orange) for a model trained sequentially on disjoint class subsets of CIFARIO. (a) Mean distance. (b) Cosine similarity. We see no sign of significantly more change in last layer weights, and in fact see the reverse trend in & distance.
p. p_(i>-y_00 (i)
Task ship, truck
Task horse
304.0
Task plane,
303.5
60g
303.0
302.5
302.0
-27.10
-26.95
-26.85
Mean
Figure 32: Mean and std deviation for the residual, pt for a model trained on ship v truck evaluated on cat, horse data (blue) and plane, car (orange). We see no clear separation between object and animal data. Each point represents model trained with a different seed.
Task 1: 10 Artifacts
Task 1: 10 Animals
Task 10 Animals 10 Artifacts
Task Animals or 10 Artifacts
0.7
0.5
Normalized accuracy
0.6
Accuracy
0.4
0.5
0.4
0.3
Artifact-
Animal->Anime
02
Artifact-AAnmmal
0.2
Animal- Artifact
15
20
Epoch
Epoch
Initial classification between 10 artifacts
(b) Initial classification between 10 animals
Figure 33: ImageNet Two-task 10 class sequences in which similar tasks cause less forgetting.
30

---

Published as a conference paper at ICLR 2021
Artifacts Animals
Task 2: 10 Artifacts
Task 2: Animals
1.0
1.0
Normalized accuracy
Normalizzer alizadd accuracy
0.8
04
0.4
artifact animal
0.2
artifact
animal
0.-
10
20
30
Epoch
Epoch
(a) Final classes: artifacts
(b) Final classes: animals
Figure 34: ImageNet Two-task 20-class followed by 10-class sequences in which similar categories are forgotten more.
31


Reviews:
[{"review_id": "nbhp9n3jJ96", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThe paper studies the role of (the depth of) hidden layers on catastrophic forgetting when a neural network is trained sequentially. For this, first, the paper analyses the similarity of the hidden representations through the centered kernel alignment (CKA) and subspace similarity measures. This analysis suggests that the deeper layers become more dissimilar when training is performed on a new task. Further, the subspace analysis suggests that sequential training forces the network representations of the earlier tasks to lie in the same subspace as that of the more recent task. The paper then studies the effect of different mitigation strategies for catastrophic forgetting and shows that for experience replay type approaches the performance is retained by keeping the subspaces of different tasks from overlapping from each other, whereas, in the regularization-type approaches the performance is retained by keeping the subspaces similar and resuing the previous feature mappings. Finally, the paper studies the effect of task similarity on catastrophic forgetting and shows that the intermediately similar tasks are the ones that are most prone to forgetting. The analysis is done on CIFAR-10 and a variant of CIFAR-100 datasets. \n\n**Positives**\n\n1- The experiments are done thoroughly and different hypotheses are explored in detail. \n\n\n2- I quite like the task semantics part of the paper(Section 6). The authors develop a nice and somewhat novel argument in that section although there exists a simpler explanation which I will describe below.\n\n**Negatives**\n\n1- Meta comment: I can’t help but feel that the authors had it backward. Instead of looking at which layers contribute to more forgetting, it should be which layers forgetting effect the most. The latter lens changes the perspective of looking at the problem and most of the conclusions drawn in the paper seem obvious. Perhaps, then, most of my critique of the paper would be from that perspective. \n\n\n2- **Deeper layers get more affected by the backprop**: Forgetting of previous tasks usually happens when the weights are changed from the optimal value of previous tasks. The change in weight is directly related to the norm of gradient received on a certain layer. In most deep networks, on image classification tasks this norm decreases as the gradient backpropagates to earlier layers. Hence, one sees that the weights of the earlier layers remain stable whereas the weights of deeper layers change quite a lot. So it is quite natural to observe that with training on a new task the representations of deeper layers change quite a bit. Note, I reach a similar conclusion as the authors but I don’t call it deeper layers causing more forgetting but rather forgetting causing deeper layers to change more. So to me, this contribution of the paper is stating the obvious.\n\n3- **Sec 4.1**: The authors write that they observe that lower layers can reliably be frozen with very little impact on Task 2 accuracy. This again is obvious for natural image classification-based benchmarks (take fine-tuning for example). Perhaps, this conclusion will not hold for permuted MNIST benchmark where input distribution is quite different for each task. \n\n4- **Task Semantics**: While this section is quite good and I enjoyed reading how the authors build the section but the puzzle that the authors describe at the outset can be explained more simply. Look at the gradient of the softmax-CE loss received at the logits. For a given image, it would be $p_i - y_i$, where $y_i$ is one-hot encoded (i.e.) 1 for true class and 0 for all other classes, and $i \\in \\\\{1, \\cdots, C\\\\}$. In Fig 7 (a, d), when binary classification is performed, the task 1 performance remains good when similar-looking objects are showed again for task 2 (and hence forgetting is less for similar tasks). More interestingly, Fig 7 (b), in a four-way classification task, when task 2 contains plane and car, the probability assigned to the logits of deer and dog would be very small (meaning smaller gradient and hence no change in the corresponding last layer weights), whereas the probability assigned to the logits of ship and truck would be higher which means higher gradient and more change in weights and more forgetting. A similar argument can be built for 7(e). It seems this is the artifact of the setup, single-/ multi-head, etc. Please correct me if I am wrong. \n\n5- It seems that the paper is written hastily. There are a few typos. \n*) Sec 3, line 2, we generally use\n*) Sec 4.2, when defining X and Y, the dimension of the matrix should contain \\times and not x.  \n\n---------------------------------------------------------------------------\n**Post-rebuttal**\n\nSee my post-rebuttal response below. I have some remaining questions but the authors addressed some of my concerns. Therefore, I am increasing my score. \n\n----------------------------------------------------------------------------\n\nLooks like the post-rebuttal response is not available for the authors to see. I am copy-pasting it here. \n\nI thank the authors for their detailed rebuttal. \n\n**Deeper layers get more affected by backprop:** I appreciate the authors' response. The paper they referred to does not seem to have conclusive proof of which layers change more during the training. For example, Figure 1 in that paper shows that on FCNs, the latter layers change more. On CNNs, Figures 14, 15, the results are less conclusive. Anyhow, when I wrote the review the setup that I had in mind was a solution of the previous task as an initialization of the next task and not a random initialization and standard training -- a setup that had been the subject of the referred paper. In the former case (continual learning case), the solution of the previous task should inform on the next task and I expect the earlier layer weights to have small changes. However, what the authors of this paper showed in the revised draft (Figure 31) is quite interesting. They showed that, in the L2 sense, the weights of the earlier layers change more than that of the later layers. This shows that despite the large weight changes in the earlier layers their representations don't change much (as measured by the CKA). Am I understanding that correctly? Could I request the authors to verify this observation on ResNets (I believe Figure 31 is for VGG), and perhaps, other datasets as well?\n\n**Sec 4.1**: Regarding freezing the earlier layers and not seeing any performance degradation on the subsequent tasks, I asked whether this would also hold for setups where input distribution changes (e.g.) Permuted MNIST. Could the authors please address this and add the experiment to the paper?\n\nMy other concerns are addressed, therefore, I am increasing my score. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "llKM8PHmDz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors provide an empirical investigation that the outer layers of a neural network are more responsible for the catastrophic forgetting effect than inner layers.  This is established by demonstrating graceful performance decays in training a second CIFAR classification task when freezing inner layers after training a first task.  Additionally, in sequential training, various notions of similarity reveal that outer layers change more.  The paper reveals that continual learning techniques (EWC, SI, a replay method) mitigates the change in outer layer image representations, as per these metrics.  The paper reveals that some methods cause outer layer features to live in orthogonal subspaces to those of previous tasks, and other methods cause reuse of existing outer layer features from previous tasks.  The authors present an additional observation that sometimes similar sequential tasks results in less forgetting and sometimes it results in more forgetting.  They reveal experiments that show that in a mixup dataset, intermediate levels of task similarity lead to maximal forgetting.  \n\nThe paper has a reasonably thorough set of random experiments exploring the effect of Continual Learning methods on feature representations.  That said, it is not clear how general these observations are.  For example in the EWC Paper (Kirkpatrick et al. 2017) it was observed that permutation tasks resulted in significant changes to inner layer representations (as measured by Fisher information) relative to outer layers.  At the very least this paper should reconcile or comment on this difference in observation with the presented results.  Ultimately, it is not clear to this reviewer what to take away from this paper.  The paper would be stronger if it was accompanied with an algorithmic improvement, inspired by the present observations, that improves on the state of the art.  That said, the reviewer suspects that the observations in this paper might reasonably lead to such an advance in subsequent work by the same or different authors.\n\nMinor comment:\n\nSec 4.2: Is the SubspaceSim definition correct?  The numerator is a quartic, but the denominator is a quadratic.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RvcQIs5R3oN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper investigates catastrophic forgetting phenomena, where machine learnings model trained on a new domain suffer from performance losses on older trained domains. The paper first finds that deeper layers are disproportionately responsible for forgetting. Next, it analyzes the effects of mitigation methods on higher layers and suggests that they stabilize higher layer representations by either enforcing feature re-use or storing tasks in orthogonal subspaces. Lastly, using analytic models, the paper investigates the connection between forgetting and task semantics and shows that the intermediate similarity in sequential tasks leads to maximal forgetting. The paper empirically demonstrates the above results on CIFAR10 and CIFAR100.\n\n##########################################################################\n\nPros:\n1. The layer-wise analysis of catastrophic forgetting and investigation of different mitigating forgetting methods are interesting, and the work is certainly very relevant to this venue.\n2. The motivation of the paper is clear, and the empirical results are convincing. \n3. The writing is clear and straightforward.\n4. The results can potentially help to suggest new approaches for developing and measuring mitigation methods.\n5. The related work section is sufficient.\n\n##########################################################################\n\nCons:\n1. The paper misses important discussions that I describe in the section below.\n2. Although the empirical findings seem useful, it would have been nicer to propose some new mitigation methods from these insights.\n3. The experiments are only done on image classification tasks with CIFAR10 and CIFAR100\n\n##########################################################################\n\nQuestions:\n1. What does \"N frozen stages\" represent in Figure 2? Does it correspond to freezing N lower layers (blocks)? What happens if only the higher layers are frozen during training? The conclusion that higher layers contribute to catastrophic forgetting is not evident from the paper's experiments in section 4.1.\n2. In figure 6, how does subspace similarity differ from one that does not use any mitigation methods? It is challenging to compare the results from figure 5 directly. \n3. How is the task similarity defined? I assume it is the similarity of the before-output layer activations?\n4. What are representation similarity before/after forgetting in figure 8 (b) and (c)?\n5. The paper does not discuss the potential reasons why domains with intermediate similarity have maximum forgetting. Does it relate to findings in section 6.3 (re-use and orthogonality)? Additionally, what would happen if different mitigation methods are used?\n6. What property (re-use vs. orthogonality) is advantageous in designing mitigation methods? It would be helpful to have this discussion in the paper.\n\nMinor notes:\n* All titles in the figures are relatively small and hard to read. It would be helpful if important parts are highlighted in the title.\n* On page 2, \"We generally uses m = 2\" --> \"We generally use m = 2\"\n* On page 2, \"Models\" --> \"Models:\"\n* It will be more apparent if the equation on page 3 is defined as CKA(X, Y) = ... Also, the equation numbers are missing.\n* On page 3, \\mathbb{R}^{n \\times p_1} instead of \\mathbb{R}^{n x p_1}. Also, shouldn't p_1 = p_2?\n* In page 7 \"in Figure 7a and7d\" --> \"in Figure 7a and 7d\"\n* In page 7, is \\hat{w} representing frozen weight?\n* On page 7, \\eta and t are not defined.\n* On page 7, \"puzzle in Figure  7\" --> \"puzzle in Figure 7\"; two spaces.\n* There are some \\citep and \\citet confusions.\n* In Lemma 1, the notations (e.g. norm of the set) look awkward to me.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6hWyr6PSUS3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper revisits the catastrophic forgetting problem that is common in multi-task learning. It concludes that the information encoded in the lower layers of the neural networks is more task-independent than higher layers, thus freezing the lower layer can reduce the risk of catastrophic forgetting. Another interesting finding is the semantic similarity between tasks can control the degree of forgetting. It also investigates several methods to mitigate this issue, with quantitive and analytical results to test their effectiveness.\n\nCatastrophic forgetting is a common issue, especially in the areas where we need large pre-trained neural networks. Most of the previous methods measure the degree of catastrophic forgetting only by the task performance, and rarely looked into the parameters. This paper not only conducts experiments to test the task performance, but also the weights that the neural networks learned. Various methods are proposed, and the idea of \"SubSpace Similarity\" is very interesting, which studies whether the model predicts the two tasks with different subspaces.\n\nThe conclusion of section 6 is interesting, which claims that the datasets with dissimilar representations can reduce the forgetting, and the model could learn to utilize features of different subspaces for different tasks. This explains why sometimes semantically similar tasks will forget more.\n\nHowever, the novelty of this paper is limited. It's already well-known that the lower layer of the neural networks captures more general knowledge and it's already good practice to only fine-tune the higher layers for transfer learning. E.g., in CV, it's common to use VGG only as a feature extractor. In NLP, fine-tuning high layers of the pre-trained transformer may yield better results. For example:  \"Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT\" (https://www.aclweb.org/anthology/D19-1077.pdf)\n\nThe analysis of hidden layer representation is an application of the CKA method. The following subspace analysis is similar to section 6.4 of Kornblith et al 2019 but with CKA added. Section 5 is an analysis of existing mitigation methods and no new method is proposed. The statement that the replay method results in lower subspace similarity is interesting, but no explanation follows.\n\nSome conclusions lack evidence. The results of this paper are based on CIFAR data with a few different settings, with conclusions supported by limited diversity of experiments. For example, to support the statement that representationally different tasks suffer less from forgetting, it only conducts two experiments (\"other category\" and \"2v2 + 4v2\") with three different datasets. The findings here would be more convincing with a broader range of tasks / data.\n\nThe presentation of the formulas could be improved. E.g, the denominator of eq(3) seems strange. The fonts of the first equation of section 4.2 is wrong, and the \"x\" above that equation should be replaced with \\times. Some of the symbols should be defined before using, like the \"w\" at the beginning of section 6.2. Also, given that w is fixed, using symbol \\hat{w} seems redundant. The section definition of Lemma 1 is not conventional because the curly brackets usually represent a set. The norm of equation (2) is undefined (I guess it's l2 norm because of Cauchy-Schwarz inequality).\n\nQuestion:\n\nIn fig 3, you show that higher layers are disproportionately responsible for the forgetting, but the dissimilarity in lower layers will be propagated to higher layers. How do you minimize the effect of propagation?\n\nTypo:\nThe definition of lambda in \"Maximal forgetting at intermediate similarity\" paragraph and the caption of Fig 9 are different.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies how layer-wise representation and task semantics affect catastrophic forgetting in continual learning. It presents two findings: 1. the higher layers contribute more to forgetting than lower layers, 2. intermediate-level similarity between tasks causes the maximal forgetting. It also indicates that existing methods employ either feature reuse or orthogonality to mitigate forgetting.

Pros:
- The layer-wise analysis of catastrophic forgetting and investigation of different mitigating forgetting methods are important and interesting.
- The paper is well-motivated and well-written.
- The results can potentially help to suggest new approaches for developing and measuring mitigation methods.

Cons before rebuttal:
- The paper misses discussion on and takeaways from the findings.
- How general are the findings? There is a different observation by Kirkpatrick et al. 2017.
- Limited diversity of experiments, because the experiments are only done on image classification tasks with CIFAR10 and CIFAR100.

The authors conducted more experiments and updated the paper with added explanations and results. The reviewers found the new evidence and arguments in the rebuttal to be convincing and the authors addressed most concerns.

In summary, the findings from this paper will help researchers better understanding and addressing catastrophic forgetting, and will be of interest to the community.
Hence, I recommend acceptance.

Decision: Accept (Poster)