Effect of scale on catastrophic forgetting in neural networks
Authors: Vinay Venkatesh Ramasesh
Aitor Lewkowycz
Ethan Dyer
Venue: ICLR 2022 Poster (2022)
Keywords: Catastrophic forgetting
continual learning
scaling
language modeling
image classification

Abstract: Catastrophic forgetting presents a challenge in developing deep learning models capable of continual learning, i.e. learning tasks sequentially. Recently, both computer vision and natural-language processing have witnessed great progress through the use of large-scale pretrained models. In this work, we present an empirical study of catastrophic forgetting in this pretraining paradigm.
Our experiments indicate that large, pretrained ResNets and Transformers are significantly more resistant to forgetting than randomly-initialized, trained-from-scratch models; this robustness systematically improves with scale of both model and pretraining dataset size.
We take initial steps towards characterizing what aspect of model representations allows them to perform continual learning so well, finding that in the pretrained models, distinct class representations grow more orthogonal with scale.  Our results suggest that, when possible, scale and a diverse pretraining dataset can be useful ingredients in mitigating catastrophic forgetting. 

Published as a conference paper at ICLR 2022
EFFECT OF MODEL AND PRETRAINING SCALE ON CATASTROPHIC FORGETTING IN NEURAL NETWORKS
Vinay Ramasesh, Aitor Lewkowyez, and Ethan Dyer
Google Research, Blueshift
(ramasesh, a alewkowycz, edyer edyex)8googgie..cm
ABSTRACT
Catastrophic forgetting presents a challenge in developing deep learning models capable of continual learning, i.e. learning tasks sequentially. Recently, both computer vision and natural-language processing have witnessed great progress through the use of large-scale pretrained models. In this work, we present an empirical study of catastrophic forgetting in this pretraining paradigm. Our exper- iments indicate that large, pretrained ResNets and Transformers are significantly more resistant to forgetting than randomly-inittaizzed,
coineddfroo--srrafeee mod- els; this robustness systematically improves with scale of both model and pre- training dataset size. We take initial steps towards characterizing what aspect model representations allows them to perform continual learning so well, finding that in the pretrained models, distinct class representations grow more orthogonal with scale. Our results suggest that, when possible, scale and diverse pretraining dataset can be useful ingredients in mitigating catastrophic forgetting.
Continual learning is current challenge in designing machine learning systems: when trained on multiple tasks in sequence, models tend to suffer performance losses on earlier tasks-this is known as catastrophic forgetring (McCloskey
Cohen 1989). Catastrophic forgetting occurs in many settings, from supervised learning for classification (Zenke et al., (017) to reinforcement learning for games (Kirkpatrick et al., 2017) and downstream applications such as translation (Bapna Firat, 2019) or medical diagnoses (Gupta et al. (021). While many methods have been proposed to help mitigate forgetting, none of these have fully solved the problem; see Kemker et al. (2017) for a
review.
Increasing model and dataset size have been key to the success of deep learning Kolesnikov et al., 2020 BIG-bench collaboration [021] Radford et al. 2019 Zhai al. 2021). recent trend has been the use of increasingly large models pretrained on diverse datasets, which are subsequently fine-tuned or evaluated without fine-tuning
downstream tasks of interest; see [Bommasan et al. (202T) and references therein. A particularly striking realization language models, which show consistent improvement across scales (Kaplan et al., 2020) and can perform well on diverse tasks with no finetuning (Brown et al., 20200). From a continual learning perspective, this behavior is especially dramatic as, without finetuning, any training signal for these tasks appears mixed into a drastically larger pretraining dataset, and thus more than likely appeared many gradient steps before evaluation. This observation large part inspired our work.
this has been large pretrained
Here, we empirically study continual learning in large pretrained models. Our results suggest that pretrained models are significantly more resistant to catastrophic forgetting than their randomly- initialized counterparts, and that this resistance improves with the scale of both the model and pre- training data. Most of our experiments concern lisioo--secificaall image classification; but we also conduct a preliminary investigation of language models.
Our key observations are as follows:
Resistance to catastrophic forgetting improves with model and pretraining dataset scale (Figure
Large, pretrained clasecclasiiiication models are significantly more resistant to forgetting than models trained from random initialization 6Fi

---

Published as a conference paper at ICLR 2022
Both supervised and unsupervised, SimCLR (Chen et al. 2020), pretraining improve for- getting (Supplement, Figure 12).
We perform initial experiments aimed at exploring why scale and pretraining reduce catas- trophic forgetting, showing that distinct class representations in pretrained models are more orthogonal than in trained- woi---fror--aaareee models (Figure
We observe similar phenomena in language models (Supplement, Figure 1I
RELATED WORK
Catastrophic forgetting Catastrophic forgetting was identified as a problem for neural networks as early as the late 1980s, as occurred in the single-hidden-layer networks that were studied at the time (see Delange et al. 2 for a review). Within the catastrophic forgetting literature, a number of potential solutions the problem have been devised: some involve replaying previous-task exam- ples during training on later tasks (Rebuffi et al. 2016; Lopez- Pazez & Ranzato [017]] (cauddhry et al.) 2018 Shin et al., 2017; Riemer et al. 2018), while others like elastic weight consolidation (Kirk- patrick et al. (017) limit the extent to which parameters can change once learned (Zenke et al.,
2017 Lee et al. 2017 Farajtabar et al. 2019).
Lee
Perhaps most similar to work are proposed mitigation methods which exploit the overparame- terization of modern neural networks to perform continual learning of multiple tasks, e.g. by using pruning in Mallya & Lazebnik (2088) task-specific subnetworks in Wortsman et al. (020)) and
recent works studying the impact of pretraining /Anonymous/ (2022) and overparametenizatoo via network width Mirzadeh et al. (202T) on forgetting.
Large-scale models The doservation-inittalll in language models (Kaplan et al. 2020) that model performance scales predictably with model and dataset size spurred increased interest in training and deploying increasingly large pretrained models. Large-scale models emerged in natural-language processing with models such as GPT (Radford al.) 2019) and BERT (Devlin) eet al., 2019), which are pretrained in an unsupervised manner and thus able to make use of the large corpora of natural text available on the internet. More recently, image-llassificatinn models have also benefited from pretraining on large datasets (Kolesmikov et al. (2020), both supervised and unsupervised; a key example is the Vision Transformer architecture of Dosovitskiy al. (2220), feichh-dsspiee not using the inductive bias provided by convolutions aatautiioonnnatiinnss downstream perfor- mance comparable to convolutional networks after pretraining on ImageNet-21k (Deng et al. 2009)
or JFT-300M (Sun et al. 2017). On many current NLP and computer vision benchmarks, large pretrained models achieve top performance.
All of the experiments in our work follow the cetrain-then-fineture method; recently, another paradigm, side-tuning, was proposed by Zhang et al.] (2020). which avoids forgetting altogether by leaving the pretrained model fixed and employing auxiliary networks for downstream tasks.
EXPERIMENTAL SETUP
Tasks: Most of the experiments in this paper are conducted in the standard, task split setting (Kirk- patrick et al. 2017 Zenke et al. 2017). We consider split CIFAR- 10 task sequences - the ten class dataset split and sequentially trained on two tasks of classes each. We also consider sequen- tial 10 and 50 class splits of CIFAR-100 100. Beyond split CIFAR, we study the following datasets: Oxford-IIIY pet, Oxford Flowers 102, Street View House Numbers (SVHN), Caltech-UCSD Birds 200 (CUB-200), 196, Domainnet/Clipart.
We also consider the more recent input- sopuuddistribution-sinn CIFAR- setup (Ramasesh et al. 2021 In this case each task has the same target types made up a fixed subset of the 20 CiFAR- 100 perclasses (c.g. aquatic mammals, fish, ...). The sequence of tasks consists of modifying which subclasses sampled from. For example, Task A may have aquatic mammals drawn from beaver and dolphin, while Task may have otter and whale. This setup allows one to probe forgetting without changing the model head, in a setting where task identity is not known to the model.
For our language model experiments, we consider next token prediction generative tasks and use the IMDb Reviews (Maas et af. 2011) and english Wikipedia (Foundation) datasets.
2

---

Published as a conference paper at ICLR 2022
In most of our experiments we use the Vision Transformer (ViT) and ResNet archi- tectures. For the Vision Transformers, we use the ViT-S and ViT-B configurations from Dosovitskiy et al. (2020) and smaller configuration we call ViT-xS ranging from 5.7-86.7 million parameters. For ResNets, we use five models: ResNet26, ResNer50, R0sNet101, ResNet1 and ResNer200- ranging from 14.0-62.6 million parameters; as in Kolesnikov al. (2020), these models use group
Vision models:
normalization (Wu & He 2018) and weight standarization (Qiao et al. 2019) in place of batch
normalization. Details can be found in Tables and in Appendix 1.2
Pretraining: Unless otherwise specified, we pretrain our image models in supervised manner on the ImageNet21k dataset, which contains approximately 14 million images in about 26000 distinct categories using the Adam optimizer (Kingma & Ba 2017). Further details are in the Appendix.
When fine-tuning on multiple tasks in sequence, we use the SGD optimizer with mo- mentum 8 = 0.9. The first task training employs cosine decay schedule with a linear warmup; later tasks in the sequence are trained with fixed learning rates. For the split-CIFAR- 10 task sequences, we use multi-head setup in which each head is initialized to zeros.
Finetuning:
RESULTS
3.1
FORGETTING IMPROVES WITH MODEL SCALE
Vision Transformers
ResNets
Error scaling
1.00
1.0
Best average error (Coe))
10-3
ACc (Task B)
0.9
(Took
0.92
0.88
VTxx5 16
lbrstiet 01
ResNet50
ResNe126
10-2-
Fit: Esug
16
ResNet152
ResNc1200
VIT
0.84
0.7
0.84
0.88
0.92
0.96
1.00
0.7
0.9
1.0
10a
Acc (lask Al
Acc (Task A)
Model parameters (N)
Vitxx 5.7m
R26
VIT-5
A50
R101 42.5m
8152
4200 Vi-U
14.0m
19 6m
23.5m
58.1m
62 6m 8m.7m
Figure 1: Forgetting frontier across model scales. Task A versus Task B performance for both pretrained vision transformers (left) and ResNets (center). Each point represents different choice of learning rate finetuning step for Task B. All models were pretrained on ImageNet2 for 90 epochs. (right) best average Task A/B error improves systematically with model size.
Our main finding is that for both pretrained Vision Transformers and pretrained ResNets, catas- trophic forgetting is mitigated to a large extent by scale: that is, larger models suffer less from forgetting. This is shown Figure for the two-task split CIFAR-10 sequence: Task {airplane, automobile, bird, cat, deer] + Task B {dog, frog, horse, ship, truck]. In order to make a fair compar- ison, we plot the test accuracies of the model on both tasks different steps of Task B fine-tuning for a variety of learning rates. This allows us to see, for each model, the maximum performance achievable on the new task for a given performance on the initial task. We call this the forgetting frontier.
Figure [ shows that the forgetting frontier for both VIT and ResNet models improves significantly with scale. While 7.7--mammettrr ViT-xS model loses about 6.5% in Task A accuracy over the course of learning to perform Task B, an 86.7M 7-raammtte ViT-B model loses less than half a per- cent. To put this in context, in a previous studies of split-CIFAR-11 10 task sequences, a ResNet18 model lost close to 40% accuracy with no mitigation methods, 5% when using a replay buffer, and 20% when using elastic weight consolidation (see e.g. Figure 1 of Ramasesh et al. (202T)).
is also important to note that this effect of scale and pretraining on improving forgetting frontiers does not seem be architecture-dependenn. Both pretrained vision mansformers-batte based on self-
3

---

Published as a conference paper at ICLR 2022
Vision Transforer
ResNet
10a
10
Best average error (Eaug)
10-1-
10-1
Oxford-IIIT IIT pet
10-2
10
Oxford Flowers 102
Caltech-UCSD Birds 200
cars196
DomaannnttClipart
10
10
10a
107
10a
Model parameters (N)
Model parameters (N)
Pets
Flowers
SVHN
Birds
Clipart
Cars
ViTs
-0.70
-0.99
-0.31
-0.40
-0.38
-0.40
ResNets
-0.89
-1.53
-1.06
-0.62
-0.62
-0.47
Figure 5: Scaling behavior on non-CIFAR, two-task sequences. (top) While the rate of improvement varies across datascts, continual-leariin performance in these two-task continual learning setups systematically im- proves with model size, consistent with a power-law. For a full discussion and plot of all forgetting frontiers, see Figure (bottom) Table power-law exponents.
attention and ResNets-bacce on convolutions exhibit improvements in their forgetting frontiers as they grow larger.
We see this improvement in forgetting performance beyond the simple two-task split-CIFAR-I 10 scenario. Across a 10-task sequence of 10-class split-CIFAR-100 tasks (Figure left), sequential 50-class splits of CIFAR-100 (Figure 3 right), the CIFAR- -00 single-head imput-distributiin-shii task (Figure44. and two-task sequences beyond the CIFAR dataset (Figure 2 large-scale pretrained models exhibit resistance to forgetting, and continual learning ability appears improve systemat- ically with scale. For detailed results, scaling plots, and forgetting frontiers in the 10-task scenario, see Section B.3 in the Supplement.
3.2 DEPENDENCE ON FINETUNING DATASET
It is interesting study how the improved resistance forgetting with scale depends the choice of finetuning dataset. In particular, it is worth noting that, though quite different in format, CIFAR- 10/100 are semantically quite similar to subsets of ImageNet21k. our pretraining set. To empirically investigate the dependence on finetuning dataset we have run additional experiments similar to those presented in Figure for the Oxford Pet and Flowers, Caltech Birds, Stanford Cars, street view house numbers (SVHN) and DomainNet Clipart datasets. Results are shown in Figure and in Figure [ We note particular that SVHN classes are numerals and so distinct from ImageNet21k classes, while the Clipart dataset consists of synthetic, rather than natural images. See the Supplement for further details.
3.3 FORGETTING IN PRETRAINED MODELS VS. MODELS TRAINED FROM SCRATCH
In the previous section, we saw that the amount of catastrophic forgetting in models pretrained on large datasets decreased with increasing model size, to the point that the largest models (VTT-B, with 86.7M parameters and ResNet200 with 62.6N 6M parameters) suffer very little performance loss from second-task training. In this section, investigate whether this improvement in forgetting is simply a consequence the improvement in performance on Task A. That is, do pretraining and scale have benefits with respect forgetting beyond simply improving the fine-tuned performance on downstream tasks?
To check this, compare the performance and amount of forgetting in pretrained ResNet models to models trained from scratch. We handicap the pretrained ResNets so that their maximum accuracy on Task A matches that of the corresponding wriieed-from-serate ResNet; this is done simply by
4

---

Published as a conference paper at ICLR 2022
Vision Trensformers
Reshett
Error sceling
OXford-IIIT pet
Best average (Gus))
!
0.8
0.3
10-1
(Task B)
6 10
0.4
0.4
4 4
resnot:
vit o 0.70
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
107
10
TOford Flowers 102
(Cool)
10-1:
0.8
0.8
Nec (Task
0.4
0.4
Go
10-3
resnet:
vit a -0.99
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
10'
10
SVHN
1.0
(Eao)
0.8
0.8
Rcc (Teck
0.4
0.4
10-1-
00
resnec: i-1100
vit -0.31
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
10'
10
Caltech-UCSD Birds 200
1.0
Best average (cmo))
1 10
0.8
0.8
Nec (Task Bi
0.6
0.6
2 10
0.4
0.4
10-1
resnot: a 0.62
vit -0.00
0.0 P8
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
107
10
1.0
cars196
1.0
Best average (Kava)
40-10-)
11
Dec (Takk
0.6
0.6
0.4
0.4
2x
0.2
0.2
resnel: a -8.62
vib 0.38
0.0
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
107
10
1.0
Domaannet//Cipart
1.0
4x
Acc (Teak 0)
0.6
0.6
ResNet26
2 x
Restiet101
0.2
VIT.S 16
0.2
Restbw119?
Best
resnet: a -0.47
VT.8 16
Rochut200
-0.00
0.0
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
1.00
10'
10*
Acc (Task
ACC (Task
Model gerameters (N)
Figure 2: Forgetting frontiers for two-task sequences beyond CIFARIO and CIFARI00. Fine- tuning settings for these experiments are shown in table (supplement), while the datasets them- selves are descrbed in section B.4 [ (supplemen)) As with the CIFARIO and CIFARI00 datasets, we find that model scale beneficial from the perspective of continual learning, though especially for smaller ResNets the behavior is less clean than on CIFAR. Scaling exponents also vary with dataset.
5

---

Published as a conference paper at ICLR 2022
ViT-B: Spiit-CIFAR100 10x10
VIT CIFAR100 50v50
1.00
0.98
0.9
0.8
Accuracy
0.96
Vh
0.94
0.6
VIT-S_16 ___6
0.92
0.5
VIT-x5_15 _6
VIr-B 16
0.90
400
1600 2000
0.4
0.6
1.0
Step
AcC (Task
Figure 3: Split CIFAR-100 tasks. (left) Sequential training of pretrained VIT-B model on 10-task quence, where each task consists of classifying between 10 categories in the CIFAR-100 dataset. In this contin- ual learning setup, model is able to learn all 10 tasks without much forgetting: on average, the model loses 1.8% accuracy on each task; at most, the model loses 2.9%. (right) Forgetting frontiers for vision transformers on 2-task, 50-class split CIFAR- 100, showing the same increase with scale on the split-CIFAR- R-10 version.
Vision Transformers
ResNets
1.00
1.00
0.95
0.95
Acc (Taok
0.85
0.85
0.80
0.80
ResNet26
ResNec50
viTaa5 16
Restiet101
Peshiel52
VIT8,16
0.70
0.73
0.8
Acc. (Tesk Al
Acc (Tesk
Figure 4: Single-head models CIFAR-100 distribution shift task. As with the multi-head setup, the forgetting Vision Transformers (left) and ResNets (right) largely mitigated by pretraining and scale. The full task details are given in Appendix A.I.3
reducing the number of steps for which we fine-tune. 6 shows the results of this experiment, with additional results and details in Appendix C.I] While the the same performance as their pretrained counterparts on Task A, their Task A performance degrades significantly more than the pretrained ResNets when trained Task B; pretrained ResNets forget less, and this improves with model scale.
ResNets achieve
This result suggests that pretraining and model scale confer benefits upon models which go beyond simply improving performance. From Figure 6 it is also clear (see also Figure 19 in the appendix, which plots the same data different way) that pretraining required to see the benefits of scale. Simply scaling up the model size in trained-from-seree from--cratch ResNets does not yield clear improvements in their forgetting performance.
We focused in this section exclusively ResNets (rather than Vision Transformers) because ob- taining decent performance on image classification tasks with Vision Transformers seems to require pretraining. In experiments, for example, fcrteddctom-scratch VIT-S model unable to achieve better than 75% test accuracy on five-class CIFAR- 10, while the same model, pretrained, achieved better than 98% accuracy.

---

Published as a conference paper at ICLR 2022
ResNet26
ResNet101
ResNet200
1.0
1.0
1.0
0.9
0.9
0.9
(Tosk
0.8
0.8
0.8
0.71
0.7-
0.7
0.6
0.6
0.6
0.5
0.7
0.8
0.9
1.0
0.6
0.7
0.8
0.9
1.0
0.6
0.7
0.8
0.9
1.0
Acc (Task A)
Acc (Tosk A]
Acc (Task A)
Figure 6: Pretrained versus trained-from-scrrate models. ResNet models trained from scratch (gray) exhibit inferior forgetting performance than pretrained models (colored), even for the pretrained models shown here, which were handicapped during fine-tuning, in order to match Task A performance. Futhermore, the models are unable to take advantage of model scale the forgetting frontier is largely independent of model size (see also Figure i9 the Appendices). Dashed vertical lines show the maximum accuracy on Task
3.4 DEPENDENCE ON PRETRAINING TIME, PRETRAINING DATASET SIZE, AND FINETUNING TIME
Having seen in the previous sections that pretraining on large datasets improves models' resilience to catastrophic forgetting, in this section we explore the dependence of this phenomenon on the duration of pretraining, the size of the pretraining dataset, and the number of steps for which we finetune on Task A.
First, we study the influence of pretraining time downstream forgetting. This is motivated by previous studies (e.g. Hernandez et al. (202I)) showing that downstream performance does not always monotonically improve with pretraining time and performance. By contrast, in the of forgetting we find (see Figure [or ViT-S results) that as the pretraining process continues, both downstream performance and the forgetting frontier improve. This was the case for all models (ViTs and ResNets) and downstream task sequences (CIFAR-10. CIFAR-100 100, etc.) we studied. We found evidence of forgetting frontiers getting worse with increased pretraining.
Pretraining
Finetuning
1.00
0.99
ACC (Tesk B1
0.98
31
0.97
0.96
29
0.94
10c4
15c4
20c4
0.94
0.98
1.00
Step
Acc (Task
Varying pretraining time. take pretrained checkpoints at 50k, 100k, 150k, 200k, 250k steps (shown in colored ticks at the left) for a VIT-S model pretrained on ImageNet21k. (right) The forgetting frontiers improve with increased pretraining time.
Figure 7:
Along similar lines, we show the influence pretraining data scale on downstream forgetting in Figure 8 pretraining model on varying fractions of ImageNet2lk. While the full dataset contains over 14 million images, we find that training on even one-sixteenth of the dataset (roughly the size of the standard ImageNet ILSVRC 2012 dataset) yields model which only loses about 3% performance in forgetting on Aplit-CIFAR-IU 10. This suggests that one does not need to pretrain models on unreasonably large datasets in order to reap benefits for continual learning.
7

---

Published as a conference paper at ICLR 2022
Finally, study the forgetting frontier of pretrained models fine-tuning (number of training steps). This is partly inspired by a recent study (Andreassen et al, 2021) of robustness in pretrained models, which found that during fine-tuning, pretrained models evolved some robustness to distribution shift which vanished at the end of fine-tuning. Figure [ shows forgetting frontiers for pretrained ViT-B and ResNet101 models, fine-tuned for number of steps between 100 and 5000. The results in the figure suggest that in our setting, increasing the fine-tuning steps to maximize performance on Task A does not appear to cause increased forgetting. On the contrary, even training for about ten times more steps than it takes to achieve saturating performance does not impact forgetting.
vary the amount of first-task
Varying pretrainingg delaset size:
Varying finetuning steps:
Varying Tinetuning steps: ResNer101
VIT-5 16
VIT-B 16
1.00
1.00
3.95
0.99
Acc [Tesk
Acc (lesk
3.90
00 500
0.96
0.98
3.85
1/2
0.97
staps 500 staps
1/4
0.80
1/8
1500 steps
5000
0.92
0.75
0.92
1.96
1.00
0.96
0.98
0.99
1.00
0.90 0.95
1.00
Acc (Tesk A)
Acc (Task A)
Acc (lask Al
Figure 8: Varying dataset size and finetuning time. (left) The forgetting frontier for vision transformer (VII-S16) finetuned two-task C1 sequence. pretrained varying fractions of the ImageNet2Ik dataset shows improvement with pretraining dataset size. (center. right): The forgetting frontiers models pretrained on ImageNet21k, with varying amounts fine-tuning Task A performance saturates around 500 steps; even training for ten times as many steps does not affect the forgetting fronticr.
3.5 REPRESENTATION OVERLAP AND FORGETTING
In this section we take steps toward understanding why large pretrained models seem to be resistant to forgetting. Following the work of Ramasesh et al. (202T). we measure the similarity between models representation of Task A and Task data.
In detail, if we denote the P model features (penultimate layer activations) on an input, x, by f (xx) E
RR, Ramasesh et al. (002)) propose measuring the similarity of ʃA := {f(x) a E Task A} e
A|xP and fn = ([x) a Task B} E R [Tsk B|xP using the trace overlap,
Tr(OABOT AB)
SAB (TT MTT (Tr(Onn0TTT)
Here OAB =fAfī is the matrix of inner amnrr-products between Task A and Task B features.
Ramasesh al. (202I) show empirically that forgetting maximal for intermediate values of S A.B. and derive this property analytically in a simplified model. In particular Ramasesh et al. (202T) propose that a minimal trace overlap (corresponding ent tasks) will lead to minimal forgetting. Inspired by this, we measure representation overlaps in models finetuned on split CIFAR-10, taking representations at the end of Task A training.
orthogonality of representations of differ-
We consider the 10x10 10 matrix, S, formed by taking the indices A and B to run over the different classes in CIFAR-10. The numerator the above expression can be thought of as an empirical (batch) estimate of the inner product between feature vectors for random draws from classes A and B; trace overlaps close to zero indicate that representations are nearly orthogonal.
Using the above definition, return to the original split-CIFAR-I0 setting of $3.1 and 3.3 For each of the models used in those experiments (see Tables D and 2). we compute the representation comparisons: (i) how does the representation overlap matrix
overlap matrix. We focus ciiined-from---e from-scratch ResNets? and (ii) how does the orthogonality of differ between pretrained and trained representations vary scale the model size, for both ResNets and Vision Transformers?
8

---

Published as a conference paper at ICLR 2022
averlap
Representation overlaps
1.0
Neshc:131 lstrach
ReoNsk101 (orck since)
[uck
truce
Crozs-chot evetto
0.8
h6150
ho020
(13
cal
Jutarie
0.5-
strpanc
sirplanc
06
0.3
Figure 9: Trace overlap CIFAR-10 class representations. (left) The cross-class representation overlap matrices (Equation] | are significantly lower for a pretrained ResNet101 than for one trained from scratch. (right) For both Vision Transformers and ResNets, the cross-class overlap decreases as the model size increases. These representation overlaps computed after training the model on the initial CIFAR-10 subtask. See Appendices for additional models and measurements.
Pretrained ys. trained-from-scraat representations: We show representative trace overlap ma- trices for both pretrained and wained-from-serane ResNet101 models in Figure Ileft and center) (see Figure 27 for additional models). Each element of this matrix corresponds to the represen- tation overlap between two batches, each drawn entirely from single class. The representation overlap between disjoint classes (off-diagonal class overlap matrix elements) is much smaller for the pretrained ResNets than those trained from scratch. Averaging cross-class overlaps over class pairs from different tasks quantifies this difference: for trained from-scratch ResNet101, the average overlap is 0.907, while for the pretrained model this value is only 0.466. For all sizes of ResNet, the pretrained models exhibit representations which have much smaller overlap than their mained-from- scratch counterparts.
Representation overlap across scales Figure riihht) shows the average cross-task overlap for both ResNets and Vision Transformers plotted against model size (number of parameters). For pre- trained Vision Transformers and ResNets, this overlap decreases with model size, while for trained- from-scratch ResNets, there is not a clear decreasing trend. Interestingly, Vision Transformers ap- pear form more orthogonal representations than ResNets do, a fact which can also be seen in the overlap matrices presented in supplementary Figure 26 and perhaps explains their improved forgetting performance.
These observations suggest that pretrained models store representations of different classes with scratch
much less overlap than trained-froe -recsscch models, and that this orthogonality increases with scale. As this mirrors the the observed trend in forgetting in these models, it is suggestive that the increased orthogonality of representations a partial explanation for the forgetting resilience seen large pretrained models.
4 DISCUSSION
The central observation have reported in this paper is that scaling up models pretrained large datasets is effective at mitigating catastrophic forgetting in the settings studied. Our experiments show that pretraining and scale can confer benefits beyond simply the achievable downstream loss or accuracy values.
An important direction for future work is understanding what underlies the systematic improvement of continual learning with model scale. The rate at which performance improves with model size in Figure [ with plwer-law exponent close to 11, is reminiscent of the variance limited behavior dis- cussed in Bahri et al.) (202T). Namely, when the number of model parameters is much larger than the number of training examples (as the case for split-CIFAR finetuning), performance predicted to scale as N This scaling behavior is notably not seen in Figure [ for language modeling- this is reminiscent of the disparate scaling behaviors observed in Sharma & Kaplan (2020) between language modeling and claseeelssifiiationn tasks. Ideally, developing an understanding of this be- havior would allow for predicting the rate of improvement forgetting with model size.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors would like thank Anselm Levskaya, Bill Mark, Anders Andreassen, and Maithra Raghu for their conversations during the completion of this work.
REFERENCES
Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs. The evolution of out-of-distribution robustness throughout fine-tuning, 2021.
Anonymous. An empirical investigation of the role of pre-training in lifelong learning. In Submitted to The Tenth International Conference Learning Representations, 2022. URL https://
under review.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws, 2021.
Ankur Bapna and Orhan Firat. Non-par Non-parametric adaptation for neural machine translation. In Pro- ceedings the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 1921-1931, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: RRKLitttss//aelaanooooooo..o
10.18653/v1/N19-11
1191
BIG-bench collaboration. Beyond the imitation game: Measuring and extrapolating the capabil- ities of language models. In preparation, 2021. URL https:// ctm google/
BIG bench/]
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint urXiv:2108.07255: 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Πla Sutskever, and Dario Amodei. Language models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- neurips. wttpp/ppppp///////..22/ ciates, Inc., 2020. URL www.oaoaaaaaaecooo.c Paper. pdf (457c0d6bfcb4555
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. ArXiv, abs/1812.00420.. 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. ISSN 1939-3539. doi: 10. 10100//ami.2221..033....
URL http: dx. doi org wtt...ooo...100....00000011
TPAM1 2021 3057446
Jia Deng. R. Socher, Li Fei-Fei, Wei Dong, Kai Li, and Li-Jia Li. Imagenet: large-scale hi- erarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recog- Mition(CVPRR) volume 00, pp. 245-255, 06 2009, doi: 10.109//CPPR.0095500680 URL
https:://eeexppie ieee http://ppw........eeooo/rrrrrrra ortp/abbtraat/oooomm////555833
10

---

Momentum contrast for Published as a conference paper at ICLR 2022
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1 1929, 2020.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for contin- ual learning, 2019.
Wikimedia Foundation. Wikimedia downloads. URL httph: http:///dpppppp wikimediaii...m. org
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020.
Sharut Gupta, Praveer Singh, Ken Chang, Liangqiong Qu, Mehak Aggarwal, Nishanth Thum- bavanam Arun, Ashwin Vaswani, Shruti Raghavan, Vibha Agarwal, Mishka Gidwani, Katha- rina Hoebel, Jay B. Patel, Charles Lu, Christopher P. Bridge, Daniel L. Rubin, and Jayashree Selpathy-Cramee Addressing catastrophic forgetting for medical domain expansion. CoRR, abs/2103. 13511, 2021. URL http::ttps//.stt.l org abs/ 2103.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. unsupervised visual representation learning. 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catastrophic forgetting in neural networks, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabska-Barwinskit Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forget- ting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3511-3526 2017. ISSN 0027-8424. doi: 10.1073/pnas. 161 1835114. URL. content/114/13/35
htt/:/wwpppnat pnas orgg
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V V16, pp. 491-507. Springer, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Sang- Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang, Overcoming catastrophic forgetting by incremental moment matching. Neural information processing sys- lems, pp. 4652 2017.
David Lopez- Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
11

---

Published as a conference paper at ICLR 2022
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150,
Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/www. ac www.wwwww.b....mm www.org/aaoo... http://rrrgaaaaooooo//1/11100000
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks a single network by iterative pruning, 2018.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. 1989.
Seyed Iman Mirzadeh, Arslan Chaudhry, Huiyi Hu, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Wide neural networks forget less catastrophically. 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008
O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference Computer Vision and Puttern Recognition, 2012.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference Computer Vision, pp. 1406-1415, 2019.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan L. Yuille. Weight standardization. CoRR, abs/1903. 2019. BRLhttp::/ http:///awxxxw.../a/ooooo///o 10520 10520
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
tations, 2021. URL pttpppaers/2111 /21/22212022002 074. html
Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semanties. In International Conference on Learning Represen-
net bot://orum:ldd-hhBB0
Sylvestre-Alvee Alvise Rebuffi, Alexander I Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533-5542, 2016.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. arXiv preprint arXiv:1810.1199 19100, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015.
Utkarsh Sharma and Jared Kaplan. neural scaling law from the dimension of the data manifold. CoRR, abs/2004. 1 10802, 2020. htt //tptttsss///i....../// wttpaaiivvvv/aaa//eea/eaaaaaevva org/ abs 2004. 10802
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinay Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In ICCV, 2017.
12

---

Published as a conference paper at ICLR 2022
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSII Birds 200. Technical Report CNS-TR-200 2000000-00 Califoria Institute of Technology, 2010.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition, 2020.
Yuxin Wu and Kaiming He. Group normalization. CoRR, abs/ 01111803..8449 2018. URL http:
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3987- 3995. JMLR, 2017.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers, 2021.
Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning; A baseline for network adaptation via additive side networks, 2020.
A EXPERIMENTAL DETAILS
A.1 VISION
A.1.1 SUPERVISED PRETRAINING
Here we detail the settings used to in supervised pretraining, resulting in the models used in the experiments of sections 3.1 3.3 3.4 and [8.2] We pretrain seven models: (ViT-B, ViT-S, ViT-xS, R26x1, R50x1, R101x1, R152x I. R200x1) on ImageNet21k k for 90 epochs. Pretraining is done using the Adam optimizer (3, = 0.9 and B3- = 0.999); for all models, we use batch size of 4096. Our learning-rate schedule includes a warmup of 10k steps to a maximum learning rate 10-3, followed by a linear decay to 10 10-5 For the Vision Transformer models, we use a weight decay penalty of 0.03 and dropout rate of 0. for ResNet models, we do not use weight decay dropout. These settings are largely identical to those found in Dosovitskiy et al. (2020).
A note ImageNet21k: We use the version of ImageNet21k available in TensorFlow Datasets 4.0.1 and apply minimal preprocessing, simply converting the labels to one-hot vectors and cropping the image be 224x224 pixels in size. During training, this crop is an Inception-style random crop, while during evaluation it is a deterministic central crop; during training we additionally augment the data by performing random horizontal flips and apply label smoothing, with values 0.9999 and 0.0001. Importantly, ImageNet2 lk is llss-imbalanced and heavy-tailed, with the uneven distribu- tion of label occurrences shown in Figure TO We do not balance the dataset.
A.1.2 VISION TRANSFORMER MODEL DETAILS
For full deseription of the Vision Transformer architecture, see the original paper by Dosovitskiy jt al. (2020). Abbreviated details are given here. We process 2D images of height H, width W, and C. channels, E xxxxxx xC by first reshaping them into 1D sequence of flattened patches-in this work, we exclusively use patch size 16. Each patch is mapped (via learned linear transformation) an embedding vector of dimension D) (for our models, D.s 256, Ds 516, and Dg 768, as in (T. As both our pretraining and finetuning is supervised, we prepend (learnable) '[class]' token embedding this sequence of patches, add a learnable ID position embedding, and pass the sequence through the Transformer encoder. The Transformer encoder alternates between multi- headed self-attention layers and MLP (plus residual connection) layers, with layernorm applied before both the attention and MLP layers.
For classification, we use classification head which acts on only the transformed '[class]' token representation. During pretraining, this classification head is an single-hidden-layer MLP, as in the original Vision Transformer work; during finetuning, the classification head is a linear layer (in the split-CIrs Aplit-CIFAR--I 10 setting, in the sequence). We initialize all classification layers with zeros.
multi-head setup with different classification layer for each task
13

---

Published as a conference paper at ICLR 2022
ImageNet21k Class Imbalance
10a
Occurences
10a
10
10c
2500
5000
7500
10000 12500 15000 17500 20000 Class
Figure 10: ImageNet21k is highly class-imbalanced. While roughly 7500 classes of the ImageNet21k k dataset feature over 1000 examples, there are over 3000 classes with less than 100 examples. As mentioned in the main text, in this work we do not class-balance ImageNet2 before pretraining.
Model
Resblocks
Params
Model
Layers
Width
Dim.
Hcads
Params
R26
[2. 2, 2, 2]
14.0M
R50
[3, 4,4,6, 6,
23.5M
VIT-xS
256
256
5.7M
R101
[3,4.23.3]
42.5M
VIT-S
516
516
19.6M
R152
[3, 36
58.1M
VIT-B
12
768
768
12
86.7M
R200
[3, 24.36.3)
62.6M
Table 2: Configurations of ResNet models
A.1.3 CIFAR-100 100 DISTRIBUTION SHIFT TASK
Here we provide more information about the CIFAR- 100 distribution-shift task, which was intro- duced in Ramasesh et (2021) and forms the basis of experiments in (oction/A.1.3] of the main text. The idea behind this task is to mimic way in which catastrophic forgetting might occur in practice, classcclasiiiication settings where the class labels remain constant but the distribution of the images changes. A key features of this type of setting is that the identity of the task is not known to the model at inference time, so task-specific components (such as multiple readout layers) are not allowed. For example, in medical setting, one might seek to classify images as displaying signs disease or not, and the system might be trained sequentially on images taken at various hospitals. While the CIFAR- 100 dataset features images divided into 100 distinct classes, these classes are grouped into 20 superclasses. The CIFAR- 100 distribution shift task sequence involves classifying image by its superclass; individual tasks in the sequence draw images from single subclass for each superclass. As a concrete example, in the experiments the main text we use the five superclasses aquatic mammals, fruity and vegetables, household electrical devices, trees, and vehicles-A" For the first task in the sequence, Task A, we uses the classes dolphin (for aquatic mammals), apple (for fruits and vegetables), lamp (for household electrical devices), maple tree (for trees), and bicycle (for vehicles-1), For the second task, Task B, we use the classes whale (for aquatic mammals), orange (for fruits and vegetables), television (for household electrical devices), willow (for trees), and motorcycle (for vehicles-2).
A.2 FINETUNING EXPERIMENTAL DETAILS
In this section we provide details on the finetuning procedures for the experiments in the main text. All finetuning was done using stochastic gradient descent with momentum (3 0.9), clipping gradients at unity; batch size is fixed at 512. In the first task, we used schedule with a 5-step linear
The CIFAR-100 dataset features vehicle superclasses, denoted vehicles-1 and vehicles-2
14

---

Published as a conference paper at ICLR 2022
ramp to maximum learning rate followed by cosine decay to over the full first-task training. In the second task, we used constant learning rates.
Section 3.1, Figure 1: Finetuning both the Vision Transformers and ResNets was done for 500 steps. For Vision Transformers, the first-task training used a warm-up from 7/ = 0.01 to 7 = 0.05. For each model we perform four different runs, with second-task (constant) learning rates 0.001, 0.003, 0.007, and 0.01. For ResNets, the task-A learning rate warm-up was from 7) = 0.002 to ? 0.01. The (constant) second-task learning rates we used were 0.008, 0.004, 0.002, 0.001, 0.0003, and 0.0001.
Section 3.1, Figure 2: In the 10-task Split-CIFAR 100 sequence (left), each task was trained for 200 steps; the initial task used a learning-rate schedule with a 5-step warmup from 1j = 0.01 to r 0.05 followed by cosine decay to 0 over the 200 steps, while all subsequent steps used constant 0.005. In the 2-task 50-class split-CIFAR 100 sequence (right), the settings are identical to those of the Vision Transformer training in Figure 1.
Section 3.1, Figure 3: For ResNets, we used a first-task warmup from vj = 0.001 to = 0.005, and trained each task for 300 steps. Constant second-task learning rates were 0.001, 0.0003, and 0.0001. For Vision Transformers, we trained each task for 200 steps, using first-task warmup from 7 0.01 to 17- = 0.05. Constant second-task learning rates were 0.01, 0.003, and 0.001.
Section 3.2, Figure 4: The pretrained models in this figure were the same those in Figure 1. For the trained-frone frrm-scratcchh models, we used linear warm-up from 7 = 0.006 to = 0.03, and trained each task for 15,000 steps. Constant second-task learning rates were 0.001, 0.0003, and 0.0001. Note that training here is much longer than for the pretrained models.
Section 3.3, Figure 5: Vision Transformer training settings are identical to those of Figure 1.
Section 3.3, Figure 6: In the left panel, Vision Transformer training settings are identical to those of Figure 1. In the right two panels, settings are also identical to those of Figure 1, with the exception of the number of finetuning steps, as indicated in the figure legend.
We finetune for 300 steps each task, with maximum learning rate of 7 = 0.0002 (we omit the linear warm-up up for these models, but keep the cosine decay). Constant second-task learning rates are 17 0.0002, r 0.0001, 7) 0.00004, and r) 0.00002.
Section 3.4, Figure 7:
A.3 LANGUAGE
Models are decoder-only transformers which were pretrained using Adafactor and learning rate 1.0, context length 1024 and batch size 256. We finetuned with batch size 8 and set a constant learning rate equal to the learning rate at the end of pretraining rescaled by the relative batch size, 8/256.
B ADDITIONAL EXPERIMENTS
B. FORGETTING IN LANGUAGE MODELS
Transfer learning from unsupervised pretraining is commonplace in Natural Language Processing. It has become standard to pretrain models on data crawled from the internet and then finetune on smaller, downstream tasks (Devlin et al. 2019: Radford et al.) | Raffel et al.) 2020, Brown et al., 2020). This bears similarities with both the unsupervised pretraining and the experiments in Figure 4 where: a single head was used to train on different distributions (subclasses). Indeed, as mentioned in the main text, part of the conceptual motivation for this work came from the observation that large language models with diverse pretraining perform well on tasks with no finetuning ostensibly retaining information for many gradient steps. Here, we include some preliminary empirical results probing catastrophic forgetting directly in large language models.
We consider decoder only transformer models pretrained on data scraped from the web, including C4 (Raffel et al. 2020). These models range in size from 57 million to 8.6 billion non-embedding parameters and were originally trained as part of the BlG-bench collaboration (2021). In Figure []] consider sequentially finetuning the models on Task A: IMDb Reviews for 1.5k steps followed by Task B: english Wikipedia for 10k steps. The longer finetuning time for Task is intended to highlight forgetting, these models are relatively robust.
15

---

Published as a conference paper at ICLR 2022
0.625
6 10
average (tang)
1001
Acr
Parameters
0.525
5.66c+ 07
Fit: Eavg N -0.04
0.62u-08
1.
3.82e-09
8.59e-09
0.40
0.45
10a
Acc (Task
Model parameters (N)
Figure 11: Forgetting sequential language-modeling. Token accuracy for decnder-only transformer mod- els show systematic improvement in joint Task A, Task B performance across scales.
We find that, as in the case of pretrained vision models, the best joint Task A, Task B performance improves with scale.
B.2 FORGETTING IN VISION MODELS WITH UNSUPERVISED PRETRAINING
In the main text, the models we studied were pretrained in a supervised manner on ImageNet2 However, part of the conceptual motivation for this work came from the properties of language models trained in an unsupervised fashion. Furthermore, recently, unsupervised pretraining meth- ods have yielded vision models which achieve good downstream performance; among these SimCLR (Chen et al. 2020), MoCo (He et al.] 22200. DINO (Caron et al.] (021), and BYOL (Grill al. 2020). Given the increased prevalence of such unsupervised pretraining methods, inves- tigate in this section whether unsupervised pretraining, combined with scale, provides resistance forgetting similarly to supervised pretraining. We take publicly-availabe ResNet models which were pretrained using SimCLR on the ImageNet ILSVRC-2012 dataset (Russakovsky et al. (0115), and finetune them on the two-task split CIFAR-10 sequence. The resulting forgetting frontiers, shown in Figure 12] show that as in supervised pretraining, SimCLR pretraining yields models which are resilient to forgetting, especially as scale increased. We leave thorough exploration of the continual-learnnng properties of other unsupervised pretraining methods for future work.
SinCLR-pretrained models
1.00
0.95
Acc [Task B)
0.90
ResNet50
ResNet101
ResNet152
0.80
0.85
0.90
0.95
1.00
Acc (lask A)
Figure 12: Forgetting frontiers on CIFIR-CIAARIO task for models pretrained in an unsupervised man- ner on ImageNetlk. ResNets pretrained using unsupervised SimCLR on ImageNetlk exhibit a similar trend to that observed in the supervised setting of forgetting robustness improving with model scale.
16

---

Published as a conference paper at ICLR 2022
B.3 FORGETTING IN TASK SEQUENCES WITH MORE THAN TWO SUBTASKS
In this section, we extend our results on vision past the two-task setting which was the primary focus of the main text. Specifically, how forgetting in this scenario varies as a function model and dataset scale. The specific split we use is shown in Table 3 As in the main text, we study the performance of Vision Transformer and ResNet models.
10-class, 10-subtask split of the CIFARI00 dataset study
Task
Task
Task C
Task D
Task E
Apple
Bowl
Chair
Dolphin
Lamp
Aquarium Fish
Boy
Chimpanzee
Elephant
Lawnmower
Baby
Bridge
Clock
Flatfish
Leopard
Bear
Bus
Cloud
Forest
Lion
Beaver
Butterfly
Cockroach
Fox
Lizard
Bed
Camel
Couch
Girl
Lobster
Bee
Can
Crab
Hamster
Man
Beetle
Castle
Crocodile
House
Maple tree
Bicycle
Caterpillar
Cup
Kangaroo
Motorcycle
Bottle
Cattle
Dinosaur
Keyboard
Mountain
Task F
Task G
Task H
Task
Task J
Mouse
Plain
Rose
Squirrel
Train
Mushroom
Plate
Sea
Streetcar
Trout
Oak tree
Poppy
Scal
Sunflower
Tulip
Orange
Porcupine
Shark
Sweet pepper
Turtle
Orchid
Possum
Shrew
Table
Wardrobe
Otter
Rabbit
Skunk
Tank
Whale
Palm tree
Raccoon
Skyscraper
Telephone
Willow tree
Pear
Ray
Snail
Television
Wolf
Pickup truck
Road
Snake
Tiger
Woman
Pine tree
Rocket
Spider
Tractor
Worm
Table 3: 10-subtask split of CIFAR100 used our experiments.
We find that:
As in the two-task setting, both the forgetting fronticrs and the best average accuracy im- prove with model scale as pawer-law in the number of model parameters.
When we scale the size of the pre-training dataset, we also observe an improvement with scale, though with some saturation around the full ImageNet21k dataset..
Model-scaling plots showing the best average accuracy as function of model parameters are show in Figure 13 As in the two-task split CIFAR 10 sequence, these scaling plots one way to show that the continual-learning performance of both architectures improves with scale. The power-law exponents are different in this 10-task setting than they are for the two-task setting, but interestingly all are close to unity.
Model scaling:
Another way to see the improvement in continual- -larming ability with model scale is to plot forget- ting frontiers, which we in Figures [n] this setting with more than two tasks, plotting a single forgetting frontier is not possible, but we plot two-task frontiers for all pairs of subtasks (with 10 subtasks, there are x 9/2 = 45 pairs). These frontiers are constructed in exactly the same way as the two-task frontiers, showing the test accuracies for each subtask throughout the entirety of the 10-task training. As the figures show, forgetting frontiers of larger models dominate the forgetting frontiers of smaller models of the same architecture. This complements the previous scaling-ple plot as a demonstration of improving continual- aentiuual-leamiing with scale, though the behavior here not as clean it was in the two-task setting.
Data scaling: A data-scaling plot showing the best average accuracy for a VIT-S model as a function of pretraining dataset size shown in Figure [4 Forgetting frontiers are shown in Figure
17

---

Published as a conference paper at ICLR 2022
Error across model scales:
Error across model scales: ResNet
10 lask CIFAR100
10 task CIFAR100
10
exponent: -D.88
4 10
exponent: -081
10
3 10
Emar
10-2
2x
40-2
30-2
2x 10-2
10-1
10
10
107
2 10 3 104 10" 6 107 Parameters
Parameters
Figure 13: Model scaling 10-subtask split-CIFAR100 (left) Scaling of best average with parameter count vision transformer models. (right) Scaling of best average error with parameter count for ResNet models.
Error across pretraining data scales:
VìT-S 10-task CIFAR100
exponent: -0.21
Fror
10-1
9x 10
8x 10-2
10-2
10-2
10
Fraction of ImageNet21k
Figure 14: Data scaling on 10-subtask split-CIFAR100. Scaling of best average error with pre-training dataset size (measured in a fraction of the full ImageNer21k dataset) for the ViT-S 16 model.
18

---

Published as a conference paper at ICLR 2022
Figure 15: Forgetting frontiers for 10-task Split CIFARI00, Resnet. Colors are the same Figure Df the main text. All images each row share same task the x-axis, ie.. the x-axis of the top row represents task A, x-axis of the second-frone from-----tth--opp row represents task B, etc. All images in each column share the same task for y-axis, ie. the y-axis of the leftmost column represents task B, the y-axis of the second-to-leftmee column represents task C, etc.
I
19

---

Published as a conference paper at ICLR 2022
Figure 16: Forgetting frontiers for 10-task Split CIFAR100, Vision Transformer. Colors the same as Figure of the main text. All images each row share the same task the x-axis, i.e. the x-axis of top row represents task A, x-axis of the aecond-rrom-the-top row represents task B, etc. All images in each column share the same task for y-axis, i.e. the y-axis of the leftmost column represents task B, the y-axis of the second-to-leftmest column represents task
20

---

Published as a conference paper at ICLR 2022
0.6
Figure 17: Forgetting frontiers for 10-task Split CIFARI00 versus data scale. These frontiers are plotted for VIT-S. 16 model. Colors are the same Figure left) the main text. All images in row share the same task for the x-axis, ie. the x-axis of the top row represents task A, x-axis of the second-from-the-top row represents task B, etc. All images in each column share the same task for y-axis, i.e. the y-axis of the lefumost column represents task B, the y-axis of the second-to-leftmot column represents task C, etc.
21

---

Published as a conference paper at ICLR 2022
B.4 FORGETTING IN DOWNSTREAM TASK SEQUENCES ON DATASETS OTHER THAN CIFARIO AND CIFARIO0
In this section, we study the continual-learning performance of I-mmoelss on down- stream task sequences beyond CIFARIO and CIFARI00. Specifically, we use the following datasets:
The Oxford-IIIT pet dataset (Parkhi et al. 2012), a 37-category dataset of images of pets, labeled by breed, with roughly 200 images per class. In our experiments, we create a two-task split of this dataset, with the first 19 breeds (alphabetically by English label) com- prising task A and the second 18 breeds comprising task B.
The Oxford Flowers 102 dataset (Nilsback & Zisserman 2008 102-category dataset consisting of flowers common in the the UK, labeled by species. In our experiments, we create a two-task split of this dataset, with the first 51 flower species (alphabetically by English species name) in task A and the second 51 flower species in task B.
The Street View House Numbers (SVHN) dataset (Netzer et al. 2011), a 10-category dataset where images consist of single-digit numbers from house numbers. We use the the two-task split in which task A consists of digits 0,1,2,3,4 and task B consists of digits 5,6,7,8,9.
The Caltech- CalDech-UCSD Birds 200 (CUB-200) dataset (Welinder al., 2010), a 200-category dataset consisting of photos of birds, labeled by species. As above, we create a two-task split by alphabetically sorting English species names and taking task A to consist of the first 100 categories, and task B to consist of the second 100 categories.
The Cars196 dataset, a 196-category dataset consisting of natural images of cars (Krause et al., 2013). with class labels specific models (including years) of cars, e.g. 2012 Tesla Model S or 2012 BMW M3 coupe. We split this into two tasks by using the standard label numbers in TensorFlow datasets, with labels 0-97 in task A and 98-195 in task B.
The Domainnet/Clipart dataset, a subset of Domainnet (Peng et al. 2019), consists of clipart drawings (unlike all of the above datasets, which consist of natural images) of 345 different categories including tennis racquets, diamonds, and umbrellas. We create a two- task split alphabetically, with 173 categories in task A and 172 categories in task B
We use the same basic setup as in the split-CIFRRIA task sequence, in which the first task is trained with a 5-step warmup followed by a cosine decay, while the second task is trained with a constant learning rate. The specific learning rates we use for each of these datasets are given in table
Forgetting frontiers and best-average-emee error scaling plots are shown in Figure 2 From these plots, we can see that the improvement in forgetting with scale is not unique to the CIFAR10 and CI- TAR100 task sequences we used in the main text, but occurs for all of these datasets as well. Scaling plots show that this improvement in average-task performance is a power-law function of the model parameters, but with exponents which vary across datasets.
The broader implication of these plots is that the improvement in forgetting performance with scale is not limited to downstream datasets which are essentially in-distribution of the pretraining dataset (e.g., CIFARI0 and CIFAR 100 are very similar to images found in ImageNer21k). As a proxy to quantify how 'in-distribution' the downstream datasets are, in Table 5 we show performances for both a ResNet and ViT model when after the ImageNet21k kk pretraining. We take the closeness of this head-only performance to the full finetuned performance as proxy for the dataset being in-distribution to Image using this proxy, the Oxford -IIT Pet dataset, Oxford Flowers dataset, CIFAR1O, CIFAR and CUB Birds 200 dataset are roughly in-distribution, while Domainnet/Clipart, Cars and SVHN are not. But, as we emphasized above, even for these "out-of-distribution" datasets, forgetting is largely mitigated by scale in the pretrained models.
train just linear classifier on top of weights frozen
For completeness, we show head-only and fillffintunnng performances for all ResNet and ViT models on the first task (task A) of our split-CIFARIL sequence in Table
22

---

Published as a conference paper at ICLR 2022
Vision Transformers
ResNets
Oxford-IIIT pet
Task A max learning rate
0.05
0.01
Task B learning rates
0.01, 0.007, 0.003
0.003, 0.001, 0.0003
Finetuning steps
500
500
Oxford Flowers 102
Task A max learning rate
0.03
0,003
Task B learning rates
0.003, 0.001,
0.0007, 0.0003, 0.0001
Finetuning steps
500
200
SVHN
Task A max learning
0.1
0.03
Task B learning rates
0.06, 0.04, 0.01, 0.007, 0.003
0.01, 0.007, 0.003, 0.0007, 0.0003
Finetuming steps
500
500
Calfech-UCSD Birds 200
Task max learning rate
0.05
0.1
Task B learning rates
0.016, 0.008, 0.004, 0.002, 0.001
0.03, 0.01, 0.007, 0.003, 0.0007
Finetuning steps
250
500
Domainnet/Clipart
Task A max learning rate
0.13
0.1
Task B learning rates
0.06, 500
0.04, 0.01, 0.007, 0.003
0.03, 0.01, 0.007, 0.003, 0.0007
Finetuning steps
500
Cars 196
Task A max learning rate
0.1
0.08
Task B learning rates
0.03, 0.01, 0.003, 0.001 1000
0,03, 0.01, 0.003, 0.001, 0.0003
Finetuning steps
1000
Table Finetuning settings used for two-task experiments on datasets described in section B.4
Oxford-IIIT pet
Flowers Head-only
SVHN
Hcad-only
Full
Full
Head-only
Full
ResNet101
0.9018
0.9202
0.97168
0.95752
0.3822
0.8992
ViT-S 16
0.8923
0.9099
0.9792
0.9842
0.3116
0.9229
CUB Birds 200
Clipart
Cars196
Head-only
Full
Head-only
Full
Head-only
Full
ResNet101 VIT-S_16
0.8702
0.8652
0.3977
0.7538
0.6212
0.7112
0.8047
0.8519
0.31
0.8056
0.4759
0.7237
CIFAR10
CIFAR100 Head-only
Head-only
Full
Full
ResNet101
0.8701
0.9674
0.5890
0.6924
VIT-S_16
0.7813
0.9592
0.5058
0.7977
Table 5: Comparison of head-only finetuning and full finetuning downstream datasets. Us- ing a pretrained (supervised on ImageNe(21k) ResNet101 and ViT-S_16, we train either (a) linear layer top of frozen representations, this as a proxy for the similarity between the downstream dataset and ImageNet21k; the closer the head-only performance is to the fully fulinet performance, the more similarity expect.
(b) the full model, downstream datasets. We take
Head Only
Full finetuning
Head Only
Full Finetuning
ViT-xS 16
0.7676
0.9527
ResNet26
0.8767
0.9474
ViT-S-16 16
0.7906
0.9857
ResNet50
0.9030
0,9805
V6T-B_16
0.9490
0.9937
ResNet101
0.9371
0.9896
ResNet152
0.9362
0,9913
ResNer200
0.9557
0,9933
Table 6: Comparison of head-only finetuning and full finetuning Task A of Split 10..R10. Accuracies achieved by training a linear classifier on top of frozen pretrained representations on task A split of CIFARI0 10 (used in the main text), freezing the features of the pretrained models after training on ImageNet2Ik.
23

---

Published as a conference paper at ICLR 2022
B.5 FORGETTING ON THE PRE-TRAINING TASK DURING FINETUNING
Given the strong performance of large-scale pretrained models on continual learning tasks we have seen in the main text, is natural to wonder to what extent performance drops on the original pre- training tasks throughout finetuning. One intuition might suggest that because the pretraining dataset so large. performance drops on the pretraining task will be minimal throughout finetuning, How- ever, in our experiments, we found that this was not the case. In Table 7 we show the performance drops on the pretraining tasks for all models, in two downstream tasks: CIFARIO and SVHN. In most cases, we find signficant performance drops, though these are much more pronounced when finetuning on the SVHN dataset than when finetuning on CIFARIO.
Just pretrained
Pretrained + Finetuned Pretrained + finetuned
on CIFAR10
SVHN
ResNer26
0.3787
0.1233
0.0000
ResNet50
0.4077
0.1433
0.0005
ResNet101
0,4317
0.1680
0.0015
ResNet152
0.4450
0.1940
0.0021
ResNer200
0.4459
0.2080
0.0027
ViT-xS.I 16
0.2717
0.1859
0.0085
ViT-S_16 0.3699
0.3222
0.0776
ViT-B_16 0.4765
0.4300
0.3022
Table 7: Forgetting on the pretraining task after finetuning. Top-1 accuracy on ImageNet2 Ik after pretraining, and after downstream finetuning. Unlike pretrained models performing continual learning on downstream tasks, it appears that there are significant performance drops on the original ImageNet21K 1k task.
C SUPPORTING RESULTS
In the following subsections, we present results which are directly supporting experiments or figures in the main text, but not experiments.
C.1 COMPARING PRETRAINED AND TRAINED-FROM SOI----RRO---SACCCEEOOOOO- SCRATCH RESNETS
In the main text, Figure we showed comparison between the forgetting frontiers (on the split CIFARIO task sequence) of where we handicapped the pretrained ResNet models (by training them for fewer steps) so that they matched task-A performance of the trained- from-scratch ResNets. This was done to show that even when task-A performance is the same, pretrained ResNets forget far less than their trained-frome scratch counterparts. For space, in the main text we only showed three ResNet models: ResNet26, ResNet101, and ResNet200. Here, show the forgetting frontiers for all ResNet models (Fig- ure 18). and we also show (Figure [9) the data plotted slightly differently-with all model sizes plotted the same graph-to make apparent that while the forgetting frontiers of the pretrained ResNet models clearly improve with model size, the forgetting frontiers of trained conned-frrm--se from-scratch ResNet models do not. Figure [99 also includes scaling plot the best average error vs. model parameters, supporting the conclusion that in trained-frome from-scratch ResNets, scaling does not appear benefit the continual learning performance.
ResNet models and pretrained ResNet models,
24

---

Published as a conference paper at ICLR 2022
ResNet26
ResNet50
ResNet101
1.0
1.0
1.0
0.9
0.9
0.9
0.8
0.8
0.8
0.74
0.7
0.7
0.9
1.0
Acc (lask A)
Acc (lask A)
Acc (lask A)
ResNet152
ResNet200
1.0
1.0
0.9
0.9
0.8
0.8
0.7
0.7
0.7
D.8
0.9
0.6
0.7
Acc (Task A)
Acc (Task A)
Figure 18: Forgetting frontier for ResNets trained from scratch, vs. pretrained ResNets (on Ima- geNet21kk. The pretrained ResNets were handicapped during fine-tuning, in order to match the Task A perfor- mance of the from nom-scratch model. This figure is identical to Figure 6 in the main text, with the addition of the ResNet50 and 5csNet152 models.
Trained fromfroommatt scrmshc
Pretrsined
Error scaling
Best averoge arror (aa))
0.9
Roc (Teck B)
0.8
go
ResNet20
ResNat101
ResNac152
Pretraind: a 1.14
ResMet200
From scratch: 0.04
0.6
-2
0.6
0.7
0.8
0.9
1.0
0.6
0.7
0.8
0.9
1.0
107
10a
Parometers
Figure 19: Forgetting frontier for ResNets trained from scratch, vs. pretrained ResNets (on Ima- geNet21k). The pretrained ResNets were handicapped during fine-tuning, in order to match the Task A perfor- mance of the from-scratch model. The data plotted here the here, the intention is to make clear apparent benefit with scale, while the pretrained models do (right) Scaling of the best average (task A task B) accuracy shows that only in pretrained models does the performance improve with scale.
that plotted in Figure the main text; performance the coanend-fro--scrttee models do not exhibit any
C.2 LEARNING CURVES FOR SPLIT-CIFARII VISION EXPERIMENTS
In Figure I of the main text, we plot forgetting frontiers of pretrained Vision Transformers and ResNets on the split-CIFAR1O task sequence. Corresponding to these frontiers are the learning curves (accuracies step) shown in Figures and and
25

---

Published as a conference paper at ICLR 2022
ResNet26
1.0
Accuracy
0.9
200
400
600
800
Steps
ResNet50
1.04
Accuracy
200
400
600
800
Steps
ResNet101
1.0
Accuracy
0.9
0.8
0.7
400
600
Steps
ResNet152
1.0
Accuracy
200
400
600
800
Steps
ResNet200
Task learning rate
Accuracy
0.0001
0.001
0.8
0,002
0.004
0.7
0.0003
200
400
600
800
Steps
Figure 20: ResNet Learning curves Split CIFARI0 task. Learning curves for the ResNet models of Figure The solid/dashed lines are Task A/B accuracy.
26

---

Published as a conference paper at ICLR 2022
VIT-xS_16
1.00
0.98
0.96
Accuracy
0.94
0.92
0.90
0.88
0.86
200
400
600
800
Steps
VIT-S_16 16
1.00
0.98
0.96
Accuracy
0.94
0.92
0.90
0.88
0.86
200
400
600
800
Steps
ViT-B_16
1.00
0.98
0.96
Accuracy
0.94
0.92
Task learning rate
0.90
0.01
0.003
0.88
0.86
200
400
600
800
Steps
Figure 21: VIT Learning curves on Split CIFARI0 task. Learning curves for the VIT models of Figure | The solid/dashed lines are Task A/B accuracy.
27

---

Published as a conference paper at ICLR 2022
C.3 LEARNING CURVES FOR THE LANGUAGE EXPERIMENT
For completeness, Figure 22 displays the token accuracy on task A as a function of the number of steps, corresponding to the frontiers shown in Figure 11 of the main text.
0.46
0.44
Acc (Task A)
0.42
0.40
Parameters
5.66e+07
2.62e+08
0.38
1.07e+09
3.62c+09
8.59e+09
0.36
2000
4000
6000
8000
10000
Finetuning steps
Figure 22: Learning curves in sequential language-modeling, Token accuracy as function of steps for the different models. The dashed line shows the point at which training switches from Task A to Task B.
C.4 FORGETTING FRONTIERS COLORED BY LEARNING RATE
In the forgetting frontiers of Figure of the main text, the frontiers are colored by model size; each model, however, trained with several different learning rates during the second task. In Figures 23 and 24 we plot the same frontiers, coloring them by learning rate on the left, and by proper time (learning rate multiplied by step number) the right.
We also plot scaling curve Figure 25 of the average accuracy at a fixed learning rate and step (17 0.0001 and 200), show that the power-law scaling holds even if one looks fixed hyperparameter choice.
28

---

Published as a conference paper at ICLR 2022
ResNet26
0.95
0.90
0.75
0.70
0.65
0.70
0.75
0.80
0.85
0.50
0.95
0.70
0.75
0.80
0.85
0.90
0.95
ResNet50
0.95
0.85
0.75
0.70
0.75
0.80
0.85
0.50
1.00
0.70
0.75
0.80
0.90
0.95
1.00
ResNet101
0.95
0.85
0.75
1.00
ResNet152
0.95
0.85
0.75
ResNet200
0.95
0.85
Learning rate
0.0031
3.001
0.001
0.002
0.002
0.0003
0.75
0.875
0.925
0.930 0.975 1.000
0.800 0.900
0.0
6.0
stepfir
Figure 23: Step and learning rate along Split CIFARIO forgetting frontier for ResNets. We plot the forgetting frontier of [i The left column is colored by learning rate with transparency set by step (darker) later during task training. The right column is colored by step times learning rate.
29

---

Published as a conference paper at ICLR 2022
VIT-x5
1.00
0.98
0.94
0.92
0.88
0.86
0.86
0.88
0.90
0.92
0.94
0.96
0.98
1.00
0.86
0.88
0.50
0.92
0.94
0.96
0.38
1.00
VIT-5_16
1.01
1.00
0.99
0.98
0.97
0.96
0.95
0.96
0.98
0.99
0.95
0.96
1.00
VIT-B_16
1.000
0.990
0.985
0.980
Learning rate
0.975
0.970
0.988
0.994 0.998 1.000
0.986
0.988 0.992
0.984 0.996 0.998 1.000
10.0
stepttr
Figure 24: Step and learning rate along Split CIFARIO forgetting frontier for VIT models. We plot forgetting frontier of [igure[] The left column is colored by learning rate with transparency set by step (darker) is later during task 2 training. The right column is colored by step times learning rate.
30

---

Published as a conference paper at ICLR 2022
1r= 0r000 step=200
10-1
10-2
ResNet a= -1.29
107
10"
Parameters
Figure 25: Average error scaling fixed learning and step. We plot the average Task A Task B for the setup Figure [ for fixed learning rate of 0.001 and 200 Task training steps for both ViTs and ResNets.
31

---

Published as a conference paper at ICLR 2022
C.5 REPRESENTATION OVERLAP MATRICES FOR ALL RESNET AND VIT MODELS
In the main text, we described the computation of the classwise representation overlap matrices for models the end of Task A training, showing example of such matrix for both pretrained and mained-from-scrate ResNet101 model. In Figure we show representation overlap matrices for all the ResNet models, which bears out our statement the main text that pretrained ResNet representations were much closer to orthogonal than their foined-from-scratchh counterparts. In Figure 26] we show representation overlap matrices for the (pretrained) Vision Transformer models. As with the pretrained ResNet models, the cross-class representation overlaps become closer to orthogonal as we increase the model size.
In these figures, also plot the representation overlaps for pre-trained models before being trained on task A (which was not plotted the main text). From these plots one can see that while pre- training ImageNet2Ik Ik introduces some level of orthogonality in the model representations of CIFARIO images, this orthogonality becomes much more pronounced after training the model on task A of the CIFARIO split.
ImegeNe121k and Finetuned on LIFAR13 task VIT-x5
an ImageNei221:
finetuning: VIT-xS
Figure 26: Classwise representation matrices for Vision Transformer models taken (left) after end of training on first task (Task A) a two-task split CIFAR-I0 sequence, and (right) after pretraining ImageNet2lk. Classes ordered alphabetically, left to right and down to up, i.. the order is airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.
32

---

Published as a conference paper at ICLR 2022
Trained CIFAK10
Pretrained and Finetuned CII-RR0 resnet26
imaganet2 No finetuning reanet26
resmet26
reshet50
Teshict0
resnotso
resnett.1
resneti01
resnett.2
resnet1.2
resnet1
resret200
resnet201
respet200
Figure 27: Classwise representation matrices for ResNet models. (Left) Models are trained from random initialization the first task (task A) of two-task split CIFAR-10 sequence. (Middle) Models are pretrained on ImageNet21k and then finetuned on task A of the two-task split CIFAR-10 10 sequence. (Right) Models are pretrained ImageNer21k, but finetuning is done. Classes ordered alphabetically, left to right and down to up, i.e. the order is airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck.
33


Reviews:
[{"review_id": "crdwZoeN-KZ", "reviewer": "Reviewer_r2yB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n**Strengths**:  \n(1) The paper is well-written and easy to follow.   \n(2) The main motivation of the paper (i.e., role of scale and pre-training) is significant.\n\n**Weaknesses**:\n\n(1)  I think the major problem with the arguments of the paper is that it mixes the over-parametrization and pre-training together. \nFor instance, the paper mentions: “catastrophic forgetting is mitigated to a large extent by scale: that is, larger models suffer less from forgetting”, which is not a true argument in my opinion. \nThe high-performance is due to the “pre-training” part and not the over-parametrization. In fact, [2] shows that over-parametrization by increasing depth is not very helpful for continual learning.\nThis is also visible in Fig 4. where ResNet 26 and ResNet 101 perform almost equally poorly, which suggests over-parametrization is not very helpful.\n\n(2) If we agree on the fact that the benefit comes mostly from pre-training, then the other problem is that this has also been studied before [1] for both vision and NLP benchmarks.\nSo I believe the fact that pre-training is helpful is not a new phenomenon in the CL literature.\n\n(3) The final major problem is with the experimental setup. \nFirst, pre-training on the ImageNet-21K benchmark and then continual learning on a similar dataset makes the evaluation very tricky.\nIf we look at the classes in ImageNet-21K [3], we can see that it shares many classes to CIFAR and ImageNet-1K, and the pre-trained model had already seen data from different distributions before.\nSo the model already knows the distribution of vehicles, animals, foods, etc., and then the forgetting is not well-defined here.\nI believe a correct pre-training setup should be what people are doing in the NLP literature and what authors have done in Section 3.4. \nWhen pre-training, the source task should be different than the fine-tune task. For instance, in self-supervised prompting, the model learns the representations by figuring the masked words and learning a language model. \nThen, in that setup, one can argue that the knowledge of the model is related to the language model, which can be used in different downstream tasks with different data. \nHowever, here, the model is pre-trained on the same task (image classification) on a dataset that potentially has the labels of the fine-tune task.  \nIn general, while I have no doubt regarding the benefits of pre-training, I believe the experimental setup should be more carefully designed.\n\n\n[1] Mehta et al. “An Empirical Investigation of the Role of Pre-training in Lifelong Learning”, ICML Theory of CL Workshop (2021).  \n[2] Mirzadeh et al. “Wide Neural Networks Forget Less Catastrophically.” ArXiv abs/2110.11526 (2021).  \n[3] “WordNet names”, [https://storage.googleapis.com/bit_models/imagenet21k_wordnet_lemmas.txt](https://storage.googleapis.com/bit_models/imagenet21k_wordnet_lemmas.txt)\n\n---------\n*** **UPDATE AFTER DISCUSSION PERIOD** ***   \nAfter reading other reviews, authors' responses, and the updated version of the paper, I believe major concerns have been addressed. Thus, I have increased my initial score.", "summary_of_the_paper": "The paper studies the impact of large-scale pre-training in continual learning. \nMoreover, The paper shows that pre-trained models suffer less from forgetting in vision and NLP tasks with various experiments.\n", "summary_of_the_review": "I believe the paper lacks novelty and some claims are not well-supported. Moreover, the experimental design has flaws as mentioned in the main review. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "cqP4agIe66K", "reviewer": "Reviewer_ZtJq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n-The authors open an interesting and timely discussion about the size of models and pre-training and how they affect catastrophic forgetting and lifelong/continual learning\n\n-The use of the forgetting frontier is an interesting way to visualize two task continual learning settings\n\n-Several model classes are used and the authors provide insights about the importance of pre-training\n\n\nWeakness/Comments\n\n-The choice of pre-trained model and the set of tasks is odd. CIFAR-10 is an extremely similar task to imagenet and using imagenet pre-trained model features directly should do well. The authors should include the obvious baseline of just training the head on fixed features on their graphs. It is already known that using better pre-trained models particularly for such tasks will improve performance on the down stream task. Similarly Pretraining imagenet with SIMCLR is known to give strong features for object recognition and is more over done on essentially an input data distribution that more or less includes CIFAR-10, despite different training objective.  So the proposed setup for very powerful imagenet models seems like it will mostly be fighiting changing the model weights in the first task.\n\n-The similarity of the pre-trained task and the task sequence is not properly decoupled in this work. Should we conclude that pre-training alone is the key or pre-training on very similar data distributions to the sequence that follows? I would have liked to see at least one experiment with a legitimately diverse input distribution from the pretraining distribution (e.g. different object categories from previously seen or non-natural images).\n\n-Many details of the  forgetting frontier points don't seem to be provided, for example the learning rates used or other optimization hyperparameters don’t even seem to be provided at all in the paper. Furthermore the learning rates and finetuning steps are mixed in the forgetting frontier plots making it challenging to fully interpret the results visualized. Do the larger models typically have smaller learning rates as the points with better taks 1 and taks 2 trade of? It also will make it challenging to reproduce the results. I think this visualizations should more clearly control for either finetuning step or learning rate. \n\n-Standard evaluation protocols for continual learning are not considered (e.g. forgetting or reporting an overall accuracy with a clearly stated hyperparameter search criteria). The use of the forgetting frontier is interesting but its one view and may not give a full picture. We know a larger pre-trained model on imagenet will have better performance on downstream tasks (e.g. Kornblith et al Do Better Imagenet Models Transfer Better?) so we know the first task will generally do better in this setting as the model has better imagenet performance, how do we clearly compare now the forgetting in this case where we know the first task will do better? In the experiments comparing to from scratch performance the authors handicap the models for the task A to start at the same point, can similary idea be used in other experiments somehow?\n\n-The use of pre-training seems to suggest we should evaluate forgetting on the imagenet-21k as well to get more insight into the behavior. Have the authors consider this? \n\n-The observation that the model size doesn't clearly improve forgetting when training from scratch is interesting. What if the first task is big (e.g. 1/4 of the imagenet-21k data), if followed with a 2 and 3 task how is this type of sequence really different from the pre-training formulation here?  A more natural setup seems to be to split imagenet-21k into a series of tasks and assume the first task is the “pre-trained” model. This would also end up with a situation where the CL sequence distribution does not clearly follow the pre-trained data distribution by separating them via object categories. \n", "summary_of_the_paper": "The authors study using ever larger pretrained models and suggest this can solve the problem of catastrophic forgetting without the use of any particular continual learning method. They support this claim with experiments using pre-trained models on imagenet applied to CIFAR-10 dataset as well as a Natural Language task with a large pre-trained model then applied to IMDB and english Wiki. ", "summary_of_the_review": "Studying the effects of model size and pre-training in continual learning seems pertinent. However, the claims at the moment in the paper seem to be overly general given the evaluation. The paper has experimental setups that make it hard to decouple the effects of various factors including distribution similarity of pre-training to CL sequence data, optimization hyperparemeters, finetuning steps, performance of first task. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EKtHMur3cX", "reviewer": "Reviewer_Dwjt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:** I found the paper to be well motivated. Recent works on scaling laws for transfer learning primarily focus on understanding the effects of pretrained representations when learning a single new task, but this paper extends the theme of model scaling to understand its benefits when learning multiple tasks. The experiments are extensive: authors evaluate several architectures, different pretraining paradigms (supervised and self-supervised), the impact of pretraining dataset size, amount of fine-tuning, and also show comparisons with training from scratch. I liked the representational similarity figures, which indicated representations across classes become more orthogonal with model scaling, which is likely to prevent catastrophic interference when learning new tasks.\n\n**Weaknesses:** My main apprehensions focus on the paper’s experimental setup and some missing details in the figures. I would be happy to raise my score if these two concerns are addressed.\n1. **Overlap of data distributions in pretraining vs. new tasks:** The authors pretrain several models on ImageNet-21K and evaluate on split CIFAR-10/-100 tasks (primarily). However, the two datasets have a significant class overlap and I am uncertain if one can regard training on a subset of CIFAR classes as a new task. If the pretraining data and subsequent tasks’ data have high overlap, we are unlikely to see catastrophic interference when new tasks are learned. This is also highlighted by Figure 8 of the paper, which shows that the pre-trained models have orthogonal representations across CIFAR-10 classes even *without* training the model on new tasks (please correct me if I misunderstood and the models have been fine-tuned on the new tasks). While I understand CIFAR datasets are standardly used benchmarks in the continual learning literature, past work in the field has either used randomly initialized networks or models pretrained on datasets that do not have any overlap with the downstream tasks’ data. Papers that have used ImageNet pretraining [A, B] in the past generally used datasets such as CUB Birds, Oxford Flowers, Places 365, Stanford Cars, etc. for downstream evaluation of catastrophic forgetting. *I believe it is important to include experiments on such non-/minimally- overlapping datasets to ensure subsequent tasks can be considered “new tasks”, thus ensuring better validity of the conclusions drawn from this paper.* Even for the current experiments, I think it is important that the overlap of pretraining data vs. CIFAR data is highlighted. \n\n[A] Memory Aware Synapses: Learning what (not) to forget (https://arxiv.org/abs/1711.09601)\n\n[B] PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (https://openaccess.thecvf.com/content_cvpr_2018/papers/Mallya_PackNet_Adding_Multiple_CVPR_2018_paper.pdf)\n\n2. **Training time, learning rates, and figures:** Most figures currently show accuracies of two sequentially learned tasks (e.g., 5-class CIFAR-10 splits, called Tasks A and B). Each datapoint in these figures represents a model trained with either a different learning rate or fine-tuned for a different amount of time--however, all points corresponding to the same model architecture have the same color and marker width. This makes it hard to ascertain whether fine-tuning longer on task A reduces the model’s ability for further transfer to task B, or if there are any improvements/degradation in backward or forward transfer between the two tasks as model size is scaled. Similar comments hold in regards to the effects of different learning rate scales: a large learning rate can often diminish the benefits of pretraining and it is valuable to highlight which points in the figure  correspond to which learning rates. Importantly, I note that I could not find exactly what set of learning rates was used for fine-tuning. *Overall, I think properly delineating the role of these factors is important because the paper’s main message can be misconstrued as saying scaling solves catastrophic forgetting despite how long the model is trained on a set of tasks or what hyperparameters are used.* I note that addressing this comment should primarily involve only replotting the figures by using different shades of a given color, to denote different time steps or learning rates while fine-tuning on Task B. \n\n**Major Questions/Comments**\n\n1. Was the pretraining data pre-processed? ImageNet-21K is heavy tailed and imbalanced, so papers often create a balanced version of the dataset by removing abundant data or classes with too few samples. If pre-processing was performed, this is an important detail that should be explicitly mentioned in the paper. If pre-processing was not performed, I would like to collect the authors’ thoughts on whether the imbalanced nature of this dataset affected their results. For example, would you expect better or similar results with a balanced version? I also highlight that these details are important and should at least be mentioned in the appendix.\n\n2. Change in backbone representation vs. classifier output from final linear layer: My understanding of Figure 8 is that it shows cross class representational similarity after pre-training, before any new tasks are learned. However, when the new tasks are learned, how much does the representation from the backbone change? Is it possible that with effective pre-training, the model representations remain more or less constant and only the last linear layer's output is changing? If this happens, then one possible confounding conclusion is that the benefits of scale are primarily enabling better pre-trained representations. One way of testing this may be to freeze the model and train only a linear layer on top of the model representations. This will be similar to multi-head training, but here the representations will not be allowed to change by preventing fine-tuning. \n\n3. Is self-supervised pre-training performing relatively worse than supervised pre-training? This would be in contrast with self-supervised learning literature’s results, which generally find self-supervised pretraining to be more effective for downstream transfer. Can the authors comment more on this?\n\n**Minor comments**\n1. In related work, I think methods that utilize network pruning or model compression techniques to alleviate catastrophic forgetting/interference should be cited (e.g., see [A, B, C] below). Such methods directly utilize the overparameterization of a model and are perhaps most related to the theme studied in this work. Also relevant are works focused on tuning side-networks, where a pretrained representation remains fixed but small side models are used to improve the representation quality for a given dataset.\n\n[A] PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning (https://openaccess.thecvf.com/content_cvpr_2018/papers/Mallya_PackNet_Adding_Multiple_CVPR_2018_paper.pdf)\n\n[B] Supermasks in superposition (https://arxiv.org/abs/2006.14769)\n\n[C] Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks (https://arxiv.org/abs/1912.13503)\n\n2. On page 6 and 9, the names of recent self-supervised learning methods such as DINO and BYOL are mentioned, but the corresponding papers are not cited.\n\n3. Since the paper’s analysis primarily focuses on pretrained networks and since the authors also comment “pretraining is required to see the benefits of scale”, I think explicitly including the term “pretraining” in the title will be useful. ", "summary_of_the_paper": "Following recent works studying the effect of model scaling on tasks such as transfer learning, this paper asks whether scaling up the size of a pretrained network also helps alleviate or minimize catastrophic forgetting. The authors use split CIFAR-10/-100 datasets (early experiments on NLP tasks are also included) and demonstrate the accuracy lost by a pretrained network when learning a set of tasks sequentially reduces with increase in model size. This result is shown to hold for both ResNets and Vision transformers. The topic is timely and the paper is well written. My main apprehensions relate to the experimental setup--due to the use of only CIFAR datasets for evaluation, which have significant class overlap with ImageNet, it may not be justified to think of learning a subset of CIFAR classes as a new task. Overall, I believe some claims are not completely validated yet, but this can be addressed by using a more appropriate experimental setting.", "summary_of_the_review": "I found the paper to be interesting and timely, but recommend rejection for now due to a need for better experimental setup and more details in the figures. I would be happy to raise my score if the authors justifiably address these apprehensions. \n\n**Post Rebuttal**: The authors provided detailed response to the raised concerns (though primarily in the appendix) and sufficiently backed up their proposed hypotheses. I am hence increasing my score to recommend acceptance for the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2uLdXrJC7Bj", "reviewer": "Reviewer_Tmvz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Positives:**\n- Overall, I enjoyed the direction of this paper. I think it was asking interesting questions, and the experiments were quite thorough with respect to the scale of the dataset and model\n- The results show quite a clear picture in terms of the effects of scale on forgetting in the settings the paper explores.\n**Negatives:**\n- I would like to see these results extended past the two task setting. It would be interesting to see whether these results would hold for many tasks in sequence. Many datasets used in the current literature often have anywhere from 5-20 tasks, and it's unclear how well the results would hold for those settings.\n- The paper shows some results on 10 tasks Split CIFAR-100, but there doesn't seem anything to compare the results to. Only the curves for a (single) pretrained ViT-B model are shown, and it's unclear what the takeaway of that experiment is with respect to the effects of model size, dataset size, or pretraining vs random initialization\n**Questions:**\n- Is the data used in pretraining controlled for? Specifically, there is quite an overlap in classes between the ImageNet dataset and the CIFAR datasets used. Is the effect of that accounted for?\n**Minor:**\n- The explanation of the trace overlap metric can be improved.", "summary_of_the_paper": "The paper explores the effect of scale on catstrophic. They conduct several different experiments (for both ResNets and Vision Transformers) showing:\n- the size of the model for pretrained models reduces forgetting \n- (Longer) pretraining reduces forgetting\n- More pretraining training data size reduces forgetting\n- A metric used to measure overlap between model representations showed that pretraining results in more orthoganal representations between classes. \n- The length of finetuning isn't as impactful on pretrained models as randomly initialized models\nThey perform the majority of their experiments on a two task, task incremental Split CIFAR-10 dataset. Some experiments were done on 2-task class incremental and 10-task task incremental Split CIFAR-100. They also conduct a brief set of experiments showing that scaling model size  reduces forgetting for language as well.", "summary_of_the_review": "This paper does a lot of good things and is fairly comprehensive in terms of trying different model sizes and dataset sizes. My biggest problem with this paper is that it (mostly) only explores the effects of scale on a limited 2 task setting. These results may not hold for longer task sequences that are often used in continual learning, and thus I recommend a borderline reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors explore the forgetting behavior of large-scale pre-trained models in continual learning (CL). The authors find that forgetting is mitigated by scale (or models and of pre-training datasets). The authors also make preliminary observations to try to explain why this happens. This manuscript is somewhat in line with recent results around scaling laws and in a sense, this paper extends the study of scaling laws to the CL setup with a focus on catastrophic forgetting, traditionally the main desiderata of CL.

The initial reviewer assessments indicated that this paper was likely below the acceptance threshold of the conference. The main perceived limitations were: 
+ CL experiments on sequences of two tasks are limiting (in CL it's common to look at sequences of at least 10 tasks)
+ The effect of the pre-training data on the results was not properly assessed. In particular, it is likely that the pre-training data and downstream/test data were very similar.
+ Other aspects such as missing hyperparameter values and empirical settings were also raised.

The authors really came through and obtained lots of additional empirical results. In particular, the authors show that their results mostly hold on longer task sequences and even if the downstream task was very different from the pre-training tasks. Further, the authors provided precise answers to all the reviewer comments and also ran a few more studies to answer more specific reviewer questions (including a few to answer some excellent suggestions from reviewer ZtJq). 

Overall, this is a good contribution and I imagine one that could have a significant impact in the field and give rise to follow-up work. Congratulations! 

In preparing the final version of the manuscript, I would strongly suggest that the authors incorporate all results discussed in their replies in their paper. Further, and as was suggested by reviewer Dwjt, I think it would be very useful to add the study of the longer task sequences and of the different downstream tasks to the main paper and not the appendix as is done currently.
Decision: Accept (Poster)