Effect of scale on catastrophic forgetting in neural networks

Continual learning is a current challenge in designing machine learning systems : when trained on multiple tasks in sequence , models tend to suffer performance losses on earlier tasks—this is known as catastrophic forgetting ( McCloskey & Cohen , 1989 ) . Catastrophic forgetting occurs in many settings , from supervised learning for classification ( Zenke et al. , 2017 ) to reinforcement learning for games ( Kirkpatrick et al. , 2017 ) and downstream applications such as translation ( Bapna & Firat , 2019 ) or medical diagnoses ( Gupta et al. , 2021 ) . While many methods have been proposed to help mitigate forgetting , none of these have fully solved the problem ; see Kemker et al . ( 2017 ) for a review . Increasing model and dataset size have been key to the success of deep learning ( Kolesnikov et al. , 2020 ; BIG-bench collaboration , 2021 ; Radford et al. , 2019 ; Zhai et al. , 2021 ) . A recent trend has been the use of increasingly large models pretrained on diverse datasets , which are subsequently fine-tuned or evaluated without fine-tuning on downstream tasks of interest ; see Bommasani et al . ( 2021 ) and references therein . A particularly striking realization of this has been large pretrained language models , which show consistent improvement across scales ( Kaplan et al. , 2020 ) and can perform well on diverse tasks with no finetuning ( Brown et al. , 2020 ) . From a continual learning perspective , this behavior is especially dramatic as , without finetuning , any training signal for these tasks appears mixed into a drastically larger pretraining dataset , and thus more than likely appeared many gradient steps before evaluation . This observation in large part inspired our work . Here , we empirically study continual learning in large pretrained models . Our results suggest that pretrained models are significantly more resistant to catastrophic forgetting than their randomlyinitialized counterparts , and that this resistance improves with the scale of both the model and pretraining data . Most of our experiments concern vision—specifically image classification ; but we also conduct a preliminary investigation of language models . Our key observations are as follows : • Resistance to catastrophic forgetting improves with model and pretraining dataset scale ( Figure 1 ) . • Large , pretrained image-classification models are significantly more resistant to forgetting than models trained from random initialization ( Figure 5 ) . • Both supervised and unsupervised , SimCLR ( Chen et al. , 2020 ) , pretraining improve forgetting ( Supplement , Figure 11 ) . • We perform initial experiments aimed at exploring why scale and pretraining reduce catastrophic forgetting , showing that distinct class representations in pretrained models are more orthogonal than in trained-from-scratch models ( Figure 8 ) . • We observe similar phenomena in language models ( Figure 9 ) . 1 RELATED WORK . Catastrophic forgetting Catastrophic forgetting was identified as a problem for neural networks as early as the late 1980s , as it occurred in the single-hidden-layer networks that were studied at the time ( see Delange et al . ( 2021 ) for a review ) . Within the catastrophic forgetting literature , a number of potential solutions to the problem have been devised ; some involve replaying previous-task examples during training on later tasks ( Rebuffi et al. , 2016 ; Lopez-Paz & Ranzato , 2017 ; Chaudhry et al. , 2018 ; Shin et al. , 2017 ; Riemer et al. , 2018 ) , while others like elastic weight consolidation ( Kirkpatrick et al. , 2017 ) limit the extent to which parameters can change once learned ( Zenke et al. , 2017 ; Lee et al. , 2017 ; Farajtabar et al. , 2019 ) . Perhaps most similar to our work are proposed mitigation methods which exploit the overparameterization of modern neural networks to perform continual learning of multiple tasks , e.g . by using pruning in Mallya & Lazebnik ( 2018 ) or task-specific subnetworks in Wortsman et al . ( 2020 ) and recent works studying the impact of pretraining Anonymous ( 2022 ) and overparameterization via network width Mirzadeh et al . ( 2021 ) on forgetting . Large-scale models The observation—initially in language models ( Kaplan et al. , 2020 ) —that model performance scales predictably with model and dataset size spurred an increased interest in training and deploying increasingly large pretrained models . Large-scale models emerged in natural-language processing with models such as GPT ( Radford et al. , 2019 ) and BERT ( Devlin et al. , 2019 ) , which are pretrained in an unsupervised manner and thus able to make use of the large corpora of natural text available on the internet . More recently , image-classification models have also benefited from pretraining on large datasets ( Kolesnikov et al. , 2020 ) , both supervised and unsupervised ; a key example is the Vision Transformer architecture of Dosovitskiy et al . ( 2020 ) , which—despite not using the inductive bias provided by convolutions—attains downstream performance comparable to convolutional networks after pretraining on ImageNet-21k ( Deng et al. , 2009 ) or JFT-300M ( Sun et al. , 2017 ) . On many current NLP and computer vision benchmarks , large pretrained models achieve top performance . All of the experiments in our work follow the pretrain-then-finetune method ; recently , another paradigm , side-tuning , was proposed by Zhang et al . ( 2020 ) , which avoids forgetting altogether by leaving the pretrained model fixed and employing auxiliary networks for downstream tasks . 2 EXPERIMENTAL SETUP . Tasks : Most of the experiments in this paper are conducted in the standard , task split setting ( Kirkpatrick et al. , 2017 ; Zenke et al. , 2017 ) . We consider split CIFAR-10 task sequences – the ten class dataset is split and sequentially trained on two tasks of 5 classes each . We also consider sequential 10 and 50 class splits of CIFAR-100 . We also consider the more recent input-distribution-shift CIFAR-100 setup ( Ramasesh et al. , 2021 ) . In this case each task has the same target types made up a fixed subset of the 20 CIFAR-100 superclasses ( e.g . aquatic mammals , fish , ... ) . The sequence of tasks consists of modifying which subclasses are sampled from . For example Task A may have aquatic mammals drawn from beaver and dolphin , while Task B may have otter and whale . This setup allows one to probe forgetting without changing the model head , in a setting where task identity is not known to the model . For our language model experiments , we consider next token prediction generative tasks and use the IMDb Reviews ( Maas et al. , 2011 ) and english Wikipedia ( Foundation ) datasets . Vision models : In most of our experiments we use the Vision Transformer ( ViT ) and ResNet architectures . For the Vision Transformers , we use the ViT-S and ViT-B configurations from Dosovitskiy et al . ( 2020 ) and a smaller configuration we call ViT-xS – ranging from 5.7-86.7 million parameters . For ResNets , we use five models : ResNet26 , ResNet50 , ResNet101 , ResNet152 , and ResNet200 – ranging from 14.0-62.6 million parameters ; as in Kolesnikov et al . ( 2020 ) , these models use group normalization ( Wu & He , 2018 ) and weight standarization ( Qiao et al. , 2019 ) in place of batch normalization . Details can be found in Tables 1 and 2 in Appendix A.1.2 . Pretraining : Unless otherwise specified , we pretrain our image models in a supervised manner on the ImageNet21k dataset , which contains approximately 14 million images in about 26000 distinct categories using the Adam optimizer ( Kingma & Ba , 2017 ) . Further details are in the Appendix . Finetuning : When fine-tuning on multiple tasks in sequence , we use the SGD optimizer with momentum β = 0.9 . The first task training employs a cosine decay schedule with a linear warmup ; later tasks in the sequence are trained with fixed learning rates . For the split-CIFAR-10 task sequences , we use a multi-head setup in which each head is initialized to zeros . 3 RESULTS . 3.1 FORGETTING IMPROVES WITH MODEL SCALE . Our main finding is that for both pretrained Vision Transformers and pretrained ResNets , catastrophic forgetting is mitigated to a large extent by scale : that is , larger models suffer less from forgetting . This is shown in Figure 1 for the two-task split CIFAR-10 sequence : Task A { airplane , automobile , bird , cat , deer } → Task B { dog , frog , horse , ship , truck } . In order to make a fair comparison , we plot the test accuracies of the model on both tasks at different steps of Task B fine-tuning for a variety of learning rates . This allows us to see , for each model , the maximum performance achievable on the new task for a given performance on the initial task . We call this the forgetting frontier . Figure 1 shows that the forgetting frontier for both ViT and ResNet models improves significantly with scale . While a 5.7M-parameter ViT-xS model loses about 6.5 % in Task A accuracy over the course of learning to perform Task B , an 86.7M-parameter ViT-B model loses less than half a percent . To put this in context , in a previous studies of split-CIFAR-10 task sequences , a ResNet18 model lost close to 40 % accuracy with no mitigation methods , 5 % when using a replay buffer , and 20 % when using elastic weight consolidation ( see Figure 1 of Ramasesh et al . ( 2021 ) for an example ) . It is also important to note that this effect of scale and pretraining on improving forgetting frontiers does not seem to be architecture-dependent . Both vision transformers—based on self-attention— and ResNets—based on convolutions—exhibit improvements in their forgetting frontiers as they grow larger , as long as they are pretrained . We see this improvement in forgetting performance beyond the simple two-task split-CIFAR-10 scenario . Across a 10-task sequence of 10-class split-CIFAR-100 tasks ( Figure 2 left ) , sequential 50-class splits of CIFAR-100 ( Figure 2 right ) , and the CIFAR-100 single-head input-distributionshift task ( Figure 3 ) , large-scale pretrained models exhibit resistance to forgetting , and continual learning ability appears to improve systematically with scale . For detailed results , scaling plots , and forgetting frontiers in the 10-task scenario , see Section B.2 in the Supplement . 3.2 DEPENDENCE ON FINETUNING DATASET . It is interesting to study how the improved resistance to forgetting with scale depends on the choice of finetuning dataset . In particular , it is worth noting that , though quite different in format , CIFAR10/100 are semantically quite similar to subsets of ImageNet21k , our pretraining set . To empirically investigate the dependence on finetuning dataset we have run additional experiments similar to those presented in Figure 1 for the Oxford Pet and Flowers , Caltech Birds , Stanford Cars , street view house numbers ( SVHN ) and DomainNet Clipart datasets . Results are shown in Figure 4 and in Figure 17 . We note in particular that SVHN classes are numerals and so distinct from ImageNet21k classes , while the Clipart dataset consists of synthetic , rather than natural images . See the Supplement for further details . 3.3 FORGETTING IN PRETRAINED MODELS VS. MODELS TRAINED FROM SCRATCH . In the previous section , we saw that the amount of catastrophic forgetting in models pretrained on large datasets decreased with increasing model size , to the point that the largest models ( ViT-B , with 86.7M parameters and ResNet200 with 62.6M parameters ) suffer very little performance loss from second-task training . In this section , we investigate whether this improvement in forgetting is simply a consequence of the improvement in performance on Task A . That is , do pretraining and scale have benefits with respect to forgetting beyond simply improving the fine-tuned performance on downstream tasks ? To check this , we compare the performance and amount of forgetting in pretrained ResNet models to models trained from scratch . We handicap the pretrained ResNets so that their maximum accuracy on on Task A matches that of the corresponding trained-from-scratch ResNet ; this is done simply by reducing the number of steps for which we fine-tune . Figure 5 shows the results of this experiment , with additional results and details in Appendix C.1 . While the trained-from-scratch ResNets achieve the same performance as their pretrained counterparts on Task A , their Task A performance degrades significantly more than the pretrained ResNets when trained on Task B ; pretrained ResNets forget less , and this improves with model scale . This result suggests that pretraining and model scale confer benefits upon models which go beyond simply improving performance . From Figure 5 it is also clear ( see also Figure 19 in the appendix , which plots the same data in a different way ) that pretraining is required to see the benefits of scale . Simply scaling up the model size in trained-from-scratch ResNets does not yield clear improvements in their forgetting performance . We focused in this section exclusively on ResNets ( rather than Vision Transformers ) because obtaining decent performance on image classification tasks with Vision Transformers seems to require pretraining . In our experiments , for example , a trained-from-scratch ViT-S model was unable to achieve better than 75 % test accuracy on five-class CIFAR-10 , while the same model , pretrained , achieved better than 98 % accuracy .

Summary: The paper explores the effect of scale on catstrophic. They conduct several different experiments (for both ResNets and Vision Transformers) showing: - the size of the model for pretrained models reduces forgetting  - (Longer) pretraining reduces forgetting - More pretraining training data size reduces forgetting - A metric used to measure overlap between model representations showed that pretraining results in more orthoganal representations between classes.  - The length of finetuning isn't as impactful on pretrained models as randomly initialized models They perform the majority of their experiments on a two task, task incremental Split CIFAR-10 dataset. Some experiments were done on 2-task class incremental and 10-task task incremental Split CIFAR-100. They also conduct a brief set of experiments showing that scaling model size  reduces forgetting for language as well.