An Empirical Investigation of the Role of Pre-training in Lifelong Learning

1 INTRODUCTION . The contemporary machine learning paradigm concentrates on isolated learning ( Chen & Liu , 2018 ) i.e. , learning a model from scratch for every new task . In contrast , the lifelong learning ( LL ) paradigm ( Thrun , 1996 ) defines a biologically-inspired learning approach where models learn tasks in sequence , ideally preserving past knowledge and leveraging it to efficiently learn new tasks . LL has the added benefit of avoiding periodical re-training of models from scratch to learn novel tasks or adapt to new data , with the potential to reduce both computational and energy requirements ( Hazelwood et al. , 2018 ; Strubell et al. , 2019 ; Schwartz et al. , 2020 ) . In the context of modern machine learning where state-of-the-art models are powered by deep neural networks , catastrophic forgetting has been identified as a key challenge to implementing successful LL systems ( McCloskey & Cohen , 1989 ; French , 1999 ) . Catastrophic forgetting happens when the model forgets knowledge learned in previous tasks as information relevant to the current task is incorporated . Mitigating or preventing this phenomenon is critical to achieving true LL . At the same time , transfer learning ( TL ) has shown impressive results in both computer vision ( CV ; Zhuang et al . 2021 ) and natural language processing ( NLP ) applications ( Howard & Ruder , 2018 ; Peters et al. , 2018 ; Devlin et al. , 2019 ) .2 The modern TL paradigm involves pre-training a fixed architecture , like ResNet ( He et al. , 2016 ) or BERT ( Devlin et al. , 2019 ) , using copious amounts of data , and then fine-tuning the learnt parameters on target tasks . Given the tremendous success of pre-trained models , there has been increased interest in understanding their role in improving generalization ( Erhan et al. , 2010 ; Neyshabur et al. , 2020 ) , speed of convergence ( Hao et al. , 2019 ) , 1The code is available in the supplemental material . 2One of the original motivations for transfer learning was as a way to enable lifelong learning , discussed in a NIPS-95 workshop on “ Learning to Learn ” ( Pan & Yang , 2009 ) . successful transfer ( He et al. , 2019 ; Pruksachatkun et al. , 2020 ) , and out-of-distribution robustness ( Hendrycks et al. , 2020 ; Tu et al. , 2020 ) . Despite these efforts , the role of pre-trained initializations in lifelong learning settings has been under-explored . In contemporary work , it has been shown that pre-trained models can be used as feature extractors ( i.e. , pre-trained weights are frozen ) for task-sequential learning ( Hu et al. , 2021 ) . Because the pre-trained weights are explicitly frozen in this setting , the model undergoes no catastrophic forgetting . In contrast , fine-tuning pre-trained weights updates the pre-trained model parameters and is susceptible to severe forgetting . This is typically the most accurate and thus common TL paradigm ( Peters et al. , 2019 ) , and the one we consider in this work . To the best of our knowledge , there is no prior work systematically analyzing the role of pre-trained initialization on catastrophic forgetting in lifelong learning scenarios . Figure 1 shows that simply changing the network initialization to generic pre-trained weights can significantly reduce forgetting on the first task when doing sequential training on five tasks . This observation motivates us to ask : Does pre-training implicitly alleviate catastrophic forgetting , and if so , why ? To answer this question we conduct a systematic study on existing CV and NLP benchmarks and observe that pre-training indeed leads to less forgetting . We also investigate the effect of the type of pre-trained initialization by analyzing the extent to which three pre-trained Transformer language model variants ( Sanh et al. , 2019 ; Devlin et al. , 2019 ; Liu et al. , 2019 ) undergo forgetting , observing that increasing the capacity of the model and diversity of the pre-training corpus play an important role in alleviating forgetting . To further stress-test these models under realistic scenarios , we introduce a dataset with 15 diverse NLP tasks and observe a considerable increase in forgetting on this dataset . We hypothesize that pre-trained weights already have a good inductive bias to implicitly alleviate forgetting . To explain this behavior , we build upon two separate observa- tions — Hao et al . ( 2019 ) ; Neyshabur et al . ( 2020 ) show that in the context of TL , pre-trained weights lead to a flat loss basin when fine-tuning on a single task . Mirzadeh et al . ( 2020b ) argues that the geometric properties of the local minima for each task play a vital role in forgetting , and they suggest modifying the hyper-parameters ( learning rate decay , batch size , dropout ) to promote flat minima . To verify the above hypothesis , we analyze the loss landscape of the first task as the model is trained sequentially on subsequent tasks . For pre-trained initializations , we see that minima obtained after training on a sequence of tasks still remain in the relatively low loss basin of the first task when compared with random initialization.3 These observations hint at the flatness of the minima reached in the case of pre-trained initialized models . To quantify the flatness of the loss landscape , we evaluate a sharpness metric ( Keskar et al. , 2017 ) and verify that pre-trained weights indeed lead to flat basins in comparison to random weights while training sequentially . These analyses help us showcase that continual training from pre-trained weights induces wide task minima , therefore , implicitly alleviating forgetting . To further mitigate forgetting , we explicitly optimize for flat loss basins by minimizing the current task loss and the sharpness metric . Concretely , we use the Sharpness-Aware Minimization ( SAM ) procedure ( Foret et al. , 2021 ) to seek parameters that lie in the neighborhoods having uniformly low loss values ( Section 5 ) and report improved results across many experimental settings . Our main contributions can be summarized as follows : • We observe that initializing models with pre-trained weights results in less forgetting compared to random weights despite achieving higher performance on each task . We bolster this observation with a systematic study validating that this behavior persists across applications ( NLP , CV ) and prominent approaches : Elastic weight consolidation ( Kirkpatrick et al. , 2017 ) , A-GEM ( Chaudhry et al. , 2018 ) , and episodic replay ( Chaudhry et al. , 2019 ) . 3Linearly interpolating loss across sequentially trained task minima confirms that models initialized with pre-trained weights undergo a more gradual change in loss compared to random initialization ( Appendix E.1 ) . • To understand the role of varying pre-trained initializations , we analyze a suite of pre-trained Transformer language models and showcase that model capacity and diversity of the pretraining corpus do play a role in alleviating forgetting . We also show that sequential training on diverse tasks is still challenging for pre-trained initialized models by introducing a new , more challenging benchmark for LL in NLP consisting of 15 diverse NLP tasks . • We hypothesize and verify empirically that pre-trained models alleviate forgetting as they have an implicit bias towards wider task minima . The effect of these wider minima is that changes in weights from learning subsequent tasks result in a smaller change to the current task loss , which helps reduce forgetting . We further show that explicitly seeking flat basins during sequential fine-tuning results in even less forgetting . 2 PRELIMINARIES . 2.1 PROBLEM SETUP : LIFELONG LEARNING ( LL ) . We consider a setup where we receive a continuum of data from different tasks in sequential manner : ( x1 , y1 , t1 ) , · · · , ( xi , yi , ti ) , · · · . Each triplet ( xi , yi , ti ) consists of a task descriptor ti ∈ T , input data xi ∈ Dti , target labels yi ∈ Yti and satisfies ( xi , yi ) iid∼ Pti ( X , Y ) . Following ( Chaudhry et al. , 2019 ) , we consider an explicit task descriptor ti because the same input xi can appear in multiple different tasks but with different labels . For example , given a product review , we could annotate it with sentiment polarity and grammatical acceptability judgments . Given the observed data , our goal is to learn a predictor f : X × T → Y where we want to evaluate test pairs ( x , t ) from previously observed tasks ( backward transfer ) and the current task at any time during LL of our model . 2.2 DATASETS AND TASK SEQUENCES . We perform extensive experiments on widely adopted task-incremental learning benchmarks ( Chaudhry et al. , 2019 ; Ebrahimi et al. , 2020 ; Wang et al. , 2020 ) across both CV and NLP domains . These benchmarks help us evaluate our method to be consistent with the literature . Most of the existing works consider Split MNIST , Split CIFAR-10 , and Split CIFAR-100 benchmarks , which are homogenous ; different tasks in these benchmarks share the same underlying domain . Given the generic nature of the pre-trained initialization , we want to investigate forgetting when subjected to a sequence of diverse tasks . Therefore , we also consider datasets spanning diverse CV and NLP tasks . CV benchmarks . We perform our experiments on 5-dataset ( diverse ) and Split CIFAR-50 / Split CIFAR-100 ( homogenous ) . 5-dataset consists of 5 diverse 10-way image classification tasks : CIFAR10 ( Krizhevsky & Hinton , 2009 ) , MNIST ( LeCun , 1998 ) , Fashion-MNIST ( Xiao et al. , 2017 ) , SVHN ( Netzer et al. , 2011 ) , and notMNIST ( Bulatov , 2011 ) . It is one of the largest datasets used for LL experiments ( Ebrahimi et al. , 2020 ) with overall 180.9k train examples ( see Table 4 in the Appendix B for task-specific statistics ) . Split CIFAR-50 takes the first 50 classes of the CIFAR-100 image classification dataset ( Krizhevsky & Hinton , 2009 ) and randomly splits them into 5 homogenous 10-way classification tasks . Each task contains 5k/1k ( train/test ) examples . We built this dataset as a homogenous counterpart to 5-dataset by mimicking its task structure ( 10 classes/task ) and the number of tasks . Further , we note that Split CIFAR-50 ( 10 classes/ task ) is more challenging than Split MNIST/ CIFAR-10 ( 2 classes/ task ) because of the more number of classes per task . Therefore , in this work , we prefer Split CIFAR-50 over MNIST/CIFAR-10 for our experimentation . Split CIFAR-100 splits the CIFAR-100 dataset into 20 disjoint 5-way classification tasks , with each task containing 2.5k/0.5k ( train/test ) examples . Due to a large number of tasks in this dataset , it is regarded as one of the most challenging and realistic CV benchmarks for LL ( Chaudhry et al. , 2018 ) . NLP benchmarks . We perform our experiments on Split YahooQA ( homogenous ) and 5-datasetNLP/ 15-dataset-NLP ( diverse ) . Split YahooQA consists of 5 homogenous 2-way classification tasks and is built from a 10-way topic classification dataset ( Zhang et al. , 2015 ) by randomly splitting topics into different tasks . Each task includes around 279k/12k train/test examples . 5-datasetNLP consists of text classification datasets ( Zhang et al. , 2015 ) from 5 diverse domains : AGNews , Yelp , Amazon , DBPedia , and YahooQA . Following the data processing procedure mentioned in ( de Masson d ’ Autume et al. , 2019 ) , we have 115k/7.6k ( train/test ) examples per task . 15-datasetNLP : In order to study the role of different pre-trained initializations in LL , we introduce 15-dataset- NLP , a novel suite of diverse tasks for LL . It consists of 15 text classification tasks covering a broad range of domains . Although there exists 5-dataset-NLP benchmark , we show that our introduced benchmark proves more challenging than the previous setup ( see Section 3.3 ) . 15-dataset-NLP consists of single sentence or sentence pair classification tasks : CoLA ( Warstadt et al. , 2019 ) , BoolQ ( Clark et al. , 2019 ) , SST-2 ( Socher et al. , 2013 ) , QQP4 , YahooQA ( Zhang et al. , 2015 ) , Yelp ( Zhang et al. , 2015 ) , Event Factuality ( Poliak et al. , 2018 ) , Argument Aspect Mining ( Stab et al. , 2018 ) , Explicit Discourse Marker Prediction ( Prasad et al. , 2019 ; Kim et al. , 2020 ) , QNLI ( Wang et al. , 2018 ) , Rocstory ( Mostafazadeh et al. , 2016 ) , MNLI ( Williams et al. , 2018 ) , SciTail ( Khot et al. , 2018 ) , Implicit Discourse Relation Classification ( Prasad et al. , 2019 ; Kim et al. , 2020 ) , and Emotion Detection ( Saravia et al. , 2018 ) . For more details on this dataset , see Appendix B . Task sequences . One of the desiderata of a LL method is to be robust to different task sequences as task ordering is unknown beforehand . Hence , we run all of our experiments with 5 random task sequences and report average performance ( see Appendix B.1 for task sequences ) .

Summary: The authors present a comparative study on the impact of using pre-trained models on task incremental learning problems. They split computer vision and NLP datasets into multiple tasks and evaluate performance between pre-trained and randomly initialised models. They conclude that task incremental learning benefits from pre-training due to the flatter loss landscapes, and propose to use an optimisation constraint to learn better models.