An Empirical Investigation of the Role of Pre-training in Lifelong Learning
Authors: Sanket Vaibhav Mehta
Darshan Patil
Sarath Chandar
Emma Strubell
Venue: ICLR 2022 Submitted (2022)
Keywords: Lifelong Learning
Continual Learning
Catastrophic Forgetting
Pre-training

Abstract: The lifelong learning paradigm in machine learning is an attractive alternative to the more prominent isolated learning scheme not only due to its resemblance to biological learning, but also its potential to reduce energy waste by obviating excessive model re-training. A key challenge to this paradigm is the phenomenon of catastrophic forgetting. With the increasing popularity and success of pre-trained models in machine learning, we pose the question: What role does pre-training play in lifelong learning, specifically with respect to catastrophic forgetting? We investigate existing methods in the context of large, pre-trained models and evaluate their performance on a variety of text and image classification tasks, including a large-scale study using a novel dataset of 15 diverse NLP tasks. Across all settings, we observe that generic pre-training implicitly alleviates the effects of catastrophic forgetting when learning multiple tasks sequentially compared to randomly initialized models. We then further investigate why pre-training alleviates forgetting in this setting. We study this phenomenon by analyzing the loss landscape, finding that pre-trained weights appear to ease forgetting by leading to wider minima. Based on this insight, we propose jointly optimizing for current task loss and loss basin sharpness in order to explicitly encourage wider basins during sequential fine-tuning. We show that this optimization approach leads to performance comparable to the state-of-the-art in task-sequential continual learning across multiple settings, without retaining a memory that scales in size with the number of tasks.

Under review as a conference paper at ICLR 2022
AN EMPIRICAL INVESTIGATION OF THE ROLE OF PRE- TRAINING IN LIFELONG LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The lifelong learning paradigm in machine learning is an attractive alternative the more prominent isolated learning scheme not only due its resemblance to biological learning, but also its potential to reduce energy waste by obviating excessive model re-training. A key challenge to this paradigm the phenomenon of catastrophic forgetting, With the increasing popularity and success of pre-trained models in machine learning, we pose the question: What role does pre-training play in lifelong learning, specifically with respect to catastrophic forgetting? We investigate existing methods in the context of large, pre-trained models and evaluate their performance on a variety of text and image classification tasks, including a large-scale study using a novel dataset of 15 diverse NLP tasks. Across all settings, observe that generic pre- vre-training implicitly alleviates the effects of catastrophic forgetting when learning multiple tasks sequentially compared to randomly initial- ized models. We then further investigate why pre-training alleviates forgetting in this setting. We study this phenomenon by analyzing the loss landscape, finding that pre-trained weights appear to ease forgetting by leading wider minima. Based on this insight, we propose jointly optimizing for current task loss and loss basin sharpness in order to explicitly encourage wider basins during sequential fine-tuning. We show that this optimization approach leads to performance compa- rable to the state-of-the-ard in cask-sequentil continual learning across multiple settings, without retaining memory that scales in size with the number of tasks.
INTRODUCTION
The contemporary machine learning paradigm concentrates on isolated learning (Chen & Liu, 2018) i.e., learning model from scratch for every task. In contrast, the lifelong learning (LL.) paradigm (Thrun, 1996) defines a biologically-inspire. learning approach where models learn tasks in sequence, ideally preserving past knowledge and leveraging it to efficiently learn new tasks. LL has the added benefit of avoiding periodical re-training of models from scratch to learn novel tasks or adapt to new data, with the potential to reduce both computational and energy requirements (Hazelwood et al., 2018; Strubell et al., 2019; Schwartz et al., 2020). In the context of modern machine learning where state-of-the-art models are powered by deep neural networks, catastrophic forgetting has been identified as a key challenge to implementing successful LL systems (McCloskey & Cohen, 1989; French, 1999). Catastrophic forgetting happens when the model forgets knowledge learned in previous tasks as information relevant to the current task is incorporated. Mitigating preventing this phenomenon is critical to achieving true LL.
At the same time, transfer learning (TL) has shown impressive results in both computer vision (CV; Zhuang et al. 2021) and natural language processing (NLP) applications (Howard & Ruder, 2018; Peters et al., 2018; Devlin et al., 2019).2 The modern TL paradigm involves pre-training fixed architecture, like ResNet (He et al., 2016) or BERT (Devlin et al., 2019), using copious amounts of data, and then fine-tuning the learnt parameters on target tasks. Given the tremendous success of pre-trai models, there has been increased interest in understanding their role in improving generalization (Erhan et al., 2010: Neyshabur et al., 2020), speed of convergence (Hao et al., 2019),
1 Thee code is available in the supplemental material.
2One of the original motivations for transfer learning was way to enable lifelong learning, discussed in NIPS-95 workshop on "Laamnnng Learn" (Pan & Yang, 2009).

---

Under review as a conference paper at ICLR 2022
successful transfer (He et al., 2019; Pruksachatkun et al., 2020), and outtof-diitribution robustness (Hendrycks et al., 2020; Tu et al., 2020). Despite these efforts, the role of pre-trained initializations in lifelong learning settings has been under-explored. In contemporary work, it has been shown that pre-trained models can be used as feature extractors (i.e., pre-trained weights are frozen) for task-sequential learning (Hu et al., 2021). Because the pre-trained weights explicitly frozen in this setting, the model undergoes no catastrophic forgetting. In contrast, fine-tuning pre-trained weights updates the pre-trained model parameters and susceptible to severe forgetting. This is typically the most accurate and thus common TL paradigm (Peters et al., 2019), and the one we consider in this work. To the best of our knowledge, there is no prior work systematically analyzing the role of pre-trained initialization on catastrophic forgetting in lifelong learning scenarios.
Figure shows that simply changing the network initial- ization to generic pre-trained weights can significantly reduce forgetting on the first task when doing sequential training on five tasks. This observation motivates us to ask: Does pre-training implicitly alleviate catastrophic forgetting, and if so, why? To answer this question we conduct a systematic study on existing CV and NLP bench- marks and observe that pre-training indeed leads to less forgetting. We also investigate the effect of the type of pre-trained initialization by analyzing the extent to which three pre- rre-trined Transformer language model variants (Sanh et al., 2019; Devlin et al., 2019; Liu al., 2019) undergo forgetting, observing that increasing the capac- ity of the model and diversity of the pre-training corpus play an important role in alleviating forgetting. To further stress-test these models under realistic scenarios, we intro- duce dataset with 15 diverse NLP tasks and observe considcrable increase in forgetting on this dataset.
90
Accuracy
70
60
50
Inhlaization
11com
crotrsired
4
Tisks Learned
Figure 1: Pre-trained and randomly ini- tialized DistilBERT on Split YahooQA dataset. Performance of the first task vi- sualized over sequential learning of tasks (averaged over 5 runs). Both models start with approximately equal average task accuracy, but pre- rre-trained initializa- tion leads to significantly less forgetting.
We hypothesize that pre-trained weights already have good inductive bias to implicitly alleviate forgetting. To explain this behavior, we build upon two separate observa- tions- - Hao et al. (2019); Neyshabur et al. (2020) show that in the context of TL., pre-trained weights lead to a flat loss basin when fine-tuning on single task. Mirzadeh et al. (2020b) argues that the geometric properties of the local minima for each task play a vital role in forgetting, and they suggest modifying the paper-paraettrss (learning rate decay, batch size, dropout) to promote flat minima.
To verify the above hypothesis, we analyze the loss landscape of the first task as the model is trained sequentially on subsequent tasks. For pre-trained initializations, we see that minima obtained after training on sequence of tasks still remain in the relatively low loss basin of the first task when compared with random initialization.' These observations hint the flatness of the minima reached in the case of pre-trained initialized models. To quantify the flatness of the loss landscape, we evaluate a sharpness metric (Keskar et al., 2017) and verify that pre-trained weights indeed lead to flat basins in comparison to random weights while training sequentially. These analyses help us showcase that continual training from pre-trained weights induces wide task minima, therefore, implicitly alleviating forgetting. To further mitigate forgetting. we explicitly optimize for flat loss basins by minimizing the current task loss and the sharpness metric. Concretely, we use the Sharpness-Awee -Aware Minimization (SAM) procedure (Foret et al., 2021) to seek parameters that lie in the neighborhoods having uniformly low loss values (Section 5) and report improved results across many experimental settings. Our main contributions can be summarized as follows:
We observe that initializing models with pre-trained weights results in less forgetting compared to random weights despite achieving higher performance on each task. We bolster this observation with a systematic study validating that this behavior persists across applications (NLP, CV) and prominent approaches: Elastic weight consolidation (Kirkpatrick et al., 2017), A-GEM (Chaudhry et al., 2018), and episodic replay (Chaudhry al., 2019).
'I Linearly interpolating loss across sequentially trained task minima confirms that models initialized with pre-trained weights undergo more gradual change in loss compared random initialization (Appendix E.
2

---

Under review as a conference paper at ICLR 2022
To understand the role of varying pre- pre-trained initializations, analyze suite of pre-trained Transformer language models and showcase that model capacity and diversity of the pre- training corpus do play role in alleviating forgetting. We also show that sequential training on diverse tasks is still challenging for pre-trained initialized models by introducing a new, more challenging benchmark for LL in NLP consisting of 15 diverse NI.P tasks.
We hypothesize and verify empirically that pre-trained models alleviate forgetting as they have implicit bias towards wider task minima. The effect of these wider minima is that changes in weights from learning subsequent tasks result in a smaller change to the current task loss, which helps reduce forgetting. We further show that explicitly seeking flat basins during sequential fine- tuning results in even less forgetting.
2 PRELIMINARIES
2.1 PROBLEM SETUP: LIFELONG LEARNING (LL)
We consider setup where we receive continuum of data from different tasks in sequential manner: (31, (11,31, (1),, (Ii, Vi, li).-... Each triplet (I:;, Mis 44) consists of a task descriptor 4 E T, input data z; E De., target labels y/ E Vv. and satisfies (z,, std Pt, (X, Y). Following (Chaudhry et al., 2019), we consider an explicit task descriptor t, because the same input 2; appear in multiple different tasks but with different labels. For example, given a product review, we could annotate with sentiment polarity and grammatical acceptability judgments. Given the observed data, our goal is to learn a predictor f X' x T + y where we want to evaluate test pairs (x, 1) from previously observed tasks (backward transfer) and the current task any time during LL of our model.
2.2 DATASETS AND TASK SEQUENCES
We perform extensive experiments on widely adopted mask-incremental learning benchmarks (Chaudhry et al., 2019; Ebrahimi al., 2020; Wang et al., 2020) across both CV and NLP do- mains. These benchmarks help us evaluate our method to be consistent with the literature. Most of the existing works consider Split MNIST, Split CIFAR-10, 10, and Split CIFAR-I00 benchmarks, which are homogenous; different tasks in these benchmarks share the same underlying domain. Given the generic nature of the pre-trained initialization, we want to investigate forgetting when subjected to a sequence of diverse tasks. Therefore, we also consider datasets spanning diverse CV and NI.P tasks.
CV benchmarks. We perform our experiments on 5-dataset (diverse) and Split CIFAR-50 / Split CIFAR-100 (homogenous). 5-dataset consists of 5 diverse 10-way image classification tasks: CIFAR- 10 (Krizhevsky & Hinton, 2009), MNIST (LeCun, 1998), Rashion-MNISS (Xiao al., 2017). SVHN (Netzer et al., 2011), and notMNIST (Bulatov, 2011). It is one of the largest datasets used for LL experiments (Ebrahimi et al., 2020) with overall 180.9k train examples (see Table in the Appendix B for task-specific statistics). Split CIFAR-50 takes the first 50 classes of the CIFAR- 100 image classification dataset (Krizhevsky & Hinton, 2009) and randomly splits them into 5 homogenous 10-way classification tasks. Each task contains 5k/1k (train/test) examples. We built this dataset as homogenous counterpart to 5-dataset by mimicking its task structure (10 classes/task) and the number of tasks. Further, we note that Split CIFAR-50 (10 classes/ task) is more challenging than Split MNIST/ CIFAR-I0 (2 classes/ task) because of the more number of classes per task. Therefore, in this work, we prefer Split CIFAR-50 over MNIST/CIFAR- 10 for our experimentation. Split CIFAR-100 splits the CIFAR-100 dataset into 20 disjoint 5-way classification tasks, with each task containing 2.5k/0.5k (train/test) examples. Due to a large number of tasks in this dataset, is regarded as one of the most challenging and realistic CV benchmarks for LLL (Chaudhry et al., 2018).
NLP benchmarks. We perform our experiments on Split YahooQA (homogenous) and 5-dataset- NLP/ 15-dataset-NLP (diverse). Split YahooQA consists of 5 homogenous 2-way classification tasks and is built from a 10-way topic classification dataset (Zhang et al., 2015) by randomly splitting topics into different tasks. Each task includes around 279k/12A train/test examples. 5-dataset- NLP consists of text classification datasets (Zhang et al., 2015) from diverse domains: AGNews, Yelp, Amazon, DBPedia, and YahooQA. Following the data processing procedure mentioned in (de Masson d' Autume et al., 2019), we have 115k/7. /..6/ (train/test) examples per task. 15-dataset- NLP: In order to study the role of different pre-trained initializations in LI,, introduce 15-dataset-
3

---

Under review as a conference paper at ICLR 2022
NLP, novel suite of diverse tasks for LL. It consists of 15 text classification tasks covering broad range of domains. Although there exists 5-dataset-NLP benchmark, we show that our introduced benchmark proves more challenging than the previous setup (see Section 3.3). 15-dataset-NLP consists of single sentence or sentence pair classification tasks: CoLAA (Warstadt et al.. 2019), BoolQ (Clark al., 2019), SST-2 (Socher et al., 2013), QQP YahooQA (Zhang al., 2015), Yelp (Zhang et al., 2015), Event Factuality (Poliak et al., 2018), Argument Aspect Mining (Stab et al., 2018), Explicit Discourse Marker Prediction (Prasad et al., 2019; Kim al., 2020), QNLI (Wang 2018), Roestory (Mostafazadeh et al., 2016), MNLI (Williams et al., 2018), SeiTail (Khot et al., 2018), Implicit Discourse Relation Classification (Prasad al., 2019; Kim al., 2020), and Emotion Detection (Saravia et al., 2018). For more details on this dataset, see Appendix B.
al.,
Task sequences. One of the desiderata of a LLL method is to be robust to different task sequences as task ordering is unknown beforehand. Hence, we run all of our experiments with 5 random task sequences and report average performance (see Appendix B. for task sequences).
2.3 EVALUATION
Let St,, denote the accuracy the task r after training on task t. After model finishes training on the task i, we compute the average accuracy (At), forgetting (Ft) and learning accuracy (I.A,) metrics as proposed by Lopez-Paz & Ranzato (2017); Riemer al. (2019). F. (also referred as backward transfer) measures the influence of learning task i on the performance of all previously seen tasks T, (1 S T < t). As the model learns multiple tasks in the sequence, we hope that knowledge acquired during LL should aid the learning of new tasks (forward transfer). LA, measures the learning capability when model sees the new task (indirectly measuring forward transfer). Say we learn the 1th task, then A,, F3 and LA, are defined as follows:
A
S
F== t t- re(L,-,tt1 x-1)
E max
(ST'. - St,r)
LA,
Σ
ST.
(1)
2.4 METHODS
We compare our method with offe--o-tthear baselines (Chaudhry et al., 2019; Mirzadeh al., 2020b). We first consider the finetune (FT) approach, where we simply fine-tune the model on each task in sequence with no additional constraints on learning. Elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) is reguulriizaiinn-baa based approach that tries mitigate forgetting by limiting learning for parameters important to previously learned tasks, as measured by the Fisher information matrix. A-GEM (Chaudhry et al., 2018) 8) and episodic replay (ER) (Chaudhry et al., 2019) methods augment the base model with episodic memory module which retains examples from the previously seen tasks. Following (Chaudbry et al., 2019), select examples for storage. Stable SGD (Mirzadeh et al., 2020b) controls training dynamics by varying the learning rate, learning rate decay, dropout and batch size (see Appendix A). (Prabhu et al., 2020; Hussain et al., 2021) show that simple ER method outperforms all of the previous methods under realistic task-incremental learning setting and we compare our method with ER.
retain one example per task per class and randomly
DOES PRE-TRAINING IMPLICITLY ALLEVIATE FORGETTING?
Having defined the formal problem definition, evaluation metrics, and methods for alleviating the forgetting phenomenon, in this section we conduct experiments to tease apart the role of pre-training for LL. We are interested in answering the following questions: (Q1) How much does pre-training help in alleviating the forgetting?
(Q2) Do pre-trained weights undergo similar forgetting on diverse and homogeneous tasks? (Q3) How do different pre-trained initializations affect forgetting?
Experimental design. To answer these questions convincingly, we conduct experiments on the above discussed CV and NI.P datasets. We utilize the DistilBERT hase (Sanh et al., 2019) architecture for text classification and the ResNet-18 (He et al., 2016) architecture for image classification. To isolate the effect of pre-training we consider two variants for each of these architectures: pre-trained models (DistiBBERT-PT. ResNet-18-PT) and randomly initialized models (DistilBERT-R, ResNet-18-R)
httn:///wwwwwgoooooooommm//nnnns Release-Quession-Pairss
4

---

Under review as a conference paper at ICLR 2022
Table 1: Comparing performance in terms of accuracy, forgetting, and learning accuracy across methods after training on the last task (averaged over 5 runs). f indicates higher, + lower is better. We observe that pre-trained models undergoes significantly less forgetting compared the randomly initialized models. Augmenting the Finetune baseline with SAM (Section 5) results in performance competitive with SOTA, and augmenting the ER baseline with SAM often outperforms SOTA.
w/o Pretraining (ResNet-18-8) DistUBERT-R
w/ Pretraining 18--P DisHIBRRTPPT
Accuracy(%)
Forgetting())
LA(S)
Accuracy(5)
Forgetring(())
LA(%)
Split-YahooQA
Finctune
73.10 (+:4.69
26.39 (#:5 88)
94.21 (#:0.0)
87.66 (+:3772)
9.45 (#4.666
95.22 (=0.02)
Finetune + SAM
73.45 (1:4.04)
25.94 (1:5.04)
94.20 (:10.03)
88.53 (112.1))
8.37 (1.3.52)
95.22 (100.00)
EWC
76.06 (d:3.09)
22.69 (4:3.88)
94.21 (#:0.02)
89.52 (#:3.35)
7.13 (4:4200
95.22 (+0.02)
ER
77.19 (#:3.33)
21.28 (::4.17)
94.22 (2:0.01)
89.35 (=:0.69)
7,33 (=0090)
(=20000)
SAM
77.48 (::11.0)
20.91 (1:1.76)
94.21 (1:0.02)
(::0.69)
(2:0.1)
95.20 (2:004)
S-dataset-NLP
Finetune
44.27 (d::499)
36.66 (1:6.27)
73.59 (1:0.07)
64.28 (#:4.53)
16.73 (77)67))
77.67 (+00066
Finetune SAM
45.95 (+#4,99
34.30 (#:6.25)
73. 39 (=00055
66.41 2+2.7
13.93 (#3.48)
77.56 (=0006)
(::4499)
(1:6.19)
73.60 (::0.44
(::3.30)
13.60 (#4.155)
77.64 (+0007)
ER
56.34 (..005
21.58 (13390)
73.60 (110.10)
70.17 (:1)
9.39 (122004)
77.68 (1.0.66)
SAM
(1:3.93)
21 (1:5.03)
73.43 (1:0.12)
(:11.21)
1.51)
77.53 (40004)
Split CIFAR-50
Finetune
42.76 (1:3.14)
23.68 (d:1.09)
66.44 (::2.14)
86.29 86:29 (:::111)
7.11 (1.0.92)
93.40 (1:0.66)
Finetune SAM
50.34 (1:2.19)
14.95 (1:2.14)
(1:1.00)
90.45 (:11.111)
4,24 (110.98)
94.69 (1:0.40)
EWC
45.28 (+:2.53)
20.65 (77)
65.93 (.2))
86.15 (5:0.85)
A-GEM
47.34 (+:2.65)
21.08 (=:1955)
68.42 (=0.711)
87.25 (+:0.95)
6.166660.33
(3.004)
(1:1.6)
(122655
86.16 (::1005)
(1:0.8))
93.30 (1:046)
ER + SAM
50,75 (+00553
16.88 (+00.5)
67.63 (+00668
(++1.6)
(=11)4)
94.40 (+222
Stable SGD
46.02 (4::233)
12.07 (4:0.42)
58.09 (::2.45)
84.06 19)
(=1.66)))
89.21 (=0773
5-dataset
Finetune
(::2.53
(+:2.58)
(#:199)
$7.19 (0:
38,28 (#5.00)
95.47 (=00200)
Finctune SAM
47,58 (+:3775)
40.60 (+:3 98)
88. 18.19 (+ 1.3
70.40 (++437)
25.61 (#4.44)
96.01 (0)
BWC
34.99 (#:4.94)
50.05 (::6.53)
85.05 (+: 15)
56.67 (+3.75)
38.75 (+3.78)
95.42 (0116
A-GEM
46.05 (1:6.75)
39.48 (1:7.13)
85.17 (1:2553
71.98 (1:2.27)
23.00 (122227)
94.98 (1.022
ER
50.58 (::4.50)
35.02 (115.35)
85 63.6016 (11)
70.73 (H:1.52)
14116
94.89 (1:0.19)
SAM
60.26 (:99444)
27.32 (+:4.05)
59 (3))
77.40 (+:391)
18.23 (#3.00)
95.63 (8)
Stable SGD
50,20 (+:7.04
40,31 (1:7.81)
90.51 (::0.97)
71.32 (#:2.71)
20.53 (#2444)
91.86 (=:0.84)
Split CIFAR-100
Finetune
38.89 (.12.20)
39.11 (112.02)
77.96 (.10.96)
81.98 (.12.97)
13.83 (12.555)
95.81 (110.50)
Finetune SAM
48.93 (::4.78)
28:53
77.46 (1:087)
88.31 (#11.72)
8.60 1.22)
96.91 (#0.55)
EWC
37.37 (=4)))
40.12 (2:182)
71.47 (#:1
81.29 (+2.52)
95.81 (=0661)
A-GEM
(::3.48)
32.01 (1:3.77)
(20095)
(1:1.88)
(.002)
95.70 (10069)
ER
48.60 (++1.6))
29.84 (+))
18.10
84.41 (+22177
11.36 (+177))
95.75 (+0.53)
ER + SAM
60.53 (1:0.45)
20.91 (::0.69)
81.39 (1:0.70)
88.41 (+:0.71)
8.61 (=2020)
96.73 (=0.53)
Stable SGD
52.94 1.711
21.02 (1:2.01)
73.83
86.64 (1.2.17)
(11.54)
91.75 (10.74)
For our study, we need ensure that there are as few confounding factors as possible. Therefore, we keep all other hyperparameters the same and vary only the initialization (for more details refer to Appendix A). To measure the severity of forgetting, we ideally want sufficient training samples to ensure either a pre-ttrined model or randomly initialized model of the same capacity can achieve similar learning accuracy on each task. To control for this behavior we either select large training corpus whenever available (e.g.. 279k examples/task for Split YahooQA) or run our experiments for multiple epochs (5 epochs for CV benchmarks). Finally, the ResNet- 18-PT model was pretrained on subset of Imagenet consisting of 733 classes where classes that were semantically similar to the classes in the CIFAR- 100 dataset were removed (more details Appendix B).
3.1 HOw MUCH DOES PRE-TRAINING HELP IN ALLEVIATING FORGETTING?
From Table 1, we see that pre-trained models 1eesNet-18-PPP DistiBBRRT-PT) undergo significantly less forgetting in comparison to models with random initializations (ResNet-18-B. DistilBERT-R). This trend holds across all three methods. For text classification tasks (Split YahooQA, 5-dataset- NLP), we see that both models have comparable learning accuracy (LA) and significantly less forgetting for DistilBERT-PPT. This can be completely attributed to the pre-trained initialization. Now on 5-dataset, ResNet- 18-PT (38.28) undergoes less forgetting when compared to ResNet- 18-R (51.51). Specifically, despite task accuracy starting at a higher base for ResNet- 18- PT, the absolute forgetting value is still lower compared ResNet-188 models. Additionally, this effect also holds
5

---

Under review as a conference paper at ICLR 2022
when considering sequentially finetumed pre-trained model (with additional regularization to alleviate forgetting) to a randomly initialized model trained with LL methods. For example, on 5-dataset-NLP, sequentially finetuning DistilBERT-PT undergoes less forgetting (16.73) compared to competitive ER method (21.58) when applied to DistilIBERT-R, This raises an interesting research direction explicitly focusing learning generie features while training sequentially apart from just focusing on the forgetting aspect of LL. Further, in Appendix C we discuss task-specific results.
3.2 DO PRE-TRAINE) WEIGHTS UNDERGO SIMILAR FORGETTING ON DIVERSE AND HOMOGENEOUS TASKS?
From Table 1, we see that ResNet- 18-PT does not undergo a significant amount of forgetting when sequentially fine-tuned on Split CIFAR-50, Split CIFAR-100 100 (homogenous tasks). On Split CIFAR- 50, forgetting is around 7 accuracy points. Surprisingly, the competitive ER method also undergoes similar amount of forgetting, thereby raising a question about the applicability of these datasets when studying forgetting in the context of the pre-trained models. It may be possible to manually cluster tasks based upon semantic closeness, rendering severe interference to make these benchmarks more challenging (Ramasesh et al., 2020). Given the generic nature of the pre-trained initialization, we ask: What happens when we train the model sequentially on diverse tasks? To answer this question, we conduct experiments on 5-dataset and 5-dataset-Ni.P From Table 1, empirically observe that pre-trained models are susceptible to forgetting when exposed to diverse tasks. Particularly, DistilBERT-PT undergoes 16.73 point drop accuracy when trained on 5-dataset-NLP. Similarly, ResNet- 18-PT undergoes 38. 28 point drop in accuracy when trained on 5-dataset.
3.3 How DO DIFFERENT PRE-TRAINED INITIALIZATIONS AFFECT FORGETTING?
To examine the impact of varying pre-trained initialization on forgetting, we evaluate different pre- trained Transfomer models, DistiBBERT (Sanh et al., 2019), BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), on text classification tasks. From the previous subsection, we observe that pre- trained models are relatively more susceptible to forgetting on LL of diverse tasks. In response, we conduct a thorough investigation on the 5-dataset-NLP. From Table 2, we observe that when keeping the pre-training corpora the same and increasing the capacity of the model DistilBERT (66M), BERT-base (110M), and BaRT-large (336M) - we observe that larger models undergo less forgetting on sequential training of diverse NL.P tasks. Further, to explore the impact of the diversity of the pre-training corpora, we compare BERT-base (1 10M) with RoBERTa-base (125M). We observe that the RoBERTa-base model performs far superior BERT-base, thus hinting at the necessity of diverse pre-training corpora to implicitly alleviate forgetting. To stress-test these models, we experiment with the 15-dataset-NLPP We observe that by increasing the number of tasks in the sequence, pre-trained models undergo severe forgetting, Surprisingly, the RoBERTa- base model put- BERT- despite having many fewer parameters. Empirically, we infer that diversity of pre-training corpora plays a vital role in easing forgetting during LL of diverse tasks.
4 EXPLORING THE LOss LANDSCAPE
To better understand how pre-training reduces forgetting, we perform experiments analyzing where models are situated in the loss landscape after training on each task. We denote model parameters after training on task i as wi. If we define forgetting as the increase in loss for given task during training (instead of decrease in accuracy). Mirzadch et al. (2020b) show that the forgetting can actually be bounded by:
XT (w - Li (w() R 2 Aw L' S 2
mer |wul?
(2)
where L1 (w) represents the loss on Task with parameters w, Au W2 W1, and Amar is the largest eigenvalue of L (w]). The magnitude of the eigenvalues of L can be used to characterize the curvature of the loss function (Keskar et al., 2017). and thus Xmaax can be thought of proxy for the flatness of the loss function (lower is flatter). From Equation 2, we can see that the flatter the minima, the less forgetting occurs in the model.
We hypothesize that the one explanation of improvements from pre- pre-training shown in the previous section might be because pre-training leads to a more favorable loss landscape. Specifically, pre-

---

Under review as a conference paper at ICLR 2022
Table 2: Comparing performance in terms of average accuracy, forgetting, and learning accuracy for sequential finetuning after training on the last task. 1 indicates higher is better, + indicates lower is better. All metries are averaged across runs. Overall, we observe that models pre-trained on diverse corpora (RoBERTa-base) undergo less forgetting across both 5 and 15 diverse tasks. Augmenting the Finetune baseline with SAM (Section 5) results in performance competitive with SOTA, and augmenting the ER baseline with SAM often outperforms SOTA.
S-datasctNNLP (1 epoch)
15-dataset-NLPP (1 epoch)
Accuracy(56) 1
Forgetting! (%) L
LA(%)
Accuracy(5) 1
Forgetting(%) J
LA(%) 1
DistiBERT
Finetune
64.28 (d:4.53)
16.73 (1:5.67)
77.67 (::0.06)
46.99 (::3.50)
18.80(244.03) (2:4.03)
64.41 (#1.11)
Finetune SAM
66.41 (1:2.76)
13.93 (1.3.88)
77.56 (110.66)
47.51 (..3.12)
16.46 (1.3.84)
62.53 (1.0.88)
ER
70.17 (.00)
9.39 ( 04)
77.68 (2:0.06)
53.22 3.33
13.05 (#4.07)
65.32 (200.59)
SAM
71.07 (1=1.21)
8.07 (::155)
77.53 (=00.04)
53.46 (+:2.04)
11.00 (#3.05)
63.07 (=1.04)
BERT-base
Finetune
(::2766
13:57 (#:3.37)
78.40 (::0.07)
(#:2. 82)
(9:3.01
70.76 (#00277)
Finetune SAM
70.82 (1:2.08)
9.53 (1.2.52)
78.44 (1:0.07)
55.11 (11261))
16.38 (1:3.21)
70.39 (1.0.65)
70.61 (4:2.14)
(1:2.61)
(1:0.07)
56.39 (::2.93)
15.70 (23300)
71.03 (.3).8
ER SAM
72.97 (#11.54)
6,88 (=:1.89)
78.47 (=0007)
57.82 (=11.5))
13.65 1.91)
70.53 (=44)7)
RoBERTaa buse
Finctune
(+1.66)
9.52 (+:2.03)
79.04 (+0007)
$5.48 (+t3.10)
21.02 (+300
75.10 (=0.52)
Finetune + SAM
72.56 (:::122)
7.83 1.5))
78.83 (1:0.02)
57.82 (::1.72)
15.40 (#1.88)
72.08 1.36)
ER
73.69 (1.0.76)
6.74 (.10.93)
79.09 (1.0.10)
60.92 (:11.44)
15.32 1.58)
75.22 (1.0.12)
ER + SAM
74.33 (4:0.62
5.68 (::0.76)
78.85 (2:0.02)
62.14 (::4)5)
12.17 (#2009)
73.30 (=:0,79)
BERT-Large
Finetune
70.98 (112.02)
10:22 (1.2.53)
79.15 (1:0.03)
53.82 (11))2
23.42 (111.43)
75.68 (10.41)
Finetune SAM
(:22)
(+:0.03)
(#:3.38)
17.11 (#:4.26)
74.64 (#2.118)
ER
73.46 1.15)
7.17 (L.1.4)
79.20 (110007)
61.06 (:12.55)
15.06 (1.2.87)
75.12 122)
SAM
(1:0.62)
(1:0.83)
79.15 (1:0.09)
(::1255)
13.73 (2:2.77)
74.53 (#: 1.49)
training results in wider, flatter minima for each task. The effect of these wider minima is that the change in weights from learning on future tasks results in smaller change the actual loss for the current task, which leads to less forgetting. We verify this idea in two parts. First we use loss contours and then interpolate between model checkpoints show that the flat loss basins lead to smaller changes in loss. Next we compute a sharpness metric to show that pre- pre-training leads to flat loss basins. All models analyzed in this section are trained using the Finetune method.
4.1 Loss CONTOUR
In Figure 2, we visualize the contours of the test loss for the first task. On those contours, we plot the locations of the model (Wi, W2, wa) after training on each of the first three tasks. Pre-training results in significantly wider optima. In fact, as the model is trained on tasks sequentially, the pre-trained model still remains mostly at the same loss level as compared to randomly initialized models, despite moving approximately the same (or even more) euclidean distance away from the original model. For example, in the plot for the pre-trained model 5-dataset (Figure 2e), the model after the second task (w2)) remains at the same Task 1 loss level as after just training on Task 1 (wi). It is a couple of loss levels higher for task 3 (wa). For the randomly initialized model (Figure 2a), the euclidean distances between the model parameter vectors are approximately the same as for the pre-trained model, but the differences in Task loss levels are significantly higher. We provide more instances of these visualizations along with loss interpolation plots in Appendix E.
4.2 SHARPPESS
As another measure of the wideness of the minima, we calculate sharpness metric (Keskar et al., 2017) for the model on each task as goes through training. The metric tries to find the maximum value of the loss in the neighborhood of the minima, and calculates the difference between the maximum and the minimum loss value, scaled by the loss value. The maximization is performed in subspace of the entire parameter space R". specified by projection matrix A E Rx For our experiments, we randomly sample our matrix A and set p 100 as in Keskar et al. (2017). The neighborhood of the metric is given by:
C. (= E R (((A+x)) +1) S 2 <(((At)) + 1)Vi E (1.
(3)
7

---

Under review as a conference paper at ICLR 2022
1.0
1.5-
4/
10.0
14.0
12.0
1.0
7.5
5.0
3.0
4.0
1.8
2,4
1.5
2.1
6.0
1.0
12
0,5-
8.0
0.5
6.0
4.5
2.0
0.9
0.
0.
4.0
3.0
1.0
8.6
04
0.0
0.3
0.0
02
0.0
1.0
2.0
0.0
5.0
10.0
0.0
0.0
5.0
10.0
0.0
(a) 5datasct w/o PT
(b) CIFAR-50 w/o PT
(c) 5dataset-N NL..sc w/o PT
(d) YahooQA w/o PT
#2
14.0
21
6.0
#
12
1.0-
12.0
15
7.5
1.5
8.0
4.5
6.0
2.0
0.6
0.5
3.0
0.5
0.4
2.0
0.0
1.5
0.0
0.0
02
2.0
4.0
10.0
000
10.0
0.0
(e) Sdataset w/PT
(f) CIFAR-50 w/PT
(g) 5dataset-NLB w/PT
(h) YahooQA w/PT
Figure 2: Loss contours for Task I on each dataset. Each contour shows the location of the model parameters after training on each of the first three tasks. The top row shows contours for randomly initialized models and the bottom row shows contours for pre- pre-trained initialized models.
Table 3:
Average sharpness (lower is flatter) of minima across tasks in a 100 dimensional random subspace. Pre-training (PT) leads flat minima for each task in training by an order of magnitude.
5 10-4
10
Dataset
Method
w/o PT
w/ PT
w/o PT
w/ PT
Finetune Finetune SAM
2.07(++.56)
0.09(+0.02)
5.63(=1.64)
0.20 0.25(10.02)
5-dataset
0.69(100.1)
0.11
(.8(=
Finetune
2.26(#0.68)
0.18(+0.06)
6.13(=145)
0.44(#0.11)
Split CIFAR-50
Finetune SAM
0.73(+0.14)
0.24(+0.02)
0.55(+0.04) (0.32)
where A is the pseudo inverse of A, x the parameter vector and is hyperparameter controlling the size of the neighborhood. The metric is then defined as:
Φ2. (maxyec, f(r + Ay)) f(x) x 100
(4)
+ f(r)
where f(x) denotes the loss value with parameters We caleulate the sharpness metric at e 5 x 10-4 and 10-3 After training on each task, we compute the sharpness values of the minimum reached by the model on that task. We then take the sharpness value of the run be the mean of the values across the sequence of tasks. We present the mean and standard deviation across 5 runs. The sharpness values for 5-dataset and Split CIFAR R-50 are presented in Table and for rest of the datasets see Table 6 and Table 7 (in Appendix D). We see that for all datasets, the sharpness values for the pretrained initialized models are significantly lower than the values for the randomly initialized models.
5 LIFELONG LEARNING WITH SHARPNESS AWARE MINIMIZATION (SAM)
In the previous section, we looked at how initialization plays an important role in overcoming forgetting as pre-trained initializations favor flat loss basins. On the other hand, (Mirzadeh et al., 2020b) suggests modifying the training regime by varying learning rate decay, batch size, and dropout regularization such that inherent noise in the stochastic gradients leads to flat basins in the loss landscape. However, the procedure for tuning these hyperparameters before continual learning of tasks is ill-defined, thereby rendering their strategy less helpful and quite expensive. Doing a sweep over just the hyperparameter values suggested in the original paper requires 48 separate runs.
8

---

Under review as a conference paper at ICLR 2022
Furthermore, these hyperparameters cannot be reused across different architectures and datasets, and thus we need to do even more sweeps. To alleviate these issues, we propose to explicitly seek flat basins while sequential fine-tuning of the pre-trained weights. Towards this objective, employ the Sharpness- Minimization (SAM) procedure (Foret et al., 2021) that seeks parameters that lie in the neighborhoods having uniformly low loss values by jointly minimizing the task loss value and sharpness metric. SAM defines the sharpness of the loss function f at parameters æ as:
max f(x + E) f()) |e||مSp
(5)
where maximization region an S ball with radius p for p = 2 in Equation (5), SAM problem can be defined in terms of the following minimax optimization:
min max f( + |||=||
(6)
|
The gradient of the result of the maximization problem can be approximated as
0²(x)
Ve max f(x +c) Vaf(t) ||e||aSp
vzf(x) x+((x)
(7)
x+2(x)
T
where
v.f(z)
E(x) =
(8)
To make the optimization simpler, the second order term in the gradient is dropped, leaving with
Vx max f(x f(x (x)()) x+((x)
||claSS
The full derivation of this gradient can be found in Foret et al. (2021).
From the results Tables I and 2, we can see that SAM results a consistent improvement in performance over non-SAM counterparts. Simply adding SAM to the sequential finetune baseline makes the approach competitive with, and sometimes outperform, state-of-the-ate baselines like ER and Stable SGD, and with minimal hyperparameter tuning required (we used the default value of the p = 0.05 parameter for all experiments). Finally, since SAM is simply an optimizer, it can be combined with existing continual learning methods. Combining SAM with ER results in method that outperforms all existing baselines.
RELATED AND FUTURE WORK
We focus on several closely related lines of work studying the optimization and loss landscapes of deep learning and lifelong learning models here, and discuss some more distant related work in Appendix F. Hao et al. (2019) show that for single-task generalization, pre-training leads to wider optima for BERT models. Keskar et al. (2017) explore how larger batch sizes lead to sharper minima and worse generalization catastrophic forgetting can be impacted by the training regime, and show that certain hyperparameter settings produce wider minima which lead to less catastrophic forgetting. Finally, Mirzadeh et al. (2020a) compare minima that result from multitask learning and continual learning, and show that the the minima resulting from continual learning are linear mode connected to the optimal sequential multitask minima, but not to each other, which results in forgetting and corresponding drop in performance. These works all either explore the relation between pre-training and flatness of minima in single-task settings, between flatness of minima and model generalization capability. We extend this line of work by examining whether benefits from pre-training can persist across training on several tasks, assessing the effects of pre-training on loss landscapes over the course of lifelong learning, and validating hypothesis explaining the effects of pre-training on lifelong learning.
the single-task learning setting. Mirzadeh et al. (2020b) look at how
Based on our findings and results, a potential future work could explore where the multitask minima are in relation to the pre-trained initialization, as Mirzadeh et al. (2020a) show that the sequential multitask minima are linear mode connected to minima after each task in lifelong learning. The flainess of the minima for every model starting from pre-trained initialization could suggest way to regularize the sequential training process with the pre-trained initialization such that that the model ends up at the multitask minima. One final takeaway from these results is that lifelong learning methods should focus on creating more general representations instead of simply reducing catastrophic forgetting, more general representations appear to result in more robust learning.

---

Under review as a conference paper at ICLR 2022
7 ETHICS STATEMENT
Large neural network models are known to mimic and potentially emphasize biases found in the data, including biases that can negatively impact the individuals who interact with these increasingly ubiquitous models. Our work does not analyze this aspect of these models as it pertains to lifelong learning, but without such analysis unknown whether lifelong learning could further exacerbate negative societal biases exhibited in these models. We leave this important analysis to future work.
REPRODUCIBILITY STATEMENT
We include the code, and instructions needed to reproduce the main experimental results in the supplemental material. We specify all the training details including the hyper-parameters for our method and baselines ones. The relevant training details are provided in Sections 3, 4. 5 and Appendix A. Every experiment in the paper was run across random seeds and is reported with error bars (See Tables 1, 2 and Figure 5). We cite all datasets and baselines used as part of our experiments in Section 2. In the supplemental material, we provide a way to aggregate our new 15-dataset-NLP.
REFERENCES
Mohamed Abdelsalam, Mojtaba Faramarzi, Shagun Sodhani, and Sarath Chandar. lire: Incremental implicitly-refnne classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11038-11044 2021.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3366-3375, 2017.
Yaroslav Bulatov. Notmnist dataset. Technical report, Google (Books/OCR), 2011. URL http: hwwpsssswwwww.b.o..p..o...o..... http:/pp...../.1///.1 notnnisttttttoooommmmm.mmmmmmmmm dataset html.
Massimo Caccia, Pau Rodríguez, O. Ostapenko, Fabrice Normandin, Min Lin, Lucas Page-Caccia, Issam H. Laradji. I. Rish, Alexandre Lacoste, David Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. In NeurIPS, 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning, arXiv preprint arXiv: 10486 2019.
Zhiyuan Chen and Bing Liu. Lifelong machine learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1-200 2018.
Christopher Clark, Kenton Lee, Ming-
King-Wei Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936. 2019.
Cyprien de Masson d' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems, pp. 13122-13131, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
10

---

Under review as a conference paper at ICLR 2022
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Sayna Ebrahimi, Franziska Meier, Roberto Calandra, Trevor Darrell, and Marcus Rohrbach. Ad- versarial continual learning. In Andrea Vedaldi. Horst Bischof, Thomas Brox, and Van-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 386-402, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58621--
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre- training help deep learning? In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 201-208. JMLR Workshop and Conference Proceedings, 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. aggosscg meta-learing for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1135, JMLR. org, 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine Online meta-learning In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 1920-1930. PMLR, 09-15 Jun 2019. URL http: http:////ceeeeee...eeee././...ee mll press/v97/ finn1 9a. ht.l.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware aware minimization for efficiently improving generalization. International Conference on Learning Representations. 2021.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3 (4):128-133) 135, 1999.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4134-41 4143, 2019.
K. Hazelwood, S. Bird, Brooks, Chintala, U. Diril, D. Dzhulgakov, M. Fawzy, B. Jia, Y. Jia, A. Kalro, J. Law, K. Lee, J. Lu, P. Noordhuis, M. Smelyanskiy, L. Xiong, and X. Wang. Applied Machine Learning at Facebook: Datacenter Infrastructure Perspective. In 2018 IEEE Interna- lional Symposium on High Performance Computer Architecture (HPCA), pp. 620-629, 2018. doi: 10.1109/HPCA.20888 8.0009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 770-778, 2016.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918-4927, 2019.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
2020.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 208-339,400 2018
Wenpeng Hu, Qi Qin, Mengyu Wang, Jinwen Ma, and Bing Liu. Continual learning by using information of each class holistically. 2021.
Aman Hussain, Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. Towards
robust experimental framework and benchmark for lifelong language learning. 2021.
11

---

Under review as a conference paper at ICLR 2022
Khurram Javed and Martha White. Meta-learning representations for continual learning. arXiv preprint arXiv: 1eti:..99....88... 12588, 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httph: /opprr/ieee.......... net forum? Hd=HloyRlg@g.
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In Second AAAI Conference on Artificial Intelligence, 2018.
Najoung Kim, Song Feng, Chulaka Gunasekara, and Luis Lastras. Implicit discourse relation classification: We need to talk about evaluation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.
2020.
James Kirkpatriek, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnisl// 1998.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
Vincenzo Lomonaco, Davide Maltoni, and Lorenzo Pellegrini. Rehearsal -fhee continual learning over 2019.
small non-iid batches. arXiv preprint
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. arXiv preprint arXiv:2010.04495, 2020a.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the role training regimes in continual learning. arXiv preprint arXiv:2006.0695: 2020b.
Nasrin Mostafazadch, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. A corpus and cloze evaluation for deeper under- standing of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 839 849, San Diego, California, June 2016. Association for Computational Linguistics. doi: 1ttp///NN11-19990 10.16655////1161109999 URL httph:/t////
web. wwt.:e/...g..ttt..ooooooeoo/ee/e hwwgotoohoooooooooooooooooo /11 6 6-109..
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NIPS), 2011.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint 2r0iv:2088. /687, 2020.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 12410))135-1135, 2009.
12

---

Under review as a conference paper at ICLR 2022
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettl Deep contextualized word representations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computatioonal Linguistics: Human Language Technologies, Volume (Long Papers), pp. 2227-2237, 2018.
Matthew Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP-2019, pp. 7-14, 2019.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 67-81, 2018.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pp. 524-540.
Springer, 2020.
Rashmi Prasad, Bonnie Webber, Alan Lee, and Aravind Joshi. Penn discourse treebank version 3.0. In I.DC2019005. Philadelphia: Linguistic Data Consortium., 2019.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang. Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate transfer learning with pretrained language models: When and why does work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5231-52477 Online, July 2020. Association for Computational Linguistics. doi: 10011863/v1120000 20r100000..lmmmi.447777777 URL ht https: http:::/aalannthlloggeo www../.........mm........e...... 467.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technicul report, OpenAl, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 1910 10683, 2019.
Vinay V Ramasesh, Ethan Dyer. and Maithra Raghu, Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv preprint vrXiv:2007.074000 2020.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019. URL https: /openreieep....iie.. net/forum? diteeiiid--siccccccco
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 041666 2016.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 108, 2019.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contex- tualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empiriaal Methods in Natural Language Processing, pp. 3687-3697, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DI8-111 https: www. acl web....eb..... org/ heggtanoooogyy// /D18/111111111 1444.
URL
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. Commun. ACM, 63(12): 54-63, November 2020. ISSN 0001-0782. doi: 10.1 00.1145/3381811 URL https:: .org/ 831.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
13

---

Under review as a conference paper at ICLR 2022
Shagun Sodhani, Sarath Chandar, and Yoshua Bengio. Toward training recurrent neural networks for lifelong learning. Neural computation, 32(1):1-35, 2020.
Christian Stab, Tristan Miller, Benjamin Schiller, Pranav Rai, and Iryna Gurevych. Cross-topic argument mining from heterogeneous sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3664-3674, 2018.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650, 2019.
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? In Advances in neural information processing systems, pp. 640-646, 1996.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621-633 6-33, 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018.
Zirui Wang, Sanket Vaibhav Mehta, Barnabas Poczos, and Jaime G Carbonell. Efficient meta lifelong- learning with limited memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 535-548, 2020.
Alex Warstadt, Amanpreet Singh, and Samuel Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technolo-
gies, Volume (Long Papers). pp. 1112-1122 1122, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.00 07747, 2017.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3987- 3995. JMLR. org, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classifi- cation. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 1, pp. 649-657, 2015.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1):43-76, 2021. doi: 10. 10.1100/PROC.22200000555
14

---

Under review as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
Vision Experiments For all vision experiments, we the full ResNet- 18 He et al. (2016) archi- tecture, with the final linear layer replaced (the number of outputs corresponds to the total number of classes in all given tasks). During inference, only the subset of outputs corresponding the given task is considered. All images are resized to 224 x 224, and normalized with μ (0.485, 0.456, 0.406) and o (0.229. 0.224, 0.225). We used SGD optimizer with the learning rate set to .01 for all methods (we did hyperparameter search for both pre-trained and randomly initialized models and found the learning rate 0.01 resulted a good learning accuracy for both pre-trained and randomly initialized models). The batch size was set to 10 for the Split CIFAR-50 and Split CIFAR- 100 experiments and 64 for the 5-dataset experiments. The memory per class for ER was set to 1, and the À parameter for EWC was also set to 1.
For Stable SGD, we performed hyperparameter sweep over the parameters specified in the original paper, namely:
initial learning rate: [.25 (Spli-CIIAAR1OO-Ramddd) Split-CIFAR5O-Randde Sdata- Random), 11, _01 (Spii-CIFFAR000-PT, Split-CIFAROO-PT), .001 (5data-PT))
learning rate decay: [0.9 (Spli--CIFAR5O-Randdoo) Sdata-Random, Spli-CIFARR10-PTT) 0.85 (Split-CIFAR) 100-100-RO 100---omo-.,. Split-CIFARSO-PT), 0.8 (5data-PT))
batch size: [10 (all), 64]
dropout: [0.5 (5data-Random) 0.25 Salii-CIFAR10O-Randdde Split-CIFAR5O-Randdo Split-CIFAR IO0-PT, Split-CIFARSO-PT, Sdata-PT)]
NLP Experiments For most of the text classification experiments, we use the Transformer architec- ture based text encoder, DistilBERT-bass (Sanh et al., 2019) encode input. In single sentence text classification task, x, is an input sentence to be classified. In a sentence-par classification task, concatenation of xr and ui sentences separated by a [SEP] symbol is considered DistilBERT produces a contextual representation of each token in x; including a special beginning of the sentence token symbol [CLS]. We use the representation of the [CLS] symbol from model as features for a linear task classifier. We have a separate classifier for each task. We mainly set hyper-parane paper-parameters to default implementation from HuggingFace." We use Adam as our optimizer, set dropout 0.1, the base learning rate to 2e-5, batch size to 32 and the maximum total input sequence length after tokenization 128. For EWC, we set the regularization strength À to 100 (as this ended up with comparable LA across other methods) and for ER, following (Chaudhry et al., 2019), the memory per class per task is set to 1. For SAM, we set p = 0.02 for all models (random as well as pre-trained) on 5-dataset-NLP and 15-dataset-NLP For Split YahooQA we set p 0.001.
input 24-
B DATASETS
ImageNet pre-training corpus For a fair comparison between pre-trained and randomly initial- ized models, we wanted to explicitly control for and remove the overlap between pre- pre-training and downstream tasks. Publicly available ResNet models are pre-trained on ImageNet that overlaps with CIFAR-100 in terms of class labels. Therefore, we make sure that the subset of the ImageNet corpus we use does not have any visually and semantically overlapping classes with the CIFAR- 100 dataset. We use the publicly available (Abdelsalam et al., 2021) two-level class hierarchies for ImageNet, where semantically and visually similar labels are grouped under one super-category. We iterate over all CIFAR-100 labels and drop the complete super-category from ImageNet corresponding to each of these labels. For example, CIFAR- 100 contains a castle class and we have a building super-category in ImageNet that contains castle, palace, monastery, church, etc.. We remove all building-related labels from our pre-training dataset. In total, we remove 267 classes and pre-train the ResNet- 18-PT model on the remaining subset of the ImageNet dataset.
15

---

Under review as a conference paper at ICLR 2022
Table 5-dataset statistics. ITrainl, IDev,, (Testl denotes the number of examples in train, dev, test splits respectively. ILI denotes the number of classes for each task.
Dataset
ITrainl
iDevl
Test
[I]]
MNIST
51,000
9,000
10,000
10
notMNIST
15,526
2,739
459
10
Fushion-MNIST
9,574
1,689
1,874
10
CIFARTO
42,500
7,500
10,000
10
SVHN
62,269
10,988
26,032
10
Table 5: 15-dataset-NLP: Task/Dataset description and statistics. All tasks either single sentence or sentence pair classification. ITaiiil, IDevl, I'est denotes the number of examples in train, dev, test splits respectively. ILI denotes the number of classes for each tasks.
Task
Dataset/
Domain(s)
ITrainl
IDevl
iTestl
ILI
Metrics
Corpus
Text source(s)
Linguistic Acceptability
CoLA
Journal articles
7,695
856
1,043
2
Matthews correlation
& books
Boolean Question
BoolQ
Google queries, Wikipedia passages
8,483
944
3,270
Acc.
Answering
Sentiment Analysis
SST-2
Movie reviews
9,971
873
872
Acc.
Paraphrase Detection
QQP
Quora questions
10,794
4,044
4,043
Acc. &
A Categorization
YahonQA
Yahoo! Answers
13,950
4,998
4,998
10
Acc.
Review Rating Prediction
Yelp
Business reviews
12.920
3,999
3,998
5
Acc.
Event Factuality
Decomp
FactBank
10,176
4,034
3,934
2
Acc.
Argument Aspect Detection
AAC
Web documents
10.893
2,025
4,980
3
Acc. & Fl
Explicit Discourse
DISCONN8
Penn Discourse TreeBank
9,647
1,020
868
8
Acc. &
Marker Prediction
Question Answering
QNLI
Wikipedia
9,927
5,464
5,463
Acc.
NLI
Binary Sentence
RocBSO
Roc story, corpus
10,000
2,400
2,400
Acc.
Order Prediction
Natural Language
MNLI
speech, fiction, govt. reports
11,636
4,816
4,815
3
Acc.
Inference
Multi-choice Science
ScITAIL.
Science exams
11,145
1,305
1,304
2
Acc.
QA
Implicit Discourse
PDTB2L1
Penn Discourse TreeBank
13,046
1,183
1,046
4
Acc. & F1
Relation Classification
Emotion
Emotion
Twitter
9,600
2,000
2,000
6
Acc. & Fl
Delection
5-dataset-NLP consists text classification datasets (Zhang et al., 2015) from five diverse do- mains: (1) news article classification (AGNews, 4-way classification); (2) Yelp sentiment analysis (5-way classification); (3) Amazon sentiment analysis (5-way classification): (4) Wikipedia article classification (DBPedia, 14 14-way classification); and (5) question and answer topic categorization (YahooQA, 10-way classification). We follow the data processing procedure mentioned in (de Mas- d' Autume al., 2019) and have 115, 000 training examples and 7, 600 test examples per task.
16

---

Under review as a conference paper at ICLR 2022
One of the objectives of our work is to study the role of different pre-trained initializations in lifelong learning. To enable this study, we introduce 15-dataset-NLP, a novel suite of diverse tasks for lifelong learning. It consists of fifteen text classification tasks covering broad range of domains and data sources. Although there exists a setup with 4 tasks spanning datasets, 5-dataset-NI.P (de Masson d' Autume al., 2019), we show that introduced benchmark proves more challenging (see Table 2 and Section 3.3) than the previous setup for the Transformer models (e.g., DistilBERT, BERT, RoBERTa) considered in our study.
15-dataset-NLP benchmark consists of single sentence sentence pair classification tasks. We design our benchmark from existing tasks such that (1) the overall dataset includes various domains, (2) different tasks are (dis)similar each other, thereby, facilitating both transfer and interference phenomena. All tasks under consideration differ in dataset size (from 8.5k-400k), so for our experi- ments, only use between 8.5-14k training examples from each task. Lifelong learning from highly imbalanced data is an interesting problem, and we feel that our introduced benchmark can be used to investigate this problem as well. As our data is gathered from publicly available sources, for some tasks we do not have access to hidden test examples. In such cases, we consider dev examples as test split and sample examples from train split for validation We describe the tasks below and Table details the evaluation metrics and train/dev/test split sizes for each task.
1. Linguistic acceptability aims identifying whether the given sequence of words is grammatical sentence. The Corpus of Linguistic Acceptabilityy (CoLA) ((Warstadt et al., 2019) consists of English sentences annotated with their grammatical judgements. The data spans multiple domains, specifically books and journal articles.
2. Boolean QA is reading comprehension task of answering yes/no questions for a given passage. The Boolean Questions (BoolQ) (Clark et al., 2019) dataset consists of short pas- sages with yes/no questions about the passage. The questions are sourced from anonymous Google users and paired up with passages from Wikipedia articles.
3. Sentiment analysis a binary classification task identifying the polarity (positive/negative sentiment) of a given text. The Stanford Sentiment Treebank (SST-2) (Socher et al., 2013) corpus consists of sentences from Rotten Tomatoes movie reviews annotated with their sentiment.
4. Paraphrase detection aims at identifying whether two sentences are semantically equivalent. The Quora Question Pairs (QQP) corpus constitutes of question pairs from Quora' website annotated for semantic equivalence of question pairs.
5. Q&& categorization is a topic classification task of categorizing question and answer text pairs into existing topics. The Yahoo! Answers Comprehensive Questions and Answers (YahooQA) (Zhang et al., 2015) corpus contains data corresponding to the ten largest categories from Yahoo! Webscope program.
6. Review rating prediction is a five-way classification task of predicting the number of stars the user has given in review given the corresponding text. The Yelp (Zhang et al., 2015) dataset contains business reviews obtained from the Yelp Dataset Challenge (2015).
7. Event factuality prediction is the task of determining whether an event described in the text occurred. The factuality annotations from the Decomp corpus are recast into NLI structure and we use the modified dataset from Diverse NLI Collection (Poliak et al., 2018).
8. Argument aspect mining is concerned with the automatic recognition and interpretation of arguments (assessing the stance, source, and supportability for a given topie). The Argument Aspect Corpus (AAC) (Stab et al., 2018) has over 25,000 arguments spanning eight topics annotated with three labels (no argument, supporting argument, opposing argument). Stab et al. (2018) collected the data from web documents representing a range of genre and text types, including blogs, editorials. forums, encyclopedia articles.
9. The explicit discourse marker prediction task aims at classifying the discourse markers between sentences. Specifically, words like 'and', 'but', "because", 'if', 'when', 'also', 'while', 'as' mark the conceptual relationship between sentences (DISCONN8) and are
We plan to release sampled example indices for replicability of our results Qusluuss-QQusstionPPaaiss
17

---

Under review as a conference paper at ICLR 2022
considered labels for this task as discussed in (Prasad al., 2019; Kim al., 2020).
We use examples from the Penn Discourse TreeBank 3.0 marked for explicit discourse relationship for our experimentation.
10. Question-ans answering NLI (QNLI) is task adapted from the SQuAD by converting it into the sentence pair classification task (Wang et al., 2018). QNLI is binary classification task of detecting whether the context sentence contains the answer the question.
11. Binary Sentence Ordering (BSO) is binary classification task to determine the order of two sentences. This task is similar to pre-training objectives considered in recent works. We use Roc Stories (RocBSO) (Mostafazadeh et al., 2016) corpus for constructing the dataset for this task.
12. Natural language inference (NLI) is a three-way classification task of predicting whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The Multi-Genre Natural Language Inference (MNLI) (Williams et al., 2018) corpus consists of sentence pairs from different sources (transcribed speech, fiction, and government report) annotated for textual entailment.
13. Multi-choice QA is a reading comprehension task wherein given a passage and question, models need to pick up the right option out of provides ones. Khot al. (2018) cast the multiple-choice science exam questions into an NLI structure to convert them to the binary classification task. We use the SciTAIL. (Khot et al., 2015 8) dataset released by them for our experimentation.
14. Implicit discourse relation classification is common task of identifying discourse relations between two text spans or arguments. The Penn Discourse TreeBank 3.0 (PDTB3L1) (Prasad et al., 2019; Kim et al., 2020) contains hierarchical annotation scheme (top-level senses, fine-grained level-2 senses) and we use top-level senses comprising of four labels (expansion, comparison, contingency, temporal) for our experimentation.
15. Emotion detection is classification task of detecting the emotions from given text snippet. We use Emotion dataset (Saravia et al., 2018) which contains Twitter messages with six emotions: anger, fear, joy, love, sadness, and surprise.
B. TASK SEQUENCES
The task sequences for the Split CIFAR-50 and Split CIFAR-100 experiments were generated by randomly sampling classes without replacement for each task, similar to Chaudhry et al. (2019). Thus, the sequences were different for every random seed, but since same seeds, each method was trained and tested on the same 5 sequences.
each method with the
For Split YahooQA, we created 5 tasks by using disjoint groups of consecutive classes (e.g. {0, 1), (2,3).... ...). These tasks were than randomly ordered for each task sequence, and each method trained and tested using the same 5 random sequences.
For 5-dataset, we randomly select the following dataset orders:
Seq1 SWWN:--owwwwyy---eeoooowwwEETT--
Seq2 SVWNN-NWWWUS--HnnNNSSSHEEHHEIII- Seq3 UIZA:-O-SSWW----nnnNSSS---aaaa-- Seq4 TuuNNN----aaaon--SSSSS----------
Seq5 CIAAR-O--NNNTT---UUUNSSSSVVVVV--
For 5-dataset-NLP. we randomly select the following dataset orders (first 4 are consistent with
(de Masson d' Autume et al., 2019)):
Seql Yelp Ytg---Glwwws-PPPededi---aooo--o-
Seq2 DBPedii-Phlooo---AANWWWWAAAA----
Seq3 Reh-YYalnoA--AAMmmo---DDDdddda--
Seq4 McNuwss-eepp-Annnooo--Vloo------
18

---

Under review as a conference paper at ICLR 2022
For 15-dataset-NLP, we randomly select and use the following 5 orders:
Seq5 Sut.uuueSSSS.aaaaoooooooAAAAAAA-
C TASK-SPECIFIC RESULTS
In order to understand the evolution of task-specific performance during continuous training, we visualize the task-specific results in Figures 3 and 4. Specifically, compare the performance of pre-trained and randomly initialized ResNet- 18/ DistilBERT, for the first three tasks sequence, across five random task ordering, when evaluated 5-dataset/ 5-dataset-NLPP (diverse tasks). In general, we see that both models start with approximately equal task accuracy (except for CIFAR-10), but pre-trained initialization leads lesser forgetting than randomly initialized models (consistent with our observation in Figure for Split YahooQA), Moreover, given the heterogeneous nature of the downstream tasks, we see that performance gains (in terms of forgetting) from pre-trained initialization vary across different tasks.
5-dataset-NLP For example, in the case of DBPedia (Figures 3c, 3d, 30) and AGNews (Figures 3b, 3f, 3j) datasets, see mre-trained DistilBERT undergoes little to almost forgetting. One plausible explanation for these results is that both datasets are for the article classification tasks, DBPedia is Wikipedia article classification (14 classes) and AGNews is news article classification (4 classes), and share similar domains with the pre-training corpora (Wikipedia and Books). On the other hand, a significant forgetting in the case of Yelp (Figures 3a, 3g. 3k) and Amazon datasets (Figure 31). Both of these datasets are review sentiment classification tasks (5 classes). We know that the reviews domain (noisy text from Yelp.com and Amazon.com) is less similar with the pre-training corpora (clean text from Wikipedia and Books), and might be one of the reasons behind the drop performance. Further, note that as we train on the sequence of tasks, we expect to see positive/ negative transfer from related/ unrelated tasks. For example, we see that the performance on Yelp improves significantly after training on Amazon (Figures 3a, 3g, 3n), demonstrating an example of positive transfer from the related task.
5-dataset Here, we report that the forgetting is more severe for SVHN (Figures 4a, 4d, 4h) and CIFAR-10 (Figures 4g, 41, 4m) compared MNIST (Figures 4e, 4n), notMNIST (Figures 4b, 4f, 4i, 4o). Although SVHN and MNIST both are digit recognition tasks, we believe that the realistic nature (house numbers in Google Street View images) SVHN images makes them more susceptible to forgetting, even in the case of pre- pre-trained ResNet-18 18 models.
19

---

Under review as a conference paper at ICLR 2022
100
Intekatee
100
Accurccy
20
Intulliee
AGhows
DéPodia Amazon
YanooOA
AcNows
DBPedoo
Amazon
Yshoo
Amazon Tesc Sequuence
YahooGA
Tase Sequancee
Takk Sequercce
(a) Yelp (Seq1)
(b) AGNews (Seq1)
(c) DBPedia (Seq1)
1000
100
Accurey
randor
precalned
randon
++ srwcsited
8pPedia
YanooOA
MGNews
Amazon
Pahos0A
AGNews
Amazon
Ydl
AcNews
Amazon Task Siquarncc
Task Sequance
Task Sequence
(d) DBPedia (Seq2)
(c) YahooQA (Seq2)
(f) AGNews (Seq2)
Intodrarce
100
Indta pation
rencom
prectalnet
Acarney
Aecorwyy
PahowgA
Aranon
Task Scquence
Toak Sequance
Task Soquence
(g) Yclp (Seq3)
(h) YahooQA (Seq3)
(i) Amazon (Seq3)
1000
Inttlbatin
100
Intill cation
Bibaloaconn
Accurety
Achows
Yshoo
DiPedis
Yahoo
DBPeda
Task Sequence
Teak Sezuance
Task Sequuince
(i) AGNews (Seq4)
(k) Yclp (Seq4)
(I) Amazon (Scq4)
100
Fitis bysan
rendor
randem
Acarney
Intokkttie
middon
pwosied
Amazon
Banedl
Tick Soouance
Task Sequence
Task Soquance
(m) YahooQA (Seq5)
(n) Yelp (Seq5)
(o) DBPedia (Seq5)
Figure 3: Evolution of task accuracy during sequential training on 5-dataset-NLP. We compare the performance of pre-trained and randomly initialized models, for first three tasks in a sequence, across five different random task orderings (Seql....Seq55). We see that both models start with approximately equal task accuracy, but pre-trained initialized models undergo lesser forgetting than randomly initialized models.
20

---

Under review as a conference paper at ICLR 2022
100
Hisken
100
Accurccy
Accorddy
Accurecy
55
20
SVHN
1otHMSST FNNST OFAR-10
MNIST
noTRRIST
F.RMIST
CIAAR 10
MNST
9 HMMST
OFAR 10
MNIST
Time Sequence
Tase Sequarcce
Tank Seqquence
(a) SVHN (Seq1)
(b) notMNIST (Seq1)
(c) Fashion Minioon---ITTTT-- (Seq1)
1.00
In Uudrationn
100
Accurey
Intokiotve
randor
swossied
SVHN
noMMINST F-MNIST OFAR-10
'MIIST
NOTMNIST
FMNIST
CIFAR-10
ROEMMST
S-
OFAR-10
Tasic Seqquance
Sequance
Tan Sequence
(d) SVHN (Seq2)
(c) MNIST (Seq2)
(f) notMNIST (Seq2)
1.00
100
FVis pww-tratee
randim
rendor
Acarney
Aeanney
Aecowey
60
811A8-10
RUMMIST
Tasi Sequance
Task Soquence
Tisk Soquznce
(g) CIFAR-10 (Seq3)
(h) SVHN (Seq3)
(i) notMNIST (Seq3)
100
100
1000
Accurety
Aocuracy
Acrurecy
SOMMSST
MNIST
P-SESST
CIFAR-10
SVAN
8
Tinic Sequerce
Task Sequence
Tisk Sequance
(i) notMNIST (Seq4)
(k) Mashion-MNIST (Seq4)
()) CIFAR-10 (Seq4)
Intolmacce
100
rendor
pre-tainee
Acarney
Acconnty
Accorery
60
8PAS-10
F-NNIST
WWIST
P-NNIST
REMMST
Tisk Soquance
Task Sequence
Task Soquance
(m) CIFAR-10 (Seq5)
(n) MNIST (Seq5)
(o) noLMNIST (Seq5)
Figure 4: Evolution of task accuracy during sequential training on 5-dataset. We compare the performance of pre-trained and randomly initialized models, for first three tasks in a sequence, across five different random task orderings (eqq). We see that both models start with approximately equal task accuracy (except for CIFAR-10). but pre-trained initialized models undergo lesser forgetting than randomly initialized models.
21

---

Under review as a conference paper at ICLR 2022
D SHARPNESS
The matrix A E x used for projecting the parameters onto a subspace is randomly sampled and then normalized row-wise. Since this matrix is very large, the computation of the pseudo inverse A+ (required for calculating the bounds in Equation 3) is very memory intensive and unstable. Instead, we directly calculate A+x x by finding the least squares solution to Ab = x. To find the maximum referenced in Equation 4, we the L-BFGS-B B algorithm. We set the maximum number of iterations for the algorithm to 10, and to speed up computation, we directly provide the gradients along with the loss the algorithm, instead of using the default 2-point finite difference gradient estimation.
For ResNet-18 (n 11M), set p 100. However, for DistilBERT (n 66M) when we set p = 100, we notice extremely small values for the sharpness metric. With the increase in the number of parameters, n, we should ideally increase random subspace projection dimension p. Setting larger p(> 100) values for DistilBERT, however, leads to memory issues relating to allocating space for A and computing the bounds (even with the more efficient method discussed above). So instead of evaluating the sharpness metric in a random manifold, we perform the maximization in the entire space TR. (basically setting A I..). According to Keskar et (2017), when is small enough and A = I.. the sharpness metric in Equation 4 relates to the largest eigenvalue of fx)(). In Table 7, we report sharpness values for DEstilBERT on 5-dataset-NLP and Split YahooQA datasets for E {5e 1e_,, "). We see that values in the case of pre-trained models (w/ PT) are lower compared to randomly initialized models (w/o PT), thereby, validating the relative flatness of the task minima in the case of pre- pre-trained models.
Table 6: Average sharpness (lower is flatter) of minima across tasks in a 100 dimensional random subspace. ResNet- 18-PT (w/ PT) has lower sharpness in comparison to BesNet-18-B (w/o PT). Pre-training reduces the sharpness of minima for each task in training by an order of magnitude.
5 x 10
10
w/o PT
w/ PT
w/o PT
w/ PT
Split CIFAR-100 2.26(+0.58)
0.10(+0.02)
5.89(+1.33)
0.24(+0.06)
Table 7: Average sharpness (lower is flatter) of tasks minima. DistilBERT-PT (w/ PT) reduces the sharpness in comparison to DistilBERT-R (w/ PT).
5 x 10
10
5 x 10
PT
w/PT
PT
W/PT
PT
w/PT
5 5-dataset-Ni.
32.67 + 1.17
18.27111
213.61 + 11.46
125.97 + 10.49
596.82 :: 13.70
552.09 +: 17.28 422.85 + 44.31
Splii YahooQA
10.41 4 0.39
8.77 0.44
53.23 = 7.02
43.03 + 4.21
545.06 6.40
E LOSS LANDSCAPE
E LINEAR MODEL INTERPOLATION
Ideally, to reduce forgetting, as the model sequentially trains on tasks, its loss on previous tasks should change minimally. This would be satisfied if the loss surface between model checkpoints were flat. To visualize this, we linearly interpolate between Wi and the other checkpoints of the model, tracking the test loss on task I and task 2. This can be interpreted as viewing a slice of the contour plots shown in Figure along the line that connects Wa to each checkpoint. The results are shown in Figure 5, with first row for task and second row for task 2. The pre-trained plots are shown in hues of blue, and the random plots shown in hues of red. These plots show that the pre-trained initialized models experience a much more gradual increase in the loss compared to the randomly initialized models, even when interpolating to checkpoints created after training on several tasks.
We used the implementation provided by scipy at wttp:////lasssoooooeeee///////ee hhtl
22

---

Under review as a conference paper at ICLR 2022
12.1
03
2
66
0.8
16
3.2
34
66
34
(a) 5-dataset (T1)
(b) Split CIFAR-50 (TI)
(c) S-
(d) Split YahooQA (TI)
03
32
3.4
0.6
0.8
10
30
6.2
0.
0.8
10
84
38
(c) 5-dataset (T2)
(f) Split CIFAR-50 (T2)
(g) 5-datasct-NaL NLP (T2) (h) Split YahooQA (T2)
Figure 5: Loss interpolation plots for each dataset. Blue is pre-trained models, red is randomly initialized models. We interpolate between the checkpoint after task (TI) task 2 (T2) to the checkpoint after every other task, tracking the loss in the process. In general, the loss landscape is flatter along these paths for pre-trained initialized models compared randomly initialized models.
Moreover, these results hold for task as well, thereby verifying that pre-trained initialized models lead to flatter task minima for subsequent tasks.
48.0
45.0
48.0
48.0
36.0
36.0
3.0
36.0
36.0
2.0
24.0
30.0
2.0
2.0
24.0
30.0
1.0-
18.0
1.0
180
1.0
18.0
1.0
18.0
6.0
49
6.0
0.0
86
6.0
0.0
5.0
0.0
2.0
4.0
6.0
0.0
5.0
0.0
5.0
(a) Finetune (TI)
(b) Finetune + SAM (TI)
(c) ER (TI)
(d) SAM (TI)
48.0
48.0
42.0
42.0
42.0
36.0
36.6
96.0
36.0
2.0
24.0
24.0
2.0
24.0
24.0
10-
18.0
1.0
12.0
12.
12.0
12.0
12
4
2
0.00
0.0
(e) Finetune (T2)
(f) Finetune + SAM (T2)
(g) ER (T2)
(h) ER SAM (T2)
Figure 6: Loss contours for task (TI) and task (T2) of Split CIFAR-50. The top row visualizes loss contours for task 1 where W1, W2, W3 minima obtained after sequential training tasks 1, 2, and 3, respectively. Similarly, the bottom row visualizes loss contours for task 2 after sequential training on tasks 2, 3, and 4. All of the above models start with random weights. SAM (Finetune SAM, ER + SAM) leads to wide task minima compared to Finetune and ER methods.
E.2 Loss CONTOURS
In this section we present loss contours for task 1/ task for all task sequences (refer to Section B.1 for more details) for 5-dataset-NLP, Split YahooQA, Split CIFAR-50, and 5-dataset. In line with our observation from the sharpness and linear model interpolation analyses, pre-trained initialized models lead to flatter task minima for subsequent tasks well.
23

---

Under review as a conference paper at ICLR 2022
E.2.1
Loss CONTOURS: SAM AND ER
We plot loss contours for Task 1/ Task 2 of Split CIFAR-50 (Figure 6) and 5-dataset (Figure 7), under continual training from randomly initialized weights, and compare them across four different methods: Finetune, Finetune + SAM, ER, and ER SAM. We notice that SAM (Finetune SAM and ER + SAM) leads to wide task minima (Task Task 2) across both datasets as compared to Finetune and ER methods. Moreover, from Table 1, we see that for Split CIFAR-50, Finetune SAM (14.95), ER + SAM (16.88) undergoes lesser forgetting than Finetune (23.6 68) and ER (20.63) methods. These results convincingly demonstrate the effectiveness of SAM when used with vanilla SGD and/ or ER methods. Similarly, we see that for 5-dataset, ER SAM (27.32) undergoes lesser forgetting than Finetune SAM (40.92), which in turn improves loss contours between Finetune and ER methods and do not notice any stark difference terms of flatness. However, in the presence of SAM, qualitatively we that ER SAM (Figures 7d, 7h) leads to a flat loss basin in comparison to Finetune + SAM (Figures 7b, 7f).
Finetune (51.51). Next, we compare the
2.0
18.0
2.6
186
15.0
2.0
2.0
1.5
12.6
1.9
9.0
1.0
9.0
1.0
9.0
1.0
0.5
6.0
6.0
0.5
6.
6.0
3.0
3.0
3.0
3.0
(a) Finetune (T1)
(b) Finetune SAM (T1)
(c) ER (TI)
(d) ER SAM (T1)
2.0
2.0
24.0
24.0
24.0
24.0
1.0
18.0
10
15.
1.0
18.0
1.-
18.0
12.0
12.0
12.0
12.0
22
0.0
21
8/2
24
(e) Finetune (T2)
(f) Finetune + SAM (T2)
(g) ER (T2)
(h) ER SAM (T2)
Loss contours for SVHN (T1) and MNIST (T2) of 5-dataset. The top row visualizes loss contours for SVHN where W1, W2, W3 minima obtained after sequential training SVHN, MNIST, and nonMNIST, respectively. Similarly, the bottom row visualizes loss contours for MNIST where W2, W3, W4 are minima obtained after sequential training on tasks MNIST, nonMNIST, and Fashion-MNIST.. All of the above models start with random weights. SAM (Finetune + SAM, ER + SAM) leads to wide task minima compared Finetune and ER methods.
Figure 7:
24

---

Under review as a conference paper at ICLR 2022
E.2.2
Loss CONTOURS: 5-DATASET-NUA
24
27
3.0
4.0
21
4.
27
24
4.0
30
2
2A
4.2
24
21
20
20
15
2
6.
2
60
5/5
1910
0.0
5-3
10.0
0.0
0.
50
10/3
60
5.0
33.0
00
5.0
10.0
Seq w/o PT
(b) Seq
w/o PT
(c) Seq 3 w/o PT
(d) Seq 4 w/o PT
(e) Seq 5 w/o PT
24
d+
24
2.4
d'
24
2,7
30-
200
20
12
15
20
1.5
12
15
12
20
15
12
2
//
(f) Scq 1 w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Seq 4 w/ PT
(i) Seq 5 w/ PT
Figure 8: Loss contours for Task on 5 task sequences of 5-dataset-NLP. Each contour shows the location of the model parameters after training sequentially Task 1 (w1), Task 2 (w2), Task 3 (Wy). The top row shows contours for randomly initialized models (w/o PT) and the bottom row shows contours for pre-trained initialized models (w/ PT).
18
20
20
10
0.0
0.
50
20
0.2
5.3
06
16.2
5.0
16.0
80
(a) Seq w/o PT
(b) Seq 2 w/o PT
(c) Seq 3 w/o PT
(d) Seq 4 w/o PT
(c) Seg 5 w/o PT
20
5.6
27
5.0
27
4.0
244
4.2
14
14
15
15
12
1S
20-
15
0.0
16
8%
2+
46
n4
03
66
(f) Seq 1 w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Seq 4 w/ PT
(i) Seq 5 w/ PT
Figure 9: Loss contours for Task 2 on 5 task sequences of 5-dataset-NL.P. Each contour shows the location of the model parameters after training sequentially Task 2 (w2), Task 3 (wa), Task 4 (Wa). The top row shows contours for randomly initialized models (w/o PT) and the bottom row shows contours for pre-trained initialized models (w/ PT).
25

---

Under review as a conference paper at ICLR 2022
E.2.3 Loss CONTOURS: SPLIT YAHOOQA
5.0
4.-
40
16
75
05
05
13
63
12
10.0
5.0
50
15.0
0.
100
68
50
15.0
0.
5.6
10.0
(a) Seq w/o PT
(b) Seq w/o PT
(c) Seq
w/o PT
(d) Seg 4 w/o
(e) Seg 5 w/o
3.0
66-
#
5.
09
6a
18
08
84
08
54
16
46
64
06
66
15
2.-
05
2.2
83
20
05
53
13
0.
6.
03
6.3
0.-
12
06-
62
6.3
41
62
06
02
60
(f) Seq w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Seq w/ PT
(i) Seq 5 w/ PT
Figure 10: Loss contours for Task 1 5 task sequences of Split YahooQA. Each contour shows the location of the model parameters after training sequentially on Task (w1), Task 2 (W2), Task 3 (Wa). The top row shows contours for randomly initialized models (w/o PT) and the bottom row shows contours for pre- pre-trained initialized models (w/ PT).
11
28
29
08
05
40
63
63
79
03
83
16
0c
64
36
64
15
26
13
0.0
0.
60
30
62
(a) Seq w/o PT
(b) Seq 2 w/o PT
(c) Seq 3 w/o PT
(d) Seq 4 w/o PT
(e) Seq 5 w/o PT
122
1.2
1.2
1.
2.2
111
66-
1.1
4'
1.
11
5.1
09
09
ta
4.0
30
46-
08
04
4.0
08
6a
64
06
6.4
2.0
2.-
2,2
20-
2%
d5
0
20
10.0
18.0
5.2
(f) Seq 1 w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Seq 4 w/ PT
(j) Seq 5 w/ PT
Figure 11: Loss contours for Task on 5 task sequences of Split YahooQA Each contour shows the location of the model parameters after training sequentially on Task 2 (w2), Task 3 (wa), Task 4 (w4). The top row shows contours for randomly initialized models (w/o PT) and the bottom row shows contours for pre-trained initialized models (w/ PT).
26

---

Under review as a conference paper at ICLR 2022
E.2.4 Loss CONTOURS: SPLIT CIFAR-50
4.
380
310
210
010
380
010
3.0
2.0
120
24
126
2.0
10
66
64
80
90
48
8.0
(a) Seq w/o PT
(b) Seq w/o PT
(c) Seq 3 w/o PT
(d) Seq 4 w/o PT
(e) Seq 5 w/o PT
10-
350
15.0
30
15.2
15.0
150
320
12
12.5
120
10
0.
10
0.6
60
30
0.
10
3.
255
25
500
25
75
(f) Seq w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Soq
(j) Seq w/
Figure 12: Loss contours for Task 1 on 5 task sequences of Split CIFAR-50 with 5 epochs of training
on each task. Each contour shows the location the model parameters after training sequentially on Task (wi), Task 2 (w2), and Task (w3). The top shows contours for randomly initialized models (w/o PT) and the bottom row shows contours for pre-trained initialized models (w/ PT).
4.
550
30-
520
120
120
3.0
10
40
4.
64
48
60
10
3.0
34
10
5.0
50
(a) Seq w/o PT
(b) Seq w/o PT
(c) Seq 3 w/o PT
(d) Seq 4 w/o PT
(c) Seq 5 w/o PT
350
/0
usa
iso
510
200
3.0
200
2.
46
4.
1.0
60
1.0
40
60
60
100
63
10
10
30
30
33
23
5.0
25
(f) Seq 1 w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Seq w/ PT
(j) Seq 5 w/ PT
Figure Loss contours for Task 2 on 5 task sequences of Split CIFAR-50 with 5 epochs of training on each task. Each contour shows the location of the model parameters after training sequentially Task (W2), Task 3 (W3), and Task (W,). The top row shows contours for randomly initialized models (w/o PT) and the bottom shows contours for pre- mr-itrieed initialized models (w/ PT).
27

---

Under review as a conference paper at ICLR 2022
E.2.5
Loss CONTOURS: 5-DATASET
310
276
273
20
840
210
25
21.
210
20
243
240
241
1.-
350
120
150
130
123
150
20
35.0
10
60
47
6.8
41
1.
(a) Seq
w/o PT
(b) Seq 2 w/o PT
(c) Seq 3 w/o PT
(d) Seq 4 w/o PT
(e) Seq 5 w/o PT
240
240
218
156
1.0
15
21.8
15.5
150
21.
120
12.4
12.
15
120
/'
21
1.4
(f) Seq 1 w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Seq 4 w/ PT
(i) Seq 5 w/ PT
Figure 14: Loss contours for Task on 5 task sequences of 5-dataset with 5 epochs of training on
each task. Each contour shows the location of the model parameters after training sequentially on Task (W1), Task 2 (W2), and Task (Wy). The top row shows contours for randomly initialized models (w/o PT) and the bottom row shows contours for pre-trained initialized models (w/ PT).
210
360
25
215
240
2.0
21.
243
4.0
300
210
240
215
26-
310
150
10-
150
126
1.
124
20
2300
125
30
8
10
62
8
60
(a) Seq
w/o PT
(b) Seq 2 w/o PT
(c) Seq 3 w/o PT
(d) Seq 4 w/o PT
(e) Seq 5 w/o PT
3.
340
2.6
26.2
210
15
250
1..
120
12.2
120
24
20
25
(f) Seq 1 w/ PT
(g) Seq 2 w/ PT
(h) Seq 3 w/ PT
(i) Seq 4 w/ PT
(i) Seq 5 w/ PT
Figure 15: Loss contours for Task on 5 task sequences of 5-dataset with 5 epochs of training on each task. Each contour shows the location of the model parameters after training sequentially on Task (W2), Task 3 (w3), and Task (w4). The top row shows contours for randomly initialized models (w/o PT) and the bottom row shows contours for pre-trained initialized models (w/ PT).
F ADDITIONAL. RELATED WORK
Transfer learning from generic pre-trained models has enabled significant recent progress in ML (Zhuang et al., 2021). This trend started the CV field with the ImageNet dataset (Deng et al., 2009). Transfer learning in NI.P has witnessed its own "ImageNet revolution" where large models pre-trained on self-supervised tasks have shown impressive results across many language understanding tasks (Peters al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2019; Liu et al., 2019).
Lifelong learning approaches focus the idea of mitigating the catastrophic forgetting phenomenon and can be categorized into three groups: (1) Regularization based approaches that augment the loss function with extra penalty terms preventing important parameters learned previous tasks from significantly deviating while training on the new task (Kirkpatrick et al., 2017: Zenke et al., 2017); (2) memory-based approaches that augment the model with episodic memory for sparse experience
28

---

Under review as a conference paper at ICLR 2022
replay of previous task examples (Lopez-Pazz Pazez-PPa&ee & Ranzato, 2017; Chaudhry al., 2018; Wang et al., 2020) (3) network expansion- based approaches that dynamically expand the network based upon new tasks (Rusu et al., 2016; Aljundi et al., 2017; Sodhani et al., 2020). We analyze eegularization-bssee and memory-based approaches in this work.
Meta-learning involves creating models that learn to learn over time. Several works have applied meta-learning approaches to the task of lifelong learning (Riemer et al., 2019; Finn et al., 2019; Javed & White, 2019; Wang et al., 2020). Caccia et al. (2020) propose two-phase continual learning scenario where the first phase is pre-training (using MAML (Finn et al., 2017)) and the second phase involves continual deployment with task revisiting. They make the point that many scenarios (Lomonaco et al., 2019), it would be unrealistic to deploy agents with no pre- training in lifelong learning setting. Whereas some of these works do use pre-trained initializations for their models, many do not, and none have extensively studied the effect of pre-training on lifelong learning.
29


Reviews:
[{"review_id": "wx_EPFxkBI", "reviewer": "Reviewer_UX5R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Empirical analysis papers like this one, aiming to identify and understand best training practices, are sorely needed in machine learning. Authors have made substantial efforts in building multiple datasets, comparing different settings, and trying to understand what their findings mean. The proposed loss constraint solution appears to work well, and improves overall model performance. \n\nDespite the large amount of work carried out, I have several concerns about the proposed work and its conclusions.   \nFirstly, the main conclusions of the different experiments are not providing significant novel insights. It is well known that model pre-training provides significant performance boosts and robust transfer training solutions. It is expected that pre-training would yield superior performance in datasets somewhat similar to the pre-training dataset. Furthermore, comparing model weights and loss landscape between a robust pre-trained model and a randomly initialised, potentially not converged model does not seem particularly fair. The large gap between learning accuracies in certain settings highlights this issue. The flat loss landscape is a more interesting, yet not novel, conclusion as acknowledged in section 6. \n\nOne key limitation is that, besides analysing forgetting, the experiments proposed here provide little insights with regards, specifically, to the task incremental setting. For example, task dissimilarity and ordering could have an impact on how beneficial pre-training is. What happens if the first task is highly different from the pre-training dataset?  What is  the impact of the choice of the pre-training dataset? This was briefly discussed when comparing different BERT models, but the main conclusion was that larger and more diverse pre-training datasets are better, which is also a well known result. Authors had a great opportunity to analyse the impact of the dissimilarity between the pre-training and task specific datasets on forgetting with their 5(+)-datasets settings. It would have been very interesting to study the impact of ordering tasks differently depending on their distance to the pre-training dataset. \n\nFurthermore, reported results only provide overall accuracy and forgetting. Task specific results would provide much more insights, especially when dealing with heterogeneous tasks. Are observed differences consistent across tasks? Are there tasks where pre-training is detrimental, or less beneficial?  \n\nThe presentation of the paper could be improved greatly. There is a lot of content, which makes the paper extremely condensed with a lot of essential information relegated to supplementary material such as the discussion on related work. The issue is also reflected in the presentation of the results. Different experiments are reported in the same table, which is referenced multiple times at different points in the paper. This leads the reader to wonder where to look in the table, and what the SAM method is, not knowing that it will be introduced at the end of the paper. This creates confusion. \n\nIn addition, authors provide very little information on datasets. It is not stated in the main paper why these specific datasets are chosen, whether their order was curated, and how tasks were constructed. Again, a lot of the essential information is relegated to the supplementary material when this constitutes a key contribution. \n\nSimilarly, while there exist a large set of methods aiming to address the task incremental learning problem, authors selected 4 without clear justification. Why were these methods selected over others?\n\nMinor comments:\n\nHow were semantically similar classes between ImageNet and CIFAR determined?\n\nPlease double check citation formats. Some were not accurately formatted in the text. \n\nPlease add line number to facilitate the review process and point out typos.\n", "summary_of_the_paper": "The authors present a comparative study on the impact of using pre-trained models on task incremental learning problems. They split computer vision and NLP datasets into multiple tasks and evaluate performance between pre-trained and randomly initialised models. They conclude that task incremental learning benefits from pre-training due to the flatter loss landscapes, and propose to use an optimisation constraint to learn better models. ", "summary_of_the_review": "In summary, authors provide a study with a lot of potential, but unfortunately the paper in its current state does not provide substantially novel conclusions or innovations. While the proposed experiment are substantial and do provide additional confirmations, main conclusions are mostly well known (pre-training achieves better performance, larger datasets and models are better), or already discussed in the lifelong learning community (flat loss landscape).  The proposed SAM optimisation constraint is a direct application of another work, and its interaction with different continual learning methods, which often rely on optimisation constraints, is not discussed. \nFinally, the paper is presenting a lot of content which is severely hurting presentation and readability, with essential elements relegated to supplementary material or missing. I would recommend refocusing the work or submitting in settings where page limit is not a concern. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ul6uFydd9Fw", "reviewer": "Reviewer_xCMa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n - This paper takes a different and interesting approach to tackle one of the most important issues of lifelong learning i.e., catastrophic forgetting. As pre-training of large models on large diverse datasets has become a norm, this approach looks more practical to me than fine-grained changes in rehearsal-based and regularization based approaches.\n - The main hypothesis that pre-training helps in reducing catastrophic forgetting over a sequence of datasets/tasks has been supported theoretically and empirically with diverse quantitative results and qualitative analyses.\n - To strongly demonstrate the validity of main hypothesis, this paper provides comprehensive experiments, including both quantitative results and qualitative analysis: (1) using prominent approaches of lifelong learning in recent years consisting of rehearsal based (A-GEM, ER) and regularization based (EWC) approaches, (2) for Computer Vision (CV) and Natural Language Processing (NLP) applications, (3) on homogeneous (Split CIFAR-100, Split Cifar-50, Split YahooQA) and diverse (5-dataset, 5-dataset-NLP, 15-dataset-NLP) sequential datasets.\n - For example, it has been demonstrated that a simple fine-tuning (without regularization or rehearsal) of a pre-trained model on a sequence of diverse tasks (5-dataset-NLP) undergoes significantly less forgetting than rehearsal-based ER method with randomly initialized weights, which begs the question that if instead of specifically focusing on forgetting aspect of lifelong learning would it be more useful to learn generic features on a larger scale?\n - In case of diverse sequential datasets (5-dataset, 5-dataset-NLP) simple fine-tuning of a pre-trained model performs worse than ER method which shows that pre-trained models are susceptible to forgetting when learning on diverse tasks. However, larger models and models pre-trained with diverse pre-trained corpora undergoes less forgetting which shows that diversity of pre-trained corpus is a relevant factor during lifelong learning.\n - Experiments have been performed with multiple different sequences of tasks in each dataset to test robustness of the proposed approach.\n - Loss contour plots and Sharpness metric evaluations further validates the hypothesis that pre-trained models have bias for wider loss basins for fine-tuning tasks.\n - It has been shown that in addition to pre-trained weight initialization, it further helps in reducing forgetting if sharpness of the loss basin is taken into account as an additional training objective via Sharpness-Aware Minimization (SAM) training procedure. Experimental results confirm that SAM eases catastrophic forgetting and improves the performance of underlying model whether it is \"fine-tuning of a pre-trained model\" or \"episodic replay (ER)\".\n\nWeak points:\n - Following the setup of experiments of ER method with and without SAM training procedure, it would be interesting to see the experiments with model initialized with pre-trained weights in ER method to see if fine-tuning of pre-trained weights and rehearsal of examples from previous tasks complement each other or not.\n - To see the effects of SAM optimization procedure qualitatively, I would recommend to plot loss contours for Task 1 (as shown in Figure 2) for Finetune + SAM models.\n - Also, it would be interesting to see loss contours for Task 1 using models trained with ER method with randomly initialized weights to compare Finetune and ER methods qualitatively.", "summary_of_the_paper": "This paper explores the catastrophic forgetting in lifelong learning from the perspective of model initialization. It further shows that large pre-trained models have an implicit bias towards wider loss basin when fine-tuning on sequential tasks which ultimately results in less forgetting of tasks when training on subsequent tasks. Furthermore, by explicitly seeking wider loss basins during lifelong learning training as an additional optimization objective further reduces the forgetting regardless of the lifelong learning algorithm and base model used.", "summary_of_the_review": "Although there is no novel algorithm or model introduced in this paper, but the direction it explores to minimize catastrophic forgetting in lifelong learning methods via pre-trained weight initialization is interesting and practical as large pre-trained models have become a norm because of the wide range of diverse knowledge they can capture. Even though the Finetune method is not the best performing method in all of the experiments (however it outperforms in most of the experiments), the key conclusion I can take form this paper is that instead of completely focusing on improving upon fine-grained details of forgetting aspect of lifelong learning, the focus should also be on learning more general representations as they appear to result in robust lifelong learning.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "08a_xLngqSl", "reviewer": "Reviewer_6rM5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n- The paper is very well-written and easy to follow.  \nThe paper shows that pre-weights can alleviate catastrophic forgetting and suggests an explanation of it - pertained weights lead to wider minima. With that, the authors show that explicitly seeking flat basins during sequential fine-tuning results in even less forgetting. Again, the coherent structure and detailed analysis are a plus. \n\nConcerns\n-  Missing baselines.  Various methods have been proposed in the continual/lifelong learning line of work. Although it is hard to exhaustively evaluate all possible methods, some popular approaches are still missing, such as iCaRL and PackNet (parameter isolation-based approaches). Therefore, the authors may group their baselines based on method types, similar to the structures in the survey paper (e.g., https://arxiv.org/pdf/1909.08383.pdf). \n- Limited setting. The paper focuses only on the Tabula rasa case, where incremental learners trained from scratch and do not require pretraining on large labeled datasets. A more realistic setting is to learn from pre-trained models. For example, a model is trained on the first 50 classes of CIFAR-100 and then incrementally learns the remaining classes. Is the paper trying to suggest CL benchmarks should use pre-trained weights instead of training from scratch? \n- The CV datasets are relatively small.  The methods are evaluated on CIFAR datasets while in the literature (again checking the survey paper above),  larger datasets like ImageNet, iNaturalist, are used for evaluation.  Can the authors provide the analysis there? \n- Other evaluation metrics, like Backward/Forward transfer? \n", "summary_of_the_paper": "This paper provides an empirical study on pretraining for lifelong learning. The paper suggests that the generic pre-training implicitly alleviates the effects of catastrophic forgetting in lifelong learning and the reason is the pre-trained weights can ease forgetting by leading to wider minima. The model is evaluated on a range of datasets in CV and NLP to support the findings. \n\n\n\n", "summary_of_the_review": "The paper provides an empirical study to understand the role of pre-training in lifelong learning.  Given that the paper draws conclusion based on empirical results, it's important to have a systematic study over the existing continual learning settings and baselines (e.g.,  various training/evaluation settings,  evaluation metrics such as Backward/Forward transfer,  datasets, etc.). The current manuscript pushes one step further towards understanding the role of pretraining, but still more studies can be added.  \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an empirical study on the impact of pertained model on lifelong learning. It concludes that the generic pertaining can benefit the lifelong learning duet the flatter loss landscape and evaluates on CV and NLP tasks. The paper is well written with detailed analysis. However, there is concerns on its limited setting and the conclusion is known in the community and based on empirical studies.
Decision: Reject