Pretrained Language Model in Continual Learning: A Comparative Study
Authors: Tongtong Wu
Massimo Caccia
Zhuang Li
Yuan-Fang Li
Guilin Qi
Gholamreza Haffari
Venue: ICLR 2022 Poster (2022)
Keywords: Continual Learning
Pre-trained Language Model

Abstract: Continual learning (CL) is a  setting in which a model learns from a stream of incoming data while avoiding to forget previously learned knowledge. Pre-trained language models (PLMs) have been successfully employed in continual learning of different natural language problems. With the rapid development of many continual learning methods and PLMs, understanding and disentangling their interactions become essential for continued improvement of continual learning performance. In this paper, we thoroughly compare the continual learning performance over the combination of 5 PLMs and 4 CL approaches on 3 benchmarks in 2 typical incremental settings. Our extensive experimental analyses reveal interesting performance differences across PLMs and across CL methods. Furthermore, our representativeness probing analyses dissect PLMs’ performance characteristics in a layer-wise and task-wise manner, uncovering the extent to which their inner layers suffer from forgetting, and the effect of different CL approaches on each layer. Finally, our observations and analyses open up a number of important research questions that will inform and guide the design of effective continual learning techniques.

Published as a conference paper at ICLR 2022
PRETRAINED LANGUAGE MODEL IN CONTINUAL LEARNING: A COMPARATIVE STUDY
Tongtong Wu' Massimo Caccia", Zhuang Li", Yuan-Fang Li", Guilin Qi', Gholamreza Haffari 'Southeast University "Monash University 'MILA (wutong8C23, gqi}eseu. edu. en, massimo,, . p. cacciaigmail..................am com
(tongtong. wu, thuang.1i, yuanfang. li, gholamreza haffari, @menash.
ABSTRACT
Continual learning (CL) is a setting in which model learns from stream of in- coming data while avoiding to forget previously learned knowledge. Pre-trained language models (PLMs) have been successfully employed in continual learning of different natural language problems. With the rapid development of many con- tinual learning methods and PLMs, understanding and disentangling their inter- actions become essential for continued improvement of continual learning perfor- mance. In this paper, we thoroughly compare the continual learning performance over the combination of PLMs and 4 CL approaches 3 benchmarks in 2 typ- ical incremental settings. Our extensive experimental analyses reveal interesting performance differences across PLMs and across CL methods. Furthermore, representativeness probing analyses dissect PLMs performance characteristics in layer-wise and task-wise manner, uncovering the extent which their inner lay- ers suffer from forgetting, and the effect of different CL approaches on each layer. Finally, our observations and analyses open up a number of important research questions that will inform and guide the design of effective continual learning techniques.
INTRODUCTION
Continual Learning (CL) methods aim at training model from stream of non-i.i.d. samples, relieving catastrophic forgetting (CF) while limiting computational costs and memory footprint. Throughout the years, many methods have been proposed to address the CL problem in computer vision and robotics (Kirkpatrick et al.) 2017 Serrà et al.) 2018 Buzzega et al. 2020). CL in NLP is still a nascent topic, as reflected by the relatively smaller number of proposed methods (Biesialska et al. 2020 Li et al. 2021I. However, it is not always easy precisely measure the merits of these works. This is partly due to the subtle differences of-the-art approaches only stand out in the setting where they were originally conceived. Moreover, pretrained language models (PLMs) have been widely applied in CL methods NLP, and their addition further complicates the clear understanding of model performance (Han et 2020 Wang
the way methods are evaluated: many state-
ct al. 2019 Cao et al. 2020).
Although many works that apply PLM in CL point out the CF problem present in PLMs, there re- main three significant issues that require further investigation. (1) Many existing CL works employ specific PLMs such BERT (Devlin et al. 2019 Rogers et al. 2020), whereas more recent and sophisticated PLM structures, c.g. ALBERT (Lan et al. 2020) and XLNET (Yang et al.) 2019), have been developed afterwards. A natural question would be whether the later PLMS could better mitigate CF in continual learning (Mosbach et al.] 2021 Lee et al.) (220aa). (2) The existing works focus on input, output, and gradients information PLMs while ignoring the effect of model archi- tecture on CF (Biesialska et al.) 2020). We believe opening up the black-box of PLMs may lead to a deep understanding of their characteristics and thus better algorithmic design (Wallat et al. 2020). (3) Some existing works focus alleviating forgetting in trivial setting, during knowledge trans- fer between two tasks, which not realistic in general continual learning with more tasks or even without task boundary (Tiang et al. 2020).

---

Published as a conference paper at ICLR 2022
In this paper, conduct in-depth exploration of continual learning through extensive empirical analysis of a number of PLMs and CL methods. We analyze not only the performance differences across the combinations of PLMs and CL methods, but also the idiosyncrasies of each PLM.
Our main contributions are as follows:
We design a rigorous benchmark for comprehensive study of continual learning NLP. We conduct experiments over (I) two primary continual learning setting, including task- incremental learning and caass-nncremenal learning, (2) three benchmark datasets with different data distributions and task definitions, including relation extraction, event classifi- cation, and intent detection; (3) four CL approaches with six baseline methods implemented for systematic comparison; and (4) five pretrained language models.
We evaluate and contrast the performance the above combinations of settings, PLMs and CL methods, providing a comprehensive comparative study from number of perspectives.
We dissect the performance characteristics of different PLMs with number of layer-wist wise probing analyses.
Our observations and insights give rise to a number of open research questions that can guide the design and optimization of better PLM-oriented continual learning methods and CL-oriented pretraining strategies.
To encourage more research on continual learning in NLP, we release the code and dataset as an open-access resource on https::/githhi wwtp///cc//ototg0/2/2/2LL...
BACKGROUND
In this section, we provide the required background on pre-trained language models as well as continual-learing settings, methods and evaluation metrics.
2.1 PRETRAINED LANGUAGE MODELS
It now become the best practice to incorporate PLMs into NLP systems for many problems such as question answering, machine reading comprehension, summarization, to name a few. Usually, making use of PLMs in such systems leads to significant performance gains in (weakly) supervised learning. In this work, we investigate the use of PLMs in continual learning. We now briefly introduce five typical PLMs that evaluated in our work.
BERT (Devlin et al., 2019) is the most representative PLM, which uses bi-directional deep Trans- formers (Vaswani et al. (017) the backbone. BERT adopts the Masked Language Modeling (MLM) and the Next Sentence Prediction (NSP) as the self-supervision tasks for pre-training.
ALBERT (Lan et al. 2020) ALBERT is lite version of BERT which the parameters of all Transformer blocks shared across all layers and factorizes the embedding matrices into two small size of matrices. Instead of NSP. ALBERT predicts the order of two consecutive textual seg- ments. Although ALBERT utilizes significantly less memory, fine-tuning it on downstream tasks can achieve close performance to that of BERT.
RoBERTa (Liu et al. 2019) RoBERTa has almost the same architecture as BERT, while it differs in terms of three training procedures. RoBERTa removes the NSP loss, trains a model with bigger size and longer sequences, and creates dynamic MLM masks static ones used in BERT.
GPT2 (Radford al.) Unlike the aforementioned PLMS which are all masked language models, GPT2 is autoregressive language model predicting one token at time from left to right. GPT2 often used for natural language generation, while the aforementioned PLMs are mostly used for natural language understanding.
XLNET
Yang et al. 2019) Unlike the masked language models, GPT2 can not utilize the context from the backward side. XLNET is autoregressive language model well, while it resolves this problem by adopting a new objective called Permutation Language Modeling, enabling the model to take advantage of both forward and backward contexts.
2

---

Published as a conference paper at ICLR 2022
2.2 CONTINUAL LEARNING SETTINGS
Continual learning (CL) focuses on the development of learning algorithms able to accumulate knowledge on non-stationary data. CL approaches are benchmarked on their ability to learn a se- quence of tasks without forgetting previously aequired knowledge. They are typically evaluated on incremental classification settings.
Some popular incremental learning scenarios in CL (van de Ven & Tolias 2019: Zeno et al. 2018) include: class-incremental learning and task-incremental learning. In the training time of these in- cremental learning settings, a CL algorithm experiences each task sequentially (only once), and informed about the distribution shift (aka the task boundary). They, however, differ in their assump- tions about the evaluation.
task-incremental learning, the model relies on the task identity (or a task label) to make its prediction. Conversely, in cass-incremental learning, the methods have to perform task inference, implicitly (Aljundi) 2019) or explicitly (Lee et al.) 2020b). We will cover both settings in our study. A lesser known setting, namely Domain Domain-incrementul incremental learning, also laxes the task-ID dependence, but shares the same output head for each task. For more details on these settings, (van de Ven & Tolias 2019
2.3 CONTINUAL LEARNING APPROACHES
Many continual- approaches have been proposed recently, e.g., see Delange al. (2021) for an overview, which can be categorised as follows:
Rehearsal- Approach The simplest way to reduce forgetting is to store samples from the past and reuse them to complement the learning of new tasks. its simplest form, old samples are replayed with new data. This strategy, known as experience replay (ER), (Rolnick et al. 2019) is often hard beat baseline. Numerous rehearsal-basee based methods have been developed increase ER's performance or efficiency (Aljundi et al.) 2019a] Caccia et al. 2019 Hayes et al. 2018).
Instead of replays, some methods use the old sample to perform constrained optimization to prevent increasing the loss old tasks (Lopez-Paz Paz. & Ranzato 2017 Aljundi et al.. 2019b).
Regularization- Approach Also know prior-based approaches, these methods prevent significant updates to the parameters that are deemed important for the previous tasks. Their first in- stantiation appeared in the Blastic-Weight Consolidation (EWC) (Kirkpatrick al. 2017), where the previously learned weights are restrained from drifting via an L.2 regularization loss. Regularization- based strategy often rely on the task boundaries consolidate their knowledge during training. They often fail on long tasks sequences or settings where the task identity is not observable (Farquhar & Gal 2018 Lesort et al. 2019b Chaudhry al. 2018). Despite these findings, prior-focused meth- ods are actively researched (Zeno et al. 2018 Nguyen et al. 2018).
Dynamic Architecture Approach Also know parameter-issolation methods, this family of al- gorithm alleviates forgetting by using different subset of parameters for fitting different tasks. One popular approach, called Hard Attention to the Task (HAT) (Serrà et al. 2018). achieves parameter freezing through an attention mask that is learned concurrently at every tasks. Other similar strate- gies have also been proposed (Yoon et al. 2017; Schwarz et al., 2018). Similarly regularization- based methods, dynamic architectures usually assume the availability of test-time task labels, and thus are not straightforwardly applicable in more realistic settings. A notable exception can be found in Ostapenko et al. (2019).
Hybrid Approach Some hybrid strategy from different strategy also exist. E.g. Dark Experience Replay (DERPP) Buzzega et al. (2020) proposes a method at the intersection of rehearsals and regularization based methods.
The rapid growth of continual learning has lead researchers to work on empirical studies (De Lange et al. 2019 Lesort et al. 2021b), surveys (Hadsell et al. 2020 Khetarpal al. 2020 Lesort al. 202 laj Mundt et al. 2020; 202T) as well as CL-specific software (Normandin et al. 202 [ Douillard
& Lesort 2021
Lomonaco al. 202T). Further approaches are detailed in Appendix
3

---

Published as a conference paper at ICLR 2022
3 BENCHMARKING CONTINUAL LEARNING OF PLMS
In this section, we first compare the performance of different combinations of CL methods and PLMs on three benchmark datasets, and we explore the following three research questions: (1) Does the catastrophic forgetting problem exist in PLMs during continual learning? (2) Which continual learning approach is the most efficient for PLMs and why? (3) Which PLM is the most robust for continual learning and why?
3.1 EXPERIMENTAL SETUP
Methods. We adopt 5 representative PLMs for evaluation, i.., ALBERT (Lan et al., 2020). BERT (Devlin et al. 2019) GPT2 (Radford et al.) (019). RoBERTa (Liu et al. 2019), and XL- NET (Yang et al. 2019). By combining a PLM and linear classifier as the backbone model, we investigate the following 4 approaches for the comparative study: (1) Vanilla uses the model learned on previous tasks as initialization and then optimizes the parameters for the current task. This baseline greedily trains the model on each task without accessing data from previous tasks, and is thus severely prone to catastrophic forgetting. It serves as a weak lower bound in terms of the average accuracy. (2) Joint trains the entire training set simultaneously in the conven- tional supervised learning setup. Therefore, Joint does not suffer from forgetting and represents the performance upper bound. (3) EWC (Kirkpatrick et al. 2017) is regularization-basee methods, extending the loss function with a regularization term that slows down the updates of the impor- tant network weights. (4) HAT (Serrà et al.) 2018) is a dynamic architecture method, employing a heuristic strategy to prevent intransigence by allocating additional units to the network when needed. (5) ER a rehearsal-based method, interleaving old samples with current data in training batches. (6) DERPP (Buzzega et al. 2020) is hybrid method, combining the strategy of rehearsal and reg- ularization which prevents the prediction logits memorized samples from changing. Following DERPP (Buzzeaa et al. 2020), we simulate the sequence of tasks according to the class-incremental learning setting (i.e. Class-IL or CIL) and task-incremental learning setting (i.e. Task-IL or TIL), splitting the training dataset into partitions of classes/tasks. Training details and pearameeess are in Appendix [
Metrics. To measure the FWT and BWT abilities of the CL models, assume access to a test set for each task. After the model learning on the training set of task t, we present the evaluation results on all T tasks. We adopt three evaluation metrics in (Lopez-Paz & Ranzato 2017): (1) Forward transfer FWT = -ー ET Ar. b;; (2) Backward Transfer BWT -
AT, A,,
And average accruacy Aug. ACC + ΣL= Ar, where A is the accuracy of models on the test set of ith task after model learning on the tth task and b; is the test accuracy for task i at random key performance measure, measuring the accuracy of past tasks after the model has moved on to learning new tasks.
initialization. In addition to FWT and BWT, we consider average accuracy
Datasets. We evaluate methods 3 datasets with distinct label distributions, covering the following domains. CLINCC50 (Larson et al. (019) is an intent classification dataset with labels evenly distributed, including 150 classes and 100 instances per class. Maven (Wang et al. 2020) is a tailed event detection dataset with 163 classes, each of which is labelled at least 15 instances, reaching total of 47.921 instances. WebRED (Ormandi et al.) 202T) is a severely long-tailed relation classification dataset with 243 classes, each of which is labelled with at least 15 instances, reaching a total of 31,441 instances. Adhering to (Wu et al., 202 I), we randomly split each dataset into disjoint tasks, and each tasks contains 10 classes. For each class, we randomly split the dataset set into train, validation and test set by 10:2:3. To reduce the amount of computation, we cap the number of instances per class for both Maven and WebRED to 1000. The data distributions of the three datasets is visualized in Appendix B
3.2 RESULTS AND DISCUSSIONS
The main evaluation results over all studied PLMs and CL methods, in terms of accuracy, are sum- marized in Table T For the Task-IL setting, Figure shows a comparison between the PLMs (a d) and between CL methods (c-h) in terms of accuracy, backward transfer, forward transfer, and train- ing time. Performance measure is averaged across the three datasets. From these results we can
4

---

Published as a conference paper at ICLR 2022
Settings
Class-IL
Task-IL
PLMs (Parameters)
Vanilla
EWC
ER
DERPP
Joint
Vanilla
EWC
HAT
ER
DERPP
Joint
C
6.04
6.29
63.07
73.24
94.58
15.24
14.09
19.96
77.96
86,96
96.84
ALBERT(L 177M)
M
6.25
6.21
36.56
36.43
83,28
16.86
14.34
22.31
57.40
55,42
92.29
3.41
3.80
22.65
19.52
61.19
14.66
11.94
16.95
47.21
43.7
93.33
C
15.09
12.11
82.13
86.29
95.69
51.22
41.71
34.56
92.00
94,71
97.07
BERT(109,57M)
M
7.62
6.85
$7.06
50.38
99.81
31.24
24.18
33.53
39.74
66.40
99.98
4.21
4.02
36.11
29.42
69.78
15.75
21.00
21.11
59.88
47.47
95.56
C
9.78
9.76
74.47
83.67
95.18
3
30.76
27. 62
86.58
92.20
97.00
(PT2(124.33M)
M
6.27
6.26
50.17
46.31
93.39
20.99
23.54
20.25
69. 69.70
62.82
99.24
W
4.10
4.53
29.77
22.64
66.06
19.60
20.50
16.81
58.26
46.83
93.41
C
14.89
11.20
83,04
83.62
96.31
50.76
37.18
38.09
91. 84
90.38
97.89
RoBERTa(124.44M)
M
7.00
6.85
52.35
46,49
99 46
26,90
28.97
33.24
72.01
62.10
99.96
W
5.15
3.99
36.24
35.10
71.89
23.88
17.08
19.31
61.99
54.41
95.06
C
8.53
10.20
72.22
78.67
95.64
36.51
35.71
48.67
87.64
89.38
97.38
XLNET((16.81M)
M
6.40
8.98
53.26
46.81
99.69
24.76
32.15
39.74
75.12
61.19
99.96
4.90
4.13
39.36
30.13
68.81
22.82
20.73
24.43
64.28
Table 1: Accuracy benchmark datasets with two continual learning setting, where "C" is sequen- tial CLINC150 dataset, "M" is the sequential MAVEN dataset, and "W" is the sequential WebRED dataset.
make the following observations on the three research questions that we posed previously. More analysis can be found in D and
Method
joint
vanilla
cr
ewc
derpp
-75
(a) Overall evaluation results in meekeincremmetall leamning grouped by PLM, including (1) accuracyf. (2) back- ward transfert, (3) forward transferf, and (4) time..
PLM
albert
80
bert
gp12
roberta
xtnet
(b) Overall evaluation results in aask-incremental learning grouped by continual learning method, including (1) accuracy1. (2) backward transfer?, (3) forward transfert, and (4) time].
Performance comparisons between PLMs (first row) and CL methods (second row) under the Task-incremental setting. For each row, the upper row is the results grouped by PLMs and the lower row is the results grouped by CL. methods. For each column, from left to right the label is accuracy, backward transfer, forward transfer, and time (in hours) respectively. Performance results are averaged across the three datasets. Note that the time spent by Joint in (a) and (b) is the time spent on training with the entire training set once.
Figure I:
For PLMs, catastrophic forgetting is serious. For each PLM, the performance gap between Vanilla and Joint indicates the PLM's tendency towards forgetting. For all PLMs on all three datasets, we can consistently observe significant performance gaps between the two methods, where the gaps are even more pronounced in CIass-IL. This observation indicates that, without specialised algorithms, direct adoption of PLMs in a CL environment will result in severe performance penalty.
Imbalanced data is natural yet harder in continual learning than in conventional supervised learning. Comparing the performance different datasets, we notice that PLMs often achieve
5

---

Published as a conference paper at ICLR 2022
better performance more evenly distributed data (i.e. CLINC150 better than Maven better than WebRED). Compared to the conventional supervised setting (Joint), continual learning methods, including the well-performing methods such ER and DERPP, are much more negatively impacted by data imbalance. This can be more easily observed in Figure Tb-1).
Experience replay is the most robust method while regularization is not. Comparing the four dif- ferent CL methods, as shown in Figure a-)2, we can observe that the methods adopting memory- replay (ER and DERPP) achieve much higher accuracy and higher backward transfer than the others (EWC and HAT). For dynamic architecture-based methods, HAT is the only method benefiting from continual learning. as shows the positive Forward Transfer (FWT) in Figure Ja-3, b-3). However, the higher training time, the lower accuracy, and the requirement for task label (thus inapplicable for Class-IL.) do limit the applicability of HAT. Counterintuitively, the regularization-bant EWC shows the worst performance in our experimental setting. DERPP, which adopts both regu- larization and experience replay, reaches the highest accuracy on CLINC150. Surprisingly, per- forms worse than ER on the other two imbalanced datasets (Figure (b-1,2)), which indicates that regularization-bast based methods may not be as robust as those based on experience replay.
method
BERT is still good option for continual learning. Comparing different PLMs. there is no obvious difference on average accuracy and backward transfer, as shown in Figure 1,2) RoBERTa and GPT2 show the lowest FWT values in Figure a--3, which means that their representative ability has been interfered the most during continual learning. For XINEE, Table 3 shows that conducts the lowest computation per instance yet Figure | a--4) shows that it spends the longest time in training. In comparison, ALBERT has the least parameters and approx. equal time to obtain competitive performance. As shown in Figure [a). BERT is still good choice for continual learning scenarios as it achieves the highest accuracy while being competitive on the other three metrics.
3.3 NEW RESEARCH QUESTIONS
The above results show that among all PLMs, BERT is the most robust, and that among all CL methods, those based on experience replay are the most performant. These observations lead to the following important research questions.
(1) What happens inside the black box of BERT during continual learning?
(2) What is the performance difference across PLMs and across the layers inside each PLM?
(3) Why are replay-based methods more robust than regularization-based methods?
(4) In which layers does replay make the most contributions?
We further explore these research questions in Sections and
4 MINING THE SECRETS OF LAYERS IN BERT
In this section, we dive into BERT to provide detailed analysis on the severity of forgetting on each of its layers, addressing RQ (1) of Section 3.3 Performing such an analysis further allows us to gain an understanding of the possible reasons of the robustness of experience replay-based methods, addressing RQ (3) of Section 3.3
Generally speaking, the representative ability of PLM increases with the increase of layers since such a bottom-up process can gradually gain more task-specific features. Hence, we put forward two possible reasons to catastrophic forgetting on BERT. (1) The continual learning problem with the PLM's ability to retrain the inner representations, which leads to catastrophic forgetting. (2) The continual learning of incremental tasks has minor influence on PLMs, but the classifier layer is the bottleneck. To investigate the above assumptions, we designed the following probing experiments.
4.1 PROBING SETTINGS
To investigate the representation ability per layer a PLM, we follow Saunshi et al. (2019) and propose prototype- based probing method (See Appendix E). which takes the layer- mean representation of instances belonging to the same class in the test set as the prototype representation of the class, and re-classify test instances by distance-based classification for the layer.

---

Published as a conference paper at ICLR 2022
100
100
30
02
40-
10
(a) BERT Vanilla (layer)
(b) BERT + ER (layer)
(c) BERT DERPP (layer)
100
100
100
40-
10
32
(d) BERT Joint (layer)
BERT+Vanila (c) BERT+ Vanilla (task)
(f) BERT ER (task)
100
20
12
24
24
BERT DERPP (task)
BERT Joint
Figure 2: Layer-wise and task-wise probing of PLMs during continual learning, where cl. means classifier illustrating the prediction accuracy, number k E [1, 12] means representative of the k-th layer in PLM.
Hence, we can use the mean classification accuracy short-hand for the representative ability for each layer, and obtain 7 N results to track the performance per layer during continual learning, where T is the number of learned tasks, i.e., time steps, and N is the number of layers in PLM, typically 12. To compare the the inner representative ability and the model output, we also track the prediction accuracy per task, which is denoted by el Note that, by comparing layer 12 and cl could learn the gap between the remaining representative ability of PLM and the performance of a backbone model. To provide fair comparison, we obtain the results for joint multi-task training by retraining new backbone model from scratch at each time step, rather than retraining model over seen data sequentially.
Sigureeisooss the probing results, in which we probe BERT with Vanilla, Joint and two CL methods based on experience replay, namely ER and DERPP. The analysis is done in two settings, layer-wise and task-wise.
Layer-wise Probing. row (a-d), the x-axis the layers of BERT, and each line represents the accuracy after the model is trained with task f, as measured by the testsets of tasks I up to t.
Task-wise Probing. In row (e-h), the x-axis represents the tasks, and each line represents the representativeness of each inner layer as well as the backbone model (clf), as measured by the testsets of tasks up to after the model trained with task
4.2 RESULTS AND DISCUSSIONS
Catastrophic forgetting occurs both the last and middle layers. As shown in Figure ((-d)), com- paring the layer-wise performance of Vanilla the other three methods, obvious drop appears

---

Published as a conference paper at ICLR 2022
ALBERT Vanilla
(b) GPT2 Vanilla
(c) RoBERTa Vanilla
(d) XLNET Vanilla
1 K 10
5.2.4 9 1015
(e) ALBERT buffer size
(1) GPT2 + buffer size
(g) RoBERTa + buffer size (h) XLNET + buffer size
Figure 3: The buffer size analysis of ER. The first row is the detailed layer-wise analysis for different PLMs with Vanilla; The second row is the averaged layer-wise probing for ER with various buffer sizes. The color-scale represents the probing performance.
in Vanilla between layer 12 and the final classifier layer for all tasks except task 1. This result in- dicates that although BERT still maintains a high representative ability at the last time step, yet classifier has already lost the ability to classify previously learned classes. Furthermore, as shown in Figure [e),, the bottom (i.e. earlier) layers of BERT are consistently less representative than the top layers, and the drop is the most significant in the 6th and the classifier layers.
Regularization of outputs cannot prevent the forgetting in the bottom layers. Besides experience replay, DERPP also adopts regularization the final classifier layer. Comparing ER and DERPP in Figure ( and (g), DERPP can better constrain the cl layer to maintain the representativeness of BERT, but is less effective than ER constraining the drop of the middle layers. Such differences methods.
may partially explain why ER could be more robust than regularization-base
The layer-wise and task-wise probing experiments give rise to the following research questions that may shed lights on further improvements of CL methods. (I) The regularization of the middle layers may improve continual learning performance. (2) The magnitude of the difference between the last and penultimate layer shows that it is mostly a badly calibrated output layer that causes the forgetting.
UNDERSTANDING THE EFFECTIVENESS OF EXPERIENCE REPLAY
In this section, we probe the possible reasons behind the effectiveness of experience replay. We do
so by trying understand why BERT is robust among all PLMs (RQ (2) Section (.3), and by analyzing the effect of replay on different layers of PLMs (RQ (4) in Section 3.3). See Appendix G for more results.
5.1 PROBING SETTING
We extend the probing method in the above section the following two probing settings.
Layer-wise Probing for Vanilla. As shown in Figure 3 d),,), we evaluate the layer-wise represen- tativeness of different PLMs (except BERT) with Vanilla.
Averaged Layer-wise Probing for ER with Various Buffer Size. Figure Ble-h) shows the mean accuracy per layer across time steps for ER with different buffer sizes. i.c.. the number of seen training instances stored for the future replay. Doing so allows us to investigate on which layers do experience replay make the most contributions.
8

---

Published as a conference paper at ICLR 2022
5.2 RESULTS AND DISCUSSION
The layer-wise representativeness varies significantly across PLMs. As shown in Figure a--d), although the prediction of cl is similar across PLMs, the performance characteristic of each PLM on their inner layers varies lot. For example, due to its parameter-sharing mechanism, the hidden layers of ALBERT are more fragile than BERT (Figure 2(a)) and RoBERTa (Figure C)), with larger accuracy gaps across tasks. Figure e- shows that which layers benefit the most from the increase in buffer size differs across PLMs. ER improves the performance of layer 12 and fl layers in XLNET, but for RoBERTa and GPT2 it is mainly the cl layer.
Compared with the classification layer, the representation of some inner layers maintains a high performance. We notice similar layer-wise performance curve in both BERT and RoBERTa, as shown in Figure a and Figure (]c). In both cases, layers 7 to 12 seems to be robust during continual learning. For XLNET, the robust layers are 10, with a dramatic drop layer 11. Moreover, some fragile layers can be observed for each PLM, such as layer 6 of BERT in Figure (e) and layer 12 for GPT2, and layer 11 for XLNET.
The robust layers in pretrained model may provide us the free lunch that we can extract features for metric-based classification without extra computation and buffer memory. The probing over five pretrained language model shows the existence of robust layer in BERT, XLNET, and RoBERTa. However, the post-selection of pretrained model and configuration may not always feasible real- world applications, we believe it is an important direction to predict and dynamically detect the robust layer for better continual learning performance.
CONCLUSION
In this paper, we conduct the first comprehensive comparative study that sheds light on the perfor- mance characteristics of continual learning across representative language models and CI. methods, as well as detailed layer-wise analysis within language models. The insights gained from this study open up new research questions that will inspire further research continual learning based on pre-trained self-supervised models.
Our comparative study pretrained language models for continual learning provides more in- sights on different aspects of performance, uncovers new opportumities for the development of NLP- specific continual learning methods in which language models play central role. Broadly speaking, the future impact of our work is threefold: (1) Criteria and strategies for selecting a PL.M for con- tinual learning. (2) Techniques for utilizing the layer-wiss insights of specific PLM for continual learning. Given the fragile layer and robust layer of a PLM, novel dynamic adapter structures, replay or regularization methods may be explored for more robust continual learning. (3) The open-source toolkit and best practices for consistent evaluation of CL performance.
ACKNOWLEDGMENTS
Research in this paper was partially supported by the National Nature Science Foundation of China (U21A20488). The first author was partially supported by the Chinese Scholarship Council. The computational resources for this work were supported by the Multi-modal Australian ScienceS Imaging and Visualisation Environment (MASSIVE) www.mmassive.org.uu We would like thank the anonymous reviewers for their insightful comments.
REFERENCES
Rahaf Aljundi. Continual learning in neural networks. arXiv preprint arXiv: 2019.
URL httpthts///axi...ooo org/ abs 1910.
Rahaf Aljundi, Lucas Eugene Belilovsky, Massimo Caccia, Min Lin, Lau- rent mal
Charlin, and Tinne Tuytelaars Online continual learning with maxi- interfered retrieval. In Advances
Neural Information Processing Sys- tems 32, pp. 11849-11860. 2019a. URL. http://papers. ni nips cc cc/pperr

---

Published as a conference paper at ICLR 2022
pdf
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample
selection for online continual learning. In Advances in Neural Information Prucessing
Systems 32, pp. 11816-11825. 2019b. URL http::/pappes.aeeppppppppap wcppppppppcccccpcpp..ppcpcpcpc 9354 115d-gradient -asett-aaee---ee same------deee--e-eeeeeee wwweesseeennneeeee -ertetiio--oor--ooo ott.a:/aaaaaaaaaaaaaaaaa.aaaaaar pdf
Magdalena Biesialska, Katarzyna Biesialska, and Marta Costa-jussà. Continual lifelong learning 2020.
in natural language processing: A survey. In Proceedings of COLING, pp.
URL ALtDo://bo...ogg1....81/////
wwwieooooaiiinnaeeeeee 574
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and SIMONE CALDERARA. Dark experience for general continual learning: a strong, simple baseline. In Proceeding, www.ipeurpp...
pp. 15920-15930, 2020. URL
Lucas Caccia, Eugene Belilovsky, Massimo Caccia, and Joelle Pineau. Online learned continual compression with adaptative quantizatiion module. arXiv preprint arXiv:1911.08019, 2019. URL
https::/axxiv.rrgg// http://aaxw....r//////1111111
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: new approach to continual learning. NeurlPS, 2020.
05856
Pengfei Cao, Yubo Chen, Jun Zhao, and Taifeng Wang. Incremental event detection via knowledge consolidation networks. In Proceedings of EMNLP, pp. 707-717, 2020. URL https:
oxg/10.18653/v1/220
emnlp main.52
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018. URL
https://arxiss wwt. wwwpg//b////oe 1801 10112
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. Continual learning: A comparative study on how to defy forget- ting classification tasks. arXiv preprint arXiv:1909.000
2019.
M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting classification tasks. JEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. URL
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. of pp.. 4171- 4186, 2019. URL https::p www aclweb. org/ anthology/M11-1122
Arthur Douillard and Timothée Lesort. Continuum: Simple management of complex continual learning scenarios. ArXiv, abs/2102.06253, 2021.
Sebastian Farquhar and Yarin Gal. Towards robust evaluations of continual learning, arXiv preprint arXiv:1805.09733, 2018.
09733
Raia Hadsell, Dushyant Rao, Andrei Rusu, and Razvan Pascanu. Embracing change: Continual learning deep neural networks. Trends in Cognitive Sciences, 24:1028-100 1040, 12 2020. doi: 10.101 t/i. w0t.s........0000.
Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Continual relation learning via episodic memory activation and reconsolidation. In Proceedings of ACL, pp. 6429 6440, 2020. URL hntpps: : h/www//wwwww. aclwešs http:/rrooaooogg//2222..3..
ac col-maine 1emainn5777 573
10

---

Published as a conference paper at ICLR 2022
Tyler L. Hayes, Nathan D. Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. CoRR, abs/1809.05922,, 2018. URL http:://xxii..oo////aaaa 05922
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu, Yee Whye Teh, and Razvan Pas- canu. Task agnostic continual learning via meta learning. ArXiv, abs/1906.002011, 2019. URL
petps://aarxir org/ abs 1906. 05201
Khurram Javed and Martha White. Meta-learning representations for continual learn- ing. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32. pp. 1818- 1828. Curran Associates, Inc., 2019. URL http:: nips cc/paper/ 8458-meta-
pdf
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao.
SMART: Robust and efficient fine-tuning for pre-trained natural language models through prin- cipled regularized optimization. In Proceedings of ACL. pp. 2177-2190 2020. URL hetps:
www. ac aclweb. org/ anthology 2020 acl mace--miiie mw....ain......m.....m.......... 197
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives. 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Ctrabska-Barwinskee Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11<(13)33221-3526, 2017.
URL htLhttts: wwww pr.s. //
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self-superviseed learning of language representations. In ICLR, 2020. htttaa/:/opeeeeeeeeeeeeee/ooroo
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of EMNLP-IJCNLP, pp. 1311-1316, 2019. URL https:/ htttpp://www.acrm
D19-10
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR, 2020a.
Eorum? i -inimiiddddiiiiii-
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In Proceedings of ICLR, 2020b.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Jean-François Goudou, and David Filliat. Generative models from the perspective of continual learning. In IJCNN International
Joint Conference on Neural Networks, 2019a. Utt/ttts::
ouvertes.
Er/hal-01951950
Timothée Lesort, Andrei Stoian, and David Filliat. Regularization shortcomings for continual learn- ing. arXiv preprint 19120099,,,,3449, 2019b.
Timothée Lesort, Massimo Caccia, and Irina Rish. Understanding continual learning settings with data distribution drift analysis. 202
Timothée Lesort, Thomas George, and Irina Rish. Continual learning deep networks: an analysis of the last layer. 2021b.
Zhuang Li, Lizhen Qu, and Gholamreza Haffari. Total recall: a customized continual learning method for neural semantic parsers. CoRR, abs/2109.051888 2021.

---

Published as a conference paper at ICLR 2022
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov, Roberta: A robustly optimized BERT pretraining approach. CoRR, 10tp//990.1192,, 2019.
abs /et.//197000 1907.11692
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange. Mare Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-to-end library for continual learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning in Computer Vision Work- shop, 2021.
David Lopez-Pazz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp- 6470-6479, 2017.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. On the stability of fine-tuming BERT: Misconceptions, explanations. and strong baselines. In ICLR, 2021. URL https:// openreview htt/://ee///oooe/oo forum idd
Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge active and open world learning. 2020.
Martin Mundt, Steven Lang, Quentin Delfosse, and Kristian Kersting. Cleva-compass: A contin- ual learning evaluation assessment compass to promote research transparency and comparability. 2021.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In International Conference on Learning Representations, 2018. URL https:
10628
Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Matthew Riemer, Pau Rodriguez, Julio Hurtado, Khimya Khetarpal, Timothée Lesort, Laurent Charlin, Irina Rish, and Massimo Caccia. Sequoia towards a systematic organization of continual learning research. Github repository, 2021. URL http///gttttttteee com. http//eebiice// httm//eeerree/eeuuiaaaa
Robert Ormandi, Mohammad Saleh, Erin Winter, and Vinay Rao. Webred: Effective pretraining and finetuning for relation extraction the web, 2021. URL
2102.09688
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to
remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition, pp. 11321-11329, 2019. URL. thecvf wtt.c//c//ooctce
https:
Continual CVPR 2019 paper....... htmml
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language
models are unsupervised multitask learners.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In ICLR, 2018.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how BERT works. Trans. Assoc. Comput. Linguistics, 8:842-866, 2020.
12

---

Published as a conference paper at ICLR 2022
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances Neural Information Processing Systems, pp. 348- 358. 2019. https:://axtt:i org/
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khande- parkar. A theoretical analysis of contrastive unsupervised representation learning. In Pro- ceedings of ICML, pp. 5628-5637, 2019.
saunshi 9a html
Jonathan Schwarz, Jelena Luketina, Wojeiech M Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In ICML, 2018. URL httpp://aaa....rrr/a////88...0.0
Joan Serrå, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of ICML, volume 80, pp. 4555-4564, 2018. URLppttt://proceeeiiii
wtt.sssmw......e.eemeeeeemeeseee www.sppasssseeeeeeeeess http///00/eeerrlae html
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances Neural Information Processing Systems, pp. 2990-2999, 2017. URL
https: /tas:///axivv wwt./ao/////ooooooo//ooo.//..o.r abs / 705. 08690
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st In- ternational Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
Risto Vuorio. Dong- Yeon Cho, Daejoong Kim, and Jiwon Kim. Meta continual learning. arXiv preprint
2018.
Jonas Wallat, Jaspreet Singh, and Avishek Anand. Bertnesia: Investigating the capture and forgetting of knowledge in BERT. CoRR, abs/2010.09313, 2020. URL httpss://axxii.org////////// abs/ 2010.09313
Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. Sen- tence embedding alignment for lifelong relation extraction. In Proceedings of NAACL-HI.T, pp- 796-806, 2019. URL /ttp::////o..r111111111///1////
1086
Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li. Yankai Lin, and Jie Zhou. MAVEN: Massive General Domain Event Detection Dataset. In Proceedings of EMNLP, pp. 1652-1671, 2020.
Tongtong Wu, Xuckai Li, Yuan-Fang Li, Reza Haffari, Guilin Qi, Yujin Zhu, and Guoqiang Xu. Curriculum-mer meta learning for order-robust continual relation extraction. CoRR, abs/2101.01926, 2021. URL https:t:: http:///wwwaa.ooo///////2...... 01926
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of NeuriPS, volume 32, pp. 5754 2019. htRL:stttts://rrccdd.ngg
wttpeeedddiinnnunnniieeeeeeeeeee cc/ hatp://aao1111eee666631116666612 Pawp:eee.coopppppppppp. pdf
Jaehong Yoon, Funho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708.015:: 2017.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. 2018. URL / 101.3...1......101111 .pff
13

---

Published as a conference paper at ICLR 2022
A FURTHER RELEVANT APPROACHES TO CONTINUAL LEARNING
Instead of using previously-storee data for replay, some methods will train generative model along- side their classifier and perform the rehearsal on generated samples. These methods are often re-
ferred to generative replay[Shin [hhin et al. 2017) Lesort et al. (2019a); Ostapenko al. (2019).
Furthermore, the field of meta-learning, now interrelating with continual learning, as provoked the emergence of new methods. In meta-continual learning JJaved & White (20T9); Vuorio et al.) (2018), algorithms are learning how to continually learn. The hope is that the continual-learning problem
can be solved in a data-driven way by the learning algorithm itself. In continual-meta learning, the task-agnostic CL setting, i.e., where the task boundaries aren't provided during training, is tackled. The algorithms perform fast adaptation Riemer al. (2018) to adapt the current task He et al. 2019); ; Cacciaa et al. (2020).
B DATASET DISTRIBUTIONS
The distribution of each dataset are summarized as Figure
(a) Training instances distribation in CLINC150
(b) Training instances distribution in MAVEN
(c) Training instances distribution in WebRED
Figure 4: Data distribution
C MODEL TRAINING AND PYPER-PARAMETERS
Hyperparameter selection. For each combination of PLM and CL scheme, we perform a com- bined grid-search for Class-II. and Task-II, choosing the configuration that achieves the highest final accuracy averaged on the two settings.
Typically, summarize the details of implementation follows:
Training. To provide a fair comparison among CL methods, we train all the networks using the AdamW Mosbach al. (2021) optimizer, and select 10e-5 as the learning rate for all pretrained
14

---

Published as a conference paper at ICLR 2022
Key
Vanilta
num checkpoint
As the size of pretrained model is rynically lorge, we only keep the check point from lost task for regularization.
EWC
1,000,000
The weighr for panalty, selected from [0.1, 160, 1,000, 50,000. 1.000,00,, 10,000.000) weight updating fisher information,
?
0.7,
buffer size
200
500 for MAVEN and WEBRED. Refer to DERPP
ER
Sampling Method
Reservoir
0.7
The weight for panalty, selected from [0.1. 0.7. 100.
sealing pamaeter quantliization, The threshold for clamping. refer HAT The threshold for embedding clamping. refer to HAT
HAT.
coofitureabadd
50
emberdaeesahod
buffer
200
500 for MAVEN and WEBRED.
Sampling Method
0
0.5
The weight for logits distillation, selected from [0.1. 0.5, 0.7. I, 10. 100). The weight for experience replay, selected from (0.1. 0.5. 0.7, 10, 100).
Table The hyper-parametent of each baseline method.
ALBERT
BERT
GPT2
RoBERTa
XLNET
GMac
3.9
4.25
4.25
2.83
Parameters
11.78M
109.57M
124.53M
124.74M
116.81M
Table Experimental computation complexity per instance.
backbone models. We deliberately hold batch size and minibatch size from the hyperparameter space, thus avoiding the flaw caused by the different numbers of update steps for different methods.
D COMPUTATION COST
To provide perspetive on computational complexity, we also measure the giga multiply-accumulate operations (GMac) per instance (spans of 50 tokens randomly selected from Maven) for each PLM and its number of parameters, as shown in Table 3
E PROBING METHOD
As shown in Algorithm | tion on validation dataset.
we do layer-wise probing with the layer-wise prototype-based classifica-
F FINAL STAGE PROBING
As shown in Figure 5 and Figure pretrained language models show different layer- robustness at the last stage of task- iess-incremettll and inas-innremental settings.
G OVERALL PERFORMANCE ON PLMS
As shown in Figure 7 pretrained language models demonstrate different task-wise representability during continual learning.
15

---

Published as a conference paper at ICLR 2022
Algorithm 1: Function of Layer Evaluation Evaluate.Layer()
Input: Classification model he; Validation Dataset D Feature layers Ay, Candidate Feature Layers
A...
Sample examples (1t, v&) E K C Deal
Detach layer-wise hidden states h4 E [el for an KC and I E Ae
Get layer-wise prototype v' via averaging iif by class
Get layer-wise distance dist/ + ahh vd)) per example
layer- wise representability accA. [ Σx U(arg max[dist]]] yc)
return acc
100
vanis
90
o0
80
80
70
accuracy
accuracy
60
60
50
SHEEEN Patt
40
30
20
20
11
11
laver
laver
(a) ALBERT
(b) BERT
100
10
1000
13
(c) RoBERTa
GPT2
XLNET
Figure 5: The layer-wise performance of PLMS during task-incremental learning.
16

---

Published as a conference paper at ICLR 2022
van
variiz
80
despp
80
Jesp
accuracy
accuracy
60
40
20
20
2
4
6
7
8
9
10
11
12
dl
2
4
6
8
9
10
11
12
cIT
(a) ALBERT
(b)
BERT
50
kayer
(c) RoBERTa
(d) GPT2
XLNET
Figure 6: The layer-wise performance of PLMs at the last stage of class-incremental learning.
vani CWC_00
70
60
65
derpp
jaine
60
40
55
30
50
20
45
0-
8
10
12
14
100
12
14
task
task
(a) ALBERT
(b) BERT
20
10
30
TIVL
(c) RoBERTa
(d) GPT2
(e) XLNET
Figure 7: The task-wise performance of PLMs during class-incremental learning.
17


Reviews:
[{"review_id": "zhqKTyU59e", "reviewer": "Reviewer_EuaL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strenghts:\n\nThe analysis of PLM is carried out over different continual learning techniques, NLP tasks, PLM variants and finally by layer-wise probing analysis. Understanding the strengths and weaknesses of each PLM is very desirable research progress. The authors also provide new research questions that arise from the analysis and point to interesting unsolved research challenges.\n\nEvaluation starts off with the expected lower and upper bound of performance, and then moves on to disentangle FWT and BWT (backward, forward performance) when using various CL techniques.   The chosen data sets are not well behaved, i.e. experience imbalances, which makes the results more realistic (less artificial). Overall, this study provides a necessary step towards exploring future continual learning methodology and explores many important factors on eight pages.\n\nFrom the batch-learning adaptation literature on PLMs one may expect baselines such as various adapter block versions or 'anti forgetting hacks', but it is understandable that the authors did not test these, since adapters would likely introduce complexity per increment and quickly become impractical. As the authors mention, CL specific future extensions to adapters are conceivable, but a work of their own.\n\n### Minor weaknesses (easily fixable suggestions for improvement -- 1 content page left for improvements):\n\n- Some plots seem to be a very small, and may be enlarged to use the 9th page.\n- Fig 2 plots should share a larger model legend (on the figure top or bottom), so the bars can become wider and easier to distinguish\n- Fig 2 color could be made more distinguishable, especially since the plots are narrow.\n- Tab 1 could underline the best non joint performance — makes it easier to glance\n- sec 4. That Transformer layers, except the classification layer, do not adapt much during fine-tuning, is a known result (hence assumptions 1 and 2 in the paper), which should be cited — see BERTOLOGY Primer by Anna Rogers for references. Here intermediate layers are shown to forget as well, so this is a nice new finding, that can be contrasted.\n- Fig 3: I assume the figure color-scale is probe performance? Also, the buffer size, e.g. er-200 should be explained with an example. Is it 200 er samples, 200 samples/ class? In section 5.1 it should be (re-)mentioned what the buffer size means.\n\n### Spelling errors (not all listed): using a TTS app/function makes it easy to find these\n\n- sec 2.3. \"eg\".\n- sec 3.2:\n    - \"could be find\" ... can be found\n    - tendency to forgetting ... to forget/ towards forgetting\n    - reach the highest .. reaches\n- 4\n    - interferes ... with the PLMs ability to retrain _ representations\n- 4.1\n    - Joint ... Joint multi-task training\n    - from sketch ... from scratch\n- 4.2.\n    - methods making sense ... make sense\n    - The sentence \"(2) The classification layer clf is typically the most fragile of BERT, where continual learning learning methods making sense.\" is not understandable.\n\n## Questions to the authors:\nNone, regarding clarity.\n", "summary_of_the_paper": "This paper explores the continual learning performance when combining different PLMs and common continual learning methods with 3 challenging NLP classification tasks.\n\nTo benchmark these combinations the methods are evaluated in task-incremental and class-incremental learning settings over various NLP end-tasks, which covers common learning settings in continual learning and NLP. There is also a layer-wise performance analysis to identify which layers keep or forget task relevant information during training.\n\nOverall the paper shows that forms of replay outperform other methods like regularization.", "summary_of_the_review": "The problem, proposed CL algorithms, benchmarks and evaluation metric are suitable to answer the research questions. The performance and layer-wise analysis is conductive to deepen an understanding for the shortcomings and potential opportunities of PLMs for continual learning. Unsurprisingly, variants of ER are the most effective technique in incremental CL with PLMs, which is somewhat disappointing, but also a standard outcome in CL.\n\nThe paper is mostly well written, and the experiments (sub research questions) are logically structured. Some minor details can be improved and have been pointed out in the review. Insights are valuable and provide a solid foundation for followup studies. The details in the appendix were interesting and added to the paper and its reading flow, e.g. by moving cumbersome details like hyperparameters to a dedicated appendix section.\n\nI thus feel confident to recommend this paper for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "v17qRS6HB4h", "reviewer": "Reviewer_XW45", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nOverall the study is very thorough covering both the correct range of options for each axis studied and a set of relevant cross-axis multiple variable experiments. The organization is good (but not perfect, see below), its strengths are that the different options considered along each axis are clearly laid out ahead of time, with the exception of the continual learning strategies. The layer-wise analysis in particular is interesting and tells a coherent story, despite the challenges of displayed complicated 3D data. Overall, it seems that recently many studies compare quantitatively against multiple PLMs, which ultimately appear similar due to only slightly different performance numbers. This study's most successful contribution in my opinion is an exploration of the qualitative differences among PLMs in the continual learning setting.\n\nWeaknesses:\n- It would be really great to see how the insights after analysis can be used to improve performance. It's probably not absolutely required given the focus on probeing, but it would go a long way towards validating the insights.\n- Adjusting the numbers to achieve the 5/4/3/2 cuteness gets slightly in the way of understanding, unfortunately. The main reason for this is that the \"veins of CL methods\" has a different number over the course of the paper, which makes it hard to identify when a given list of N things is a list of the \"veins of CL methods\". Specifically, in the abstract and intro this number is 4, in section 2.3 this number is 3, in section 3.1 this number is 6, in Table 1 this number is 5 (two different sets of 5), in Figure 1 this number is 6, and in Figure 2 this number is 4. For a paper that has so much going on and so many different lists of different sizes, keeping these consistent would make it much easier for the reader to understand at any point what exactly this given list of N items is referring to. Along the same lines, it would help to be consistent with the language around each set of N things. For example, the \"veins of CL methods\" are called at least \"veins\", \"schemes,\" and \"approaches\" at different points. \n- Section 3.2, Table 1, and Figure 1 are relatively weak in my opinion. What am I supposed to conclude? I can look at the table and see the different results, but so what? What should I be drawing my eye to in the table (bold would help)? The Figure here is too small to even attempt to parse.\n - It would be very helpful to have a sentence that gives intuition about what's being measured with the accuracy metric. The definition is there, but it took me a second to realize that the intuitive idea is that it's measuring the accuracy of past tasks after the model has moved on to learning new tasks. \n", "summary_of_the_paper": "The authors perform a comprehensive study of how pretrained language models work in the continual learning setting. The authors study 5 relevant pretrained language models (masked and unmasked) and somewhere between 3 and 6 continual learning strategies depending on where in the paper they are counted. In addition to a thorough everything-by-everything evaluation, the authors hone in on the details of how the different models and CL approaches are reflected in the transformer layers. The authors find that the different language models studied perform relatively differently, both qualitatively and quantitatively, and these insights may provide useful for directing future improvements.", "summary_of_the_review": "Overall the authors perform a quite deep study of using pretrained language models for continual learning. Despite some weak points in the analysis of the quantitative results and inconsistent organization/language around the CL approaches, the thoroughness of the study, in particular the analysis at a layer-by-layer level, is likely of interest to the broader community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fKlyAPB6drG", "reviewer": "Reviewer_7fZV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the authors have conducted quite a lot of experiments, the phenomena shown in experiment results is hardly surprising to me. It is not surprising that the pre-trained language models would have forgetting issues when fine-tuned on downstream tasks. It is also not surprising that rehearsal-based methods perform the best for pre-trained models. \n\nMoreover, the paper draws a conclusion that BERT is the most robust one and is a good option if a continual learning process is going to be conducted. Based on this, the authors provide a few analyses on BERT’s ‘secret’ for continual learning. However, compared with other pre-trained models, I don’t see that BERT is significantly better than others given the figures and tables. I feel from the figures and tables, BERT and other models look similar. The authors didn’t give a comprehensive explanation on how they read such information or a concrete quantitative comparison to support this claim.\n", "summary_of_the_paper": "\nThis paper conducts an empirical study on the catastrophic forgetting of pretrained language models. On two continual learning settings (class incremental and task incremental), the paper evaluates multiple pre-trained models on different data sets, to see how severe the catastrophic forgetting issue is for these pre-trained models. Then the paper also tests the effectiveness of multiple continual learning methods on such pre-trained models and draws some conclusions.\n", "summary_of_the_review": "A thorough empirical analysis with unsurprising conclusions", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bj8K6YPwt9J", "reviewer": "Reviewer_P6Qd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think a comparative study paper should suffice at least two conditions to be considered for a publication at a venue like ICLR. First, it should present a novel view on the problem, and second, it should draw a novel conclusion out of the experiments. Although the paper could be a good survey for readers who want to learn about continual learning, I think its viewpoint is not new and its conclusion is not surprising. While it is helpful to know that rehearsal works better than regularization in most datasets, this is not entirely a surprising result. I think it is a common belief that rehearsal-based is more robust against catastrophic forgetting, while regularization-method is more space-efficient in that it doesn't have to store examples. The fact that the last layer suffers from catastrophic forgetting is also not a surprising result, given that the lower layers are known to encode linguistic features and the upper layers encode task-specific features.", "summary_of_the_paper": "As the title suggests, the paper is a comparison of recent continual learning methods that prevent catastrophic forgetting and their effectiveness in some text classification tasks using popular pretrained language models such as BERT, RoBERTa, etc. The paper divides continual learning methods into three categories: (1) rehearsal-based, (2) regularization-based, and (3) dynamic architecture. The experimental results show that rehearsal based methods are superior to the other two, and also that BERT is generally better than other candidates. The paper then proposes a new probing techniques to find out what makes rehearsal-based method better and what's happening inside BERT. The paper finds that the last layer has the biggest catastrophic forgetting and lower layer is less impacted.", "summary_of_the_review": "While the paper can be helpful for readers who want to learn about continual learning in text classification using pretrained language models, the paper does not seem to bring sufficient a novel viewpoint or conclusion to be published at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a comparison and analysis of continual learning methods for pretrained language models. The authors categorise continual learning methods into three categories, those that use cross task regularisation, those that employ some form of experience replay of previous training examples, and those that dynamically alter the network architecture for each task. Evaluation results from representative examples of these three paradigms are then presented and analysed. In general methods that incorporate experience reply appear to perform the best, while analysis of the predictive power of individual layers of the pretrained models suggests that some network layers are more robust to catastrophic forgetting than others, and that this also varies across architectures (BERT, ALBERT, etc.).

In general the reviewers agree that this is a well conducted study that provides an interesting contribution to an important area of research. They also generally agree that the many of the results are unsurprising given the properties of the algorithms explored and prior work in this area. The main point of difference then becomes how valuable it is to present a thorough study of existing algorithms that confirms our assumptions. I believe that the current work raises enough interesting questions to make it a useful contribution to researchers working in continual learning. In particular the results analysing the relative differences in catastrophic forgetting across different layers in models suggests interesting avenues for follow on work.
Decision: Accept (Poster)