Pretrained Language Model in Continual Learning: A Comparative Study

1 INTRODUCTION . Continual Learning ( CL ) methods aim at training a model from a stream of non-i.i.d . samples , relieving catastrophic forgetting ( CF ) while limiting computational costs and memory footprint . Throughout the years , many methods have been proposed to address the CL problem in computer vision and robotics ( Kirkpatrick et al. , 2017 ; Serrà et al. , 2018 ; Buzzega et al. , 2020 ) . CL in NLP is still a nascent topic , as reflected by the relatively smaller number of proposed methods ( Biesialska et al. , 2020 ) . However , it is not always easy to precisely measure the merits of these works . This is partly due to the subtle differences in the way methods are evaluated : many state-of-the-art approaches only stand out in the setting where they were originally conceived . Moreover , pretrained language models ( PLMs ) have been widely applied in CL methods in NLP , and their addition further complicates the clear understanding of model performance ( Han et al. , 2020 ; Wang et al. , 2019 ; Cao et al. , 2020 ) . Although many works that apply PLM in CL point out the CF problem present in PLMs , there remain three significant issues that require further investigation . ( 1 ) Many existing CL works employ specific PLMs such as BERT ( Devlin et al. , 2019 ; Rogers et al. , 2020 ) , whereas more recent and sophisticated PLM structures , e.g . ALBERT ( Lan et al. , 2020 ) and XLNET ( Yang et al. , 2019 ) , have been developed afterwards . A natural question would be whether the later PLMs could better mitigate CF in continual learning ( Mosbach et al. , 2021 ; Lee et al. , 2020a ) . ( 2 ) The existing works focus on input , output , and gradients information of PLMs while ignoring the effect of model architecture on CF ( Biesialska et al. , 2020 ) . We believe opening up the black-box of PLMs may lead to a deep understanding of their characteristics and thus better algorithmic design ( Wallat et al. , 2020 ) . ( 3 ) Some existing works focus on alleviating forgetting in a trivial setting , during knowledge transfer between two tasks , which is not realistic in general continual learning with more tasks or even without task boundary ( Jiang et al. , 2020 ) . In this paper , we conduct an in-depth exploration of continual learning through extensive empirical analysis of a number of PLMs and CL methods . We analyze not only the performance differences across the combinations of PLMs and CL methods , but also the idiosyncrasies of each PLM . Our main contributions are as follows : • We design a rigorous benchmark for comprehensive study of continual learning in NLP . We conduct experiments over ( 1 ) two primary continual learning setting , including taskincremental learning and class-incremental learning ; ( 2 ) three benchmark datasets with different data distributions and task definitions , including relation extraction , event classification , and intent detection ; ( 3 ) four CL approaches with six baseline methods implemented for systematic comparison ; and ( 4 ) five pretrained language models . • We evaluate and contrast the performance of the above combinations of settings , PLMs and CL methods , providing a comprehensive comparative study from a number of perspectives . • We dissect the performance characteristics of different PLMs with a number of layer-wise probing analyses . • Our observations and insights give rise to a number of open research questions that can guide the design and optimization of better PLM-oriented continual learning methods and CL-oriented pretraining strategies . • To encourage more research on continual learning in NLP , we will release the code and dataset as an open-access resource . 2 BACKGROUND . In this section , we provide the required background on pre-trained language models as well as continual-learning settings , methods and evaluation metrics . 2.1 PRETRAINED LANGUAGE MODELS . It now become the best practice to incorporate PLMs into NLP systems for many problems such as question answering , machine reading comprehension , summarization , to name a few . Usually , making use of PLMs in such systems leads to significant performance gains in ( weakly ) supervised learning . In this work , we investigate the use of PLMs in continual learning . We now briefly introduce five typical PLMs that are evaluated in our work . BERT ( Devlin et al. , 2019 ) is the most representative PLM , which uses bi-directional deep Transformers ( Vaswani et al. , 2017 ) as the backbone . BERT adopts the Masked Language Modeling ( MLM ) and the Next Sentence Prediction ( NSP ) as the self-supervision tasks for pre-training . ALBERT ( Lan et al. , 2020 ) ALBERT is a lite version of BERT which sets the parameters of all Transformer blocks shared across all layers and factorizes the embedding matrices into two small size of matrices . Instead of NSP , ALBERT predicts the order of two consecutive textual segments . Although ALBERT utilizes significantly less memory , fine-tuning it on downstream tasks can achieve close performance to that of BERT . RoBERTa ( Liu et al. , 2019 ) RoBERTa has almost the same architecture as BERT , while it differs in terms of three training procedures . RoBERTa removes the NSP loss , trains a model with bigger size and longer sequences , and creates dynamic MLM masks vs static ones used in BERT . GPT2 ( Radford et al . ) Unlike the aforementioned PLMs which are all masked language models , GPT2 is an autoregressive language model predicting one token at a time from left to right . GPT2 is often used for natural language generation , while the aforementioned PLMs are mostly used for natural language understanding . XLNET ( Yang et al. , 2019 ) Unlike the masked language models , GPT2 can not utilize the context from the backward side . XLNET is an autoregressive language model as well , while it resolves this problem by adopting a new objective called Permutation Language Modeling , enabling the model to take advantage of both forward and backward contexts . 2.2 CONTINUAL LEARNING SETTINGS . Continual learning ( CL ) focuses on the development of learning algorithms able to accumulate knowledge on non-stationary data . CL approaches are benchmarked on their ability to learn a se- quence of tasks without forgetting previously acquired knowledge . They are typically evaluated on incremental classification settings . Some popular incremental learning scenarios in CL ( van de Ven & Tolias , 2019 ; Zeno et al. , 2018 ) include : class-incremental learning , domain-incremental learning , and task-incremental learning . In the training time of these incremental learning settings , a CL algorithm experiences each task sequentially ( only once ) , and is informed about the distribution shift ( aka the task boundary ) . They , however , differ in their assumptions about the evaluation . In task-incremental learning , the model relies on the task identity ( or a task label ) to make its prediction . Conversely , in class-incremental learning , the methods have to perform task inference , implicitly ( Aljundi , 2019 ) or explicitly ( Lee et al. , 2020b ) . We will cover both settings in our study . A lesser known setting , namely Domainincremental learning , also relaxes the task-ID dependence , but shares the same output head for each task . For more details on these settings , see ( van de Ven & Tolias , 2019 ) . 2.3 CONTINUAL LEARNING APPROACHES . Many continual-learning approaches have been proposed recently , e.g. , see Delange et al . ( 2021 ) for an overview , which can be categorised as follows : Rehearsal-based Approach The simplest way to reduce forgetting is to store samples from the past and reuse them to complement the learning of new tasks . In its simplest form , old samples are replayed with new data . This strategy , known as experience replay ( ER ) , ( Rolnick et al. , 2019 ) is often a hard to beat baseline . Numerous rehearsal-based methods have been developed to increase ER ’ s performance or efficiency ( Aljundi et al. , 2019a ; Caccia et al. , 2019 ; Hayes et al. , 2018 ) . Instead of replays , some methods use the old sample to perform constrained optimization to prevent increasing the loss on old tasks ( Lopez-Paz & Ranzato , 2017 ; Aljundi et al. , 2019b ) . Regularization-based Approach Also know as prior-based approaches , these methods prevent significant updates to the parameters that are deemed important for the previous tasks . Their first instantiation appeared in the Elastic-Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) , where the previously learned weights are restrained from drifting via an L2 regularization loss . Regularizationbased strategy often rely on the task boundaries to consolidate their knowledge during training . They often fail on long tasks sequences or settings where the task identity is not observable ( Farquhar & Gal , 2018 ; Lesort et al. , 2019b ; Chaudhry et al. , 2018 ) . Despite these findings , prior-focused methods are actively researched ( Zeno et al. , 2018 ; Nguyen et al. , 2018 ) . Dynamic Architecture Approach Also know as parameter-isolation methods , this family of algorithm alleviates forgetting by using different subset of parameters for fitting different tasks . One popular approach , called Hard Attention to the Task ( HAT ) ( Serrà et al. , 2018 ) , achieves parameter freezing through an attention mask that is learned concurrently at every tasks . Other similar strategies have also been proposed ( Yoon et al. , 2017 ; Schwarz et al. , 2018 ) . Similarly to regularizationbased methods , dynamic architectures usually assume the availability of test-time task labels , and thus are not straightforwardly applicable in more realistic settings . Further approaches are detailed in Appendix A . 3 BENCHMARKING CONTINUAL LEARNING OF PLMS . In this section , we first compare the performance of different combinations of CL methods and PLMs on three benchmark datasets , and we explore the following three research questions : ( 1 ) Does the catastrophic forgetting problem exist in PLMs during continual learning ? ( 2 ) Which continual learning approach is the most efficient for PLMs and why ? ( 3 ) Which PLM is the most robust for continual learning and why ? 3.1 EXPERIMENTAL SETUP . Methods . We adopt 5 representative PLMs for evaluation , i.e. , ALBERT ( Lan et al. , 2020 ) , BERT ( Devlin et al. , 2019 ) , GPT2 ( Radford et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and XLNET ( Yang et al. , 2019 ) . By combining a PLM and a linear classifier as the backbone model , we investigate the following 4 approaches for the comparative study : ( 1 ) Vanilla uses the model learned on previous tasks as initialization and then optimizes the parameters for the current task . This baseline greedily trains the model on each task without accessing data from previous tasks , and is thus severely prone to catastrophic forgetting . It serves as a weak lower bound in terms of the average accuracy . ( 2 ) Joint trains on the entire training set simultaneously in the conventional supervised learning setup . Therefore , Joint does not suffer from forgetting and represents the performance upper bound . ( 3 ) EWC ( Kirkpatrick et al. , 2017 ) is a regularization-based methods , extending the loss function with a regularization term that slows down the updates of the important network weights . ( 4 ) HAT ( Serrà et al. , 2018 ) is a dynamic architecture method , employing a heuristic strategy to prevent intransigence by allocating additional units to the network when needed . ( 5 ) ER is a rehearsal-based method , interleaving old samples with current data in training batches . ( 6 ) DERPP ( Buzzega et al. , 2020 ) is a hybrid method , combining the strategy of rehearsal and regularization which prevents the prediction logits of memorized samples from changing . Following DERPP ( Buzzega et al. , 2020 ) , we simulate the sequence of tasks according to the class-incremental learning setting ( i.e . Class-IL or CIL ) and task-incremental learning setting ( i.e . Task-IL or TIL ) , splitting the training dataset into partitions of classes/tasks . Training details and hyper-pearameters are in Appendix C Metrics . To measure the FWT and BWT abilities of the CL models , we assume access to a test set for each task . After the model learning on the training set of task t , we present the evaluation results on all T tasks . We adopt three evaluation metrics as in ( Lopez-Paz & Ranzato , 2017 ) : ( 1 ) Forward transfer FWT = 1T−1 ∑T−1 i=2 AT , i − b̃i ; ( 2 ) Backward Transfer BWT = 1 T−1 ∑T−1 i=1 AT , i − Ai , i ; And average accruacy Avg . ACC = 1T ∑T i=1AT , i , where At , i is the accuracy of models on the test set of ith task after model learning on the tth task and b̃i is the test accuracy for task i at random initialization . In addition to FWT and BWT , we consider average accuracy as a key performance measure , measuring the accuracy of past tasks after the model has moved on to learning new tasks . Datasets . We evaluate our methods on 3 datasets with distinct label distributions , covering the following domains . CLINC150 ( Larson et al. , 2019 ) is an intent classification dataset with labels evenly distributed , including 150 classes and 100 instances per class . Maven ( Wang et al. , 2020 ) is a long-tailed event detection dataset with 163 classes , each of which is labelled at least 15 instances , reaching a total of 47,921 instances . WebRED ( Ormandi et al. , 2021 ) is a severely long-tailed relation classification dataset with 243 classes , each of which is labelled with at least 15 instances , reaching a total of 31,441 instances . Adhering to ( Wu et al. , 2021 ) , we randomly split each dataset into disjoint tasks , and each tasks contains 10 classes . For each class , we randomly split the dataset set into train , validation and test set by 10:2:3 . To reduce the amount of computation , we cap the number of instances per class for both Maven and WebRED to 1000 . The data distributions of the three datasets is visualized in Appendix B .

Summary:  This paper conducts an empirical study on the catastrophic forgetting of pretrained language models. On two continual learning settings (class incremental and task incremental), the paper evaluates multiple pre-trained models on different data sets, to see how severe the catastrophic forgetting issue is for these pre-trained models. Then the paper also tests the effectiveness of multiple continual learning methods on such pre-trained models and draws some conclusions.