Pretrained Language Model in Continual Learning: A Comparative Study

1 INTRODUCTION . Continual Learning ( CL ) methods aim at training a model from a stream of non-i.i.d . samples , relieving catastrophic forgetting ( CF ) while limiting computational costs and memory footprint . Throughout the years , many methods have been proposed to address the CL problem in computer vision and robotics ( Kirkpatrick et al. , 2017 ; Serrà et al. , 2018 ; Buzzega et al. , 2020 ) . CL in NLP is still a nascent topic , as reflected by the relatively smaller number of proposed methods ( Biesialska et al. , 2020 ) . However , it is not always easy to precisely measure the merits of these works . This is partly due to the subtle differences in the way methods are evaluated : many state-of-the-art approaches only stand out in the setting where they were originally conceived . Moreover , pretrained language models ( PLMs ) have been widely applied in CL methods in NLP , and their addition further complicates the clear understanding of model performance ( Han et al. , 2020 ; Wang et al. , 2019 ; Cao et al. , 2020 ) . Although many works that apply PLM in CL point out the CF problem present in PLMs , there remain three significant issues that require further investigation . ( 1 ) Many existing CL works employ specific PLMs such as BERT ( Devlin et al. , 2019 ; Rogers et al. , 2020 ) , whereas more recent and sophisticated PLM structures , e.g . ALBERT ( Lan et al. , 2020 ) and XLNET ( Yang et al. , 2019 ) , have been developed afterwards . A natural question would be whether the later PLMs could better mitigate CF in continual learning ( Mosbach et al. , 2021 ; Lee et al. , 2020a ) . ( 2 ) The existing works focus on input , output , and gradients information of PLMs while ignoring the effect of model architecture on CF ( Biesialska et al. , 2020 ) . We believe opening up the black-box of PLMs may lead to a deep understanding of their characteristics and thus better algorithmic design ( Wallat et al. , 2020 ) . ( 3 ) Some existing works focus on alleviating forgetting in a trivial setting , during knowledge transfer between two tasks , which is not realistic in general continual learning with more tasks or even without task boundary ( Jiang et al. , 2020 ) . In this paper , we conduct an in-depth exploration of continual learning through extensive empirical analysis of a number of PLMs and CL methods . We analyze not only the performance differences across the combinations of PLMs and CL methods , but also the idiosyncrasies of each PLM . Our main contributions are as follows : • We design a rigorous benchmark for comprehensive study of continual learning in NLP . We conduct experiments over ( 1 ) two primary continual learning setting , including taskincremental learning and class-incremental learning ; ( 2 ) three benchmark datasets with different data distributions and task definitions , including relation extraction , event classification , and intent detection ; ( 3 ) four CL approaches with six baseline methods implemented for systematic comparison ; and ( 4 ) five pretrained language models . • We evaluate and contrast the performance of the above combinations of settings , PLMs and CL methods , providing a comprehensive comparative study from a number of perspectives . • We dissect the performance characteristics of different PLMs with a number of layer-wise probing analyses . • Our observations and insights give rise to a number of open research questions that can guide the design and optimization of better PLM-oriented continual learning methods and CL-oriented pretraining strategies . • To encourage more research on continual learning in NLP , we will release the code and dataset as an open-access resource . 2 BACKGROUND . In this section , we provide the required background on pre-trained language models as well as continual-learning settings , methods and evaluation metrics . 2.1 PRETRAINED LANGUAGE MODELS . It now become the best practice to incorporate PLMs into NLP systems for many problems such as question answering , machine reading comprehension , summarization , to name a few . Usually , making use of PLMs in such systems leads to significant performance gains in ( weakly ) supervised learning . In this work , we investigate the use of PLMs in continual learning . We now briefly introduce five typical PLMs that are evaluated in our work . BERT ( Devlin et al. , 2019 ) is the most representative PLM , which uses bi-directional deep Transformers ( Vaswani et al. , 2017 ) as the backbone . BERT adopts the Masked Language Modeling ( MLM ) and the Next Sentence Prediction ( NSP ) as the self-supervision tasks for pre-training . ALBERT ( Lan et al. , 2020 ) ALBERT is a lite version of BERT which sets the parameters of all Transformer blocks shared across all layers and factorizes the embedding matrices into two small size of matrices . Instead of NSP , ALBERT predicts the order of two consecutive textual segments . Although ALBERT utilizes significantly less memory , fine-tuning it on downstream tasks can achieve close performance to that of BERT . RoBERTa ( Liu et al. , 2019 ) RoBERTa has almost the same architecture as BERT , while it differs in terms of three training procedures . RoBERTa removes the NSP loss , trains a model with bigger size and longer sequences , and creates dynamic MLM masks vs static ones used in BERT . GPT2 ( Radford et al . ) Unlike the aforementioned PLMs which are all masked language models , GPT2 is an autoregressive language model predicting one token at a time from left to right . GPT2 is often used for natural language generation , while the aforementioned PLMs are mostly used for natural language understanding . XLNET ( Yang et al. , 2019 ) Unlike the masked language models , GPT2 can not utilize the context from the backward side . XLNET is an autoregressive language model as well , while it resolves this problem by adopting a new objective called Permutation Language Modeling , enabling the model to take advantage of both forward and backward contexts . 2.2 CONTINUAL LEARNING SETTINGS . Continual learning ( CL ) focuses on the development of learning algorithms able to accumulate knowledge on non-stationary data . CL approaches are benchmarked on their ability to learn a se- quence of tasks without forgetting previously acquired knowledge . They are typically evaluated on incremental classification settings . Some popular incremental learning scenarios in CL ( van de Ven & Tolias , 2019 ; Zeno et al. , 2018 ) include : class-incremental learning , domain-incremental learning , and task-incremental learning . In the training time of these incremental learning settings , a CL algorithm experiences each task sequentially ( only once ) , and is informed about the distribution shift ( aka the task boundary ) . They , however , differ in their assumptions about the evaluation . In task-incremental learning , the model relies on the task identity ( or a task label ) to make its prediction . Conversely , in class-incremental learning , the methods have to perform task inference , implicitly ( Aljundi , 2019 ) or explicitly ( Lee et al. , 2020b ) . We will cover both settings in our study . A lesser known setting , namely Domainincremental learning , also relaxes the task-ID dependence , but shares the same output head for each task . For more details on these settings , see ( van de Ven & Tolias , 2019 ) . 2.3 CONTINUAL LEARNING APPROACHES . Many continual-learning approaches have been proposed recently , e.g. , see Delange et al . ( 2021 ) for an overview , which can be categorised as follows : Rehearsal-based Approach The simplest way to reduce forgetting is to store samples from the past and reuse them to complement the learning of new tasks . In its simplest form , old samples are replayed with new data . This strategy , known as experience replay ( ER ) , ( Rolnick et al. , 2019 ) is often a hard to beat baseline . Numerous rehearsal-based methods have been developed to increase ER ’ s performance or efficiency ( Aljundi et al. , 2019a ; Caccia et al. , 2019 ; Hayes et al. , 2018 ) . Instead of replays , some methods use the old sample to perform constrained optimization to prevent increasing the loss on old tasks ( Lopez-Paz & Ranzato , 2017 ; Aljundi et al. , 2019b ) . Regularization-based Approach Also know as prior-based approaches , these methods prevent significant updates to the parameters that are deemed important for the previous tasks . Their first instantiation appeared in the Elastic-Weight Consolidation ( EWC ) ( Kirkpatrick et al. , 2017 ) , where the previously learned weights are restrained from drifting via an L2 regularization loss . Regularizationbased strategy often rely on the task boundaries to consolidate their knowledge during training . They often fail on long tasks sequences or settings where the task identity is not observable ( Farquhar & Gal , 2018 ; Lesort et al. , 2019b ; Chaudhry et al. , 2018 ) . Despite these findings , prior-focused methods are actively researched ( Zeno et al. , 2018 ; Nguyen et al. , 2018 ) . Dynamic Architecture Approach Also know as parameter-isolation methods , this family of algorithm alleviates forgetting by using different subset of parameters for fitting different tasks . One popular approach , called Hard Attention to the Task ( HAT ) ( Serrà et al. , 2018 ) , achieves parameter freezing through an attention mask that is learned concurrently at every tasks . Other similar strategies have also been proposed ( Yoon et al. , 2017 ; Schwarz et al. , 2018 ) . Similarly to regularizationbased methods , dynamic architectures usually assume the availability of test-time task labels , and thus are not straightforwardly applicable in more realistic settings . Further approaches are detailed in Appendix A . 3 BENCHMARKING CONTINUAL LEARNING OF PLMS . In this section , we first compare the performance of different combinations of CL methods and PLMs on three benchmark datasets , and we explore the following three research questions : ( 1 ) Does the catastrophic forgetting problem exist in PLMs during continual learning ? ( 2 ) Which continual learning approach is the most efficient for PLMs and why ? ( 3 ) Which PLM is the most robust for continual learning and why ? 3.1 EXPERIMENTAL SETUP . Methods . We adopt 5 representative PLMs for evaluation , i.e. , ALBERT ( Lan et al. , 2020 ) , BERT ( Devlin et al. , 2019 ) , GPT2 ( Radford et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and XLNET ( Yang et al. , 2019 ) . By combining a PLM and a linear classifier as the backbone model , we investigate the following 4 approaches for the comparative study : ( 1 ) Vanilla uses the model learned on previous tasks as initialization and then optimizes the parameters for the current task . This baseline greedily trains the model on each task without accessing data from previous tasks , and is thus severely prone to catastrophic forgetting . It serves as a weak lower bound in terms of the average accuracy . ( 2 ) Joint trains on the entire training set simultaneously in the conventional supervised learning setup . Therefore , Joint does not suffer from forgetting and represents the performance upper bound . ( 3 ) EWC ( Kirkpatrick et al. , 2017 ) is a regularization-based methods , extending the loss function with a regularization term that slows down the updates of the important network weights . ( 4 ) HAT ( Serrà et al. , 2018 ) is a dynamic architecture method , employing a heuristic strategy to prevent intransigence by allocating additional units to the network when needed . ( 5 ) ER is a rehearsal-based method , interleaving old samples with current data in training batches . ( 6 ) DERPP ( Buzzega et al. , 2020 ) is a hybrid method , combining the strategy of rehearsal and regularization which prevents the prediction logits of memorized samples from changing . Following DERPP ( Buzzega et al. , 2020 ) , we simulate the sequence of tasks according to the class-incremental learning setting ( i.e . Class-IL or CIL ) and task-incremental learning setting ( i.e . Task-IL or TIL ) , splitting the training dataset into partitions of classes/tasks . Training details and hyper-pearameters are in Appendix C Metrics . To measure the FWT and BWT abilities of the CL models , we assume access to a test set for each task . After the model learning on the training set of task t , we present the evaluation results on all T tasks . We adopt three evaluation metrics as in ( Lopez-Paz & Ranzato , 2017 ) : ( 1 ) Forward transfer FWT = 1T−1 ∑T−1 i=2 AT , i − b̃i ; ( 2 ) Backward Transfer BWT = 1 T−1 ∑T−1 i=1 AT , i − Ai , i ; And average accruacy Avg . ACC = 1T ∑T i=1AT , i , where At , i is the accuracy of models on the test set of ith task after model learning on the tth task and b̃i is the test accuracy for task i at random initialization . In addition to FWT and BWT , we consider average accuracy as a key performance measure , measuring the accuracy of past tasks after the model has moved on to learning new tasks . Datasets . We evaluate our methods on 3 datasets with distinct label distributions , covering the following domains . CLINC150 ( Larson et al. , 2019 ) is an intent classification dataset with labels evenly distributed , including 150 classes and 100 instances per class . Maven ( Wang et al. , 2020 ) is a long-tailed event detection dataset with 163 classes , each of which is labelled at least 15 instances , reaching a total of 47,921 instances . WebRED ( Ormandi et al. , 2021 ) is a severely long-tailed relation classification dataset with 243 classes , each of which is labelled with at least 15 instances , reaching a total of 31,441 instances . Adhering to ( Wu et al. , 2021 ) , we randomly split each dataset into disjoint tasks , and each tasks contains 10 classes . For each class , we randomly split the dataset set into train , validation and test set by 10:2:3 . To reduce the amount of computation , we cap the number of instances per class for both Maven and WebRED to 1000 . The data distributions of the three datasets is visualized in Appendix B .

Summary: The authors perform a comprehensive study of how pretrained language models work in the continual learning setting. The authors study 5 relevant pretrained language models (masked and unmasked) and somewhere between 3 and 6 continual learning strategies depending on where in the paper they are counted. In addition to a thorough everything-by-everything evaluation, the authors hone in on the details of how the different models and CL approaches are reflected in the transformer layers. The authors find that the different language models studied perform relatively differently, both qualitatively and quantitatively, and these insights may provide useful for directing future improvements.