Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models (2024)

Abstract: Incremental Learning (IL) has been a longstanding problem in both vision and Natural Language Processing (NLP) communities. In recent years, as Pre-trained Language Models (PLMs) have achieved remarkable progress in various NLP downstream tasks, utilizing PLMs as backbones has become a common practice in recent research of IL in NLP. Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance and propose various techniques to overcome this issue. However, we find that this assumption is problematic. Specifically, we revisit more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (ClassIncremental and Task-Incremental) and reveal that most of them severely underestimate the inherent anti-forgetting ability of PLMs. Based on the observation, we propose a frustratingly easy method called SEQ* for IL with PLMs. The results show that SEQ* has competitive or superior performance compared with state-ofthe-art (SOTA) IL methods yet requires considerably less trainable parameters and training time. These findings urge us to revisit the IL with PLMs and encourage future studies to have a fundamental understanding of the catastrophic forgetting in PLMs. The data, code and scripts are publicly available 1.

heading: 1 Introduction
text: Learning knowledge incrementally without much forgetting is an essential ability of human beings but still an unsolved challenge for neural networks in achieving human-level intelligence (French, 1999). Incrementally learning a sequence
∗*Corresponding author 1https://github.com/zzz47zzz/codebase-for-incremental-
learning-with-llm and https://github.com/qianlimalab/codebase-for-incremental-learning-with-llm
of tasks can be formulated into the paradigm of Incremental Learning (IL) and has been impeded by catastrophic forgetting (Kirkpatrick et al., 2017). Catastrophic forgetting refers to neural networks forgetting previous knowledge after learning new tasks (McCloskey and Cohen, 1989).
Recent years have witnessed significant breakthroughs in Pre-trained Language Models (PLMs) in vision and NLP tasks. Most recent studies of IL use PLMs as the backbone and design various methods for alleviating catastrophic forgetting in NLP tasks. However, is forgetting really catastrophic in PLMs? More specifically, how can we quantify forgetting and how much knowledge is forgotten in various IL scenarios when using various backbones and methods on various tasks? More recently, (Tao et al., 2023) reveal for the first time that BERT-like models have a strong anti-forgetting ability in the task-incremental setting. Why does this happen? Does it hold for a more challenging setting, such
14848
as class-incremental learning, and for other model architectures, such as GPT-like models?
To answer the above questions, we carry out extensive experiments to explore forgetting in more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (ClassIncremental and Task-Incremental) with various model architecture (encoder only and decoder only) and scales (from 19M to 1.21B number of parameters). Through extensive experiments, we have several major findings:
• The popular assumption that PLMs suffer from catastrophic forgetting does not hold. Even under sequential fine-tuning (SEQ), the PLMs maintain the knowledge without much forgetting (Sec. 3.2). From the probing perspective, most existing IL methods do not learn incremental knowledge for PLMs (Sec. 4.2).
• By combining SEQ with simple strategies (Sec. 4.1), we propose SEQ* and find that SEQ* has competitive or even superior performance than SOTA IL methods (Figure 1, Sec. 4.2).
• The inherent anti-forgetting ability of PLMs comes from both the pre-training stage as well as the architecture of Transformer (Sec. 3.4). Randomly initialised PLMs learn incrementally when SEQ is performed on a sequence of tasks.
• The forgetting of SEQ is due to the deviation of the classifier from the PLM rather than the loss of old knowledge in the PLM. (Sec. 3.5).
Our study urges the NLP community to revisit and deepen the understanding of the forgetting in PLMs.

heading: 2 Experimental Settings

heading: 2.1 Problem formulation
text: Formally, the goal of IL is to learn a model fθ : x → y ∈ Y from a sequence of tasks D = {D1,D2, · · · ,DT }, where the t-th task Dt = {(xti, yti)}i=1 contains input samples xti ∈ Xt and labels yti ∈ Yt. In Class-Incremental Learning (CIL), the label sets of different tasks are exclusive: Y1 ∩ Y2 · · · YT = ∅, and the task id is unknown
during inference. In Task-Incremental Learning (TIL), the label sets of different tasks may be overlapping: Y1 ∩ Y2 · · · YT ̸= ∅, and the task id is required during inference. In general, CIL is much more challenging than TIL because PLMs suffer from inter-task forgetting much more seriously than intra-task forgetting (Tao et al., 2023). Appendix A provides detailed description and evaluation metrics.

heading: 2.2 Tasks and Datasets
text: We consider four types of downstream tasks in our experiments: Text classification, intent classification, relation extraction, and named entity recognition. We use the following eight datasets: Topic3Datasets (containing AGNews, DBPedia, and YaHoo (Zhang et al., 2015)) for text classification; CLINC150 (Larson et al., 2019) and Backing77 (Casanueva et al., 2020) for intent classification; FewRel (Han et al., 2018) and TACRED (Zhang et al., 2017) for relation extraction; OntoNotes5 (Hovy et al., 2006), I2B2 (Murphy et al., 2010), Few-NERD (Ding et al., 2021) for named entity recognition. Detailed descriptions are provided in Appendix B.

heading: 2.3 Backbones
text: We consider two popular architectures as backbone PLMs: encoder-only and decoder-only. For encoder-only backbones, we use bert-base-cased and bert-large-cased (Devlin et al., 2019), the most popular choices in previous IL studies. The encoder-only backbones are typically used as discriminant models, and linear layers are added for downstream tasks. We use the GPT2 (Radford et al., 2019) and Pythia suite (Biderman et al., 2023) for decoder-only backbones. Pythia is based on GPT-NeoX (Black et al., 2022), which contains 8 model sizes and 154 pre-training checkpoints, enabling research in interpretability and learning dynamics. The decoder-only backbones are typically used as generative models, and no additional linear layers are required since the output target is natural language. The detailed description is provided in Appendix C.

heading: 3 Revisiting the Forgetting from the Probing Perspective

heading: 3.1 How to Measure the Forgetting in PLMs?
text: This subsection describes how to measure the forgetting inside PLMs during IL. Specifically, we
utilize the probing technique, an effective method to evaluate the representation ability of backbones on target tasks (Chen et al., 2023a; Tao et al., 2023; Davari et al., 2022; Wu et al., 2021).
To probe the knowledge in PLMs for all tasks in IL, we add probing classifiers on top of the PLM and train the probing classifiers on all tasks in IL. Then, we evaluate the PLM and the probing classifiers on all tasks and obtain the probing performance. The probing performance is the upper bound performance when the classifiers do not forget. For clarity, the performance evaluated with the original model is called the observed performance. We note that measuring probing performance will not affect the training process of IL since the backbone PLM is frozen when training probing classifiers. Furthermore, the original classifiers only predict the classes of learned tasks, while the probing classifiers predict the classes of all tasks in IL. We provide an illustration in Figure 2 and the formal definition in Appendix A.
We consider four metrics for our probing study: linear probing, cosine linear probing, prototype probing, and cosine prototyping. Linear probing has been widely adopted in previous works. In linear probing, the probing classifier is a linear layer. Cosine linear probing adopts a cosine linear layer as the probing classifier. Specifically, the logits are computed as the cosine similarities between classifier weights and extracted features. Hou et al. (2019) show that utilizing cosine linear layers mitigates bias towards new classes in IL. In prototype probing, the probing classifier is a linear layer
whose weight matrix is calculated as the class feature centres. Previous IL studies (Zhou et al., 2023; Chen et al., 2023b; Ma et al., 2023) show that using class feature centres as prototypes for classification is effective. Cosine prototype probing further utilizes cosine normalization when calculating logits. Further discussion is provided in Appendix D.1.

heading: 3.2 Is Sequential Fine-tuning Really the Lower Bound?
text: Sequential fine-tuning (SEQ) has long been regarded as the lower bound of IL. In this subsection, we revisit SEQ from the probing perspective, and we find that SEQ is severely underestimated when using PLMs for IL.
The backbone was small and randomly initialized in early studies exploring IL (Kirkpatrick et al., 2017; French, 1999; McCloskey and Cohen, 1989). They find that SEQ usually results in models forgetting all previous knowledge when learning new tasks. Recent IL studies in NLP (Razdaibiedina et al., 2023; Zheng et al., 2024a; Huang et al., 2021; Sun et al., 2019; Qiu et al., 2024) also observe that SEQ leads to worse performance. However, in the era of PLMs, fine-tuning has proven to be effective for adapting PLMs to different domains or downstream tasks (Aghajanyan et al., 2020; Devlin et al., 2019; Radford et al., 2018; Zheng et al., 2023). If fine-tuning really causes PLMs to forget nearly all previous knowledge in IL, it should also cause PLMs to forget all pre-trained knowledge when adapting to new tasks. Obviously, this assumption is not true since fine-tuning is still effective
for PLMs (OpenAI, 2023). Furthermore, prior research (Tao et al., 2023) also finds that BERT-like models suffer from little forgetting under the taskincremental setting by the probing study.
The observed and probing performance on classincremental intent classification with generative models are summarized in Figure 3. The results on other IL settings, downstream tasks and backbones are in Appendix D.2. Figure 3a shows that the observed performance drops dramatically from approximately 98% to 10% as more new tasks are learned, in line with our understanding of catastrophic forgetting. However, Figure 3c describes an entirely different phenomenon. The PLMs achieve high probing performance after learning the first task. And the linear probing performance has barely decreased since the second task. In other words, PLMs preserve the knowledge to classify all 15 tasks even when adapting to only new tasks sequentially. This phenomenon is contradictory to
what we know about catastrophic forgetting and SEQ.
Indeed, the probing performance is high since all tasks’ data is available when training the probing classifiers, while the observed performance is poor since the original classifiers only train on the data from the current task. However, with the above observation, we can boost the observed performance with simple strategies, which will be described in Section 4.1.

heading: 3.3 What is the Best Metric for Probing Performance?
text: In Figure 3, we find that the ranking of four probing metrics is as follows: linear > cosine linear, cosine prototype > prototype probing. This subsection will explain why linear probing is the best metric for probing study.
First, we need to understand what the features (i.e., last hidden states), word embeddings of PLMs, and the class embeddings in probing classifiers “look like” respectively. The detailed description is in Appendix D.3. The histograms of the L2 norm and the cosine similarity of features, word embeddings and class embeddings are in Figure 4. Figure 4a shows that the features occupy a nar-
row cone in the vector space rather than being uniform in all directions, which has been discussed in (Ethayarajh, 2019). More surprisingly, Figure 4b shows that the learned (output) word embeddings are nearly orthogonal to the features. We infer that the cross-entropy loss encourages all word embeddings except the ground truth one to become farther away from the feature during pre-training. In other words, the cross-entropy loss encourages a large difference in logits, and the word embeddings to be orthogonal to the features in order to distinguish logits better. Therefore, it is not surprising that linear probing has the best performance, considering that the word embedding layer is essentially a linear layer. From this point of view, it is also not surprising that the performance of prototype probing is poor since the prototypes (class feature centres) also fall in the narrow cone space, and it is not an optimal solution for distinguishing logits.
Then, why does cosine normalization degrade the performance of linear probing but improve prototype probing? Figure 4c and 4d are the L2 norm of the features and word embedings. We find that the norm of word embeddings has a larger discrepancy than features. It indicates that the norm of word embeddings contains the prior knowledge obtained from pre-training. Therefore, the cosine linear probing ignores the difference in the norm of features and thus has poorer performance compared with linear probing. For prototype probing, the prototype falls in a narrow cone space, and the similarity between the prototype and features is large and close to each other. In this case, cosine normalization can eliminate the interference of the norm and establish the relationship between logits and cosine similarity between features. We provide the detailed analysis and full results with different backbones in Appendix D.3. An illustration of different types of probing metrics is in Figure 7.

heading: 3.4 What is the Role of Pre-training in IL?
text: In this subsection, we reveal that the key to the antiforgetting ability of PLMs lies in both the Transformers’ architecture and the pre-training knowledge.
We evaluate the linear probing performance on checkpoints with a different number of pre-training steps: {0,16,128,1k,10k,143k(final)}. We load the pre-trained checkpoints (or randomly-initialized checkpoints at step 0) and evaluate their linear probing performance before and after IL using SEQ. Figure 5 shows two main phases in pre-training:
overfitting and generalization. In the first phase (step 0 - step 128), the model memorizes the pretraining corpus, and the linear probing performance decreases. In the second phase (step 1k - step 143k), the model gradually learns the pre-training knowledge and the linear probing performance increases. However, when the model further generalizes to the pre-training corpus (step 10k - step 143k), the linear probing performance of small backbones (Pythia-70 m and 160m) decreases again due to the gap between pre-training and downstream tasks. This gap can be eliminated when adapting to downstream tasks (Figure 5a and Figure 3c). For larger backbones (Pythia-410 m, 1b, and 1.4b), the model can be adapted to new tasks directly without this gap.
Besides, we have the following interesting findings: (1) Pre-training indeed improves the linear probing performance in IL (Figure 5b and 5d). (2) Apart from pre-training, the architecture of the Transformer is also a key factor in the high linear probing accuracy during SEQ. When the downstream task is relatively simple, such as intent classification, even the randomly-initialized models achieve high linear probing performance (Figure 5b). Pre-training brings considerable improvements when the downstream task is more complex, such as relation extraction (Figure 5d). (3) More surprisingly, SEQ improves the linear probing per-
formance of models from nearly all pre-training steps (Figure 5a v.s. 5b; Figure 5c v.s. 5d). This shows that Transformers’ architecture can incrementally absorb new knowledge even when just sequential fine-tuning on new tasks. The detailed settings, visualization of features, and additional results on text classification are provided in Appendix E.

heading: 3.5 What is really forgotten in SEQ?
text: As discussed in Sec. 3.2, SEQ’s linear probing performance showed little degradation or even improvement across all settings. Therefore, the reason for forgetting must lie in the classifier. In this subsection, we will take a closer look at the forgetting in classifiers.
Existing studies (Wu et al., 2019; Hou et al., 2019; Zheng et al., 2024b) find that the model tends to predict new classes during IL. They refer this phenomenon as the class imbalance problem between old and new classes. In SEQ, the class imbalance problem is more severe since only new classes are learned. We also observe that the logits of new classes are much larger than those of old classes in a SEQ model. Because both features and class embeddings determine the magnitude of logits, the features occupy a narrow cone space, and their norms are relatively close, we can infer that the forgetting is caused by either (1) the norm
of class embeddings or (2) the cosine similarity between features and class embeddings.
For the first reason (i.e., class norm), we compare the class embedding norm between learned linear classifiers and linear probing classifiers in Figure 6a and 6b. Surprisingly, the class embedding norm of new tasks is not larger than those of old tasks in the observed classifier of SEQ. It indicates that the class norm is not the primary reason for the forgetting in SEQ.
For the second reason (i.e., cosine similarity), we compare the moving distance of class embeddings between the observed and probing classifiers in Figure 6c and 6d. The moving distance of the class embeddings of task t at task t+k is computed as follows: (1) When the model finishes training on task t, we compute the cosine distance between all pairs of class embeddings from task t and class feature centres from all tasks and obtain a cosine similarity matrix Ctt. (2) When model finishes training on task t+ k, we compute the cosine distance between all pairs of class embeddings from task t and class feature centres from all tasks and obtain a cosine similarity matrix Ctt+k. (3) Then, the moving distance of task t’s class embeddings is calculated as the average absolute difference between the cosine similarity matrix Ctt and C t t+k. The moving distance measures how the class embeddings move relatively to all class feature centres since they have been learned. If the classifier does not forget a class, the distance from its class embeddings to all class feature centres should remain constant. In other words, its moving distance will be zero if the classifier does not forget how to classify this class with the features extracted by PLMs. We provide an illustration in Figure 25. The detailed settings, definition, and additional results with frozen bert-large-cased are provided in Appendix F.
Figure 6c and 6d show that the class embeddings of observed classifiers change significantly compared with those of probing classifiers. It indicates that the forgetting happens because the old class embeddings are pushed away from their initial and optimal position. The cosine similarity matrices are visualized in Figure 26.

heading: 4 Revisiting the SOTA Methods in IL

heading: 4.1 SEQ*: Boosting the Performance of SEQ
text: In this subsection, we propose SEQ* based on the findings about the forgetting in SEQ.
In the previous section, we have the following findings about SEQ: (F1) The PLMs do not learn new knowledge in SEQ about the downstream IL tasks; (F2) The PLMs achieve the highest probing performance once being adapted to downstream tasks, and there is little performance degradation when learning on more new tasks (See Figure 3c and Figure 5a); (F3) The classifier forgets dramatically, while the PLMs do not. The reason is that the class embeddings are pushed away from the initial learned optimal position.
Therefore, we propose the following strategies for closing the gap between the probing and observed performance in SEQ: (S1) Freeze the PLMs after warm-up; (S2) Freeze the old classifiers when learning new tasks; (S3) Use cosine linear classifiers only when no old data is available in a CIL scenario. Otherwise, use linear classifiers; (S4, optional) Pre-allocate future classifiers. We call the method with the above strategies as SEQ*, and an illustration is provided in Figure 27.
The rationale for the above strategies is as follows: (S1) is proposed according to (F1). Furthermore, we propose to warm up (i.e., full-parameter fine-tuning) PLMs in only the first task according to (F2). In practice, warm-up onlt for 1-3 epochs brings considerable improvement across backbones and datasets. (S1) and (S2) preserve the relative position of class embeddings with respective to class feature centres to avoid the issue in (F3). When both PLMs and classifiers of old tasks are frozen, only the norm of new class embeddings may lead to the biased prediction towards new classes. Because cosine linear layers are not optimal for exploiting PLMs’ knowledge, we propose (S3) to avoid bias prediction. In other words, we use linear classifiers for the TIL scenario and the CIL scenario where old data is stored. Finally, we propose (S4) for better forward compatibility (Zhou et al., 2022). (S4) is marked as an optional strategy since it requires additional information on the number of total tasks. Therefore, we report the two variants of SEQ*, i.e., w/ and w/o (S4), when comparing with SOTA methods. We provide detailed discussion and explanation in Appendix G.

heading: 4.2 Comparing SOTA methods with SEQ*
text: In this subsection, we evaluate SEQ* under extensive settings. Despite its simplicity, SEQ* has competitive or even superior performance in most settings.
We provide the result under the CIL scenario in
Figure 1, Table 1 and 2 in the main manuscript due to the space limitation. We provide the introduction and training details of SOTA methods in Appendix H. The full results on other backbones, datasets, and IL settings are summarized in Table 7, 8, 10, 11, 12, 13, 9, 15 in Appendix I. All baselines and SEQ* use the same backbone PLM for IL. In all settings except for sentence-level classification tasks with discriminant backbones, SEQ* and all baselines store no old samples.
From the results in Table 1 and 2, we have
the following findings: (1) SEQ* shows comparable or better performance on all datasets. Using proper classifiers, fixing old classifiers, and preallocating future classifiers improve SEQ significantly. We highlight that we do not aim to show SEQ* achieves SOTA performance across all settings. Instead, we aim to demonstrate that SEQ* serves as a comparable baseline in most IL settings and should be considered in further IL studies.
(2) SEQ* does not perform best when the PLM is required to absorb new knowledge, or there are overlaps between new and old tasks. For example, Few-NERD contains fine-grained entities, such as “Airport” and “Hotel”, which PLMs may not have seen during pre-training. In Topic3Datasets, “Sci/Tech” and “Computers & Internet” belong to two different tasks. Intuitively, the PLM need to adjust the class boundary to avoid overlapping between classes.
Furthermore, we compare the linear probing performance between SEQ* and SOTA methods in Figure 3 and 14. The results show that the difference in the linear probing performance is small compared with the observed performance. The improvement between “BeforeIL” and “AfterIL” mainly comes from the adaptation from pretraining to downstream tasks (Figure 3,9,11,12). It explains why freezing PLMs after warm-up is effective. It also explains why prompt-based methods (Razdaibiedina et al., 2023; Wang et al., 2022) are effective even if only a tiny portion of the parameters are learned. Furthermore, the performance gap between SEQ* and linear probing performance still exists. The reason is that backward knowledge transfer from new tasks to old tasks is prohibited in SEQ*.
We compare the training time and the number of trainable parameters between SEQ* and SOTA methods in Table 4. SEQ* requires much less training time and trainable parameters for each task.

heading: 5 Related Work
text: Previous studies have assessed catastrophic forgetting by measuring performance degradation on old tasks. However, there is limited understanding of probing performance in incremental learning. Davari et al. (2022) use linear probing to reveal that representations still experience significant drift due to parameter updates. Wu et al. (2021) conduct layer-wise probing studies on BERT, revealing catastrophic forgetting in the top and middle layers. They observed that, although BERT maintains high representational ability at the last incremental step, the classifier loses the ability to classify previously learned classes. Chen et al. (2023a) conduct linear probing on k-shot samples from the next task, revealing a strong correlation between retaining past information and learning efficiency on new tasks. Tao et al. (2023) utilize linear probing to illustrate that BERT is inherently resilient to catastrophic forgetting, even without buffer data in task-incremental learning. In this study, we further investigate the influence of probing metrics, backbone and classifier architecture, pre-training steps, datasets, and IL methods on the probing performance. Furthermore, we analyze the forgetting of classifiers from the perspective of norm and cosine similarity. Finally, we propose simple but effective strategies for SEQ and conduct extensive experiments to validate its effectiveness.

heading: 6 Conclusion
text: Incremental learning is a key pillar of human cognition and intelligence. As PLMs have become popular in recent years, more and more IL studies adopt PLMs as the backbone model. However, we reveal that existing studies ignore the inherent anti-forgetting of PLMs and design methods based on a problematic assumption. Our findings encourage the IL community to revisit the assumption of catastrophic forgetting in PLMs and re-evaluate the
proposed IL algorithms by comparing them with frozen-based methods such as SEQ*.
We suggest two future directions for IL with PLMs: (1) design IL benchmark where domainspecific knowledge is required; (2) design IL algorithms that update the knowledge of PLMs with limited time, computation cost and memory budget.

heading: Limitations
text: There are two limitations of this study: (1) We only focused on the IL of classification tasks and did not explore the forgetting of general forms of knowledge in PLMs; (2) We did not fully understand the internal mechanism of how PLMs incrementally learn the knowledge under SEQ.

heading: Ethical Considerations
text: The ethical considerations of our research are carefully addressed to ensure compliance with relevant standards and transparency. To this end, we provide the following clarifications for reproducibility:
• We provide a detailed setting of our experiments.
• The source code, data, and scripts will all be publicly available.
• Our findings are in alignment with observed empirical outcomes.

heading: Acknowledgements
text: We thank the anonymous reviewers for their helpful feedbacks.The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant No. 62272173), the Natural Science Foundation of Guangdong Province (Grant Nos. 2024A1515010089, 2022A1515010179), the Science and Technology Planning Project of Guangdong Province (Grant No. 2023A0505050106), and the National Key R&D Program of China (Grant No. 2023YFA1011601).

heading: Appendix

heading: A Problem Formulation 13
text: A.1 Overview of IL . . . . . . . . . . 13 A.2 Evaluation Metric for IL . . . . . 13

heading: B Datasets 14

heading: C Backbones 14
text: C.1 Discriminant Backbones . . . . . 14 C.2 Generative Backbones . . . . . . 15

heading: D Revisiting IL with Probing Study 15
text: D.1 Four Probing Metrics . . . . . . . 15 D.2 Probing Performance with Differ-
ent Backbones . . . . . . . . . . . 15 D.3 A Closer Look at Features, Word
Embeddings, and Class Embeddings 16

heading: E The Role of Pre-training 19

heading: F The Forgetting in Classifiers 21
text: F.1 Overview of the Forgetting in Clas-
sifiers . . . . . . . . . . . . . . . 21 F.2 The Bias in the L2 Norm of Class Embeddings . . . . . . . . . . . . 22 F.3 The Bias in the Cosine Similarity
between Features and Class Embeddings . . . . . . . . . . . . . . 22

heading: G SEQ*: Boosting Performance of SEQ 23
text: H Introduction of Baselines 24

heading: I Revisiting SOTA Methods 25
text: I.1 CIL with Generative Backbones
for Sentence-Level Tasks . . . . . 25 I.2 CIL with Discriminant Backbones for Sentence-Level Tasks . . . . . 26 I.3 CIL with Discriminant Backbones for Word-Level Tasks . . . . . . . 26 I.4 TIL for Sentence-Level Tasks . . . 26

heading: A Problem Formulation

heading: A.1 Overview of IL
text: Incremental Learning (IL) aims to learn a model on new tasks incrementally without forgetting previous knowledge. In this paper, we only consider classification tasks, which are popular and challenging settings in existing studies. Formally, IL aims to learn a model fθ : x → y ∈ Y from the sequence of tasks D = {D1,D2, · · · ,DT }, where
the t-th task Dt = {(xti, yti)}i=1 contains input samples xti ∈ Xt and labels yti ∈ Yt. There are three popular scenarios in IL: Class-Incremental Learning (CIL), Task-Incremental Learning (TIL), and Domain-Incremental Learning (DIL). In CIL, the label sets of different tasks are exclusive: Y1 ∩ Y2 · · · YT = ∅, and the task identity is unknown during inference. In TIL, the label sets of different tasks may be overlapping: Y1∩Y2 · · · YT ̸= ∅, and the task identity is required during inference. In DIL, the label sets of different tasks are the same: Y1 = Y2 = YT . Under the data replay setting, a buffer M is introduced for storing old representative instances. In the main experiments of this research, we consider the most challenging scenario, CIL, where catastrophic forgetting occurs most severely. The result on TIL is also reported when compared with state-of-the-art methods.

heading: A.2 Evaluation Metric for IL
text: We adopt average accuracy (Chaudhry et al., 2018) as the metric for evaluation. Specifically, the average accuracy at task t is defined as the following
At = 1
t
t∑
i=1
at,i, (1)
where at,i represents the accuracy evaluated on the test set of task i after training the model incrementally from tasks 1 to t. The average accuracy indicates the performance on all learnt tasks. In the main manuscript, we report the average accuracy after learning the final task, i.e., AT . Besides, we report the average incremental accuracy Ā in the appendix. The average incremental accuracy is computed as follows:
Ā = 1 T
T∑
t=1
At (2)
It indicates the average of the average accuracy over all incremental steps.
We note that the probing accuracy is calculated as the average of the test accuracy on all T tasks:
Aprob = 1
T
T∑
i=1
aprob,i, (3)
aprob,i represents the accuracy evaluated on the test set of task i after training probing classifiers. The probing accuracy is the performance when the classifier is optimal. According to the probing
experiments in Sec. 3, the PLMs almost do not forget. Therefore, the probing performance can be regarded as the upper bound performance when using PLMs for IL.

heading: B Datasets
text: The statistics of the eight datasets are summarized in Table 5. For text classification, we construct Topic3Datasets from AGNews, DBPedia, and YaHoo (Zhang et al., 2015). We remove Sports, Business & Finance,Science & Mathematics from YaHoo since they overlap with the classes in AGNews. We subsample 3000 training samples and 2000 test samples for each class (only 1900 test samples for the four classes in AGNews). The class order is obtained by sorting class names alphabetically and shuffling them using the random seed 1.
We use the default class order for Clinc150, Banking77, FewRel, and TACRED. We follow (Shao et al., 2023) to convert the class name to semantic labels for generative backbones. For example, in TARCRED, org:founded_by is converted to organization related: founded by.
For Few-NERD, Ontontes5, and I2B2, the class
order is obtained by sorting class names alphabetically. We use the BIO schema for tagging. The # class represents the number of entities in Table 5.
For sentence-level classification tasks, we report the accuracy. For word-level classification tasks, we report the macro-f1 due to the class imbalance.

heading: C Backbones
text: The statistics of the 9 backbones are summarised in Table 6. We download the pre-trained weights from Huggingface (Wolf et al., 2019).

heading: C.1 Discriminant Backbones
text: For discriminant backbones (i.e., encoder-only backbones), we use the [CLS] feature for sentencelevel classification tasks and the feature of last hidden states for word-level classification tasks. We do not use prompts for discriminant backbones. When learning each new task, we add a linear layer on top of the backbone for classification. For example, in the bert-base case, we add a linear layer whose input dimension and output dimension are 768 and 10 for learning the first task in CLINC150. Then, we add another linear layer with the same architec-
ture as the previous one when learning the second task. In the CIL scenario, the output logits over all learned categories are obtained by concatenating the logits from all classifiers. In the TIL scenario, the output logits are the logits from the classifier with the same task ID as the input sample.

heading: C.2 Generative Backbones
text: For generative backbones (i.e., decoder-only backbones), we train the model to output the class name with causal language modelling loss. We do not use generative backbones for word-level classification tasks because it requires special design on input and output format (Zhao et al., 2022) and the evaluation is different from that of sequential labelling models (Monaikul et al., 2021). For text and intent classification, we use the following prompt: “Input sentence: {text}\n The label: {label}{eos token}”. For relation extraction, we use the following prompt: “Input sentence: {text}\n The relationship between {head entity} and {tail entity} is {label}{eos token}”. We note that we use the same prompt for all baseline models unless they have special designs on prompts. Following Sun et al. (2019), only the causal language modelling loss of “{label}{eos token}” is optimized. For the probing study, we use the last hidden states of the last word as the feature.
The max sequence length is 256 in Topic3Datasets, 50 in CLINC150, 64 in Banking77, 100 in FewRel, and 128 in other datasets. We use exactly the same backbone for all baselines unless they have special designs on backbones.

heading: D Revisiting IL with Probing Study

heading: D.1 Four Probing Metrics
text: Linear Probing trains a new linear layer on top of the backbone model. We do not use bias for linear probing classifiers because there is no significant difference between the probing performance. We train the linear probing classifier for 20 epochs with an initial learning rate of 0.001. The linear probing classifier is trained on the training data from all T tasks jointly using cross-entropy loss. The Adam (Kingma and Ba, 2014) optimizer is used, and the batch size is set as 128. We note that the training data from all tasks is mixed for optimization. Otherwise, the probing performance is degraded significantly.
Cosine Linear Probing is the same as linear
probing except that the cosine similarity is adopted for calculating logits. We use the same training process as the linear probing classifier to train the cosine linear probing classifier. Hou et al. (2019) propose to use cosine linear layers for IL to avoid prediction bias towards new classes. However, we find that using cosine linear layers does not improve probing performance.
Prototype Probing requires no training of probing classifiers. It calculates the class feature centre for each class using all training data. It makes predictions of a test sample according to its Euclidean distances to all class centres. The prototype probing classifier can be regarded as a linear classifier with a weight matrix specified as all class centres.
Cosine Prototype Probing is the same as prototype probing except that the cosine similarity is adopted for calculating logits. The idea of using prototypes is widely adopted in IL (Ma et al., 2023; Cui et al., 2021; Han et al., 2020). However, we reveal that using prototypes for classification may not be the best option.

heading: D.2 Probing Performance with Different Backbones
text: We provide the probing performance of SEQ on class-incremental intent classification in Figure 3 and 8, relation extraction in Figure 9 and 10, text classification in Figure 13 and 14, and named entity recognition in Figure 12 and 11. We summarize the findings as follows:
• The linear probing performance is significantly higher than the other three metrics across backbones, tasks, and datasets.
• For all probing metrics, the probing performance always increases when learning the first task.
• For generative backbones, the probing performance has not decreased or even increased since the second task. It indicates that the smaller backbone can adapt to downstream tasks by SEQ. The larger backbone can adapt to downstream tasks without training and maintain the knowledge during SEQ.
• For discriminant backbones, using the linear classifier maintains the probing performance, while the cosine linear classifier degrades the probing performance significantly. There is
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(a) Observed Performance
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(b) Lin. Prob
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(c) Cos.Lin. Prob
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(d) Proto. Prob
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(e) Cos.Proto. Prob
(a) Observed Performance
(b) Lin. Prob
(c) Cos.Lin. Prob
(d) Proto. Prob
(e) Cos.Proto. Prob
(a) Observed Performance
(b) Lin. Prob
(c) Cos.Lin. Prob
(d) Proto. Prob
(e) Cos.Proto. Prob

heading: D.3 A Closer Look at Features, Word Embeddings, and Class Embeddings
text: To investigate the significant difference between the four probing metrics, we plot the histogram of
0 1 2 3 4 5 6 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(a) Observed Performance
0 1 2 3 4 5 6 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(b) Lin. Prob
0 1 2 3 4 5 6 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(c) Cos.Lin. Prob
0 1 2 3 4 5 6 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(d) Proto. Prob
0 1 2 3 4 5 6 # Task Has Learned
0
20
40
60
80
100
Ac cu
ra cy
( %
)
bert-large-cased+Linear bert-large-cased+CosineLinear bert-base-cased+Linear bert-base-cased+CosineLinear
(e) Cos.Proto. Prob
(a) Observed Performance
(b) Lin. Prob
(c) Cos.Lin. Prob
(d) Proto. Prob
(e) Cos.Proto. Prob
Figure 12: The observed and probing performance on class-incremental named entity recognition. The dataset is I2B2. The backbones are discriminant models. Other settings are the same as Figure 8.
the features, the (output) word embeddings, and the class embeddings in (cosine) linear classifiers. In this research, the features of PLMs refer to the last hidden states of the last word in generative
backbones and the [CLS] token in discriminant backbones. The class prototype refers to the class feature centres. The class embeddings refer to the row vectors of the weight matrix in linear layers. For example, for a linear layer whose input and output dimensions are 768 and 20, respectively, its weight matrix has the shape 20× 768. Then, each row vector corresponds to a certain category, and its shape is 1× 768. When using linear classifiers for prediction, the logits of a certain category are computed as the dot product between the feature and the class embeddings of that category.
During pre-training, the logits over vocabulary are computed as the dot product between features and word embeddings. In the probing study, the logits over categories are computed as the dot product between features and class embeddings. Therefore, we need to figure out the relationship between features, word embeddings, and class embeddings. We note that there is a dense layer (i.e., linear layer) between backbones and linear classifiers in BERT, and we ignore it for simplicity.
The histogram of the cosine similarity and L2 norm is provided in Figure 15 (Pythia-410m), 16 (Pythia-160m), 17 (bert-large-cased), 18 (bert-basecased). The dataset is CLINC150. We note that the PLMs are loaded directly without fine-tuning. The features are computed on the whole training set of CLINC150. The word embeddings are loaded directly from PLMs. The class embeddings of the linear probing classifiers are obtained by training
probing classifiers. From the result, we have the following findings:
• The features of PLMs have high cosine similarity, indicating that they fall in a cone space.
• The features are almost orthogonal to the word embeddings in all backbones except for Pythia-160m.
• The features are almost orthogonal to the class
embeddings in all backbones.
• The features have high cosine similarity with the prototypes.
• The L2 norm of word embeddings, class embeddings have large discrepancy for all backbones except for Pythia-160m.
These findings explain why the linear classifier is the best option to utilize the backbone’s knowledge.
Specifically, the word embedding layer is also a linear classifier for pre-training, and thus, the features are most discriminative when multiplied with the class embeddings of linear classifiers. Furthermore, the discrepancy of the L2 norm of word and class embeddings suggests that the norm contains the prior knowledge obtained from pre-training. We illustrate the four probing metrics in Figure 7.
The word embeddings in Pythia-160m have high cosine similarity with features. We speculate the reason is that the parameters are not enough for generalization with causal language modelling loss.

heading: E The Role of Pre-training
text: The result of text classification is shown in Figure 19. It shows a similar trend as intent classification and relation extraction in Figure 5.
We further analyze why even a randomlyinitialized model (step 0) achieves high probing performance. We use t-SNE (Van der Maaten and Hinton, 2008) to visualize the features of randomly initialized models and PLMs in Figure 20 and 21.
Figure 20a shows that randomly initialized models extract discriminative features with the Trans-

heading: 100 101 102 103 104 105 106
text: 60 40 20 0 20 40 60
40
20
0
20
40
Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9 Class 10 Class 11 Class 12 Class 13 Class 14 Class 15 Class 16 Class 17 Class 18 Class 19 Class 20
(a) Before SEQ w/o pre-training 40 20 0 20 40 60
40
20
0
20
40
Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9 Class 10 Class 11 Class 12 Class 13 Class 14 Class 15 Class 16 Class 17 Class 18 Class 19 Class 20
(b) After SEQ w/o pre-training 60 40 20 0 20 40 60
60
40
20
0
20
40

heading: 60 Class 1
text: Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9 Class 10 Class 11 Class 12 Class 13 Class 14 Class 15 Class 16 Class 17 Class 18 Class 19 Class 20
(c) Before SEQ w/ pre-training 60 40 20 0 20 40 60
40
20
0
20
40
60 Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 Class 8 Class 9 Class 10 Class 11 Class 12 Class 13 Class 14 Class 15 Class 16 Class 17 Class 18 Class 19 Class 20
(d) After SEQ w/ pre-training
40 20 0 20 40
(a) Before SEQ w/o pre-training 60 40 20 0 20 40 60
(b) After SEQ w/o pre-training 60 40 20 0 20 40 60
(c) Before SEQ w/ pre-training 60 40 20 0 20 40 60
(d) After SEQ w/ pre-training
Figure 21: The t-SNE visualization of features on FewRel. The backbone model is Pythia-410m. Only the first 10 classes are visualized for clarity. (a)(b): the backbone model is randomly initialized without pre-training; (c)(d) the backbone model is pre-trained.
former architecture, which is overlooked in previous IL studies. With SEQ or pre-training, the representation ability is enhanced, and the category boundaries become clearer. For the harder dataset, FewRel, the randomly-initialized model also learns the downstream knowledge through SEQ (Figure
21a vs 21b). It explains why the performance is improved by SEQ without pre-training.
In summary, we highlight that both pre-training and the architecture of Transformers are the key factors of the anti-forgetting ability of PLMs. Most existing studies (Scialom et al., 2022; Peng et al.,
2023) only attribute it to the pre-training stage. Our findings are consistent with the recent advance in the incremental learning dynamics of Transformers (Tarzanagh et al., 2023). Tarzanagh et al. (2023) discover that the rank of attention head weights gradually increases during training. We leave a deeper analysis of the incremental learning dynamics of the Transformers architecture to the future.

heading: F The Forgetting in Classifiers

heading: F.1 Overview of the Forgetting in Classifiers
text: Recall that we train probing classifiers on top of PLMs and achieve superior performance during SEQ. In contrast, the observed performance has
dropped consistently since the second task. In previous studies (Wu et al., 2019; Hou et al., 2019), they attributed the reason for catastrophic forgetting to the bias prediction between new and old categories. Indeed, a model trained with SEQ consistently achieves high performance on the new task while the accuracy on old tasks becomes nearly zero. In other words, the model predicts larger logits on new classes and smaller logits on old classes. Then, we investigate the dynamics of L2 norm and cosine similarity between features and class embeddings during SEQ.

heading: F.2 The Bias in the L2 Norm of Class Embeddings
text: We summarize the results of SEQ when the backbone is Pythia-410m in Figure 6, frozen Pythia410m in Figure 22, bert-large-cased in Figure 23, and frozen bert-large-cased in Figure 24. The class embeddings in each task are sorted according to the norm for clarity. The results show no apparent tendency in the norm of the class embeddings in both observed classifiers and probing classifiers. The norm of the class embeddings of newer classes even decreases when the backbone is bert-large-
cased. Therefore, the norm is not the critical factor in the bias prediction towards new classes in SEQ.

heading: F.3 The Bias in the Cosine Similarity between Features and Class Embeddings
text: First, we define the moving distance to measure how the cosine similarity between features and class embeddings changes during SEQ. After a new task is learned, the relative position between features and class embeddings is optimal for this task. Intuitively, the forgetting happens only when the relative position changes. Ideally, if all pairs of
features and class embeddings remain at the same angle, forgetting will not occur. In practice, we measure the cosine similarity between class embeddings and class feature centres instead of all features. An illustration is provided in Figure 25.
Formally, we define the moving distance of class embeddings of the i-th task at the i+ k-th task as follows:
MDii+k = 1
mn
m=|Yall|∑
m=1
n=|Yt|∑
n=1
|Cii+k[m,n]−Cii[m,n]|
(4) The Yall and Yt represent the label set of all T tasks and the t-th task respectively. C represent the cosine similarity matrix between all pairs of class embeddings and class feature centres. Cii+k represents the cosine similarity matrix between all pairs of class embeddings from task i and all class feature centres, and it is measured at task i + k. Cii+k[m,n] is the entry of the cosine similarity matrix Cii+k at position [m,n]. This entry represents the cosine similarity between the n-th class embeddings of task t and the m-th class embeddings of all T tasks.
We summarize the moving distance of SEQ when the backbone is Pythia-410m in Figure 6, frozen Pythia-410m in Figure 22, bert-large-cased in Figure 23, and frozen bert-large-cased in Figure 24. The results clearly show that the old class em-
beddings in observed classifiers have drifted out of position since they were learned. Besides, the moving distance becomes larger when the backbone is frozen. In contrast, the old class embeddings in the probing classifier do not deviate much from the initial position.
We provide instances of the cosine similarity matrix in Figure 26. The cosine similarity matrix of observed classifiers changes significantly when new tasks are learned. The forgetting is more apparent when we focus on the diagonal.
In summary, we reveal that the change of relative position between class embeddings and features leads to the forgetting of classifiers. Additionally, the old knowledge is preserved if the relative position is maintained, as in probing classifiers.

heading: G SEQ*: Boosting Performance of SEQ
text: We illustrate the proposed four strategies in Figure 27. (S1) Freeze the PLMs after warm-up. For generative backbones, we train models with causal language modelling loss to generate the groundtruth class name in the warm-up stage. For discriminant backbones, we train models with classifiers to predict the class ID with cross-entropy loss. We warm up 1 epoch for discriminant backbones and 3 epochs for generative backbones. After the warm-up stage at the first task, all parameters of backbones are frozen and will not be updated in
the subsequent incremental tasks. (S2) Freeze the old classifiers. When the model finishes training on the current task, we freeze the classifier of the current task. When learning more tasks, we only update the new classifier while the old classifiers are frozen.
(S3) Use cosine linear classifiers only when no old data is available in the CIL scenario. Otherwise, use linear classifiers. We use linear classifiers even without old samples for named entity classification because the non-entity tokens (i.e., “Other” tokens) can be regarded as old samples from previous tasks (Zheng et al., 2022).
(S4, optional) Pre-allocate future classifiers. For example, when learning the 10 classes in the first task of CLINC150, the future classifiers from task 2 to task 15 are pre-allocated. The future classifiers are trained in advance in the first task through the softmax layer even when there are no instances from task 2 to task 15. This strategy is especially effective for generative backbones. The reason may be that it enhances the forward compatibility of classifiers, and thus, new classifiers are easier to adapt to new classes when old classifiers are frozen.
H Introduction of Baselines
Except for Topic3Datasets, we train all baselines for 5 epochs for each incremental task. On Topic3Datasets, we train all baselines for 3 epochs for each incremental task. The learning rate of backbones and classifiers are 1×10−5 and 1×10−3 respectively. We use AdamW (Loshchilov and Hutter, 2018) optimizer. We use RTX 3090 GPUs for our experiments. Each experiment is repeated three times, and the average and standard deviations are reported. We train SEQ* and all baselines with exactly the same training settings for fair comparison. We search the best hyper-parameters for each compared method and use the same hyper-parameters across backbones and datasets.
The introduction of SOTA methods is as follows:
• SEQ: SEQ refers to sequential fine-tuning. For generative backbones, SEQ trains models to output the class name with causal language modelling loss. For discriminant backbones, SEQ trains models with classifiers to predict the correct class ID.
• LAMOL (Sun et al., 2019): LAMOL trains a generative model with question-answering
and generation targets and generates pseudo samples before learning each new task. The weight of the generation target λ = 0.25. The ratio of pseudo samples with respect to the training data of new task γ = 0.20. The top-k sampling for generating pseudo samples K = 20. There are two variations of LAMOL, including LAMOL_t and LAMOL_g. The difference is whether or not to use task-specific tokens for generation.
• L2KD (Chuang et al., 2020): L2KD proposes to add knowledge distillation targets based on LAMOL. We implement the word-level (Word-KD) variation since it performs best on text classification tasks.
• LAMOL_KD: LAMOL_KD further utilizes knowledge distillation based on LAMOL_t. Unlike L2KD, the teacher model in LAMOL_KD is trained on all previous tasks. The new data is for learning both LOAMOL targets, and the pseudo data is for word-level knowledge distillation as regularization terms. LAMOL_KD is an extension to LAMOL.
• PCLL (Zhao et al., 2022): PCLL is build upon LAMOL. Furthermore, PCLL utilizes the targets of varational autoencoders and wordlevel knowledge distillation to train generative models.
• LFPT5 (Qin and Joty, 2021): LFPT5 further utilizes knowledge distillation based on LAMOL. Besides, LFPT5 utilizes prompt tuning instead of fine-tuning the whole model. The number of tokens for prompt tuning is 10.
• AdapterCL (Madotto et al., 2021): AdapterCL learns one adapter (Houlsby et al., 2019) for each task. During inference in CIL, the model predicts the task ID according to causal language modelling loss, which requires T forward passes for each instance.
• LoRA (Hu et al., 2021): LoRA learns one LoRA adapter for each task. We only compare LoRA in the TIL setting. We set the rank r = 4 and the scaling parameter α = 8. We use the implementation of LoRA in the PEFT library (Mangrulkar et al., 2022).
• ProgPrompt (Razdaibiedina et al., 2023): Progressive Prompt learns soft prompt for each
task progressively. Following (Razdaibiedina et al., 2023), we Use a residual two-layer MLP to encode soft prompt, and the number of soft prompt tokens for each task is 5. ProgPrompt can only be used for the TIL setting because the task ID is required during inference.
• ER: Experience replay stores representative old samples and jointly optimizes both new and old samples when learning new tasks.
• DER++ (Buzzega et al., 2020): DER++ is based on data replay. Besides, DER++ adds an MSE loss to regularize the logits of old samples between teacher and student.
• CLSER (Arani et al., 2022): CLSER is based on DER++. Besides, CLSER additionally stores two models (i.e., fast model and slow model) for selecting teacher logits when computing the MSE loss.
• SpanKL (Zhang and Chen, 2023): SpanKL converts the named entity recognition task from entity-level annotation to Span-level annotation, using a binary cross-entropy loss function and a distillation loss, with the distillation loss coefficient λ is 1.
• OCILNER (Ma et al., 2023): OCILNER utilizes contrastive learning to adaptively detect entity clusters and utilizes two distancebased methods to label non-entity tokens. The threshold for labelling Non-entity is the median class similarity of all classes in the memory.
• ExtendNER (Monaikul et al., 2021): ExtendNER utilizes knowledge distillation to review old entity types, and the distillation loss coefficient λ is 2.
• DLD (Zhang et al., 2023c): DLD improves the knowledge distillation method in ExtendNER, dividing it into negative terms and positive terms for knowledge distillation, and the distillation loss coefficient λ is 2.
• SelfTrain (Rosenberg et al., 2005; De Lange et al., 2019): SelfTrain utilizes the teacher model to directly label current data and train it together with samples of new entity types.
• RDP (Zhang et al., 2023b): RDP utilizes the previous model to pseudo-label new data and
increases self-entropy loss to improve the confidence of model prediction. The self-entropy loss coefficient λ is 0.1.
• CPFD (Zhang et al., 2023a): CPFD designed a pooled feature distillation loss and proposed a confidence-based pseudo-label annotation method. The feature distillation loss coefficient λ is 2.
• ICE (Liu and Huang, 2023): ICE includes two methods, ICE_O and ICE_PLO. These methods freeze the backbone model and the old classifiers. ICE_O combine the non-entity logits with the new task logits to obtain the output probability during training. ICE_PLO uses all previous logits and new logits during training. Unlike ICE, SEQ* additionally warm-up the PLMs and pre-allocating future classifiers. Furthermore, ICE is limited to class-incremental information extraction, while SEQ* can be applied to both sentence and word-level classification tasks.
• CFNER (Zheng et al., 2022): CFNER proposes a causal effect framework to alleviate the forgetting of old entity types and uses curriculum learning methods to reduce the impact of noisy labels. The number of matched tokens K is 3.

heading: I Revisiting SOTA Methods

heading: I.1 CIL with Generative Backbones for Sentence-Level Tasks
text: The results of SOTA methods and SEQ* with Pythia-410m, Pythia-160, gpt2-base, gpt2-large are provided in Table 7, 8, 9. We compare AdapterCL only on GPT2 because Adapter is not implemented for Pythia. From the results, we have the following findings:
SEQ* shows competitive or even superior performance on all datasets. The warm-up strategy (S1) is effective when the gap between pre-training and downstream data is large (CLINC150, Banking77, FewRel, TACRED). In contrast, it is not so effective when the gap is small (Topic3Datasets). Freezing old classifiers (S2) is more important than freezing backbones. The result also validates that the forgetting results from classifiers instead of backbone PLMs. Using cosine linear classifiers (S3) is crucial when the IL scenario is CIL, and no old samples are stored. Otherwise, models will be
strongly biased towards new classes. Pre-allocating future classifiers (S4) are very effective for generative backbones and brings considerable improvements. The reason may be that training future classifiers in advance reduces the overlapping between class embeddings.
The LAMOL-based methods achieve superior performance on the topic classification datasets. It indicates that generating pseudo samples is effective when the downstream data have close data distribution with the pre-training data. However, their performance is much worse than SEQ* when more tasks are learned (Clinc150), or the gap between pre-training and downstream data is larger (FewRel, TACRED). LFPT5 has the worst performance in our settings. LFPT5 was originally designed for T5 (Raffel et al., 2020), and we find that the decoder-only models do not learn to generate pseudo samples by just learning soft prompts.
SEQ* fails to achieve superior performance on gpt2-base. We find the training loss hard to decrease when adding cosine classifiers on top of the gpt2-base. On gpt2-large, SEQ* outperforms AdapterCL significantly while updating much fewer parameters.

heading: I.2 CIL with Discriminant Backbones for Sentence-Level Tasks
text: We compare SEQ* with two strong rehearsal-based methods, DER++ and CLSER. Both DER++ and CLSER store teacher models for knowledge distillation and require updating all parameters in PLMs.
From the results in Table 10 and 11, we find that both DER++ and CLSER bring considerable improvement upon ER. However, SEQ* again shows competitive or superior performance on all datasets. It indicates that knowledge distillation is effective for preserving knowledge. However, fully finetuning PLMs causes more forgetting. Therefore, stability is more important than plasticity when designing IL algorithms for PLMs.
Besides, we have the following minor findings: (1) Pre-allocating future classifiers with discriminant backbones is less effective than generative backbones. (2) Freezing only old classifiers achieves the best performance on CLINC150 and TACRED. It shows that freezing backbone PLMs may not be necessary for some datasets. The relative position between old class embeddings and features may still be preserved when SEQ with frozen old class embeddings. (3) Using bert-large-cased may not lead to better results than bert-base-cased.
We empirically find that training with bert-largecased is more unstable than bert-base-cased.

heading: I.3 CIL with Discriminant Backbones for Word-Level Tasks
text: We evaluate SEQ* on class-incremental named entity recognition since its popularity. In classincremental named entity recognition, the nonentity tokens (“Other” tokens) can be regarded as old samples (Zheng et al., 2022). Existing methods such as (Monaikul et al., 2021; Zheng et al., 2022) utilize the Other tokens for knowledge distillation. And Zhang et al. (2023b), Zhang et al. (2023c), Zhang et al. (2023a) also verify the effectiveness of applying knowledge distillation. However, the results in Table 12, 13 show that SEQ* outperforms them by a large margin on OntoNotes5 and I2B2. On the challenging dataset Few-NERD, SEQ* beat 7 SOTA methods for class-incremental named entity recognition.
When using bert-large-cased, the performance of CFNER, DLD, and SpanKL becomes more unstable. In contrast, SEQ* are more robust to the choice of PLMs and show consistent superior performance.
Finally, we also find that the choice of classifier is crucial for class-incremental named entity recognition. When using cosine linear classifiers, the performance is degraded significantly. Therefore, we use linear classifiers for all baselines and SEQ* for a fair comparison, although CFNER, DLD, RDP, and CPFD adopt cosine linear classifiers in their original implementation. We speculate the reason is that using cosine linear classifiers prevents models from learning the prior distribution between classes, which is crucial for class-imbalanced tasks.
The result in Table 14 shows that the linear probing performance of both SEQ* and SOTA methods increases. It indicates that all models enable PLMs to adapt to downstream tasks. Furthermore, SEQ*, SelfTrain and ExtendNER have higher linear probing performance than SEQ (Lin). It indicates that forgetting causes the degradation of linear probing performance, and it can be alleviated by freezing PLMs or knowledge distillation.

heading: I.4 TIL for Sentence-Level Tasks
text: We consider LoRA and ProgPrompt for the baselines in TIL. LoRA has been widely adopted for adapting PLMs to downstream tasks. The result in Table 15 shows that SEQ* achieves competitive performance but requires much fewer new parameters
to learn compared with LoRA. ProgPrompt relies heavily on the choice of PLMs and shows poor performance on Pythia-410m and 160m. Since the scenario of TIL is much simpler than that of CIL,
and all methods achieve high accuracy, we did not conduct as many experiments in TIL as in CIL.

heading: Future Work
text: Besides, we have the following minor findings: (1) Pre-allocating future classifiers with discriminant backbones is less effective than generative backbones. (2) Freezing only old classifiers achieves the best performance on CLINC150 and TACRED. It shows that freezing backbone PLMs may not be necessary for some datasets. The relative position between old class embeddings and features may still be preserved when SEQ with frozen old class embeddings. (3) Using bert-large-cased may not lead to better results than bert-base-cased. We empirically find that training with bert-largecased is more unstable than bert-base-cased