Incremental Sequence Labeling: A Tale of Two Shifts (2024)

Abstract: The incremental sequence labeling task involves continuously learning new classes over time while retaining knowledge of the previous ones. Our investigation identifies two significant semantic shifts: E2O (where the model mislabels an old entity as a non-entity) and O2E (where the model labels a non-entity or old entity as a new entity). Previous research has predominantly focused on addressing the E2O problem, neglecting the O2E issue. This negligence results in a model bias towards classifying new data samples as belonging to the new class during the learning process. To address these challenges, we propose a novel framework, Incremental Sequential Labeling without Semantic Shifts (IS3). Motivated by the identified semantic shifts (E2O and O2E), IS3 aims to mitigate catastrophic forgetting in models. As for the E2O problem, we use knowledge distillation to maintain the model’s discriminative ability for old entities. Simultaneously, to tackle the O2E problem, we alleviate the model’s bias toward new entities through debiased loss and optimization levels. Our experimental evaluation, conducted on three datasets with various incremental settings, demonstrates the superior performance of IS3 compared to the previous state-of-the-art method by a significant margin. The data, code, and scripts are publicly available 1.

heading: 1 Introduction
text: The conventional sequence labeling task typically involves categorizing data into a predetermined set of fixed categories (Lample et al., 2016). However, this approach may need to be revised in natural language processing scenarios, such as the named entity recognition task, where new types of entities continuously emerge. Adapting a fixed set of
∗*Corresponding author 1https://github.com/zzz47zzz/codebase-for-incremental-
learning-with-llm and https://github.com/qianlimalab/codebase-for-incremental-learning-with-llm
categories becomes challenging when faced with the dynamic nature of new entity classification requirements. Consequently, continuous model updates are essential to accommodating evolving entity types. Previous studies have advocated for adopting continual learning (Parisi et al., 2019; Monaikul et al., 2021), also known as lifelong learning or incremental learning. Continual learning is a paradigm designed to train models capable of adapting to the continual addition of new categories in real-world scenarios while ensuring that knowledge of old categories is retained. For instance, voice assistants like Siri frequently encounter new event types, such as pandemics, need to understand and provide health-protecting information based on the user’s latest intent. (Monaikul et al., 2021).
Due to constraints imposed by storage limitations and privacy concerns, there exists a shortage of training data about the old categories (He and Zhu, 2022). Additionally, the manual relabeling of all categories within the new training dataset would incur substantial costs and time investment (De Lange et al., 2021; Bang et al., 2021). Conse-
777
quently, the model undergoes continuous updates using a freshly acquired dataset comprising the new categories. As depicted in Figure 1, the model undergoes training based on the current ground-truth label and undergoes testing using the full groundtruth label.
The incremental sequence labeling task faces a significant challenge known as the catastrophic forgetting problem, as extensively discussed in previous studies (McCloskey and Cohen, 1989; Robins, 1995; Goodfellow et al., 2013; Kirkpatrick et al., 2017; Zheng et al., 2024). This issue manifests as semantic shifts, leading to a decrease in the discriminative power of entity classes (Zhang et al., 2023b; Ma et al., 2023). In this paper, we decompose the problem into two primary semantic shifts in the incremental sequence task: E2O and O2E. The first semantic shift, E2O, arises from the presence of non-entities, potential old entities (mislabelled as non-entities), and new entities in the new dataset. Progress has been made in addressing E2O through methods falling into three categories: (1) Methods based on knowledge distillation: For instance, RDP proposes a knowledge distillation loss incorporating inter-task relations (Zhang et al., 2023b). At the same time, CFPD introduces a pooled feature distillation loss to alleviate catastrophic forgetting (Zhang et al., 2023a). (2) Methods based on pseudo-labels: OCILNER utilizes class prototypes to label new data (Ma et al., 2023), and CPFD employs old models to label predictions of new data. (3) Methods based on freezing models: Examples include ICE (Liu and Huang, 2023), which freezes the backbone model and old classifiers to maintain the stability of the old classes at the expense of learning new classes.
Existing methods primarily focus on addressing the E2O shift, neglecting the bias towards the emergence of new classes and the consequential second semantic shift, O2E. To address both semantic shifts, we propose a novel framework called Incremental Sequential Labeling without Semantic Shifts (IS3). IS3 consists of two key components: First, we apply the knowledge distillation method to tackle the E2O shift. Second, we address the O2E shift on two fronts. At the loss function level, we introduce a debiased cross-entropy loss function to mitigate the model’s impact on old class distributions, reducing its inclination towards new entities. At the optimization level, we introduce a prototype-based approach to balance the imbalanced contributions of old and new entities during
batch updates, which aims to increase the involvement of old entities in the optimization process. Importantly, IS3 adopts a storage-efficient approach, maintaining only one prototype per class with minimal storage costs. Class feature centers serve as prototypes, ensuring no direct correspondence to actual sample information and mitigating privacy leakage concerns.
The contribution of our work can be summarized as follows:
• We propose a novel perspective on the semantic shift problem in incremental sequence labeling task by categorizing the catastrophic forgetting problem into E2O and O2E.
• We propose a novel framework, Incremental Sequential Labeling without Semantic Shifts (IS3), to solve the two semantic shifts simultaneously.
• We conduct experiments under nine CIL settings on three datasets, and our method outperforms the previous state-of-the-art methods.

heading: 2 Related Work
text: Incremental Learning The model continually acquires new tasks intending to achieve optimal performance on tasks previously learned (Gepperth and Hammer, 2016; Wu et al., 2019; van de Ven et al., 2022; Zheng et al., 2023b). There are three main categories of current incremental learning methods: regularization-based, rehearsal-based, and architecture-based. Regularization-based methods place constraints on model weights (Kirkpatrick et al., 2017; Zenke et al., 2017), representations of intermediate layer features (Hou et al., 2019; Douillard et al., 2020), and output probabilities (Li and Hoiem, 2017; Zheng et al., 2023a). Rehearsal-based methods overcome forgetting by saving some of the data containing the old classes for learning with the new classes (Lopez-Paz and Ranzato, 2017; Shin et al., 2017). Alternatively, architecture-based approaches involve dynamically expanding the network structure to allow for more data as new classes are added (Hou et al., 2018; Yan et al., 2021). Incremental Sequence Labeling The traditional sequence labeling task is the task of labeling each token of a one-dimensional linear input sequence, which requires each token to be categorized according to its contextual content(Rei et al., 2016; Akbik et al., 2018). However, previous methods
can only recognize classes in a fixed set. Therefore, continuous learning paradigms are introduced in sequence labeling tasks, including incremental named entities (Monaikul et al., 2021; Zheng et al., 2022; Zhang et al., 2023a), incremental event detection (Cao et al., 2020; Yu et al., 2021), and so on.
Methods for incremental sequence labeling tasks can be categorized into distillation-based, rehearsalbased, and other approaches. Distillation-based methods encompass ExtendNER (Monaikul et al., 2021), which is the pioneer in applying knowledge distillation to incremental sequence labeling task, RDP (Zhang et al., 2023b) with a relational distillation approach, and CPFD (Zhang et al., 2023a) utilizing pooled features distillation loss. CFNER (Zheng et al., 2022) introduces a causal framework for extracting new causal effects in entities and nonentities. Rehearsal-based approaches include KCN (Cao et al., 2020) and KD+R+K (Yu et al., 2021), both employing rehearsal samples to address the class imbalance and catastrophic forgetting in incremental event detection. L&R (Xia et al., 2022) proposes a learn-and-review framework by training a new backbone model and a generative model simultaneously, generating synthetic samples of the old class to be trained with new samples. OCILNER (Ma et al., 2023) uses rehearsal samples to compute class feature centers as class prototypes, generates an entity-oriented feature space through comparative learning, and annotates new data with pseudo-labels using class prototypes. Other methods encompass span-based and freezing modelbased approaches, among others.
The mentioned methodologies primarily focus on preserving the existing knowledge of the model and do not explicitly consider the implications of transitioning from non-entity to entity semantics. In contrast, our proposed method, IS3, provides a fresh perspective on model forgetting by addressing the model’s inclination towards new classes during task adaptation. IS3 not only addresses issues related to model mislabeling, indirectly mitigating the problem of semantic migration from entity to non-entity, but also handles the challenge of semantic migration from non-entity to entity. By recognizing and addressing the model’s bias towards new classes during adaptation, our approach offers a comprehensive solution to the dynamic challenges associated with transitioning between different semantic categories.

heading: 3 Problem Formulation
text: Formally, the objective of incremental sequence labeling is to acquire knowledge through a series of tasks T = {T1, T2, . . . , TN}. Each task contains its dataset Dt = {(xi, yi)|yi ∈ Yt} where (xi, yi) is a pair formed by the input token sentence and the label corresponding to each token in the sentence and Yt stands for the current label set. Notably, yi only labels the token corresponding to the current task t, and the other tokens are labeled as O class (potential old entities Y1:t, and unseen entities Yt+1:N ). At task t (t > 1), the new model Mt learns only from the new dataset and is expected to perform well on the learned classes ⋃t i=1 Yi.

heading: 4 Method
text: In this section, we systematically address the catastrophic forgetting problem by decomposing it into two distinct semantic shift challenges (Section 4.1). Subsequently, we present a comprehensive framework designed to address these semantic shifts individually, focusing on E2O in Section 4.2 and O2E in Section 4.3. The overarching goal is to effectively mitigate the catastrophic forgetting problem, as illustrated in Figure 4.

heading: 4.1 Two semantic shift problems
text: In the incremental sequence labeling task, semantic shift can be decomposed into entity to non-entity semantic shift and non-entity to entity semantic shift, which are abbreviated as E2O and O2E.
E2O refers to the model incorrectly categorizing entities as non-entities during the learning process.
This misclassification stems from the incremental sequence labeling task, where only new entities are labeled in the new dataset, potentially causing old entities to be erroneously labeled as non-entities. For instance, in Figure 2, the name "Amy" is mistakenly labeled as a non-entity. This misclassification induces a gradual shift in the semantics of old entities towards non-entities, leading to a blurred boundary between the two classes. Several previous approaches have addressed this bias issue. Methods like RDP focus on designing improved distillation techniques to maintain the stability of the model’s old entities. Similarly, OCILNER utilizes comparative learning to obtain a more discriminative feature space, clarifying the classification boundaries between entities and non-entities. These strategies aim to mitigate the impact of E2O, ensuring a more accurate preservation of entity semantics during incremental sequence labeling tasks.
O2E signifies the model incorrectly labeling nonentities or old entities as new entities during the learning process. As seen in Figure 2, our observations indicate that the model maintains good discrimination between old entities. However, Figure 3 shows a bias towards new entities in predictions during incremental learning. Our research identifies two key contributing factors to this bias.
The first factor is related to the classifier dimension’s predisposition. When learning new entities, the ordinary cross-entropy function induces the
model to fit and converge faster on the distribution of new entities by excessively penalizing the classifier dimension associated with old entities. This over-penalization of old entities results in a pronounced bias in significant classification scores towards the new classes.
The second factor involves a tendency at the feature optimization level. The current dataset mainly contains samples of new entities with minimal representation from other entities, including potential old and future new entities, to facilitate effective learning of new entities. As a result, in the same batch, the probability of old entities participating in model optimization is much lower than the probability of new entities’ participation. Consequently, there is a predisposition towards new categories at the feature optimization level. Addressing these aspects is crucial for mitigating the O2E semantic shift and achieving more balanced and accurate predictions during incremental sequence labeling tasks.
Notably, the E2O and O2E problems are interconnected. If the O2E problem occurs in the model during incremental sequence labeling, it can gradually blur the boundaries among entities. It can also indirectly contribute to the E2O problem, ultimately impacting the model’s discriminative ability. We will address these two semantic biases separately to mitigate catastrophic forgetting during incremental sequence labeling.

heading: 4.2 Solving E2O problem via knowledge distillation
text: When learning the current task t, the model Mt is trained on the training examples with the current entities, which often leads to catastrophic forgetting for the old entities. To alleviate the E2O problem, we use knowledge distillation (Hinton et al., 2015). This method preserves the prior knowledge by distilling the output probabilities from the old Mt−1 to the current model Mt. Therefore, the objective function for solving the E2O problem can be expressed as:
Lkd = 1
|Dt|
|Dt|∑
i=1
ŷt−1i log ŷ t i , (1)
where ŷt−1i and ŷ t i represent the output probabilities of the current model and the old model respectively. Through Eq.1, "Amy" in Figure 4 corrects the current model’s incorrect labeling via the output
probabilities provided by the old model, thus maintaining discriminative properties between old entities.

heading: 4.3 Solving O2E problem
text: In this section, we address the O2E problem at the debiased loss and feature optimization levels.

heading: 4.3.1 Debiasing in Ordinary Cross Entropy
text: The overall model parameters are defined as Θ = {θ, ω}. The model’s backbone fθ : X → Rd extracts feature embeddings of dimension d from the inputs. Following the backbone, a linear classifier produces logits Φ(·) = ωT · fθ(·) : X → R|Yt|, where ω represents the classifier weights for the corresponding dimensions. As the number of classes of recognizable entities increases as well, the dimension of the classifier increases. The model is trained by a cross-entropy loss function, which is defined as:
Lce = − 1
|Dt|
|Dt|∑
i=1
yi log ( eΦyi (xi)∑
y′∈Yt e Φy′ (xi)
)
= 1
|Dt|
|Dt|∑
i=1
log[1 + ∑
y′̸=yi eΦy′(xi)−Φyi (xi)],
(2) where yi denotes the label of the new entity for the current incremental step t and y′ denotes the labels
not equal to yi. Figure 3 shows that confusion matrix of previous method at incremental step 4. It clearly shows that most predictions are biased towards the recent entity (class 4). We find that such a bias can be found in the cross-entropy loss function. When learning new entities, the model’s gradient update for old entities is defined as:
∂Lce ∂ωy′ ∝ eΦy′(xi)(y′ ̸= yi), (3)
where the gradient update for old entities is proportional to the classification score for that entity. During the incremental sequence labeling process, this gradient update exhibits an overly penalizing effect on the old entity probability distributions. It shows up as an excessive reduction in the output probability score of the old entity. We provide a more detailed explanation and derivation in Appendix A.
We assume the old model has learned the optimal representation of old entities. Therefore, the new entities should have a smaller impact on the knowledge of old entities. Otherwise, because of the absence of rehearsal samples of the old entities, the model will face catastrophic forgetting of the old entities. In addition, the new entity was not in the predefined set, and a change from a non-entity to a new entity occurs during learning. Therefore,
it is reasonable to have a penalizing effect on nonentities, and the debiased cross-entropy loss function is defined as follows:
LDebiasce = 1
|Dt|
|Dt|∑
i=1
log[1+ ∑
y′̸=yi eδΦy′(xi)−Φyi (xi)],
(4) where δ is the correction factor for the gradient update of the old entity weights (excluding nonentity weights), δ ∈ [0, 1]. When δ → 0, the model will no longer penalize the learning of old entities. When δ → 1, Eq.4 degenerates to the traditional cross-entropy loss function.

heading: 4.3.2 Learning with Prototypes
text: In Section 4.1, we elucidate the reasons behind the emergence of O2E at the feature optimization level. In this section, we introduce the utilization of class centers of old entities as class prototypes during the learning process of new entities. Following each task training, we compute prototypes using feature representations from the training set and store them. These prototypes then participate in training the model classifier for the subsequent task alongside the feature representations of new entities.
The class prototypes of old entities serve two essential purposes: firstly, they participate in optimization alongside new entities in each batch, ensuring a balanced optimization process among entities. Secondly, these class prototypes act as anchors in the feature space, mitigating the issue of over-labeling new entities. As depicted in Figure 4, the introduction of old prototypes reduces the potential over-labeling of new entities, enhancing the precision of new entity learning.
To this end, we defined the loss function of prototypes as follows:
Lpro = − t−1∑
i=1
ỹi log
( eω
TPi ∑|Yt|
j=0 e ωTPj
) , (5)
where ỹi stands for the label of the old prototype and Pi, Pj stand for old prototypes, defined as follows:
Pt = 1
|Dt|
|Dt|∑
i=1
fθ(xi). (6)
Our approach differs from OCILNER’s approach, which uses prototypes in two ways:(1) OCILNER’s approach stores old samples for calculating prototypes. However, in this paper, we only use the training data in each incremental step for
calculating prototypes and do not introduce replay samples. (2) OCILNER uses prototypes to label new datasets and adopts a cosine similarity as the threshold for entity labeling. However, in this paper, we found that some of the real non-entities also have a high cosine similarity with entities, which can easily produce wrong labeling for real nonentities and exacerbate semantic migration from entities to non-entities. Therefore, to avoid incorrectly labeling non-entities, we use only class centers as prototypes and do not use prototypes for labeling existing entities.
In summary, the objective function of our method is defined as follows:
L = LDebiasce + αLpro︸ ︷︷ ︸ LO2E +βLkd︸ ︷︷ ︸ LE2O . (7)

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Datasets We conducted experiments on three widely used datasets: i2b2 (Murphy et al., 2010), OntoNotes5 (Hovy et al., 2006), and MAVEN (Wang et al., 2020). We divide the dataset into disjoint slices according to categories. In each slice, we keep only the category labels visible to the current task, and the rest of the labels are labeled as non-entities. Settings We sort the above slices according to initial letter and train them in a FG-a-PG-b manner. FG means that the pre-trained model is trained with a entity types as the initial model, and PG means that the initial model is trained with b entity types at each following incremental step. Baselines We consider the following state-ofthe-art methods for incremental sequence labeling: Self-Training (Rosenberg et al., 2005; De Lange et al., 2019), ExtendNER (Monaikul et al., 2021), CFNER (Zheng et al., 2022), DLD (Zhang et al., 2023c), RDP (Zhang et al., 2023b), OCILNER (Ma et al., 2023), ICE (Liu and Huang, 2023), CFPD (Zhang et al., 2023a). Detailed descriptions of the baselines and their experimental setup are provided in Appendix C. Implementation Details We use bert-base-cased model from HuggingFace (Wolf et al., 2019) as backbone, with a hidden dimension of d = 768. In addition to this, we have included supplemental experiments with the roberta-base model on the relevant datasets in Appendix E. We use the AdamW (Loshchilov and Hutter, 2018) optimizer,
with learning rate 1e−6 and 1e−3 for backbone and classifier. We report the mean and standard deviation results over five runs. Metrics Considering that each of the categories should have a comparable degree of contribution in the test, we use Macro F1 to evaluate the performance of the model. We use the last step Macro F1 result in AT , and the average Macro F1 result in Ā, on all incremental steps as evaluation metrics. AT and Ā are defined in Appendix D.

heading: 5.2 Results and Analysis
text: Comparisons with State-Of-The-Art To validate the effectiveness of our approach, we conducted exhaustive experiments on the i2b2, OntoNotes5, and MAVEN datasets. We used the Finetune Only (FT) approach as a lower bound for comparison. Table 1 and Table 8 display the results of the experiments conducted on i2b2. Due to space limitations, we provide the results on MAVEN in Table 6 and OntoNotes5 in Table 7 and Table 9. In detail, we show the experimental results under nine incremental learning settings through Figure 6. Our method consistently outperforms the previous state-of-the-art method in multiple settings, from FG-1-PG-1 to FG-8-PG-2. The poor performance of the previous method may be attributed to the ignorance of O2E. As shown in Figure 7, during the learning process, the previous method, ExtendNER, incorrectly confuses new entities with non-entities due to O2E errors, and old entities with non-entities due to E2D errors. These issues together lead to
poor prediction results for the model. We have effectively mitigated the above problems through our framework IS3, which strikes a good balance between maintaining old entities and learning new ones.
To further demonstrate the effectiveness of our method, we visualize the feature representation through T-SNE (Van der Maaten and Hinton, 2008). As shown in Figure 9, the ExtendNER method faces serious E2O and O2E problems, with new entities and non-entities overwriting old ones, leading to catastrophic forgetting. Our method successfully addresses the issue of semantic bias that arises when the model learns a new task. Ablation Study We explored the validity of the components of our approach through ablation experiments, and the results are shown in Table 2. We removed the debiased cross-entropy loss LDebiasce and prototype loss Lpro modules, respectively. These results demonstrate the essential roles
played by both LDebiasce and Lpro modules. The LDebiasce reduces the penalizing effect of the new entity on the old entity and enhances the discrimination between the old and new entities by improving the prediction confidence of the old entity. The Lpro corrects the bias of modeling new entities by shrinking the scope of over-labeling new entities through old prototypes. Hyper-Parameter Analysis Figure 10 shows the results of different hyper-parameter choices on OntoNotes5 with the setting FG-1-PG-1. We consider two hyper-parameters: the correction factor in the debiased cross-entropy loss δ and the weight of the prototype loss β. The results show that δ around 0.5 reaches the best result, indicating that a moderate penalty effect reduction favors model performance. As β keeps increasing, it makes the model overfit for the old prototype, leading to a decrease in model performance. Case Study We provide an example in Figure 8 to demonstrate that the previous method suffers from an O2E offset when learning a new entity ORG, overwriting the old entity EVENT as a new entity. Simultaneously, the model inherits past O2E issues (labeling [O] as [DATE]). Additionally, it suffers from E2O, which fails to recognize the old entity accurately. Our method effectively balances these two offset problems and is more conducive to model learning.

heading: 6 Conclusion
text: In this paper, we introduce a novel perspective on the catastrophic forgetting problem in incremental sequence annotation, identifying and addressing both E2O and O2E semantic shifts. Bridging gaps in previous research, we propose the IS3 framework to tackle both issues. Comprehensive experiments on three datasets demonstrate that our IS3 method significantly outperforms previous stateof-the-art approaches. This work provides a fresh
ro F
1
outlook on the incremental sequence labeling task and offers effective solutions to mitigate the catastrophic forgetting problem.
Limitations
While the proposed method effectively mitigates catastrophic forgetting to some extent, its reliance
on the predictions of old models for preserving existing knowledge can result in accumulated prediction errors, which may lead to poor model performance in more incremental steps. Moreover, the current method does not thoroughly explore the relationship between the penalty effect and the dataset, leaving potential avenues for future research.

heading: Acknowledgements
text: The work described in this paper was partially funded by the National Natural Science Foundation of China (Grant No. 62272173), the Natural Science Foundation of Guangdong Province (Grant Nos. 2024A1515010089, 2022A1515010179), the Science and Technology Planning Project of Guangdong Province (Grant No. 2023A0505050106), and the National Key R&D Program of China (Grant No. 2023YFA1011601).

heading: A Derivation of Debiased Cross-entropy Loss Function
text: The overall model parameters are defined as Θ = {θ, ω}. The model’s backbone fθ : X → Rd extracts feature embeddings of dimension d from the inputs. Following the backbone, a linear classifier produces logits Φ(·) = ωT · fθ(·) : X → R|Yt|, where ω represents the classifier weights for the corresponding dimensions and Yt represents the current label set. The softmax probability of new entity is defined as: pyi =
eΦyi (xi)∑ y′∈Yt e Φy′ (xi) . The
derivation of Debiased Cross-entropy Loss Function is proved as follows:
∂Lce ∂ωy′ = ∂Lce ∂pyi · ∂pyi ∂Φy′(xi) · ∂Φy′(xi) ∂ωy′
= − 1 ln 2 · pyi · fθ(xi) · ∂pyi ∂Φy′(xi) = fθ(xi)
ln 2 · py′ ∝ eΦy′(xi)(y′ ̸= yi)
(8)
for the same input xi, fθ(xi) ln 2 can be viewed as a constant. Therefore, the gradient penalty of the new entity over the old entity is proportional to the probability value of the old entity.

heading: B Datasets
text: Table 3 shows the detailed description of each dataset. Table 4 shows the format of inputs and outputs for Sequence labeling task. FL means the full ground-truth label for all steps. During the learning process, we will label unseen entities as non-entities [O].

heading: C Baselines
text: The introduction about the baselines in the experiment and their settings are as follows:
• SelfTrain (Rosenberg et al., 2005; De Lange et al., 2019): SelfTrain utilizes the labels generated by the predictions of the old model on the new dataset, combined with the labels of the new entities, to guide the training of the new model.
• ExtendNER (Monaikul et al., 2021): ExtendNER introduces knowledge distillation to review the knowledge of old entities, aiming to align the outputs of the old and new models for old entities using KL divergence. In contrast to SelfTrain, ExtendNER retains specific
structural information through the probability distribution of the model output. The coefficient of the distillation loss λ = 2.
• CFNER (Zheng et al., 2022): CFNER proposes a unified causal framework to extract causality from both new entity types and the Other-Class and employs curriculum learning to alleviate the impact of label noise and introduce a self-adaptive weight to balance the causal effects between new entity types and the Other-Class. The number of matched tokens K = 3, the initial value of balancing weight λbase = 2 and the initial value of confidence threshold δ1 = 1.
• DLD (Zhang et al., 2023c): DLD decomposes a prediction logit into two terms, measuring the probability of an input token belonging to a specific entity type or not. The coefficient of the distillation loss λ = 2.
• RDP (Zhang et al., 2023b): RDP introduces a task relation distillation scheme with two aims: ensuring inter-task semantic consistency by minimizing inter-task relation distillation loss and enhancing model prediction confidence by minimizing intra-task selfentropy loss. The coefficient of inter-task relation distillation loss λ1 = 0.3 and the coefficient of intra-task self-entropy loss λ2 = 0.1.
• OCILNER (Ma et al., 2023): OCILNER introduces a novel representation learning method aimed at acquiring discriminative representations for entities and non-entities, which can dynamically identify entity clusters within non-entities. The threshold for relabeling samples βi = 0.98− 0.05 ∗ (t− i), where t is the current step, and i is the id of the old task.
• ICE (Liu and Huang, 2023): ICE freezes the backbone model and the old entity classifiers, focusing solely on training new entity classifiers. This approach includes two methods: ICE_O and ICE_PLO. The former combines logits of non-entity with logits of new entities for output probability computation during training, while the latter combines all previous logits with new entity logits.
• CFPD (Zhang et al., 2023a): CPFD introduces a pooled feature distillation loss that
Inputs Xinhua news agency , Beijing , August 31st FL B-ORG I-ORG I-ORG O B-GPE O B-DATE I-DATE
Inputs There were no direct effects of the earthquake ’ s
FL O O O O B-Influence O O B-Catastrophe O O shaking due to its low intensity. B-Motion O O O O O
adeptly balances the trade-off between retaining knowledge of old entity types and acquiring new ones and a confidence-based pseudolabeling method for the non-entity type. The balancing weight λ = 2.

heading: D Metrics
text: The last step Macro F1 result AT and the average Macro F1 result Ā are defined as follows:
at = 1
|Dt|
|Dt|∑
i=1
1(argmax y′∈Yi Φt,y′(xi) = yi), (9)
where at represents the F1 score of the tth entity, |Dt| repesents the number of entities and 1(·) is the indicator for Φt,y′(xi) = yi.
AT = 1
N
N∑
j=1
aj , (10)
where AT stands for the MacroF1 score at incremental step t.
Ā = 1 N
N∑
k=1
ATk , (11)
where Ā stands for the average MacroF1 score for all incremental steps.

heading: E Additional Experimental Results
text: Table 6 highlights the superiority of our method IS3 over previous methods on MAVEN. However, due
to its larger number of classes, the performance of the model decreases in subsequent incremental steps. Table 7 shows the results of the experiments conducted on OntoNotes5. Our method IS3 achieves improvements over the previous SOTA ranging from 5.47% to 10.52% in MarcoF1 score, and 3.89% to 6.53%, under four settings (FG-1-PG1, FG-2-PG-2, FG-8-PG-1, and FG-8-PG- 2) of the OntoNotes5 dataset.
As shown in Figure 11, the previous method exhibits a rapid decrease in probability distribution with increasing incremental steps, coinciding with a decline in the F1 score. In contrast, our approach IS3 effectively mitigates the model’s penalization of old entities, thereby maintaining good performance.
We compare the runtime and the number of updated parameters in Table 5. The results indi-
cate that the proposed IS3 requires less than 5% more training time than SEQ and less training time than most previous methods, such as CFNER and CPFD.

heading: Future Work
text: Moreover, the current method does not thoroughly explore the relationship between the penalty effect and the dataset, leaving potential avenues for future research