SKD-NER: Continual Named Entity Recognition via Span-based Knowledge Distillation with Reinforcement Learning
Authors: Yi Chen
Liang He
Venue: EMNLP 2023 Main (2023)
Keywords: Continual learning
Named Entity Recognition
Knowledge Distillation
Reinforcement Learning
TL;DR: We utilize knowledge distillation (KD) to preserve memory and employ reinforcement learning strategies during the KD process to optimize the soft labeling and distillation losses generated by the teacher model.

Abstract: Continual learning for named entity recognition (CL-NER) aims to enable models to continuously learn new entity types while retaining the ability to recognize previously learned ones. However, the current strategies fall short of effectively addressing the catastrophic forgetting of previously learned entity types. To tackle this issue, we propose the SKD-NER model, an efficient continual learning NER model based on the span-based approach, which innovatively incorporates reinforcement learning strategies to enhance the model's ability against catastrophic forgetting. Specifically, we leverage knowledge distillation (KD) to retain memory and employ reinforcement learning strategies during the KD process to optimize the soft labeling and distillation losses generated by the teacher model to effectively prevent catastrophic forgetting during continual learning. This approach effectively prevents or mitigates catastrophic forgetting during continuous learning, allowing the model to retain previously learned knowledge while acquiring new knowledge. Our experiments on two benchmark datasets demonstrate that our model significantly improves the performance of the CL-NER task, outperforming state-of-the-art methods.

SKD-NER:
Continual Named Entity Recognition via Span-based Knowledge Distillation with Reinforcement Learning
Yi Chen LLinn Hel.2.,, Lei Wang Zhenxiang Han 1.2
School of Computer Science and Technology, Xinjiang University, Urumqi 830017, China *Xinjiang Key Laboratory of Signal Detection and Processing, Urumqi 830017, China Department of Electronic Engineering, and Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China tmail 1. com, hel teliang@tsinghua edu. cn
Stephen263 tmaammiilll
Abstract
Saplan
Continual learning for named entity recognition (CL-NER) aims to enable models to continu- ously learn new entity types while retaining the ability to recognize previously learned ones. However, the current strategies fall short of ef- fectively addressing the catastrophic forgetting of previously learned entity types. To tackle this issue, we propose the SKD-NER model, an efficient continual learning NER model based on the span-based approach, which in- novatively incorporates reinforcement learn- ing strategies to enhance the model's ability against catastrophic forgetting. Specifically, we leverage knowledge distillation (KD) to retain memory and employ reinforcement learning strategies during the KD process to optimize the soft labeling and distillation losses gener- ated by the teacher model to effectively prevent catastrophic forgetting during continual learn- ing. This approach effectively prevents or miti- gates catastrophic forgetting during continuous learning, allowing the model to retain previ- ously learned knowledge while acquiring new knowledge. Our experiments on two bench- mark datasets demonstrate that our model sig- nificantly improves the performance of the CL- NER task, outperforming state-of-the-art meth- ods.'
Sepbes
07c
Taktl
Nav
Yol
15 10he Sax
Nav Yokk
Ta60 kg
3ead
An illustration of the sequence labeling approach in SKD-NER. Assuming a model learns taskl:Per and task2:Loc in OntoNotes5 sequentially, the left side represents the conventional sequence label- ing method, while the right side represents the approach we adopted.
Figure 1:
et al., 2022; Parisi et al., 2019). However, continual learning has always faced catastrophic forgetting, which has become a pervasive issue for continual lcarning NER tasks (McCloskey and Cohen, 1989; Robins, 1995; Kirkpatrick et al., 2017). Specifi- cally, simply fine-tuning the NER system based on new data often results in a significant drop in per- formance on previously learned tasks, which poses major challenge for achieving human-level intel- ligence in continual learning for NER (CL-NER). This is in contrast the natural ability of humans to learn new entity categories without forgetting previously learned ones.
1 Introduction
The introduction of continual learning methods has enabled systems to continuously learn from new data and reduce their dependence on initial train- ing data. Moreover, these methods facilitate model updates and fine-tuning, enhancing their scalability (Shin et al., 2017). By combining continual learn- ing with NER tasks, systems can significantly im- prove their ability to perceive the constantly chang- ing real world amidst the emergence of new tasks and data sources, these functionalities can be for- mulated paradigms of continual learning (Jin
In the context of continual learning, the model training process is typically divided into CL steps, with each step being specific to the current task. In the case of CL-NER, only new entity types rec- ognized in each CL step. However, this approach can lead a situation that is easily overlooked.
whereby entity types that are not required be rec- ognized in the current step (c.g.. ORG) may need to be learned in the future or have been learned in
*Conresponding author.
'Code is available at http://ggihaa..co///hhhnn22775/

---

the past. In traditional sequence labeling methods,
fits well with our sequence labeling approach. This approach offers several advantages: 1) It allows for knowledge distillation and retention of old knowl- edge during the CL process. 2) The introduction of reinforcement learning effectively reduces la- bel noise while also addressing the issue of catas- trophic forgetting that arises due to frequent weight updates for the same entity in different steps based different categories. The proposed framework of the model can be used as a reusable framework for continuous learning knowledge distillation and applied to other model migration or continuous learning domains.
22
14
Fedilanal Scouetee HOC Gur aequecc late RE
80
0
We evaluated our model on two Named Entity Recognition (NER) datasets, namely OntoNotes5(Hovy et al., 2006), and Fewnerd(Ding et al., 2021). The experimental results demonstrate that our proposed SKD-NER model significantly outperforms existing continuous learning NER models and achieves a new state-of-the-art (SOTA) performance. Notably, SKD-NER almost elimi- nates catastrophic forgetting on relatively simple OntoNotes, thereby achieving "continuous learn- ing" in the true sense. Our contributions can be summarized as follows:
Figure 2: An illustration of the impact of sequence labeling methods on OntoNotes5.
entity that is not a required type in the current step is assigned global O tag to indicate that the model does not need to recognize this type in the current task(refer to Figurel). However, this can result in each entity's s category needing frequent parameter updates in each different CL step, due to this incoherent optimization. We believe this exacerbates catastrophic forgetting and label noise interference. To test this hypothesis, we conducted experiments, the results of which are presented in Figure2.
We propose a Continual Named Entity Recog- nition model and a reinforcement learning- based knowledge distillation framework that ensures the model's effectiveness for continu- ous learning, which can be further leveraged in other fields of continuous learning knowl- edge transfer.
With the goal of addressing the issues of catas- trophic forgetting and label noise in CL-NER (Le- cun et al., 1998), we propose SKD-NER. To miti- gate the adverse effects of frequent parameter up- dates due incoherent optimization in continuous learning, our model first computes an entity clas- sification matrix for each segment of text to be processed, inspired by the multi-headed attention mechanism (Vaswani et al., 2017). This matrix computes score for the different entity classes to be recognized, thereby converting the entity recog- nition problem into a binary classification problem. In the CL setting, we enable the model perform well in multi-label learning on the span classifica- tion, while equipping it with knowledge distillation at the span entity level based on the Bernoulli dis- tribution it produces. To address the label noise problem in the incremental learning process, we introduce a reinforcement learning strategy for the student model in the knowledge distillation process (Jiang, 2023), using the most suitable knowledge distillation method for the current student model. For model prediction, we introduce a multi-label classification loss function (Su et al., 2022), which
Innovatively introduce reinforcement learning strategies to support Continual Named Entity Recognition, while optimizing traditional se- quence labeling methods and loss functions to address catastrophic forgetting and label noise problems in continuous learning.
Through extensive experiments, we demon- strate that our approach achieves state-of-the- art performance in Continual Named Entity Recognition and can be scamlessly integrated plug-and-plas module to further enhance the performance of other Continual Named Entity Recognition models.
2 Related Work
2.1 Continual Learning NER
Recent research has expanded the application of Continual Learning (CL) from Computer Vision
to Natural Language Processing (NLP) tasks, par-

---

ticularly NER. While most CL-related works in compuler vision focus on accuracy-oriented tasks like image classification, their direct application to CL-NER has shown unsatisfactory performance due to challenges in preserving old knowledge in Other-class samples.
developing models that can learn new tasks while retaining the memory of previously learned knowl- edge. For example, Ruder (2019) uses RL to fine- tune pre-trained models with a combination of dif- ferent reinforcement learning strategies to adapt them to new tasks. Bo et al. (2019) leverages a selector to choose source domain data that is close to the target and accepts rewards from both the discriminator and transfer learning module. Rama- murthy al. (2022) present NLPO (Natural Lan- guage Policy Optimization), a policy optimization- based RL algorithm that dynamically learns task- specific constraints on language distribution.
Chen and Moschitti (2019) pioneered the study of knowledge transfer in sequence labeling NER models from source to target domains with new entities, using a neural adapter module to han- dle diverse entity distributions. Following this, AddNER, ExtendNER (Monaikul et al., 2021), and LR (Xia et al., 2022) were developed under class- incremental setting for CL-NER, employing se- quence labeling methods with knowledge distilla- tion. AddNER uses a multi-head approach, while ExtendNER and LR employ single-head layouts with different strategies for handling O tags and old entity mentions. However, current methods such as AddNER and ExtendNER still face forward incom- patibility issues and require cumbersome coopera- tion with knowledge distillation. LpaanL, and Chen, 2023) explores the potential of span- based models for solving CL.NEE with Excellent forward compatibility to solve this problem.
Inspired by these studies, in this work, we use RL to expertly select appropriate distillation tem- perature and loss weights for knowledge distilla- tion to support continuous learning. This approach successfully mitigates the impact of catastrophic forgetting on model recognition accuracy during continuous learning.
Our Approach
In this section, we will first introduce the task set- ting for Continual Learning Named Entity Recog- nition (NER). After presenting the overall structure of the SKD-NER model, we provide a detailed explanation of how the model integrates reinforce- ment learning strategies.
In the context of CL-NNE, token-noise is also a very important issue be concerned about, self- training (Rosenberg et al., 2005; De Lange al., 2019) has been a straightforward solution for learn- ing old knowledge from Other-class samples. How- ever, this approach suffers from error propagation between models. Recently, Das et al. (2022) have proposed contrastive learning and pretraining tech- niques to address the problem of token-noise in few-shot NER. CENER (Zheng et al., 2022) pro- poses a causal framework in CL-NER that explic- itly addresses the challenges posed by noien- In contrast, Our approach utilizes reinforcement learning strategies to adjust the process of knowl- edge distillation, optimizing the process and ensur- ing the forward compatibility of the model, while also addressing the issue of token-noise to some extent.
3.1 Problem Formulation
Taking into account the non-overlapping nature of entity types in continual learning NER tasks, we follow recent work and propose CL-NER (Xia et al., 2022) in an incremental setting. Given a se- ries tasks Tı, T2, .... ; Tn and their corresponding training sets D1, D2, Dn. for each task Tn, a new entity type to be recognized and its training set Dn with annotations for the current entity type are defined.
Specifically, we first define a task Ti and train a model Mj on the dataset Dj to recognize the en- tity type Fg. Then, task T3 defines the model M2 to recognize a new entity type Ez on the dataset D2. It is noteworthy that M2 is obtained through knowledge distillation with reinforcement learning based on M1 to achieve the ability to recognize entity type E2 while retaining the ability to rec- ognize entity type E,. Similarly, in the following incremental steps, we train the previous model Mn 1 on the dataset D, to obtain the new model Mn, while incorporating a reinforcement learning
2.2 Reinforcement learning
Reinforeement learning has found wide application in natural language processing, including machine translation, dialogue systems, and text summariza- tion. More recently, researchers have focused on the potential of RL in the field of continuous learn- ing for natural language processing, with the aim of

---

Toski
Sigkes Clas van
New York Testenty
Stpphos
SKi
Choe
Previous Task Distillation Label
Previous Model Jeccerr modicl
Peettvon
KL Diverganze Luss
Choo
0%
Dinder
Task2 1o0
Ster
Akylan Chen vas
Nen York
Skglluo
Yadarday
Class Imbelance
Cumust Task
Injut
Loss
Figure 3: Overall architecture of SKD-NER comprises a shared contextual encoder that is applicable to all tasks and different span representation layers each entity type in each task. The Bernoulli KT. loss and Span loss are calculated on the corresponding entity-relevant relevant span matrices for previously learned entities and currently learned entities, respectively. The RL-KD strategy layer is responsible for identifying the critical parameters for knowledge distillation.
strategy into the knowledge distillation process to ensure the recognition of all entity types defined so far and to reduce catastrophic forgetting and the impact of label noise the model.
matrix H E Raxdd, for each token as:
h1, h2, ......... hn T2,, mn)
Span Prediction Layer. Many scholars have thoroughly explored the generation of Span Predic- tion from tokens and have achieved quite effective results. However, due to the structural bias of Span Prediction which has not been fully understood, Fu et al. (2021) treated span prediction combiner to re-identify named entities from the outputs of different systems. In order further improve the recognition accuracy of NER models for nested or overlapping discontinuous entities, Zhang et al. (2021) used text syntax dependency to guide the construction of a graph convolutional model to achieve Span Prediction, while Su et al. (2022) employed a multi-head attention mechanism to compute the span matrix. We adopted the latter method, using the starting position token and end- ing position token of the entity processed by two feedforward layers for dot product calculation obtain the prediction the span. Now that we have obtained the representation hú of the sentence, the process of representing spans can be described as follows:
3.2 SKD-NER Model
We have introduced innovative and effective SKD-NER model (refer to Fig. 3) that can learn diverse entity types in a sequential manner for each task. The model takes an imput sentence X compris- mn]. We define "span" as a cohesive sequence of tokens that initiate with x and culminate with zj. where 1 S S S n. At the (-th incremental step, the SKD-NER model endeavors to represent each span in a matrix h4- In this matrix, each span consisting of contiguous tokens Siy is assigned label corresponding to the current K -th entity class. The SKD-NER model consists of the contextual encoder, Span prediction layer, label loss layer, and the RL-KD (reinforce- ment learning for knowledge distillation) strategy layer.
system
ing n tokens: [2 22, .2,,.,, -..,-..,))
Contextual Encoder. Given an input sentence X comprising tokens [21, T2, ..., ]. to capture the dependence between tokens within input sentences, we link each token in X with its corresponding representation in pre-training language model (e.g., BERT). We define F. = [c1, f2, en] E guxde to represent the embedded vectors of input X. After PLM processing, We end up with new
s(i,j) = SpanPre[h,, har.. hy]
(2)
sa(t,)) = PFN**(h) (FNNi((h))
(3)

---

Where represents the a-th type of entity to be identified, represents the starting position to- ken, and j represents the ending position token. In order to fully utilize boundary information, we in- troduced the Relative Position Encoding (ROPE)
(see Fig,4), which will act on the Bernoulli distri- butions qi of soft distillation labels for each span of each old entity type. Specifically, during the soft distillation process, the original probability distri- bution is typically normalized by the softmax and then multiplied by a temperature factor I' to obtain smoothed probability distribution:
during the span prediction process Su et al. (2022). This encoding explicitly injects relative position information into the model. Specifically, after in-
exp (zi/T)
jecting ROPE position encoding, span (i can be represented follows:
9: E,exy exp (2//T)
(7)
These pseudo-labels are used to compute the Bernoulli KL divergence loss of the current model M (student model):
(a() FFN R,, T FFN R,) = =FFN (h.)
(4)
( FFN
(p Me+1 KL
KD
PPBk-Pi -P
(8)
Label Loss Layer. Building on the span-based method, we generated a marked global upper- triangle matrix for each sentence be learned. To address this upper-triangle matrix, we devised scoring function characterize the relationship be- tween span and the current entity type as follows:
Here, E denotes all the learned entity recogni- tion types up to the current step, p represents the soft distillation labels generated by the teacher model, and * represents the labels produced by the student model for the previous entity types.
Evelustion Performance of Golden labels
Reward
(SjA(ij) E Pa
0 i (4,5) $ No inf i>;;
(5)
Cument Model (Student mock))
Here, Pa represents the positive set of entities of type a and Na represents the negative set of entities of type a. To enhance the ability of contin- uous learning NER to recognize entities in the cur- rent task, introduced span-based cross-entropy loss. This loss not only encourages the model to better learn boundary information but also ensures forward comparability of the model's predictions during the continuous learning process.
Processed soft labels for KD
Select value of Temperature
Previous model (beacher model)
(x):aa)))
Agent
Envirsoment
Cyrn
log 1+
Σ 1
exp ((-1) (111)(tsa((h.i)) (6)
Semantic represrattion
labels
Span loss
3.3 RL-KD strategy layer
Figure 4: Overall architecture of RL-KD consists of three parts: State, Action, and Reward. This module adjusts the critical parameters knowledge distillation to optimize model performance.
To preserve the model's recognition ability for pre- viously learned entity types, we employ knowledge distillation (Gupta et al., 2019; Hinton et al., 2015) to prevent catastrophic forgetting. Specifically, in the K-th incremental step (K' > 1), we first use the previously learned model M2 (teacher model) make a one-pass prediction the entire current training set D;. up to the entity type Eş learned in the previous step of the current task. During this process, we introduce a reinforcement learn- ing strategy to optimize the distillation temperature
As with all knowledge distillation works, after one- pass prediction at each step, the final knowledge distillation loss is:
L - aL BLKD
(9)
State. Our reinforcement learning approach maintains a sequence of environment states

---

81, 82, ...y 8j. They summarize the input instance and the features of the teacher model, enabling wise decisions to be made accordingly. We design as a vector of real numbers F(sy), which in- cludes the concatenation of three features. The first feature is the vector representation R(z,) E Rd of the input instance 24. In this paper, we use the score matrix obtained by the span prediction layer the semantic representation input. The second feature is the prediction by the teacher model M₁ on the current imput text sequence 2, for all k types already identified before the (k- + 1)-th task. The third feature is the loss of the student model on the input text sequence I4, which is the actual loss of the new entity types that the student model needs to recognize for the current task.
the distillation loss KD (BQq.7 to train the student model. To incentivize better model generalization, we use the accuracy metric on the development set Do and the student model loss as the reward, where is a hyperparameter balancing the reward from the training set and the development set. Note that the reward is not given immediately after each step is taken. Instead, it is deferred until the completion of the entire batch training.
reward -y (-LOR Lpr.)
(13)
+ +11= 7) Accuracy Do
4
Experiments
4.1 Settings
Datasets. We conduct experiments on two widely used datasets, i.c., OntoNotes5 (Hovy et al., 2006), (Ding et al.. 2021). Meanwhile, we follow recent works (Monaikul et al.. 2021; Xia et al., 2022) to convert the widely used standard NER corpora into separated datasets acting series of CL synthetic tasks in maassincremmntal setting. Both datasets, OntoNotes5 and FewNerd, reflect the model's per- formance in continual learning tasks to some ex- tent. OntoNotes5 requires only one entity type to be learned for each task, which can reflect the model's performance in simple continual learning tasks. FewNerd, on the other hand, requires multi- ple entity types to be learned for each task, thereby reflecting the model's performance in more com- plex continual learning tasks closer real-world scenarios. We placed the more detailed dataset settings in the appendixA.
Action. The soft labels generated by the teacher model are associated with the distillation temper- ature and the distillation loss weight during the knowledge distillation process. The agent adjusts the distillation temperature and the weight of the distillation loss for the current teacher model. The policy function (a(s): 9z) determines a distribution over actions on state, from which an action value aj E {0, 1} is sampled. 0 represents the trainable parameters in the policy function.
To (s), az) = temp (8), (,), klweight (s)) (10)
temp (z), a, [ AF (0,)] ag)
(11)
klweight (8), 0z) = min (weight +BF (8;)) + (1 a5) weight, 0.1]
Training. We use bert-large-case (Devlin et al., 2018) as the contextual encoder for our model. To split the original training/development set into a series of CL tasks, we randomly divide the sam- ples into unrelated tasks, following previous work (Monaikul et al., 2021). We pre-define fixed der of classes as the setting for the CL order, as in previous CL work (Hu al., 2021). However, avoid excessive randomness in the experiments, we pre-define multiple entity class learning orders for both datasets, and the final experimental results are the average of the results under multiple learning orders. The pre-defined entity class learning orders and more detailed training settings are listed in the appendixB.
(12)
where F(s)) E Re (C +) is the state vector and trainable parameter 0 (A E Rd+(C+1), b E R-) The result of the above strategy function definition is adjusted value assigned to two key parameters in the knowledge distillation process.
Reward. The reward function is related to the performance of the student model trained from the distillation of the teacher model. We define a batch of training instances, denoted as & {r,, 1..11........................1.. Ti m 1). where h represents the batch ID and represents the batch size. For each instance zi < < m 1), we con- struct state vector 8yk for each teacher model Mg. and sample an action Ojk according to the pol- icy To T0($},a) af)) 10q.10). For all sampled ajk- we integrate the average of the KL loss (Eq.8) into
Metries. Due the class imbalance problem in NER, we use Micro Fl and Macro F1 to measure the model performance. We report the Micro Fl and Macro Fl of all learned types up to each step,

---

OntoNotes5
Method
Step1
Step2
Step3
Step4
Step5
Step6
6
AddNER
82.53
83.65
84.46
85.02
85.43
84.97
2.44
EXtendNER
82.69
83.16
84.34
84.54
85.06
84.83
2.14
L&R
92.06
88.09
85.69
83.79
83.38
83.02
-9.04
SpanKL
85.6
87.97
88.34
88.84
88.63
88.12
2.52
SKD-NER(Ours)
87.33
91.47
92.36
90.76
88.54
88.17
0.84
Table 1: Macro-F1 of different five models at each step on OntoNotes5, and represents the degree of forgetting of the model, which is the difference between the Fl of the final model and the Fl of the first step.
07:CR
Diand
CrE
SKY4L
5038
Figure 5: The anti-forgetting performance of four models on various entities OntoNotes5.
and unless otherwise noted, these results are the average of the results obtained from all pre-defined learning orders.
4.2 Results
Comparisons with State-Of-The-Art. We adopt a more realistic datasct scenario by randomly divid- ing the samples into unrelated tasks, and we com- pare our method with previous baselines on these datasets. The experimental results on OntoNotes5 are summarized in Table 1, and Figure 5. In most cases, method achieves the best performance. In particular, we use delta value quantify the difference between the final results of each model's continual learning and the results of the first step of learning, which largely represents the model's 's anti- forgetting ability. Our model outperforms the previ- ous state-of of the-art by large margin in this metric. Additionally, in Figure 6, we observe an interesting phenomenon that for particular entity, the predic- tion accuracy of our model distilled models after one even two steps of distillation is higher than the accuracy on the previous predictions made on the same entity. We attribute this improvement generalization ability to the knowledge distillation
Baselines. We consider five baselines this
work: ExtendNER and AddNER (Monaikul et al.. 2021), L&R(Xia et al., 2022), CFNER(Zheng et al., 2022), and SpanKL(Zhang and Chen, 2023). Ex- tendNER was the previous state-of-the-an art method in CL-NER, and L&R was the recent state-of-the- art method in CL-NER. CENER extracts causal effects CL-NER tasks and achieves advanced performance multi-entity type tasks. SpanKL. also uses a span-level named entity recognition model for continuous learning, and it performs well in terms of forgetfulness resistan. It is worth noting that these basclines different dataset splitting methods, and we believe that the dataset splitting method is an extremely important but easily over- looked issue in continual learning tasks. To better approximate real-world scenarios, we randomly divide the samples into unrelated tasks.

---

under reinforcement learning, where future distil- lation labels may be more accurate and consistent than the originally learned labels. Due to space constraints, we have included the results on the Fewnerd dataset in the appendixD.
Method
OntoNotesS
Step1
Step2
Step3
Step4
Step5
Step6
ORG
85.2
82.32
82.4
82.16
82.14
81.92
-1.28
ExtendNER
PER
91.31
90.1
88.38
88.92
89.38
89.43
-1.88
GPE
ORG
57.33
88.62
87.55
87.87
85.76
85.39
-1.94
SKD-NEE
PER
94,03
93.98
93.51
93.22
92.4
92.1
(Ours)
GPR
94.64
94.79
16
95.2
94.06
93.4
-1.24
Ablation Study. We ablate our method, and the results are summarized in Table 2. To validate the effectiveness of the proposed reinforcement learn- ing knowledge distillation method, we also apply this method to traditional sequence labeling meth- ods and BCE-loss methods. Specifically, in w/o SL and w/o SPL, our model still applies the rein- forcement learning knowledge distillation method, while in w/o SL RL, we remove both the rein- forcement learning policy and the span sequence labeling method. The results show that the rein- forcement learning policy plays sigmificant role in our framework. Additionally, the new sequence la- beling method also helps the model further combat catastrophic forgetting. Due to the space limitation the article, we put the results of FewNerD into the appendix D.
Table The Anti-CF performance analysis, ana- lyzed the Anti-CF performance of our model and Ex- tendNER at each step on three types of entities, and used 6 to represent the total forgetting amount.
labels in the model predictions as the predicted la- bels. From the figure, can that compared with ExtendNER, SKD-NER has higher values on the diagonal of the confusion matrix. This indicates that SKD-NER less affected by incorrectly prop- agated labels and has more accurate discrimination between different entity types than ExtendNER. These results are consistent with the improvements shown in Table 1.
OntoNotes5
Method SKI--NER(Our))
Stepl
Step2
Step3
Step4
Step5
Step6
87.33
91.47
92.36
90.76
88.54
88.17
w/o SL & RL
82.57
82.76
83:45
82.19
83.79
82.67
w/o SL
83.69
83.59
84.17
84.26
84.76
84.43
w/o RL
86.83
86.37
86.38
85.2
85.54
85.38
wlo SPI.
86.79
91.13
92.07
90.04
88.43
88.14
Table 2:
The ablation study of our method on OntoNotes5, SL: sequence labeling, RL: reinforcement learning strategy, w/o SPL:Replace span loss with BCE loss.
Figure 6: The aconfusion matrix of two models on various entities in OntoNotes5.
Anti-CF performance analysis. We perform Anti-CF performance analysis on the model un- der OntoNotes5, and the results can be seen in Table 3. For three entities, we track the forgetting situation after 6 steps compared to the initial state. In most cases, our model achieves the best perfor- mance. The experimental results demonstrate that our model essentially solves the CF problem on OntoNotes5, which relatively simple dataset. We also put FewNerd's 's results in the appendix D, its results also prove the validity of our method. Label noise reduction. To validate our hypothe- sis that SKD-NER alleviates the label noise prob- lem in continual learning and leads to improve- ments, we plot the normalized confusion matrix between different entity types based on the final predictions (Figure 6). Specifically,, we use the BB TB-X" X' (X denotes a specific entity type) labels in the ground truth as the true labels and the B X'
Conclusion
This paper proposes an effective continual learning named entity recognition (NER) model, SKD-NER. aiming maintain high accuracy for existing en- tity types while identifying new entity types. We propose continual learning NER model based on the span method, SKD-NER, which combines re- inforcement learning policy. We use knowledge distillation (KD) to preserve memories in the con- tinual learning process and adopt reinforcement learning policy with multi-label classification loss for prediction, effectively alleviating the impact of label noise in continual learning. The experimental results demonstrate that the proposed model not only outperforms oate-offthheart methods but also almost solves the catastrophic forgetting problem on OntoNotes5.

---

6 Ethics Statement
Long Papers), pages 3198-3213, Online. Association for Computational Linguistics.
For ethical considerations, we provide the follow- ing clarifications: (1) We conduct all experiments on existing datasets sourced from public scientific research. (2) We describe the statistical data of the datasets and the hyperparameter settings of our method. Our analysis and experimental results are consistent. (3) Our work does not involve sensitive data or sensitive tasks.
Jinlan Fu, Xuanjing Huang. and Pengfei Li. 2021. Span- ner: Named entity re-frecognition as span prediction. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 35, pages 6718-6726.
Prakhar Gupta, Yatin Chaudhary, Thomas A Runkler, and Hinrich Schütze. 2019. Neural topic modeling with continual lifelong learning- In International Conference on Machine Learning, pages 3907-3917.
7 Limitations
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, 2015. Distilling the knowledge in neural network. arXiv 2(7).
Although the proposed model can partially solve the catastrophic forgetting problem, there is still significant room for improvement in more complex dataset testing. Additionally, due to the introduc- tion of knowledge distillation with reinforcement learning policy, the training time of our model is slightly longer than that of baselines.
preprint
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% Solution. In Proceedings of the Human Lon- guage Technology Conference of the NAACL, Com- panion Volume: Short Papers, pages 57-60, New York City, USA. Association for Computational Lin- guistics.
References
Xinting Hu, Kaihua Tang. Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. 2021. Distilling causal effect of data in class-incremental learning. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3957-3966.
Wen Bo, Honglei Li, Shiyu Wang, Jun Guo, Tongliang Liu, and Dacheng Tao. 2019. A minimax game for instance based selective transfer learning. In KDD, pages 34-43.
Liang Chen and Alessandro Moschitti. 2019. Transfer learning for sequence labeling using source model and target data. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 33, pages 6260-6267.
Fei YuanllLinjun ShoulJjan Peil|Wutao LinllMing GongllYan FullDaxin Jiang. 2023. Reinforced Multi-Teacher Selection for Knowledge Distil- lation. https://aaai.rrgpapers///// (tspppprr//1224-reiffoooo multi-teacher-selecnne fortooo---oooooeee wor--ffoo---kleee www..eege-disstllleeddddeeeeeede
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. 2022. CONTaiNER: Few-shot named entity recognition via contrastive learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 6338-633 6353, Dublin, Ireland. Association for Computational Linguistics.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022. Lifelong pretraining: Continually adapting language models to emerging corpora. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4764 4780, Seattle, United States. Association for Computational Linguistics.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2019. Continual learning: A comparative study on how defy forgetting in clas- sification tasks. arXiv preprint arXiv: htw:/999.0880 1909.08383, 2(6).
Diederik P Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei A. Rusu. Kieran Milan, John Quan, Tiago Ramalho, Ag- Demis Hassabis, Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3511335
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training training of deep bidirectional transformers for language understand- ing. arXiv preprint
nieszka
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang. Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD; A Few-shot Named Entity Recognition Dataset. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1:
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 1 1):2788-2222

---

Michael McCloskey and Neal J. Cohen. 1989. Catas- trophic Interference in Connectionist Networks: The Sequential I Learning Problem. In Gordon H. Bower, editor, Psychology of Learning and Motivation, vol- ume 24, pages 109-165. Academic Press.
Yu Xia, Quan Wang, Yajuan Lyu, Yong Zhu, Wenhao Wu, Sujian Li, and Dai Dai. 2022. Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages 2291-2300, Dublin, Ireland. Association for Computational Linguistics.
Natawut Monaikul, Giuseppe Castellucci, Simone Fil- ice, and Oleg Rokhlenko. 2021. Continual learning for named entity recognition. In Thirty-Fifth Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, EAAI 2021, Virtual Event, February 2-9, 2021.
Yuanmeng Zhang, Qianqian Yang, Yanghua Liu, and Maosong Sun. 2021. Syntax- syntax-enhanced entity and relation extraction with graph convolutional networks. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, pages 7255-7265
Yunan Zhang and Qingcai Chen. 2023. A neural span- based continual named entity recognition model. Proceedings of the AAAI Conference on Artificial Intelligence.
German I. Parisi, Ronald Kemker, Jose L.. Part, Christo- pher Kanan, and Stefan Wermter. 2019, Continual lifelong learning with neural networks: A review. Neural Networks: The Official Journal of the Inter- national Neural Network Society, 113:54-71.
Junhao Zheng, Zhanxian Liang, Haibin Chen, and Qianli Ma. 2022. Distilling causal effect from miscel- laneous other-class for continual named entity recog- nition. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3602-3615, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines. and building blocks for natural language policy optimiza- tion.
A Detailed dataset settings
OntoNotes5 annotates 18 entity types, but prac- tice, to ensure sufficient training samples for each entity type, we filtered out entity types with less than 50 training samples and selected the follow- ing types for training: Organization (ORG), Person (PER), Cco-Political Entity (GPE), Date (DATE), Cardinal (CARD), and Nationalities and Religious Political Group (NORP). For FewNerd, we fol- lowed recent research to construct each task by using a coarse-grained type, and each task con- tains multiple -ea-grriied entity types related to the coarse-grained type. The coarse-grained types include Location (LOC), Person (PER), Organi- zation (ORG), Other (OTH), Product (PROD), Building (BUID), Art (ART), and Event (EVET). Each coarse-grained type contains roughly 10 fine- grained types within it. For example, Product (PROD) contains the following fine-grained entity types: airplane, car, food, game, other, ship. soft- ware, train, weapon.
Anthony Robins. 1995. Catastrophic forgetting, re- hearsal and pseudorehearsal. Connection Science, 1(2):123-146.
Chuck Rosenberg, Martial Hebert, and Henry Schnei- derman. 2005. Semi-supervised self-training of ob- ject detection models. In Proceedings of the Seventh IEEE Workshops on Application of Computer Vision (WACVMOTION 05)- Voluue I- I-Volume 01, pages 29-36. IEEE
Sebastian Ruder. 2019. Neural machine translation and saweencc-tossqqeeeee models: A tutorial using ex- amples from numerous languages.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In Proceedings of the 31st International Con- ference on Neural Information Processing Systems, NIPS' 17, page 2994-3003, Red Hook, NY, USA. Curran Associates Inc.
Jianlin Su, Ahmed Murtadha, Shengfeng Pan, Jing Hou, Jun Sun. Wanwei Huang, Wen, and Yunfeng Liu.
B Detailed training settings
2022. Global pointer: Novel efficient baan-based approach for named entity recognition.
The experiments are run on GeForce RTX 3080 Ti GPU. Each experiment is repeated 5 times. During the evaluation process, we only retain the labels of new entity types and set the other labels in the vali- dation set as other classes. For example, we erase the annotations of PER on the samples assigned to the LOC task learning. At each CL step, we select
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Ilaia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31 International Conference on Neural Information Processing Sys- lems, pages 5998-6008

---

OntoNotes Permutations
ORG -> PER > GPE -> DATE CARD NORP
2: DATE -NORP- + PER CARD + ORG GPE
3: GPE -> CCARD + ORG - NORP DATE - PER
4: NORP ORG + DATE PER GPE CARD
5: CARD - GPEE NORP ORG + DATE
6: PER + DATE CARD - GPE + NORP - ORG
Few-NERD Permutations
1: LOC -> PER - ORG OTH PROD BUID ART EVET
2: ORG + PROD + ART EVET + OTH + PER + LOC BUID
3: PROD > EVET - OTH PER ART 1.C + BUID ORG
4: BUID + OTH + PROD + PER + ORG + LOC - ART + EV
Table 4: Different permutations tasks used on OntoNotes and Few-NERD.
the model with the best validation performance for testing and the next step of learning. For testing, we retain the labels of all entity types identified until the current lask. The pre-defined entity class learning orders are showen in the table
We conducted comparative experiments, abla- tion studies, and other key experiments on SKD- NER model the FewNERD dataset. The experi- mental results demonstrate that our model achieves state-of-the-art performance in mitigating catas- trophic forgetting and entity recognition accuracy in CL-NER, outperforming baselines in most cases.
C Hyper-parameters
In the experiments, we set the dropout rate to 0.1, and we use d = 50 for all subsequent feed- forward networks in span predictions. For the ini- tial model loss, set initial distillation temperature 1. All parameters are tuned using the Adam optimizer (Kingma and Ba, 2015), with a learning rate (Ir) 5 x 10-5 for the bert encoder and x 10-3 for the remaining networks. After BPE tokenization widely used in PLMs, we limit the maximum sentence length to 256, and we only use the representation of the first subword piece to represent the word after the Bert contextual encoder.
SparKL
1, and we set the
OTH FROD
EVST
CLSUP
Figure The anti-forgetting performance of SpanKL. models FewNERD.
D Additional Experimental Results
ExtriR
55
49 082
R04
6.85
FIED
AlTai
AfTax
CL.
Figure 7: The anti-forgetting performance of Extend- NER models on FewNERD.
Figure 9: The anti-fogg performance of SKD-NER models on FewNERD.

---

FewNERD
Method
Stepl
Step2
Step3
Step4
Step5
Step6
Step7
Step8
AddNER
64.01
62.25
61.87
61.16
61.37
62.34
63.88
63.67
-0.34
ExtendNER
64.06
59.02
57.03
55.79
55.65
56.03
56.78
56.09
-7.97
L&R
68.13
66.72
64.51
63.44
60.97
61.23
60.88
60.32
-7.81
SpanKL SKD-NER(Ours) 72.09
67.82
64.04
63.24
62.08
64.13
61.98
63.01
62.04
-5.78
71.8
68.79
68.13
67.15
66.84
66.88
67.14
-4.95
Table 5: Macro-Fi of different five models at each step FewNerd, and å represents the degree of forgetting of the model, which is the difference between the Fl of the final model and the Fl of the first step.
FewNERD
Method
Step1
Step2
Step3
Step4
Step5
Step6
Step7
Step8
SKD NER(OUrs)
72.09
71.8
68.79
68.13
67.15
66.84
66.88
67.14
w/o SL&RL
64.05
59.67
58.68
57.79
56.64
56.03
56.98
57.02
w/o SL
68.63
66.89
64.51
64.32
62.08
61.53
63.05
62.89
w/o RL
67.76
65.23
64.43
63.08
65.19
63.27
64.66
64.07
w/o SPL
72.08
72.01
67.63
67.56
67.02
66.69
66.74
67.03
Table 6: The ablation study of our method on FewNERD, SL: sequence labeling, RL; reinforcement learning
strategy, w/o SPL:Replace span loss with BCE loss.
FewNERD
Method
Step1
Step2
Step3
Step4
Step5
Step6
Step7
Step8
LOC
70.95
72.07
72.66
72.92
73.16
73.07
72.76
72.68
1.73
ExtendNER
PER
63.23
63.62
63.8
64.13
63.95
62.97
62.57
62.04
-1.19
ORG
57.97
58.16
58.45
58.14
57.11
56.81
56.42
55.96
-2.01
LOC
75.09
76.02
75.93
76.0
76.14
75.97
75.96
75.95
0.86
SKD-NER
PER
66.36
66.26
66.22
65.97
65.95
65.52
65.03
65.18
-1.18
ORG
61.22
61.95
61.96
61.75
61.55
61.01
61.24
61.18
-0.04
Table The Anti-CF performance analysis, we analyzed the Anti-CF performance of our model and ExtendNER each step three types of entities, and used to represent the total forgetting amount.


Reviews:
[{"review_id": "rHcj7q4QNS", "reviewer": "Reviewer_JuEv", "summary": "", "questions": "Question A: The motivation in the introduction should be better explained. For example, label noise (Line 071) is not explained; Performance comparison (Figure 2) alone is not enough to test the hypothesis.\nThe necessity of dynamically adjusting the hyper-parameters of distillation is expected.\n\nQuestion B: Experimental results are not found to support your claim that “our method can be seamlessly integrated as a plug-and-play module to further enhance the performance of other Continual Named Entity Recognition models.”\n\nQuestion C: Others might have difficulty reproducing the results due to the instability of reinforcement learning. Will you release the code?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive suggestions from reviewers.\nGeneral response: \n\nFirstly, experiments on the Ontonotes5 and FewNerD datasets demonstrate that our approach achieves state-of-the-art performance in continual learning for named entity recognition on both datasets. Secondly, our main contribution lies in the innovative application of reinforcement learning strategies to the parameter adjustment in the knowledge distillation process. Compared to traditional knowledge distillation methods, our approach has been proven to be more effective in achieving a balance between recognizing new entities and preserving the ability to recognize old entities. Furthermore, we conducted additional experiments to extend our method to other continual learning models.\n\n1.Motivation\n\nOur motivation for conducting experiments stems from an investigation into the underlying principles of knowledge distillation. The distillation temperature, as a parameter controlling the degree of softening of old entities, and the distillation loss weight, as an indicator balancing distillation loss (retaining the ability to recognize old entities) and model loss (learning new entities), have both been experimentally studied with fixed values in previous work. However, this approach inevitably increases the impact of label noise during the model's knowledge distillation process. Therefore, we made attempts combining reinforcement learning strategies. Experimental results demonstrate that our method effectively improves the model's performance while mitigating catastrophic forgetting.\n\n2.In our future work, we expect to further refine our method as a plug-and-play module. This article aims to emphasize the significant contribution of reinforcement learning strategies in improving model accuracy. Currently, we have conducted preliminary experiments to validate the applicability of our method to other continual learning models. Below are the experimental results demonstrating the enhanced performance of the ExtendNER model for continual learning named entity recognition using our method. We have incorporated reinforcement learning techniques to improve its knowledge distillation process while preserving the model's original settings. It can be observed that our method effectively mitigates catastrophic forgetting and enhances the overall named entity recognition accuracy of the final model.\n\n|      Method       |       |       | Ontonotes |       |       |       |\n| :---------------: | :---: | :---: | :-------: | :---: | :---: | :---: |\n|                   | step1 | step2 |   step3   | step4 | step5 | step6 |\n|     ExtendNER     | 82.69 | 83.16 |   84.34   | 84.54 | 85.06 | 84.83 |\n| ExtendNER with RL | 82.69 | 82.09 |   84.56   | 84.61 | 85.35 | 84.94 |\n\n3.Code: We will release our code in the near future for experimental purposes and model reproduction.\n\n4.Typos Grammar Style And Presentation Improvements: We sincerely appreciate your meticulous review, and we will accordingly examine and amend any grammatical issues in the paper while enhancing the style of expression.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper introduces a general reinforcement learning-based knowledge distillation to adaptively adjust the hyper-parameters of distillation loss which makes sense.", "reasons_to_reject": "The motivation should be better organized and explained. The necessity of dynamically adjusting the hyper-parameters of distillation is expected.", "questions_to_authors": "Question A: The motivation in the introduction should be better explained. For example, label noise (Line 071) is not explained; Performance comparison (Figure 2) alone is not enough to test the hypothesis.\nThe necessity of dynamically adjusting the hyper-parameters of distillation is expected.\n\nQuestion B: Experimental results are not found to support your claim that “our method can be seamlessly integrated as a plug-and-play module to further enhance the performance of other Continual Named Entity Recognition models.”\n\nQuestion C: Others might have difficulty reproducing the results due to the instability of reinforcement learning. Will you release the code?\n", "paper_topic_and_main_contributions": "This paper addresses the issues of catastrophic forgetting and label noise by a reinforcement learning-based knowledge distillation. The reinforcement learning is used to adjust the temperature and the weight of the distillation loss for different entity types.\nExperimental results show that the performance is higher than the previous state-of-the-art by 0.05 and 3.47 on two datasets.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A: The motivation in the introduction should be better explained. See Question A for details.\nB: Table 3 is redundant compared with Figure 5. Replacing Table 3 in the analysis of “Anti-CF performance analysis” can better support the claim of “our method almost eliminates catastrophic forgetting on relatively simple OntoNotes”.\nC: Upper-bound performance of multitask learning could be added in Table 1 and Table 5 for better comparison.\nD: Line 426 missing dataset name\nE: Line 483 State-of-the-art\nF: Inconsistent nouns e.g. FewNERD, Fewnerd, Few-NERD, FewNerD\nG: Title of Figure 6\nH: Inconsistent format in Table 4\n"}}, {"review_id": "b1tgYuvD6B", "reviewer": "Reviewer_gUVE", "summary": "", "questions": "- In section 3.2, hoe many span matrix will be used for the whole continual learning? is there any novelty of span matrix compared to Su et al. 2022?\n- In section 3.3, how many RL steps will be used to update the model to reach the expected performance for each entity type classification?\n- If the teacher model is not trained well, how do you avoid the training errors transferred to the student model?\n- For experimental results, there is only Macro-F1 scores reported in all tables, but the Micro-F1 is not reported according to line 458 and 459. Also the standard division should be better reported, which could better show the effect of multiple ordering and different entity types to the average F1.\n- Although different ablation studies are conducted, it is hard to find some deeper analysis or insights from those discussions. Maybe authors could provide some case studies or error analysis of why the proposed method has large performance dropping with steps increasing compared to other baselines.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive suggestions from reviewers.\nGeneral response: \nFirstly, experiments on the Ontonotes5 and FewNerD datasets demonstrate that our approach achieves state-of-the-art performance in continual learning for named entity recognition on both datasets. Secondly, our main contribution lies in the innovative application of reinforcement learning strategies to the parameter adjustment in the knowledge distillation process. Compared to traditional knowledge distillation methods, our approach has been proven to be more effective in achieving a balance between recognizing new entities and preserving the ability to recognize old entities. Furthermore, we conducted additional experiments to extend our method to other continual learning models.\n\n1.In our future work, we expect to further refine our method as a plug-and-play module. This article aims to emphasize the significant contribution of reinforcement learning strategies in improving model accuracy. Currently, we have conducted preliminary experiments to validate the applicability of our method to other continual learning models. Below are the experimental results demonstrating the enhanced performance of the ExtendNER model for continual learning named entity recognition using our method. We have incorporated reinforcement learning techniques to improve its knowledge distillation process while preserving the model's original settings. It can be observed that our method effectively mitigates catastrophic forgetting and enhances the overall named entity recognition accuracy of the final model.\n\n|      Method       |       |       | Ontonotes |       |       |       |\n| :---------------: | :---: | :---: | :-------: | :---: | :---: | :---: |\n|                   | step1 | step2 |   step3   | step4 | step5 | step6 |\n|     ExtendNER     | 82.69 | 83.16 |   84.34   | 84.54 | 85.06 | 84.83 |\n| ExtendNER with RL | 82.69 | 82.09 |   84.56   | 84.61 | 85.35 | 84.94 |\n\n2.In Table 1, we measure the model's entity recognition capability at each step by taking the average accuracy across all entity types. To validate the degree of forgetting for each entity type, we selected entity types with higher quantities from the dataset (Table 3) for evaluation. The results are as follows. Our method significantly improves catastrophic forgetting in the PER and GPE types. Although our method exhibits a higher degree of forgetting in the ORG type compared to the baseline, our final entity recognition accuracy is significantly higher than the baseline. Similarly, our experiments in Table 1 demonstrate that our method achieves state-of-the-art performance in entity recognition accuracy at the final step of the model.\n\n|  Method   |      |       |       | Ontonotes |       |       |       |       |\n| :-------: | :--: | :---: | :---: | :-------: | :---: | :---: | :---: | :---: |\n|           |      | step1 | step2 |   step3   | step4 | step5 | step6 |   δ   |\n| ExtendNER | ORG  | 83.2  | 82.32 |   82.4    | 82.16 | 82.14 | 81.92 | -1.28 |\n| ExtendNER | PER  | 91.31 | 90.1  |   88.38   | 88.92 | 89.38 | 89.43 | -1.88 |\n| ExtendNER | GPE  | 92.81 | 91.56 |   91.91   | 91.27 | 91.35 | 91.34 | -1.47 |\n|  SKD-NER  | ORG  | 87.33 | 88.62 |   87.55   | 87.87 | 85.76 | 85.39 | -1.94 |\n|  SKD-NER  | PER  | 94.03 | 93.98 |   93.51   | 93.22 | 92.4  | 92.3  | -1.73 |\n|  SKD-NER  | GPE  | 94.64 | 94.79 |   95.16   | 95.2  | 94.06 | 93.4  | -1.24 |\n\n3.If the delta value is positive, it indicates an improvement in the model's accuracy at the final step compared to the first step. If it is negative, it indicates forgetting compared to the first step. Although our delta values are not the highest in Table 1, our method achieves the highest model performance at the final step, assuming no forgetting occurs in Table 1.\n\n4.Forgetting in Figure 5: Since our method focuses more on the distillation temperature and weights during the knowledge distillation process, its underlying principle in improving model accuracy lies in balancing the model's ability to learn new entity recognition while preserving the ability to recognize old entities. Due to the influence of temperature adjustment, the model may pay more attention to learning new entities in the next step. However, the final model's recognition accuracy (Table 1) and the degree of forgetting for most individual entities (Table 3) have significantly improved, demonstrating the effectiveness of our method.\n\n5.Span matrix: For each sentence, we use a span matrix to discriminate each entity type. In comparison to Su et al.'s work, our innovation primarily lies in two aspects: firstly, applying the Span Matrix to the field of continual learning, motivated by the use of the Span Matrix to recognize nested entities and improve model accuracy. Secondly, we introduce reinforcement learning strategies into the knowledge distillation process, thus enhancing the performance of continual learning models.\n\n6.RL steps: To achieve the desired performance for entity classification, we set 20 batch sizes as one batch, and after each batch training, rewards are given to update the model.\n\n7.Teacher model: The problem you raised is indeed a major challenge in the knowledge distillation process. There are typically two approaches to address this issue. First, improving the teacher model to enhance its accuracy (Xia et al., 2022). Second, improving the knowledge distillation process. By innovatively employing reinforcement learning strategies to adjust key parameters in the knowledge distillation process, our model can adjust the softness of learning old model labels based on feedback. In Table 1, our initial teacher model accuracy is significantly lower than the L&R model, but ultimately, our model's accuracy is significantly higher than L&R. These experimental results also demonstrate that our proposed method can alleviate this issue to a certain extent.\n\n8.Micro-F1: Due to the limitations of the article length requirement, we did not report the Micro-F1 scores. This is because in the continual learning problem, we have equal attention to the performance of each category. However, our model also achieves state-of-the-art performance in Micro-F1. Below are some performance results for Micro-F1:\n\n|     Method     |       |       | Ontonotes |       |       |       |\n| :------------: | :---: | :---: | :-------: | :---: | :---: | :---: |\n|                | step1 | step2 |   step3   | step4 | step5 | step6 |\n|   ExtendNER    | 81.39 | 83.04 |   83.87   | 84.45 | 84.98 | 84.54 |\n| SKD-NER（ours) | 87.33 | 91.33 |   92.07   | 90.27 | 87.48 | 87.39 |\n\n9.text suggestions: Thanks, we will revise the teaser texts accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a span-based matrix knowledge distillation approach enhanced with reinforcement learning method is introduced \n- Achieve sota on two datasets cross single and multiple entity types classification ", "reasons_to_reject": "- there is one contribution mentioned in the introduction, i.e., the proposed method can be applied to other model migration or continual learning domains as a plug-and-play module, which is not proved based on the experimental results.\n- Experimental results $\\delta$ the degree of forgetting, is not correctly used because the first step only classifies the first entity type, but the final model has already classified all entity types. Then the difference between these two average scores under all entity types and multiple learning orders cannot correctly show whether the model can still forget the earlier entity types. In addition, the value of delta should be the larger the better, to my understanding. However, its value from table 1 and 5 does not show that the proposed approach outperforms other baselines.\n- In the figure 5, the proposed model has the largest F1 score performance dropping for almost each entity type compared to other baselines, although each entity type F1 has a higher starting point. This situation seems like that the proposed model does not mitigate the catastrophic forgetting, and it just improves the each entity types F1 at the corresponding step instead. This situation needs a better exploration and explanations.", "questions_to_authors": "- In section 3.2, hoe many span matrix will be used for the whole continual learning? is there any novelty of span matrix compared to Su et al. 2022?\n- In section 3.3, how many RL steps will be used to update the model to reach the expected performance for each entity type classification?\n- If the teacher model is not trained well, how do you avoid the training errors transferred to the student model?\n- For experimental results, there is only Macro-F1 scores reported in all tables, but the Micro-F1 is not reported according to line 458 and 459. Also the standard division should be better reported, which could better show the effect of multiple ordering and different entity types to the average F1.\n- Although different ablation studies are conducted, it is hard to find some deeper analysis or insights from those discussions. Maybe authors could provide some case studies or error analysis of why the proposed method has large performance dropping with steps increasing compared to other baselines.", "paper_topic_and_main_contributions": "This paper proposes a span-based knowledge distillation approach powered with reinforcement learning to solve continual named entity recognition and catastrophic forgetting issue. Authors argue that their method achieves sota on single and multiple entity types classification on two datasets. Experimental results show that reinforcement learning plays a significant role.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q1Bmr4AaLz", "reviewer": "Reviewer_8unw", "summary": "", "questions": "The paper only discusses continuous learning for single NER tasks. However, in the era of large language models, a single model can handle multiple tasks with a significant number of parameters. Can the proposed method be applied to continuous learning scenarios with large models? Or does the author have any solutions for continuous learning with large models?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive suggestions from reviewers.\nGeneral response: \n\nFirstly, experiments on the Ontonotes5 and FewNerD datasets demonstrate that our approach achieves state-of-the-art performance in continual learning for named entity recognition on both datasets. Secondly, our main contribution lies in the innovative application of reinforcement learning strategies to the parameter adjustment in the knowledge distillation process. Compared to traditional knowledge distillation methods, our approach has been proven to be more effective in achieving a balance between recognizing new entities and preserving the ability to recognize old entities. Furthermore, we conducted additional experiments to extend our method to other continual learning models.\n\n1.Code: We will release our code in the near future for experimental purposes and model reproduction.\n\n2.Experimental results comparison in Table 1: Thank you for pointing out the issue. In Table 1, although ExtendNER and L&R both use bert-base as the encoder, the remaining baselines such as SpanKL use bert-large as the encoder. The experimental results show that our method still achieves state-of-the-art performance. Additionally, we have compared the results of ExtendNER using bert-large as the encoder with our model, and the results are as follows:\n\n|     Method     |       |       | Ontonotes |       |       |       |\n| :------------: | :---: | :---: | :-------: | :---: | :---: | :---: |\n|                | step1 | step2 |   step3   | step4 | step5 | step6 |\n|   ExtendNER    | 82.76 | 83.85 |   84.79   | 84.61 | 85.94 | 85.23 |\n| SKD-NER（ours) | 87.33 | 91.33 |   92.07   | 90.27 | 87.48 | 87.39 |\n\n3.The citation format of the paper: Thank you for bringing up this matter. We will carefully review and revise the citation format in our paper.\n\n4.Limitations: In our analysis of limitations, we have identified the presence of forgetting in the model's entity recognition at each step. Although the final model achieves state-of-the-art accuracy, addressing catastrophic forgetting remains a concern that needs to be addressed by the model.\n\n5.Continuous learning with large models: Our proposed method is an attempt and exploration of applying continual learning techniques to large language models. After conducting thorough experiments, we believe that our approach of using reinforcement learning strategies to adjust key parameters and improve the knowledge distillation process holds significant practical value. We have also conducted related explorations, and knowledge distillation can not only be applied to continual learning with large models but also contribute to model compression, further facilitating the deployment of large models. The previous challenge in knowledge distillation lies in the fixed distillation temperature and distillation loss weight, which lead to severe catastrophic forgetting. In contrast, our proposed method serves as a plug-and-play module that intuitively enhances the performance of the knowledge distillation process and stabilizes the model's memory capacity. We believe this is highly relevant in the era of large language models.\n\n6.text suggestions: We will revise the teaser texts accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written, and the formatting is quite pleasing.\n2. The practice of using Reinforcement Learning (RL) to select appropriate distillation temperature and loss weights for supporting continuous learning is quite innovative.\n3. The proposed method achieves the state-of-the-art CNER performance and is model-agnostic.", "reasons_to_reject": "1. The effectiveness of RL-based strategies is often not very stable. And the author did not submit code for review.\n2. The experimental results are not rigorous enough as there were no multiple runs for averaging, and no significance tests were conducted.\n3. The experimental results comparison in Table 1 is unfair. The ExtendNER [1] and L&R [2] papers use \"bert-based-cased\" as the encoder, while your SKD-NER paper uses \"bert-large-cased\" (line 442).\n4. The citation format of the paper is not standardized. The formal published version should be cited instead of the Arxiv version. For example, when citing BERT, please ensure to use the appropriate formal publication reference. The authors are requested to double-check all other citations for accuracy.\n5. The analysis of the limitations in the paper is too simplistic.\n\n\n[1] Monaikul, Natawut, et al. \"Continual learning for named entity recognition.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 15. 2021.\n\n[2] Xia, Yu, et al. \"Learn and review: Enhancing continual named entity recognition via reviewing synthetic samples.\" Findings of the Association for Computational Linguistics: ACL 2022. 2022.", "questions_to_authors": "The paper only discusses continuous learning for single NER tasks. However, in the era of large language models, a single model can handle multiple tasks with a significant number of parameters. Can the proposed method be applied to continuous learning scenarios with large models? Or does the author have any solutions for continuous learning with large models?", "paper_topic_and_main_contributions": "This paper introduces a novel Continual Named Entity Recognition (CNER) model and a knowledge distillation framework using reinforcement learning. It combines reinforcement learning strategies with traditional sequence labeling methods to tackle issues like catastrophic forgetting and label noise in continuous learning. Extensive experiments demonstrate that the approach achieves state-of-the-art performance in CNER and can be easily integrated as a plug-and-play module to improve the performance of other CNER models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Try to avoid situations where there is only one word on a single line, causing excessive white space and making it less aesthetically pleasing, as seen in line 28."}, "extra_text": {"Missing_References": "Ma, Ruotian, et al. \"Learning “O” Helps for Learning More: Handling the Unlabeled Entity Problem for Class-incremental NER.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023."}}]

Author rebuttal:

Meta-review: The authors consider using reinforcement learning to improve the continual learning for named entity recognition, by using RL to control parameter tuning in a knowledge distillation module for memory preservation, so that new entities can be learned while existing knowledge is also maintained. Results show that the method is effective, and the same method can also be used for other tasks. The content is nicely organized and nicely written.

It would be a nice addition if the authors can discuss further the motivation behind the model design, and give stronger evidence in the stability and significance of the results. The authors promised to release code for RL, which can be useful.
Decision: Accept-Main