SmartSpanNER: Making SpanNER Robust in Low Resource Scenarios
Authors: Min Zhang
Xiaosong Qiao
Yanqing Zhao
shimin tao
Hao Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: SpanNER
Named Entity Head
SmartSpanNER
Multi-task Learning
TL;DR: This paper proposes SmartSpanNER, a multi-task learning approach that introduces the Named Entity Head prediction task into SpanNER, to address the sensitivity of SpanNER to the amount of training data.

Abstract: Named Entity Recognition (NER) is one of the most fundamental tasks in natural language processing. Span-level prediction (SpanNER) is more naturally suitable for nested NER than sequence labeling (SeqLab). However, according to our experiments, the SpanNER method is more sensitive to the amount of training data, i.e., the F1 score of SpanNER drops much more than that of SeqLab when the amount of training data drops. In order to improve the robustness of SpanNER  in low resource scenarios, we propose a simple and effective method SmartSpanNER, which introduces a Named Entity Head (NEH) prediction task to SpanNER and performs multi-task learning together with the task of span classification. Experimental results demonstrate that the robustness of SpanNER could be greatly improved by SmartSpanNER in low resource scenarios constructed on the CoNLL03, Few-NERD, GENIA and ACE05 standard benchmark datasets.

SMARTSPANNER: Making SPANNER Robust in Low Resource Scenarios
Min Zhang, Xiaosong Qiao, Yanqing Zhao, Shimin Tao, Hao Yang
Huawei Translation Services Center, Beijing, China (zhangmin1 166, /iw.ii/ioioooooooooooooooooooooe zhaoyaqqingg taoshimin, yanghao3 yenugaa
Abstract
Reporter FUn
Carl Dinnon' of BBritiin'' 's' ITN' filed" this" report".
Named Entity Recognition (NER) of the most fundamental tasks in natural language pro- cessing. Span-level prediction (SPANNER) is more naturally suitable for nested NER than se- quence labeling (SEQLAB). However, accord- ing our experiments, the SPANNER method is more sensitive to the amount of training data, Le.. the F1 score of SPANNER drops much than that of SEQLAB when the amount of training data drops. In order to improve the robustness SPANNER in resource scenarios, propose simple and effective method SMARTSPANNEER which introduces a Named Entity Head (NEH') prediction task SPANNER and performs multi-task learn- ing together with the task of span classifica- tion. Experimental results demonstrate that the robustness of SPANNER could be greatly im- proved by SMARTSPANNER in low resource scenarios constructed the CoNLL03, FEW- NERD, GENIA and ACE05 standard bench- mark datasets.
rtn
6FF
Pb
Figure 1: A sentence from ACE05 with 5 nested NEs. The superscript of each word indicates its index in the sentence.
70
SpanNER SmartSpanNER
60
50
40
30
20
10
ACE:5
Figure Averaged F1 scores of SPANNER and SMARTSPANNER methods over 10 independent runs on the test datasets of CoNLL03, FEW-NERD, GENIA and ACE05, where the models trained 1,000 sen- tences randomly sampled from the training data with the maximum epoch set to 10.
1 Introduction
NER is fundamental information extraction task and plays an essential role in natural language pro- cessing applications such as information retrieval, question and answering, machine translation and knowledge graphs (Liu et al., 2022). The goal of NER is to extract named entities (NEs) into pre- defined categories, such as person (PER), location (LOC), organization (ORG) and geo-political en- tity (GPE). With the rapid evolution of neural ar- chitectures (Hochreiter and Schmidhuber, 1997;
al., 2020; Lin et al., 2020) to span prediction (Xu et al., 2017: Mengge et al., 2020; Tan al., 2020; Fu et al., 2021).
pointed out that nested enti- ties are very common in the field of NER (Finkel and Manning, 2009), i.e., a named entity can con- tain or embed other entities as illustrated in Fig. 1. Compared with SEQLAB, SPANNER has an obvi- ous advantage: All candidate entities be cas- ily found with different sub-sequences, which is straightforward for nested NER (Fu et al., 2021).
Kalchbrenner et al., 2014; Vaswani et al., 2017) and large pretrained models (Devlin al., 2019;
Brown et al., 2020; Lewis et al., 2020), recent years have seen the paradigm shift NER systems from sequence labeling (Chiu and Nichols, 2016; Luo
However, experiments reveal that the per- formance of SPANNER drops much more than that of SEQLAB when the amount of training data drops, which makes SPANNER hard low re- source scenarios. In order mitigate this problem, we propose a novel method SMARTSPANT
'NEH is the first word of named entity, for example, "Carl" is the NEH of "Carl Dinnon" in Fig. 1. If named entity has only one word, the NEH itself.

---

this paper. By introducing Named Entity Head (NEH) prediction task for each word in given sen- tences, we perform multi-task learning together with the task of span classification for NER. We conduct experiments on both flat and nested stan- dard benchmark datasets (CoNLL03, FEW-NERD, GENIA and ACEO5). Experimental results demon- strate that SMARTSPANNER could improve the ro- bustness in low resource scenarios significantly as shown in Fig Our contributions are summarized as follows:
Percentage
10%
15%
20%
50%
100%
sentences
1,498
2,248
2,997
7,493
14,987
SEQLAB
82.26
83,90
85.78
87.81
91.01
SPANNER
26.93
46.13
76.61
88.98
90.72
Table I: F1 scores of SEQLAB and SPANNER methods on the test dataset of CoNLLO3 with different percent- ages of training data and the maximum epoch 10.
nesses of SPANNER have been systematically in- vestigated by Fu et al. (2021), its performance in low resource scenarios is not discussed. To the best of our knowledge, we are the first to highlight that the performance of SPANNER drops much more than that of SEQLAB when the amount of training data drops, which poses a challenge to the robust- ness of SPANNER in low resource scenarios. In order to address this challenge, we propose novel method SMARTSPANNER.
To the best of our knowledge, we are the first to highlight the robustness problem of SPAN- NER in low resource scenarios.
To mitigate this problem, we develop simple and effective method, named SMARTSPAN- NER. By introducing the task of NEH pre- diction, SMARTSPANNER can achieve signifi- cant gains in low resource scenarios.
It should be noted that the low resource scenarios in this paper refer to those with at least 1,000 la- beled sentences, different from the settings for few- shot NER (Huang et al., 2021; Ding et al., 2021) and more common real-world applications.
We provide an in-depth analysis of the rea- sons for the strong robustness of the method SMARTSPANNER.
3 Methodology
2
Related Work
3.1 Problem Description
There has been a long history of research involv- ing NER (McCallum and Li, 2003). Traditional approaches are based on Hidden Markov Model (HMM; Zhou and Su, 2002) or Conditional Ran- dom Field (CRF; Lafferty et al., 2001). With the de- velopment of deep lcarning technology (Hinton and Salakhutdinov, 2006), SEQLAB methods such as LSTM-CRF (Huang et al., 2015) and BERT-LSTM- CRF (Devlin et al., 201 9) achieve very promising results in the field of NER. However, these methods cannot directly handle the nested structure because they can only assign one label to each token.
The commonly-used NER standard benchmark dataset CONL.O03 (English) (Tjong Kim Sang and De Meulder, 2003) is selected to show the different sensitivities of SEQLAB and SPANNER methods.? 10%, 20%, 50% and 100% of the training data are used to train the models respectively, and the Fl scores the test dataset are reported in Table 1. From Table 1, although the performances of SE- QLA and SPANNER on the entire training data are almost the same, the Fl score of SPANNER drops much more than that SEQLAB when the training data drops, i.c., the robustness of SPANNER in low resource scenarios needs to be greatly improved.
As it is pointed out that named entities are of- ten nested (Finkel and Manning, 2009), various approaches for nested NER have been proposed in recent years (Wang et al., 2022; Shibuya and Hovy, 2020). One of the most representative directions is opan-based methods that recognize nested en- tities by classifying sub-sequences of a sentence (Xu al., 2017; Mengge et al., 2020; Tan al., 2020; Fu et al., 2021; Zaratiana al., 2022; Weng and Zhang, 2023). SPANNER methods are nat- urally suitable for the nested structure because nested entities can be easily detected different sub-sequences. Although the strengths and weak-
3.2 NEH for SPANNER
Given a sentence S {wI..... w. with n words. and a span (f.j) denoting the sub-sequence in S which starts with w; and ends with Wy, wy is an NEH if there is an NE span (i, For example, the words "Reporter"", "Carl and "Britain in Fig. are NEHs, and the other words are not. From the above definition, we could conclude that w; is an NEH, which is a necessary but not suf-
Details of implementation are descrihed in Experiments.

---

Joint Inference
Only the start with the predicted NBNs used for cbssiiictiio
PER
ORG
GPE
Span Classification
NEH Prediction
Joint Training
Shared
Shared Encoder
Reporter
Dinnon
Britains
report
Reporter
Dinnon
Breahs
fied
report
(a) Namned Entlty Head Predication
(b) Span Classification
Figure 3: An overview of SMARtSPANNe method, which consists of (a) NEH prediction and (b) span classification. The two parts are jointly trained under the multi-task learning framework, and jointly determine the final results.
ficient condition for span (i,j) to be NE. That is say, if w; is not NEH, the span (f,j) must be NE. Therefore, if we introduce the NEH pre- diction task for SPANNER, the number of spans for semantic tag classification could be greatly reduced in training and inferring. For example, without the NEH prediction task, the number of spans in Fig. 1 (all sub-sequences for the sentence with 11 words) for SPANNER is 66. If the three NEHs in Fig. are correctly predicted, the number of spans (the sub-sequences starting with the words "Reporter"" "Carl-" and "Britain (r) could be reduced 28.
entity, and span classification aims classify spans corresponding semantic tags. The two parts are jointly trained under multi-task learning framework with shared encoder. The encoder is applied to a sentence to obtain contextual word rep- resentation, which shared downstream NEH prediction and span classification tasks. In this work, the NEH prediction task is treated as special type of span classification, where the span width is and the number of classes is 2. The span classifi- cation task is to aggregate the span information for multi-class classification. During inference, only the words predicted as NEHs are used generate candidate spans for span classification.
These are the basic ideas of our method SMANESPANNER. With the introduction of the NEH prediction task, the number of spans for se- mantic tag classification could be greatly reduced, the difficulty of semantic tag classification in SMARTSPANNER will be easier than in SPANNER. Meanwhile, the NEH prediction task is obviously easier than semantic tag classification (fewer cat- egories and more balanced positive and negative samples). Therefore, with the help of the NEH prediction task, SMARTSPANNER reduces the dif- ficulties of NER and could be more robust in low resource scenarios.
3.3.1 Encoder
Considering a sentence with n tokenized words [w:] we convert the words to their contextual embeddings with BERT (Devlin et al., 2019) en- coder. We generate the input sequence by concate- nating [CLS] token, [w and [SEP] token, and series of L stacked Transformer blocks (TBs) to project the input sequence of contex- tual vectors, i.e.,
3.3
SMARTSPANNER
ho: ha+1 TB; Mr.... M., [SEP]).
An overview of SMARTSPANNER method is shown Fig. 3, which consists of two parts: NEH prediction and span classification. NEH prediction aims to predict whether a word is the first word of
3.3.2 Span Classification
Considering a span (i,j), we use the boundary em- beddings and span length embedding to represent

---

CoNLLO3
FEW-NERD
GENIA
ACEO5
Train
Dev
Test
Train
Dev
Test
Train
Dev
Test
Train
Dev
Test
# sentences
14,987
3,466
3,684
131,767
18,824
37,648
15,022
1,669
1,855
7.285
968
1,058
il entities
23,499
5,942
5,648
339,617
48,633
96,676
47,006
4,461
5,596
24.700
3,218
3,029
# nested entities
3,222
328
448
2,797
352
339
avg length
13.59
14.82
12.60
24.48
24.59
24.45
26.49
25.67
27.05
18.82
18.77
16.93
Table Statistics of CoNLLO3, FEW-NERD, GENIA and ACE05 datasets, where the number of entity types 4, 66, and respectively.
the span (Fu al., 2021):
During inference, only the words predicted NEHs are used to generate candidate spans for span classification.
s(t,j) = h4; h3s 2j <nals
(2)
Therefore, in order to keep the training and infer- ring data distributions consistent, training the span classification model in SMARTSPANNEL only re- quires part of the spans that are all needed in SPANNER. A selection strategy is designed for each span (i.j) in training:
where 2j-i+1 is the span length embedding. which could be obtained by learnable look-up table.
Next, we feed the span representation into multi-layer perceptron (MLP) classifier, and ap- ply a softmax layer to obtain the probability P, to predict its semantic tag.
Pg softmax(MLP(s((.))))
1, if is an NEH
Select Select((i,)))
1, if rand sp
(8)
Finally, we minimize the cross-entropy loss func-
0, otherwise
tion:
where rand a randomly generated float number in [o, 1] and sp is a hyper-parameter of the selection threshold (0.05 used in this paper).
C. - Σ (y log (1=( (4)
An example for this selection strategy is pro- vided to compare the number of total training sam- ples in SPANNER and SMARTSPANNER: Given sentence having tokens and d named entities, we could get the number of training samples gen- erated by this sentence for SPANNER (Nsn) and SMART SPANNER (Nus)) respectively when the max span width is set to the value m:
where the number of semantic tags, and y denotes label indicating whether the span (1, is in tag t.
3.3.3 NEH Prediction
In this work, we treat NEH prediction as binary classification for a special span (i, (i.e.. only word Wi, 1 < i I n) in sentence S. So the NEH probability Pa is:
(m 1) m/2 N n d m (n d) sp
No
(9)
Ps softmax(MLP(s/) ()));
(5)
It should noted that the training samples of NEH prediction are included in N (the first term in the right). Supposing n = 100, = 5, m 10, sp = 0.05, we could have N... =
and the cross-entropy loss function is:
Cr = (1/ log Pa + (1 - log(1 - PA)), (6)
RJ 197. This means the training samples are greatly reduced in SMARTSPANNER and thus the training time is also greatly reduced. This is the reason why we name the method "SMART".
where yA denotes whether the word w; is NEH.
3.3.4 Joint Training and Inference
We We jointly minimize the following loss for training:
L - Ch + (1 W.C.,,
(7)
Experiments
4.1 Datasets
where CA and C. are the losses of the NEH predic- tion task and span classification task, and w is the pprammprr to balance the two tasks.
Four standard benchmark datasets CoNLL03 En- glish (Tjong Kim Sang and De Meulder, 2003),

---

CoNLLO3
FEW-NERD
GENIA
ACEOS
# Sents
Methods
P
R
F1
P
R
F1
P
R
F1
P
R
F1
SPANNER
45.33
10.92
17.28
0.00
0.00
0.00
42.73
17.28
23.06
66.58
12.46
20.44
1,000
SMARTSPANNER
77.85
67.75
72.45
49.74
12.19
19.57
41,36
65.46
50.66
51.03
60.60
55.38
SEOLAB
75.24
82.52
78.38
14.47
20.99
17.10
SPANNER
73.98
59.11
65.69
21.85
0.00
0.01
38.26
45.67
41.54
53.92
45.02
48.98
2,000
SPARTSPANNER
87.16
80.66
83.78
48.54
27.02
34.69
$4.50
77.35
63.94
62.64
76.37
68.82
SEQLAB
80.85
86.37
83.52
27.08
35.31
30.71
SPANNER
87.24
84,98
86.09
55.39
19.29
28.62
57,00
73.72
64,28
63,99
77.62
70.14
5,000
SMARTSPANNER
89.98
89.88
89.93
57.97
58.56
58.26
65.67
81.20
72.61
70.49
84.33
76.79
SEQLAB
85.07
89.20
87.08
44.64
53.11
48.50
SPANNER
90.15
91.29
90.72
67.62
70.99
69.26
70.26
81.94
75.65
73.11
85,83
78.96
ALL
SMARTSPANNER
91.25
91.56
91.40
67.45
70.23
68.81
71.61
82.22
76.54
77.51
87.05
82,00
SEQLAB
89.87
92.19
91.01
67.05
69.40
68.20
Table 3: Overall results of SPANNER, SMARTSPANNER and SEQLAB on CoNLLO3, FEW-NERD, GENIA and ACE05. The P. R and F1 are the mean values 10 independent runs. Best F1 scores are bold. "# Sents" stands for the number of sentences used for training.
FEW-NERD (SUP) (Ding et al., 2021), GENIA (Kim et al., 2003) and ACE0S English are selected for evaluation, where CoNLLO3 and FEW-NERD are for flat NER and the others are for nested NER. The statistics of these datasets are shown in Table 2. It should be pointed out that we follow Shibuya and Hovy (2020)'s preprocessing steps* to split GENTA and ACE05 into train, development, and test sets.
to 50, the max span width is set to 207. The MLP for span classification takes two layers, which is the same as the setting by Fu et al. (2021). Con- sidering that NEH classification is less challenging compared to span classification, we use single-layer MLP for NEH classification.
The training epoch number is set 10, and the batch size is set to 16. During training, the SPANNER, SMARTSPANNER and SEQLAB mod- els are optimized by AdamW (Loshchilov and Hut- ter, 2019) with the learning rate set to 0,00001 and a linear warmup scheduler. The values of the hlyer-parametted
4.2 Experiment Settings
experiments, we implement the SPANNER and SMARTSPANNER methods based on the source codes by Fu et al. (2021), where the pretrained model
for joint training and sp for span selection in SMARTSPANNER: and 0.05 respectively. All models are trained using single NVIDIA V100 GPU.
is used as the en- coder. And we implement BERT-CRF the SE- QLAB method for comparison.
set to 0.2
The training datasets for low resource scenarios are constructed by random. shuffle on the entire training sentences and extracting the first 1,000, 2,000 or 5,000 sentences, where the random seeds in 10 runs are set to 1, 2, 3, 4, 5, 6, 7, 8, and 42 respectively. In addition, the random seed for the results in Table 1 is set to 42, The development datasets of CoNLLO3, GENIA and ACE05 are all used in the constructed low resource scenarios. As the development data in FEW-NERD is very large (18,824 sentences), we choose the first 2,000 sen- tences as the development data when the training data contains 1,000, 2,000 or 5,000 sentences.
4.3 Main Results
The results of SPANNER, SMARTSPANNER and SEQLAB on the test datasets of CoNLLO3, FEW- NERD, GENIA and ACE05 are reported in Table 3, where the values of the precision (P), recall (R) and F1 score are included (SEQLAB only for flat NER datasets, not for nested).
From the results of SPANNER and SMARTSPAN-
NER on 1,000, 2,000, and 5,000 training sentences, is obvious that the F1 scores of SMARTSPAN- NER are greatly better that those of SPANNER on all the four datasets. Specially, when the train- ing data is 1,000 sentences, compared with SPAN- NER, SMARTSPANNER has the most obviously improvement in F1 scores (17.28% to 72.45% on CoNLLO3, 0.00% to 19.57% on FEW-NERD,
For the SPANNER and SMARTSPANNE ods, the embedding size of the span width is set
meth-
*catalog. og. 1dc upenn. edw/LLC200e http::::///gtbbb/////p/ppp cow. nested-ner-tacl2222
Pht t/githu. httpa/////gtuuubbc/mmmeeeee www..//w///./..eeeeeae...aaa SbAnaaNER
"This setting is to ensure the proportions of entities with a length exceeding 20 in the four datasets do not exceed 1%.
'https://hugginn co/bert-bal

---

Sents
Methods
CONTLA3
FEW-NERD
GENTA
ACEO5
F Sents
Methods
Tusks
A CAT
&PS
#NS
#PS/ANS
SN
6.9
10.0
10.5
8.0
SSN
NEN
3,072
15,553
175.1
1,000
4.9
6.4
7.1
5.7
1,000
SP
3,341
47,168
1/3.1
SSN
4 29%
36%
33%
29%
SN
SP
3,341
207.281
1/62.0
13.7
19.8
21.2
15.9
SSN
NEH
6,100
30,874
1/.1
2,000
9,9
12.8
14.2
11.3
2,000
6,688
87,177
1/3.1
SSN
28%
30%
33%
29%
SN
SP
6,668
412.529
161.9
SN
33.6
49.5
53.7
35.1
SSN
NRH
15,884
78.179
1/5.0
5,000
24.4
31.9
35.7
26.0
5,000
16,973
222.745
1/13.1
SSN
SN
SP
16,973
1,083,351
162.1
27%
35%
34%
26%
NRH
22,614
114,524
1/811
100.1
1311.3
162.4
58.3
SSN
24,614
323,921
1/132
ALL
SSN
72.6
844.8
108.0
41.2
SN
SP
24,614
1,545,866
1/62.8
1 27%
36%
33%
29%
The value 24,155 24,700 shown Table 2. This training. width
exceeding
experisteuts)
Table 4: Training time (in seconds) of SPANNER (SN) and S SMARTSPANNER (SNN) methods on the datasets of CoNLLO3, FEW-NERD, GENIA and ACEOS.
Table 6: Descriptions of ACE05's training data for the tasks in SMARTSPANNER (SSN) and SPANNER (SN) methods, where # CAT, # PS and # NS mean the number of classification categories, the number of positive samples and the number of negative samples
Sents
Methods
CoNTL03
FEW-NERD
GENTA
ACEOS
40.5
1029.2
37.6
12.7
1,000
SSN
14.7
301.7
12.2
5.4
respectively.
1. 166%
C 71%
68%
57%
SN
40.7
1026.7
37.9
12.7
2,000
14.0
280.3
11.8
SSN
66%
73%
.9%%
60%
Table 5 report the rumning time of the two meth- ods on the four datasets (CoNLLO33 FEW-NERD,
SN
41.0
1041.0
37.8
12.7
5,000
13.5
277.3
11.5
5.2
SSN
J 177%
73%
70%
59%
GENIA and ACE05) during training and inferring. For training, the running time is the average time to train one epoch. For inferring, the running time is the time to complete NER on the test datasets.
SN
41.1
1041.7
37.8
12.7
ALL
13.5
270.1
IL11
5.2
SSN
1 67%
74%
71%
59%
Inferring time (in seconds) of SPANNER (SN) and SMARTSPANNER (SNN) methods on the test datasets of CoNLL03, FEW-NERD, GENIA and ACE05.
Table 5:
From Table 4, it could be seen that SMARTSPAN- NER takes at least 26% less training time than SPANNER both all low resource settings and the entire data of all the four datasets. This because much fewer negative samples for span classifica- tion are used in the training of SMARTSPANNER. according to the selection strategy.
23.06% to 50.66% on GENIA, and 20.44% to 55.38% on ACE05). Therefore, SMART SPANNER is much more robust in low resource scenarios. From the comparison results of SEQLAB and SMARTSPANNER on the two flat NER datasets, can be seen that the F1 scores of SMARTSPANNER are better than those of SEQLAB on all low resource settings except on the 1,000 sentences of CONLLO3. It is worth mentioning that SMARTSPANNER is more effective than SEQLAB on all the low re- source settings of FEW-NERD- using such set- tings poses significant challenging due to the large number of entity types in FEW-NERD.
From Table 5, it could be seen that the inferring time of SMARTSPANNER is at least 57% shorter than that of SPANNER on the four test datasets of CoNLLO3, FEW-NERD, GENIA and ACE05. This is because only the spans that start with the predicted NEHS are used for span classification in SMARTSPANNER, which reduces the number of spans for inferring greatly.
From the comparison results in Table 4 and Ta- ble 5, SMARTSPANNER is much more efficient than SPANNER during training and inferring on all low resource settings and the entire data of all the four datasets.
Therefore, by introducing the NEH prediction task, SMARTSPANNER greatly improves the ro- bustness of SPANNER in low resource scenarios, even better than SEQLAB.
4.4 Efficiency
Analysis
According to our analysis. compared with SPAN- NER, SMARTSPANNER reduces the number of samples (spans) for training and inferring. To ver- ify this, we compare the efficiencies of SPANNER and SMARTSPANNER in this section. Table 4 and
We have shown the robustness of SMARTSPANNER in low resource scenarios. In this section, we aim to take a deeper look and understand what contributes to its final performance.

---

Sentences
Sents
Tasks
P
F1
Methods
Tasks
1,000
2,000
5,000
ALL
NEH
78.03
91.84
84,33
SSN
NEH
17,909
17,909
17,909
17,909
1,000
SP
39.33
62.12
48.
SP
40.743
38,409
37,874
37,553
NEH SP
51.03
60.60
55.38 (1 7.25)
SN
SP
192,302
192,302
192,302
192,302
NEH
84,44
94.86
89.34
2,000
SP
48.60
77.98
59.87
Table 7: Number of inferring samples for the tasks in SMARTSPANNER (SSN) and SPANNER (SN) methods the test dataset of ACE05 (1,000, 2,000, 5,000 and all sentences for training respectively).
NEH + SP
62.64
76.37
68.8 82 (1 8.95)
NEH
87.66
95.91
91.60
5,000
SP
60.68
85.92
71.12
NEH + SP
70.49
84,33
76.7 79 (t 5.67)
NEH
89.76
95.91
92.73
ALL
SP
65.04
88.51
73.71
5.1 Task Analysis
NEHL SP
77.51
87.05
82.00 (1 8.29)
SMARTSPANNER has two tasks, i.e., NEH predic- tion task and span classification (SP) task, while SPANNER has one task, i.e., SP task. We first pro- vide a detailed comparison of training data for these tasks on ACE05, which are shown in Table 6.
Table 8: The average values of precision (P), recall (R) and Fl scores of the two tasks (NEH and SP) SMARTSPANNER
the test dataset of ACE05 in 10 independent runs (1,000, 2,000, 5,000 and all sentences for training respectively).
From Table 6, it could be found that the task of NEH prediction is the easiest, because the num- ber of classification categories is the smallest and the balance of positive and negative samples is the best. Meanwhile, due to the more balanced positive and negative samples brought by the se- lection strategy, the task of span classification in SMARTSPANNER is much easier than that in SPAN- NER. Although deep learning methods can solve difficult problems, they require large amounts of data. Therefore, SMARTSPPNNTN is more effec- tive than SPANNER for NER in low-resource sce- narios. Furthermore, it could be seen that the total samples of less than those of SPANNER (about 70% reduction). This is why the training time of is much less than that of SPANNER.
the results of SMARTSPANNER (only the spans that start with the predicted NEHs are used for span classification).
From Table 8, it could be seen that the F1 scores of NEH prediction tasks are higher than those of span classification tasks due to the lower task dif- ficulty. When span classification in SMARTSPAN- NER is used for all possible spans, the precision suffers because of the inconsistent distributions of training and test data (worse than that of SPAN- NER in Table 3). When NEH prediction is used before span classification (i.e., only the spans that start with the predicted NEHs are used for span classification), the precision rates are greatly im- proved (more than 10%), the recall rates are slightly decreased (less than 2%), and the F1 scores are significantly improved (more than 5%). The reason for the drop of recall rates is that the recall rates of NEH are not 100%. Two cases from the test dataset of ACE05 are shown in A.1.
for training are much
Next, we compare the number of inferring sam- ples for the tasks in SMART and SPAN- NER on the test dataset of ACE05 when 1,000, 2,000, 5,000 and all sentences are used for training, as shown in Table It could be clearly seen that the total inferring samples in SMARTSPANNER are greatly less than those in SPANNER (more than 70% reduction). This is why the inferring time of SMARTSPANNER is much less.
The analyses on CONLLO3, FEW-NERD and GENIA are provided in A.2.
5.2 Hyper-Paramete Analysis
Finally, provide the results of the two tasks in SPARTSPANNER on the test dataset of ACE05 when 1,000, 2,000, 5,000 and all sentences are used for training data. In Table 8, the rows of "NEH" list the results of the NEH prediction task, the rows of "SP" list the results of the SP task on all the possible spans, and the rows of "NEH + SP list
There are two hyper-parameters (sample selection probability sp and joint training weight w) for train- ing SMARTSPANNER. In this section, we provide a detailed analysis for the values of these two hyper- parameters respectively.
5.2.1 Sample Selection Probability
We vary the sample selection probability parameter sp from 0 to I with step 0.05, and perform 10 independent runs of SMARTSPANNER for each
"The positive samples means the spans are NEHs in NEH prediction task or NEs in span classification task, otherwise they are negative samples.

---

1,000 Sentences
2,000 Sentences
1,000 Sentences
2,000 Sentences
Precison
75-
85
80-
Precison
63-
80
Recall
75
0.0
0.2
0.4
Sentences
0.6
0.8
0.0
0.2
0.4
Sentences
0.6
0.8
1.0
0.2
5,000 Sentences
0.4
ALL Sentences
0.4
0.6
100
100
07.5
85
75.0
0.0
0.2
0.6
0.8
1.0
0.0
1.0
Figure 4: The average values of Precision, Recall and Fl scores of NEH prediction on the test dataset of ACE05 with different sample selection probabilities sp (from 0 to 1 with step 0.05) in 10 independent runs when 1,000, 2,000, 5,000 and all sentences are used for training respectively.
Figure 6: The average values of Precision, Recall and F1 scores SMARTSPANNER on the test dataset ACE05 with different joint training weight (from 0.1 to 0.9 with step 0. 1) in 10 independent runs when 1,000, 2,000, 5,000 and all sentences are used for training respectively.
1,000 Sentences
2,000 Sentences
75-
in SMARTSPANNER during inference is stable.
50
70-
From Fig. 5, it could be seen that the F1 scores of NER SMART decreases with the increase of the parameter sp, especially when the training data is small (such as 1,000 sentences). This is due to the inconsistency between training data and inferring data for span classification in SMARTSPANNER (the larger the parameter sp, the higher the inconsistency). Since the robustness of deep learning methods will improve as the amount of data increases, SMARTSPANNER is less sen- sitive to the parameter sp when the training data increases (such as all sentences). From Fig. 5, it could be seen that 0.05 is a good choice for the parameter sp in low resource scenarios.
45
Rocall
80-
40
score
1.0
0.0
0.2
0/6
1.0
5,000 Sentences
85
86-
82
75
78
70
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 5: The average values of Precision, Recall and Fl scores of SMARTSPANNER on the test dataset of ACEOS with different sample selection probabilities sp (from 0 to 1 with step 0.05) in 10 independent runs when 1,000, 2,000, 5,000 and all sentences are used for training respectively.
5.2.2 Joint Training Weight
We vary the joint training weight w from 0.1 0.9 with step 0. and perform 10 independent runs of SMARTSPANNER for each value of w on the dataset ACE0S. Fig. shows the results (Precision, Recall and F1 scores) of NER when the parameter to varies (the sample selection probability parame- ter remains at 0.05).
value of sp on the dataset ACB05. Fig. 4 and Fig. show the results (Precision, Recall and F1 scores) of NEH prediction and NER in SMARTR respectively when the parameter sp varies (the joint training weight w remains at 0.2).
From Fig. 4, it could be seen that the recall rates of NEH prediction hardly change and the precision rates improve slightly with the increase of the pa- rameter sp. Therefore, the change of the parameter sp has little effect on the task of NEH prediction. That is to say, despite the parameter sp changes, the distribution of the data for span classification
From Fig. 6, could be found that the value of parameter between 0.2 and 0.4 is suitable for SMARTSPANNER according to the F1 scores. And the sensitivity of SMARTSPANNO to parameter w decreases with the increase of training data.

---

6 Conclusion
Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Nya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates. Inc.
In this paper, it is found that the SPANNER method is sensitive to the amount of training data, i.e., the performance of SPANNER is worse than SEQLAB in low resource scenarios. In order to alleviate this problem, SMARTSPANNER is proposed by intro- ducing the NEH prediction task into SPANNER in a multi-task learning manner. The comparison re- sults of experiments designed on the CoNLLO3, FEW-NERD, GENIA and ACE05 datasets demon- strate that SMARTSPANNER is much more robust in low resource scenarios than SPANNER, and greatly reduces the running time of training and inferring. In addition, the reasons for the strong robustness of SMARTSPANNER are analyzed in depth on the dataset ACE05.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM- CNNs. Trans- actions of the Association for Computational Linguis- tics, 4:357-370.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.
Limitations
The SMARTSPANNER method proposed in this paper is very effective in low resource scenar- ios. However, when the training data contains more than 10,000 sentences, compared with SPAN- NER, the advantages of SMARTSPANNER on the CoNLLO3 and GENIA datasets are not so obvious. Furthermore, when all the training data of the FEW- NERD dataset (more than 100,000 sentences) is used, the results of SMARISPANNER even drop a bit. Therefore, SMARTSPANNER is not strongly recommended in high resource scenarios.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD, A few-shot named entity recognition dataset. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3198-3213, Online, Association for Computational Linguistics.
Jenny Rose Finkel and Christopher D. Manning. 2009. Nested named entity recognition. In Proceedings of the 2009 Conference on Empirical Methods in Natu- ral Language Processing, pages 141-150, Singapore. Association for Computational Linguisties.
Ethics Statement
In this section, we discuss the ethical consideration of this work from the following two aspects. First, for SMARTSPANNER, the code, data and pretrained models adopted from previous works are granted for research-purpose purpose usage. Second, SMARTSPAN- NER improves the robustness of SPANNER in low resource scenarios by introducing the NEH predic- tion task. Hence we do not foresee any major risks or negative societal impact of our work. However, like any other ML models, the named entities recog- nized by our model may not always be completely accurate and hence should be used with caution for real-world applications.
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. SpanNER: Named entity re-frecognition span pre- diction. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference Natu- ral Language Processing (Volume I: Long Pupers). pages 7183-7195, Online. Association for Computa- tional Linguistics.
Geoffrey E. Hinton and Ruslan Salakhutdinov. 2006. Reducing the dimensionality of data with neural net- works. Science, 31 :535555)
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 98):17355- 1780.
Jiaxin Huang, Chunyuan Li, Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng. Jianfeng Gao, and Jiawei Han. 2021. Few- shot named entity recognition: An empirical baseline study. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing, pages 10408-10423, Online and Punta Cana,
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosse,

---

Dominican Republic, Association for Computational Linguistics.
Andrew McCallum and Wei Li. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 188- 191.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015, Bidirec- tional lstm-erf models for sequence tagging. Cite arxiv:150..0199) 019991
Nal Kalchbrenner, Edward Grefenstette, and Phil Blun- som. 2014. A convolutional neural network for mod- elling sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 655-665, Baltimore, Maryland. Association for Computational Linguisties.
Xue Mengge, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. 2020. Coarse-to-Fine Pre- training for Named Entity Recognition. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6345-6354, Online. Association for Computational Linguistics.
Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi, and Junichi Tsujäi 2003. Genia corpus- a semantically annotated corpus for bio-textmining. Bioinformatics, 19 Suppl 1:180-22
Takashi Shibuya and Eduard Hovy, 2020. Nested named entity recognition via second-bes sequence learning and decoding. Transactions of the Association for Computational Linguistics,
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling se- quence data. In Proceedings of the Eighteenth In- Machine Learning. ICML '01, page 282-289. San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Chuanqi Tan, Wei Qiu, Mosha Chen, Rui Wang, and Fei Huang. 2020. Boundary enhanced neural span classification for nested named entity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 34:9016- 9023.
ternational Conference
Erik F. Tjong Kim Sang and Fien De Mculder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871 1-710, Online. Association for Computa- tional Linguisties.
BART: Denoising sequence-to-sequee
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- lems, NIPS' 17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Bill Yuchen Lin, Dong-Ho Lee, Ming Shen, Ryan Moreno, Xiao Huang, Prashant Shiralkar, and Xi- ang Ren. 2020. TriggerNER: Learning with entity triggers explanations for named entity recogni- tion. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8503-8511 Online. Association for Computational Linguisties.
Yu Wang. Hanghang Tong, Ziye Zhu, and Yun Li. 2022. Nested named entity recognition: A survey. ACM Trans. Knowl. Discov. Data, 16(6).
Maobin Weng and Weiwen Zhang. 2023. Named entity recognition based on bert-bilstm-span in low resource scenarios. 2023 15th International Conference on Computer Reseurch and Development (ICCRD), pages 32-37.
Pan Liu, Yanming Guo, Fenglei Wang, and Guohui Li. 2022. Chinese named entity recognition: The state of the art. Neurocomput., 473(C):37-53.
llya Loshchilov and Frank Hutter. 2019, Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Mingbin Xu, Hui Jiang, and Sedtawut Watcharawit- tayakul. 2017. A local detection approach for named entity recognition and mention detection. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1237-1247, Vancouver, Canada. Association for Computational Linguistics.
Ying Luo, Fengshun Xiao, and Hai Zhao. 2020. Hi- erarchical contextualized representation for named entity recognition. In The Thirth-Fourth AAAI Con- ference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial In- telligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8441 141--44 18448. AAAI Press.
Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2022. Named entity recognition as structured span prediction. In Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-loS), pages 1-10, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

---

GuoDong Zhou and Jian Su. 2002. Named entity recog- nition using an HMMM-based chunk tagger. In Pro- ceedings of the 40th Annual Meeting of the Associa- tion for Computational L Linguistics, pages 473-480, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
which are shown in Table 11, Table 12 and Table 13, From Table 11, could be seen that SMARTSPAN- NER also obtains much more balanced positive and negative samples on the three datasets. As shown in Table 11 and Table 12, SMARTSPNNNER requires significantly fewer samples for training and inferring on the three datasets than SPANNER, enabling much faster training and inferring. Ta- ble 13 demonstrates the results of the two tasks in SMARTSPANNER on the three datasets, which are consistent with the results on ACE05.
A Appendix
1,000 Sentences
2,000 Sentences
90
85-
90
80
For reference, we report the results of NEH pre- diction in SMARTSPANNER on ACE05 with differ- ent joint training weight w in Fig. 7.
75
Recel
70-
Fi score
0.6
0.2
0.1
5,000
Sentences
According to the results (Zaratiana et al., 2022), setting larger epoch value will lead to better re- sults for SpanNER. Therefore, we just increase the training epoch number from 10 25 and conduct three independent experiments on the four datasets (using random seeds 1, 2 and 42 to obtain 1,000 training sentences). The comparison results are re- ported in Table 9. It can be seen that setting a larger epoch number only results in a sigmificant improve- ment of F1 score for SPANNER on CoNLLO3 (with minor improvements on GENIA and ACE05, and change on FEW-NERD). Moreover, the superiority of SMARTSPANNER over SPANNER remains evident across all four datasets, highlight- ing the robustness of SMARTSPANNER in low- resource scenarios.
96
0.2 0.4
0.6
0.8
0.2
0.4
0.6
0.8
Figure 7: The average values of Precision, Recall and Fl scores of NEH prediction in SMARTSPANNER on the test dataset of ACE05 with different joint training weight w (from 0.1 0.9 with step 0.1) in 10 indepen- dent runs when 1,000, 2,000, 5,000 and all sentences are used for training respectively.
SPANNER
SMARTSPANNER
Dataset
P
R
F1
P
R
F1
CONLLO3
74.25
56.11
63.67
88.98
88.87
88.92
FEW-NERD
0.00
0.00
0.00
47.81
29.90
36.67
GENIA
50.09
30.64
37.81
61.72
71.61
66.28
ACHO5
12.05
19.60
76.44
73.23
Table 9: Averaged P. R and F1 values of SPANNER and SMARTSPANNER methods over independent runs, where the models are trained on 1,000 sentences ran- domly sampled from the training data with the maxi- mum epoch set 25.
A.1
Case Study
Two cases from ACEOS are shown in Table 10, where the results of SPANNER and SMARTSPAN- NER are provided. From Table 10, it could be clearly seen that SMARTSPANNER performs better for NER than SPANNER, and the predicted NEHs are beneficial to improve the precision of NER in SMARISPANNER..
A.2 Supplement Experimental Results We provide the results of task analysis on the datasets CoNLLO3, NEWD and GENIA,

---

Sentence
He was' the governor" of my² state of Texas
where 10 there!! are a whole lor of doctors 18
Ground Truth NEs
(0, O, PER), (2, 17, PER), (5, 5, PER), (5, 17, GPE),
Case
(Total Ground Truth NEHs
(8, GPE), (10, 10, GPE), (13, 17, PER)
(0, 0), (2, 2), (), (8, 8), (10, 10), (13, 13)
(Total 6)
NEs by SPANNER
(0, 0. PER, 0.8446), (5, 8, GPE, 0.4203),
(2 correct, 2 incorrect)
(5, 17, PER, 0.4736), (8, 8, GPE, 0.6953)
NEHs by SMARTSPANNER
(0, 0, 0.9961), (2, 2,, (5, 5, 0.99955,, (8, 8, 0.9005),
(6 correct, I incorrect)
(10, 10, 0.9729), (11, 11, 0.7172), (13, 13, 0.9433)
(0, O, PER, 0.9465), 5,, PER, 0.8316),
NEs by SMARTSPANNER (5 correct, 1 incorrect)
(8, GPE, 0.9542), (10, 10, GPE, 0.7904),
(11, 17, PER, 0.6768), (13, 17, PER, 0.8246)
(0, O, PER, 0.9465), 5, 5, PER, 0.8316),
NEs by SMARTSPANNER
(6, 8, GPE, 0.6781), (8, 8, GPE, 0,9542),
without NEHs
(10, 10, GPE, 0.7904), (11, 17, PER, 0.6768),
(5 correct, 3 incorrect)
(13, 17, PER, 0.8246), (17, 17, PER, 0.7493)
Sentence
A traveler' was driving" through' Arkansas" when' he" lost his way and got off the main highway 17
Case
Ground Truth NEs
(0, 1, PER), (5, GPE), (7, 7, PER), (9, 9, PER).
(Total 5) Ground Truth NEHs
(14, 16, FAC)
(0, 0), (5, 5), (7, 7), (9, 9), (14, 14)
(Total 5)
(0, 1, PER, 0.8230), (0. PER, 0.7019),
NES by SPANNER
(5, 5, GPE, 0.7611), (7, PER, 0.8407),
(4 correct, 1 incorrect)
(9, 9, PER, 0.5256)
NEHS by SMARTSPANNER
(0, 0.9886), (5, 0.9680), (7, 7, 0.9959), (9, 9, 0.9899),
(5 correct, 0 incorrect)
(14, 14, 0.9180)
(0, 1, PER, 0.9403), 5, PER, 0.5006),
NEs by SMARTSPANNER (5 correct, incorrect)
(5, 5, GPE, 0.9541), (7, 7, PER, 0.8839),
(9, 9, PER, 0.7767), (14, 16, FAC. 0.4102)
(0, 1, PER, 0.9403), (0. 5, PER, 0.5006),
NES by SMARTSPANNER
(1, 1, PER, 0.6204), (5. 5, GPE, 0.9541),
without NEHs
(7, 7, PER, 0.8839), (9, 9, PER, 0.7767),
(5 correct, 2 incorrect)
(14, 16, FAC, 0.4102)
Two cases from ACE05. The superscript of each word indicates its index in the sentence, the ground truth NEs are represented by triples (NE start word index, NE end word index, NE type), and the ground truth NEHs are represented by special spans (same start and end indexes). The NEs predicted by SPANNER and SPANTSP represented by four-tuples (predicted NE start word index, predicted NE end word index, predicted NE type, predicted probability), and the NEHs predicted by SMARTSPANNES are represented by triples (predicted start index, predicted end index, predicted probability). The triples or four-tuples in red indicate incorrect predictions. The rows of "NEs by S SMARTSPP SBARTSPANNER without NEHs" show the predicted NEs by SMARTSPANN when all possible spans are used for NE prediction (i.e., the predicted NEHs are not used for span filtering).
Table 10:

---

FEW-NERD
Methods
Tasks
# CAT
PS
#SS/#NS CAT
APS
#NS
PS//NSS
CAT
P8
#N8
1,620
12,446
2
2,538
22,90
18.5
2,899
25,223
18.1
SSN
1,000
5
1,630
22,636
1/4.0
67
2,598
46.351
VI7A
6
3018
49,478
1/16.4
3N
1,620
150,140
19277
67
2,598
311.226
1119
3018
333,425
DITT0
NEH
2
3.244
24,794
177.6
2
5,214
44,146
18.5
5,705
46,968
1882
88N
2,000
3.244
44,843
67
5,214
92,903
6054
1/66.5
SN
5
3,241
295,933
1/9222
67
5,214
619,037
17118.7
6
6051
677,073
DILL8
NEH
7,862
2
12,942
109,808
178.5
14,133
117,807
1
5,000
5
7,862
107,335
1/13.7
67
12,942
229,465
1/17.7
6
15,423
254,543
1/16.5
SN
7,862
727,192
1/92.5
67
12,942
1.538.134
1/18.8
15.423
1,705,363
1/110.6
SSN
NEH
23.499
181,068
1/7.7
2
339,00
2,580,060
188.5
2
43.536
354,703
188.1
ALL
33,499
319,074
1/3.6
67
339,602
6,013,526
1/1777
6
46.984
766,069
1/16.3
SN
23,499
2,146.052
1/913
67
339,602
10,291,745
1/118.6
6
46,984
5,133,236
1/109.3
Table 1I: Descriptions of the training data of CoNLLO3, FEW-NERD and GENIA for the tasks in SMARTSPANNER (SSN) and SPANNER (SN) methods, where CAT means the number of classification categories, # PS means the number of positive samples, and # NS means the number of negative samples.
CONLLA3 (# Sentences)
FEW-NERD (P Sentenees)
GENIA Sentenees)
Methods
1,000
2,000
5,000
A13.
1,000
2,000
5,000
ALI
1,000
2,000
5,000
ALL
NGI
919,162
919,162
919,162
46,578
46,878
46,578
SSN
SP
69,758
62,204
60,424
60,323
1,097,569
1,581,779
1,519,050
1,452,749
93,219
85,390
80,286
78,402
SN
SP
477,887
477,887
487
477,887
11,595,325
11,595,325
11,595.328
11,595,325
616,491
616.491
616,491
616,491
Table 12: Number of inferring samples for the tasks in SMARTSPANNER (SSN) and SPANNER (SN) methods
on the test datasets of CoNLLO3, FEW-NERD and GENIA (1,000, 2,000, 5,000 and all sentences for training respectively).
CoNLL03
FEW-NERD
GENIA
# Sents
Tasks
F1
P
R
F1
P
R
F1
NEH
82.87
95.13
88.52
72.45
91,88
81.01
66.58
88.62
76.02
1,000
SP
70.65
69.25
69.94
48.45
12.18
19.46
28.52
69.57
40.45
NEH SP
77.85
67.75
72.45 (1 2.51)
49.74
12.18
19.57 (1 0.11)
41.36
65.46
50.66 (? 6.21)
NEH
91.81
97.01
94.34
78.94
93.35
85.54
72.20
90.39
80.27
2,000
SP
74.00
81.40
77.52
45.81
27.16
34.07
37.28
82.04
$1.26
NEH + SP
87.16
80.66
83.78 (f 6.26)
48.54
27.02
34.69 (t 0. 62)
$4.50
77.35
63.94 (? 12.68)
NEH
95.56
97.66
96.60
82.89
93.96
88.08
78.56
90.20
83.98
5,000
SP
82.23
90.46
86.15
52.50
59.40
55.74
49.14
86.72
62.74
NEH + SP
89.98
89.88
89.93 (1 3.78)
57.97
58.56
58.26 (t 2.52)
65.67
81.20
72.61 (1 9.87)
NEH
96.42
97.95
97.18
87.31
94,56
90.79
81.71
89.64
85.49
ALL
SP
89.44
91.94
90.68
61.74
71.55
66.28
55.92
88.36
68.47
NEH + SP
91.25
91.56
91.40 (1 0.72)
67.45
70.23
68.81 (1 2.53)
71.61
82.22
76.54 (1 8.07)
Table 13: The average values of precision (P), recall (R) and F1 scores of the tasks (NEH and SP) in on the test datasets of CONLLO3, FEW-NERD and GENIA in 10 independent runs (1,000, 2,000, 5,000 and all sentences for training respectively).


Reviews:
[{"review_id": "8W2Mjn7Uhj", "reviewer": "Reviewer_Y6rd", "summary": "", "questions": "1. Shibuya and Hovy 2020 present better results than the ones found in this paper. 84.3 vs 82.0 on ACE05 and 77.4 vs 76.5 on GENIA. I couldn’t find any comparison in the current draft. Any reason why this comparison was not made in the current paper?\n2. I see two sets of negative samples,  hard negatives that share the start token with an entity and soft negatives that are randomly sampled using `sp`. It would be useful to perform the following ablation. This will tease apart the effectiveness of each component.\n        - SpanNER w/ original negative samples\n        - SpanNER w/ hard negatives\n        - SpanNER w/ hard + soft negatives\n        - SpanNER w/ hard + soft negatives, joint learning (aka SmartSpanNER)\n3. Additional empirical analysis that reports differences between nested and flat entities in these benchmarks, maybe at varying depths (1st, 2nd, 3rd, and 4th levels; see Table 1 in Shibuya and Hovy, 2020).\n4. In Table 4, is the training time in seconds?\n5. Why not just use h_i instead of s(i,i) in NEH (equation 5)?\n6. I appreciate average scores over multiple runs, but it would be great if you can include the standard deviations in Table 3.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n\n*Q1. Shibuya and Hovy 2020 present better results than the ones found in this paper. 84.3 vs 82.0 on ACE05 and 77.4 vs 76.5 on GENIA. I couldn’t find any comparison in the current draft. Any reason why this comparison was not made in the current paper?*\n\nA1: The aim of this paper is to improve the performance of SpanNER in low-resource scenarios by introducing the auxiliary task (NEH prediction). Therefore, we focused on comparing it with the SpanNER method. We will add comparisons with Shibuya and Hovy 2020 in the revision.\n\n*Q2. I see two sets of negative samples, hard negatives that share the start token with an entity and soft negatives that are randomly sampled using sp. It would be useful to perform the following ablation. This will tease apart the effectiveness of each component. - SpanNER w/ original negative samples - SpanNER w/ hard negatives - SpanNER w/ hard + soft negatives - SpanNER w/ hard + soft negatives, joint learning (aka SmartSpanNER)*\n\nA2: Good idea!  We will add this ablation experiment in the revision.\n\n*Q3. Additional empirical analysis that reports differences between nested and flat entities in these benchmarks, maybe at varying depths (1st, 2nd, 3rd, and 4th levels; see Table 1 in Shibuya and Hovy, 2020).*\n\nA3: Thank you for pointing out this! We will report the analysis in the revision.\n\n*Q4. In Table 4, is the training time in seconds?*\n\nA4: Yes, we have indicated it in the description of Table 4.\n\n*Q5. Why not just use h_i instead of s(i,i) in NEH (equation 5)?*\n\nA5: Because we treat NEH as a special span, we represent it this way. In fact, we can use h_i to represent s(i,i) in equation 5.\n\n*Q6. I appreciate average scores over multiple runs, but it would be great if you can include the standard deviations in Table 3.*\n\nA6: Thank you very much for your appreciation! We will include the standard deviations in the revision.\n\n*Q7:  It would be interesting to see the effectiveness of the proposed method in true low-resource settings. This could include NER on specialized domains, maybe? (see Zhang et al., 2021: https://academic.oup.com/jamia/article/28/9/1892/6307885)).*\n\nA7: Your suggestion is quite valuable, and we will consider validating our method in the true low-resource settings.\n\nThanks again for your insightful comments, which are very instrumental in improving the quality of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The choice of auxiliary task is well-motivated and shows strong results across all datasets. As highlighted by the results, the auxiliary task is simpler than the NER task itself and doesn't adversely affect the core NER task. It also provides signficant gains at inference time.\n2. The negative sampling strategy includes hard and soft negatives. Hard negatives constitute non-entity spans that share the head, and soft negatives are randomly sampled. This strategy provides clear gains in training time.\n3. The analysis section is quite thorough. Paper reports average results across 10 independent runs across all experiments, studies the impact across varying training sizes, and sampling probabilities.", "reasons_to_reject": "Some weaknesses include,\n\n1. The paper motivates span-based methods on their ability to deal with nested entities. However, it doesn't analyze the effectiveness of the proposed method on nested entities from GENIA and ACE05 datasets. It doesn't compare against the state-of-the-art work (Shibuya and Hovy, 2020) on these two datasets. Interestingly, this paper underperforms the results from Shibuya and Hovy, 2020.\n2. The two core contributions of this paper, the auxiliary task, and the sampling strategy seem quite effective. But it is hard to tease apart the effect of each. Some additional ablations that compare the effectiveness of hard, soft negatives, and joint learning can improve the claims of this paper.\n3. A minor point about the choice of datasets in this paper. All the datasets in the paper contain large amounts of training data. It would be interesting to see the effectiveness of the proposed method in true low-resource settings. This could include NER on specialized domains, maybe? (see Zhang et al., 2021: https://academic.oup.com/jamia/article/28/9/1892/6307885)).", "questions_to_authors": "1. Shibuya and Hovy 2020 present better results than the ones found in this paper. 84.3 vs 82.0 on ACE05 and 77.4 vs 76.5 on GENIA. I couldn’t find any comparison in the current draft. Any reason why this comparison was not made in the current paper?\n2. I see two sets of negative samples,  hard negatives that share the start token with an entity and soft negatives that are randomly sampled using `sp`. It would be useful to perform the following ablation. This will tease apart the effectiveness of each component.\n        - SpanNER w/ original negative samples\n        - SpanNER w/ hard negatives\n        - SpanNER w/ hard + soft negatives\n        - SpanNER w/ hard + soft negatives, joint learning (aka SmartSpanNER)\n3. Additional empirical analysis that reports differences between nested and flat entities in these benchmarks, maybe at varying depths (1st, 2nd, 3rd, and 4th levels; see Table 1 in Shibuya and Hovy, 2020).\n4. In Table 4, is the training time in seconds?\n5. Why not just use h_i instead of s(i,i) in NEH (equation 5)?\n6. I appreciate average scores over multiple runs, but it would be great if you can include the standard deviations in Table 3.", "paper_topic_and_main_contributions": "This paper studies the named entity recognition task of both flat and nested entities. The paper contrasts the two formulations, standard sequence labeling, and the span-level prediction. They highlight the challenges of span-level prediction in low-resource settings and present a joint learning framework that significantly improves the system's performance.\n\nThey introduce an auxiliary task of named entity head prediction that identifies words that constitute the start of any NE span. They jointly train this with the standard NE span prediction and report results on four datasets, CoNLL03 and Few-NERD (flat), GENIA, and ACE05 (flat+nested). They also propose a negative sampling strategy that mines hard negatives based on the entity head.\n\nThey present strong results on all four datasets, especially in low-resource scenarios.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UIbsrMhuDI", "reviewer": "Reviewer_f3si", "summary": "", "questions": "* The max span width K is set to 20 in the paper which is arbitrary. I think one should check for the max entity width in the training set to decide this hyperparameter. A lower size of K should also increase the speed of the baseline Spanner. An analysis of the effect of K on speed and performance can quantify this issue.\n\n* Why is NEH and Span classification have 1 and 2 layers respectively? What is the rationale for this decision?\n\n* Suggestion for improving the baseline: training the model for more steps; tune the learning rate and the weight decay; etc.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n\n*Q1: The max span width K is set to 20 in the paper which is arbitrary. I think one should check for the max entity width in the training set to decide this hyperparameter. A lower size of K should also increase the speed of the baseline Spanner. An analysis of the effect of K on speed and performance can quantify this issue.*\n\nA1: Thumbs up for your attention to experimental details!  The reason for setting this parameter to 20 is to ensure that the proportions of spans with a length exceeding 20 in the four datasets (train, dev and test) do not exceed 1%, which are shown as follows. \n\n|Dataset | Train | Dev | Test|\n|  ----  | ----  |----  |----  |\n|CoNLL03| 0.000% | 0.000% | 0.000% |\n|FEW-NERD| 0.004% | 0.002% | 0.004%|\n|GENIA| 0.000% | 0.000% | 0.000% |\n|ACE05| 0.312% | 0.124%| 0.330%|\n\nWe apologize for not clarifying this in the paper and will provide an explanation in the revision.\n\nWe quite agree that a lower size of K could increase the speed of the baseline Spanner. Additionally, a smaller K size will also enhance the speed of SmartSpanNER. We will analyze the effect of K on speed and performance in the revision.\n\n*Q2: Why is NEH and Span classification have 1 and 2 layers respectively? What is the rationale for this decision?*\n\nA2: We greatly appreciate your attention to the details of the model. As mentioned in the paper, our implementation is based on the code (https://github.com/neulab/SpanNER) provided by Fu et al. (2021), where a two-layer network is used for Span classification. Therefore, we have also adopted this setting. Considering that NEH classification is less challenging compared to Span classification (which is also validated in our experiments), we have used a single-layer network for NEH classification. Additionally, we adopted the greedy decoding method provided by Fu et al. (2021) for the flat NER task. We apologize for not explaining this in the paper.\n\n*Q3: Suggestion for improving the baseline: training the model for more steps; tune the learning rate and the weight decay; etc.*\n\nA3: Great suggestion! We will definitely follow it to improve the baseline performance.\n\n*Q4: Missing References: Named Entity Recognition as Structured Span Prediction(https://aclanthology.org/2022.umios-1.1) (Zaratiana et al., UM-IoS 2022)*\n\nA4: Very practical paper, we will cite it in the revision.\n\nThanks again for your insightful comments, which are very instrumental in improving the quality of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The idea is interesting as it prevent from enumerating from large number of candidates in span-based NER.\n* The model is apparently more data-efficient than the standard span-based NER.\n* The paper contains detailed analysis..", "reasons_to_reject": "* While the authors claimed that the standard Spanner is sensitive to training data size, it seems like the model is rather poorly tuned. For instance, Zaratiana et al., (2022) use the standard Spanner on CoNLL-2003 with 1000 data obtained around 77 F1 (Table 6), while this paper reports an F1 score of 17.28. I suggest the authors review their baselines. Consequently, the authors claim that Sequence Labeling models are more data-efficient than Spanner, which is not supported by evidence. \n\n* The authors omit crucial details about decoding: as the Spanner uses a sort of local span classification, a structured decoding should be applied to ensure well-formedness of the output. For instance, for flat NER, there should be no span overlapping; and for Nested NER, completely nested spans should be allowed but no clashing. \n\n* I understand that comparison to SOTA is not required, but the results on Nested NER are very low compared to existing models, demonstrating again that the models are not well-tuned.", "questions_to_authors": "* The max span width K is set to 20 in the paper which is arbitrary. I think one should check for the max entity width in the training set to decide this hyperparameter. A lower size of K should also increase the speed of the baseline Spanner. An analysis of the effect of K on speed and performance can quantify this issue.\n\n* Why is NEH and Span classification have 1 and 2 layers respectively? What is the rationale for this decision?\n\n* Suggestion for improving the baseline: training the model for more steps; tune the learning rate and the weight decay; etc.", "paper_topic_and_main_contributions": "This paper introduces SmartSpanner, a novel approach to enhance the robustness of Spanner i.e span-level prediction method for Named Entity Recognition. SmartSpanner incorporates a Named Entity Head (NEH) prediction task alongside span classification to address standard Spanner shortcomings: reduce sensitivity to training data size and to reduce the number of candidate span both during training and inference. The author conducted experiment on diverse range of datasets of flat and nester NER to validate their hypothesis.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Named Entity Recognition as Structured Span Prediction(https://aclanthology.org/2022.umios-1.1) (Zaratiana et al., UM-IoS 2022)"}, "extra_text": {}}, {"review_id": "nkqyfswX6D", "reviewer": "Reviewer_o3MJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n\n*Q1: There does not seem to be a major problem, but a minor problem is that the development set is not used to determine the two hyperparameters, w and sp, and they seem to be determined based on an analysis using a test set.*\n\nA1: We fully agree with this point and will consider making improvements to the experiment.\n\n*Q2: Missing References: The following paper may address a similar setting: Weng and Zhang, Named Entity Recognition Based on BERT-BiLSTM-SPAN in Low Resource Scenarios*\n\nA2: Thank you for providing the paper. We will cite it in the revision.\n\n*Q3: Typos Grammar Style And Presentation Improvements: Line 024: information extract task -> information extraction task; The font in Tables 4, 5, 10, and 11 is too small.*\n\nA3: Thank you for pointing out these issues. We will fix them in the revision.\n\nThanks again for your insightful comments, which are very instrumental in improving the quality of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The problem that motivates the study and the model for solving the problem are explained clearly.\n- This paper shows experimentally that span-based NER models do not perform robustly when the training data size is small.\n- This paper proposes SmartSpanNER, which models NER as a multitask learning problem with two tasks: named entity head (NEH) prediction and span classification and shows through experiments that it performs relatively robustly even with small training data.\n- After discussing the efficiency of learning and inference between SpanNER and SmartSpanNER, the paper shows through experiments that the latter is capable of learning and inference in a shorter period of time.", "reasons_to_reject": "- There does not seem to be a major problem, but a minor problem is that the development set is not used to determine the two hyperparameters, w and sp, and they seem to be determined based on an analysis using a test set.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper points out that span-based named entity recognition (SpanNER) models do not perform well in low-resource settings with about 1000 sentences of training data, and proposes SmartSpanNER, which is based on multi-task learning with two tasks: one to predict the named entity head (NEH) and the other to classify spans, to address this problem. Experiments using four representative datasets, CoNLL03 and Few-NERD as flat NER data, and GENIA and ACE0 as nested NER data, show that SpanNER's performance degrades significantly when training data is small, and SmartSpanNER performs relatively robustly even when training data is small. In addition, a comparison of training and inference times between SpanNER and SmartSpanNER was also conducted, showing that SmartSpanNER can be trained and inferred in a relatively short time because the introduction of NEH narrows down the number of candidates.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- The following paper may address a similar setting: Weng and Zhang, Named Entity Recognition Based on BERT-BiLSTM-SPAN in Low Resource Scenarios", "Typos_Grammar_Style_and_Presentation_Improvements": "- Line 024: information extract task -> information extraction task\n- The font in Tables 4, 5, 10, and 11 is too small."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an auxiliary task for span-level prediction for NER to improve efficiency/robustness. They provide a thorough evaluation on multiple benchmarks, showing the effectiveness of their approach. Although in their original submission, their baselines were extremely low (17 vs 77SOTA); this has been resolved in the rebuttal. One of the reviewers still worries about the authors using more hyperparameter tuning for the proposed method compared to the baseline (private discussion); but from section 4.2 it seems like they were tuned the same way (and the rebuttal only adds the number of epochs). Therefore, the results seems sound (with a much better baseline, trends remain). In conclusion: a simple and motivated approach is introduces, which improves performance/robustness, this seems a valuable contribution.

Remaining weaknesses as reported by the reviewers are: no real low-resource setting is tested, no separate evaluation on nested entities, no ablation of the sampling strategy that was also introduced (besides the auxiliary task), tuning of 2 hyperparameters on test data. Furthermore, I think a (qualitative) analysis of which cases have improved could be expected from a long paper.
Decision: Accept-Findings