Type-Aware Decomposed Framework for Few-Shot Named Entity Recognition
Authors: Yongqi Li
Yu Yu
Tieyun Qian
Venue: EMNLP 2023 Findings (2023)
Keywords: Named Entity Recognition
Few-Shot Learning
TL;DR: A decomposed framework for solving few-shot named entity recognition.

Abstract: Despite the recent success achieved by several two-stage prototypical networks in few-shot named entity recognition (NER) task, the over-detected false spans at span detection stage and the inaccurate and unstable prototypes at type classification stage remain to be challenging problems.
In this paper, we propose a novel Type-Aware Decomposed framework, namely TadNER, to solve these problems.
We first present a type-aware span filtering strategy to filter out false spans by removing those semantically far away from type names. We then present a type-aware contrastive learning strategy to construct more accurate and stable prototypes by jointly exploiting support samples and type names as references.
Extensive experiments on various benchmarks prove that our proposed TadNER framework yields a new state-of-the-art performance.

Type-Aware Decomposed Framework for Few-Shot Named Entity Recognition
Yongqi Lit, Yu Yu', Tieyun Qian 1.2,*
School of Computer Science, Wuhan University, China Intellectual Computing Laboratory for Cultural Heritage, Wuhan University, China [liyongqi, Yu. Yul Yu1024, qty (ww). edu.cn
Abstract
Applesos Foundid Culforiiiin
Apple fmandod
Spin
Apple
Califernta
Despite the recent success achieved by several two-stage prototypical networks in few-shot named entity recognition (NER) task, the over- detected false false spans at the span detection stage and the inaccurate and unstable prototypes the type classification stage remain to be chal- lenging problems.
Caliionia
Apple
Type
Eata Dop
dever F'llertng
Calffrraa
B96
Japle
(e) Over-detected falss spans
(b) fter type-aware flltering
Typrance
this paper, we propose a novel Type-A Awar ware Decomposed framework, namely TadNER, to solve these problems. We first present rype-aware span filtering strar- egy to filter out false spans by removing those semantically far away from type names. We then present type-aware contrastive learning strategy to construct more accurate and stable prototypes by jointly exploiting support sam- ples and type names references. Extensive experiments on various henchmarks prove that our proposed TadNER framework yields a new state-of-the-art performance.
pracepe
Inaccurate and unstable prototypes Aferr typ--aware contrastiee learning
Figure 1: (a) shows over-detected false spans, (b) shows spans got by adopting our type-aware span filtering strategy. (c) shows inaccurate and unstable prototypes. (d) shows prototypes got by adopting our type-aware contrastive learning strategy.
stream in few-shot NER. These methods need to simultaneously learn the complex structure consist- ing of entity boundary and entity type. When the domain gap is large, their performance will drop dramatically because it is extremely hard cap- ture such complicated structure information with only a few support examples for domain adaptation. This leads to the insufficient learning of boundary information, resulting that these methods often mis- classify entity boundaries and cannot obtain very satisfying performance.
1 Introduction
Named entity recognition (NER) aims to detect entity spans and classify them into pre-defined cat- egories (entity types). When there are sufficient labeled data, deep learning-based methods (Huang et al., 2015; Ma and Hovy, 2016; Lample et al., 2016; Chiu and Nichols, 2016) can get impressive performance. In real applications, it is desirable to recognize new categories which are unseen in training/source domain. However, collecting extra labeled data for these new types will be surely time- consuming and labour-expensive. Consequently, few-shot NER (Fritzler et al., 2019; Yang and Kati- yar, 2020), which involves identifying unseen en- tity types based on few labeled samples for each class (i.e., support samples) in test domain, has attracted great research interests in recent years. End-to-end metric learning based methods (Yang and Katiyar, 2020; Das et al., 2022) are the main-
Recently, there an emerging trend in adopting two-stage prototypical networks (Wang et al., 2022; Ma al., 2022c) for few-shot NER, which decompose NER into two separate span extraction and type classification tasks and perform one task at each stage. Since decomposed methods only need to handle one single boundary detection task at the first stage, they can find more accurate boundaries and obtain better performance than end-to-end approaches.
While making good progress, these two-stage prototypical networks still face two challenging problems, i.e., the over-detected false spans and the
*Corresponding author. 'Our code and data will he available at https://gitthll com/NL /NN PWM- WHU/TadNER.

---

inaccurate and unstable prototypes in correspond- ing stages. (1) At the span extraction stage test phase, the decomposed approaches usually recall many over-detected false spans whose types only exist in the source domain. For example, "1976" in Figure (a) belongs to a DATE type in the source domain since there are many samples like "Obama was born in 1961" in training, and thus it is easily recognized a span by the span detector. However, there no such label in the test domain and "1976" is thus assigned a false LOC type. (2) The pro- totypccal networks in decomposed methods target at learning a type-agnostic metric similarity func- tion to classify entities in test samples (i.e., query samples) via their distance to prototypes. Since the prototypes are constructed using very few support samples in the type-agnostic feature space, they might be inaccurate and unstable. For example, in Figure (c), a prototype is just the support sample in one-shot NER and thus deviates far away from the real class center.
2 Method
In this section, we formally present our proposed TadNER. The overall structure of our TadNER is shown in Figure 2. Note that the type-aware contrastive learning and type-aware span filtering strategies take effect at the type classification stage in the training and test domain, respectively.
Task Formulation Given sequence X {I1, 22,,,,,.., with N tokens, NER aims to sign each token 24 corresponding label Vi T U {O}, where T is the entity type set and O denotes the non-entity label. For few-shot NER, a model trained in a source domain dataset D soorrce with the entity type set Tsoorce = (41, f.,mm The model is then fine-tuned in a test/target domain dataset Diarget with the entity type set Tharges {t1, .2.p. using a given support set Starget- The entity token set and corresponding label set in Starget are denoted as E {ef, e E&r and ys = {vi V2 ..., vir), where of E Tharget is the label and M the number of entity tokens. The model supposed to recognize entities in the query set Qtarges of the target domain. Besides, Toouree and Tharget have no or very little overlap, making few-shot NER very challenging. More specifically, in the n-way k-shot setting, there labels in Tharget and k examples associated with each label in the support set Starget-
Based the above observations, we propose Type-Aware Decomposed framework, namely TadNER, for few-shot NER. Our method follows the span detection and type classification learning scheme in the decomposed framework but moves two steps further to overcome the aforementioned issues.
Firstly, we present type-aware span filtering strategy filter out false spans by removing those semantically far away from type names 2. By this means, the over-detected spans like "1976" whose types do not exist in test domain can be removed due to the long semantic distance to type names, as shown in Figure| (b).
2.1 Source Domain Training
The source domain training consists of span detec- tion and type classification stages. The procedure is shown in Figure 2 (a).
2.1.1 Span Detection
Secondly, we present a rype-aware contrastive learning strategy to construct more accurate and stable prototypes by jointly leveraging type names and support samples as references. Through this way, the type names can serve as the guidance for prototypes and make them not deviate too far away from the class centers even in some extreme outlier cases, as shown in Figure (d).
The span detection stage is formulated as se- quence labeling task, similar to an existing decom- posed NER model (Ma et al., 2022c). We adopt BERT (Devlin et al., 2019) with parameters O, as the PLM encoder fa, Given an input sentence , the encoder produces con-
X = {x1, X2, .... NN). textualized representations for each token as:
H hu bsl = fo, <xx),,,,
(I)
Extensive experimental results on 5 benchmark datasets demonstrate the superiority of our TadNER over the ofate-of-the-art decomposed methods. In particular, in the hard intra Few-NERD and 1 1-shot Domain Transfer settings, TadNER achieves 8% and 9% absolute Fl increase, respectively.
where E Ner 3. H is then fed into a classi- fication layer consisting of a dropout layer (Sri- vastava et al., 2014) and a linear layer to get the probability distribution P = |p(xx),...P(xxx) p(xN)] (p(x;) C RIC C = (I,O)) uuing a softmax
'In this paper, r denotes the hidden size the PLM. 'In Appendix A.6, we perform a detailed analysis using the 10, BIO, and BIOES tagging schemes.
NNote that though type assignments are unknown in few- shot NER, the type names (labels) in test domain are provided.

---

@a Source Domain Training
(b) Target Domain Inference
Type-Rarre
Contrastive lose
Charss
Apple
Apple
Apple
Feataane Spaice
Concatoontiin
Cdlerra
Cdlieasa
dite
dite
140
Entty& Obel
Lobel-xstty
Type-Aware Span Fillering
Lacake
Sungliian
Appie
BERT Type Classification
BERT Type Classiiiation
Obama was born 1961
person
Applehvas Callforniala in
Vollewaagos was founded in Germany
organization location
See
Clasdfication Layer
PER
DATE
C Classffication Layer
Ove-driated Tabs
tealy Type de
Cerrectly detected span
BERT- Span Detection
- Span Detoction
Barack Obama was born in 1961
Apple was founded in California 1976 Datry Srb
The overall structure of our proposed TadNER framework. (a) Training in the source domain. (b) Inference on the query set by utilizing the support samples in the target domain. Note that the source and target domains have different entity type sets.
Figure 2:
function:
another encoder foz with parameters # to obtain contextual representations:
p(x:) softmar(Dropout b)),
where W c RlC and b E IR. Cl are the weight matrix and bias.
h; [oz(es) @ foa (Map()))
(4)
After that, the training loss is formulated by the averaged cross-entropy of the probability distribu- tion and the ground-truth labels:
h = fo. (Map().) @ fo, (ex).
(5)
where ① is the concatenation operator, and he" and h/ denote two kinds of type-aware representations of the entity-token e; which are obtained in entity- label order and label-entity order, respectively.
Expan CrossEntropy(uu. p(x:)),
where ==0 when the i-th token is O-token, 4/=1 otherwise. Specifically, denote the training loss of span detection stage as Lxpan- During the train- ing procedure, the parameters (0,, W, b) are up- dated to minimize Lspan.
Type-Aware Contrastive Learning To learn a generalized and type-aware feature space, which can further be used for constructing more accurate and stable prototypes, we borrow the idea of contrastive learning (Khosla et al., 2020) and use two kinds of type-aware token representations mentioned above to construct positive and negative pairs shown in Figure 2 (a), i.e., those with the same label in different orders positive pairs and those with different labels as negative pairs. The type-aware contrastive loss calculated as:
2.1.2 Type Classification
Representation Given an input sentence X, we only select entity-tokens E = {e1,e2, e2, eM) (E X) with ground-truth labels Y (11, U2,,.,, for the training of this stage. For the entity type set Tsmuror (t,,t,, t.,, of the source domain Dsouree, we manually convert them into their corresponding type names T Tsoures Map(Taource) = (1, (,., (.
Σ exp(sim(h) h21)/))
M
11,1
Lrype
log
1=1
exp exp(sim(hp!) (h))m))))))
After that, to obtain tokens with type name in- formation, which are further used for calculating contrastive loss, we concatenate entity tokens with their corresponding labels in two orders, i.e., entity- label order and label-entity order. Here we use
Z. = 1=|1 M. wc
h h
sim(h; ol b//b))
(8)
ΣM
Σmー (hil h)o")
(M is the function used to convert label to a type name, c.g. "PER" to "person"". Please refer to Appendix A.7 for type names of all datasets.
where M is the number of entity tokens in a batch and Z, is the set of positive samples with the same

---

label type Vr. Here we adopt the dot product with a normalization factor as the similarity function sim(). We also add a temperature hyper-parameter T for focusing more on difficult pairs (Chen et al., 2020). During the source domain training, the parameters 02 are updated to minimize Liype.
is far from all the golden type names. Based on this assumption, we calculate a threshold % with the fine-tuned encoder foz using entity tokens and corresponding type names in the support set:
?
min Sor (et) fog (Map(u())" Ist5
(11)
2.2 Target Domain Inference
This threshold Yt is used to remove the over- detected false spans. And the remaining candidate spans will be assigned corresponding labels.
As illustrated in Figure (b), during the target do- main inference, we first extract candidate spans in query sentences and then remove over-detected false spans via the type-aware span filtering strat- egy. Finally, we classify remaining candidate spans into certain entity types to get the final results.
Type-Aware Prototype Construction We can construct a type-aware prototype for each entity type ty E Tiarget, which more accurate and stable owing to the generalized and type-aware feature space learned in the source domain:
2.2.1 Span Detection
The span detector with its parameters {01, W,b trained in the source domain is further fine-tuned with samples in the support set Starget in the target domain to minimize Expan in (3). To alleviate adopt a loss-based early
Pj fgy (Map(t,)) ) @
(e))
(12)
12,
IEZ
2, 1515 M. t,)
(13)
the risk of over- over-fitting, stopping strategy, i.e., stopping the fine-tuning pro- cedure once the loss rises 3 times continuously. where 33 is happe--aramettr.
where + is the concatenation operator and Zj de- notes the set of entity words with the label type ty in the support set.
After fine-tuning the span detector, we use it to detect entity words of query sentences in Qtarget and then consider continuous entity words as a candidate span, e.g., "Barack Obama". Finally, we obtain the candidate span set Cspan containing all candidate spans, which will be assigned entity types at the type classification stage.
Inference
For each remaining candidate span 8t, assign it label type tj Tearget with the high- est similarity:
Mproed
arg max (h; Pj E
(14)
= fag (8c) @ (a)
(15)
2.2.2 Type Classification
Domain Adaption Benefiting from the gener- alized and type-aware feature space trained in the source domain, we can further get a domain- specific encoder Jog via fine-tuning with the fol- lowing loss:
where Pj is the sype-aware prototype representation corresponding to the label type łj. and Upred is the predicted label type of the candidate span h; is the self-concatenated representation of s; for consistency with the dimension of the prototype Pj- The entire procedure of inference in the target domain presented in Appendix A.I.
Lishel
Σ
M
s(e Map()())
M
s(ef, Map(ts))
3 Experiments
s(p.4) 4) (oz(o) for TT)
(10)
3.1 Evaluation Protocal
Type-Aware Span Filtering As we illustrate in the introduction, the span detector may generate some over-detected false spans whose type names only belong the source domain, since the seman- ties of entity type names are not considered the span detection stage. To solve this problem, we propose a type-aware span filtering strategy during the inference phase to remove these false spans. In- tuitively, the semantic distance these false spans
Ding et al. (2021) propose a large scale dataset Few-NERD for few-shot NER, which con- tains 66 fine-grained entity types across 8 coarse- grained entity types. It contains intra and in- ter tasks where the train/dev/test sets are divided according to the coarse-grained and fine-grained types, respectively. Besides, following Das et al. (2022), we also conduct Domain Transfer experi- ments, where data are from different text domains
Datasets

---

Intra
Inter
Paradigms Models
1-22-hhl
shott
1-22-hot
S- 3-10hhhol
Avg.
Avg.
way
10 way
10 way
5 way
10 way
5 way
10 way
PriiBBERT
20.76-n.84
15.05=044
42.54-0.94
35 13
28.44
38.83=1.49
32.45-0.79
58.7940.44
52.9240.37
45.75
NNShot
25.78.0.91
18.27
36. 18,0.9
27. 27.38
26.90
47.24_1.00
38.87.021
53.64,0.3
49,57,2.73
47.83
One-slage
StructSbot"
30,21=0.00
21.03e1113
38.00e1.29
26/42+0.0
28.92
51,882069
43.342010
57.3220.5
49,5723.05
50.53
FSI.S"
30.38-2.85
28.31+405
46.85+3.49
40.76+3.18
44.52-4.59
44.01=1.3
59.74x2.51
56.67e1.75
51.24
CONTaINER
41.51a0007
36.62a0:01
57.83a0.01
51.04a0.21
46.75
50.92.029
47.0
47.02s0.24
63.35
60.14
55.36
ESD
36.08u1.60
30.00u0.70
52.14:1.50
42.1512.60
40.09
59.29
52.16.0.79
69.0610.80
64.0010:43
61.13
Two-stage
DecomposedMetuNER
49,48=0.85
42.84g0.45
62.92s0.57
57.31
53.14
64.75=0.35
58.65.0.43
71.49.0.47
68.11
65.75
TadNER
60.78:0.32
55.44:0.0
67.94s0.17
60.8740.21
61.26
64.83=0.14
64.0620.19
72.12s0.12
69,94.0.15
67.74
Table 1: F1 scores with standard deviations for Few-NERD. denotes the results reported by Ma et al. (2022c).
denotes the results reported by our replication using data of the same version. The best results are bold and the second best ones are underlined.
1-shot
S-shol
Paradigms Models
121:2
CONLA
WNUT
GUM
Avg.
12B2
CoNI.L
WNUT
GUM
Avg.
ProteBERT
13.4=3.0
49.9
17.4449
17.8
24.6
17.9=18
61.3s%.1
22.8s4.5
19.5e3.4
30.4
One-stage
NNShot
15.3
61.2+10.4
22.7474
10.5229
27.4
22.0s15
74.1
27.325.
15.9
34.8
StructShot
21.4238
62.4110.5
24.2a8.0
7.812.1
29.0
30.322.1
74.8
30.416.5
13.311.3
37.2
FSL.S*
18.3+3.5
50.9165
14.3+5.5
12.6:28
24.0
25.4=2.
63.9-33
24.0+32
18.8 #2.
33.1
CONTANNER
21.5+1.7
61,2e10.7
27.5e19
18.5+49
32.2
36.7=2.1
75.8+17
32,5+18
25.2+2.7
42.6
DecomppseeMetaNNER
15.5=3.0
61.219.2
27,7a5.3
20.3:4.
31.2
19.8=2.
75.2
29.8
33.5+2.4
39.6
Tivo-ttae
TadNER
39.3=38
70.4s10.6
32.8e4.8
24.2s4.1
41.7
45.2a2.3
80.5a3.6
34.5a44
35,1+22
48.8
Table 2: FI scores with standard deviations for Domain Transfer. denotes the results reported by Das et al. (2022). denotes the results reported by our replication. Since no previous two-stage methods have conducted experiments under this setting, choose the strong DecomposedMetaNER for reproduction experiments, and denotes the results reported by our replication. The best results are in bold and the second best ones are underlined.
(e.g., Wiki, News). We take OntoNotes (Gen- eral) (Weischedel al., 2013) as our source do- main, and evaluate few-shot performances on 12B2 (Medical) (Stubbs and Uzuner, 2015), CoNLL (News) (Tjong Kim Sang and De Meulder, 2003), WNUT (Social) (Derczynski et al., 2017) and GUM (Zeldes, 2017) datascts.
NERD and Domain Transfer, respectively. We have the following important observations: 1) Our model demonstrates superiority under Few-NERD settings. Notably, in the more challenging intra task, our TadNER achieves an average 8.2% in- crease in F1 score. Besides, our model outperforms basclines by 10.5% and 9.2% under 1-shot and 5-shot Domain Transfer settings, respectively. 2) Particularly, when provided with very few samples (e.g., g.. 1-shot), the improvements become more significant, which is very attractive property. 3) The performance of DecomposedlMetaNER, a competing model, severely deteriorates under certain settings, such as I2B2. This is primarily due to the presence of numerous sentences without entities, leading to multiple false detected spans. In contrast, our TadNER effectively mitigates this issue through the tape-aware span filtering strategy, successfully removing false spans and achieving promising results.
Baselines We compare our proposed TadNER with many strong baselines, including one-stage and !wo-stage types. The one-stage base- lines include ProtoBERT (Snell et al., 2017), NNShot (Yang and Katiyar, 2020), Struct- Shot (Yang and Katiyar, 2020), FSLS (Ma et al., 2022a) and CONTaiNER (Das et al., 2022). Note that FSLS also adopts type names. The two-stage baselines include ESD (Wang al., 2022) and the DecomposedMetaNEN (Ma et al., 2022c) 6
3.2 Main Results
Table 1 and 2 report the comparison results between our method and baselines under Few-
"Results are tested with the latest version of data from
http::://ningddig97..g. .it. Lo/fewnerd/. which is corresponding with https: /ttps://github. com/ microsoft/vert-noo hwtpeeetpppppsseeeesssss/////ppe DecomposeddetaaMERffee -e
Please refer to Appendix A.2-A.5 for more deseriptions about datascts, evaluation methods, hasclines and implemen- tation details.

---

C1: Query sentence:
with the promotion of emrespor the burkish third league the end the 2011 season
DecaM::sedMetaNER:: organization-sporttitoo sportsteam: emrespor (v), turkish tff third league (x)
TadNER (ours):
organizationsportsseemm. emrespor (v) wewpanittttooosotsllaaeoee turkish tff third league (v)
C2: Query sentence:
Leicestershire beat Somerset by an innings and 39 runs in two days.
DecomposedMettNERR, ORG: L eicestershire (v) LOC; Somerset (x), two
TadNER (ours): ORG: L Leicestershire (v), Somerset (v)
Figure 3: Case study. CI and C2 are from Few-NERD intra and CoNLL2003 Cross datasets, respectively, and organization-sporttteamm organization--portsleaagg ORG and LOC are entity types.
3.3 Ablation Study
strates that the training objective in the source do- main training phase aligns well with the target do- main finetuning phase via task decomposition and contrastive learning strategy, despite having dif- ferent entity classes. As a result, the model can effectively utilize the provided support samples from the target domain, enhancing its performance in few-shot scenarios.
To validate the effectiveness of the main compo- nents in TadNER, we introduce the following vari- ant baselines for the ablation study: 1) TadNER w/o Type-Aware Span Filtering (TASF) removes the type-aware span filtering strategy and directly feeds all spans detected span detection stage type classification. 2) TadNER w/o Type Names (TN) further replaces type names with random vectors when calculating contrastive loss and constructs class prototypes using only the support samples. 3) TadNER w/o Span-Finetune skips the target do- main adaptation of the span detection stage. 4) Tad- NER w/o Type-Finctune skips the target domain adaptation of the type classification stage.
3.4 Case Study
To examine how our model accurately constructs prototypes and filters out over-detected false spans with the help of type names, we randomly select one query sentence from Few-NERD intra and CoNLL2003 for case study. We compare TadNER with DecomposedMetaNFRR (Ma et al., 2022c), which also belongs to the two-stage methods.
1-sbot
5-shot
Models
Avg.
1262
CONLL
WNUT GUM
1292
CONLL WNUT GUM
As shown in Figure 3, in the first case, our model correctly predicts "turkish tff third league" as "organization-ssortticee
TadNER
70.4
wol TASF
21.2
68.5
31.6
24.2
27.4
80.1
34,3
35.1
40.3
DecomposedMetaNNER identifies type, while a wrong "organizationnsportttaan"" type. Since the type name and the entity span have an overlapping word "league", incorporating the type name into the construction of the prototype will make the identification much easier. Conversely, without the type name, it would be hard to distinguish two categories of entities because they both represent "sports-related" organizations".
wol TN
20.0
65.6
28.3
20.3
26.2
76.3
33.8
33.2
38.0
Span-Finctune
37.0
52.5
150
50.8
wo Type Finetune 37.6
68.3
32.3
20.3
45.2
76.3
33.6
27.9 42.7
Table 3: Results (F1 scores) for ablation study under Domain Transfer settings. The best results are in bold.
From Table 3, we can observe that: 1) The removal of the type-aware span filtering strategy leads to drop in performance across most cases, particularly in entity-sparse datasets like 12B2, where a large number of false positive spans are detected. Besides, for entity-dense datasets like GUM, the performance is not harmed by the span filtering strategy, which proves the robustness and effectiveness of our model in various real- woold applications. 2) The omission of type names also results in significant decrease in performance, in- dicating that our model indeed learns type-aware feature space, which plays crucial role in few- shot scenarios. 3) The elimination of finetuning in the span detection and type classification stages ex- hibits a substantial performance drop. This demon-
In the second case, DecomposedMetaNER incor- rectly identifies "two" as an entity span and then assigns it a wrong entity type "LOC", since there are many samples like "TTh two sides had not met since Oct. 18" in the source domain Ontonotes, where "two" is an entity of "CARDINAL" type. In contrast, our TadNER successfully removes this false span via the type-aware span filtering strategy.
3.5 Impact of Type Names
To further explore the impact of incorporating the semantics type names and whether model perfor-

---

mance is sensitive to these converted type names. We perform experiments with the following vari- ants of type names: 1) Original type names, which are used in our main comparision experiments. 2) Synonymous type names. We generate three syn- onyms for each original type name as variants using ChatGPT. These synonyms were automatically gen- erated to explore the effect of different but related type names on model performance. 3) Meaningless type names, e.g,, "label 1" and "label 2". 4) Mis- leading type names, e.g., "person" for "LOC" and "location" for "PER" in the CoNLL dataset. Please refer to Appendix A.7 for details.
module in a decomposed NER framework, in or- der to explore whether it can address the issue of unstable and inaccurate prototypes. Here we adopt the same 10 samplings used in the 1-shot Domain Transfer experiments.
Somsinng hoox
Samping reee
Surniinng hnxex
Samplngg reee
Figure 5: Impacts of prototypes by different methods under 1-shot Domain Transfer setting. The horizontal and vertical coordinates indicate the n-th sampling and the accuracy of type classification, respectively.
Sway P1
Figure 4: Fl Scores on Few-NERD Intra and CoNLL 2003 with different variants of type names.
As shown in Figure 5, proposed TadNER achieves a significant improvement over Decom- posedMetaNER on each dataset and is more stable across different samplings. Besides, removing type names causes a sharp performance drop in some cases for TadNER w/o TN, indicating that the in- corporation of type names indeed helps construct more stable and accurate prototypes. Moreover, Vanilla CL performs extremely poorly due to the in- troduction of an additional projection layer, which is crucial component employed in various con- trastive learning methods (Chen et al., 2020; Das et al., 2022). However, the inclusion of this layer hampers the model's 's capacity to acquire adequate semantics related to entity classification.
As shown from the Figure 4, we can make the following observations: 1) All three variants of syn- onym type names have comparable performance, indicating that our method is robust to different ways of transforming type names. However, the best way is still the direct transformation method, such "person" for "PER", which is how we ob- tain the original type names. 2) Irrelevant or in- correct information meaningless and misleading type names leads to a significant degradation in model performance, indicating that the semantics associated with entity classes are more suitable as anchor points for contrastive learning.
3.6 Impact of Type-Aware Prototypes
3.7 Error Analysis
In order to investigate the effectiveness of our pro- posed strategy for solving the problem of inaccu- rate and unstable prototypes in the type classifi- cation stage, we further perform an analysis of the impact of stability and quality of prototypes. We select three baselines as our compared meth- ods: 1) TadNER w/o Type Names (TN) (the second variant baseline in the ablation study). 2) Decom- posedMetaNER (Ma et al., 2022c). 3) Vanilla Con- trastive Learning (CI., which adopts token-token contrastive loss and was proposed by Das et al. (2022). We use it to train the type classification
We conduct an error analysis to examine the de- tailed types of errors made by different models. The error statistics are shown Table 4.
We can observe that: 1) Our TadNER makes fewer errors than baselines overall. Notably, it significantly reduces false negatives, indicating its ability to accurately recall more correct entities. 2) Both TadNER and FSLS can effectively reduce "Type" errors by incorporating type names. How- ever, though FSLS has less "Type" errors than our TadNER, produces a much larger number of un-

---

Fabe Preilive
Faller Negutive
support samples, 2) since the feature space learnt in the type classification stage is well generalized and type-aware, we can directly adopt the represen- tations of type names as prototypes of novel entity types. To demonstrate the promising performance of our model under zero-shot settings, we select SpanNER (Wang et al., 2021) as a strong baseline, which is a ccromposed-baseee method and good at solving zero-shot NER problem.
Models
Span
Type
Spun
Type
False
Num.
Rasio
Num,
Ratio
Num.
Ratio
Num.
Ratio
PRLS
990
65.48
169
1468.
87.5%
169)
12.58
CONTANER
581
63.3%
S11
36.7%-
628 55.1%
511
44.9% 2531
ESD
562
56.5%
133
43.5%
932
68.9%
433
51.7% 2360
43.8%
485
43.1%
TaINER
786
51.2%
182
188%
450
71.2%
182
28.8%
1600
Table 4: Error analysis for different methods under the Few-NERD Intra 5-way 1~2-shot setting. We select the first 300 episodes for analysis. "False Positive" and "False Negative" denote the incorrectly extracted entities and unrecalled entities, respectively. "Span" " and "Type" denote the error is due to incorrect span/type.
Domain Transfer
Model
12B2
CoNLL WNUT
GUM
Avg.
SpanNER (0-shot)
8.02
23.63
24.82
6.57
15.76
recalled samples, i.e., false negatives. 3) Our Tad- NER still suffers from inaccurate span prediction, which inspires our future work.
TadNER (0-shot)
17.13
43.14
25.06
7.62
23.24
Table 6: F1 scores under Domain Transfer zero-shot settings.
3.8 Model Efficiency
As shown in Table 6, our proposed TadNER per- forms better than SpanNER (Wang et al., 2021) under every case. The reason for this may be that the type classification of SpanNER is based on a traditional supervised classification model, which performs worse generalization in cross-domain sce- narios. Besides, compared with previous metric- based methods (Das al., 2022: Ma et al., 2022c) for few-shot NER, which heavily rely on support
Compared to one-stage approaches, c.g., CON- TaiNER, two-stage models require more param- eters, longer training and inference times. To have a close look at the time cost induced by two-stage models, we perform a model efficiency analysis and show the results in Table 5.
Paradigms
Models
#Para.
Train Adapt Inference
F1
FSLS
222M
1871s
10s
14ms
30.38
One-stage
CONTaiNER
112M
980s
Is
17ms
41.51
sets and had no zero-shot capability, our method is more inspirational for future zero-shot NER works.
112M
2601s
35ms
36.08
Two-stage
DoompoodMecnNER
222M
35-955
2s
37ms
49.48
TodNER
3796s
1.5g
32ms
60.29
Related Work
Table 5: Model efficiency analysis for different methods under the Few-NERD Intra 5-way 1~~2-shot setting.
Few-Shot NER Few-shot NER methods can be categorized into two types: prompt-based and metric-based. Prompt-based methods focus on leveraging pre-trained language model knowledge for NER through prompt learning (Cui et al., 2021; Ma et al., 2022b; Huang et al., 2022; Lee et al., 2022). They rely on templates, prompts, or good examples to utilize the pre- pre-trained knowledge effec- tively. Metric-based methods aim to learn feature space with good generalizability and classify test samples using nearest class prototypes (Snell et al., 2017; Fritzler et al., 2019; Ji et al., 2022; Ma et al., 2022c) or neighbor samples (Yang and Katiyar,
From Table 5, can be seen that two-stage mod- els indeed require longer training and inference time than one-stage models. However, two-stage models often get better performance. In particular, our TadNER is the most effective one among both one-stage and two-stage models, and it achieves F1 improvement of 45% and 67% over CON- TaiNER and ESD. It is also the most efficient one among three two-stage models in terms of the in- ference time.
3.9 Zero-Shot Performance
2020; Das et al., 2022).
Since there is no domain-speciffe support set under zero-shot NER settings, it is extremely challenging and rarely explored. While we believe our pro- posed TadNER can obtain certain zero-shot ability after training in the source domain for the following two reasons: 1) the model can extract entity spans in the span detection stage before fine-tuning with
There are also some efforts to improve few-shot NER by incorporating type name (label) seman- tics (Hou et al., 2020; Ma et al., 2022a). These methods usually treat labels as class representa- tives and align tokens with them, yet neglecting the joint training of entity words and label representa- tions. Hence they can only use either support sets

---

labels as class references. Instead, our method exploits support samples and type names simulta- neously, which helps construct more accurate and stable prototypes in the target domain.
Limitations
Our proposed TadNER mainly focuses on the type classification stage of few-shot NER and simply adopt token classification for detecting entity spans. There might be better solutions, e.g., using global boundary matrix. However, due to its high GPU memory requirements, do not include it in our current framework. This drives us to find more efficient and powerful span detector for better few- shot NER performance in the future.
Task Decomposition and Contrastive Learn- ing Recently, decomposed methods have emerged as effective solutions for the NER prob- lem (Shen et al., 2021; Wang et al., 2021; Zhang et al., 2022; Wang et al., 2022; Ma et al., 2022c). These methods can learn entity boundary informa- tion well in data-limited scenarios and often get better results. However, the widely used proto- typical networks in these methods may encounter inaccurate and unstable prototypes given limited support samples at the type classification stage. Be- sides, they may face the problem of over-detected false spans produced at the span detection stage. Our method can address these two issues via the proposed type-aware contrastive learning and type- aware span filtering strategies.
Ethics Statement
Our work is entirely at the methodological level and therefore there will not be any negative social impacts. In addition, since the performance of the model is not yct at a practical level, it cannot be applied in certain high-risk scenarios (such as the I2B2 dataset used in our paper) yet, leaving room for further improvements in the future.
Our method is also inspired by contrastive learn- ing (Chen et al., 2020; Khosla et al., 2020). Due to its good generalization performance, two recent methods (Das al., 2022; Huang et al., 2022) bor- row this idea for few-shot NER, which construct contrastive loss between tokens or between the to- ken and the prompt. However, they are both the end-to-end approach and thus have the inherent drawback that cannot learn good entity boundary information. In contrast, our method is a decom- posed one and our contrastive loss is constructed between tokens with additional type name informa- tion, which can find accurate boundary and learn aype-aware feature space.
A Acknowledgments
This work was supported by the grant from the Na- tional Natural Science Foundation of China (NSFC) project (No. 62276193). It was also supported by the Joint Laboratory on Credit Science and Tech- nology of CSCI- Suuhnn University.
References
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
5 Conclusion
In this paper, propose novel TadNER frame- work for few-shot NER, which handles the span detection and type classification sub-tasks at two stages. For type classification, we present type- aware contrastive learning strategy to learn type- aware and generalized feature space, enabling the model to construct more accurate and stable pro- totypes with the help of type names. Based on it, we introduce type-aware span filtering strategy for removing over-detected false spans produced the span detection stage. Extensive experiments demonstrate that our method achieves superior per- formance over previous SOTA methods, especially in the challenging scenarios. In the future, we will try to extend TadNER to other NLP tasks.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Trans- actions of the Association for Computational Linguis- rics, 4:357-370.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang. and Yue Zhang. 2021. Template- named entity recognition us- ing BART. In Findings of the Association for Com- putational Linguistics: ACI-IJCNLPP 2021, pages 1835-1845, Online. Association for Computational Linguistics.
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. 2022. CONTaINER: Few-shot named entity recognition via contrastive learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume Long Papers), pages 6338-6355 Dublin, Ireland, Association for Computational Linguistics.

---

Leon Derezynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNU12017 shared task on novel and emerging entity recogni- tion. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguis- tics.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. 2020. Su- pervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 260-270, San Diego, California. Association for Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dong- Ho Lee. Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng. Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sckiya, Jay Pujara, and Xiang Ren. 2022. Good examples make a faster learner: Simple demonstration- based learning for low-resource NER. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2687-2701 Dublin, Ireland. Association for Computational Linguisties.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD. A few-shot named entity recognition dataset. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the I 1Ith International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 3198-3213, Online, Association for Computational Linguistics.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named en- tity recognition task. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing, SAC 19, page 993-1000, New York, NY, USA. As- sociation for Computing Machinery.
Jie Ma, Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan Roth. 2022a. Label semantics for few shot named entity recognition. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 1956- 1971, Dublin, Ireland. Association for Computational Linguistics.
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1381- 1393, Online. Association for Computational Linguis- tics.
Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2022b. Template- free prompt tuning for few-shot NER. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5721-5732, Seattle, United States. Association for Computational Linguistics.
Yucheng Huang, Kai He, Yige Wang, Xianli Zhang, Ticliang Gong, Rui Mao, and Chen Li. 2022. COP- NER: Contrastive learning with prompt guiding for few-shot named entity recognition. In Proceedings of the 29th International Conference on Computational Linguistics. pages 2515-2527, Gyeongju, Republic of Korea. International Committee Computational Linguisties.
Tingting Ma, Huiqiang Jiang. Qianhui Wu, Tiejun Zhao, and Chin- Yew Lin. 2022c. Decomposed meta- learning for few-shot named entity recognition. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 1584-1596, Dublin, Ire- land. Association for Computational Linguistics.
Zhiheng Huang. Wei Xu, and Kai Yu. 2015, Bidirec- tional lstm-crf models for sequence tagging. arXiv preprint 1xxiv:1508.0199)
Xuezhe Ma and Eduard Hovy. 2016. End-to-end se- quence labeling via bi-directional I. LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074 Berlin, Germany. Association for Computational Linguistics.
Bin Ji, Shasha Li. Shaoduo Gan, Jie Yu, Jun Ma, Huijun Liu, and Jing Yang. 2022. Few-shot named entity recognition with entity-level prototypical network enhanced by dispersedly distributed prototypes. In Proceedings of the 29th International Conference on Computational Linguisties, pages 1842-1854, Gyeongju, Republic of Korea. International Com- mittee on Computational Linguistics.
Yongliang Shen, Xinyin Ma. Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021. Locate and la- bel: A two-stage identifier for nested named entity

---

recognition. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 2782-2794, Online. Association for Computa- tional Linguistics.
Yi Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning- In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6365-6375, Online. Association for Computational Linguistics.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical Networks for Few-shot Learning. In Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Amir Zeldes. 2017. The gum corpus: Creating multi- layer resources the classroom. Lang. Resour, Eval.. 51(3):881-62..
Xinghua Zhang, Bowen Yu, Yubin Wang, Tingwen Liu, Taoyu Su, and Hongbo Xu. 2022. Exploring modu- lar task decomposition in cross- rross-domain named entity recognition. In Proceedings of the 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 301-311
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, (5(1):1929-3 1958..
Amber Stubbs and Özlem Uzuner. 2015. Annotating longitudinal clinical narratives for de-identification: The 2014 12b2/uthealth corpus. Journal of biomedi- cal informatics, 58:S20-S05 S29..
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-20033 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HILT-NACCI. 2003, pages 142- 147.
Peiyi Wang, Runxin Xu, Tianyu Liu, Qingyu Zhou, Yunbo Cao, Baobao Chang, and Zhifang Sui. 2022. An enhanced span-based decomposition method for few-shot sequence labeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5012-5024, Seat- tle. United States. Association for Computational Linguistics.
Yaqing Wang, Haoda Chu, Chao Zhang, and Jing Gao. 2021. Learning from language description: Low-shot named entity recognition via decomposed framework. In Findings the Association for Computational Linguistics: EMNLP 2021, pages 1618-1630, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5.0 2t119. Linguistic Data Consortium, Philadelphia, PA.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of the natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.

---

Algorithm 1 Procedure of target domain inference in TadNER.
Dataset
Domain
# Classes
# Sentences
# Entities
Few-NERRD
Wikipedia
66
188.2k
491.7k
12B2214 14
Medical
29.2k
Require: Support set Stargeti Query set Qtarget; Class set
CO
News
4
20.7k
35. 1k
Turgeti Encoders Sa., Sez:
GUM
Wiki
3.5k
6.1k
Output: Query set predictions Srenut
WN
Social
5.7k
3.9K
1: Eprev = 20; Lyree Rt, (Any large positive value);
OntoNotes
General
76.7k
104.2k
2: Expan Cpren
3: while Wapae Lyrce do
Table 7: Dataset statistics
4:
Lyree Lapan:
5:
Compute loss Lispan using Eq. (3);
Update foz + Sdl to reduce Lapans
A.3 Details of Evaluation Methods
7:
end while
Episode-level Evaluation Following Ma et al. (2022c), we adopt the episode- evaluation method for the Few-NERD dataset. Each episode consists of support set and query set, both given in the n-way k-shot form. In each episode, the model trained in the source domain tested on the query set by utilizing the support set. To make fair comparisons, we obtain the Micro Fl score with the episode-data processed by Ding et al. (2021). We report the mean Fl score with standard deviation using 3 different seeds.
8:
Lprev 30; Lyree R+ (Any large positive value);
9:
Liahtl Lyree
10:
while Liabel Lpreu do
11: Eyreer Liabet;
12: Compute loss Liabel using Eq. (9);
13: Update parameters O2 B2 to reduce Liatel;
14:
end while
15:
Cepan = 0::
16: for X; Qtarget do
17: Extract candidate entity spans C% from sentence
X; according Section 2.2.1;
18: Cepan Copen Capani
19: end for
20: Calculate threshold tu for span filtering using Eq. (1I);
21: Calculate all prototypes in Terreet using Eq. (12);
Dataset-level Evaluation Yang and Katiyar (2020) point that sampling test episodes may not reflect the real-world performance due to various data distributions, and they propose to sample sup- port sets and then test the model in the original test set. Each support set consists of & examples corre- sponding to each label. The final Micro Fl scores and standard deviations ent sampled support sets. Thus, following Yang and Katiyar (2020) and Das et (2022), we also adopt this evaluation schema for Domain Transfer settings. For fair comparisons, we use the support sets sampled by Das et al. (2022) 8
22: The prototype of class i denoted Pj:
Sresule ():
23: for 8, in Cspon do
24:
max sim
max t Ttrrgett
(s) .pp))
25:
if max_ sim/ > 7/. then
26:
Assign the label ypred to using Eq. (14);
27:
Sresuit = Sresutt U [8:];
28:
else
29:
Remove this candidate span 8;:
obtained using differ-
30:
end if
31:
end for
A Appendix
A.1 Target Domain Inference Algorithm
A.4 Baselines
Algorithm describes the process of domain adap- tation using support set in the target domain and prediction on the query set. Lines 1-7 describe the target domain adaptation process for the span detec- tion stage. Lines 8-14 describe the target domain adaptation process for the type classification stage. Lines 15-19 describe the extraction of candidate entity spans the query set using the fine-tuned span detector. Lines 20-31 describe the candidate entity span filtering and entity type classification using awpe- prototypes.
ProtoBERT (Fritzler et al., 2019) adopts token-level prototypical network, where the prototype of each class is obtained by averaging token samples of the same label, and the label of each unlabeled token in the query set determined by its nearest class prototype.
NNShot (Yang and Katiyar, 2020) pre-trains BERT by traditional classification methods in the source domain training phase, and decides the class of each unlabeled token by the nearest neighbor at the token level in the target domain inference phase.
StructShot (Yang and Katiyar, 2020) is based on NNshot and uses an abstract transition probability for Viterbi decoding during testing.
A.2 Statistics of Datasets
Table 7 shows statistics of various datasets used in
our experiments.
httpps: htttss///sttttu..ommmmmpppprrp// httm://sunlggrruup/CNTaaatte

---

ESD (Wang et al., 2022) uses span-level protolypical network, which designs multiple prototypes for O-tokens and uses inter- and cross-span attention for better span representation. FSLS (Ma et al., 2022a) adopts two encoders, one for obtaining type names representations and the other for token representations, During the training procedure, the Euclidean distance between tokens and their corresponding type name semantics minimized. During prediction, the label for token is determined based on the closest type name semantics. We chose this baseline to demonstrate the superiority of our approach over existing approaches using the semantics of type names. CONTaiNER (Das et al., 2022) first trains BERT in the source domain using token-level contrastive learning loss function, then fine- tumettmes the trained model the support set, and finally use the nearest neighbor method proposed in NNShot (Yang and Katiyar, 2020) for target domain inference phase. DecomposedMetaNER (Ma et al., 2022c) a decomposed approach that incorporates model- agnostic meta-lcarning strategy into traditional prototypical network to learn model-agnostic model and more fully exploits the support set.
Learning rale Batch Dropout rale temperature
(le-5, 3e-5, le-4] { 32, 64, 128)
(0.1, 0.2, 0.5] (0.01. 0.05, 0.1] (1, 4, 6,
Early stopping threshold
Table 8: Hyper-parameters search space in our experi- ments.
A.6 Analysis of Tagging Schemes in the Span Detection Stage
Table and Table 10 show the span detection and overall performance under the Domain Transfer settings. We observe that: 1) The three tagging schemes have their own advantages and disadvan- tages. 10 and BIO schemes can achieve higher recall, BIOES achieve higher precision. 2) The IO tagging scheme can achieve the best over- all performance in most settings, except for the GUM dataset. Therefore, the IO scheme selected for the span detection stage in this paper. 3) The type-aware span filtering strategy proposed in this paper shown robust and positive effects across dif- ferent tagging schemes. Even when dealing with entity-dense datasets, where incorrect entity spans are minimal, this strategy does not significantly impair performance. In future work, we can try to combine the advantages and disadvantages different tagging schemes to further improve the performance of the span detection stage.
A.5 Implementation Details
Following previous methods (Ding et al., 2021; Das et al., 2022; Ma et al., 2022c), we
A.7 Detailed Type Names
bert-base-uncased model (Devlin al., 2019) from HuggingFace (Wolf al., 2020) as our en- coder fo, and fo..
Tables 11 and 12 show the type names used in our TadNER framework. Tables 13 and 14 show the variant type names used in the analysis experiments the impact of type names in Section 3.5.
During the source domain training procedure, we use AdamW (Loshchilov and Hutter, 2019) as the optimizer with learning rate of 3e-5 and 1% linear warmup steps, and the batch size is set to 64. We set the temperature hyper-parameter 0.05 in E6) and keep dropout rate as 0.2 in the classification layer of the span detection.
As for the early stopping strategy in 2.2.1, we found that the fewer samples face a higher risk of over-fitting, and lower 3 threshold is required. So set =2 2 in all 1-shot settings and = 6 in all other cases. Table 8 shows the searching space of each hyper-parameter. Besides, implement our framework with Pytorch 10 and train with V100-16G GPU.
http:///ggggnnneeee..ooooteeeeee "https://pytorch.oo

---

12B2
CoNLL
Stage
Filtered
Schema
I-shot
5-shot
T-shot
5-shot
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
19.62
70.59
25.12
71.71
84.28
90.68
No
BIO
19.84
67.89
30.49
22.36
75.76
34.40
72.01
84.27
77.15
85.87
88.78
87.24
BIOES
19.71
60.46
29.47
23.89
70.19
35.53
70.38
80.93
74.89
84.02
87.77
85.72
Span
IO
53.79
41.54
45.33
55.78
52.84
53.82
78.65
83.25
80.47
87.86
89.56
88.67
BIO
54.20
40.83
45.63
53.24
55.64
53.63
77.22
84.29
80.18
87.11
88.78
87.90
BIOES
52.77
34.04
39.80
57.46
50.97
53.32
74.39
80.72
77.00
84.65
87.65
86.00
14.14
47.18
21.57
17.83
51.11
26.35
65.37
72.73
68.47
79.06
81.32
80.14
No
BIO
14.92
49.32
22.74
16.65
$4.40
25.40
63.66
74.08
68.01
77.88
80.27
79.00
BIOES
14.18
42.01
21.02
17.44
49.36
25.69
61.84
70.69
65.62
76.36
79 /6
77.75
Span Type
10
47.24
35.77
39.32
46,92
44.33
45,20
68.89
72. 77.70
70.38
79.81
81 .1
80.53
35.42
45.18
47.00
74.07
70.52
78.75
80.26
BIOES
45.47
28.69
33.80
47.54
42.15
44.10
65.26
70.62
67.46
76.80
79.46
77.99
Table 9: span detection.
WNUT
GUM
Stage
Filtered
Schema
I-shot
5-shot
I-shot
5-shot
Precision
Recall
F1
Precision
Recall
F1
Precision
Recall
F1
Precision
Recall
IO
38.42
65.42
47.37
40.70
65.6< 65.64
49.13
45.93
45.70
45.72
56.41
64.25
60.04
BIO
40.89
63.85
48.82
38.28
68.60
48.92
44.86
45.67
45.05
53.99
64.34
58.64
BIOES
56.14
65.24
48.32
64.07
62.22
Span
63,50
65.27
Yes
BIO
43.61
61.33
49.41
38.74
68.15
49.17
45.55
45.74
45.44
54.74
64.40
59.11
BIOES
45.78
54.39
48.14
42.45
65.06
50.92
54.58
47.92
50.88
60.88
63.56
62.15
10
25.86
43.19
31.62
28.59
45.35
34.26
24.64
23.90
24.21
33.39
37.02
35.09
BIO
27.34
42.11
32.52
25.69
45.83
32.77
24.97
25.24
24.99
33.14
39.04
35.81
BIOES
28.94
36.92
31.74
28
44.37
34.48
30.20
26.65
28.23
37.38
39.02
38.15
Bpan+Type
10
42.60
32.84
28.95
45
34.51
24.65
87
24.20
33.39
37. 47.02
35.09
Ves
BIO
29.79
41 22
33.56
26.06
45,80
33.06
24.98
25.
24.99
33.14
39.04
35.81
BIOES
31.72
36.31
32.85
28.95
44.37
34.72
30.21
26.64
28.22
37.38
39.02
38.15
Table 10: span detection.

---

Dataset
Tabels
Type names
AGE
age
BIOID
hiometric ID
Labels
Ty pe names
CITY
city
art- btwoscccssssrorrra
broadcast program
COUNTRY
country
art urt-iilmn
film
DATE
date
art-music
music
DEVICE
device
DOCTOR
doctor
art painting att-writennart
painting
EMAIL
email
writien
person-actorr
actor
FAX
fax
person-artistauutnn
author
HEALTHPLAN
health plan number
person athlete
athlete
HOSPITAL
hospital
person-dinector
director
12B2'14
IDNUM
ID number
person other
other person
LOCATION OTHER
location
person-politien polittiiiinn
politician
person scholar
scholar
MEDICALRECORD
medical record
pence-soolder
soldier
ORGANIZATION
organization
product airplane
sirplane
PATTENT
patient
product-car
car
PHONE
phone number
product food
food
PROFESSION
profession
product game
game
STATE
state
prodact-olber
other product ship
product- ship
STREET
street
product-softwan
software
URL
url
product
USERNAME
usemame
product-weapo
weapon
ZIP
zip code
otber astronomy thing
astronomy thing
other-award
award
PER
person
other biologything
biology thing
LOC
Incation
other chemicalthing
chemical thing
CONLL'03
ORG
organization
other-currency
currency
MISC
miscellancous
wcher-edtcattonaldeeee
educcational degree
abstract
abstract
other god
god
animal
animal
other-tanguage
language
other-law
law
event
event
Few-NERD
other-liviiee hiviniiggg
living thing
object
object
other medical
medical
organization
organization
building-airport
airport
GUM
person
person
building-hospiinn
bospital
place
place
huilding- hotel
plant
plant
building library
library
quantity
quantity
building other
other building
building-restarran
restaurant
substance
substance time
building sportsfucility
sports facility
time
bui miidiing- theater
theater
event attaac/bbttlee
attack battle
corporation
corporation
wwrrhni licaryconflict
war military conflict
creative-work
creative work
event-
event-disaster
election
disaster
IN
location
group
group
location
other
other event
person
person
event protest
protest
product
product
event-sportsevent
sports event
location bodiesofwater
bodies of
CARDINAL
cardinal
location GPE
geographical social
DATE
date
political cutlty island
location-island
EVENT
event
locution-mountain
mountain
FAC
fac
other
other location
geographiical social
location-park
park
GPE
political entity
location-roadde ridiion-rad/railwwg
roa railway highway transit company education
LANGUAGE
lunguage
hiighway/tansit
LAW
law
organization company
organizatio---dun -ducatitioooiii
Ontonotes
LOC
location
organization government
MONEY
money
/govemienntugency
government ageney
NORP
nationality religion
organization media/newspaper
media newspuper other organization
ORDINAL
ordinal
organization other
ORG
organization
organication-Pe onganiattion-poliicaapattt
politicalputt
political party religion
PERCENT
percent
show organization
PERSON
person
organization sportsleague
sports league sports team
PRODUCT
product
oeganization-sponn sporiisteam
QUANTITY
quantity
TIME
time
WORK_OK SORKKOF_AN OF_ART
work art
Table 11: Original labels and their corresponding natural-language-fome type names of Few-NERD.
Table 12: Original labels and their corresponding naturalllanguage-fomm type names of datasets under Domain Transfer settings.

---

Original Type Names
Synonym
Synonym
Synonym 3
broadcast program
television show
TV program
telecast
film
movie
motion picture
cinema
music
melody
tuncs
songs
other art painting
different art
alternative art
diverse art
artwork
canvas
picture
written art
literature
written work thespian
prose
actor
performer
artist
artist author
creative writer
author
wordsmith
athlete
sportsman wwoman
player
competitor
director
filmmaker
supervisor
manager
other person
someone else
another person lawmaker
another individual
politician
statesman/worann
public servant
scholar
academic
intellcctual
researcher
soldier
military personnel
serviceman/woman
trooper
airplane
aircraft
plane
jet
car
automobile
nourishment
farc
food
cuisine
nourishment
fure
game
sport
competition
match
other product
different product
alternative item boat application
various commodity
ship
vessel
craft
software
program
computer program railcar
train
locomotive
railway vehicle firearm
weapon
armament
arm
astronomy thing
celestial object
astronomical entity prize
heavenly body recognition
award
accolade
biology-thing
biological entity chemical substance
living organism compound
life form
chemical thing
element
currency
money
cash
legal tender
disease
illness
sickness
disorder certification
educational degree
academic qualification
diploma
god
deity
divine heing
higher power communication
language
tongue
speech
law
legislation
legal system
jurisprudence
living thing
organism
creature
being
medical
healthcare
medicinal
therapeutic
bodies of water
Waterways
aquatic features
lakes and rivers jurisdiction
geographical social political entity island
Territory
region
Isle
islct
key
mountain
Peak
summit
range
other location
spot
place
park
Garden
reserve
recreational area transportation network
road railway highway transit company
Route
thoroughfare
Corporation
firm
enterprise
education
Learning
schooling
instruction
government agency
Public body
administrative department journalism
authority
media newspaper
Press
news organization
other organization
Institution
establishment
association
political party
Political group
faction
party organization
religion
Faith
belief system
spirituality
show organization
Production company
entertainment group
performance troupe
sports league
Athletic association
sporting federation competitive squad
league organization
sports team
Athletic club
sporting roster
Table 13: Variant type names Few-NERD Intra setting.

---

Original Type Names
Synonym
Synonym 2
Synonym 3
cardinal
Primary
fundamental
principal
date
Day
time
appointment
event
Occasion
happening
function
fac
Facility
building
structure
geographical social political entity
Territory
region
jurisdiction
language
Tongue
spocch
communication
law
Regulation
rule
statute
location
Place
site
spot
money
Currency
funds
finances
nationality religion
Citizenship
faith
belief system ordered
ordinal
Sequential
numbered
organization
Institution
establishment proportion human
association
percent
Percentage
rate
person
Individual
character
product
Item
good
merchandise
quantity
Amount
volume
measure
time
Duration
period
interval
work of art
Artwork
creation
masterpiece
person
Individual
human being
somebody
location
Place
site
spot
organization
Institution
establishment
company
miscellaneous
Various
diverse
mixed
Table Variant type names for Domain Transfer setting. Here we show the type names the OntoNotes dataset and the CoNLL2003 dataset.


Reviews:
[{"review_id": "xlFJULBSFp", "reviewer": "Reviewer_WiN1", "summary": "", "questions": "In the reading of this paper, there are the following questions:\n1)It seems that on the basis of the mainstream span method, the paper only adds label names to inject information into the model, but does not carry out greater innovation in the method or model. The results of the experiment appear to be improved, but there is no good formula or theoretical basis for the effectiveness of the information added or removed.\n2)The article proposes to delete the type far from the type name. In the example Figure 1 given in the article, there is only one word between 1976 and California. If we want to delete the influence of 1976, the window distance is very small, which seems to be helpful only for this particular case, but for most of the samples, Whether it is appropriate, and whether it will cause some important boundary information to be missed. Because in most cases, we want to be able to detect more information from a long text.\n3)The paper wants to use type names to obtain richer semantics and prototype representation, but in contrast, the mainstream large models such as GPT contain richer semantics. What is the semantic advantage of the proposed type names compared with the big model? Besides, why does the paper not use the big model to get the semantics?\n4)The article's description of the type of deletion is too simple, and there is no detailed description of the basis and distance of deletion, so there is no way to judge the effectiveness of this step. The article seems to give the threshold value in Formula 11 but does not explain the value in the formula, indicating that the description is very vague.\n5)Regarding the fine-tuning proposed in the article, may I ask how to divide the support data of the test set during the fine-tuning process? In the case of 1shot, it cannot be divided and fine-tuned the support, which will lead to type loss. How the fine-tuning proposed in the article solves this problem is not explained in the article.\n6)The article argues that adding type names can be helpful for prototyping. In the result and analysis part of the article, only the influence on acc after adding the name is given, and the effect of the prototype is not intuitively seen. It seems that the article should give a vector diagram of the prototype to illustrate the correction or aggregation effect of the prototype.\n7) Is the model proposed in this paper an experiment based on the DecomposedMetaNER model? If so, is the classification of the training set, verification set, and test set the same as that of DecomposedMetaNER? Can you show more details of the processing", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We hope the following clarifications can address the reviewer’s concerns.\n> Q1. It seems that on the basis of the mainstream span method, the paper only adds label names to inject information into the model, but does not carry out greater innovation in the method or model. \\\nThe results of the experiment appear to be improved, but there is no good formula or theoretical basis for the effectiveness of the information added or removed.\n\n1\\. To the best of our knowledge, no related work has explicitly identified and addressed the issues of \"over-detected false spans\" and \"inaccurate and unstable prototypes\" in the context of few-shot NER. In other words, these are new problems for few-shot NER. In order to solve these two issues, we propose to inject type name information and design the type-aware span filtering strategy and the type-aware contrastive learning strategy, respectively.\n\nAlthough our approach may seem simple and straightforward, this does not necessarily imply a lack of innovation. \nFirstly, recognizing a new problem is always critical to the research community, and it might be more important than solving the problem itself. \nSecondly, our recognized two problems may appear in the few-shot scenarios of other NLP tasks, such as the few-shot event detection task (extracting the event trigger word first and then classifying the event type based on the trigger word).\nOur discoveries and solutions could provide some insights to the researchers when tackling these tasks, too.\n\n2\\. Early probabilistic models such as Semi-CRFs [1] had good formalized analysis. Recent data driven deep learning methods can achieve better performance yet often lacking the theoretical basis. To compensate for this shortcoming, for the added type name information, we conducted as much experimental analysis as possible. As can be seen, in Section 3.5, there are a large number of experiments with \"Synonyms\", \"Meaningless\" and \"Misleading\" variants of type names to demonstrate the effectiveness of type name injection.\n\n[1] Sarawagi S, Cohen W W. Semi-markov conditional random fields for information extraction. Advances in neural information processing systems, 2004, 17.\n\n> Q2. The article proposes to delete the type far from the type name. In the example Figure 1 given in the article, there is only one word between 1976 and California. If we want to delete the influence of 1976, the window distance is very small, which seems to be helpful only for this particular case, but for most of the samples, Whether it is appropriate, and whether it will cause some important boundary information to be missed. Because in most cases, we want to be able to detect more information from a long text.\n\nWe guess that the reviewer might have misunderstood the main reason why \"1976\" was misclassified as \"LOC\".\nThe reviewer may think that the misclassification of \"1976\" into \"LOC\" was influenced by the semantic of the nearby token \"California\". However, the error is mainly due to the bias brought by the domain gap in the few-shot scenario.\n\nFirst of all, we are focusing on the NER task setting with a fixed set of entity types (which is also the most common in its practical applications). For the example in Figure 1 (a) and (b), we only aim to extract entities of the \"ORG\" and \"LOC\" type. So even if \"1976\" is an entity of the \"DATE\" type, we don’t want to extract it.\n\nSecondly, assuming under a specific few-shot NER setting (Figure 2), the entity type sets of the training and test sets are {\"PER\", \"DATE\"} and {\"ORG\", \"LOC\"}, respectively. The model needs to be trained on the training set first, and then be fine-tuned using few support samples in the test set.\nSince entities of the \"DATE\" type have appeared in the training set, the span detection module will be trained to extract the span of the \"DATE\" type entity. During the inference on the test set, as shown in the Figure 1 (a), \"1976\" is extracted at the span detection stage. \nNote that **for this test set, our predefined entity type set does not include the \"DATE\" type**, i.e., we do not want to retain the entity span of \"1976\". Continuing to classify it would force it to be assigned an entity type of either \"ORG\" or \"LOC\" (as these are the only two types in this test set), resulting in an obviously incorrect classification result. Therefore, the type-aware span filtering strategy is designed to alleviate this problem.\n\n> Q3. The paper wants to use type names to obtain richer semantics and prototype representation, but in contrast, the mainstream large models such as GPT contain richer semantics. \\\nWhat is the semantic advantage of the proposed type names compared with the big model? Besides, why does the paper not use the big model to get the semantics? \n\n\n**Firstly**, we suppose the reviewer's concern is about \"using a larger model to get a richer semantics may perform better than incorporating type names\". However, the two main issues we've identified and tried to address, i.e., \"over-detected false spans\" and \"inaccurate and unstable prototypes\", are not caused by the lack of rich semantic information in the LM.\nInstead, they are caused by the bias from the domain gap and the scarcity of samples under few-shot settings, respectively.\n\nFor the issue of \"over-detected false spans\", the main reason is that at the entity span detection stage, the span information learned by the model from the source domain is inevitably transferred to the target domain. When the entity type sets of the source domain and target domain are different, some entity spans that only belong to the source domain type set rather than the target domain type set will be incorrectly extracted (this is also discussed in question 2 above). Therefore, this problem is due to the bias from the domain gap, not a lack of rich semantic information obtained through the LM.\n\nFor the issue of \"inaccurate and unstable prototypes\", the main reason is that the prototype under the few-shot settings is constructed with very few samples (1/5-shot), which may deviate from the actual category center in some scenarios (Figure 1c).\nThis would lead to a reduction in the model's performance (Section 3.6). Hence, even using a larger model to obtain the representation of entity words, there may still exist such a prototype bias caused by the scarcity of samples.\n\nFor these two reasons, we think it is unnecessary to conduct experiments with a big model just for obtaining richer semantics.\nTherefore, we only used the BERT model in our experiments, which is frequently adopted by community researchers, including all the baselines we compared, to validate the effectiveness of our proposed method.\n\n**Secondly**, we are not sure whether the reviewer's concern is about the necessity of small PLMs like BERT for this problem. If this is the case, the answer is YES because LLMs cannot perform better than fine-tuned SLMs, e.g., 5-shot + settings in our task, due to the lack of source-domain NER data. A newly published paper [1] in ACL 23 also get the similar conclusion with the help of a large corpus in the source domain.\n\n[1] Chen et al., 2023. Learning In-context Learning for Named Entity Recognition. ACL 2023, pages 13661–13675.\n\n\n> Q4. The article's description of the type of deletion is too simple, and there is no detailed description of the basis and distance of deletion, so there is no way to judge the effectiveness of this step. \\\nThe article seems to give the threshold value in Formula 11 but does not explain the value in the formula, indicating that the description is very vague.\n\nSorry for the confusion caused by the lack of detailed descriptions, which is mainly due to space constraints that prevent us from elaborating on the corresponding information in the main body.\n\n1. For detailed process of the type-aware span filtering (deletion), please refer to lines 20-31 of the algorithm in Appendix A.1. Per your suggestion, we will add a more detailed description of the type-aware span filtering strategy in the corresponding methodology section.\n\n2. The symbols used in Eq. 11 were mentioned in the previous paragraphs. Specifically, $e_i^s$ refers to the $i_{th}$ entity token in the support set (line 130), and the Map() function is for converting the original class labels into corresponding type names, e.g., \"PER\"->\"person\" (page 3, footnote 4). The main purpose of this formula is to obtain a type name-aware threshold using samples in the support set, which would be used for the false span filtering. We will add detailed descriptions of these symbols in Eq. 11 to make it more clear to read.\n\n> Q5. Regarding the fine-tuning proposed in the article, may I ask how to divide the support data of the test set during the fine-tuning process?\\\nIn the case of 1 shot, it cannot be divided and fine-tuned the support, which will lead to type loss. How the fine-tuning proposed in the article solves this problem is not explained in the article. \n\nWe suppose the reviewer's concern about \"how to divide the support set into a training set and a validation set\" during the fine-tuning process.\n\nIn fact, for the few-shot (1-shot or 5-shot) NER task, the number of samples available in the divided validation set is too limited to prevent the overfitting problem.\nThus, in our experiments, we do not divide the support set into a training set and a validation set, but use it as a whole for fine-tuning the model.\nTo address the overfitting problem, we employ a loss-based early-stopping strategy, which is a commonly used approach in previous methods like CONTaiNER [1] during the fine-tuning step.\n\nIn Appendix A.5, we provide specific implementation details. \nDuring the fine-tuning process of the span detection module and type classification module, we monitor the loss. \nIf the loss continues to rise for β times (where β is a hyperparameter), we stop the fine-tuning process to mitigate the risk of overfitting. We will clarify this in the methodology section to ensure clarity and avoid any potential confusion.\n\n[1] Das et al., 2022. CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning. ACL (1) 2022: 6338-6353.\n\n> Q6. The article argues that adding type names can be helpful for prototyping. In the result and analysis part of the article, only the influence on acc after adding the name is given, and the effect of the prototype is not intuitively seen. It seems that the article should give a vector diagram of the prototype to illustrate the correction or aggregation effect of the prototype.\n\nThanks for the suggestion. \nWe conducted visualization experiments as soon as we received comments from the reviewer, i.e., we visualized the distribution of prototypes and test samples without and with type names.\n\nThe visualization results show that the entity prototypes of \"ORG\" and \"MISC\" types are obviously deviated from the corresponding actual sample centers, which is due to the scarcity of samples in the given support set. In contrast, the type-aware entity prototypes of the \"ORG\" and \"MISC\" types, which incorporate type name semantics, are able to alleviate the problem of prototype bias caused by the scarcity of samples. \n\nBut, unfortunately, due to the format restriction of the rebuttal, we are unable to show the images here to the reviewer.\nWe will add this visualization analysis in the paper to intuitively show the effect of type names on the prototype correction.\n\n> Q7. Is the model proposed in this paper an experiment based on the DecomposedMetaNER model? If so, is the classification of the training set, verification set, and test set the same as that of DecomposedMetaNER? Can you show more details of the processing?\n\n\nNo, the experiment of our proposed model is not based on that of the DecomposedMetaNER [1] model.\nThe only similarity between our approach and DecomposedMetaNER [1] is that it is also a decomposed framework, i.e., entity spans are extracted first and then entity types are categorized, which is common in NER [2,3].\n\nBesides, the division of the training set, validation set, and test set is the same as that of DecomposedMetaNER [1]. \nDue to the space limit, we can only place a detailed description of the dataset in Appendix A.3. We apologize for the inconvenience.\n\n1. We adopt the Few-NERD dataset made public by Ding et al. (2021) [4], which discloses the division of training/validation/test sets and few-shot sampling results (link: https://ningding97.github.io/fewnerd/).\n2. For the Domain Transfer dataset, we use the division and sampling data made public by Das et al. (2022) [5] (link: https://github.com/psunlpgroup/CONTaiNER).\n\nOur TadNER and all the compared baselines, e.g., DecomposedMetaNER [1], conduct experiments using these two publicly available data divisions and samplings.\n\n[1] Ma et al., 2022. Decomposed Meta-Learning for Few-Shot Named Entity Recognition. ACL 2022, pages 1584–1596.\\\n[2] Shen et al., 2021. Locate and Label: A Two-stage Identifier for Nested Named Entity Recognition. ACL 2021, pages 2782–2794.\\\n[3] Zhang et al., 2022. Exploring Modular Task Decomposition in Cross-domain Named Entity Recognition. SIGIR 2022. Association for Computing Machinery, New York, NY, USA, 301–311.\\\n[4] Ding et al., 2021. Few-NERD: A Few-shot Named Entity Recognition Dataset. ACL-IJCNLP 2021.\\\n[5] Das et al., 2022. CONTaiNER: Few-Shot Named Entity Recognition via Contrastive Learning. ACL (1) 2022: 6338-6353.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates a clear structure and presents promising experimental results.", "reasons_to_reject": "1) The overall logic of the paper is relatively clear, but this paper lacks sufficient novelty and appears to be a limited improvement over the DecomposedMetaNER paper.\n2) In the introduction of part of the work, the article lacks details and does not elaborate clearly. \nThese issues will be elaborated on in the following question and answer section.", "questions_to_authors": "In the reading of this paper, there are the following questions:\n1)It seems that on the basis of the mainstream span method, the paper only adds label names to inject information into the model, but does not carry out greater innovation in the method or model. The results of the experiment appear to be improved, but there is no good formula or theoretical basis for the effectiveness of the information added or removed.\n2)The article proposes to delete the type far from the type name. In the example Figure 1 given in the article, there is only one word between 1976 and California. If we want to delete the influence of 1976, the window distance is very small, which seems to be helpful only for this particular case, but for most of the samples, Whether it is appropriate, and whether it will cause some important boundary information to be missed. Because in most cases, we want to be able to detect more information from a long text.\n3)The paper wants to use type names to obtain richer semantics and prototype representation, but in contrast, the mainstream large models such as GPT contain richer semantics. What is the semantic advantage of the proposed type names compared with the big model? Besides, why does the paper not use the big model to get the semantics?\n4)The article's description of the type of deletion is too simple, and there is no detailed description of the basis and distance of deletion, so there is no way to judge the effectiveness of this step. The article seems to give the threshold value in Formula 11 but does not explain the value in the formula, indicating that the description is very vague.\n5)Regarding the fine-tuning proposed in the article, may I ask how to divide the support data of the test set during the fine-tuning process? In the case of 1shot, it cannot be divided and fine-tuned the support, which will lead to type loss. How the fine-tuning proposed in the article solves this problem is not explained in the article.\n6)The article argues that adding type names can be helpful for prototyping. In the result and analysis part of the article, only the influence on acc after adding the name is given, and the effect of the prototype is not intuitively seen. It seems that the article should give a vector diagram of the prototype to illustrate the correction or aggregation effect of the prototype.\n7) Is the model proposed in this paper an experiment based on the DecomposedMetaNER model? If so, is the classification of the training set, verification set, and test set the same as that of DecomposedMetaNER? Can you show more details of the processing", "paper_topic_and_main_contributions": "The paper proposes a novel approach called TadNER for few-shot named entity recognition (NER), which aims to overcome two main challenges in this task: over-detected false spans and inaccurate and unstable prototypes. TadNER incorporates a type-aware span filtering strategy and a type-aware contrastive learning strategy.\n\n1)The type-aware span filtering strategy is designed to filter out false spans by identifying and removing those that are semantically far away from type names. This is achieved by considering the distance between each span and the nearest type name and filtering out those that exceed a predefined threshold.\n\n2)The type-aware contrastive learning strategy is designed to construct more accurate and stable prototypes by leveraging type names and support samples as references. This is achieved by jointly optimizing a contrastive loss function that encourages the prototypes to be close to their corresponding support samples and far from other samples in the same type or other types.\n\n3)The proposed TadNER approach is evaluated on various benchmarks, and the results demonstrate that it achieves a new state-of-the-art performance, outperforming existing approaches.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OEQKIE9Nax", "reviewer": "Reviewer_RZMJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We hope the following clarifications can address the reviewer’s concerns.\n> Q1. In some parts, the paper is really hard to follow. In method section, the only real examples are presented in the (small) picture, and it is hard to connect each formula and description to the phases in the image. In general, the LaTeX formatting should be more tidy (see for example Section 2.1.1).\n\nSorry for the inconvenience, we will enrich the caption in Figure 2 so that the readers can connect the formula with the real example in the picture more clearly.\n\n> Typos: There are a lot of errors due to the wrong use of equation in LaTeX (main of them can be found in Section 2.1.1).\n\nThanks for your suggestion! We will fix them per your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The goal of the paper is clear, and the approach seems promising, although it is an application of ideas that already exist for the same kind of task. The use of a big number of baselines and the strong evaluation proposed bring robustness to the whole process.\nThe most interesting part of the study, in my opinion, is the semantic connection between the transformers and the labels (as described in Appendix A.8).", "reasons_to_reject": "In some parts, the paper is really hard to follow. In method section, the only real examples are presented in the (small) picture, and it is hard to connect each formula and description to the phases in the image.\nIn general, the LaTeX formatting should be more tidy (see for example Section 2.1.1).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes TadNER, a type-aware decomposed framework for named-entity recognition (NER). Its main goal is the enhancement of performance in few-shot NER, i.e. the extension of the set of spans and labels using only few examples.\nThe authors deal with the problem by two points of view: (i) filter out spans semantically far from the accepted type names; (ii) build stable prototypes of labels, to avoid deviation form the class centers.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a lot of errors due to the wrong use of equation in LaTeX (main of them can be found in Section 2.1.1)."}, "extra_text": {}}, {"review_id": "5OSiM9ZZ5o", "reviewer": "Reviewer_NUFY", "summary": "", "questions": " - Q1: In Table 2, where did you get the scores marked as \"from Ma et. al (2022c)\" from? Maybe I'm mistaken, but I can not find these values in the publication: https://aclanthology.org/2022.findings-acl.124/\n - Q2: How is fθ2 defined? Is it really the same as fθ1? But it looks like Map(yi) = t′i can be a token sequence (some of the mapped types in the appendix consist of multiple words). But in this case, equation (4) seems to only work, if fθ1 maps to R^r... in section 2.1.2, it is said that individual entity tokens ei are embedded with fθ2. However, in \"2.2.2 Type-Aware Span Filtering\" it is said that spans are filtered. Does the filtering happen token-wise? Or are all tokens of one span embedded at once and the result is then filtered (same  for the \"Inference\" section, in both sections you talk about spans)? But if fθ2 takes really a token sequence as input, Figure 2 is very misleading because \"Barack Obama\" is split into two training instances ((\"Barack\", \"person\") o (\"person\", \"Barack\")) and ((\"Obama\", \"person\" o (\"person\", \"Obama\")) where I had just assumed ((\"Barack Obama\", \"person\") o (\"person\", \"Barack Obama\")). \n - Q3: Since your method relies on useful label names: What was the effort to create the respective label name mappings? Could you give some points that this is negligible?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We hope the following clarifications can address the reviewer’s concerns.\n> Q1. In Table 2, where did you get the scores marked as \"from Ma et. al (2022c)\" from? Maybe I'm mistaken, but I can not find these values in the publication: https://aclanthology.org/2022.findings-acl.124/\n\nThis problem is caused by the different versions of the Few-NERD dataset [1] (corresponding to arXiv-V5 and arXiv-V6, respectively).\nOn the publicly available leaderboard (link: https://ningding97.github.io/fewnerd/) by the Few-NERD authors, the Few-NERD arXiv-V5 data was deleted by the authors and was replaced with the Few-NERD arXiv-V6 version when we conducted our experiements, so we had to use the Few-NERD arXiv-V6 version.\n\nResults in the camera ready version of Ma et. al's (2022c) [2] paper were obtained using the Few-NERD arXiv-V5 data.\nFor the convenience of comparison with their results, Ma et. al (2022c) [2] reproduced the results using the Few-NERD arXiv-V6 data and published the results on Github. The link is: https://github.com/microsoft/vert-papers/tree/master/papers/DecomposedMetaNER#few-nerd-arxiv-v6-version, which is also the source of results in our paper which are marked as \"from Ma et. al (2022c)\".\n\nDue to the space limit, we have to place this explanation in footnote 7 of Appendix A.3.\nTo avoid potential confusion for the readers, we will include this explanation to the main body of the paper.\n\n[1] Ding et al., 2021. Few-NERD: A Few-shot Named Entity Recognition Dataset. ACL-IJCNLP 2021.\\\n[2] Ma et al., 2022. Decomposed Meta-Learning for Few-Shot Named Entity Recognition. ACL 2022, pages 1584–1596.\n\n\n> Q2. How is $f_{θ_2}$ defined? Is it really the same as $f_{θ_1}$? \\\nBut it looks like Map($y_i$) = $t_{i}^{'}$ can be a token sequence (some of the mapped types in the appendix consist of multiple words). \\\nBut in this case, equation (4) seems to only work, if $f_{θ_1}$ maps to $R^r$... in section 2.1.2, it is said that individual entity tokens ei are embedded with $f_{θ_2}$. \\\nHowever, in \"2.2.2 Type-Aware Span Filtering\" it is said that spans are filtered. Does the filtering happen token-wise? Or are all tokens of one span embedded at once and the result is then filtered (same for the \"Inference\" section, in both sections you talk about spans)? \\\nBut if fθ2 takes really a token sequence as input, Figure 2 is very misleading because \"Barack Obama\" is split into two training instances ((\"Barack\", \"person\") o (\"person\", \"Barack\")) and (\"Obama\", \"person\" o (\"person\", \"Obama\")) where I had just assumed ((\"Barack Obama\", \n\"person\") o (\"person\", \"Barack Obama\")).\n\nSorry for the confusion due to the unclear formalization of $f_{θ_1}()$ and $f_{θ_2}()$ in the paper. \\\nGenerally speaking, $f_{θ_1}()$ and $f_{θ_2}()$ are the same context encoder with individual parameters. \nHere, we will formalize $f_{θ_1}()$ and $f_{θ_2}()$ in detail and explain the reviewer's questions one by one. For simplicity of description, let's collectively refer to $f_{θ_1}()$ and $f_{θ_2}()$ as $f_{θ}()$.\n\nFor the input $X$, the output is $V=f_θ(X)$. \n1. If $X$ is a token, then $V$ is the context embedding vector of this token. \n2. If $X$ is a word or span composed of N tokens, i.e., the type name \"nationality religion\" or entity span \"Barack Obama\", $V$ is the average of the context embedding vectors for these N tokens. \n3. If $X$ is a token sequence composed of M tokens $[x_1,…,x_M]$, like the sequence [\"Barack\", \"Obama\", \"was\",…, \"1961\"] in Figure 2, then V is the sequence of context embeddings $[emb_1,…,emb_M]$=[$f_{θ}$(Barack),…,$f_{θ}$(1961)], where the dimension of $emb_1,…,emb_M$ is the hidden layer dimension of the encoder $f_θ()$.\n\nBased on this formalization, below are our explanations for your questions:\n\n1\\. \"Does the filtering happen token-wise? Or are all tokens of one span embedded at once and the result is then filtered.\"  \\\n**Explanation**: The filtering process is span-based. The entire span is inputted to the encoder $f_{θ_2}()$ as a whole (corresponding to the second type of input to $f_{θ}()$ in the formalization above).\n\n2\\. \"Figure 2 is very misleading because 'Barack Obama' is split into two training instances\"\\\n**Explanation**: The input to $f_{θ_2}()$ in Figure 2 is a token sequence, so its output is an embedding sequence where each token $e_i$ corresponds to an output embedding $f_{θ_2}(e_i)$ (corresponding to the third type of input to $f_{θ}()$ in the formalization above).\n\nTo avoid any potential confusion, we will add this formalization of the encoders $f_{θ_1}()$ and $f_{θ_2}()$ into the methodology section.\n\n> Q3. Since your method relies on useful label names: What was the effort to create the respective label name mappings? Could you give some points that this is negligible?\n\nThe type name mappings for all the datasets used in our experiments can be seen in Tables 11 and 12 of the Appendix A.8. \\\nAs shown in Table 11, for the Few-NERD dataset, we directly use the fine-grained part of the original label as the corresponding type name, such as \"art-film\"->\"film\".\nAs shown in Table 12, for the datasets under the Domain Transfer settings, we convert the majority of labels into their most direct natural language form of type names, such as \"PER\"->\"person\" and \"AGE\"->\"age\".\\\nTherefore, the additional effort required to construct the mappings is almost negligible.\n\n\n> Typos: line 127 \"datset\"; line 938 \"doamin\"\n\nThanks a lot for your careful reading, we will fix them per your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose an effective method for few-shot NER that significantly outperforms previous approaches, especially for the very few shot setting. They explain their method quite well and conduct reasonable experiments to show the effectiveness of their method.", "reasons_to_reject": " - see Q1 (re-reported scores seem to be lower than in the cited paper)\n - see Q2 (definition of fθ2 is unclear)", "questions_to_authors": " - Q1: In Table 2, where did you get the scores marked as \"from Ma et. al (2022c)\" from? Maybe I'm mistaken, but I can not find these values in the publication: https://aclanthology.org/2022.findings-acl.124/\n - Q2: How is fθ2 defined? Is it really the same as fθ1? But it looks like Map(yi) = t′i can be a token sequence (some of the mapped types in the appendix consist of multiple words). But in this case, equation (4) seems to only work, if fθ1 maps to R^r... in section 2.1.2, it is said that individual entity tokens ei are embedded with fθ2. However, in \"2.2.2 Type-Aware Span Filtering\" it is said that spans are filtered. Does the filtering happen token-wise? Or are all tokens of one span embedded at once and the result is then filtered (same  for the \"Inference\" section, in both sections you talk about spans)? But if fθ2 takes really a token sequence as input, Figure 2 is very misleading because \"Barack Obama\" is split into two training instances ((\"Barack\", \"person\") o (\"person\", \"Barack\")) and ((\"Obama\", \"person\" o (\"person\", \"Obama\")) where I had just assumed ((\"Barack Obama\", \"person\") o (\"person\", \"Barack Obama\")). \n - Q3: Since your method relies on useful label names: What was the effort to create the respective label name mappings? Could you give some points that this is negligible?", "paper_topic_and_main_contributions": "The authors propose a method for few-shot NER build on top of two-stage methods where first span detection is performed followed by span classification. However, they improve over recent methods by incorporating: (1) an effective way of exploiting the text representations of the labels by embedding them into a common space with the entity tokens, and (2) using these embeddings to filter out false entity span candidates that are too dissimilar to the provided labels. Primarily, they construct entity prototype representations by embedding the entity tokens and the mapped class labels with the same encoder and concatenate them in two variants (i.e. in both directions: entity-label and label-entity). Then, the encoder is trained with a contrastive learning objective that pulls these representations together if they have the same label and pushes them away from each other, if this is not the case. \n\nThe authors show that this approach is very effective by conducting few-shot experiments on several datasets (Few-NERD and others) where they outperform previous work (one-stage and two-stage) on a large margin, especially in the realm of very few (1 ~ 2) training examples. Furthermore, they include an extensive ablation study (effect of: span filtering, incorporating the label text, fine-tuning of the encoders for each stage on the support set) and a quantitative as well as a qualitative error analysis.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": " - typo: line 127 \"datset\"\n - type: line 938 \"doamin\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This article primarily enhances few-shot named entity recognition tasks by utilizing label information and span filtering to address two key challenges: the issue of over-detected false spans and the problem of inaccurate and unstable prototypes. The proposed method demonstrates strong performance in the realm of few-shot named entity recognition tasks, as substantiated by a comprehensive array of experiments. After serious discussion and consideration, we appreciate this solid work but think that the innovation appears somewhat limited. Furthermore, the explanation of filtering false spans is unclear, making it difficult to determine the positive impact of label information on the model and whether it filters out useful knowledge.

Overall, we appreciate both the efforts of reviewers and authors. This work is ready to publish, and the only concern is the novelty. We hope the authors can continue to improve the paper according to the comments and your response.
Decision: Accept-Findings