Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learning
Authors: Sheng Zhang
Hao Cheng
Jianfeng Gao
Hoifung Poon
Venue: ICLR 2023 poster (2023)
Keywords: Named Entity Recognition
NER
Bi-Encoder
Contrastive Learning

Abstract: We present a bi-encoder framework for named entity recognition (NER), which applies contrastive learning to map candidate text spans and entity types into the same vector representation space. Prior work predominantly approaches NER as sequence labeling or span classification. We instead frame NER as a representation learning problem that maximizes the similarity between the vector representations of an entity mention and its type. This makes it easy to handle nested and flat NER alike, and can better leverage noisy self-supervision signals. A major challenge to this bi-encoder formulation for NER lies in separating non-entity spans from entity mentions. Instead of explicitly labeling all non-entity spans as the same class $\texttt{Outside}$ ($\texttt{O}$) as in most prior methods, we introduce a novel dynamic thresholding loss, learned in conjunction with the standard contrastive loss. Experiments show that our method performs well in both supervised and distantly supervised settings, for nested and flat NER alike, establishing new state of the art across standard datasets in the general domain (e.g., ACE2004, ACE2005, CoNLL2003) and high-value verticals such as biomedicine (e.g., GENIA, NCBI, BC5CDR, JNLPBA). We release the code at https://github.com/microsoft/binder.

Published as a conference paper at ICLR 2023
OPTIMIZING BI-ENCODER FOR NAMED ENTITY RECOGNITION VIA CONTRASTIVE LEARNING
Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon
Microsoft Research
ABSTRACT
We present bi-encoder framework for named entity recognition (NER), which applics contrastive learning to map candidate text spans and entity types into the same vector representation space. Prior work predominantly approaches NER sequence labeling or span classification. We instead frame NER as a represen- tation learning problem that maximizes the similarity between the vector repre- sentations of an entity mention and its type. This makes it easy to handle nested and flat NER alike, and can better leverage noisy self-supervision signals. A ma- jor challenge to this bi-encoder formulation for NER lies in separating non-entity spans from entity mentions. Instead of explicitly labeling all non-entity spans as the same class Out s ide (0) as in most prior methods, we introduce a novel dy- namic thresholding loss, learned conjunction with the standard contrastive loss. Experiments show that our method performs well in both supervised and distantly supervised settings, for nested and flat NER alike, establishing new state of the art across standard datasets in the general domain (e.g., ACE2004, ACE2005, CoNLL2003) and high-value verticals such as biomedicine (e.g.. GENIA, NCBI, BC5CDR, JNLPBA). We release the code at github..om//mierostttbnnnrr
INTRODUCTION
Named entity recognition (NER) the task of identifying text spans associated with named entities and classifying them into predefined set of entity types such person, location, etc. As fun- damental component in information extraction systems (Nadeau & Sekine| 2007). NER has been shown to be of benefit to various downstream tasks such as relation extraction (Mintz et al. 2009), coreference resolution (Chang et al.) 2013), and fine-grained opinion mining (Choi et al. 2006).
Inspired by recent success in open- question answering (Karpukhin et al. 2020) and entity linking (Wu et al. 2020 Zhang et al. 2021a), we propose an efficient BI-encoder for NameD Entity Recognition (BINDER). Our model employs two encoders separately map text and entity types into the same vector space, and it is able to reuse the vector representations text for different entity types (or vice versa), resulting in a faster training and inference speed. Based on the bi-encoder representations, we propose a unified contrastive learning framework for NER, which enables us to overcome the limitations of popular NER formulations (shown in Figure T]. such as difficulty in handling nested NER with sequence labeling (Chiu & Nichols 2016; Ma & Hovy (2016), complex learning and inference for span- classification Yuu al. [020 Fu et al. (202)) ). and challenges in learning with noisy supervision (Straková et al., 2019; Yan et al. 2021 Through contrastive learning, we encourage the representation of entity types to be similar with the corresponding entity spans, and to be dissimilar with that of other text spans. Additionally, existing work labels all non- entity tokens or spans as the same class Out s id (0), which can introduce false negatives when the training data is partially annotated (Das al., 2022 Aly et al. 2021). We instead introduce novel dynamic thresholding loss in contrastive learning, which learns candidate-speciff dynamic thresholds to distinguish entity spans from non-entity ones.
To the best of our knowledge, we are the first to optimize bi-encoder for NER via contrastive learn- ing. We conduct extensive experiments evaluate our method in both supervised and distantly
Das et al. (2022) applies contrastive learning for NER in few-shot setting. In this paper, we focus on supervised NER and distantly supervised NER.

---

Published as a conference paper at ICLR 2023
Vector Space
Nested NER
Noisy Supervision
Fast Speed
CJip
Acme
Sequence laheling
X
x
[CLS]
b bought
Span-based
X
classification
Seq-to-seq
Linear Layer
Linear Layer
generation
Entity Type Encoder
Text Encoder
BINDER
(Ours)
[CLS] "Pesson" Description [SEP]
[CLS] Jim bought 300 shares of Acme Corp [SEP]
Figure 1: Left: The architecture of BINDER. The entity type and text encoder are isomorphic and fully decoupled Transformer models. In the vector space, the anchor point (O) represents the special token [CLS] ] from the entity type encoder. Through contrastive learning, we maximize the similarity between the anchor and the positive token (OJim), and minimize the similarity between the anchor and negative tokens. The dotted gray circle (delimited by the similarity between the anchor and O [CLS] from the text encoder) represents a threshold that separates entity tokens from non-entity tokens. To reduce clutter, data points that represent possible spans from the imput text not shown. Right: We compare BINDER with existing solutions for NER on three dimensions: 1) whether it can be applied to nested NER without special handling; 2) whether it can be trained using noisy supervision without special handling; 3) whether it has a fast training and inference speed.
supervised settings. Experiments demonstrate that method achieves the state of the art wide range of NER datasets, covering both general and biomedical domains. In supervised NER, compared to the previous best results, our method obtains 2.4%-2.9% absolute improvement in FI on standard nested NER datasets such as ACE2004 and ACE2005, and a 1.2%-1 22%-1.9% absolute im- provement on standard flat NER datasets such as BC5-chem, BC5-disease and NCBL. In distantly supervised NER, our method obtains a 1.5% absolute improvement in F1 on the BC5CDR dataset. We further study the impact of various choices of components our method, and conduct break- down analysis at entity type level and token level, which reveals potential growth opportunities.
METHOD
In this section, we present the design of BINDER, novel architecture for NER tasks. As our
model is built upon bi-encoder framework, we first provide the necessary background for encoding both entity types and text using the Transformer-base (Vaswani et al. (017) bi-encoder. Then, we discuss our ways deriving entity type and individual mention span representations using
embedding output from the bi-encoder. Based on that, introduce two types of contrastive learning objectives for NER using the token and span-level similarity respectively.
2.1 BI-ENCODER FOR NER
The overall architecture of BINDER is shown in Figure Our model is built upon a bi-encoder architecture which has been mostly explored for dense retrieval (Karpukhin al. (020). Following the recent work, our bi-encoder also consists of two isomorphic and fully decoupled Transformer models (Vaswani et al., 2017), iee. an entity type encoder and a text encoder. For NER tasks, we consider two types of inputs, entity type descriptions and text to detect named entities. At the high level, the entity type encoder produces type representations for each entity interests (e.g. person in Figure 1) and the text encoder outputs representations for each input token in the given text where named entities are potentially mentioned (e.g. Jim in Figure I). Then, we enumerate all span candidates based on corresponding token representations and match them with each entity type in the vector space. As shown in Figure I we maximize the similarity between the entity type and the positive spans, and minimize the similarity of negative spans.
2

---

Published as a conference paper at ICLR 2023
We first formally discuss encoding both inputs using a pretrained Transformer model, BERT (De- vlin et al,, 2019 Specifically, we use 21,..., ZN to denote an input sequence of length N. When using BERT, there is prepended token [CLS] and appended token (SEP) for all in- put sequences, i.e. [CLS], (11,,,,.1....... N [SEP]. Then the output is a sequence of hidden states h [cus), h1,.... ,,,, [[ssp] IRd from the last BERT layer for each input token, where d is the hid- den dimension. Note that as [CLS] is always in the beginning, ho and h cLaj are interchangeable here. Based on this, we then discuss the way of computing entity type and text token embeddings, which are the basic building blocks for deriving our NER constrative learning objectives later.
Entity Type Embeddings The goal of entity type encoder is produce entity type embeddings that serve as anchors in the vector space for contrastive learning. In this work, we focus on predefined set of entity types & = ...... } where each entity type has one or multiple natural language descriptions. The natural language description can be formal textual definitions based on the dataset annotation guideline or Wikipedia, and prototypical instances where a target type of named entities are mentioned. For simplicity, the discussion proceeds with one description per type and we use Eg to denote sequence of tokens for the k-th entity type description. For a given entity type Eg, use BERT the entity type encoder (BERT") and add an additional linear projection to compute corresponding entity type embeddings in the following way:
h Fg [cLs] BERT (Fx);
es Linear (h (hfots));
where Linear is a learnable linear layer and C₁ E Rd is the vector representation for Eg.
Text Token Embeddings Instead of using [CLS] embeddinng done in the recent bi-encoder work for entity retrieval (Wu et al. 2020), we consider using text token embeddings as the basic unit for computing similarity with entity span embeddings. As there are multiple potential named entities not known as a prior in the input, naively using special markers (Wu et al. (2020) incurs huge computation overhead for NER. Similar to the entity type embeddings, we again use BERT as the text encoder (BERT")) and simply use the final hidden states as the basic text token representations
hf.... hy BERT" (II....,IN).
(3)
2.2 CONTRASTIVE LEARNING FOR NER
Based on the entity type embeddings and text token embeddings discussed above, we then introduce two different contrastive learning objectives for NER in this part. Here, assume span (i,j) is contiguous sequence of tokens in the input text with a start token in position and an end token in position j. Throughout this work, we use the similarity function, sim(-,-) cos(rr) where 7 is a scalar parameter.
As shown in Figure the overall goal of NER constrastive learning is to push the entity mention span representations close to their corresponding entity type embeddings (positive) and far away from irrelevant types (negative) in vector space, e.g. Person closer to Jim but away from Acme. To achieve that, we propose a multi-objective formulation consisting of two objectives based on span and token embedding spaces respectively.
Span-based Objective
We derive the vector representation for span (i.j) as below:
S,j = Linear (h?
+ h
+ D(j -)),
(4)
where h," h. are text token embeddings (Equation 3p. st., E Rd. Linear is learnable linear
layer, @ indicates the vector concatenation, D(j- i) E T is the (j - f)-th row of learnable span
width embedding matrix D E R. NNx, Based on this, the span-based infoNCE (Oord et al., 2018) can be defined as
exp(sim(si.j- ex))
l Espan log Exes; Usi, exp(sim(s'.e) ,cx))
(S)
AAthhough different BERT variants are considered later in experiments, they all follow the same way of encoding discussed here. Here, leave special tokens for simplicity.
3

---

Published as a conference paper at ICLR 2023
where the span (i, belongs entity type Ek. Sj is the set of negative spans that are all possible spans from the input text, excluding entity spans from Ek, and ey is the entity type embedding.
Position-based Objective One limitation of the span-based objective is that it penalizes all neg- ative spans in the same way, even if they are partially correct spans, e.g., spans that have the same start or end token with the gold entity span. Intuitively, it is more desirable to predict partially correct spans than completely wrong spans. Therefore, we propose additional position-based con- trastive learning objectives. Specifically, we compute two additional entity type embeddings for Eg by using additional linear layers, d Linear (Ct.s)
E, (ct.s) ) c = Linear (h where ck. C et
the type embeddings for the start and end positions respectively, h h [CLs] is from the entity type encoder (Equation 1). Accordingly, we can use two addtional linear layers to compute the corre- sponding token embeddings for the start and end tokens respectively, u, = Linear vn = Lineary (h,),, where h is the text token embeddings (Equation 3). Using eß, eg as anchors, we then define two position-basee based objectives via contrastive learning:
exp(sim(u:; e
Extur log
(6)
En um.
Um. exp(sim(u',c)) c2)
log exp(sim(v, (i.sim((,,e.))
fnnd
Ever, uv, exp(sim(v' e0)
where Ug V are two sets of positions in the input text that do not belong to the start/end of any span of entity type k. Compared with Equation 5J the main difference of position-basee objectives comes from the corresponding negative sets where start and end positions are independent of each other. In other words, the position- based objectives can potentially help the model to make better start and end position predictions.
Thresholding for Non-Entity Cases Although the contrastive learning objectives defined above can effectively push the positive spans close to their corresponding entity type in vector space, might be problematic for our model at test time to decide how close span should be before it can be predicted as positive. In other words, the model is not able to separate entity spans from non- entity spans properly. To address this issue, we use the similarity between the special token [CLS] and the entity type as a dynamic threshold (as shown in Figure (T). Intuitively, the representation of [CLS] reads the entire input text and summarizes the contextual information, which could make it a good choice to estimate the threshold to separate entity spans from non-entity spans. We compare
with several other thresholding choices in $4
To learn thresholding, we extend the original contrastive learning objectives with extra adaptive
learning objectives for non-entity cases. Specifically, for the start loss (Bquation 6) the augmented start loss pf atar is defined as
Ctort Blotart (1 (1- log exp(sim(uress: cB)
(8)
Ewer, exp(sim(u',ee))
An augmented end loss *+ can be defined similar fashion. For the span loss Equation 5), we use the span embedding So,u for deriving the augmented span loss Eppn
exp(sim(So.0- 0+))
g+ span = Blapan - (1 - 3) log Eves; exp(sim(s' ex,))
Note that we use a single scalar hyperparameter /3 for balancing the adaptive thresholding learning and original contrastive learning for all three cases.
Training Finally, we consider multi-task contrastive formulation by combing three augmented contrastive learning discussed above, leading our overall training objective
aft + 7etn + Xe+ Altpan
(10)
where o, 7, À are all scalar hyperparameters.
4

---

Published as a conference paper at ICLR 2023
Inference Strategy During inference, enumerate all possible spans that are less than the length of and compute three similarity scores based on the start/end/span cases for each entity type. We consider two prediction strategies, joint position-span and span-only predictions. In the joint position-span case, for entity type Eg., we prune out spans (i.j) that have either start or end similar scores lower than the learned threshold, i.e. sim(u; U; ek) < +simmu , or sim(v,, e < sim(v sim(V eg)) Then, only those spans with span similarity scores higher than the span threshold are predicted positive ones i.e. sim(s..) ex) sim(So.o. ex). For the span-only strategy, we just rely on the span similarity score and keep all qualified spans as final predictions. As shown later in experiments ($)) we find the span-only inference is more effective, because the joint inference is more likely to be affected by annotation artifacts. The full inference algorithm is summarized in Appendix A.5
EXPERIMENTS
We evaluate method in both supervised and distantly supervised settings. The implementation details of our method are deseribed in Appendix A.4
Evaluation Metrics We follow the standard evaluation protocol and use miero F1: predicted entity span is considered correct if its span boundaries and the predicted entity type are both correct.
Datasets In the supervised setting. we evaluate our method on both nested and flat NER. For nested NER, we consider ACE2004, ACE2005, and GENIA (Kim et al., 2003). ACE2004 and ACE2005 are collected from general domains (e-g.. news and web). We follow Luan et al. (2018) to split ACE2004 into 5 folds, and ACE2005 into train, development and test sets GENIA is from the biomedical domain. We use its v3.0.2 corpus and follow Finkel & Manning (2099 and Lu & Roth (2015) to split it into 10%//0%//19% train/dev/test splits. For flat NER, we consider CONEL22003 (Toog Kim Sang & De Meulder 2003) as well as five biomedical NER datasets from the BLURB benchmark (Gu et al. 2021): BC5 CCC--chem/disease (Li et 2016), NCBI (Dogan al.,
2014). BC2GM (Smith et al.] 2008), and JNLPBA (Collier & Kim 2004). Preprocessing and splits
follow Gu al. 2222). Appendix A.6 reports
reports the dataset statistics.
In the distantly supervised setting, we consider BC5CDR (Li et al. 2016). It consists of 1 1,500 articles annotated with 15,935 chemical and 12,852 disease mentions. We use the standard train, development, and test splits, In the train and development sets, we discard all human annotations and only keep the unlabeled articles. Their distant labels are generated using exact string matching On the training set, the distant labels have high precision (97.99 97.99 for chemicals, and 98. 16.334 % for diseases), but low recall (63.14% for chemicals, and 46.73% for diseases).
against a dictionary released by Shang et al. 2018 4
Supervised NER Results Table presents the comparison of our method with all previous meth- ods evaluated on three nested NER datasets, ACE2004, ACE2005, and GENIA. We report precision, recall, and F1. As is shown, our method achieves the state of the art performance on all three datasets. On ACE2004 and ACE2005, it outperforms all previous methods with 89.7% and 90.0% F1. Par- ticularly, in comparison with the previous best method (Tan et al., 2021), our method significantly improves F1 by the absolute points of +2.4% and +2.9% respectively. On GENIA, our method ad- vances the previous state of the art (Yu et al. 2020) by +0.3% FI. Note that the previous methods are built on top of different encoders, e.g., LSTM. BaRT- BaRT-large, and T5-base. We also report our method using BERT- encoder, which even outperforms the previous methods that use a larger encoder of BERT-large (Tan et al.] (02T) or BART-large (Yan et al.) (021). Overall, our method has substantial gains over the previous methods. Due to the space limit, we report the result on CoNLLO3 and compare with prompt-based NER and span-based NER in Appendix A. and A.2
Table 2 compares our method with all previous submissions on the BLURB benchmark. We only report Fl due to unavailability of precision and recall of these submissions. The major difference among these submissions are encoders. A direct comparison and [u et al. (202T) which formulates NER as a sequential labeling task and fine-tunes a standard CTMM+CRFFlassfffer on top of the pretrained transformer encoder. While both using PubMedBERT
be made between
method
ggthub.com/hhanginggooo 222p://uooo222////oooooo2
5

---

Published as a conference paper at ICLR 2023
the encoder, our method significantly outperforms Gu et al. (2021) across the board. Compared to the previous best submissions (Kanakarajan et al. 2021; Yasunaga et al. 2022). our method also show substantial gains BC5-chem (+1.2%) disease (+1.9%), and NCBI (1.5%).
Test scores three nested NER datasets. Bold-and-underling bold-only and underline-only indicate the best, the second best, and the third best respectively. The different en- coders are used: LSTM, BI BaRg-large, BioB BioBERT, BioBI BioBERT-large, BAI BART-large, T5b T5-base, Bb BERT-base. i Original scores of Li et al. (2020) are not repro-
Table 1:
ducible. Like Yan al. 2021) D. we report the rerun of their released code.
ACE2004
ACE2005
GENIA
Encoder
P
R
P
R
P
Lu & Roth 2015
70.0
56.9
62.8
66.3
59.2
62.5
74.2
66.7
70.3
Katryar
Cardic
(2018
73.6
71.8
72.7
70.6
70.4
70.5
77.7
71.8
74.6
Shibuya & Hovy
2020
B1
83.7
81.9
82.8
83.0
82.4
82.7
78.1
76.5
77.3
Wang ct al. 2022
BI/BioB
86.1
86.5
86.3
84.0
85.4
84.7
79.5
78.9
79.2
Li al.
2020)
BI/BioB
85.8
85.8
85.8
85.0
84.1
84.6
81.2
76.4
78.7
Yu ct al.
20,20
BI/BioB
87.3
86.0
86.7
85.2
85.6
85.4
81.8
79.3
80.5
Tän ct al.
202
BI/BioB
88.5
86.1
87.3
87.5
86.6
87.1
82.3
78.7
80.4
Yan et al. 2021
BAI
87.3
86.4
86.8
83.2
86.4
84.7
78.6
79.3
78.9
Zhang
(2222
T5b
86.5
84.5
85.4
83.3
86.6
84.9
81.0
77.2
79.1
Wan al. (002
Bb
86.7
85.9
86.3
84.4
85.9
77.9
80.7
79.3
Bb/BioB 88.3
89.1
88.7
89.1
89.8
89.5
81.5
79.6
80.5
BINDER (Ours)
BI/BioBI
89.7
89.7
89.7
89.6
90.5
90.0
83.4
78.3
80.8
Test F1 scores on five flat NER datasets from the BLURB benchmark taka.ms/blurk Bold and underline indicate the best and the second best respectively. All encoders use their base version.
Table
Encoder
BC5-chem
BC5-disease
NCBI
BC2GM
JNLPBA
Loc ct al.
2019
BioBERT
92.9
84.7
89.1
83.8
78.6
Gu el al. 2021)
PubMedBERT
93.3
85.6
87.8
84.5
79.1
Kanakarajaa
2021
BioELECTRA
93.6
85.8
89.4
84.7
80.2
Yasunaga et al. (0222)
LinkBERT
93.8
86.1
88.2
84.9
79.0
BINDER (Ours)
PuhMedBERT
95.0
88.0
90.9
84.6
80.3
Distantly Supervised NER Results Table compares test scores of our method and previous methods on BC5CDR. It presents clear advantage of our method over all previous methods in the dis- tantly supervised setting. The score is advanced by +1.5% over the previous best method (Zhou cct al. 2022). which adapts positive and unlabeled (PU) learning to obtain a high recall yet at the loss of precision. In contrast, our method maintains reasonable recall (comparable to [Shang et al.) 2018 |Png et al. (2019) and substantially improves the precision. Note that besides the dictionary used to generate distant supervisions, Zhou et al. (2022); Shang al. (2018) require an additional high-quality dictionary to estimate the noisiness of non-entity spans. Our method does not have such requirement. For reference, Table alsoo includes the supervised state the art. Our method the supervised setting achieves 91.9% Fl, outperform- ing the previous SOTA Wang et al. (202T) I by 1.0%. Comparing both settings, tantly supervised result still has an over- wver-1--point gap with the supervised one, indicating a potential to further reduce the false negative noise.
Table 3: Test scores on BC5CDR. All base- lines scores in the distantly supervised setting are quoted from Zhou et al. 2002).
BC5CDR
R
F1
Distantly Supervised
Dict/KB Matching
86.4
51.2
64.3
AutoNER (Shang et al.
2018
82.6
77.5
80.0
BNPU (Peng et al. 2019
48.1
77.1
59.2
BERT-ES [Laang ct al.
2020
80.4
67.9
73.7
Conf-MPU (Zhou et al.
2022
76.6
83.8
80.1
BINDER (Ours)
87.6
76.3
81.6
Fully Supervised
Nooralahzadeh at. 2019
92.1
87.9
89.9
Wang 2021
90.9
BINDER (Ours)
92.6
91.2
91.9
observe that the dis-

---

Published as a conference paper at ICLR 2023
4 ANALYSIS
Here we conduct extensive analyses of our method. For efficiency, all analysis in this section is done based on the uncased BERT-base encoder.
Ablation Study We compare several variants of our method and report their test scores on ACE2005 in Table [ We observe performance degradation in all these variants. Shared linear layers uses the same linear layer for span and token embeddings, and the same linear layer for en- tity type embeddings, in the hope of projecting them into the same vector space and sharing their semantics. It leads to a slightly better precision but a drop of recall. Similar changes are observed in Joint position-span inference, which adopts a more stringent strategy to prune out spans -only keep spans whose start, end, and span scores are all above the thresholds. No position-boo baaiionn-aadddde objectives only optimizes the span-baset objective, which penalizes partially corrected spans in the same way as other spans, resulting in marginal improvement of recall but a significant loss of precision.
Choices of Entity Type Descriptions Table 5 compares different choices of entity type descrip- tions. Our final model uses annotation guidelines, which outperforms other choices: (1) Atomic labels considers each entity type as an atomic label. Instead of learning an entity type encoder, we directly learn an embedding vector for each entity type. (2) Keywords uses keyword for each entity type as the input to the entity type encoder, e.g., "person" for PER. (3) Protolypical instances for each minibatch during training dynamically samples from the training data prototypical instance for each entity type and uses it as input to the entity type encoder. Unlike annotation guidelines, add special markers to indicate the start and end of entity span and use the hidden states of the start marker as entity type embeddings. At test time, we increase the number of prototypical instances three for each entity type. Larger number of prototypical instances may improve the performance, Prototypical instances may also lead to better performance in few-shot or zero-shot settings. We leave them to future exploration.
Table 4: Test scores of our method and its variants on ACE2005.
Table 5: Test scores on ACE2005 with dif- ferent entity type descriptions.
ACE2005
ACE2005
P
R
F1
P
R
F1
Our full model Shared linear layers Joint position-span inference
89.1
89.8
89.5
Atomic labels
88.9
89.6
89.2
89.3
89.3
89.3
Keywords
88.7
89.8
89.2
89.4
89.2
89.3
Prototypical instances
88.7
90.1
89.4
No position-baasee based objectives
88.7
89.9
89.3
Annotation guidelines
89.1
89.8
89.5
Thresholding Strategies Our method uses dynamic similarity thresholds to distinguish entity spans from non-entity spans. We compare the impact of dynamic thresholds in our method with two straightforward variants: (1) Learned global thresholds replaces dynamic thresholds with global thresholds, one for each entity type. Specifically, we consider the global similarity thresholds as scalar parameters (initialized as 0). During training, we replace the similarity function outputs sim(u [cis], , e in Eqquation and sim,so.o, ex)) in Equation 9with the corresponding global thresh- olds. At test times, the global thresholds are used to separate entity spans from non-entity spans. (2) Global thresholds tuned on dev introduces global thresholds after the training done and tune them on the development set. During training, instead of Equation 10 we optimize a simplified loss with- out thresholding, alstar: + YLend + joinn Table compares their test scores on the ACE2005 dataset. Dynamic thresholds have the best scores overall. Learned global thresholds performs better than global thresholds tuned on the development set, indicating the necessity of learning thresh- olds during training. Note that the global thresholds tuned on dev still outperforms all the previous methods in Table showing a clear advantage of our bi-encoder framework. In Appendix (A.7 We further visualize and discuss the distribution of similarity scores between text spans and entity types based on different thresholding strategies.
Time Efficiency Table 7 compares the training and inference speed of our method against several prominent methods. To ensure fair comparisons, all speed numbers are recorded based on the same machine using their released code with the same batch size and the same maximum sequence length.
7

---

Published as a conference paper at ICLR 2023
MRC-NER (Li et al. 2020) formulates NER as machine reading comprehension problem and employs a tross-attention encoder. Comparing with it, our bi-encoder framework has 17x and 8x speed training and inference respectively. Biaffine-NRR (Yu et al. 2020) formulates NER as a dependency parsing problem and applies a biaffine classifier to classity the typed arc between an entity span start and end. Comparing with it, our framework does not need biaffine layer and is 1.7x and 2.4x faster at training and inference. Without the need of conditional random fields (CRFs), our framework even faster than the widely used BERT-CRF framework. A drawback of BERT-CRF is that it cannot be applied to nested NER. Here we test it on a flat NER dataset CoNLL2003 (Tjong Kim Sang& De Meulder 2003). All other frameworks are tested on ACE2005.
Table 6:
Test scores of our method using dif- ferent thresholding strategies on ACE2005.
Table 7: Training and inference speed.
Speed (w/s)
ACE2005
Training
Inference
P
R
F1
MRC-NER (Li et al.
2020
147
1,110
Biaffine- NER Yu
2020
1,548
3,634
Dynamic thresholds Learned global thresholds 88.2
89.1
89.8
89.5
BERT-CRF (pytorch neural crf
2,273
8,596
89.0
88.6
Global thresholds tuned dev
86.3
88.7
87.5
BINDER (Ours)
2,571
8,886
Table 8:
Test FI score breakdowns on ACE2005 and GENIA. Columns compare Fl scores
different entity types. Rows compare Fl scores based on the entire entity span, or only the start or end of entity span. S-F1 denotes the strict F1 requiring the exact boundary match. L-F1 denotes the loose Fl allowing partial overlaps. The color signifies substantially better Fl scores than the corresponding entity span strict Fl scores.
ACE2005
GENIA
PER
GPE
ORG
FAC
LOC
VEH
WEA
ALL
Prot.
DNA
CelIT.
CellL.
RNA
ALL
S-FI gga
93.4
91.2
79.7
81.0
78.7
84.8
82.1
89.5
82.9
77.6
74.5
76.3
87.9
80.5
S-F1
sari
93.9
91.2
80.7
81.0
79.0
84.8
82.1
89.9
86.1
80.9
74.5
80.2
88.7
83.2
S-F
end
93.9
91.2
81.9
83.1
79.0
86.8
82.1
90.3
87.6
82.6
83.7
82.8
91.0
85.8
F1 span
94.4
91.4
83.0
83.1
79.4
87.2
82.1
90.8
91.6
87.4
84.8
87.2
91.7
89.9
Breakdown Table shows the test Fl scores on each entity type of ACE2005 and GENIA. We report four types of F1 scores: S-F1 is the strict Fl based on the exact match of entity spans; S-FI start the strict Fl based on the exact match of entity start words; S-F1 end the strict F1 based on the exact match of entity end words; L-F1 span is the loose Fl allowing the partial match of entity spans. We notice that S-F1 end is often substan- tially better than S-Flspan and S-Fl To explain this difference, we go through these partially corrected predictions and summarize the common errors in T The most common one is the inclusion or exclusion of modifiers. This could be due to annotation disagreement: in ACE2005, sometimes generic spans are preferred (e.g., "tomcats" vs. "f-14 tomcats"), while in some cases specific spans are preferred (e.g., "cruise ship" vs. "ship"). This issue is more common in the biomedical dataset GENIA, e.g., "human GR" is considered as wrong while "human lymphocytes" are correct, which explains the higher scores of S-Flend than S-Fl Missing genitives for person names discover some annotation errors, where names of a single person, protein, or cell line are sometimes broken into two less meaningful spans, e.g., "Dr. Germ" is annotated as two person mentions "Dr"
Performance
g
Table 9: Examples of common errors among the par- tially corrected predictions. Red indicates eтror spans. Blue indicates missing spans. The number after each span mean the span frequency in the training data.
Error Type
Ent. Type
Predicted + Gold
VEH
f-14 tomcats (0) + tomeats (0)
FAC
federal court (0) ++ court (0)
VEH
ship (29) ++ cruise ship (1)
Modifier Error
CellL.
unstimulated Tells (0) <> Tellss (553)
Prot.
human GR (0) GR (88)
CeIIT.
lymphocytes (117) human lympho-
eytes (18)
DNA
motif (0) + syntheric E6 motif (0)
Missing Genitive
PER
attendant (3) ++ attendant's (0)
PER
Dr. Germ (0) + Dr (1)/ Germ (0)
Prot.
Ag amino acid sequence (0) ++ Ag (1)/ amino acid sequence (6)
Annotation Error
Cell
EBV-rranformmed human cell line
SKW6.4 (0) <> CBV-rransoormed bu- man B cell line (0) SKW6.4 (1)
DNA
second-site LTR revertants (0) ++
sccond-site LTR (0)
another common emrors in ACE2005. We also
8

---

Published as a conference paper at ICLR 2023
and "Germ", and "-ranssaafformee human B cell line SKW6.4" is annotated as two separate cell lines "EB trransfooomd human B cell line" and "SKW6.4".
5 RELATED WORK
Supervised NER Early techniques for NER are based on hidden markov models (e.g.. Zhou & Su 20022) and conditional random fields (CRFs) (e.g.. McDonald & Pereira 2005). However, due to the inability to handle nested named entities, techniques such as cascaded CRFs (Alex al. 2007). adpated constituency parsing (Finkel & Manning 2009), and hypergraphs (Lu & Roth, 2015) are developed for nested NER, More recently, with the advance deep learning. a myriad of new techniques have been used in supervised NER. Depending the way they formulate the task, these techniques can be categorized as NER as sequence labeling (Chiu & Nichols
2016 Ma & Hovy,
2016 Katiyar& & Cardie (2018); NER as parsing (Lample al. 22016 Yu et al. 2020); NER as span
classification (Sohrab & Miwa 2018 Xia et al. 2019: Ouchi et al., 2020 Fu et al. 202I): NER
problem (Straková et al. 2019 Yan et al. 2021 and NER as machine reading comprehension (Li et al. 2020; Mengge et al. 2020). Unlike previous work, we formulate NER contrastive learning problem. The span- design of bi-encoder and contrastive loss provides us with the flexibility to handle both nested and flat NER.
sequence-to-sequeene
Distantly Supervised NER Distant supervision from external knowledge bases in conjunction with unlabeled text is generated by string matching (Gianakkooollos et al. (0177) or heuristic rules (Ren et al.] 2015 Fries et al.) 2017). Due the limited coverage external knowledge bases, distant supervision often has high false negative rate. To alleviate this issue, (Shang et al. (2018) (8) design new tagging scheme with an unknown tag specifically for false negatives: Mayhew et al. (2019) iteratively detect false negatives and downweigh them in training: Peng et al. (2019); Zhou ct al. al. 2021b) identify dictionary biases via structural causal model, and de-bias them using causal interventions. Liu et al. (022)) introduce a calibrated confidence estimation method and integrate into self-training framework. Without replying sophisticated de-noising designs, our bi-encoder framework can be directly used in distant supervision. Experiments in $ show the robustness of our contrastive learning algorithm the noise in distantly supervised NER.
2022) address overfitting to false negatives using Positive and Unlabeled (PU) learning: Zhang
Bi-Encoder The use of bi-encoder dates back to Bromley et al. (1993) for signature verification and Chopra et al. (2005) for face verification. These works and their descendants (e.g.. Yih et al. 2011 Hu et al. 2014) refer to the architecture as siamese networks since two similar inputs are encoded separately by two copies of the same network (all parameters are shared). Wsabie (Weston) et al. 2010) and StarSpace (Wu et al., 2018) subsequently employ bi-encoder to learn embeddings for different data types. Using deep pretrained transformers as encoders, Hummauu et al. (2019) compare three different architectures, bi-encoder, poly-encoder, and cross-encoder. The bi-encoder architecture afterwards has been use in various tasks, e.g., information retrieval (Huang et al. 2013 Gillick et al. 2018). open-domain question answering (Karpukhin et al., 2020), and entity link- ing (Gillick et al. 2019: Wu et al. 2020 ZZhang et al., 2021a). To our best knowledge, there is no previous work learning bi-encoder for NER.
CONCLUSIONS
We present bi-encoder framework for NER using contrastive learning, which separately maps text and entity types into the same vector space. To separate entity spans from non-entity ones, we introduce a novel contrastive loss to jointly learn span identification and entity classification. Exper- iments in both supervised and distantly supervised settings show the effectiveness and robustness of method. We conduct extensive analysis explain the success of our method and reveal growth opportunities. Future directions include further improving low- wwwferroorming types and applications in self-supervised zero-shot settings.

---

Published as a conference paper at ICLR 2023
REFERENCES
Beatrice Alex, Barry Haddow, and Claire Grover. Recognising nested named entities in biomed- ical text. In Biological, translational, and clinical language processing, pp. 65-72, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://
Rami Aly, Andreas Vlachos, and Ryan McDonald. Leveraging type descriptions for zero-shot named entity recognition and classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Nat- ural Language Processing (Volume I: Long Papers), pp. 1526-1528, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18653/v 1/2/33//112211.1 aacla//........oo 120. URL https:
//aclanthology wttp:/ogy..rgg//2...eaooo //colaucholooyyyrrrr322......._.
120
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
Kai-Wei Chang, Rajhans Samdani, and Dan Roth. A constrained latent variable model for coref- erence resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 601-612 Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
Jason P.C. Chiu and Eric Nichols. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357-370, 2016. doi: 10.1162/ tac1_a000104.
owg/ 016-1026666111
Yejin Choi, Eric Breck, and Claire Cardie. Joint extraction of entities and relations for opinion recogmition. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing. pp. 431.439, Sydney, Australia. July 2006. Association for Computational Linguis- tics. (tRttttss:: aataaccaanattocooooooooooo wrgg:/rrgg/oooo
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively. with application to face verification. In 2005 IEEE Computer Society Conference Computer Vision and Puttern Recognition (CVPR'05) volume 1, pp. 539-546. IEEE, 2005.
Nigel Collier and Jin-Dong Kim. Introduction to the bio-entity recognition task at JNLPBA. In Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications (NLPBA/BioNLP), pp. 73-78, Geneva, Switzerland, August 28th and 29th 2004. COLING.
wwwpo/g...oggoooooo 1213
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. Template based named entity recognition using BART. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1835-1845, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021. finddiiggg--c......... URL www.hhttp:/// //atts://acll http:///aaaaahhahhcc log.olo........... findings-ach acl. 161
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. CONTaiNER: Few-shot named entity recognition via contrastive learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6338- 6353, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 18.18v53/v1/ 2022.acI- long-439, URL httthtt/////// //ac////clla / antho orooorgg///00 2022 aal-l 439
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/N11--44255 URL https:
//aclanthology org N19-1111 1423
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim. Prompt-learnin for fine-grained entity typing. arXiv preprint arXiv:2 108. 10604, 2021.
10

---

Published as a conference paper at ICLR 2023
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. Special report: Nebi disease corpus: A resource for disease name recognition and concept normalization. of Biomedical Informatics, 47:1-10, feb 2014. ISSN 1532-0464.
Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. Advances in neural information processing systems, 31, 2018.
Jenny Rose Finkel and Christopher D. Manning. Nested named entity recognition. In Pro- ceedings of the 2009 Conference on Empirical Methods Natural Language Processing, pp. 141-150, Singapore, August 2009, Association for Computational Linguistics. URL https:
//aclanthology.crKk 2D0-e//D99-0000 1015
Jason Fries, Sen Wu, Alex Ratner, and Christopher Ré. Swellshark: generative model for biomed- ical named entity recogmition without labeled data, arXiv preprint arXiv:1704.000 2017.
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. SpanNER: Named entity re-frecognition as span prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7183-7195, Online, August 2021. Association for Computational Linguis- tics. doi: 10. 12t13//11112111lllllll
URL
acl acl-long.55 558
Athanasios Giannakopoulos, Claudiu Musat, Andreea Hossmann, and Michael Baeriswyl. Unsu- pervised aspect term extraction with B-LSTM & CRF using automatically labelled datasets. In Proceedings of the 8th Workshop on Computational Approaches Subjectivity, Sentiment and Social Media Analysis, pp. 180-188, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 11863//11/117-52244
URL.
org/ 522/
Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. End-to-end retrieval in continuous space. arXiv preprint vrXiv:1811.0800 2018.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene le, and Diego Garcia-Olano. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 528-537, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/ K19-1049, URL httpts::/aclaatthllle/////////
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions Computing for Healtheare (HEALTH), 3(1): 1-23, 2021.
Tao Gui, Jiacheng Ye, Qi Zhang, Zhengyan Li, Zichu Fei, Yeyun Gong, and Xuanjing Huang. Uncertainty-avee aware label refinement for sequence labeling. In Proceedings of the 2020 Confer- ence on Empirical Methods Natural Language Processing (EMNLP), pp. 2316-2326, Online, November 2020. Association for Computational Linguisties. doi: 10.18653/v122020.emm main. 181. URI. www.h:ttt/..t..... //ac: http:////...mcooooooooo//////000 http://////.000..mmmmmmmmaaa.aa emnlp-ma i 181]
Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. Convolutional neural network architectures for matching natural language sentences. Advances in neural information processing systems, 27, 2014.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heek. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM International Conference on Information &amp; Knowledge Management, CIKM '13, pp. 2333-2338, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 978 9781450322638. doi: 10.11457550551525555666.
URL Ittps: ://s:///o.... 11..:..og//1.111000.
25 2505665

---

Published as a conference paper at ICLR 2023
Samuel Humeau, Kurt Shuster, Marie-Amee Lachaux, and Jason Weston. Poly-encoders: Archi- tectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations, 2019.
Kamal raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. BioELEC- TRA:pretrained biomedical text encoder using discriminators. In Proceedings of the 20th Work- shop on Biomedical Language Processing, pp. 143-154, Online, June 2021. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/22211 bionlliiiooppppie 1.16. URL DtLp:tts:: /
org/ 2017/2021.00 www.2022iiiio.. www.lpinnnnnll....
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing Online, November 2020. Association for Computational Linguis- (EMNLP), pp. 1///220 222000.emllp-mmin.5550. UtLhttps: /aapc//alllaaa www.annnnooogyyyooooomoooooollle org/2020.
tics. doi: 10.18653/v
emr p ma in. 550
Arzoo Katiyar and Claire Cardie. Nested named entity recognition revisited. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume I (Long Papers), pp. 861-871, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/1/N18- /N1--1099.
URL htt.httt:::/eee htt///caanttii aetennoooolgg Baapehhology..oo////--17777 otg.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18673. 2020.
J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun 'ichi Tsujüi. Genia corpus-a semantically annotated corpus for bio-textmining. Bioinformatics, 19(uppl 1):ippl1)::---11:::::::: 10:1188--182, 2003.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 260-270, San Diego, California, June 2016. Association for Computational Lin- URL htthhttps//aaee http//aaaanthoooggg..oo org/N16 6 1030
guistics. doi: 10. 1/N16-
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 1240 09 2019. ISSN 1367-4803. doi: 10.1093/bioinfommats/boooz bt.2682
URL
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Al- lan Peter Davis, Carolyn Mattingly, Thomas C Wiegers, and Zhiyong Lu. Biocreative v edr task corpus: a resource for chemical disease relation extraction. Database, 2016, 2016.
Xiaoya Li, Jingrong Feng, Yuxian Meng. Qinghong Han, Fei Wu, and Jiwei Li. A unified MRC framework for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5849-5859, Online, July 2020. Asso- URL https:
ciation for Computational Linguistics. doi: 1/233///2020.aclmmm
//aclanthologys wwt.:000..g.000.o.oooo acl-mainnn 519.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang. Tuo Zhao, and Chao Zhang. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining, KDD '20, pp. 1054-1064, New York, NY, USA. 2020. Association for Com- doi: 10.1145/394486.3403311
puting Machinery. ISBN 47814993799
URL https:
7/doi. 1141/139444888 01113444886344334333333333433
Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao. Noisy-labeled NER with confidence estimation. In Proceedings of the 2021 Conference of
12

---

Published as a conference paper at ICLR 2023
the North American Chapter of the Association for Computatioonal Linguistics: Human Lan- guage Technologies, pp. 3437-3445, Online, June 2021. Association for Computational Linguis-
tics. doi: 1t1.18653/112222.naaaa maacll-min.269. ttt./t/ts/////llattoooooo.ooggg
naacl -ma 1 n...i..... 2.69
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprini
171.5 2017.
Wei Lu and Dan Roth. Joint mention extraction and classification with mention hypergraphs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp- 857-867, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/ 102. URL htt.tt/pss//cccaacoooooooooo////
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 3219-3232, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/ v1/DI8- 1360. URL htttt::/aaaeee http:/::///aaaahhlloooooo www.gyyorrDDD11110001
Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. Template- free prompt tuning for few-shot NER. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. tics. doi: 10. 10.18653/v11 10//3//112222.mn/m www.:naaallmaaan.mm naac ma in. 420
Seattle, United States, July 2022. Association for Computational Linguis-
htRpptt::: acc anthelogy..oo org/2022..
Xuezhe Ma and Eduard Hovy. End-to-end sequence labeling via bi-directional LSTM-CNN&-CRE: In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1064-1074, Berlin, Germany, August 2016. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/P16-111 1101. URL https: http:::://alaaatholoo...oo
P16-1101
Stephen Mayhew, Snigdha Chaturvedi, Chen-Tse Tsai, and Dan Roth. Named entity recogni- tion with partially annotated training data. In Proceedings of the 23rd Conference on Com- putational Natural Language Learning (CoNLL), pp. 645-655, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10. 10.18653/1//119-1066) URL
Ryan McDonald and Fernando Pereira. Identifying gene and protein mentions in text using condi- tional random fields. BMC bioinformatics, (11::117,0055....
Xue Mengge, Bowen Yu, Zhenyu Zhang, Tingwen Liu, Yue Zhang, and Bin Wang. Coarse-to- Fine Pre-training for Named Entity Recognition. In Proceedings the 2020 Conference Em- pirical Methods in Natural Language Processing (EMNLP), pp. 6345-6354, Online, November 2020. Association for Computational Linguisties. doi: 10. 20.18863/v1222 1ttp2/2222222n2../ m URL
wwg.:/..ooooooo 2020. wtt.z///22...mn1p1...mmm emnlp ma mwi.. 514
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation extrac- tion without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003-1011 Suntec, Singapore, August 2009. Association for Computational Lin- 1113
guistics. URL http:pss///claattooooo.....ee
David Nadeau and Satoshi Sekine. A survey of named entity recognition and classification. Lingvis- ficae Investigationes, 30(1):3-26 2007.
Farhad Nooralahzadeh, Jan Tore Lønning, and Lilja Øvrelid. Reinforcement-based denoising of distantly supervised NER with partial annotation. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pp. Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19. URL https://ach antho logy.org/D19-6125
13

---

Published as a conference paper at ICLR 2023
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0372 2018.
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, and Kentaro Inui. Instance-based learning of span representations: case study through named en- tity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 6452-6459, Online, July 2020. Association for Computational Linguis- URL httpps///allaathooooy..o///////p
tics. doi:
1/20200/11220aalmmmmm acl Sellmainn 1al-main.555
Minlong Peng, Xiaoyu Xing, Qi Zhang, Jinlan Fu, and Xuanjing Huang. Distantly supervised named entity recognition using positive-unlabeled learning. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2409-2419 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.188533/11/111 1231. URL https: /acclnnthooogy.orgp P11-1
Xiang Ren, Ahmed El-Kishky, Chi Wang, Fangho Tao, Clare R Voss, and Jiawei Han. Clustype: Effective entity recognition and typing by relation phrase-based clustering. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 995-1004. ACM, 2015.
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pp. 2054-2064, Brussels, Belgium, October- November 2018, Association for Computational Linguistics. doi: 10. 10.18653/w1/D18--22 URL
htttss:/acclattoo won.thhhooy.... ooyy.
org/D18 oww.r/h/oiooooo..//111.0
1230
Takashi Shibuya and Eduard Hovy. Nested Named Entity Recognition via Second-best Sequence Learning and Decoding. Transactions of the Association for Computational Linguisties, 8:605- 620, 09 2020. ISSN 2307-387X. 1162/tacl 00334
doi: 10.1162/tacl a 00334.
Larry Smith, Lorraine K Tanabe, Cheng-Ju Kuo, I Chung, Chun-Nan Hsu, Yu-Shi Lin, Roman Klinger, Christoph M Friedrich, Kuzman Ganchev, Manabu Torii, et al. Overview of biocreative gene mention recognition. Genome biology, 9(2):1-19, 2008
Mohammad Golam Sohrab and Makoto Miwa. Deep exhaustive model for nested named entity recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2843-2849, Brussels, Belgium, October- 2018. Association for Com- putational Linguistics. doi: 10. 10.18- 1309. URL https:://cccaaaae http:://ccaathhoooyy org/
D18-1300 1309
Jana Straková, Milan Straka, and Jan Hajic. Neural architectures for nested NER through lineariza- tion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pp. 1331, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/v1/PIL 1/t0//111199111111 1527. URL https: http:///aaaattoooooo.oooo org. P11- 1527
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv preprint arXiv:1406..
2014.
Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, and Yueting Zhuang. A sequence-to-set network for nested named entity recognition. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI-21, 2021. URL
21055.00 22105.08901 08901
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independee named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https:
// //acl //aclantholoyy htwpllooyyoog//0000000000
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In
14

---

Published as a conference paper at ICLR 2023
Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
URL hutpss//pprocccingg..noozzzz http:/pss..cpppprr//////EEEE
Juncheng Wan, Dongyu Ru, Weinan Zhang, and Yong Yu. Nested named entity recognition with span-level graphs. In Proceedings of the 60th Armual Meeting of the Association for Compu- tational Linguistics (Volume Long Papers), pp. 892-903, Dublin, Ireland, May 2022. As- sociation for Computational Linguistics. doi: (0.18663/11/20222alllloooo //aclanthology.orgeee http:/2222..llloon.ooooo
URL https:
Jue Wang, Lidan Shou, Ke Chen, and Gang Chen. Pyramid: A layered model for nested named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics. pp. 5918-5928, Online, July 2020. Association for Computational Linguis- URL
tics. doi:
orgp:///2oooyy.//22200000 org/2020.
acl-main.525
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. Improving named entity recognition by external context retrieving and cooperative learning. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1800-1812, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 wa..l...c--oo...1e 142. URL http:: htt.:/ttps//alllttooloooo/////// wwwphholll.o..ogoogoooo wtt.:o/ogg///a..ooo acl-long........... 142)
Jason Weston, Samy Bengio, and Nicolas Usunier. Large scale image annotation: learning to rank with joint word-image embeddings. Machine learning, 81(1):21-35, 2010.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: Statc-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods Natu- ral Language Processing: System Demonstrations, pp. 38 48, Online, October 2020. Associ- ation for Computational Linguistics. doi: 10. (tt/33/112220.mmm/// www.laantholoo...rg////32 emm.220eeemmpp.mmmmmmmmm demos.
URL https:
Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, and Jason Weston. Starspace: Embed all the things! In Proceedings of the AAAI Conference on Artificial Intelli-
gence, volume 32, 2018.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6397-6407, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. emnlp-manlee.eeeee 519
URL htttts:ss//aaaanneeees
org/2020
Congying Xia, Chenwei Zhang, Tao Yang. Yaliang Li, Nan Du, Xian Wu, Wei Fan, Fenglong Ma, and Philip Yu. Multi-grained named entity recognition. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pp. 1440-1440, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-111 1138. URL
org/P19- -138
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. A unified gen- erative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5808 5822, Online, August 2021. Association for Computational Linguistics. doi: 10. 100188655 1t..1866///1222.acll// /lttps://aclantiin
URL
wwwpor//222222 aac.aalllll accl-long........... 451
Jie Yang. Shuailong Liang, and Yue Zhang. Design challenges and misconceptions in neural se- quence labeling. In Proceedings of the 27th International Conference on Computational Linguis- tics, pp. 3879-3888 Santa Fe, New Mexico. USA, August 2018, Association for Computational Linguisties. URL htt.::ttss//// antho ogy org/ C1 1322
15

---

Published as a conference paper at ICLR 2023
Michihiro Yasunaga, Jure Leskovec, and Percy Liang. LinkBERT: Pretraining language mod- els with document links. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8003-8016. Dublin, Ireland, May URI.
2022. Association for Computational Linguistics. doi: 10.18663///2022.alllooo.ggg
wtw. 2022. aal2aaclllae acw.............255
Wen-tau Yih, Kristina Toutanova, John C. Platt, and Christopher Meek. Learning discriminative projections for text similarity measures. In Proceedings of the Fifteenth Conference on Computa- tional Natural Language Learning, pp. 247-256. Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URI. https::////... acc antho wotthology.oo org 1
Juntao Yu, Bernd Bohnet, and Massimo Poesio. Named entity recognition as dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6470-6476, Online, July 2020. Association for Computational Linguistics. doi: 10. 18.18v53/v1/ 2020.acl- 577. URL http:::tttlatattiaaita
2rc/2220..cl-mmia..5
Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Knowledge-rich self-supervised entity linking. arXiv preprint arXiv:21 2021a.
Shuai Zhang, Yongliang Shen, Zeqi Tan, Yiquan Wu, and Weiming Lu. De-bias for generative extraction unified NER task. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 508-818, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10.1 10.18653/1/20022.aal 1ong.59.. URL https:
\/aclanthology. hww.hhooogooooogg///00 wwgp//t2...2....oooooo acle-a..ccl-- acl-long........ 59
Wenkai Zhang, Hongyu Lin, Xianpei Han, and Le Sun. De-biasing distantly supervised named entity recognition via causal intervention. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing (Volume I: Long Papers), pp. 4803-4813 4813, Online, August 2021b. Association for Computational Linguistics. doi: 10. 10.118633/1/2022 110. URL htthtt:: acl-long..711
//aclanthology. wwtp::/w..o./22....1l11oo.....o.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
GuoDong Zhou and Jian Su. Named entity recognition using an HMM-based chunk tagger. In Proceedings of the 40th Annual Meeting of the Association for Compulational Linguisties, pp- 473-480, Philadelphia, Pennsylvania, USA, July 2002, Association for Computational Linguis- tics. doi: 15331///007333.107111116 URL https: /tttpp::/acaaaae /a////aaannttooooooeeeoeo wwtp://wooooy...//////22// owg/ 1060
Kang Zhou, Yuepei Li, and Qi Li. Distantly supervised named entity recognition via confidence- based multi-class positive and unlabeled learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 7198-7211, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 16653//1/2022.cc/ 498. URL http:::s//ccaattthhho htt.LLttpp////.aattoooooooooo//p wwtp://22o2...llooog.g. 498
A APPENDIX
A.1 COMPARISON WITH PROMPT-BASED LEARNING
Prompt- based learning emerges as powerful method in few-shot NER (Cui et al. 2021 Ma et al. 2022 Ding et al. 2021). Below, we compare BINDER with Tempalte BART (Cui al. 2021). representative prompt-based learning method:
Method Difference Cui et al. (2021) follows an encoder-decoder framework. Given a text span, decodes and selects among all possible templates including none-entity templates. Their candidates are templates based on a static set of entity types. In contrast, BINDER is to given an entity type select among all possible text spans. The candidates are dynamic set of text spans, which are different for different input documents. The size candidates is much larger - O(L?), where L is the max seq length, set to 256 in our experiments.
16

---

Published as a conference paper at ICLR 2023
Loss Difference Cui et al. 2021 cross-entropy maximize the likelihood of gold entity type template against templates of other entity rypes. In comparison, BINDER uses contrastive learning, given entity type, to maximize the similarity of gold text spans against other text spans. Instead of comparing entity types, BINDER compares all possible text spans from an input document. captures subtleties of each text span via contrastive learning. As shown in previous work, instance- based contrastive learning has better generalization performance than cross-entrope loss (Khosla al. 2020), due to its robustness noisy labels (Zhang & Sabuncu 2018; Sukhbaatar et al. 2014) and the possibility of better margins (Elsayed et 2018). To our best knowledge, we are the first demonstrate BINDER with contrastive loss (among text spans) significantly cross-entropy loss (among entity types/templates) in both supervised settings (see Table and Table 2 and distantly supervised settings (see Table 3)
Non-entity Handling Difference Cui et al. 2 labels all non-entity spans with non-entity tem- plates. This can introduce false negatives when the training data is partially annotated (Das al. 2022 Aly et al. 2021). Our formulation allows us to avoid using an explicit non- non-entity class, and instead to introduce a dynamic threshold based on the input document and the entity type, to dis- tinguish entity spans from non-entity spans. Our experiments show clear advantage of dynamic thresholding over explicit non-entity labeling scheme.
Computational Efficiency The encoder-decoder framework used by Cui et al. (2021) has much lower computational efficiency. First, it has double parameter size compared to BINDER (which encoder-only). Second, both its training and inference time are significantly slower than BINDER. Because the decoding process relies on cross-attention from decoder to encoder, their method has to decode all possible templates for each input document. The number of templates Niemplates Niex spuns X Neniiy types And the overall the decoding operations are (Naaccmments x N spuns X N Nentiy types). Furthermore, the decoding process has to be done in an autoregressive manner, which even reduces the time efficiency. In comparison, BINDER employs a bi-encoder framework. separately encodes entity types and input documents. Encoding entity types does not rely on input documents. At test time, it only needs to encode entity types once, and then re-use them to for NER of different input documents. Encoding entity types is only ( types), and can be done very efficiently in parallel on GPU.
Performance on CoNLLO3 To make a direct comparison with Cui al. 202 we train and evaluate BINDER on CoNLL03 (Tjong Kim Sang & De Meulder 2003). Table 10 reports the test results. BINDER outperforms Template BART well other strong baselines.
Table 10: Test scores on CoNLL03.
CoNLL03
R F1
Yang et al. (2018)
90.77
Ma & Hovy 2016)
91.21
Gui et al. 2020
92.02
Template BART (Cui et al.
2021
91.72
93.40
92.55
BINDE
93.08
93.57
93.33
A.2 COMPARISON WITH SPAN-BASED NER
We highlight the difference between our method (BINDER) and span-based NER below:
Formulation Difference In span-based NER, it is to given a text span select among all entity types. The candidates are static set of entity types, the size of which is usually small (e.g.. PER, ORG, LOC, MISC). In contrast, BINDER formulates NER as given an entity type to select among all possible text spans of input document. The candidates are a dynamic set of text spans, which different for different input documents. The size of candidates is much larger O(L)), where is the max seq length, set to 256 in our experiments.
17

---

Published as a conference paper at ICLR 2023
Loss Difference Span-based NER uses cross-entropy to maximize the likelihood of gold entity type template against templates of other entity rypes. In comparison, BINDER uses contrastive learning, given entity type, to maximize the similarity of gold text spans against other text spans. Instead of comparing entity types, BINDER compares all possible text spans from an input document. It captures subtleties of each text span via contrastive learning. As shown in previous work, instance- based contrastive learning has better generalization performance than cross-entrope loss (Khosla) al. 2020), due to its robustness noisy labels (Zhang & Sabuncu 2018; Sukhbaatar et al. 2014) and the possibility of better margins (Elsayed et 2018). To our best knowledge, we are the first to demonstrate BINDER with contrastive loss (among text spans) significantly cross-entropy loss (among entity types), in both supervised settings (see Table and T and distantly supervised settings (see Table J.
Non-entity Handling Difference Span-based NER labels all non-entity tokens or spans with the same class Out de (0). This can introduce false negatives when the training data partially annotated (Das et al. 2022; Aly et 2021). Our formulation allows us to avoid using an explicit non-entity class, and instead to introduce a dynamic threshold based on the input document and the entity type, to distinguish entity spans from non-entity spans. Our experiments show clear advantage of dynamic thresholding over traditional explicit O labeling scheme.
A.3 IMPACT OF MAXIMUM SEQUENCE LENGTII
In our experiments, we set the maximum sequence length L to 256, meaning that the number of candidate spans is (II²). Speed comparison in Table Z made based on L 256. We also perimented with smaller numbers of L (e.g., 64, 128), which resulted in better speed but slight performance degradation. Increasing L did not bring significant gain but increased memory con- sumption. Therefore, in the experiments, we set L to 256.
A.4 IMPLEMENTATION DETAILS
We implement our models based on the HuggingFace Transformers library (Wolf et al. 2020). The base encoders are initialized using SobMeedBERTT-bas--unaaa (Gu et 2021) or BioBERT (Lee et al. 2019) for biomedical NER datasets, and BFRRT-baseuncased or BERT FarT--large-uncsee (Devlin ct al. 2019) for NER datasets in the general domains. The linear layer output size is 128; the width embedding size is 128; the initial temperatures are 0.07. We train models with the AdamW opti- mizer (Loshchilov & Hutter 2017) of linear scheduler and dropout of 1. The entity start/end/span contrastive loss weights are
0.2, 7 0.2, X 0.6, and the same loss weights chosen for thresholding contrastive learning. The contrastive losses for thresholding and entity are weighted equally in the final loss. For all experiments, we ignore sentence boundaries, and tokenize and split text into sequences with a stride of 16. For base encoders, we train our models for 20 epochs with a learning rate of 3e-5 and a batch size of 8 sequences with the maximum token length of N 128. For large encoders, we train our models for 40 epochs with a learning rate of 3e-5 and a batch size of 16 sequences with the maximum token length of N 256. The maximum token length for entity spans is set to 30. We use early stop with a patience of 10 in the distantly supervised setting. Vali- dation is done every 50 steps of training, and we adopt the models that have the best performance on the development set. We report the median score of multiple runs.
A.5 INFERENCE FOR BINDER
As we can see in Algorithm [ the difference between joint position-span and span-only strate- gies is whether line 9 is used. Also, for flat NER datasets, we further carry out post-processing step remove overlapping predictions (line 19 in Algorithm D. Here, the post-processing (removeOver is carried out in a greedy fashion where higher scored span predictions with earlier start and end positions are preferred.
18

---

Published as a conference paper at ICLR 2023
Algorithm 1: Inference for BINDER.
Input: S (
}} the set of spans 8 [Ei...., Ex} the set of entity types, joint for whether using joint position-span inference, and flat for whether the inference for flat NER.
M = (}:
for Eg E do
compute start/end/span threshold scores buull; enults Snall- for (i.j) E S do
compute start/end/span similarity scores b, e. joint true and brull or e < Enail then Continue:
end if
Snall then
M M F():
end
end
end
19 flat is true then return removevverlap (M);
end
return M;
Function rencveOverlapp
M =):
sort 1 by the similarity score in descending order and break the tie by ascending in start and end positions;
for j. E&) in D do
if span (i,j) has overlap in M then
M M U j(i.),
end
end
return M/
A.6 STATISTICS OF DATASETS
Table 11 reports the statistics of supervised NER datasets.
Table 11: The statistics of supervised NER datasets.
# Annotations
Dataset
|3]
Train
Dev
Test
ACE2004
7
22,735 (5-fold)
ACE2005
26,473
6,338
5,476
GENIA
46,142
4.367
5,506
BC5-chem
5,203
5.347
5,385
BC5-disease
4.182
4,244
4.424
NCBI
5,134
787
960
BC2GM
15,197
3,061
6,325
JNLPBA
46,750
4,551
8,662
A.7 DISTRIBUTIONS SIMILARITY SCORES
Figure visualizes the distributions of similarity scores between different text spans and entity types based on different thresholding strategies. Consistent with the scores in Table 6 the majority of entity spans and non-entity spans are separable regardless the thresholding strategy. This is true even when thresholds used during training and instead tune global thresholds dev.
19

---

10 Published as a conference paper at ICLR 2023
sieamiccttresholds
learned global thres
global thres tuned
Density
2
PER
1
0
2
Density
ORG
1
0
2
Density
GPE
-10 0
-10
10
-10
10
similarity
similarity
similarity
positive spans
negative spans
threshold spans
Figure 2: The kernel density estimation of similarity scores between different spans (entity, non-entity, and threshold spans) and entity types (PER, ORG, GPE) on ACE2005 based on different thresholding strategies.
We zoom the distributions in Figure and observe that the learned global thresholds tend to make the similarities less separable between entity spans and non-entity spans.
dynamic thresholds learned global thres global thres tuned
0.10
Density
PER
0.05
0.00
0.10
Density
ORG
0.05
0.00
0.10
Density
GPE
0.05
0.00
-10
10
-10
similarity
similarity negative spans
similarity
positive spans
threshold spans
Figure 3: Zoom-in of the kernel density estimation of similarity scores in Figure 2 with the y-axis density limited to (0, 0.1).
20


Reviews:
[{"review_id": "zGwG6QRTS0E", "reviewer": "Reviewer_FXLT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework that performs the Named Entity Recognition task from the semantic matching perspective (bi-encoder here). The idea is to map candidate text spans and entity types into the same vector representation space and perform the NER task using distance metric accordingly. One important issue is separating non-entity spans from desired entity mentions. To alleviate these problems, the paper proposes a position-based objective and a dynamic thresholding approach. Experiments on both supervised and distantly supervised NER tasks demonstrate the effectiveness of the proposed approaches.", "summary_of_the_review": "This paper proposes a new framework that performs the Named Entity Recognition task with the bi-encoder model. The paper shows strong results on nested NER and distant supervised NER tasks. While the results are promising, the proposed approach is similar to the span-based NER approach, which might have limited techniqual novelty. ", "strength_and_weaknesses": "\nStrength:\n- Utilizing Bi-encoder has never been considered, while this idea is very similar to the span-based ner. (see Weaknesses)\n- The empirical results are extensive.\n\nWeaknesses:\n- The proposed approach is very similar to span-based NER. In span-based NER, the label type embeddings are randomly initialized (the final layer before softmax). However, the proposed approach utilizes a neural network to obtain that matrix. In the end, the novelty seems limited.\n- Besides nested NER, I would also prefer adding some flat NER results (CoNLL, WNUT, and so on).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "euS93-D46R7", "reviewer": "Reviewer_mFSQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper presents a bi-encoder framework for NER, which applies contrastive learning to map candidate text spans and entity types into the same vector representation space and make the representation of entity mentions be similar with corresponding entity type. It proposes span-based objectives that compare span and entity type, and position-based objectives that compare start and end tokens separately with entity type. It shows its effectiveness in nested NER with fast inference speed.", "summary_of_the_review": "Paper presents bi-encoder style NER framework with novel objectives suitable to NER and shows its effectiveness in nested NER problems. However, I think paper needs to be improved a lot in terms of writing and experiments.", "strength_and_weaknesses": "Paper presents bi-encoder style NER framework with novel objectives suitable to NER and shows its effectiveness in nested NER problems. However, I have following major concerns:\n\n(1) Lack of baseline. Span-based NER, which predicts over all the candidate spans in the sentence, has been widely explored in prompt-based learning (Ding et al., 2021, Cui et al., 2021, Ma et al., 2022… etc.) and . It should have baselines from other research lines for making this paper be more convincing.\n\n(2) No details about candidate generation. Based on window-size, I think the number of span candidates will vary a lot. This ambiguity makes Table 7 be not convincing since the inference speed might be affected by the number of span candidates.\n\nPrompt-Learning for Fine-grained Entity Typing,, Ding et al., 2021\nTemplate-based Named Entity Recognition using BART, Cui et al., 2021\nTemplate-free Prompt Tuning for Few-shot NER., Ma et al., 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RBzDtpR61UQ", "reviewer": "Reviewer_QTaJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a contrastive learning framework for NER tasks. The proposed model is flexible in predicting nested entities.  The idea is novel and experiments demonstrate the effectiveness of the proposed model. ", "summary_of_the_review": "Novel and strong results, but the details are not complete and the experiments/comparison are not complete. ", "strength_and_weaknesses": "Some important details are missed or not cleared. For example, how to set the maximum length L. What is the relation between L and the model performance (accuracy and speed). Experiments didn't include the most common dataset CoNLL 03. This framework setting is similar to the template-based NER (using prompt,\"Template-based named entity recognition using BART\") but this work didn't compare with this work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "29DRYUriPT3", "reviewer": "Reviewer_wC16", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Most of existing works approach NER as sequence labeling, span classification, or sequence-to-sequence problems. These conventional methods rely on CRF to extract entity values which performs subpar on nested NER tasks. To address these limitations, this work formulate NER as a representation learning problem. Intuitively, their method associate entity type embeddings with entity name embeddings in a common subspace. To achieve the goal in a principle manner, they cast the problem into contrastive learning framework. Moreover, they proposed two contrastive objectives for model training and dynamic thresholding for adaptive model inference. Extensive experiment demonstrated the effectiveness of their approach and ablations conveyed insightful findings. ", "summary_of_the_review": "Overall the proposed method for NER is moderately novel. It can benefit the community in this filed. The experiments and ablations demonstrated the effectiveness of their method. ", "strength_and_weaknesses": "Strength\n- This work is well-motivated. The idea of framing NER as representation learning is interesting. Although there are some works combine contrastive learning with NER, this paper proposed novel strategies to handle challenges specifically.\n- The introduced position-based objective to treat entity spans differently. Specifically, they penalize more entity span that have completely no overlap with the gold span while penalize less partially correct span. Their experiment results (as shown in Table 8) implies this objective contribute performance gains consistently and substantially. \n- The authors conducted extensive experiment in various settings and benchmarks. Their methods beats SOTA baselines almost across the board. \n- Various ablation studies have been conducted to share insightful findings.\n\nMajor Concerns\n- There are serval other works [1-3] which are quite relevant/similar with this paper as they solve NER under contrastive learning framework as well. I would appreciate the authors if they can compare their method with these works from methodology perspective.\n- I am wondering how would the model handle entity name that could be a valid value in different entity types. For instance, Willson could be a person name, while it also could be a ORG(company) name. Could you share some thoughts on this case?\n- Currently the method enumerates all candidate entity spans as negative samples for calculating contrastive losses. In the paper, the maximum token length is 256. However if we have even larger sequence length and/or large $L$, do we expect the computational cost would increase significantly? If so, is there any alternative and more efficient way to create negative samples?\n- For span-based objective, what is the dimension of $h_i^T$? I guess it should be $\\mathbb{R}^m$? Could you elaborate more about $D$? Is it learnable? and how to learn it? Is $D(j-i)$ refers to the token representations between $h_i^T$ and $h_j^T$? It seems the answer is not. So why do you using the token embeddings of $h_i^T$ and $h_j^T$ only instead of taking token embeddings between them into consideration?\n- In eq(8), is $u_{[CLS]}=Linear_B^T(h_{[CLS]}^T)$? Do you think $u_{[CLS]}$ would collapse to $e_k^B$? why or why not?\n- In eq(10), what is $\\ell_{\\text{joint}}^+$? Are you referring to $\\ell_{span}^+$?\n\n[1]. Fu, Yingwen, Nankai Lin, Ziyu Yang and Shengyi Jiang. “A Dual-Contrastive Framework for Low-Resource Cross-Lingual Named Entity Recognition.” ArXiv abs/2204.00796 (2022): n. pag.\n[2]. Ye, Hongbin, Ningyu Zhang, Shumin Deng, Mosha Chen, Chuanqi Tan, Fei Huang and Huajun Chen. “Contrastive Triple Extraction with Generative Transformer.” AAAI (2021).\n[3]. Lin, Bill Yuchen, Dong-Ho Lee, Minghan Shen, Ryan Rene Moreno, Xiao Huang, Prashant Shiralkar and Xiang Ren. “TriggerNER: Learning with Entity Triggers as Explanations for Named Entity Recognition.” ACL (2020).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster