Alignment Precedes Fusion: Open-Vocabulary Named Entity Recognition as Context-Type Semantic Matching
Authors: Zhuoran Jin
Pengfei Cao
Zhitao He
Yubo Chen
Kang Liu
Jun Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: Open-Vocabulary
Named Entity Recognition

Abstract: Despite the significant progress in developing named entity recognition models, scaling to novel-emerging types still remains challenging in real-world scenarios. Continual learning and zero-shot learning approaches have been explored to handle novel-emerging types with less human supervision, but they have not been as successfully adopted as supervised approaches. Meanwhile, humans possess a much larger vocabulary size than these approaches and have the ability to learn the alignment between entities and concepts effortlessly through natural supervision. In this paper, we consider a more realistic and challenging setting called open-vocabulary named entity recognition (OVNER) to imitate human-level ability. OVNER aims to recognize entities in novel types by their textual names or descriptions. Specifically, we formulate OVNER as a semantic matching task and propose a novel and scalable two-stage method called Context-Type SemAntiC Alignment and FusiOn (CACAO). In the pre-training stage, we adopt Dual-Encoder for context-type semantic alignment and pre-train Dual-Encoder on 80M context-type pairs which are easily accessible through natural supervision. In the fine-tuning stage, we use Cross-Encoder for context-type semantic fusion and fine-tune Cross-Encoder on base types with human supervision. Experimental results show that our method outperforms the previous state-of-the-art methods on three challenging OVNER benchmarks by 9.7%, 9.5%, and 1.8% F1-score of novel types. Moreover, CACAO also demonstrates its flexible transfer ability in cross-domain NER.

Alignment Precedes Fusion: Open-Vocabulary Named Entity Recognition as Context-Type Semantic Matching
Zhuoran Jin Pengfei 12 Zhitao Hel Yubo Chen 1.2 Kang Liu Jun Zhao 1,2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 2 The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China { Izhuoran. jin, pengfei. cao, zhitao, he @nnpr. ia. ac. cn [yubo. chen, kliu, jzhao) ) @nlpr aa.ac.cm cn
Abstract
Thaining
Barack Obama born Honoluls Hawail
Despite the significant progress developing named entity recognition models, scaling to novel-emerging types still remains challenging in real-world scenarios. Continual learning and zero-shot learning approaches have been ex- plored to handle emerrigg types with less human supervision, but they have been as successfully adopted as supervised approaches. Mcanwhile, humans possess a much larger vo- cabulary size than these approaches and have the ability to learn the alignment between en- tities and concepts effortlessly through natural supervision. In this paper, we consider more realistic and challenging setting called open- vocabulary named entity recognition (OVNER) to imitate human-lewe level ability. OVNER aims to recognize entities in novel types by their textual names or descriptions. Specifically, late OVNER as semantic matching task and propose a novel and scalable two-stage method called Context-Type SemAntiC Alignment and FusiOn (CACAO). In the pre-training stage, we adopt Dual- -aal-En for context- -tpee semantic 80M
B-PER PER
B-LOC B-LOC
Inforence
Elon Musk was born Pretonaa South Afrcc
B- PER I-PER O
O B-LOC B-Loc IoLo
[e] As exarmpii suparvisat NER wdtinng
Pre training
And Then There W ere None is novel by Agatha Christie.
And The T here Were None
Agatha Chriete
Maakke
201laaiay bay Engish arndaa
Edarr Awad wizace Pag 1ch womee drnnber
lirtek wine setggted rets plays
Fine- tuming
Barack Ohama was Hanolulú, Hawaill
PERSON
LOCATION
snall
uood dexite
bela csciidly and plosiallo...
milite in Eattle serbice chowlene
formu-
Inference
Messi captained A Acpentinato to win tiee 2022 FIFA World Cup
LOCATION
PERSON/ATHLETT
EVENT SSPORTS
In Engengaalee Incatin
An ohiid peiers compuie tea more zoant trodiee
mudi spent erpanaed speriing cous.
or glace as usok
dunuts auqioo
diftives
alignment and pre-train Dual-Encodes context-type pairs which are easily accessible through natural supervision. In the fine-tuning stage, we use cooss-Encoderreee for context-typp type semantic fusion and fine-tune base types with human supervision. Ex- perimental results show that our method out- performs the previous state-of the-art methods on three challenging OVNER benchmarks by 9.7%, 9.5%, and 1.8% Fi-score of novel types. Moreover, CACAO also demonstrates its flexi- ble transfer ability in cross-domain NER.
(b) An manple open-socaballyy NER setting
WikipedaaAacher
Wilipodia Page
Base Types
Novel Types
Figure I: Difference between (a) supervised NER set- ting and (b) open-vocabulary NER setting.
such as PERSON, LOCATION, ORGANIZATION etc. Despite its challenging nature, recent methods have achieved high performance on several super- vised benchmarks like CoNLL03 (Tjong Kim Sang and De Meulder, 2003) and OntoNotes 5.0 (Prad- han et al., 2013). However, practical challenge remains in the real-world applications of NER. As depicted in Figure (a), in the supervised setting, traditional methods depend the pre-defined en- tity ontology and limited annotated training data. Therefore, these methods can only recognize a fixed number of entity types observed during the training stage. As novel entity types continually
1 Introduction
Named entity recognition (NER) is fundamental task in natural language processing, focusing on locating named entities mentioned in unstructured text and classifying them into pre-defined types,
'Our code and data is publicly available at https:// gi gitbub... com/ ninzhuc
huhraaa an /OVNER/

---

emerge in real scenarios, it becomes non-trivial problem for NER models to handle the novel- emerging types on the fly inference time.
supervision and facilitating human-computer inter- action. As shown Figure 1 (b), we devise a two- stage framework for OVNER. the pre-training stage, based on Wikipedia articles and anchors, we first construct a large-scale pre-training corpus called Wikipair80M. Wikipair80M contains 80M context-type pairs which are freely available on Wikipedia. Then, we pre-train the recognizer to expand its vocabulary, simulating human learning by natural supervision. In the fine-tuning stage, we further fine-tune the recognizer on base types (e.g., LOCATION) with costly annotations to real- ize open-vocabulary recognition. These two stages enable the recognizer to generalize novel types (e.g., EVENT/ without any annotation. More specifically, we formulate OVNER as semantic matching task and propose novel two-stage method called Context-Type SemAntiC Alignment and FusiOn (CACAO). In the pre- training stage, we adopt Dual-Encoder (two en- coders) architecture for efficient training on large- scale, easily accessible context-type pairs. For context-type semantic alignment, embeds entity context and type descriptions into a universal dense space independently via in-instance and in-batch contrastive learning. In the fine-tuning stage, we employ Cross-Encoder (single encoder) architecture for effective tuning on limited NER an- notations. For context-type semantic fusion, Cross- Encoder captures the intricate interactions between
To solve the challenge of novel-emerging types, a straightforward approach is to collect more train- ing examples for new types and then re-train the model from scratch. The large cost of data col- lection and model re-training limits the usage of this approach. To tackle this problem, Cao al. (2020) and Monaikul et al. (2021) propose con- tinual learning approaches for sequence labeling, which require the model to learn new types incre- mentally without forgetting previous types. Contin- ual learning approaches are inefficient due to the requirement of collecting enough annotated data for new types. Wang et al. (2022) propose contin- ual few-shot NER reduce the need for annotated data. Hence, Aly al. (2021) propose more effi- cient zero-shot learning approach for NER, which can recognize unseen entity types without any an- notated data. However, zero-shot learning methods learn a limited set of base types and struggle to generalize to novel types in practical applications. There is large performance gap between the cur- rent zero-shot NER and supervised NER models. Thus, existing zero-shot NER unpractical.
Compared to the limited types of the above ap- proaches, humans use thousands of vocabulary to describe the entity types. In cognitive psychology, humans can learn to connect visual objects and se- mantic categories effortlessly through natural super- vision, so they have the ability to recognize unseen categories with a wide-rane renging vocabulary (Lake et al., 2017: Zareian et al., 2021). Inspired by this, we argue that humans can expand their vocabulary by reading millions of articles, and learn the align- ment between entities and concepts through natural supervision. As illustrated in Figure 1 (b), take the sentence "And Then There Were None is novel by Christie." in Wikipedia article as example. We can connect entity context "Christie" and de- scriptive concepts (e.g., "British detective fiction writers"") of the entity "Agatha Christie" through the anchor (i.e., hyperlink) naturally.
context and all candidate types based on the repre- sentations encoded by Dual-Encoder. Experimental results on three challenging OVNER benchmarks demonstrate that approach significantly outper- forms previous state-of-the-ard methods. Our contributions are summarized as follows:
To solve the challenge of novel-emerging types, propose more realistic and chal- lenging setting called open-vocabulary named entity recognition (OVNER), which aims recognize entities in any novel type by their names or descriptions.
To imitate human-level ability, we formulate OVNER semantic matching task and pro- pose a two-stage method called Context-Type SemAntiC Alignment and FusiOn (CACAO), which enables the model to learn from both natural and human supervision.
To imitate human-level ability, this work con- siders more realistic and challenging setting called open-vocabulary named entity recognition (OVNER). Open-vocabulary setting requires that the model can recognize entities in any novel type by their textual names or descriptions. is an essential step toward reducing laborious human
To evaluate the effectiveness of CACAO, we conduct thorough experiments and show that

---

our method outperforms the previous state-of- the-art methods on three challenging bench- marks by 9.7%, 9.5%, and 1.8% Fi-score of novel types. CACAO also demonstrates its flexible transfer ability in cross-domain NER.
Architecture
We use Dual-Encoder architecture to model context-type pairs similar entity linking (Wu et al., 2020; Zhang et al., 2022a,b; Leszezyn- ski et al., 2022). As shown in Figure 2 (a) and (b), Dual-Encoder consists of two independent trans- former encoders, called Context-Eneoder Ectxt and Type--Enodder Esype. Both input context and type description are encoded into vectors:
2 Task Definition
this section, we formulate open-vocabulary named entity recognition task as follows. At the pre-training time, large-scale context-type pairs (c, t) are available. At the fine-tuning time, the model can only access the annotated data of base types TB but need to expand the vocabulary to rec- ognize novel types TN at test time. Since To O Tv 0, it is very challenging for model gener- alize to unseen types. Each type E TBUTN has its name and detailed description. Given a sequence of n tokens x = { .2..... . En NER model aims to assign each token 2; to its cor- responding tag label y, v/ ...,, v/ [Tl-ITx/), where v/ E {0, } denotes whether the i-th token belongs to the j-th type.
[b.,1: ho,2, ..., hanl Extxt (xe)
(1)
h, = red (Eeypo (2:)):
(2)
where H, and æt are token sequences of context and type respectively. Specifically, we construct the input of context as:
[CLS] context [SEP]
where context are word-pieces tokens of input con- text. We construct the input type as:
as the context c, the
[CLS] name [SEP] description [SEP]
where and description word-pieces tokens of type name and description. Ectxt and Etype are two pre-trained language models (Sun et al., 2022; Zhao et al., 2023), such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). red (-) is a pooling function that transforms multiple vectors into one vector. We experiment with three red (.) functions: using the output vector of the [CLS] token, using the output vector of the name token, and computing the mean of all output vectors.
3 Methodology
Most prior studies solve supervised NER via the sequence labeling paradigm, which conducts multi- class classification on each token in the sentence. However, the sequence labeling paradigm converts all types into indices directly, ignoring the seman- tie information of the types. Therefore, we formu- late OVNER as a semantic matching task. More specially, we propose a two-stage method called Context-Type SemAntiC Alignment and FusiOn (CACAo) to solve OVNER problem, which illus- trated in Figure 2. CAcAO consists of context-type semantic alignment pre- pre-training stage (Section 3.1) and open- vocabbuacyy named entity recognition fine- tuning stage (Section 3.2). We will detail the model architectures, score functions, and optimization ob- jectives of these two stages.
Scoring The similarity score between the token in context c and the type is computed as:
h h sim (bey he) [hell [
(3)
Optimization For efficient pre-training, we op- timize Dual-Encoder by context-type contrastive representation learning. Contrastive learning (Chen et al., 2020; Gao al., 2021) aims to learn effec- tive representation by pulling positive examples together and pushing apart negative examples. We argue that (1) the token semantics are determined by its context; (2) the token and its type should have the same semantics. Therefore, we can treat all the context-type pairs as positive pairs 1 where Te. denotes the token in the context be- longs to the type I. As shown in Figure 2 (a) and (b), devise two contrastive strategies: (1) in- instance contrastive pre-training: we treat all the tokens in the context c except Øe,+ as negative examples. Take Figure (a) as an example, the
3.1 Context-Type Semantic Alignment Pre-training
In the context-type semantic alignment pre-training stage, we first construct large-scale pre-training corpus based on Wikipedia articles and anchors. The corpus contains 80M context-type pairs which are freely available on Wikipedia. Then, we pre- train the recognizer to expand its vocabulary, simu- lating human learning by natural supervision for a rich and complete context-type semantic space.

---

Dallee Negatee
Staakr Seymenbrinnn
Mesoi
Arpenitaa....
FIBA
Contert Enooder
Type Eacoder
+
M cxplanned guntine PerseniA thiete aliledo Lo 2022 PISX World
sports
Cruss Encuder
phystcal snerght...
CmssAraiion
(e] In instanee Contrative Pre training
Mesii
Person/AAthoe
OoO
O
O
O
Musoi
Arganiina
FIFA
thilee
LecationGPP
Eventi
Barack Obama
O
O
Person/Pollttrtaa
Type Enooder
Encodar
Contect Encoder
Type Eacodee
Mesd captained mgprinn win the 2022 FIRA Wodl Cup
Persen/Atblete As siblete
Location/G PE pergasphcal which is polliicl....
Fvuu//Fledion Az election focnal derision process waich popelation...
Mesaa capramed Acpentina the 2022 FIBA Wodld Cup.
[CLS] Personii thieee (SEP) istietess plasical
oee sports
competas spocts
OVNOOVEERRR NER-spiiiiic Fine tuning
Barack Obama Hondolls Hovell
(CLS) Person Politiciin [SEP]A politicion a person octive parts pollico,
person holding seeking an elected...
(b) Ir-batch Coutrastive Pre training
Figure 2: The two-stage framework of our proposed CACAO for open-vocabulary named entity recognition, which consists context-type semantic alignment pre-training stage (Left Figure (a) and (b)) and open-vocabulary named entity recognition fine-tuning stage (Right Figure (c)).
type Person Cerson/Athlete Athlete the anchor, the token "Messt" is a positive example, and the other tokens in the context "Messi captained Argentina to win the 2022 FIFA World Cup." are negative examples. The training objective of in-instance contrastive pre-training becomes:
Then, we can get the final optimization goal of the pre-training stage:
Epre li Hi + HeCrat
(8)
where 141 and μl2 are hyper-parameters.
To encourage the model to learn entity semantics from context and prevent the model from relying
c ha,
Linst -log
En ch be
on superficial statistical cues. We randomly mask the entity tokens in context with [MASK] token in probability p.
where denotes the temperature hyperparameter; (2) in-batch contrastive pre-training: con- struet the in-batch negative examples similar Li et al. (2022d). Take the j-th instance in mini- batch of m instances, the training objective of in- batch contrastive pre-training becomes:
Data Collection need collect large-scale, easily accessible context-type pairs (c, L). We collect the pre-train corpus through KILT (Petroni et al., 2021), which is based on the 2019/08/01 Wikipedia snapshot and contains 5.9M Wikipedia pages. Each Wikipedia page is assigned: (1) a unique Wikipedia id; (2) a list of paragraphs; (3) list of anchors: (4) list of categories. Each anchor refers the entity mention in the text, which can be linked its Wikipedia page. Each category a descriptive concept (defini- tion) of an entity. For example, the entity "Agatha Christie" is defined as "Murder-suicide fiction".
For CACAO pre-training, we
sim h2 b/)/7
Lac = - -log
(5)
Dm sim bi b/t
siml h/) (e..bb, D/T
Ces
m -log
sim simmmi, (I..
1==]
e
Ciat
(Cr2e Lo2)).

---

"English women dramatists" and "British novels adapted into films". Therefore, we can naturally connect the entity mentions contexts and descrip- tive concepts in Wikipedia pages through anchors. To collect more entity types, we also link the entity to Wikidata (Vrandečié and Krötzsch, 2014) via its Wikipedia id. For each entity, We use the instance of and subclass of property values its corresponding entity types (Chen et al., 2022). In this way, we automati- cally construct a large-scale pre-training corpus called Wikipair80M, containing 80M context-type pairs, such as ("And Then There Were None is a novel by Christie.", British detectiv fiction writers). We also obtain a collec- tion of 120K entity types which can serve as the foundation for much larger vocabulary.
all the type representations [he, he. [TT encoded by Dual-Encoder as input:
[h......... ble hi............. PTTT
(9)
Perces ([hets-.. han ba ha/Ta))
The similarity score between the token in context and the type is computed as:
Scoring
score (hes h.) = sigmoid (sim (hc, ht) /T). (10)
Optimization Since None type is hard to de- scribe using natural language, we ignore the None type and formulate NER binary classification problem. Then, we fine-tune the model using the following binary cross-entropy (BCE) loss:
TTll Sest log Pe,
1
Dbox
LTB
3.2 Open-Voceet Vocabulary Named Entity Recognition Fine-tuning
i=1 j=l
(11)
+ (1- v2.s) log (1-Pes. Pecs)
the open-vocabulary named entity recognition fine-tuning stage, we first fine-tune the recognizer on base types TB with human annotations. Then, we expand the type query embeddings to include novel types TN for OVNER at inference time.
where P', score (h hiss) denotes the prob- ability of the i-th token in context c belongs to j-th type, vi, E {0, l} denotes the ground truth whether the i-th token in context belongs to the j-th type.
Architecture Fuul--Encoder computationally efficient for pre-training because it encodes the in- put of context and type independently. However, Dual-Encoder ignores the interactions between con- text and type. Li et al. (2020) and Aly et al. (2021) formulate NER as MRC-style task and encode the input context and entity types jointly. At infer- ence time, every type must be concatenated with the input context and fed into the model through a forward pass. Due to the high time complexity of inference, this method may not work for hundreds of types in real-time systems.
At inference time, we predict the type te of the i-th token as follows:
Inference
argmax(Pos). if max(pea) 6 None,
16,1 =
otherwise,
(12)
where d is threshold hapereparameter.
Experiments
4.1 Evaluation Benchmarks and Metries
We employ ture the rich interactions between context and all candidate types efficiently and effectively. Re- cently, cross-attention modules have been success- fully used to learn cross-modal interactions in multi-modal tasks (Wei et al., 2020; Kim et al., 2021; Li et al., 2021b), motivating us to leverage cross-attention modules for exploring context-type interactions. As shown in Figure 2 (c), Cross- Encoder Erross is based Context-Encodes Ertxt and Type-Ennoder Eype- Cross-Encoder is com- posed of stack of k = cross-attention layers and forwafo networks. context representations [h,,1 h ..,.....hh.. henl and
architecture to cap-
Evaluation Benchmark So far, there is benchmark for evaluating OVNER models. There- fore, we adapt OntoNotes 5.0 (Pradhan et al., 2013) and FEW-NERD (Ding et al., 2021) datasets into the OVNER setting. OntoNotes 5.0 is a widely used NER dataset containing eleven entity types and seven value types. Few-NERD is the largest few-shot NER dataset, which contains INTRA and INTER two sub-settings with cight coarse-grained and 66 fine-grained types. Following the previ- ous zero-shot NER method (Aly al., 2021), we construct three challenging OVNER benchmarks, namely OV-OntoNotes, OV-NERD-INTRA, and OV-NERD-INTER, Within these benchmarks, we
takes the

---

OV-OntoNotes
OV-NERD-INTRA
OV-NERD-INTER
Method
PLM
Base
Novel
All
Base*
Novel
All
Base
Novel
All
Baseline Methods
MRC
BERT-base- SERRT-baee-SQuree SQUAD
80.5
11.7
36.7
19.4
19.4
66.2
10.1
40.7
MRC
DERT-Iarge-SQUar
81.3
14.4
38.7
21.3
21.3
67.5
11.5
42.0
BEM
BERT-base-MNLL
78.6
14.8
38.0
16.6
16.6
64.4
9.4
39.4
BEM
BERRT-arge-MNIJ
79.8
17.2
40.0
18.7
18.7
65.2
10.3
40.3
SMXM
BERT-base
79.7
18.5
40.8
20.2
20.2
66.9
11.2
41.6
SMXM
BERT-large
80.8
21.8
43.3
23.4
23.4
68.0
13.4
43.2
Our Method
CACAO
BERT-base
82.7
23.6
45.1
33.1
33.1
74.2
22.9
50.9
Table I: Experimental results (macro-averaged Fl-score, %) of different methods on OV-OntoNotes, OV-NERD-
INTRA OV-NERD-INTER benchmarks. Bold denotes results. indicates that test set no base types, since the entity types are mutually disjoint in the training and test set of OV-NERD-INTRA.
OV-OntoNootes Novel
OV-NERD-INTRA
DV-NERD-INTER
Method
Base
All
Base
Novel
All
Base
Novel
All
CACAO
82.7
23.6
45.1
33.1
33.1
74.2
22,9
50.9
w/o pre-training
82.3
0.0
29.9
14.5
14.5
54.9
6.8
33.0
w/o in-instance
81.9
21.6
43.5
32.1
32.1
19.7
49.4
w/o in-batch
81.6
20.7
42.9
31.7
31.7
72.9
20.1
48.9
w/o cross-encoder
80.7
25.4
45.5
30.4
30.4
73.0
19.4
48.6
w/ self-attention
82.5
21.0
43.4
31.3
31.3
74.0
22.3
50.5
Table 2: Ablation study of CACAO on OV-OntoNotes, OV-NERD-INTRA and OV-NERD-INTER benchmarks.
consider those types seen in the training set base types. The test may contain both novel types and base types, depending on the specific bench- mark setting. The statistics and type splits of the three benchmarks are shown in Appendix A.
shot text classification textual entailment task and classify whether class description is entailed by the text. Following Aly al. (2021), we adjust BEM for OVNER. We design template "This text is about ... to convert each entity type into a hypothesis, then treat the target mention in context as the premise, and ask the BEM model whether the target mention can entail an entity type. We adopt BERT-base and BERT-large fine-tuned on MNLI (Williams et al., 2018) dataset.
Evaluation Metric We consider macro-averaan macro-averaged Fl-score to evaluate the performance due to the im- balance in annotated samples per type. We experi- ment with three random seeds and report the per- formance of base types, novel types and all types.
(3) SMXM: Aly et al. (2021) present the first approach for zero-shot named entity recognition. SMXM uses the cross-attention encoder to model the sentence and the entity type descriptions and explores three approaches to modeling None type. We adopt BERT-base and BERT-large for SMXM.
4.2 Baseline Methods
We compare the proposed approach CaCAo with the following methods:
(1) MRC: Li et (2020) formulate the NER task as machine reading comprehension task and construct natural language queries for entity types. For example, extracting entities with the Person type is transformed extracting answer spans to the question "which person is mentioned in the texi?". We adopt BERT-base and BERT-large fine- tuned on SQuAD (Rajpurkar al., 2016) dataset.
All three baseline methods and our CACAO need to be fine-tuned base types. The implementation details of CACAO shown in Appendix B.
4.3 Overall Results
Table shows the results of different methods on OV-OntoNotes, OV-NERD-INTRA and OV- NERD-INTER benchmarks. We note the following
(2) BEM: Yin et al. (2019) propose to treat zero-

---

Bukkigg Event
Locatinn
Organted.
Evont
Location
Cehar
Product
23
50
23
30
10
Figure 3: Fl-scores of different novel entity types on
Figure 4: Fl-scores of different novel entity types on OV-NERD-INTER.
LENERR-INNRA,
Pooler
OV-NERD-INTRA
OV-NERD-INTER
OV-NERD-INTRA Novel
OV-NERD-INTER Novel
Desc
red[ (-)
Base
Novel
All
Base
Navel
All
Base
Base
All
[C5S]
27.3
27.3
70.4
18.8
46.9
Wiki
31.1
31.1
74.2
22.9
50.9
name
22.6
22.6
67.5
13.6
43.0
WordNet
23.6
23.6
66.4
12.3
41.8
31.1
31.1
74.2
22.9
50.9
GPT-3.5
31.8
31.8
71.7
19.7
48.1
Table 3: Effects of different pooling functions.
Table 4: Effects of different type descriptions.
key observations throughout our experiments:
4.4 Ablation Results
(1) Our method outperforms all the baseline methods by a large margin and achieves new state- of-the-art performance on three benchmarks. More specially, compared with the previous state-of- the-art model SMXM, our method achieves 1. 1.8%, 9.7%, and 9.5% improvements of Fl-score of novel types on OV-OntoNotes, OV-NERD-INTRA and OV-NERD-INTER benchmarks, respectively. Ex- perimental results demonstrate that our proposed CACAO is very effective for the open- open-vocabulary named entity recognition task.
To demonstrate the effectiveness of our proposed CACAO, we conduct ablation studies as follows: 1) w/o pre-training, which removes the entire pre-training stage; 2) w/o in-instance, which re- moves in-instance contrastive learning from the pre-training stage; 3) w/o in-batch, which removes in-batch contrastive learning from the pre-training stage; 4) w/o cross-encoder, which removes Cross- Encoder from the fine-tuning stage; 5) w/ self- attention, which replaces the cross-attention lay- ers with the self-attention layers in Cross-Encodes From Table we have the following observations: (1) We observe significant performance drop of 23.6%, 18.6%, and 16.1% in the Fl-score of CACAO when dealing with novel types without pre-training. This highlights the effectiveness of alignment pre-training in improving generaliza- tion to unseen types. We also find that in-batch contrastive pre-training is more efficient than in- instance contrastive pre-training. The reason may be that in-instance contrastive pre-training will in- troduce false negative examples in instance.
(2) Our CACAO performs well not only on novel types but also on base types. indicates that CA-
CAO can learn rich and complete context-type se- mantic space, and expand its vocabulary from large- scale pre-training. Furthermore, we also observe that the performance improvement of CACAo is more significant on OV-NERD-INTRA compared OV-OntoNotes. This suggests CacAo has effec- tive fine-grained entity type recognition ability.
(3) MRC or BEM design template to convert each entity type name into question a hypoth- esis. Compared with them using type names, our CACAO and SMXM adopting entity type descrip- tions achieve better performance. The reason is that type descriptions can help the model understand the meaning of types and generalize to new types.
(2) When we remove Cross -noos-E, the perfor- mance drops significantly on OV-NERD-INTRA and OV-NERD-INTER, but improves on OV- OntoNotes. This phenomenon may be attributed to the fact that Few-NERD has much more entity

---

Method
PLM
Politics
Science
Music
Literature
AI
Average
LST-NERT
BERT-bass
70.44
66.83
72.08
67.12
60.32
67.36
LANERI
BERT-base
71.65
69.29
73.07
67.98
61.72
68.74
CP-NER
T5-base
73.41
74.65
78.08
70.84
64.53
72.30
LST-NER w/ DAPTI
BERT-base
73.25
70.07
76.83
70.76
63.28
70.84
LANER w/ DAPTI
BERT-base
74.06
71.83
78.78
71.11
65.79
72.31
CP-NER w/ DAPTI
T5-base
76.35
76.83
80.28
72.17
66.39
74.40
CACAO (Ours)
BERT-base
81.26
77.78
82.60
75.73
68.95
77.26
Table 5:
Experimental results (miero-averaged Fl-score, %) of different methods on CrossNER dataset. i indicates the results taken from the original paper (Chen al., 2023b).
types than OntoNotes. Cross-Encoder works well with larger number of types, capturing the intri- cate interactions between context and types. In ad- dition, find that the cross-attention mechanism performs better than the self-attention mechanism.
description generated by text-davinci-003 Table 4 demonstrates that Wiki, with its well- crafted descriptions, can serve strong baseline compared WordNet. Meanwhile, using GPT- 3.5 to automatically generate descriptions can also achieve good performance and reduce human labor. Please refer Appendix D for entity descriptions.
4.5 Discussion and Analysis
Performance of Different Entity Types We an-
4.6 Transfer Ability
alyze the performance of different entity types OV-NERD-INTRA and OV-NERD-INTER. As illustrated in Figure 3 and 4, we find that CA- CAO achieves good performance on Person and Building types and yields a bad perfor- mance on Event: type. Moreover, CACAO can barely recognize GPE, ShowOrganizat i in and Election types. A possible explanation is these types are difficult to define and describe in single sentence. For instance, deseribe GPE "gcograahiccl/scciapppliica entity, including countries, cities, states". Therefore, a good type description is crucial for recognizing novel types.
We also demonstrate the transfer ability of our CA- CAO in cross-domain NER. We conduct experi- ments on the CrossNER dataset (Liu et al., 2021). Following the previous studies (Zheng et al., 2022; Hu al., 2022: Chen et al., 2023b), we adopt CoNLL 2003 (Sang and De Meulder, 2003) as the source domain which contains four types: PER, LOC, ORG, and MISC. Then, we use CrossNER as the target domain which contains five separate domains: politics, natural science, music, litera- ture and Al. Table 7 presents the detailed statistics of these datasets. To evaluate the effectiveness of CACAO, we compare it with several cross cross-domain NER baselines, including:
Impact of Entity Type Representations We ex- periment with three red (-) pooling functions to represent the entity types: 1) [CLS]: using the output vector of the [CLS] token; 2) name: us- ing the output vector of the name token; 3) mean: computing the average of all output vectors. As illustrated in Table 3, mean can provide more com- prehensive information about entity types, leading to better performance than [CLS] and name.
(1) DAPT: Liu et al. (2021) propose domain- adaptive pre-training (DAPT), which leverages a large domain-related corpus pre-train the model, similar the first-stage pre-training of our method. (2) LST-NER: Zheng et al. (2022) build label graphs in both the source and target label spaces based on BERT-base for cross-domain NER.
(3) LANER: Hu al. (2022) propose an auto- regressive cross-domain NER framework based on BERT-base to improve label information transfer. (4) CP-NER: Chen al. (2023b) introduce col- laborative domain-prefix tuning for cross-domain NER based on T5-base (Raffel et al., 2020), which obtains the state-of- ofate-of-the-ar performance on the
Impact of Entity Type Descriptions We also analyze the effect of entity type descriptions by conducting experiments with three different type descriptions: 1) Wiki: the most relevant sentence describing the type that we carefully selected on Wikipedia; 2) WordNet: the type definition in WordNet (Miller, 1995); 3) GPT-3.5: the type
CrossNER dataset.

---

To make fair comparison, first train on the source domain with sufficient data, and then adapt to the target domain with a very small amount of data. As shown in Table 5, CACAO can consis- tently obtain better performance than the previous statc-of-the-ar method CP-NER w/ DAPT across all target domains, with an average 2.86% Fl-score improvement. Experimental results demonstrate the effectiveness of our OVNER method under cross-domain NER scenarios. We think there are several reasons for the remarkable results:
training samples. Aly et al. (2021) present the first approach for zero-shot named entity recognition and leverage the entity type descriptions to transfer knowledge from observed to unseen classes. Never- theless, existing ZSNER methods are still far from practical performance due to their unnecessarily harsh constraint. In this paper, we consider more realistic setting, in which not only the annotated samples of base types but also large number of low-cost context-type pairs are available.
(1) CACAO can learn complete context-type semantic space, and expand its vocabulary from large-scale pre-training. Pre-training is essential to improve the domain generalization of our model.
5.3 Open-Vocabulary Object Detection
Open-vocabulary object detection (OVD) (Zareian et al., 2021; Zang et al., 2022; Li et al., 2022b; Gu et al., 2022; Zhou et al., 2022; Xu et al., 2022; Kuo et al., 2022), which is concerned with the prob- lem of detecting novel objects guided by natural language, has gained increasing attention from the community. Zareian al. (2021) first propose the open-vocabulary object detection task to bridge the performance gap between zero-shot learning and supervised learning. They first pre-train the model on image-caption pairs and then fine-tune the model on downstream detection datasets. There are some studies (Li et al., 2021c, 2022a.c: Jiao et al., 2022; Lu et al., 2022a) in the NLP commu- nity that use indirect supervision to alleviate the data scarcity issue. Inspired by these outstanding works, we propose to solve open- apen-vocabulary named entity recognition with context-type semantic align- ment, further improving the zero-shot performance on novel-emerging entity types.
(2) CACAo leverages text descriptions to rep- resent entity types, which can contain more type semantic information. We guess entity descriptions can better realize the transfer of knowledge from the source domain to the target domain.
5 Related Work
5.1 Supervised Named Entity Recognition
Named entity recognition (NER) a long-standing study in natural language processing. Traditional approaches for NER are based on sequence label- ing (Ju et al., 2018; Wang et al., 2020), assigning a single tag to each word in a sentence. To han- dle overlapping entities in nested NER, span-based methods (Luan et al.. 2019; Fu et al., 2021; Li et al.. 2021a; Shen et al., 2021: Chen et al., 2023a) first extract all candidate spans and then classify these spans. Besides, Li et al. (2020) and Shen et al. (2022) propose to reformulate the NER task as a machine reading comprehension (MRC) task. In addition, some methods (Yan et al., 2021; Ma et al., 2022; Chen et al., 2022: Lu et al.. 2022b) attempt to formulate NER as an autoregressive sequence gen- eration task and achieve promising performance. Das al. (2022) propose to solve few-shot NER via contrastive learning. Recently, Zhang et al. (2023) present bi-encoder framework, which sep- arately maps text and entity types into the same vector space. These methods have achieved high performance in the supervised setting, yet they all require costly annotations. Hence, it is hard for them to handle novel-emerging types.
6 Conclusion
In this paper, we consider more realistic and chal- lenging setting called open-vocabulary named en- tity recognition (OVNER) to imitate human-level ability. We formulate OVNER as a semantic match- ing task and propose two-stage method called Context-Type SemAntiC Alignment and FusiOn (CacAo). In the pre-training stage, we first con- struct a corpus containing 80M context-type pairs, then pre-train Dual- Encoder through natural super- vision for context-type semantic alignment. In the fine-tuning stage, we fine-tune Cross- Encoder base types with human supervision for context-type semantic fusion, so that the model can recognize novel types by their descriptions. Experimental re- sults show that our method outperforms the state-of- the-art art methods on three challenging benchmarks.
5.2 Zero-Shot Named Entity Recognition
Zero-shot named entity recognition (ZSNER) aims to recognize entities novel types without any

---

Limitations
Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1516-1522 Online. Association for Computational Linguisties.
Although our approach has worked well on open- vocabulary named entity recognition task, there are still some limitations to be resolved in the future: (I) The core idea of OVNER is bridging the gap between zero-shot NER and supervised NER by learning from natural supervision. In this paper, we pre-train the recognizer on large-scale context-type pairs as natural supervision. Despite its effective- ness, pre-training requires significant amounts of computational resources. In the future, We will ex- plore general way to leverage natural supervision; (2) In this paper, solely evaluate our method CaCAo on several OVNER benchmarks. In fact, our method can be adapted to solve all kinds of entity-related information extraction tasks, includ- ing named entity recognition, entity typing, entity clustering, and entity linking. We leave it as an open question for future work. (3) Our method requires the provision of textual descriptions to rep- resent entity types. Although we think it may be an easier way to provide collect entity descrip- tions for each type. We believe that alleviating the limitation of type descriptions could be further extension of our work in the future.
Pengfei Cao, Yubo Chen, Jun Zhao, and Taffeng Wang. 2020. Incremental event detection via knowledge consolidation networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 707-717, Online. Association for Computational Linguisties.
Jiawei Chen, Qing Liu, Hongyu Lin, Xianpei Han, and Sun. 2022. Few-shot named entity recognition with self-describing networks. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 5711-5722, Dublin, Ireland. Association for Compu- tational Linguistics.
Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, and Le Sun. 2023a. Learning in-context learn- ing for named entity recognition. arXiv preprint 2005 1038..
Ting Chen, Simon Kornblith, Mohammad Norozzi, and Geoffrey Hinton, 2020. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1507-1607. PML.R
Ethics Statement
Xiang Chen, Lei Li, Qiaoshuo Fei, Ningyu Zhang, Chuanqi Tan, Yong Jiang, Fei Huang, and Huajun Chen. 2023b. One model for all domains: collab- orative domain -neafi tuning for cross-domain ner. arXiv preprint arXiv:2301. 231.:2301110410..
To ensure the reproducibility of our paper, we will release all source codes, large-scale pre-training corpus, benchmark datasets, and all trained check- points upon the acceptance of this paper.
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. 2022. CONTaiNER: Few-shot named entity recognition via contrastive learning. Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6338-6353, Dublin, Ireland. Association for Computational Linguistics.
Acknowledgements
This work is supported by the National Key Re- search and Development Program of China (No. 2020A A010440006400), the National Natural Science Foundation of China (No. 6197621162176255 )
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers). As- sociation for Computational Linguistics.
This work is also supported by the Strategic Pri- ority Research Program of Chinese Academy of Sciences (Grant NoAX022700000 ), the Youth In- novation Promotion Association CAS, and Yunnan Provincial Major Science and Technology Special
Plan Projects (No.202202AA0000004))
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang. Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition datasct. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 3198-3215, Online. Association for Computational Linguistics.
References
Rami Aly, Andreas Vlachos, and Ryan MeDonald. 2021. Leveraging type descriptions for zero-shot named entity recognition and classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ih International

---

Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. SpanNER: Named entity re-frecognition as span pre- diction. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 7183-7195, Online. Association for Computa- tional Linguistics.
Megan Leszezynski. Daniel Fu, Mayee Chen, and Christopher Re. 2022. TABi: Type-aware bi- encoders for open-domain entity retrieval. In Find- ings of the Association for Computational Linguis- fics: ACL 2022, pages 2147-2166, Dublin, Ireland. Association for Computational Linguisties.
Bangzheng Li, Wenpeng Yin, and Muhao Chen. 2022a. Ultra-fine entity typing with indirect supervision from natural language inference. Transactions of the Association for Computational Linguistics, 10:607- 622.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceeding of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguisties.
Boyi Li, Kilian Q Weinberger. Serge Belongie, Vladlen Koltun, and Rene Ranftl. 2022b. Language-driven semantic segmentation. In International Conference on Learning Representations.
Xiuye Gu. Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. 2022. open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations.
Dongfang Li, Baotian Hu, and Qingeai Chen. 2022c. Prompt-based text entailment for low-resource named entity recognition. In Proceedings of the 29th International Conference on Computational Linguis- tics, pages 1896-1903, Gyeongju, Republic of Korea. International Committee on Computational Linguis- tics.
Jinpeng Hu, He Zhao, Dan Guo, Xiang Wan, and Tsung- Hui Chang. 2022. A label-aware autoregressive framework for cross-domain NER. In Findings of the Association for Compulational Linguistics: NAACL 2022, pages 2222-2232. Seattle, United States. Asso- ciation for Computational Linguistics.
Fei Li. ZhiChao Lin, Meishan Zhang, and Donghong Ji. 2021a. A span-based model for joint overlapped and discontinuous named entity recognition. In Proceed- ings the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Popers). pages 48 18-44828, Online. Association for Computational Linguistics.
Yizhu Jiao, Sha Li, Yiqing Xie, Ming Zhong, Heng Ji, and Jiawei Han. 2022. Open-vocabulary argument role prediction for event extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5404 5418, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recog- nition. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 1446-1459) New Orleans, Louisiana, Association for Computa- tional Linguistics.
Jiacheng Li, Jingbo Shang, and Julian McAuley. 2022d. UCTopie: Unsupervised contrastive learning for phrase representations and topic mining. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers), pages 6159-6169, Dublin, Ireland. Association for Computational Linguistics.
Wonjae Kim, Bokyung Son, and Iodoo Kim. 2021, Vilt: Dision-and-languaee transformer without convolu- tion region supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021b. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Sys- tems, volume 34, pages ciates, Inc.
PMLR. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Rep- resentations.
Curran Asso-
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5850 On- line. Association for Computational Linguistics.
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergio- vanni, and Anelia Angelova. 2022. F-ylm: Open- vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209. 156399
Zhucong Li, Zhen Gan, Baoli Zhang, Yubo Chen, Jing Wan, Kang Liu, Jun Zhao, and Shengping Liu. 2021c. Semi-supervised noisy label learning for chinese clin- ical named entity recognition. Data Intelligence. 3(3):389400
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenen- baum, and Samuel J. Gershman. 2017. Building ma- chines that learn and think like people. Behavioral and Brain Sciences, 40:e253.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 1rxxiv:907.110
Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards ro- bust linguistic analysis using OntoNotes. In Proceed- ings of the Sevenleenth Conference on Computational Natural Language Learning, pages 143-152, Sofia, Bulgaria. Association for Computational Linguistics.
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Zi- wei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021, Crossner: Evaluating cross- domain named entity recognition. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 13452-13460.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-tex: trans- former. The Journal of Machine Learning Research. 2185555485-5551
Keming Lu. I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2022a. Summarization as indirect supervision for relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6575-6594, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas, Association for Computational Linguistics.
Yaojic Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022b. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5755-5772, Dublin, Ireland. Association for Computational Linguistics.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independern named entity recognition. arXiv preprint cs/0306050.
Yi Luan, Dave Wadden, Lubeng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), J, pages 3036 3046, Minneapolis, Minnesota. Association for Computational Linguistics.
Yongliang Shen, Xinyin Ma. Zeqi Tan, Shuai Zhang, Wen Wang. and Weiming Lu. 2021. Locate and la- bel: A two-stage identifier for nested named entity recegnition.
Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Nutu- ral Language Processing (Volume I: Long Papers), pages 2782-277 2794, Online. Association for Computa- tional Linguistics.
Ruotian Ma, Xin Zhou, Tao Gui. Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2022. Template- free prompt tuning for few-shot NER. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5721-5732, Seattle, United States. Association for Computational Linguistics.
Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yuet- ing Zhuang. 2022. Parallel instance query network for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics |Volume 1: Long Papers), pages 947-961, Dublin, Ireland. Association for Computa- tional Linguisties.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM.
Miinn Sun, Xiang-Yane Liu, Xi-Peng Qiu, and Xuan-Jing Huang. 2022. Paradigm shift in natural language processing. Machine Intelligence Research. 19(3):16--183.
Natawut Monaikul, Giuseppe Castellucci, Simone Fil- ice, and Oleg Rokhlenko. 2021. Continual learn- ing for named entity recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 13(15)::13300113377
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Longuage Learning at HILT-NAACI. 2003, pages 142- 147.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human
Language-independee
Denny Vrandečié and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Communi- cations of the ACM, (7))))88--8

---

Jue Wang. Lidan Shou, Ke Chen, and Gang Chen, 2020. Pyramid: A layered model for nested named entity recognition. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 5918-5928, Online. Association for Computa- tional Linguistics.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang. and Chen Change Loy. 2022. Open- apen-vocabulary detr with conditional matching. arXiv preprint arXiv:2233.1188
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. 2021. Open-vocabulary object detec- tion using captions. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 14393-14402.
Rui Wang, Tong Yu, Handong Zhao, Sungchul Kim, Subrata Mitra, Ruiyi Zhang, and Ricardo Henao. 2022. Few-shot class-incremental learning for named entity recognition. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 5-82,, Dublin, Ireland. Association for Computational Lin- guistics.
Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. 2023. Optimizing bi-encoder for named entity recognition via contrastive learning. In The Eleventh International Conference on Learning Representa- tions.
Xi Wei, Tianzhu Zhang, Yan Li, Yongdong Zhang, and Feng Wu. 2020. Multi-modality cross attention net- work for image and sentence matching. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages (0941-10950)
Sheng Zhang, Hao Cheng. Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Nau- mann, Jianfeng Gao, and Hoifung Poon. 2022a. Knowledge-rin -rcch self superviinn for biomedical en- tity linking. Findings of the Association for Com- putational Linguistics: EMNLP 2022, pages 868- 880, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies. Volume 1 (Long Papers), pages l1 12-1 122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Wenzheng Zhang, Wenyue Hua, and Karl Stratos. 2022b. EntQA: Entity linking as question answering. In International Conference on Learning Representa- tions.
Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2023. Transformer: A general framework from machine translation others. Machine Intelligence Research, 20(4):514-538
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407, Online. Association for Computa- tional Linguistics.
Junhao Zheng, Haibin Chen, and Qianli Ma. 2022. Cross-domain named entity recognition via graph matching. In Findings of the Association for Com- putational Linguistics: ACL 2022. pages 2670-2680, Dublin, Ireland. Association for Computational Lin- guistics.
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. 2022. A simple baseline for open-wocabulary semantic segmentation with pre-trained dision-langual model. In Com- puter Vision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings, Part XXIX, pages 1336-753.4 Springer.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. 2022. Detecting twenty- thousndd classes using image-level supervi- sion. In Computer Vision ECCV 2022, pages 350- 368, Cham. Springer Nature Switzerland.
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, and Xipeng Qiu. 2021. A unified generative framework for various NER subtasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joins Conference on Natural Language Processing (Volume 1: Long Popers), pages 5808-5822, Online. Association for Computational Linguistics.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.

---

A Benchmark Details
Table presents the statistics of OV-OntoNotes, OV-NERD-INTRA and OV-NERD-INTER bench- marks. Table 7 presents the statistics of CoNLL 2003 and CrossNER datasets. Additionally, we also provide detailed type splits in Table 8, 9 and 10.
B Implementation Details
Our implementation is based on HuggingFace's Transformers library and PyTorch. We initial- ize Context-Encoder and Type-Encoder with pre- trained BERT-base. Then, initialize Cross- Encoder with the last three layers of pre-trained BERT-base. We use the Adam algorithm (Kingma and Ba, 2015) to optimize model parameters. The learning rate is initialized as le-5 and 5e-6 with a linear decay for pre-training and fine-tuning stages. The batch size is set to 100 and 30 for pre-training and fine-tuning stages. We also keep dropout of 0.1 and weight decay of 0.01. The hyperparame- ter temperature 7 is 0.05, 141 is set to 0.5, 12 is set to 0.3, mask probability p is threshold 5 essential for inference. In general, us- ing the default å = 0.5 can yield good performance. To achieve optimal performance, we tune the model on the development set with 5 {0.4, 0.5, 0.6). All experiments are conducted with eight NVIDIA GeForce RTX A6000 GPUs.
0.5. The
C Case Study
As depicted in Table 11, we perform a case study to validate the effectiveness of our method on the OV-NERD-INTRA benchmark. The results demon- strate that our approach excels in accurately clas- sifying entities. However, it faces challenges in precisely determining entity boundaries.
D Type Descriptions
We present detailed entity descriptions of the OntoNotes and FEW-NERD datasets used in this paper shown in Table 12 and 13.

---

OV-OntoNotes
OV-NERD-INTRA
OV-NERD-INTER
Statistics
Train
Valid
Test
Train
Valid
Test
Train
Valid
Test
Overall Types
35
66
# Novel Types
7
14
31
13
30
Table 6: Statistics of OV-OntoNotes, OV-NERID-INTR and OV-NERD-INTER benchmarks.
Statistics
CoNLL 2003
CrossNER
Domain
News
Politics
Natural Science
Music
Literature
Artificial Intelligence
# Train
15.0k
200
200
100
100
100
# Test
3.7k
651
543
456
416
431
# Entity Types
4
10
17
13
11
12
Table Statistics of CoNLL 2003 and CrossNER datasets.
Split
Types
Train
PERSON, GPE, ORG, DATE
PERSON, GPE, ORG, DATE, FAC, LOC, TIME, QUANTITY, CARDINAL,
Dev
WORK OF _AR,, LANGUAGE
PERSON, GPE, ORG, DATE, NORP, MONEY, ORDINAL, PERCENT, EVENT, PRODUCT, LAW
Test
Table 8: Type split of OV-OntoNotes. Blue denotes hase type, red denotes novel type.
Split
Types
preduct- product-train, product-softwarc product-ship, product- other, product-game, product-food, product-car, product-airplane, person- soldier, person-scholar, person-politician person-other, person-directon person-athlete, person-artist/author, person-actor, other-medical, other- livingthing, other-law, other- lahhrrlanggage, other-god, other-educatiinnallerree other-disease, other-currency, other-chemicaltinng, other-biologything, other-award, other- attrrrattoooottinngg... art-writtenart, art-painting, art-other, art-music, art-film, art-broadcastprogrrn
Train
event-sportsevent,, event-protest, event-other, event-election, event-
disaster, ween--atakk/aattt/aa///ilatarrro building- theeter, building- sportsfacility, quilding-restauran building-other, building-librant building- hotel, muilding-hospitee building-airport
Dev
building-airport
building-hotel, building-library,
building-othh other,
building palldingssorttffaciiittt.. building muilding-theater,
event-disaster, event-election.
event-other, event-protest, event-sportsevent, location-GPE GPE, location-
bodiesofwater, location-island, location-mountain. location-other,
Test
location-park,
organization-comee company,
ceaniizationeducaatinn organization- www.oonnowwweneeee//oe/eeenennee
organization-mmciiannwwwwas media/ncwssaperr organization-other organization-
politicalparty, organization--eligion organzzaionnshoorrgnnaatiio organization-sportsleqgeee organization-spoorssean
Table Type split OV-NERD-INTR Blue denotes the base type, red denotes the novel type.

---

Split
Types
art brtoocddcastprorram,, art-film, building-hotel, building-restaurant, solliiingssootsfcciitttt.t event-
art-other, building-airport,
attac//autt/varmmilaaycoiiiiii event-disaster,
event-protest,
location-GPE: location-island,
location-
oou//ralwayhighvaaytrasi,, arganization-compan
organization-
Train
education, argamization-mediaaeewwppp organization-other, organization-
sportsleague, organization-sportttiee other-astronommyhing, other-award,
other- bihor-biologything, other-disease, other-god, other-language, other-law,
person- aruts,,rrist/authoor person-director, person-politician, person-soldier, product-airplann product-food, product-other, product-ship, product- software
art- brrrroddcastprorramm art-film, building-hotel, building-restaurant, building-sportsfacilitty event- hwwcc//bate/wwa//iliiyycccomo
art-other, building-airport,
event-disaster, event-protest,
location GPE, location-island,
location-mountain mountain, location- location-
http///awwawyiggghhattttiiii coamiizationcomaaan organization-
education, mwwiiiaatioommdddddeesspppppr organization-othhe, organization-
sportsleague, crmniiaation-ppotsttaaa other-astrnommything other-award,
Dev
other- other-biologything other-disease, other-god, other-language, other-law,
person-artisie arttst,artis/auuhhorr person-director, person-politician, person-soldier, product-airplane, product-food, product-other, product-ship, product- software, art-painting, building-library, building-other event-other, location- park, organization-religie organization-shooorgamne showorganization, other- chemicalthing, other-currency, person-other, person-scholar, product-game, product-train
art-other, building-hoie building-restauran,, attack/batlee/ar/miihia http://balleawa/mmiaaracnoiiiic
ort-broadcastprogram, art-film,
building-airport,
event- event-protest,
event-disaster,
location-GPP location-island,
location location-mountiin, location-
road/ailwayyhighwaalrassss anganization-coppan organization- education, sportsleague, aigamization-spoottttamm, other-astronomythingg awher-
organization-o organization-
other- biolorrbooooythingg other-discase, other-god, other-language, other-law,
person person-artist/auue artist/author, person-director person-politician, person-soldier, product-airplane, product-food, product-other, product-ship, product- software, art-music, art-painting, art-writtenart, building-hospital building-library, building-other, building-theater, event-election, event-other, event-sportseevett,
Test
location-other, location-parke
wggnizztoo-geemmmeen/geemmemmmee
organization-polliicalaaa
organizationn-rlliionn, araeiizztioonshoooogaanztzzooooo other-chemicaltingg
other-currency, offerr-dducationaalgeee,, other-livingthing, other-medical, person-actor, person-athlete, person-other, person-scholar, product-car, product-game, product-train, product-weapon
Table 10: Type split of OV-NERD-INTER Blue denotes the base type, red denotes the novel type.

---

Sentences with Gold Entities
Sentences with Predicted Entities
The [Beinecke Rare Book] Library and [Manuscript JBuiding at Yale Univer- sity has an archive of his collected papers.
The [Beinecke Rare Book and Manuscript Li- braty]aaaddig-uear at Yale University has an archive of his collected papers.
[Walker CirquelluenionMommii is a prominent glacier-filke filled cirque at the west side the ter- minus of [McCleary clacier]seenimm.oo in [Cook Mountains| Moumtion-Mouminn
[Walker Cirque Boniioo--Biiewwwtater is a prominent glacier-filled cirque at the west side the ter- minus of [McCleary clacier]neetimmcoon in [Cook
Mountains Jt Location-Moummiin
JwariiorS Spartsstiinn--n knocked out in the quarter-finals of the [Auckland Nines brgmmizaiinnSSeetttm by eventual runners up [Cronulla Sutherland Sharks Sharks] Siggmizaioo-ppunsess
The
The [Warriors] Warrioos (rggunizziion Sportssamm were knocked in the quarter- quarter-finals of the [2015 NRL Auckland Nines J Jonpeizaniee-Sperlleae by eventual runners up
2015
JNRL.ogganatiimSSSyr
[Cronulla Sutherland Sharks (arpmizaiionpputtteec
stands above [Brothers
Wa-
stands above [Brothers Water Reseooee and the [Ullsatte-Ambleeidee]] Lecation-OOhe Other [road]Lacuion RoadRaaaaay/HHhhhee
Tellocutee-Bddesatteee and the Ammlesde
road Locutio Roaghaaa
The wreck was sold [Thursday Is- The wreck was sold [Thursday Is- land] according to the [Sydney Morning Herald Organiizaaion Medis/Nowppper on June 1883.
Lonconn according to the [Sydney Morning Heraidlormmiintionnn Mraaainneeeeeeeeee///oeeeoo on June 1883.
Table 11: Case Study on OV-NERD-INTRA Blue denotes the entites with gold annotations, green denotes the entites with true predictions, and red denotes the entites with false predictions.
Name
Description
PERSON
person name [SEP] people, including fictional.
nationality, other, religion, political [SEP] nationalities or religious or
NORP
political groups.
FAC
facility [SEP] buildings, airports, highways, bridges, etc.
ORG
organization [SEP] companies, agencies, institutions, etc.
GPE
geographical/socia/poliiiii entity [SEP] countries, cities, states.
LOC
location [SEP) non-GPE locations, mountain ranges, bodies of water.
PRODUCT
product [SEP] vehicles, weapons, foods, etc. (Not services).
DATE
date [SEP] absolute or relative dates or periods.
TIME
time [SEP] times smaller than day.
PERCENT
percent [SEP] percentage (including "%").
MONEY
money [SEP] monetary values, including unit.
QUANTITY
quantity [SEP] measurements, as of weight or distance.
ORDINAL
ordinal [SEP] first, second.
CARDINAL
cardinal [SEP] numerals that do not fall under another type.
EVENT
event [SEP] named hurricanes, battles, wars, sports events, etc.
WORK_OF OFLART
work of art (SEP) titles of books, songs, etc.
LAW
law [SEP] named documents made into laws.
LANGUAGE
language [SEP] any named language, English, Chinese.
Table 12: Entity descriptions of OntoNotes.

---

Name product-weapon
Description
weapon [SEP] weapon, arm armament is any implement or device that be used to deter, threaten, inflict physical damage, harm, kill. Weapons are used increase the efficacy and efficiency of activities such as hunting, crime, law enforcement, self-defense, warfare, or suicide.
product-train
train [SEP] in rail transport, a train a series connected vehicles that run along railway track and transport people or freight. Trains are typically pulled or pushed by locomotives, though some are self-propelled, such as multiple units.
product-softwarc
software [SEP] at the lowest programming level, exccutable code con- sists of machine language instructions supported by individual proces- sor sor-typically central processing unit (CPU) or a graphics processing unit (GPU). Machine language consists of groups of binary values signifying processor instructions that change the state of the computer from its preced- ing state.
product-ship
ship [SEP] ship is large watercraft that travels the world's oceans and other sufficiently deep waterways, carrying cargo passengers, or in support of specialized missions, such as defense, research, and fishing. Ships are generally distinguished from boats, based size, shape, load capacity, and purpose.
product-other
product [SEP] in marketing, product an object, or system, or service made available for consumer use as of the consumer demand; it is anything that can be offered to a market to satisfy the desire or need of a customer. game [SEP] game is a structured form of play, usually undertaken for entertainment or fun, and sometimes used as an educational tool.
product-game
product-food
food [SEP] food is any substance consumed by an organism for nutritional support. Food is usually of plant, animal, or fungal origin, and contains es- sential nutrients, such carbohydrates, fats, proteins, vitamins, minerals. car [SEP] car automobile is motor vehicle with wheels. Most defini- tions of cars say that they run primarily on roads, seat one to eight people, have four wheels, and mainly transport people instead of goods.
product-car
product-airplane
airplane [SEP] an airplane or aeroplane (informally plane) a fixed-wing aircraft that is propelled forward by thrust from a jet engine, propeller, or rocket engine. Airplanes come in variety of sizes, shapes, and wing configurations. The broad spectrum of uses for airplanes includes recreation, transportation of goods and people, military, and research.
person-soldier
soldier [SEP] soldier is person who member of an army. A soldier can be a conscripted or volunteer enlisted person, non-commissioned officer, an officer.
person-scholar
scholar [SEP] a scholar a person who pursues academic and intellectual activities, particularly academics who apply their intellectualism into exper- tise in an area of study. A scholar can also be an academic, who works as professor, teacher, researcher at university.
person-politician
politician [SEP] politician is a person active in party politics, or person holding or seeking elected office in government. Politicians propose, support, reject and create laws that govern the land and by extension its people. Broadly speaking, politician can be anyone who seeks achieve political power in a government.

---

person-other
person [SEP] a person is being that has certain capacities attributes such reason, morality, consciousness or self-consciousness, and being a part of a culturally established form of social relations such as kinship, ownership of property, or legal responsibility.
person-director
director [SEP] a film director controls a film's artistic and dramatic aspects and visualizes the screenplay (or script) while guiding the film crew and actors in the fulfilment of that vision. The director has key role in choosing the cast members, production design and all the creative aspects of filmmak- ing.
person person-athlete
athlete [SEP] an athlete (also sportsman or sportswoman) a person who competes in one or more sports that involve physical strength, speed, or endurance.
person artist/author
artist/author [SEP] an artist is a person engaged an activity related to creating art, practicing the arts, or demonstrating an art. An author is the writer of a book, article, play, mostly written work.
person-actor
actor (SEP) actor or actress is person who portrays a character in performance. The actor performs in the flesh in the traditional medium of the theatre in modern media such as film, radio, and television. The actor's interpretation of role-the art of acting pertains the role played, whether based on a real person or fictional character.
other-medical
medical [SEP] medicine the science[1] and practice[2] of caring for a patient, managing the diagnosis, prognosis, prevention, treatment, palliation of their injury or disease, and promoting their health. Medicine encompasses a variety of health care practices evolved to maintain and restore health by the prevention and treatment of illness.
other-livingting
living thing (SEP) various forms of life exist, such plants, animals, fungi, protists, archaea, and bacteria.
other-law
law [SEP] law is set of rules that are created and are enforceable by social or governmental institutions to regulate behavior, with its precise definition a matter of longstanding debate. It has been variously described as a science and as the art of justice.
other-language
language [SEP] language is a structured system of communication. The structure of a language is its grammar and the free components are its vocabulary. Languages are the primary means of communication of humans, and can be conveyed through spoken, sign, written language.
other-god
god [SEP] in monotheistic thought, God is usually viewed as the supreme being, creator, and principal object of faith. God usually conceived of as being omnipotent, omniscient, omnipresent, and omnibencvolent, as well as having an eternal and necessary existence.
other-
educational degree [SEP] an academic degree is qualification awarded to students upon successful completion of a course of study in higher education, usually college university. These institutions commonly offer degrees various levels, usually including undergraduate degrees, master's, and doctorates, often alongside other academic certificates and professional degrees.
educationaldegree
other-disease
disease [SEP] disease is a particular abnormal condition that negatively affects the structure or function of all or part of an organism, and that is not immediately due to any external injury. Diseases are often known to be medical conditions that are associated with specific signs and symptoms. A disease may be caused by external factors such as pathogens or by internal dysfunctions.

---

other-currency
currency [SEP] currency standardization of money in any form, in use circulation medium of exchange, for example banknotes and coins. A more general definition is that currency a system of money common use within a specific environment over time, especially for people in nation state.
other- chemicalthing
chemical thing [SEP] chemistry is the scientific study of the properties and behavior of matter. It is natural science that covers the elements that make up matter to the compounds composed of atoms, molecules and ions: their composition, structure, properties, behavior and the changes they undergo during reaction with other substances.
bthor-bioloyything
biology thing [SEP] biology is the scientific study of life. It is a natural science with a broad scope but has several unifying themes that tie it together as a single, coherent field. For instance, all organisms are made up of cells that process hereditary information encoded in genes, which can be transmitted to future generations. Another major theme is evolution, which explains the unity and diversity of life.
other-award
award [SEP] an award, sometimes called distinction, is something given to a recipient token of recognition of excellence in certain field.[1][2] When the token is medal, ribbon or other item designed for wearing, is known as decoration.
other-
astronomy thing [SEP] astronomy a natural science that studies celestial objects and phenomena. uses mathematics, physics, and chemistry order to explain their origin and evolution. Objects of interest include planets, moons, stars, nebulae, galaxies, and comets.
astronomything
organization-
sports team [SEP] sports team is group of individuals who play sports (sports player).[1] usually team sports, on the same team. The number of players in the group depends on type of the sports requirements.
sportsteam
organization-
sports league [SEP] a sports league group of sports teams or individual athletes that compete against each other and gain points in a specific sport. At its simplest, it may local group of amateur athletes who form teams among themselves and compete on weekends.
sportsleague
organization- showorganization
show organization [SEP] show, an artistic production, such as: Concert, Radio show, Talk show, Television show, Theatre production.
organization- religion
religion [SEP] religion is usually defined as social-cultural system of desig- nated behaviors and practices, morals, beliefs, worldviews, texts, sanctified places, prophecies, ethics, or organizations, that generally relates humanity to supernatural, transcendental, and spiritual elements.
organization- politicalparty
political party [SEP] a political party is an organization that coordinates candidates to compete in a particular country's s elections. It is common for the members of party to hold similar ideas about politics, and parties may promote specific ideological or policy goals.
organization-other
organization [SEP] an organization or organisation, is an entity- -ucch company, an institution, or association compiini one more people and having particular purpose.
organization- media/newspaper
media/newsppppe [SEP] the news media or news industry are forms of mass media that focus on delivering news to the general public or a target public. These include news agencies, print media (newspapers, news magazines), broadcast news (radio and television), and the internet (online newspapers etc.).

---

organization-
governmmentggovemmeetteegn [SEP] government or state agency, some- times an appointed commission, is a permanent or semi-permanent organi- zation the machinery of government that responsible for the oversight and administration of specific functions, such as an administration.
govemmentagencyy
organization- education
education [SEP] an educational institution place where people of dif- ferent ages gain an education, including preschools, childcare, primary- elementary schools, secondary-high schools, and universities. They provide a large variety of learning environments and learning spaces.
organization- company
company (SEP) company, abbreviated co., legal entity representing an association of people, whether natural, legal mixture both, with specific objective.
location-
road/railway/trassi [SEP] a road a linear way for the conveyance traffic that mostly has an improved surface for use by vehicles (motorized and non-motorized) and pedestrians.
way/tani
location-per
park [SEP] a park is an area of natural, semi-natural or planted space set aside for human enjoyment and recreation or for the protection of wildlife natural habitats.
location-other
location [SEP] in geography, location or place are used denote a region, country, city, town, village or hamlet (point, line, or area) on Earth's surface or elsewhere, such as.
location-mountain
mountain [SEP] a mountain an elevated portion of the Earth's crust, generally with steep sides that show significant exposed bedrock.
island [SEP] an island (or isle) is an isolated piece of habitat that is sur- rounded by dramatically different habitat, such as water.
location-
bodies of water [SEP] body of water or waterbody (often spelled water body) is any significant accumulation of water on the surface of Earth another planet. The term most often refers to oceans, seas, and lakes, but it includes smaller pools of water such as ponds, wetlands, or more rarely, puddles.
bodiesofwater
location-Gii GPEE
GPE [SEP] the FAO geopolitical ontology is an ontology developed by the Food and Agriculture Organization of the United Nations (F'O) to describe, manage and exchange data related to geopolitical entities such countries, territories, regions and other similar areas.
event-sportsseeen
sports event [SEP] multi-sport event is organized sporting event, of- ten held over multiple days, featuring competition in many different sports among organized teams of athletes from (mostly) nation-states. The first ma- jor, modern, multi-sport event international significance was the Olympic Games.
event-protest
protest (SEP) a protest (also called demonstration, remonstration or remon- strance) is public expression of objection, disapproval or dissent towards an idea or action, typically political one. Protests can be thought of as acts of cooperation in which numerous people cooperate by attending, and share the potential costs and risks of doing
event-other event-election
event [SEP] an occurrence; something that happens.
election [SEP] an election is a formal group geeision-making process by which a population chooses an individual or multiple individuals to hold public office.
event-disaster
disaster [SEP] a disaster is serious problem occurring over short or long period of time that causes widespread human, material, economic or environmental loss which exceeds the ability of the affected community or society to cope using its resources.

---

attac//baate/vaammlllaaryooiii [SEP] war is an intense armed conflict attack/militaryconfiit between states, governments, societies, or paramilitary groups such as mer- cenaries, insurgents, and militias. is generally characterized by extreme violence, destruction, and mortality, using regular or irregular military forces. theater [SEP) theatre or theater a collaborative form of performing art that uses live performers, usually actors or actresses, to present the experience of a real or imagined event before a live audience in specific place, often stage.
event-
building-thean
building- sportsfacility
sports facility (SEP) sports complex is group of sports facilities. For example, there are track and field stadiums, football stadiums, baseball stadiums, swimming pools, and Indoor arenas.
building-restaraan
restaurant [SEP] restaurant a business that prepares and serves food and drinks to customers.
building-other
building [SEP] a building, or edifice, is an enclosed structure with a roof and walls standing more or less permanently in one place, such as house or factory (although there's also portable buildings).
building-libbn -ibrary
library [SEP] library is collection of materials, books or media that are accessible for use and not just for display purposes.
building-hoter
hotel [SEP] a hotel is an establishment that provides paid lodging on short-term basis.
building- hospital
hospital [SEP] a hospital is a health care institution providing patient treat- ment with specialized health science and auxiliary healthcare staff and medical equipment.
building-ainge
airport [SEP] an airport is an aerodrome with extended facilities, mostly for commercial air transport.
art-writtenart
written [SEP] literature is any collection of written work, but it is also used more narrowly for writings specifically considered be art form, especially prose fiction, drama, and poetry.
art-painting
painting [SEP] painting the practice of applying paint, pigment, color or other medium to solid surface. Painting is an important form in the visual arts, bringing in elements such as drawing, composition, gesture. narration, and abstraction.
art-other
art [SEP] art is a diverse range of human activity, and resulting product, that involves creative imaginative talent expressive of technical proficiency, beauty, emotional power, conceptual ideas.
art-music
music [SEP] music is generally defined as the of arranging sound to create some combination of form, harmony, melody, rhythm or otherwise expressive content.
art-film
file [SEP] a film - also called a movie, motion picture, moving picture, picture or photoplay - is a work of visual art that simulates experiences and otherwise communicates ideas, stories, perceptions, feelings, beauty, or atmosphere through the use of moving images. These images are generally accompanied by sound and, more rarely, other sensory stimulations.
art-
broadcast program [SEP] broadcast programming the practice of organiz- ing or ordering (scheduling) of broadcast media shows, typically radio and television, in a daily, weekly, monthly, quarterly or season-long schedule.
broadcastprogram
Table 13: Entity descriptions FEW-NERD.


Reviews:
[{"review_id": "CVJYRBJh8W", "reviewer": "Reviewer_tqWV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews. Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! \n\n> **Response to Rejection 1**: The work does not bridge well to past related research work in the community. This work criticizes past related research directions. OVNER should also have certain limitations, such as the provision of descriptions being necessary.\n\nWe are sorry for making this confusion! We talked about past related research work (including supervised methods, continual learning methods and zero-shot methods) in the introduction, with the aim of presenting the motivation for our method **\"open-vocabulary named entity recognition\"**. **We think all of the previous work in the community is outstanding and interesting, making solid contributions to the development of NER**. Our work is also based on their research and aims to enhance the NER model's ability to handle novel-emerging types during the inference stage. Our intent is to make a meaningful and practical contribution to the community's research. We will revise the introduction so that our work bridges better with past related outstanding work.\n\nFurthermore, we agree with you that our approach has certain limitations. Compared to annotating the training data for new types, we think it may be an easier way to provide or collect entity descriptions for each type. As shown in Table 4 (Page 7), we also explore using GPT 3.5 to automatically generate descriptions instead of collecting descriptions from Wikipedia. Experimental results show that GPT 3.5 can also achieve good performance and reduce human labor. **We believe that alleviating the limitation of type descriptions could be a further extension of our work in the future**.\n\n> **Response to Rejection 2**: Does OVNER cover Domain-Adapted NER scenarios? Does it include scenarios with large domain differences?\n\nThanks for your valuable suggestion! **We think OVNER covers Domain-Adapted NER scenarios, especially those with large domain differences.** According to your suggestion, we conduct experiments on the Cross-NER benchmark [1], and **are pleasantly surprised to find that our method has flexible transfer ability and performs well in cross-domain NER scenarios**. Following the previous work [2, 3, 4], we use CoNLL 2003 as the source domain which is labeled with four types: PER, LOC, ORG, and MISC. Then, we use CrossNER as the target domain which contains five separate domain: politics, natural science, music, literature and AI. **Table 1** presents the detailed statistics of these datasets.\n\n**Table 1:** The statistics of CoNLL 2003 and CrossNER.\n\n| Datasets     | CoNLL 2003 | CrossNER |    CrossNER     | CrossNER |  CrossNER  |        CrossNER         |\n| ------------ | :--------: | :------: | :-------------: | :------: | :--------: | :---------------------: |\n| Domain       |    News    | Politics | Natural Science |  Music   | Literature | Artificial Intelligence |\n| #Train       |   15.0k    |   200    |       200       |   100    |    100     |           100           |\n| #Test        |    3.7k    |   651    |       543       |   456    |    416     |           431           |\n| #Entity Type |     4      |    10    |       17        |    13    |     11     |           12            |\n\nTo evaluate the effectiveness of our method, we compare it with several cross-domain NER baselines, including: \n- DAPT [1]: DAPT refers as domain-adaptive pre-training, which leverages a large domain-related corpus to pre-train the model, similar to the first-stage pre-training of our method.\n- LST-NER [2]: LST-NER builds label graphs in both the source and target label spaces for cross-domain NER tasks.\n- LANER [3]: LANER proposes a novel autoregressive cross-domain NER framework to improve label information transfer.\n- CP-NER [4]: CP-NER introduces collaborative domain-prefix tuning for cross-domain NER based on text-to-text generative PLMs, which obtains SOTA performance on the CrossNER benchmark.\n\nTo make a fair comparison, we first train on the source domain with sufficient data, and then adapt to the target domain with a very small amount of data. As shown in **Table 2**, **our method can consistently obtain better performance than SOTA method CP-NER + DAPT across all target domains, with an average 2.86% F1-score improvement**. Experimental results demonstrate the effectiveness of our OVNER method under cross-domain NER scenarios. We think there are several reasons for the surprising results:\n\n- Our method can learn a rich and complete context-type semantic space, and expand its vocabulary from large-scale pre-training. **Pre-training is essential to improve the domain generalization of our model**.\n- Our method uses entity descriptions to represent types, which can contain more type semantic information. We guess entity descriptions can better **realize the transfer of knowledge from source domain to target domain**.\n\n**Table 2:** The performance of baselines and our CACAO on CrossNER.\n\n| **Models**         | **Politics** | **Science** | **Music** | **Literature** |  **AI**   | **Average** |\n| ------------------ | :----------: | :---------: | :-------: | :------------: | :-------: | :---------: |\n| LST-NER            |    70.44     |    66.83    |   72.08   |     67.12      |   60.32   |    67.36    |\n| LANER              |    71.65     |    69.29    |   73.07   |     67.98      |   61.72   |    68.74    |\n| CP-NER             |    73.41     |    74.65    |   78.08   |     70.84      |   64.53   |    72.30    |\n| LST-NER+DAPT       |    73.25     |    70.07    |   76.83   |     70.76      |   63.28   |    70.84    |\n| LANER+DAPT         |    74.06     |    71.83    |   78.78   |     71.11      |   65.79   |    72.31    |\n| CP-NER+DAPT        |    76.35     |    76.83    |   80.28   |     72.17      |   66.39   |    74.40    |\n| CACAO (Our Method) |  **81.26**   |  **77.78**  | **82.60** |   **75.73**    | **68.95** |  **77.26**  |\n\n[1] CrossNER: Evaluating Cross-Domain Named Entity Recognition, Liu et al. 2021.\n\n[2] Cross-domain Named Entity Recognition via Graph Matching, Zheng et al. 2022.\n\n[3] A Label-Aware Autoregressive Framework for Cross-Domain NER, Hu et al. 2022.\n\n[4] One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NER, Chen et al. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This work proposes a new Setting in an attempt to address an important challenge for NER in recent years, which is how to discover new entity types. Acceptance of this work may be useful for subsequent research in the NLP community.\n2. The work is relatively complete and somewhat sound. While proposing OVNER, the CACAO framework is proposed to address it. The design and results of the experiment can relatively support their claims.", "reasons_to_reject": "1. The work does not bridge well to past related research work in the community. This work criticizes past related research directions. OVNER should also have certain limitations, such as the provision of descriptions being necessary.\n2. Does OVNER cover Domain-Adapted NER scenarios? Does it include scenarios with large domain differences?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a more challenging task called Open Vocabulary Named Entity Recognition(OVNER) in response to a real-world need. The authors started with the motivation that the current NER approach does not work well to discover novel entity types. The goal of the newly proposed OVNER is to automatically recognize new types when given a description beside the name. For OVNER, the work also proposed a framework called CACAO and experimentally validated its effectiveness.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uUh2CxobHE", "reviewer": "Reviewer_3cyn", "summary": "", "questions": "Question A: Continual learning NER requires sufficient annotated data for new type, while continual few-shot NER does not. Comparing OVNER with this paradigm makes the discussion more complete in the introduction (Line 064).\n\nQuestion B: It seems that the performance of novel types remains largely lower than that of base types. Discussion about future efforts to improve the performance and error analysis are expected.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews. Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! \n\n> **Response to Rejection 1**: The paper only compares with discriminative methods, it is expected to compare with generative methods or LLMs which might perform better in some cases.\n\nThanks for your insightful advice! **Considering that our model adopts the discriminative method, and previous generative methods do not perform very well in recognizing novel types, we only compare with several discriminative methods**. According to your suggestion, **we compare our method with several generative methods under cross-domain settings**. We use CoNLL 2003 as the source domain which is labeled with four types: PER, LOC, ORG, and MISC. Then, we use CrossNER [1] as the target domain which contains five separate domains: politics, natural science, music, literature and AI. To evaluate the effectiveness of our method, we compare it with several generative methods, including: \n\n- BARTNER [2]: BARTNER uses the pre-trained BART model to generate entity spans, formulating the NER task as a sequence generation problem.\n- LightNER [3]: LightNER utilizes a pluggable prompting method for pre-trained generative model to improve NER performance in low-resource settings.\n- CP-NER [4]: CP-NER introduces collaborative domain-prefix tuning for cross-domain NER based on text-to-text generative PLMs, which obtains SOTA performance on the CrossNER benchmark. CP-NER also leverages a large domain-related corpus to pre-train the model (DAPT), similar to the first-stage pre-training of our method.\n\nTo make a fair comparison, we first train on the source domain with sufficient data, and then adapt to the target domain with a very small amount of data. As shown in **Table 1**, **our method can consistently obtain better performance than SOTA method CP-NER + DAPT across all target domains, with an average 2.86% F1-score improvement**. Experimental results demonstrate the effectiveness of our OVNER method under cross-domain NER scenarios. We think there are several reasons for the surprising results:\n\n- Our method can learn a rich and complete context-type semantic space, and expand its vocabulary from large-scale pre-training. **Pre-training is essential to improve the domain generalization of our model**.\n- Our method uses entity descriptions to represent types, which can contain more type semantic information. We guess entity descriptions can better **realize the transfer of knowledge from source domain to target domain**.\n\n**Table 1:** The performance of generative methods and our CACAO on CrossNER.\n\n| Models             | Politics  |  Science  |   Music   | Literature |    AI     |  Average  |  Average  |\n| ------------------ | :-------: | :-------: | :-------: | :--------: | :-------: | :-------: | :-------: |\n| BARTNER            |   69.90   |   65.14   |   65.35   |   58.93    |   53.00   |   62.46   |   62.46   |\n| LightNER           |   72.78   |   66.74   |   72.28   |   65.17    |   35.82   |   62.56   |   62.56   |\n| CP-NER             |   73.41   |   74.65   |   78.08   |   70.84    |   64.53   |   72.30   |   72.30   |\n| CP-NER+DAPT        |   76.35   |   76.83   |   80.28   |   72.17    |   66.39   |   74.40   |   74.40   |\n| CACAO (Our Method) | **81.26** | **77.78** | **82.60** | **75.73**  | **68.95** | **77.26** | **77.26** |\n\n[1] CrossNER: Evaluating Cross-Domain Named Entity Recognition, Liu et al. 2021.\n\n[2] Cross-domain Named Entity Recognition via Graph Matching, Zheng et al. 2022.\n\n[3] A Label-Aware Autoregressive Framework for Cross-Domain NER, Hu et al. 2022.\n\n[4] One Model for All Domains: Collaborative Domain-Prefix Tuning for Cross-Domain NER, Chen et al. 2023.\n\n> **Response to Question 1**: Continual learning NER requires sufficient annotated data for new type, while continual few-shot NER does not. Comparing OVNER with this paradigm makes the discussion more complete in the introduction (Line 064).\n\nYes, we agree with you that continual few-shot NER requires only a small amount of annotated data to learn to recognize novel types. Compared with the outstanding continual few-shot NER methods [5], our method has the following differences: **OVNER focuses on generalizing novel types without any annotation, aiming to train a once-and-for-all recognizer**. To make our paper clearer and more complete, we will compare OVNER with this paradigm in our revised version.\n\n[5] Few-Shot Class-Incremental Learning for Named Entity Recognition, Wang et al. 2022.\n\n> **Response to Question 2**: It seems that the performance of novel types remains largely lower than that of base types. Discussion about future efforts to improve the performance and error analysis are expected.\n\nThanks for your insightful suggestion! We can observe a performance gap between base types and novel types, which demonstrates that OVNER is a challenging and unsolved task. In Section 4.5 (Page 7), we analyze the performance of different entity types on OV-NERD-INTRA and OV-NERD-INTER. We find that **our model does not perform well when the entity types are difficult to define and describe in a single sentence**, such as TIME, PERCENT, QUANTITY and GPE types. Besides, we also perform a case study to analyze our method. As shown in Table 9 (Page 15), our approach excels in accurately classifying entities. However, **it faces challenges in precisely determining entity boundaries**.\n\nTherefore, we can improve our method from the following directions:\n\n- **Provide several entity demonstrations to model** to enhance original type descriptions. We find this to be effective for identifying those hard-to-describe types.\n- Design new pre-training objectives to **improve the model's ability to determine entity boundaries**.\n- Increase the pre-training corpus or **use the retrieval-augmented method** to expand the model's vocabulary, so that the model can better handle long-tail types.\n\n> **Writing Comments**: Typos Grammar Style And Presentation Improvements\n\nWe really appreciate your suggestions on the details of our paper writing! We will revise our paper according to your suggestion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is easy to follow and well-motivated.\nAbundant experimental results verify the effectiveness of the approach.\n", "reasons_to_reject": "The paper only compares with discriminative methods, it is expected to compare with generative methods or LLMs which might perform better in some cases.", "questions_to_authors": "Question A: Continual learning NER requires sufficient annotated data for new type, while continual few-shot NER does not. Comparing OVNER with this paradigm makes the discussion more complete in the introduction (Line 064).\n\nQuestion B: It seems that the performance of novel types remains largely lower than that of base types. Discussion about future efforts to improve the performance and error analysis are expected.", "paper_topic_and_main_contributions": "This paper tackles a realistic open-vocabulary NER problem, i.e., recognizing novel types of entities given only their surface names/descriptions and annotation of base types.\nThe paper formulates OVNER as a semantic matching task and proposes a two-stage method. It first pretrains dual encoders on distantly-annotated context-type pairs for alignment based on contrastive learning, then it finetunes the cross-encoder on base type supervision.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "A: “Our Proposed Method” in Table 1 can be omitted."}, "extra_text": {}}, {"review_id": "03ddnktwqI", "reviewer": "Reviewer_79yH", "summary": "", "questions": " Could you provide some examples of cases where the proposed method fails or makes errors, and how the method could be improved to handle such cases?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and insightful reviews. Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! \n\n> **Response to Rejection 1**: The paper mainly focuses on the English language, and it is unclear how well the proposed method would generalize to other languages.\n\nThis is a good idea! **Our method can be easily extended to multilingual.** Since Wikipedia covers over 300 language editions, we can easily build the pre-train **multilingual** corpus by following the data collection method proposed in our paper. Besides, we can replace the original backbone model with a multilingual pre-trained model, such as mBERT and XLMR.\n\n> **Response to Rejection 2**: The paper does not provide a detailed analysis of the computational complexity and efficiency of the proposed method, which could limit its practical applicability in some scenarios.\n\nThanks for your valuable advice! **Our method is designed for practical NER applications**. **We first analyze the computational complexity of our method**. Since the input sentences and type descriptions are independently encoded via Dual-Encoder, **we can encode all entity types in advance**. Then, we employ Cross-Encoder to capture the rich interactions between context and all candidate types efficiently and effectively. At the inference stage, our method only need to encode the input sentence once, and then fuse the context and all types once, so its computational complexity is $O(1)$. Instead, MRC-style methods can only extract entities of one type per inference, so the model needs to be run $N$ times, where $N$ is the number of entity types. Due to the high time complexity $O(N)$ of inference, MRC-style methods may not work for hundreds of types in real-time systems.\n\n**We use the FLOPs-per-token estimates to calculate the compute cost of our method**: $C_{CACAO}=l_{input}\\times\\theta_{BERT}+(l_{input}+N)\\times\\theta_{BERT}$, where where $l_{input}=200$ is the number of tokens per input, $\\theta_{BERT}=110M$ is the parameter size of BERT.\n\nThe compute cost of MRC-style methods is: $C_{MRC}=N\\times(l_{input}+l_{description})\\times\\theta_{BERT}$, where $l_{description}=50$ is the number of tokens per type description. Our method is efficient when the number of entity types $N=100\\ or\\ 1000$ is large in practical applications. If the online system has extremely high requirements for speed, **our method can directly discard Cross-Encoder, and only adopt Dual-Encoder for inference**. We can also use Faiss to perform efficient similarity search.\n\nAccording to your suggestion, we will provide a detailed analysis of the computational complexity and efficiency in the revised version.\n\n> **Response to Rejection 3**: The paper does not provide a detailed analysis of the impact of different hyperparameters on the performance of the proposed method, which could limit the understanding of the robustness of the method.\n\nThanks for your suggestion! We provide the detailed hyperparameters in Appendix B. In fact, **our method is robust to hyperparameters**. For example, the threshold $\\delta$ is essential for inference. In general, using the default $\\delta=0.5$ can yield good performance. To achieve optimal performance, we tune the model on the development set with $\\delta = \\{0.4, 0.5, 0.6\\}$. \n\n> **Response to Rejection 4**: It is better to compare with other LLM-based NER methods, which seems to be more suitable for zero-shot and open-vocabulary.\n\nThanks for your suggestion! Because several recent work [1] show that LLMs are not effective information extractors in general, given their unsatisfactory performance in most settings. Besides, different from the LLM with the high latency and budget requirements, our method is designed for practical NER systems with low compute cost and latency. So we did not compare our method with LLM-based NER methods. We will try your suggestion and compare with LLM-based NER methods in the revised version.\n\n[1] Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!, Ma et al. 2023.\n\n> **Response to Question 1**: Could you provide some examples of cases where the proposed method fails or makes errors, and how the method could be improved to handle such cases?\n\nWe think your suggestion is well! In Section 4.5 (Page 7), we analyze the performance of different entity types on OV-NERD-INTRA and OV-NERD-INTER. We find that **our model does not perform well when the entity types are difficult to define and describe in a single sentence**, such as TIME, PERCENT, QUANTITY and GPE types. Besides, we also perform a case study to analyze where our proposed method fails or makes errors. As shown in Table 9 (Page 15), our approach excels in accurately classifying entities. However, **it faces challenges in precisely determining entity boundaries**.\n\nTherefore, we can improve our method from the following directions:\n\n- **Provide several entity demonstrations to model** to enhance original type descriptions. We find this to be effective for identifying those hard-to-describe types.\n- Design new pre-training objectives to **improve the model's ability to determine entity boundaries**.\n- Increase the pre-training corpus or **use the retrieval-augmented method** to expand the model's vocabulary, so that the model can better handle long-tail types.\n\n> **Missing References**\n\nWe really appreciate your suggestions on the details of our paper writing! Based on your suggestions, we now add the discussions of missing references [2, 3, 4] in the related work section.\n\n[2] Ultra-fine entity typing with indirect supervision from natural language inference, Li et al. 2022.\n\n[3] Prompt-based Text Entailment for Low-Resource Named Entity Recognition, Li et al. 2022.\n\n[4] Open-vocabulary argument role prediction for event extraction, Jiao et al. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The strengths of this paper include proposing a novel and scalable two-stage method for open-vocabulary named entity recognition , which is a challenging and interesting task in natural language processing.\n2. The paper provides a thorough analysis of the proposed method and its limitations, as well as directions for future work. \n3. The paper also evaluates the proposed method on multiple datasets and releases the pre-trained models and code for reproducibility.\n", "reasons_to_reject": "1. The paper mainly focuses on the English language, and it is unclear how well the proposed method would generalize to other languages.\n2. The paper does not provide a detailed analysis of the computational complexity and efficiency of the proposed method, which could limit its practical applicability in some scenarios.\n3. The paper does not provide a detailed analysis of the impact of different hyperparameters on the performance of the proposed method, which could limit the understanding of the robustness of the method.\n4. It is better to compare with other LLM-based NER methods, which seems to be more suitable for zero-shot and open-vocabulary.", "questions_to_authors": " Could you provide some examples of cases where the proposed method fails or makes errors, and how the method could be improved to handle such cases?", "paper_topic_and_main_contributions": "This paper is about Open-Vocabulary Named Entity Recognition (NER), which is the task of identifying and classifying named entities in text, where the set of entity types is not fixed in advance. The paper proposes a novel and scalable two-stage method called Context-Type SemAntiC Alignment and FusiOn (CACAO) to recognize entities in novel types by their textual names or descriptions. The method is formulated as a semantic matching task and pre-trained on 80M context-type pairs, making it easily accessible through natural supervision. The paper addresses the challenge of open-vocabulary named entity recognition and makes several contributions towards a solution, including proposing the CACAO method, evaluating it on multiple datasets, and releasing the pre-trained models and code for reproducibility. The paper also provides a thorough analysis of the proposed method and its limitations, as well as directions for future work.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Li B, Yin W, Chen M. Ultra-fine entity typing with indirect supervision from natural language inference[J]. Transactions of the Association for Computational Linguistics, 2022, 10: 607-622.\n\nLi D, Hu B, Chen Q. Prompt-based Text Entailment for Low-Resource Named Entity Recognition[C]//Proceedings of the 29th International Conference on Computational Linguistics. 2022: 1896-1903.\n\nJiao Y, Li S, Xie Y, et al. Open-vocabulary argument role prediction for event extraction[J]. arXiv preprint arXiv:2211.01577, 2022."}}]

Author rebuttal:

Meta-review: Summary:
This paper addresses a challenging open-vocabulary Named Entity Recognition (NER) problem, which involves recognizing new entity types based solely on their surface names/descriptions and annotations of base types. The approach formulates OVNER as a semantic matching task and presents a two-stage method. Initially, it pretrains dual encoders on context-type pairs with distant annotations for alignment using contrastive learning. Subsequently, it fine-tunes the cross-encoder with supervision on base types.

Reason To Accept:
1.This paper is well-structured and provides clear motivation. It introduces a novel approach to address a significant challenge in NER, specifically the discovery of new entity types. The acceptance of this work could serve as a valuable contribution to future research in the NLP community.
2.The research is comprehensive and reasonably robust. The paper includes a thorough analysis of the proposed method and its limitations, along with suggestions for future research directions.
3.The paper's strengths lie in its proposal of a novel and scalable two-stage method for open-vocabulary named entity recognition, a challenging and intriguing task within natural language processing.
4.The approach is thoroughly validated through extensive experiments, with evaluations conducted across multiple datasets. Additionally, the paper promotes reproducibility by releasing pre-trained models and code.

Reason To Reject:
1.The paper lacks a clear connection to prior related research in the field and is critical of previous research directions. It is essential to acknowledge certain limitations in the proposed approach, such as the necessity of providing descriptions.
2.Some experiments need to be considered. 1) The paper only compares with discriminative methods, it is expected to compare with generative methods or LLMs which might perform better in some cases. 2) The paper primarily focuses on the English language, leaving uncertainty about the generalizability of the proposed method to other languages.
3.Some details and analysis are lack. 1) A comprehensive analysis of the computational complexity and efficiency of the proposed method is lacking, potentially limiting its practical applicability in real-world scenarios. 2) The paper does not delve into an in-depth examination of how different hyperparameters affect the method's performance, hindering our understanding of its robustness.
Decision: Accept-Findings