NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval
Authors: Uri Katz
Matan Vetzler
Amir David Nissan Cohen
Yoav Goldberg
Venue: EMNLP 2023 Findings (2023)
Keywords: zero shot ner
retrieval
exhaustive search
TL;DR: new dataset for fine-grained NER, zero shot NER and NERetrieval task

Abstract: Recognizing entities in texts is a central need in many information-seeking scenarios, and indeed, Named Entity Recognition (NER) is arguably one of the most successful examples of a widely adopted NLP task and corresponding NLP technology. Recent advances in large language models (LLMs) appear to provide effective solutions (also) for NER tasks that were traditionally handled with dedicated models, often matching or surpassing the abilities of the dedicated models. Should NER be considered a solved problem? We argue to the contrary: the capabilities provided by LLMs are not the end of NER research, but rather an exciting beginning. They allow taking NER to the next level, tackling increasingly more useful, and increasingly more challenging, variants. We present three variants of the NER task, together with a dataset to support them. The first is a move towards more fine-grained---and intersectional---entity types. The second is a move towards zero-shot recognition and extraction of these fine-grained types based on entity-type labels. The third, and most challenging, is the move from the recognition setup to a novel retrieval setup, where the query is a zero-shot entity type, and the expected result is all the sentences from a large, pre-indexed corpus that contain entities of these types, and their corresponding spans. We show that all of these are far from being solved. We provide a large, silver-annotated corpus of 4 million paragraphs covering 500 entity types, to facilitate research towards all of these three goals.

NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval
Uri Katz"1
Matan Vetzler*1
Amir DN Cohen'
Yoav Goldberg" 1.2
Bar-Ilan University
2 Allen Institute for Al
(urikacid, metan. zer, yoav, goldbbrgg 8gma com
Abstract
Netrieve nentions about: (Slnosaura)
(Hama11)) (Rivezs)
Recognizing entities in texts is central need in many information-seeking scenarios, and in- deed, Named Entity Recognition (NER) ar- guably one of the most successful examples a widely adopted NLP task and correspond- ing NLP technology. Recent advances in large language models (LLMs) appear provide ef- fective solutions (also) NER tasks were traditionally handled with dedicated models, often matching or surpassing the abilities of the dedicated models. Should NER considered solved problem? We argue to the contrary: the capabilities provided by LLMs are not the end of NER research, but rather an exciting begin- ning. They allow taking NER to the next level, tackling increasingly more useful, and increas- ingly more challenging, variants. We present three variants of the NER task, together with dataset to support them. The first is move to- wards more fine-grained and intersectional entity types. The second is a move towards zero-shot recognition and extraction of these fine-grained types based entity-type labels. The third, and most challenging. is the move from the recognition setup a novel retrieval setup. where the query is zero-shot entity type, and the expected result is all the sentences from a large, pre-indexed corpus that contain entities of these types, and their corresponding spans. We show that all of these are far from being solved. We provide large, silver-annotate corpus of 4 million paragraphs covering 500 entity types, facilitate research towards all of these three goals.
the Cretaneus porice,
deltas of
nighty have occasionally encecntared eaxly zarnols Bidelphodom,
Mabul, pEsaz aaard cenaluew, ecovenging the rera'ns et the dincesarte kille.
secent oxcazation in Miesourt River unrstled scens whaze Weleeissptes ded souningly unbuabed by pack of Rapancramus.
Along the sedisentacy depoxita of the Iusotitan) tootgrints dotnotde ctesely with these snallex Morganseddo,, refleating intricate ecological intersations Juranaie puz.od.
Figure 1: An example of the NER retrieval process for multiple entity types. The retrieval system utilizes entity recognition to retrieve any text that mentions entities from all queried entity types.
it is arguably of the few pre-Large Language Models natural language technologies that gained wide popularity also outside the NLP research com- munity, and which is routinely used in NLPP applica- tions. The NER task has seen large improvements over the past years. Approaches based on fine- tuning of pre-trained models like BERT, RoBERTa, and T5 significantly improve the accuracy of super- vised NER models (Devlin et al., 2019; Lee et al.. 2020; Phan et al., 2021; Raman al., 2022), while instruct-tuned LLMs (Ouyang et al., 2022; Chung et al., 2022) seem to offer zero-shot NER capabili- ties ("What are all the animal names mentioned this text? (text)"). Moreover, LLMs demonstrate that many language understanding tasks can be per- formed end-to-end while bypassing the need for a dedicated entity identification stage. Is entity iden- tification and extraction solved made redundant by pre-trained-basee end of the research on the NER task? We argue that this is not the end of NER research, but rather new beginning. The phase transition in abilities enables us to expand our expectations from named-entity
1 Introduction
Identification and extraction of typed entities in text (Named Entity Recognition, "NER") central task in natural language understanding, which is both useful in and of itself and as central compo- nent in other language understanding tasks. Indeed,
models and LLMs? Is it the
"Both authors contributed equally this work.
'NERetrieee dataset available at https: //g/ thub. com/katzurike /tte//i//.e...... Net.eNeeerri.. rieve

---

recognition models and redefine the boundaries of the task.
2 Named Entities
After motivating the need for explicit model- ing of named entity identification also in a world with strong end-to-end models $2, we highlight our desiderata for the next-generation entity identifica- tion modules $3. These are based on deficiencies we identify with current approaches and abilities $7:
The identification of entity mentions is a well- established and central component in language pro- cessing, where the input is text, and the goal is to identify all spans in the text that denote some cate- gory of entities. Entity classes can be as broad and generic (PERSON, LOCATTIN,, ORGANIZATION), or specialized EASE), and they may range in specificity (ANIMAL, MAMMAL, DOG ROG_BREED, ...). The predominant setup is the supervised one, in which the set of entity types is given in advance, and specialized model is trained to perform the task for the given set of types.
domain (GENE, PROTEIN, DIS-
1. Supervised models work well on the identifi- cation of broad and general categories ("Per- son". "Animal"),, but still struggle on more nuanced or specialized entity types ("Politi- cian", "Insect"), as well as on intersection types ("Freech Politicians").
Perfectly solving the task requires a combina- tion of strategies, relying on text-internal cues to- gether with external knowledge. For some entity types (e.g., MACHINE LEARNING RESEARCHER) it is common for the text to not provide sufficient hints regarding the type of the mention, and the system should rely on its internal knowledge of the category mental list of ML researchers) in order to identify strings of this category. Yet, also in these cases, the system should rely on the text to disambiguate when possible, e.g. mentions of "Michael Jordan" in text about sports should drive the system against the MACHINE LEARNING RE- SEARCHER hypothesis, even if its knowledge lists "Michael Jordan" one. In other cases, the cat- egory membership can be inferred based on cues in the surrounding text (mentioning of an entity barking suggests its a DOG), or the string itself (entities of type EDUCATIONAL INSTITUTION of- ten contain words such as University, College or School).
2. While LLM-based models exhibit some form of zero-shot NER, this ability degrades quickly when faced with more fine-grained and specialized types. Both supervised and zero-shot setups fail when applied out-of- domain, identifying many non-entities enti- ties.
3. While zero-shot NER abilities allow us to identify entities in a given text, it does not allow us efficiently retrieve all texts that mention entities of this type ("fetch me all paragraphs mentioning an insect").
We thus spell out our vision of next-generatin NER: named entity identification which is:
1. Robust across domains.
2. Can deal with fine-grained, specific, and inter- sectional entity types.
3. Support a zero-shot setup in order to accom- modate novel entity types.
Named entities in the era of LLMs LLMs demonstrate increasing levels of ability to perform "end-to-end" language tasks rather than relying on pipeline components. However, we argue that the identification and extraction of entities from text is often useful in and of itself, and not merely component in larger tasks like summarization
4. Extends the zero-shot setup from identifica- tion to retrieval, allowing to index set of documents such that it efficiently supports re- trieval of all mentions of entities of given zero-shot type. This retrieval should be ex- haustive, focusing on the retrieval of all rele- vant mentions in the collection, and not only the top ones.
(unless the question "which birds are mentioned in this text", in which case it reduces to NER).
This is most pronounced when we consider the retrieval case, in which user may be interested in all documents that contain a specific entity ("all news items from 2010-2020 mentioning a French Politician and a Butterfly"), but is relevant also
We formally define these in the form of NER tasks ($3), and present a silver-annotated dataset that supports research towards these directions ($5).

---

in the identification case where user might be interested in scanning a stream and identifying all mentions of a given type. NER-related tasks remain highly relevant also in the age of LLMs.
be performed on wide range of texts that do not contain entities of that type.
3.2 Finer-Grained, Hierarchical and Intersectional categories
Fine-grained, zero-shot NER retrieval build- ing block While the ability the identify entities of particular type is useful in and of itself, it be- comes significantly more useful when used as a building block for an informmttion-seeiine applica- tion, in which we look for texts that contain several properties. For example, consider the scenario in Figure/ where domain expert seeks to analyze the interplay between dinosaurs, mammals, and the river ecosystems they inhabit. By utilizing the NER retrieval system, the researcher can extract passages containing entities from all three specified types, which they could later further refine. This results in a refined dataset, illuminating the cocxistence and interactions of these entities in ancient fluvial environments. Such a fine-graing theegrriddd, zero-shot ap- proach revolutionizes how experts sieve through vast scientific literature, ensuring comprehensive insights.
The world of entities is wide and diverse. Over the years, academic supervised-NER research tended towards the recognition of a narrow and restricted set of coarse-grained entity types, such as PERSON, ORGANIZATION, LOCATION (Tjong Kim Sang, 2002) or domain specialized types such as GENE, CHEMICAL, BODY-PART, and on in the scien- tific domain (Settles, 2004). Recent datasets aim extend the set of categories: the OntoNotes corpus features 11 "general purpose" coarse-grain cate- gories (Weischedel et al., 2013), and Few-NERD (Ding et al., 20 has 8 coarse-grain -rain categories, which are then subdivided into 66 fine-grained ones. However, even Few-NERD's fine-grained cat- egories are rather coarse-grained, taking the coarse- grained category and going one step further down the hierarchy, e.g. from PERSON POLITICIAN, from BUILDING AIRPORT, from LOCATION MOUNTAIN. The world of entity types is vastly richer and broader: categories can be deeply nested into hierarchies (ANIMAL -> INVERTEBRATE -> INSECT - BUTTERFLY] and arranged into inter- sectional categories that combine different proper- ties of the entity intersectional (FRENCH POLITI- CIAN, MIDDLE EASTERN VEGETARIAN DISHES).
3 Next-Generation NER
We now elaborate a sequence of increasingly more challenging criteria for an ideal NER engine, which we believe are now within reach, yet far from being achieved at present and require further research.
While each intermediate ability is useful on its own, when combined they culminate in what we consider to be the holy grail: Transforming NER from Named Entity Recognition to Named Entity Retrieval.
Despite significant progress in supervised NER, it is evident that NER systems, even those based on pre-trained language models, still struggle to grasp the underlying nuances associated with entity types (Fu al., 2020), particularly when considering fine-grained categories (Ding et al., 2021; Malmasi et al., 2022). Already with Few-NERD's conserva- tive fine nnee-graied categories, we observe low super- vised NER scores (I1 well below 60%) for many entity types. Hierarchical and Intersectional entity types further impose additional challenges, such scarcity of data, the need to detect entities from very long tail which often requires knowledge ex- ternal to the text, and the need make fine-grained distinctions between closely related types. In sec- tion $7.2 7.2 we demonstrate the deterioration of super- vised NER accuracy as we move towards deeper levels of hierarchy and towards intersections.
3.1 Cross-domain Robustness
While current supervised NER systems perform well on established benchmarks (Yamada et al., 2020; Zhou and Chen, 2021), they are in fact rather brittle when applied broadly. For example, a sys- tem trained to identify CHEMICAL names, when applied to texts (even scientific) that do not mention any chemicals, will still (erroneously) recognize many entities as belonging to the chemical class. We quantify this tendency in Section 7. The next- generation NER system should work also when applied broadly.
As LLMMs seem to excel gathering vast and relevant world knowledge in their pre-training stage and using in their predictions, this raises hopes
To facilitate this, evalua- tion for a NER system for a given entity type should
Formal Requirement:

---

towards accurate entity recognition systems that can handle the rich of possible entity types, at various levels of granularity.
the text all and only spans that mention an entity of this type. Unlike the supervised case, here the desired entity type not known at train time, but rather provided to the model as an additional input at test time.
Task definition: Fine-grained Supervised NER The supervised NER task is well established in its form: training stage is performed over texts an- notated with typed, labeled spans, where the types come from predefined set of types denoting tities. At test time, the system gets input new, unlabeled texts, on which it should identify and mark labeled spans of the entity types seen during training. Future supervised NER systems should retain the established task definition, but be trained and tested on a much richer set of entity types, which includes a wide range of fine-grained hier- archical, and intersectional entity types. Ideally, we should aim for the training set to be small as possible.
Like in the supervised case, the ideal zero-shot model will be able support fine-grained, hierar- chical, and intersectional types.
3.4 From entity identification exhaustive retrieval
Traditional NER tasks focus on the identification (or recognition) of entities within text. This quires model process the text for any entity of interest, which can become both slow and ex- pensive when users want to extract mentions of arbitrary entity types from a large text collection, where most documents do not contain an entity of interest ("what are all the snakes mentioned in this corpus").
3.3 From supervised to zero-shot based NER
While supervised NER is effective, it requires the collection of dedicated training Users would like identify entities of their novel types of in- terest, without needing to collect data and train a model. This desire becomes more prominent as the range of entity types to identify grows and becomes more granular: constructing supervised NER sys- tems for each category becomes impractical. We thus seek a zero-shot system, that can identify en- tity types not seen in training, based on their name description.
We thus propose the challenge of NERetrieve: moving the NER task from zero-shot recognition to zero-shot retrieval. In this setup, model is used to process large corpus and index it, such that, at test time, a user can issue entity-type queries for novel entity types, and the model will return all mentions of entities of this type in the collection (like in the zero-shot setup), but without scanning all the documents in the collection.
Crucially, unlike standard retrieval setups which are focused on finding the most relevant documents in collection, the NERetrieve setup is exhaustive: we aim to locate all the relevant documents in the indexed collection.
The zero-shot task is substantially more challeng- ing than the supervised one, requiring the model to rely on its previous "knowledge" to both scope the boundaries of the type to be identified given a class name or a description, as well as to identify potential members of this class (e.g. knowing that Lusotitan atalaiensis is dinosaur) and identifying supporting hints for class membership (if someone obtained Ph.D. from X then X is an educational institute).
Task definition: Exhaustive Typed-Entity Re- trieval In the exhaustive retrieval setup. the entity type label (such as "DoG BREEDs") functions as a query to retrieve a comprehensive collection of all relevant documents containing mentions of entities belonging to that specific category, During testing. the model will encounter previously unseen queries (entity types) and will be required to retrieve all available documents and mark the relevant spans in each one.
Recent experience with LLMs suggests that, for many instances, the model indeed possesses the required knowledge, suggesting the zero-shot setup is achievable (Wang et al., 2022b; Agrawal et al., 2022; Hu al., 2023; Wei et al., 2023).
3.5 To Summarize
Task definition: Zero-shot Fine-grained NER In the zero-shot setup, the model can be trained however its designer sees fit. At test time, the model is presented with text, and an entity cat- egory name ("dog breeds"), and should mark on
Supervised NER is still not "solved" for the general case, and future techniques should work towards working also with fine-grained and specialized en- tity types, and being robust also on texts that do not naturally mention the desired entity types. A

---

step further will move from the supervised case to a zero-shot one, where the entity type to be identified is not known at train time, but given as input at test time. Evidence from LLMs suggests that such a task is feasible, yet still far from working well in practice. Finally, we envision moving zero-shot NER from recognition to retrieval, in a setup that assumes single pre-processing step over the data, indexing, and allowing to retrieve all mentions of entities of given category which is unknown at indexing time and only specified at query time.
proaches to the entity-typing task could be parts of a solution for the various NER tasks we propose.
Few-shot NER Work on Few-shot NER (Wang et al., 2022b; Ma et al., 2022; Das et al., 2022) sumes a variation on the supervised setup where the number of training samples is very small (be- low 20 samples). While this setup indeed facilitates schema-frre freee NER, we argue that it still requires sig- nificant effort from the user, and that a zero-shot ap- proach (which given LLMs seems within reach) is highly preferable. Our vision for NER is zero-shot, not few-shot. Our dataset, however, can also cas- ily support the few-shot setup for researchers who choose to tackle a challenging, fine-grained variant of Compared to FewNERD (Ding et al., 2021), the current facto standard for this task, we pro- vide more and more fine- tne-grained entity types.
4 Relation to Existing Tasks
Fine-grained NER While the "canonical" NER task in general-domain NLP centers around very broad categories (PERSON, LOCATION, ORGANI- ZATION, ...), there have been numerous attempts to define more fine-grained types, often by refining the coarse- grrsse-rraned ones. Sekine and colleagues (Sekine et al., 2002; Sekine, 2008) converged on a schema consisting of 200 diverse types, and Ling and Weld (2012) introduced FIGER, a Freebase- based schema based on the Freebase knowledge graph, comprising 112 entity types. Ding et al. (2021) was inspired by FIGER schema when an- notating "Few-Nerd", which consists of 66 fine- grained entity types and is the largest fine- grainea NER dataset that was annotated by humans. In contrast to the schema-based sets that aim to define a universal category set that will cover "everything of interest", we argue that such set is not real- istic, and each user has their own unique entity identification needs. Hence, the label set should be not only fine-grained but also dynamic, letting each user express their unique needs. Our vision for NER is thus zero-shot over very fine- grained types and our selection of 500 categories is thus not supposed to reflect a coherent "all-encompassing" set, but rather an eclectic set of fine-grained and often niche categories, representative of such niche information needs. Entity typing, closely related field to NER, aligns with our proposed vision.
Zero-shot NER Zero shot NER is already gain- ing momentum (Hu et al., 2023; Wang al., 2023). We support this trend, and push towards a finer- grained and larger scale evaluation.
Entity Retrieval and Multi-Evidence QA Entity-retrieval tasks aim to retrieve information about entities based on a description. For example, Hasibi et al. (2017) introduced the LISTSEARCH subtask in (Bpedda-Ennti v2, in which the user en- ters fine-grained queries with the aim of retrieving a comprehensive list of DBpedia matching the query, and Malaviya et al. (2023) introduced QUEST, a dataset for entity retrieval from Wikipedia with diverse set of queries that feature intersectionality and negation operations, mirroring our desire for fineregraindd and intersectional types. In (Amouyal et al., 2022; Zhong et al., 2022), the set of entities to be retrieved is defined based on a question that has multiple answers ("hho is or was a member of the Australian Army?").
These tasks all focus on retrieving set of enti- ties based on a description. In contrast to this type- focused view, the NERetrieve task is concerned instcad with mentions within documents and re- quires finding not only the set of entities but also all mentions of these entities in a collection of text. This includes complications such as resolving am- biguous mentions, finding aliases of entities, and dealing with ambiguous strings that may or may not correspond the entity.
Choi et al. (2018) introduced the concept of Ultra-Fine Entity Typing: using free-form phrases to describe suitable types for given target entity. While this echoes our interest in ultra-finee fine--nnnnnnn finengrained types, in their task the model is presented with an in-context mention and has to predict category- names for it. In contrast, in the zero-shot NER tasks, the user gets to decide on the name, and the model should recognize it. Nonetheless, ap-
These entity-retrieval tasks described this sec- tion can be an initial component in a system that attempts to perform the complete NERetrieve task.

---

5 The NERetrieve Dataset
Wikipedia. For each paragraph, we mark the spans of entities that appear in it, from set of 500 entity types. The paragraphs were selected to favor cases with multiple entity types in the same paragraph. The selection of entity types was carefully curated to include diverse range of characteristics. This includes combination of low and high-level hier- archy (Insects and Animal), First and second-order intersections (Canoeist and German Canoeist), eas- ier (Company) and more challenging types (Christ- mas albums), as well as types with both few and numerous unique entities. This approach ensures a balanced representation of different entity char- acteristics in the dataset. Due to the increasing specificity in entity types, each corpus span might be marked for multiple entity types.
Having spelled out our desired future for NER tasks, devise dataset that will facilitate both research and evaluation towards the range of pro- posed tasks, from fine- me--griieed supervised recogni- tion to zero-shot exhaustive retrieval.
Firstly, to support supervised fine-grained, hier- archical and intersectional Named Entity Recog- nition (NER), the dataset is designed to contain large number of entity types (500) selected such that they include wide array of fine-grained. domain-specific, and intersectional entity types, as well as some casier ones. Each entity type encom- passes a substantial number of entities and entity mentions, which are scattered across thousands of distinct paragraphs for each entity type, thereby providing a rich, varied resource for effective su- pervised fine- grained NER tasks, and allowing eval- uation also on specific phenomena (e.g., only hier- archical types, only intersectional, and so on). The large number of classes and paragraphs also pro- vides a good testbed for robustness, verifying that a NER system trained on a subset of the classes does not pick up mentions from other classes.
We provide an 80%/20% train/test split, ensuring that each split maintains an approximate 80%/20% distribution of all entity types (for the supervised NER setup). Furthermore, paragraphs are unique to each set and do not overlap. In addition, for the zero-shot NER and the retrieval setups, des- ignate 400 entity types for training and 100 for testing.
The same setup transfers naturally also to the Zero-shot fine-grained NER task, as the entity types collection is designed reflect a diverse range of interests and world knowledge, similar to what potentially causal users would expect from a zero-shot NER system. We also provide an entity- split, detailing which entity types can be trained on, and which are reserved for test time.
6 Technical Details of Dataset Creation
With more than 4 million paragraphs and 500 entity types, exhaustive human annotation not feasible. We therefore seek an effective method for creating data, while retaining high levels of both precision (if span is marked with a given type, it indeed belongs to that type) and compre- hensiveness (all spans mentioning a type are an- notated), At high level, we rely on high-quality human-curated entity lists for each entity type (part of the inclusion criteria for an entity type is the quality of its list), where each list includes several aliases for each entity. We then collect paragraphs that include relaxed matches of the entities in our entity lists.3 Due to language ambiguity, the result- ing matches noisy, and some marked spans do not refer to the desired entity (not all mentions of "the Matrix" are of the science fiction movie). We filter these based on a contextual classifier trained over a very large document collection, which we find to perform adequately for this purpose.
Finally, to support the novel exhaustive typed- entity retrieval task, the dataset is tailored suit scenarios where knowledge corpora are massive, encompassing millions of texts. This necessitates more complex methodologies than those currently available, and our datasct is specifically constructed to facilitate the evolution of such advanced tech- niques. Each entity type potentially corresponds thousands of different paragraphs with median of 23,000 relevant documents for each entity type. Each entity type label serves as a descriptive name that can be used as a query. At the same time, we kept the dataset size managcable (4.29M entries) to allow usage also on modest hardware.
A final requirement is to make the dataset free to use and distribute, requiring its construction over an open resource.
"See Appendix A.I for examples 'While no list is fully comprehensive, taking only para- graphs on which entitics from the list matched to begin with reduces the chances of the texts to include non-covered entity, compared to using random texts and annotating them.
We introduce NERetrieve, a dataset consisting of around 4.29 million paragraphs from English

---

Curating Entity Lists Each entity type in our dataset corresponds to an ontology from the Cali- graph knowledge base (Heist and Paulheim, 2020) which is in turn based on a combination of DBPedia Ontology, and Wikipedia categories and list pages. We collect 500 ontologies encompassing total of 1.4M unique entities,, and we further augment each entity with all possible aliases in the corresponding Wikidata entity page under the "Also known as" field.
tives are randomly chosen paragraphs from other entity types. The model essentially determines the likelihood of encountering entities of type X within a given text. The choice of a linear model reduces the risk of entity-specific memorization (Tänzer et al., 2022), thereby focusing on the context in- stead of specific entities. Preliminary experiments indicate its efficiency at scale. The model is run all paragraphs for each entity, discarding those predicted as low compatibility with the respective entity type.
We selected ontologies such that the resulting set covers a diversity of topics, while also ensuring that each individual ontology has high coverage, and represents a concrete meaningful category, exclud- ing abstract or time-dependent categories. Certain ontologies were identified subsets of larger on- tologies, thereby establishing a hierarchical struc- ture among the types (e.g., the "Invertebrate" on- tology part of the broader "Animals" ontology). We also ensured that many, but not all ontologies are of intersectional types.
Quality Assessment of the Resulting Dataset To assess the reliability of the proposed dataset, we randomly selected 150 paragraphs and man- ually evaluated each tagged entity and its corre- sponding entity types. This process consisted of two steps: firstly, confirming the accurate tagging of entities within the text, and secondly, ensur- ing the correctness of each (entity, entity type) } pair. The 150 paragraphs collectively con- tained 911 typed entity mentions, with each typed mention being cross-validate against Wikipedia, which serves as the source for both the entity types and the paragraphs. Of these 911 typed mentions, 94% were tagged accurately. While not perfect, consider this to be an acceptable level of accu- racy considering the datasets size, coverage, and intended use. Nevertheless, we plan to release the dataset together with a versioning and reports mech- anism that will allow future users to submit mis- takes, resulting in more accurate versions of the dataset over time.
Tagging and Filtering Entity Mentions Each entity within the dataset was mapped to all Wikipedia articles that referenced it, based on DB- pedia's link graph (Lehmann et al., 2015). As a result, the extraction of entities from Wikipedia article texts occurred solely on Wikipedia pages where at least one of the entities was explicitly mentioned. We indexed this document set with ElasticSearch, which we then queried for occur- rences of any of the 1.4M entities or their aliases, using a relaxed matching criterion that requires all the entity words to match exactly, but allows up to 5 tokens (slop 5) between words.
7 Performance of Current Models
The result of this stage matched significantly more than 4.29M paragraphs, and is exhaustive but noisy: not all spans matching an alias of a typed entity are indeed mentions of that entity. We resolved this using a contextual filtering model, which we found to be effective. We then apply this model to select paragraphs that both contain many entities of interest, and were determined by the filtering model to be reliable contexts for these entities.
The NERetrieve dataset allow us to assess the per- formance of current models on the challenging con- ditions describe in section 3.
Models For the supervised case, we train (fine- tune) Spacy? NER models over a pre-trained DcBERTa-v3-large MLM. For the zero-shot see- narios, we prompt gpt-3.5-turbo and GPT4.
7.1 Robustness on broad-scale texts
The first experiment quantifies the robustness of NER models when applied to domain-adjacent texts, texts that do not contain and are not expected to contain, entities of the types the model aims to identify.
Our filtering model is Bag-of Pag-of-Worde Words Linear SVM, trained individually for each entity type us- ing TF/IDF word features. Positive examples are from a held-out set matching the type, while nega-
For the supervised case, we selected 20 random entity types. For each entity type, we trained a
'caligraph. org
"See Appendix A.3 for entity types
wi kidaaa. org
"https://spacy"" .o/

---

Exact
Relaxed
Performance
80-
Model
P
R
Fi
R
F
75-
GPT3.5
0.23
0.27
0.22
0.32
0.37
0.31
70-
GPT4
0.38
0.56
0.41
0.53
0.75
0.57
65-
60-
Table I: Zero-shot NER Performance of GPT-3.5 and GPT-4 on 1,000 Paragraphs from NERetrieve. Para- graphs were sampled using stratified method ensure uniform distribution of each entity type.
Prectsion
Animal
Invertebrate
Insect
Butterfly
Ontologies
Figure NER performance across increasing levels of granular hierarchical classification. Model precision decreases as the entity type becomes more une-grained, owing the misclassification of similar types
strate the challenge, we selected five related in- tersected entity types: INDIAN WRITER, NORTH AMERICAN WRITER, BRITISH WRITER, POLISH WRITER, and NORWEGIAN WRITER. Here, we trained 5-way NER model on train-set contain- ing entities of all these types. The model was eval- uated on a test set containing these types as well as related entities not present in the train set, which consists of professions such as AMBASSADOR. JUDGE, DESIGNER, PAINTER, and GUITARIST. This was done to assess the model's ability to dif- ferentiate between Writers and other professions and to differentiate authors of different nationali- ties. The model achieved very low micro- average F1 score of 0.14 (precision: 0.08, recall: 0.73), demonstrating the difficulty of the task. We then combined all five writer-types above to a single Writer class. A NER model trained for this class has an F1 score of 0.44 (precision: 0.31, recall: 0.79), substantially higher but still very low.
NER model with a train set of 15,000 samples that included that specific entity type. We then tested this model on separate of 10,000 randomly selected paragraphs that did not have the targeted entity type. On average, models incorrectly iden- tified an entity (a False Positive) in 1 out of every 4 paragraphs. After manually reviewing a portion of these false positives, we confirmed that none of them actually contained the specified entity type. This stresses the brittleness of supervised models in this setting. For the Zero-shot setting, a prompted spt-bo model was remarkably more robust, only identifying a false positive in out of 500 paragraphs. This highlights the robustness of the LLMs in the zero-shot setting and suggests that such robustness should also be possible for dedi- cated, supervised models.
7.2 Supervised Fine-grained NER
7.3 Zero-Shot Fine-Grained NER with LLM
Increasing level of specificity We measure exist- ing supervised and zero-shot NER performance, as we increase the specificity of the type to be iden- tified. We consider hierarchical chain of entity types wherein each subsequent type has greater granularity levels than the previous one: ANIMAL + INVERTEBRATE - INSECT + BUTTERFLY. We train a supervised NER model for each type. Each training set consisted of 15,000 paragraphs, and evaluate all of them on all the paragraphs that contain the ANIMAL type (which also includes the other types).
To assess current LLMs ability to perform zero- shot NER over wide range of fine-grained entity types, we designed the following experiment. We randomly draw 1,000 paragraphs from the dataset, while attempting to achieve an approximately uni- form distribution of entity types in the sample. For each paragraph. we prompt the LLM with the paragraph text, and asked it to list all entities of types that appear in the paragraph, as well as an additional, random entity type (for example, for paragraph containing mentions of FOREST and PROTECTED AREA the prompt would ask to extract mentions of FOREST, PROTECTED AREA and LIGHTHOUSE We assess both GPT-3.5-turoo (Ouyang et al., 2022) and GPT-4 (OpenAI, 2023), using their function calling API features to struc- ture the output in JSON format.
The results in Figure confirm that both pre- cision and Fl indeed drop significantly as we in- crease the level of specificity.
Intersection-types Intersection types require the model to accurately distinguish closely related en- tities with fine- -na-nnd distinctions. To demon-
"See Appendix A.2 For prompt and examples.

---

Model
Recll
2023) and B5-v2 (Wang et al., 2022a). The dense models were selected as they held the leading po- sition in the retrieval category of MTEB text em- bedding benchmark (Muennighoff et al., 2023)', and exhibit particularly strong performance in the DBpediaa-ntity-22 task (Hasibi ct al., 2017), spe- cific IR task designed for entity retrieval. Addition- ally, for the sentence encoder models, we encoded each paragraph in the dataset sured cosine similarity with the query vector to de- termine the ranking. We report the performance of the IR systems with Recall @RELI metric, where RREL represents the total number of relevant docu- ments associated with given query. This metric is analogous to the R-Precision measurement (Sander- son, 2010). We compute the average of this metric over each of the 100 entity types in the test set, where the entity type serves as the query. Based on the findings Table 2, it becomes evident that despite utilizing state-of-the-art models for seman- tic search, our progress this task is significantly distant from our desired objectives. Intriguingly, the number of relevant documents within the entity type does not demonstrate a correlation' 10 with the Recall @IELI measurement, for the E5-v2-bas. experiment. This suggests that the task is not chal- lenging solety due the existence of a vast volume of texts, but rather because of the inherent complex- ity of the task itself.
BM25
0.221 + 0.16
E5-v2-base
0.331 + 0.15
E5-v2-large
0.326 = 0.14
GTE--base
0.396 + 0.16
GTE-large
0.397 0.16
RGF-hase
0.383 0.16
BGE-large
0.369
Table 2: Performance evaluation of Exhaustive Typed- Entity Mention Retrieval on the test set in Zero-Shot setup: mean and standard deviation of Recall @ @RELI across all entity types.
vector and mea-
In Table We report the Recall, Precision, and Fl scores for both the Exact Match, wherein the generated string is compared directly with the tagged entity in the dataset, and the Relaxed Match, which considers a match based on word overlap between the generated and the gold entity in the dataset. Notably, we observed substantial im- provement in all metries with GPT-4 compared to GPT-3.5-turbo. However, despite these advances, the results remain relatively low, indicating con- siderable potential for further improvement this task. These findings are consistent with other stud- ies that highlight the struggles encountered by GPT models in zero-shot NER tasks across classical NER tasks (Wang et al., 2023). This challenge be- comes even more pronounced when the entity types become more fine-grained and specific, resulting in performance that falls short when compared to supervised models (Hu et al., 2023).
8 Conclusion
We argued that NER remains a relevant and chal- lenging task even in the age of LLMs: rather than solving NER, the introduction of LLMs allows us to take the NER task to the next level.
7.4 Exhaustive Typed-Entin Mention Retrieval
To our knowledge, no current system is directly applicable to our proposed NERetrieve task: text embedding and similarity- based methods are not designed for the entity-type identification task, and retrieval systems are not designed to support ex- haustive retrieval of all items (each of our queries should return median of 23,000 paragraphs) but rather focus on returning the top-matching results. Indeed, this task stresses the limits of both retrieval and semantic embedding methods, calling for fu- ture research. We nonetheless make best-effort experiment within the existing models and meth- ods.
We highlight three challenging directions in which the task can evolve, culminating in a move from named entity recognition named entity re- trieval, and created a unique dataset that allows future explorations of these directions.
Limitations
Silver Data The dataset we provide is silver- annotated, meaning it's automatically annotated by algorithm and not manually curated. While this allows for scalability and provides large num- ber of examples, it inevitably introduces errors and inconsistencies. Silver annotations, while useful
We evaluate NER-retrieval on BM25 (Robert- son et al., 2009) for sparse retrieval, and on three state-of-the-an sentence encoder models for dense retrieval: BGE (Xiao et al., 2023), GTE (Li et al.,
http:/faacc.co/ccce/eeeeeee/eeee leaderboard
Peaason correlation coefficient r -0.06

---

for training models in the absence of extensive manually-annotan gold-standard datascts, are not a perfect substitute. Their accuracy depends on the performance of the algorithms used for annota- tion, and there may be variations in quality across different types of entities or texts.
our research community who did reach out and were even reluctant to publicly acknowledge the inhumanity and total immorality of these acts.
We fervently call for the immediate release of all those who have been taken hostage and urge the academic community to unite in condemnation of these unspeakable atrocities committed by Hamas, who claim to be acting in the name of the Pales- tinian people. We call all join us in advocating for the prompt and safe return of the abductees, as we stand together in the pursuit of justice and peace.
Exhaustiveness In our pursuit of constructing a comprehensive corpus, it is essential to acknowl- edge potential limitations inherent to our dataset. We recognize the possibilityy of underrepresentation for certain entity types or categories, particularly those that rare or lack dedicated Wikipedia article. Furthermore, nuanced details may occasion- ally go unnoticed. Nevertheless, our resource com- prises an extensive catalog of 1.4 million unique entities, indicating a significant stride toward com- prehensive data coverage. It presents notable attempt to achieve level of exhaustiveness within the bounds of feasibility while striving to exceed the limitations inherent in manual human annota- tion efforts.
This paper was finalized in the wake these events, under great stress while we grieve and mourn. It may contain subtle errors.
A Acknowledgements
We would like to thank Nicolas Heist of the Cali- graph project for his assistance and feedback. This project has received funding from the Euro- pean Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 802774 (iEX- TRACT).
While these limitations should be considered, they do not diminish the potential of our resources and our vision to stimulate innovation in NER re- search. We encourage the research community to join us in refining this resource, tackling the high- lighted challenges, and moving toward the next generation of Named Entity Recognition.
References
Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models zero-shot clinical information extractors. arXiv preprint arXiv:2205. 12689.
10 Ethics and Broader Impact
Samuel Joseph Amouyal, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022. Qam- pari: : An open-domain question answering bench- mark for questions with many answers from multiple paragraphs. ArXiv, abs/2205. 126665.
This paper is submitted in the wake of tragic ter- rorist attack perpetrated by Hamas, which has left our nation profoundly devastated. On October 7, 2023, thousands of Palestinian terrorists infiltrated the Israeli border, launching a brutal assault on 22 Israeli villages. They methodically moved from home to home brutally torturing and murdering more than 1,400 innocent lives, spanning from infants to the elderly. In addition to this horrifying loss of life, hundreds of civilians were abducted and taken to Gaza. The families of these abductees have been left in agonizing uncertainty, as no infor- mation, not even the status of their loved ones, has been disclosed by Hamas.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018 Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87-96, Melbourne, Australia. Association for Computational Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. 2022. Scaling instruction finetuned finettiio-fieetuueed language models. arXiv preprint arXiv: 10.1 11416.
The heinous acts committed during this attack, which include acts such shootings, raping, burn- ings, and beheadings, beyond any justification. In addition to the loss we suffered as a nation and as human beings due to this violence, many of us feel abandoned and betrayed by members of
Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca Passonneau, and Rui Zhang. 2022. CONTaiNER: Few-shot named entity recognition via contrastive learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume Long Papers), pages 6338-6353. Dublin, Ireland, Association for Computational Linguistics.

---

Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Human Language Technologies, pages 5721-5732. Seattle, United States. Association for Computational Linguisties.
Chaitanya Malaviya, Peter Shaw, Ming- Chang, Kenton Lee, and Kristina Toutanova. 2023. Quest: A retrieval dataset of entity-seeking queries with im- plicit set operations. ArXiv, abs/2305.11694,
Shervin Malmasi, Anjie Fang, Besnik Fetahu, Sudipta Kar, and Oleg Rokhlenko. 2022. Semeval-2022 task 11: Multilingual complex named entity recognition (multiconer). In International Workshop on Semantic Evaluation.
Ning Ding. Guangwei Xu, Yulin Chen, Xiaobin Wang. Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021. Few-nerd A few-shot named entity recog- nition dataset. arXiv preprint arXiv:2105.077 07464.
Jinlan Fu, Pengfei Liu, Qi Zhang, and Xuanjing Huang. 2020. Rethinking generalization of neural mod- els: A named entity recognition case study. ArXiv, abs/2001.0338 033444.
Niklas Muennighoff. Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putatiooal Linguistics, pages 2014-2037, Dubrovnik, Croatia, Association for Computational Linguistics.
Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisz- tian Balog. Svein Erik Bratsberg. Alexander Kotov, and Jamie Callan. 2017. Dbpedia-entity v2: A test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR '17. pages 1265-1268. ACM.
OpenAL. 2023. Gpt-4 technical report. ArXiv. abs/2303.08774.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida. Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton. Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv, abs/2203.021 02555.
Nicolas Heist and Heiko Paulheim. 2020. Entity ex- traction from wikipedia list pages. In The Semantic Web: 17th International Conference, ESWC 2020. Heraklion, Crete, Greece, May 31-Jume 4, 2020, Pro- ceedings 17, pages 327-342. Springer.
Yan Hu, Iqra Ameer, Xu Zuo, Xueqing Peng, Yujia Zhou, Zehan Li, Yiming Li, Jianfu Li. Xiaoqian Jiang, and Hua Xu. 2023. 7ero-shot clinical entity recogni- tion using chatgpt. arXiv preprint ar3iv:2303. 16416.
Long Phan, James T. Anibal, Hieu Trung Tran, Shau- rya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet 2021. Seifive: a text-to-text transformer model for biomedical literature. ArXiv, 003.0988.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jacwoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Karthik Raman, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi. and Krishna Srinivasan. 2022. Transforming sequence tagging into seq2seq task. In Conference on Empirical Methods in Natural Language Processing.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey. Patrick van Kleef, S. Auer, and Christian Bizer. 2015. Dbpedia- large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6:95
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 34).333-388
Zehan Li, Xin Zhang, Yanzhao Zhang. Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning.
Mark Sanderson. 2010. Christopher d. manning, prab- hakar raghavan, hinrich schütze, introduction to in- formation retrieval, cambridge university press. 2008. isbn- 13
Xiao Ling and Daniel Weld. 2012. Fine-grained entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 94-100.
88571-5615 xxi+ 482 pages. Natural Language Engineering, 16(1):100-133.
Satoshi Sekine. 2008 Extended named entity ontol- ogy with attribute information. In Proceedings of the Sixth International Conference on Language Re- sources and Evaluation (8), Marrakech, Mo- rocco. European Language Resources Association (ELRA).
Ruotian Ma. Xin Zhou, Tao Gui. Yiding Tan, Linyang Li, Qi 7hang, and Xuanjing Huang. 2022. Template- free prompt tuning for sho-shot NER. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics:

---

Satoshi Sekine, Kiyoshi Sudo, and Chikashi Nobata. 2002. Extended named entity hierarchy. In Pro- ceedings of the Third International Conference Language Resources and Evaluation (LREC'02), Las Palmas, Canary Islands Spain. European Language Resources Association (ELRA)).
Victor Zhong, Weijia Shi, Wen Yih, and Luke Zettle- moyer. 2022. Romqa: A benchmark for robust, multi- evidence, multi-answer question answering. ArXiv, abs/2210.14353.
Wenxuan Zhou and Muhao Chen. 2021. Learning from noisy labels for entity-centric information extraction. arXiv preprint arXiv: 10e:2210440555
Burr Settles. 2004. Biomedical named entity recogni- tion using conditional random fields and rich feature sets. In Proceedings of the international joint work- shop natural language processing in biomedicine and its applications (NLPBA/BiONLP) pages 107- 110.
Michael Tänzer, Sebastian Ruder, and Marek Rei. 2022. Memorisation versus generalisation pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 7564-7578, Dublin, Ireland. Association for Computational Lin- guistics.
Erik Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independen named entity recognition. In COLING-O2: The 6th Conference on Natural Language Learning 2002 (CoNL1-2002).
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022a. Text embeddings by weakly- supervised contrastive pre-training. arXiv preprint arXiv:2212.03533.
Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang. Wei Yu Wu, and Weiran Xu. 2022b. Instructionner: multi-task instruction- based generative framework for few-shot ner. ArXiv, abs/2203.03903.
Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. ArXiv, 10422004..044.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang. Xin Zhang. Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprini arXiv:2302. 10205.
Ralph Weischedel, Martha Palmer. Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5.0 1dc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources advance general chinese embedding.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: Deep con- textualized entity representations with antaty- self-attention. arXiv preprint arXivv2010.01005.

---

A Appendix
A.1 Example Paragraphs from the NERetrieve Dataset
Canelo Alvarez vs. Julio César Chávez Jr. paparta event/bowes) was professional boxing fight held May 6, 2017 at the T-Mobile p sugdoog/tt in Paradise, Nevada ctty/city the kocessas)- part of the Las Vegas metropolitan area. alraae decision, having been judged the winner of all 12 rounds each of the ihree ringside judges
was declared the winner by unanimous
Aristotelia devexella herthr Toseet
Pabaryyt /epeeterree//leeeee is moth of the family Gelechiidae. It was described by Annette Frances Baaant Se Remeeeeeeeee in 1925. It is found in North America, where has been recorded from Alherta, Arizona and Oklahoma.
In June 1921.
Malacrida pRot/essss velzer/Rosen writest met her future husbund, Italian journalist and for- charitable fundraising event known Alexandra Rose Day
Marchese Piero Malacrida de Saint-Auguff toemmatttttitttttttttt /toee /Decignerls mer cavalry officer, The Rite Hotel, London Londoo/uuisstt//eeeee [uiisiinn/loaaa)- They were married
December 1922, St Bortholomew the Great /tt.a////tuchh//ssasree making her the Marchesa Malacrida de Saint-August Her husband's 's family were a noble family from Lombardy. Shortly after their wedding. her busband expunded his activities into writing on interior design, and designing interions, especially luxury buthrooms, the upper class. The couple would buy flats smart Londongesey addresses. then remodel sell them, trading under name Olivotti. In 1926-1929, they lived at Upper Brook Street, Mayfair pasteser__tt tnttser__f__eedded -Lenden)-
During the intense study of the fossils it became clear that they did represent Barosaurus SerrssauruSSStnneeonn (CEnossur] but species new science. 2012 this was named Kantedocus siberi by the Swiss palaeontologist Emanuel Tschopp sscentsst) who as boy had visited the excavations, and his Por- tuguese colleague Octávio Mateus ([ciantist)- The generic name, which means small beam, combines reference to the related form Dipladocus (eincsaur) with a Crow Indian diminutive suffi kaate. The specitic name honours Siber.
In 1995, the Government of Bangladesh declared Sylheljesey) as the sixth divisional head- quarters of the country. Sytherrorry has played vital role in the Bangladeshi economy. Several of Bangladesh's finance ministers have been Members of Purliument from the city of Syyher Badar Uddin Ahmed Kamran _e polttsitaen///eerr was a longtime mayor Sylhetresey Humayun Rashid Choudhury, neoggal teot teeettreeaaaneeeeeeeeeeo tow a diplomat, served as President of the UN General Assembly and Speaker of the Bangladesh National Parliament.
Table 3: Annotated paragraphs from the NERetrieve dataset. Entities may correspond to multiple entity types, representing diverse aspects. In this example, different entity types associated with the same entity are separated by slashes.

---

A.2 Zero-shot NER with GPT Prompt and examples
The following JSON used for the function calling feature in OpenAl API ent entity_type represents a list of entity type names separated by a comma.
"name"; "zero_shot_ner",
eesccripiions 1"For each entity type: tentity_tyee Extract all relevant entities. Entity appear in more than one entity type.", "paramerers":
"type": "object",
"properties":
"entities":
"typ""; "array", iites":
"type"; "objest",
"properties":
"entity_typs";
"type": "string", "description": "Category the named entity."
"entities";
"type": "string", "description"; "A 1ist of Named entities extracted from text.*
"required": ["entities")
A performance of the complete organ works of Johann Sebastian Bach sclasties: corpoxttion]
was held on November 22, 2014 St. Peter's Lutheran Church palseious hattan. Twenty organists from The Juilliard School School techon///ausstiona instiburien) performed. Joilliand Organ Department Chair, Poul Jacobs Jacobs (erpesoe /hhooiis masician) curated the eighteen-hour performance. Johann Sebastian Bach compoae emiccianCC.asieu musicion/tazman alassioal musician] was a key figure in event.
in Man-
Jimmy Dorsey (auss ausician/Sasppeeuaa is considered one of the most important and influential alto saxophone players of the Big Band and Swing eru, and also after that Jazz saxophonists Lester Young; jasa maatettannSeeephoneee and Charlie Parkerrawes mudeetano/ both acknowl- edge him us an important influence on their styles.
Lapland Reserve (panwets is located the Scandinavian and Russian taiga ecoregion which is situated in Northern Europe between tundru in north and temperate mixed forests in the south. It is covers parts of Norway, Sweden, Finland and the northern part of European Russia, being the largest ecoregion in Europe. The ecoregion is character- ized by coniferous forests dominated by Pinus sylvestris often with understory of Juniperus convnunis, Picea abies and Picea obovad poseat] Betula pubescens and Betda pendula poreart Larix sibirica prreett is characteristic of the castern part of the ecoregion.
significant admixture of
Table 4: Example of GPT4 extracted entities and entity types. Entities may correspond to multiple entity types, representing diverse aspects. False positive entity types are marked in red, In this example, different entity types associated with the same entity separated by slashes.
Phttos://platforre www.ao.penae....m..o http:///w..aaaaaaooc///////////a

---

A.3 Entity Types Infographic
Blology and Genetics
Genre
Occupation oo
Music
Sclentiic Profession
Art
Astrnoon and
Specific Occupation
Plants
Water Bodies
Athietess and Spartspeople
Scaanie and Techrudog
Arts and Literature
Grogr aphical Features
Sports Recroation
nature Enwrenment
Animals
Medie and Entertainmene
Arts and Design
Toumum Ocoupations
and Masoioseua
Attraction Types
Sconeerbare bushings
Film and relerision
FillAs und Covennmrr
Bisthee and Commanon
aesiiaaee atd Charicmes
Find Drosk
Criminal Actvities
Geographical Lecations
spliteitty
Specific Structure
Estellanmomt
Religious Buildings
Dishes
Type of Structure
Companice aaapannis by Induseo
Relialos
mrnssorttat tation Modes
Poli Political Position
Film Genre
memoor infoama dtnu
Film Origin
Mllcary Vehicles
ain Formet
Figure 3: All the 500 entity types in the dataset are arranged topically in an easy-to sk-m diagram. Note that this diagram does not reflect all the hierarchical relations between entity types in the dataset, as it is capped at hierarchy level for aesthetic reasons.


Reviews:
[{"review_id": "BHZgp1ziI0", "reviewer": "Reviewer_Yuir", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for finding our ideas interesting to the research community and providing valuable intuitions for future research!**\n\n**Regarding your comment:**  *Considering this as a position paper, I am not quite sure what's is the main contribution of this paper:*\n \nWhile the nature of a position paper is often to spotlight key areas of interest for the research community, our paper aims to both highlight pertinent issues and provide empirical evidence to support our assertions. So this could be its own contribution. However, we believe our paper also goes beyond that. Let us try and explain what we view as our contributions in this work:\nFirst, in terms of the position expressed in the paper, we ask what is the role of named-entity-recognition tasks in the era of LLMs. We hear many voices that consider NER tasks being solved by LLMs, and we argue this is not the case: while the “standard” NER settings might be solved, we argue that the standard tasks were simple because this is what we could solve at the time, and that we now can take NER to the next level. We suggest several concrete directions as to what such next levels may be. Empirically, we present a series of experiments showing that current “standard” models, as well as LLMs, are far from solving these more challenging NER tasks. We hope this convinces readers that there is a lot to be researched around NER, also in this “new era” of LLMs.\n\nSecond, beyond expressing this position and introducing the NER tasks, we also provide a concrete dataset to facilitate research on these tasks. The dataset is designed to support the tasks in the paper including specific train/test splits for each. This is a large-scale, fine-grained NER dataset featuring a variety of domains, 494 entity types including intersectional entity types, long-tail entity types, and hierarchical relations between some (but not all) entity types. As far as we know, our dataset features the largest number of entity types available in a fine-grained NER dataset. Despite its silver-annotation procedure, we establish it to be of high quality. We hope this dataset will facilitate and promote research on future of NER.\n\nWhile this is a position paper, it is one grounded in empirical experiments, and which presents a dataset with the potential to significantly advance the field of NER during the LLM era.\n\n**Regarding your comment:** *The paper presents a silver-annotated dataset, but does not provide a baseline model on the dataset:*\n\nPart of our claim is that there are currently no good models for this set of future-looking tasks. Indeed, for the few-shot and zero-shot tasks, the best models are currently OpenAI’s LLMs, which are infeasible to run at the scale required by the size of the dataset. Thus, instead of providing comprehensive results for these tasks, we opted to demonstrate GPT3.5 and GPT4’s performance on a subset of 1000 paragraphs, showing that these LLMs are far from solving the tasks. For the supervised case we chose to highlight failure cases which we thought to be of interest (e.g., degrading performance with increasing type granularity), but we can also use the same DeBERTa-based models as a baseline for the supervised task, and report supervised baseline results on all classes for camera ready. Finally, for the retrieval task, we evaluated two state-of-the-art IR vector embedding models, e5-v2 and GTR, in both their base and large variants. Their results are detailed in Table 2. We consider them as a baseline for this task. We will make the baseline discussion clearer in camera ready.\n\nLastly, we'd like to mention that we have revised the manuscript, corrected the typos, and improved the overall writing quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed four challenges for NER tasks in the LLM era which are important to the future research. Theses points of view are interesting to the research community and provide valuable intuitions for future research. ", "reasons_to_reject": "There are some typos in the texts, it would be better the authors could improve the writing quality of this paper. Considering this as a position paper, I am not quite sure what's is the main contribution of this paper. The paper presents a silver-annotated dataset, but does not provide a baseline model on the dataset. It would be interesting to see the performance of a baseline model on the presented dataset and lead the improvement for further investigations. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses the NER task in the LLMs era and talks about the next possible directions of next germination of NER tasks. They define four versions of NER task formulation in LLMs era and present a silver-annotated dataset. ", "excitement": 3, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0M2KoiIT2W", "reviewer": "Reviewer_jvxe", "summary": "", "questions": "- I think it is good that it is a position paper and that it is mainly subjective, but in the current situation where many LLMs have a multilingual nature, is it appropriate to go in the direction of data construction and validation targeting only English?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for acknowledging our systematic summary of NER-related tasks post-LLM emergence.**\n\nPlease see our response below:\n\n**Regarding your comment:** *“As for the construction of the silver-annotated corpus of 4 million paragraphs covering 494 entity types, It is not clear whether this is a temporary corpus for preliminary research or a corpus that will be made publicly available in the future”:* \n\nApologies for any oversight on our part, but as we wrote in the manuscript, our dataset is derived from open-source resources (Wikipedia, DBpedia, Wikidata, Calligraph), and is intended for public use. We plan to release it, including task-specific splits, post-anonymity period. All experiments and dataset construction are reproducible, and we'll share the dataset, code, and details on our project's GitHub.\n\n**Regarding your comment:** *“The quality evaluation is only 91% for precision, which does not provide sufficient information to determine the actual quality of the corpus, and its usefulness as a corpus cannot be confirmed.”*\n\nFirst, since the time of our initial submission, we further worked on the dataset quality, improving its accuracy from 91% to 94%. This enhancement was achieved through a series of incremental adjustments such as improvements in the corpus parsing process, incorporating another field of aliases from Wikidata, and minor changes in the data sampling heuristics, effectively reducing potential noise generators in the dataset.\n\nWe do not consider this dataset to be a preliminary resource, but a dataset created with comprehensive efforts with the highest standards which is expected for a new NLP resource.  We found your concern to be important and we will clarify and expand the discussion around the dataset quality assessment. \n\nOur manual evaluations now indicate an accuracy of 94% for the NER labels on 250 random paragraphs, a marked improvement from the earlier 91%. It's worth highlighting that this accuracy is either on par with or surpasses established manually annotated datasets. For instance, **TACRED** reports an annotation accuracy of 93.3% [1], and **SQuAD** registers an 86.8% F1 score in human annotator validation [2].\n\nTo put this number in context, it is useful to compare to other large-scale NER datasets: \n\n(a) **FewNERD**, perhaps the largest and highest quality human annotated NER resource, claims a verified accuracy of 95%. Compared to FewNERD, our dataset has about 7x the number of classes, and more than 20x the number of sentences[3]. Given this, we believe our estimated accuracy level of 94% compares favorably. \n\n(b) **MultiCoNER**, a large-scale silver annotated multilingual NER dataset with 6 coarse entity types and around 217K original sentences (we have about x84 more entity types and more than 20x original paragraphs). They report manual quality assessment of the NER labels for 94% accuracy [4]. Again, given our substantially larger and more challenging entity types, we consider our accuracy to compare favorably.\nBoth of these datasets are well-established works in the modern NER community.\n\nTo further gauge the utility of the dataset, we reported several experiments on different NE-tasks (Supervised setup, zero shot setup,  NE mention retrieval) using our NERetrieve dataset, and for each task, our results are aligned with the trends observed in other research lines which experiment with much more simplified version of NE tasks. (see related works regarding each task). This shows the dataset indeed has discriminatory power. \n\n**Regarding your comment:** *“Even considering that it is a position paper, it is unclear what the main contribution of the paper is. It would seem that the main contribution is a discussion of NER-related tasks that will become mainstream in the future, but similar efforts have already been made for two of the proposed tasks, fine-grained supervised NER and zero-shot fine-grained NER, and they do not seem to be particularly new.”*\n\nFirst, we note that we see our contribution as consisting of two parts: (a) the position: discussing the role of NER in the era of LLMs, showing several directions in which NER tasks could evolve and increase in complexity, and **demonstrating empirically** that even state-of-the-art LLMs are far from solving these harder NER tasks, while still giving hope that we can progress in this direction. (b) providing a concrete dataset that supports research on the NER directions we propose.\n\n\nIn our paper, we discuss various phenomena that, to our knowledge, have been rarely addressed before. We illustrate these phenomena using our unique dataset. For instance, we examine the mix of intersectional granularities in the same task—such as identifying entity-type \"Writer\" from different nationalities and differentiating between professions. We also evaluate the performance of supervised models as they tackle increasing granularity in entity types, illustrated by the example experiment of  ANIMAL → INVERTEBRATE → INSECT → BUTTERFLY in Figure 1. Furthermore, we assess the capabilities of foundation models in extracting fine-grained entities across diverse domains in a zero-shot setup, thereby addressing real challenges that we foresee in the future of NER. We were able to conduct these experiments due to the comprehensive nature of the dataset we have constructed. Additionally, we introduce the task of exhaustive retrieval of entity mentions, which we believe represents a significant advancement in NER tasks. This task is not only novel but also unsupported by existing datasets. \n\n\n**Regarding your question:** *“I think it is good that it is a position paper and that it is mainly subjective, but in the current situation where many LLMs have a multilingual nature, is it appropriate to go in the direction of data construction and validation targeting only English?”*\n\nAs researchers native to a low-resource language, we deeply recognize the importance of multilingual NLP technologies. Indeed, we believe that our “position” part of the paper holds also for non-Englsh: the future of NER is relevant for all languages. However, given the effort and resources required to create the English NERetrieve dataset we present in this work, we find it infeasible to produce both an English and a multilingual dataset in the same work, while having them both be of high quality. We thus prioritized English for this initial effort (as is common in many other works), and reserve the multilingual version for future work, by us or others.\n\nWe recognize that this response is extensive, but we believe it's crucial to clarify key points. Thank you for taking the time to consider our arguments.\n\n[1] Zhang, Yuhao, et al. \"Position-aware attention and supervised data improve slot filling.\" Conference on Empirical Methods in Natural Language Processing. 2017.\n\n[2] Rajpurkar, Pranav, et al. \"Squad: 100,000+ questions for machine comprehension of text.\" arXiv preprint arXiv:1606.05250 (2016).\n\n[3] Ding, Ning, et al. \"Few-nerd: A few-shot named entity recognition dataset.\" arXiv preprint arXiv:2105.07464 (2021).\n\n[4] Malmasi, Shervin, et al. \"Multiconer: a large-scale multilingual dataset for complex named entity recognition.\" arXiv preprint arXiv:2208.14536 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It systematically summarizes NER-related tasks after the emergence of large-scale language models (LLMs).\n- Based on a large silver-annotated corpus constructed by the authors, the accuracy of several NE-related tasks has been reported.", "reasons_to_reject": "- Even considering that it is a position paper, it is unclear what the main contribution of the paper is. It would seem that the main contribution is a discussion of NER-related tasks that will become mainstream in the future, but similar efforts have already been made for two of the proposed tasks, fine-grained supervised NER and zero-shot fine-grained NER, and they do not seem to be particularly new.\n- As for the construction of the silver-annotated corpus of 4 million paragraphs covering 494 entity types, it is not clear whether this is a temporary corpus for preliminary research or a corpus that will be made publicly available in the future. In the latter case, the quality evaluation is only 91% for precision, which does not provide sufficient information to determine the actual quality of the corpus, and its usefulness as a corpus cannot be confirmed.", "questions_to_authors": "- I think it is good that it is a position paper and that it is mainly subjective, but in the current situation where many LLMs have a multilingual nature, is it appropriate to go in the direction of data construction and validation targeting only English?", "paper_topic_and_main_contributions": "The first part of this paper discusses the future direction of named entity recognition (NER) in response to the emergence of large-scale language models (LLMs). Specifically, the authors list fine-grained supervised NER and zero-shot fine-grained NER, exhaustive typed-Entity retrieval, as tasks to be addressed in the future. In the latter half of the paper, NERetrieve, a dataset consisting of around 4 million paragraphs from English Wikipedia, is constructed as data that can be used to evaluate the tasks mentioned in the first part of the paper, and the results of some simplified experiments using this data are reported.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 169: in Section 7 The next -> in Section 7. The next-\n- Line 506: graph knowledge base,( -> graph knowledge base ("}, "extra_text": {}}, {"review_id": "UhYI0mwJGR", "reviewer": "Reviewer_EbDr", "summary": "", "questions": "- Do you see any link to works on XMLC tasks? E.g. the dataset presented in [2] \n\n[2] https://aclanthology.org/2020.coling-main.513/\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for recognizing our detailed portrayal of real-world NER challenges and our dataset's contribution to advancing NLP research.**\n\nPlease see our response below:\n\n**P1.** *No reference is made to works on (extreme) multi-label classification tasks..*\n\nThank you for pointing out this direction. We see its relevance and will incorporate appropriate references in the camera-ready version.\n\n**P2.** *The selection of baseline/reference models for the newly defined tasks is not very clearly motivated*\n\nWe appreciate your feedback and will provide a clearer motivation for our choice of baseline/reference models in the revised text. Our motivations are also briefly listed here:\n\n**Task 1:** Supervised Fine-grained NER: For the supervised NER task we used DeBERTa-v3 which is a modern common usage transformer used by millions of users (based on HF monthly downloads of model variants). In our prior experience, the DeBERTa-v3 model outperforms other BERT variants, such as RoBERTA and BERT on entity-related tasks. Its superiority to other BERT variants, both in NER and in other NLP tasks, is also documented in [1,2].\n\n\n**Task 2:** Zero Shot Fine-Grained NER with LLM: to the best of our knowledge, the best performing approach for zero-shot NER today is using an instruct-tuned LLM. At the time of writing, and arguably also today, GPT3.5 and GPT4 were among the strongest available instruct-tuned LLMs, thus we believe they represent the strongest available models for this task.  Our experiments show that zero-shot-NER is a challenging task also for LLM, and which is still far from being solved. This ties to the theme of our paper: let's take this research field to the next step. \n\n**Task 3:** Exhaustive Typed-Entity Mention Retrieval - In a zero-shot configuration, we evaluated two state-of-the-art IR vector embedding models, e5-v2 and gtr, in both their base and large variants. The outcomes are detailed in Table 2. We chose these models as they are the best-performing models in the entity-retrieval task for Dbpedia-entity-v2 and were also among the top-performing models in the IR tasks benchmark of MTEB[3].\n\nWe will explicitly specify in the paper the baseline models for each task in the dataset, grounded on the results we have presented.\n\n[1] He, Pengcheng, Jianfeng Gao, and Weizhu Chen. \"Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.\" arXiv preprint arXiv:2111.09543 (2021).\n\n[2] https://ibm.github.io/model-recycling/Rankings.html\n\n[3] https://huggingface.co/spaces/mteb/leaderboard\n\n\n**Clarification questions in the presentation improvement section:**\n\nThank you for pointing out these topics, we will clarify them better in the text. Here are detailed explanations. \n\n**Q1.** *300-301: \"without scanning all the documents\": what does this mean? How can the result be \"exhaustive\" if there is a priori a filter that excludes some documents to be considered?*\n\nThe phrase \"without scanning all the documents\" doesn't imply any filtering of the content. Instead, it contrasts with the zero-shot NER approach, where each individual sample requires an inference run. In IR tasks, the standard procedure is pre-indexing the entire corpus. This allows for a quick match between a query and a paragraph based on a certain metric, such as cosine similarity, using approximate nearest neighbor search, eliminating the need to repeatedly \"scan the documents\" for every single query. We'll ensure clearer wording in our revised text.\n\n**Q2.** *589: why did you pick DeBERTa, and not some other BERT variant? Please add a motivation\nPlease see the answers for P2, about the baseline/reference models.*\n\n**Q3.**  *603: \"s.d. 0.21\": what does s.d. mean? standard deviation? on what quantity is it the standard deviation exactly?*\n\nYes, \"s.d.\" stands for standard deviation. In our experiment to assess robustness in the supervised context, we trained a NER model for each of 20 randomly selected entity types. These models were then applied to 10,000 random dataset paragraphs, excluding those containing the target entity type, to measure the false positive (FP) rate. The average FP rate was found to be 0.25, meaning, on average, 1 in 4 paragraphs had a false positive entity. The mentioned s.d. represents the standard deviation of the FP rate across these 20 models: in some models, the FP rate was higher than 0.25, and in others it was lower.\n\n\n**Q4.** *658 etc.: \"attempting to achieve an approximately uniform distribution of entity types\": please clarify how your approach would indeed amount to a uniform selection over all possible types…*\n\nSince the number of mentions of each entity type is not equal, random sampling will end with a non-balanced distribution of the classes, and each paragraph can include several entity types.\nTherefore we sampled the paragraphs in a stratified manner such that all entity types will be represented with an equal number of items. We will clarify it in the paper.\n\nWe truly appreciate the time and effort you dedicated to improving the quality of the presentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- nice description/definition of real-world use cases of challenging NER tasks\n- contribution to the broad NLP research community with a sizeable dataset supporting effective analysis of new solutions to the 3 defined challenging NER tasks", "reasons_to_reject": "- no reference is made to works on (extreme) multi-label classification tasks, e.g., on skill extraction (just as 1 example, see [1] & the multiple references in its introduction section); that line of work also pertains to an extraction task (i.c. skill extraction) with many possible labels that are organized in a hierarchical taxonomy; this is conceptually very much related to fine-grained NER\n- the selection of baseline/reference models for the newly defined tasks is not very clearly motivated\n\n[1] https://aclanthology.org/2023.acl-long.662/\n", "questions_to_authors": "- Do you see any link to works on XMLC tasks? E.g. the dataset presented in [2] \n\n[2] https://aclanthology.org/2020.coling-main.513/\n", "paper_topic_and_main_contributions": "This is a position paper, carefully assessing the state of current NER solutions, particularly those based on the most recent LLMs. Specifically, the authors define (and establish current models' performance on) three challenging NER tasks: (1) fine-grained supervised NER (with fine-grained, hierarchical and intersectional entity types); (2) zero-shot fine-grained NER (mark all mentions of a stated type, e.g., \"dog breed\", in a given text); (3) exhaustive typed-entity retrieval (e.g., return all documents that mention a dog breed + the span doing so). Moreover, the authors offer a new dataset (and describe its collection & automated annotation process) to support research & development of new models on aforementioned tasks, NERetrieve (4M paragraphs, having entity spans marked with one of the 494 considered types).\n\nThe paper contributions are (1) the definition of challenging NER tasks (including retrieval), (2) a new dataset, (2) performance analysis of recent models using that dataset for the three challenging NER tasks.\nThe authors conclude that for all of the tasks also the most recent large-scale LLM models do not solve them, thus setting out a path for future research.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "n/a"}, "extra_text": {"Missing_References": "- see earlier: works on skill extraction also amount to span extraction & classification, and thus seem worthy to refer to; see [1,2] above, as just 2 examples of such works -- there could be others that are better suited to refer to;\n\n[1] https://aclanthology.org/2023.acl-long.662/\n[2] https://aclanthology.org/2020.coling-main.513/\n", "Typos_Grammar_Style_and_Presentation_Improvements": "- Please position footnote markers (superscript numbers) *after* punctuation. (E.g., at the end of the Abstract)\n- Use curly quotes please, rather than straight ones (e.g,. 072 and following)\n- Always use a comma after \"e.g.\"\n- 067,069,071:  put quotes around references to other sections:  §2 -> (§2)  etc. \n- 119: e.g, -> e.g.,   [missing '.']\n- 150:  Buttefly -> butterfly   [no capital; missing 'r']\n- 162-63: join 2 citations within same pair of brackets?\n- 169: Section 7 -> §7   [be consistent; elsewhere you also simply use '§']\n- 187: coarse-grain -> coarse-grained   [be consistent, please]\n- 268: missing space after 'achievable'\n- 300-301: \"without scanning all the documents\":  what does this mean? How can the result be \"exhaustive\" if there is a priori a filter that excludes some documents to be considered?\n- 372: assume -> assumes   [subject is 'work' on line 370]\n- 589: why did you pick DeBERTa, and not some other BERT variant? Please add a motivation\n- 603: \"s.d. 0.21\":  what does s.d. mean? standard deviation? on what quantity is it the standard deviation exactly?\n- 604-5: and none were indeed -> and indeed, none were\n- 611: suggesting -> suggests\n- 658 etc.: \"attempting to achieve an approximately uniform distribution of entity types\":  please clarify how your approach would indeed amount to a uniform selection over all possible types. Given that you randomly draw paragraphs, wouldn't the distribution of selected types rather follow their (non-uniform?) distribution in terms of number of mentioned instances?\n- 693: NERertiev -> NERetrieve\n- Table 2: \"caption caption caption\"?\n- 722: \"analogous\": vague/imprecise; either it is exactly the same or not?\n\n"}}]

Author rebuttal:

Meta-review: This position paper presents a forward-looking study of LLMs over next-level NER research by introducing three new challenges including fine-grained, zero-shot based on labels, and retrieval for all instances for the query entity NER tasks. Reviewers appreciate that it presents insightful directions for future LLM and NER research. Reviewers share the concern and confusion about the paper contribution: while it is a position paper, it also provides a silver-annotated dataset but no baseline; the quality and usefulness of the silver-annotated dataset are uncertain; two out of three tasks are already explored in previous work. The authors provide effective responses to address these concerns. Overall, we strongly encourage the authors to incorporate the feedback to improve the paper.
Decision: Accept-Findings