ZeroNER: Fueling Zero-Shot Named Entity Recognition via Entity Type Descriptions
Authors: Alessio Cocchieri, Marcos Martínez Galindo, Giacomo Frisoni, Gianluca Moro, Claudio Sartori, Giuseppe Tagliavini
Accepted at: acl25

ZERONER: Fueling Zero-Shot Named Entity Recognition via Entity Type Descriptions

Alessio Cocchieri*   Marcos Martínez Galindo◊*   Giacomo Frisoni♣**
Gianluca Moro♣*   Claudio Sartori♣   Giuseppe Tagliavini♣
♣Department of Computer Science and Engineering, University of Bologna
◊IBM Research Europe, Dublin
♣{a.cocchieri, giacomo.frisoni, gianluca.moro, claudio.sartori, giuseppe.tagliavini}@unibo.it
◊marcos.martinez.galindo@ibm.com

Abstract

What happens when a named entity recognition (NER) system encounters entities it has never seen before? In practical applications, models must generalize to unseen entity types where labeled training data is either unavailable or severely limited—a challenge that demands zero-shot learning capabilities. While large language models (LLMs) offer extensive parametric knowledge, they fall short in cost-effectiveness compared to specialized small encoders. Existing zero-shot methods predominantly adopt a relaxed definition of the term with potential leakage issues and rely on entity type names for generalization, overlooking the value of richer descriptions for disambiguation. In this work, we introduce ZERONER, a description-driven framework that enhances hard zero-shot NER in low-resource settings. By leveraging general-domain annotations and entity type descriptions with LLM supervision, ZERONER enables a BERT-based student model to successfully identify unseen entity types. Evaluated on three real-world benchmarks, ZERONER consistently outperforms LLMs by up to 16% in F1 score, and surpasses lightweight baselines that use type names alone. Our analysis further reveals that LLMs derive significant benefits from incorporating type descriptions in the prompts.¹

1 Introduction

Named entity recognition (NER) involves identifying entity mentions and assigning them to predefined types (classes). Despite the upswing in harnessing the expressive power of pre-trained language models, current solutions mostly demand large-scale datasets and prioritize a handful of commonly occurring types (e.g., person, location, organization) (Wang et al., 2023). Real-world NER applications frequently grapple with specialized domains and lack readily available labels for non-standard types, with new targets constantly emerging. In this context, adopting or periodically retraining state-of-the-art (SOTA) models becomes impractical. Thus, the impetus shifts to zero-shot learning (ZSL) (Xian et al., 2019; Aly et al., 2021; Picco et al., 2023), where networks must generalize to entity types not experienced during training by effectively transferring knowledge gleaned from observed types. In this paradigm, prior knowledge emerges as an essential asset (Hsieh et al., 2023). Simultaneously, creating substantial ground truths remains an expensive and time-consuming activity, prone to human errors and inconsistencies that might affect model effectiveness.

The advent of large language models (LLMs) has revolutionized zero-shot NER by enabling recognition of arbitrary entity types through natural language instructions (Wang et al., 2023; Zhou et al., 2023). This represents a significant departure from

![Overview of ZERONER. The framework derives large-scale, high-quality annotations and entity-type descriptions from a teacher LLM. A student SLM is then fine-tuned on this general-domain data and evaluated on unseen entity types without any type of leakage.](page_1047_384_355_355.png)

Figure 1: Overview of ZERONER. The framework derives large-scale, high-quality annotations and entity-type descriptions from a teacher LLM. A student SLM is then fine-tuned on this general-domain data and evaluated on unseen entity types without any type of leakage.

¹Equal contribution (co-first authorship).
²Data, code, model: zeroner; research group website: unibo-nlp.github.io
the rigidity of traditional models. However, LLMs demand significant computational power. Although some LLMs are accessible through APIs (OpenAI, 2024), large-scale usage can incur prohibitive costs.

To address these challenges, we introduce ZERONER, a novel framework designed to enhance zero-shot NER capabilities in small language models (SLMs). ZERONER operates by distilling both annotations and entity type descriptions from LLMs (Figure 1). Rather than depending on large autoregressive models, our approach employs a more efficient BERT-based cross-encoder that incorporates textual descriptions as contextual information to accurately identify \emph{any} named entity across domains. First, we generate a domain-diverse distillation dataset through a frozen LLM. Then, to ensure data quality, we implement an LLM-based self-correction mechanism combined with heuristic filtering. The SLM is fine-tuned on our general-domain silver dataset, which encapsulates broad NER knowledge, and is then directly evaluated on the test set of the target dataset.

We experiment with the zero-shot-adapted versions of three widely used NER datasets: MedMentions (Mohan and Li, 2019), OntoNotes 5.0 (Pradhan et al., 2013), and LegalNER (Kalamkar et al., 2022). Unlike recent approaches, we adopt a \emph{hard} zero-shot evaluation protocol to ensure realistic performance assessment. Our evaluation adheres to the strict constraints established by Xian et al. (2019): (1) complete separation of entity types across (silver, general-domain) training, (gold, in-domain) development, and (gold, in-domain) test splits, (2) reservation of the rarest labels for development and test sets, and (3) class imbalance accounting via \emph{macro}-averaged per-class metrics. Our SLMs not only outperform all evaluated LLMs with up to 8B parameters but also exceed the performance of alternative SLMs that benefit from data contamination, i.e., training exposure to entity classes used for benchmarking.

Our contributions are as follows:
• We release a lightweight, open-source model for zero-shot NER using class descriptions, achieving SOTA results over costly LLMs.
• We prove that entity type descriptions boost LLM performance in zero-shot NER, underscoring their value for generalization.
• We advocate for a \emph{hard} zero-shot evaluation protocol that eliminates entity type leakage and ensures a more accurate assessment of zero-shot NER capabilities.
• We provide an open-source silver dataset, suitable for both research and commercial use, to promote further advances in zero-shot NER.

2 Related Work

Entity Type Descriptions. A growing body of work explores the use of textual metadata to improve generalization in NER. Obeidat et al. (2019) and Aly et al. (2021) demonstrated that injecting manually crafted definitions of entity types into encoders supports a more accurate classification. Nguyen et al. (2021) further extended this idea by incorporating structured information from knowledge graphs, allowing the model to capture cross-domain relationships between entities. Recently, interest has moved toward adapting LLMs to follow explicit entity type definitions within the prompt. Methods such as GoLLIE and SLIMER (Sainz et al., 2024; Zamai et al., 2024) fine-tune LLMs to align with natural language guidelines, achieving strong zero-shot performance on NER tasks. However, such gains come with considerable computational costs that are not applicable in resource-constrained environments. To the best of our knowledge, OpenBioNER (Cocchieri et al., 2025) is the only prior work that, like ours, investigates the use of entity type descriptions within encoder-based models for open-domain objectives. However, (i) its focus is restricted to the biomedical domain, (ii) it foresees a second-stage in-domain fine-tuning, and (iii) it does not account for type leakage between silver train annotations and evaluation splits.

Knowledge distillation. Knowledge distillation (KD) centers on transferring knowledge from large, high-capacity teacher models to more compact student models (Bucila et al., 2006; Hinton et al., 2015). In this work, we draw upon LLM-based KD in a self-supervised setting (Agrawal et al., 2022; Zhou et al., 2023), where a teacher model is used to annotate unlabeled data. Recent efforts have applied this procedure to a wide array of downstream tasks (Busotti et al., 2024), including NER. Two main directions have emerged in this space: generative and encoder-based student models.

Generative student models employ autoregressive architectures to model NER labels as output texts. InstructUIE (Wang et al., 2023) fine-tunes Flan-T5-xxl (Chung et al., 2022) on a mixture of information extraction datasets, yielding strong zero-shot NER performance. UniversalNER (Zhou et al., 2023) fine-tunes LLaMA using data annotated by
ChatGPT, often matching or surpassing ChatGPT’s own zero-shot NER results. Importantly, it requires extracting one entity type at a time, increasing LLM inference costs for standard multi-class problems. GNER (Ding et al., 2024) incorporates negative instances and adopts a BIO-tagging scheme for finer-grained supervision. However, all generative approaches are inherently limited by slow, token-by-token decoding and high resource demands.

In contrast, encoder-based student models resort to more computationally efficient architectures as backbones for KD. GLiNER (Zaratiana et al., 2023) distills UniversalNER annotations into a DeBERTa (He et al., 2021) student model. It formulates NER as a span–entity type matching problem in embedding space and has demonstrated superior zero-shot results compared to both ChatGPT and fine-tuned LLMs. Bogdanov et al. (2024) pretrain RoBERTa (Liu et al., 2019) on ChatGPT annotations through contrastive-learning, achieving results on par with LLMs in few-shot settings.

Despite these advances, a recurring limitation across current distillation frameworks lies in the use of silver datasets that include entity types overlapping with those in evaluation benchmarks. This results in evaluations conducted under soft zero-shot conditions, where type-level contamination can lead to overly optimistic assessments of generalization capabilities (see §D). As our work is grounded in hard zero-shot NER, direct comparisons with these prior approaches are not fair and therefore fall outside the scope of this paper.

3 Method

3.1 Task Definition

We formulate NER as a sequence labeling task. Given a sentence \( s = \{t_1, \ldots, t_n\} \) comprising \( n \) tokens and a description \( d_c \) for each entity class \( c \in \mathcal{C}^{test} \) in the test set, we predict a sequence of annotations \( \hat{y} \in (\mathcal{C}^{test})^n \). The classification of a token at position \( i \) is determined by \( \arg\max_{c \in \mathcal{C}^{test}} \mathcal{F}_\theta(s, t_i, d_c) \), where \( \mathcal{F} \) models the semantic affinity between \( t_i \) and \( d_c \) in the context of \( s \). Parameters \( \theta \) must be acquired without labeled data for \( \mathcal{C}^{test} \), but with labeled data and descriptions for training classes in \( \mathcal{C}^{train} \). ZERONER considers a wealth of LLM-generated examples as training data: each input-output pair \( <\tilde{s}, \tilde{y}> \) refers to a sentence \( \tilde{s} \) from an external corpus \( \mathcal{R} \) and a target sequence \( \tilde{y} \) linked to a large volume of artificial classes \( \tilde{\mathcal{C}} \) and their descriptions.

3.2 Synthetic BIO-format Dataset

The following paragraphs break down the LLM-to-dataset workflow, depicted in Figure 2. We define \( \mathcal{R} \) by randomly sampling 50k instances\(^2\) from the Pile Uncopyrighted,\(^3\) avoiding any legally encumbered content. Although ZERONER is model-agnostic, we select Llama-3.1-8B-instruct as the LLM teacher due to its strong empirical performance on the benchmarks considered and its permissive license, which allows smaller models to be trained on its outputs without any restriction.

Prompt-based tagging. We approximate expert labelers by prompting the teacher LLM. Specifically, we design a prompt template to jointly perform (1) entity recognition, (2) entity classification, and (3) domain extraction, while also handling cases of nested entities not supported by the BIO-tagging scheme (see §A). This process leads to ~350K annotated sentences. Subsequently, we employ the teacher LLM itself to evaluate its predictions using an LLM-as-a-judge approach, assessing correctness and completeness under predefined criteria (see §A). Each prediction is scored from 0 to 3 on both dimensions. To ensure quality, we retain only annotations with a completeness score of at least 2 and a correctness score of 3, resulting in a final dataset of approximately 91K sentences.

Filtering. To further promote data quality without relying on human supervision, we introduce a series of “critic rules” to regulate the accuracy and confidence of the knowledge being transferred. (1) We exclude sentences accompanied by improperly formatted generations or without any positive annotations—1.44% of the total. (2) We filter out recognized entities not mentioned in the input sentence. (3) We discard entities whose mentions consist solely of functional words, i.e., stopwords, verbs, conjunctions, or prepositions. (4) We use lemmatization to tackle the complexity arising from entity mentions having different labels, such as synonyms or inflected forms. For example, lawyers and defense lawyer can be assigned to the root lawyer. (5) We eliminate rare entity classes that appear only once, ~800 cases.

BIO-format instances. We parse the filtered LLM output and represent named entities in BIO format (short for Beginning, Inside, Outside).

\(^2\) A number of passages known to be sufficient for satisfactory NER distillation (Zhou et al., 2023).
\(^3\) The Pile Uncopyrighted
Figure 2: ZERONER pipeline for transfer dataset construction. First, the teacher LLM annotates raw sentences with entity mentions, types, and domains. Second, silver annotations undergo data cleaning for the BIO-format dataset population. Third, the teacher LLM is prompted to generate multi-view descriptions of all the detected entity types according to the most frequent domains in which they occur.

Since the LLM thrives on free generation, the derived annotations are not accompanied by offset indices communicating their token position in the original sentence. Any predicted entity type overlapped with benchmark labels (\( C^{dev} \) or \( C^{test} \)) is overwritten with the negative class (\( O \)). Given the small number of consecutive entities belonging to the same class (~1.22% on average across all datasets), we remove all I- and B- prefixes, resulting in a single label per entity class.

Data statistics. After filtering, our dataset comprises 91,768 sentences, encompassing 207,245 entities and 5,731 distinct entity types. Table 1 analyzes our transfer BIO-format dataset. The distributions of entity types and domains exhibit a pronounced heavy-tailed pattern, where the top 1% accounts for 70% of the cumulative frequencies. We unveil a broad spectrum of classes that span various disciplines. A noteworthy observation is the existence of granularity variations within particular entity types, e.g., Protein and Gene are subsets of Biological entity. These attributes greatly enhance the dataset’s potential to universally capture LLM capabilities.

<table>
  <tr>
    <th>Frequency</th>
    <th>Main entity types</th>
  </tr>
  <tr>
    <td>Top-1% (70%)</td>
    <td>Person, Organization, Location, Event, Disease, Time</td>
  </tr>
  <tr>
    <td>1%-10% (21%)</td>
    <td>Project, Research, Ingredient, Statistic, Year, Prize</td>
  </tr>
  <tr>
    <td>10%-100% (9%)</td>
    <td>Healthcare, Political, Ownership, Astronomy, Metadata</td>
  </tr>
</table>

Table 1: Entity type distribution overview. Examples of classes grouped by frequency range, along with the proportion of entity occurrences each range represents. Domains include politics, law, finance, medicine, literature, history, music, and math.

the expected class \( y_i \). For instance, given a sentence “The court ruled that the provisions of the Freedom of Information Act were applicable” and a class Law, the description “Law refers to a system of rules created and enforced through social or governmental institutions to regulate behavior” can help the model to make the clue, even without previous exposure to training examples. Plural domains could offer separate definitions for a specific class, granting more opportunities to absorb LLM knowledge. For this reason, we devise a prompting technique oriented to multi-view descriptions (see §A). Formerly collected domain labels are used to count the predominant fields in which each entity type finds mention. For every class \( \tilde{y} \), we request the LLM to generate individual descriptions linked to the three leading domains.

3.3 Entity-type Descriptions

As extra guidance during \( \mathcal{F} \) training, descriptions establish a more explicit connection between \( t_i \) and
Figure 3: Student model architecture. The target entity type description is injected into the input sentence token representations through cross-attention, aiding the model in classifying their semantic relatedness.

Ultimately, the outputs are concatenated following their ranking sequence. We note that the average number of tokens per multi-view description is 47.

3.4 Student Network

Our SLM is based on pre-trained BERT-base (Devlin et al., 2019). ZERONER’s \( \mathcal{F} \) modeling parallels the cross-attention encoder (X-ENC) presented by Aly et al. (2021), as illustrated in Figure 3. For notational convenience, we use \( \mathbb{C} \) to refer to target classes, regardless of whether they are synthetically generated (training) or come from a gold dataset (evaluation). With every description \( \mathbf{d}_c \), X-ENC produces a vector representation \( \mathbf{v}_{i,c} \in \mathbb{R}^h \) for each token \( t_i \) in sentence s:

\[
\mathbf{v}_{1,c}, \ldots, \mathbf{v}_{n,c} = \text{X-ENC}(s, \mathbf{d}_c),
\]

where \( h = 768 \) and the input tuple \( (s, \mathbf{d}_c) \) is structured in the form [CLS] s [SEP] \( \mathbf{d}_c \). The vector \( \mathbf{v}_{i,c} \) is transformed through a learnable linear layer to quantify \( l_{i,c} \in \mathbb{R} \). The value \( l_{i,c} \) indicates how likely the token \( t_i \) belongs to the entity class \( c \):

\[
l_{i,c} = \mathbf{v}_{i,c} \cdot \omega^T + b.
\]

To identify entities other than classifying them, we append the token scores \( l_{i,c_1}; \ldots; l_{i,c_{|\mathbb{C}|}} \) with the score of belonging to the negative class \( c_{neg} \):

\[
l_i = (l_{i,c_1}; \ldots; l_{i,c_{|\mathbb{C}|}}; l_{i,c_{neg}}).
\]

To derive \( c_{neg} \), we use the same class-aware encoding as in (Aly et al., 2021), which involves combining the cross-encoder representations of positive classes and then applying max-pooling. After undergoing Softmax, the top-scoring class is chosen.

<table>
  <tr>
    <th></th>
    <th colspan="2">OntoNotes-ZS</th>
    <th colspan="2">MedMentions-ZS</th>
    <th colspan="2">LegalNER-ZS</th>
  </tr>
  <tr>
    <th></th>
    <th>dev</th>
    <th>test</th>
    <th>dev</th>
    <th>test</th>
    <th>dev</th>
    <th>test</th>
  </tr>
  <tr>
    <td># sentences</td>
    <td>1,358</td>
    <td>426</td>
    <td>1,289</td>
    <td>1,048</td>
    <td>602</td>
    <td>1,227</td>
  </tr>
  <tr>
    <td># words</td>
    <td>39,349</td>
    <td>12,624</td>
    <td>37,297</td>
    <td>29,783</td>
    <td>20,623</td>
    <td>42,808</td>
  </tr>
  <tr>
    <td># entities</td>
    <td>1,735</td>
    <td>533</td>
    <td>1,710</td>
    <td>1,430</td>
    <td>808</td>
    <td>1,777</td>
  </tr>
  <tr>
    <td># compound entities</td>
    <td>219</td>
    <td>229</td>
    <td>292</td>
    <td>167</td>
    <td>387</td>
    <td>841</td>
  </tr>
  <tr>
    <td># consecutive same class</td>
    <td>11</td>
    <td>0</td>
    <td>5</td>
    <td>10</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td># types</td>
    <td>4</td>
    <td>3</td>
    <td>5</td>
    <td>5</td>
    <td>3</td>
    <td>4</td>
  </tr>
  <tr>
    <td>avg sentence length</td>
    <td>28.98</td>
    <td>29.63</td>
    <td>28.93</td>
    <td>28.42</td>
    <td>34.26</td>
    <td>34.89</td>
  </tr>
  <tr>
    <td>avg entities per sentence</td>
    <td>1.28</td>
    <td>1.25</td>
    <td>1.33</td>
    <td>1.36</td>
    <td>1.34</td>
    <td>1.45</td>
  </tr>
</table>

Table 2: Summary statistics of zero-shot NER benchmarks. For each dataset and split, we report the number of sentences, words (whitespace-separated tokens), entities (including compound and consecutive same-class entities), entity types, as well as average sentence length and entities per sentence.

3.5 Training

In ZERONER, the student SLM is trained to mimic the type-unbounded silver entity annotations of the teacher LLM, additionally exploiting its type descriptions. We manage the class imbalance induced by \( c_{neg} \) by incorporating class weights \( q_c \) into the cross-entropy loss:

\[
\mathcal{L} = - \sum_c^{|\mathbb{C}|} q_c \cdot y_{i,c} \cdot \log(p(\hat{y}_{i,c})),
\]

where \( y_{i,c} \) is the truth label and \( p(y_{i,c}) \) is the Softmax probability for class \( c \). We consistently set \( q \) to 1 for positive classes, while fine-tuning it as a hyperparameter for negative class, using the ratio \( \frac{\# \text{entities}}{\# \text{non-entity words}} \) from the training data as a reference.

To effectively manage the variety of entity types, we adopt a progressive training approach, gradually introducing the model to different classes, i.e., incremental learning. At each step, the model processes a random subset of 20 to 25 entity types (Figure 4). Any target entity types not among the sampled set are labeled as 'O'. In our methodology, the model is trained solely on entity type descriptions (e.g., “Human being able to speak and think”) rather than explicit class names such as Person. Even if a class appears in both the distilled and test datasets, variations in descriptions—often reflecting domain-specific nuances—lead the model to treat them as distinct types. To ensure that successive evaluations reflect truly hard generalization, at each training step, we also exclude from the sampled types: (1) those with labels that exactly match or contain any benchmark label (e.g., Art vs. Work
of Art), and (2) those whose descriptions exhibit high cosine similarity (above 0.95) to benchmark descriptions, thus mitigating overlap due to synonyms (e.g., Location vs. Place).4 To sum up, this filtering prevents the model from encountering nearly identical training and test data.

4 Experimental Settings

Datasets. To gauge ZeroNER holistically, we conduct experiments on three heterogeneous English-language NER datasets: MedMentions-ST21pv (Mohan and Li, 2019), OntoNotes 5.0 (Pradhan et al., 2013), and LegalNER (Kalamkar et al., 2022). MedMentions (CC0 license) holds 4K biomedical abstracts and a fine-grained class ontology rooted in the Unified Medical Language System (UMLS). OntoNotes (LDC license) is a large-scale multigenre corpus targeting domain-general entity types, including values. LegalNER (MIT license) annotates Indian court judgments published between 1950 and 2022.

Building on the recommendations of Xian et al. (2019), we create zero-shot adapted versions of these benchmarks (identified by the “-ZS” suffix). In pursuit of this, we adhere to the conversion protocol suggested by Aly et al. (2021), alternatively leaving the rarest classes in the dev and test sets. The chosen types are guaranteed to be not trivial to recognize; we omit classes whose surface form displays regular patterns (e.g., Percent, Date). After filtering, the total number of entity types within MedMentions, OntoNotes, and LegalNer is 21, 11, and 13, respectively. Annotations on split-disconnected classes are eliminated. An overview of the data distribution is shown in Table 2. Regarding class description sources for evaluation datasets, we draw on the UMLS Metathesaurus for MedMentions and annotation guidelines for OntoNotes and LegalNER. See §B, §C, and §I for details.

Baselines. We compare ZeroNER with SLMs and LLMs on pure zero-shot.
• SMXM (Aly et al., 2021). BERT-large model with X-ENC for class description integration, similar to ZeroNER. Unlike our approach, this model is fine-tuned on the train split of the target dataset—where train, dev, and test entity types are disjoint—rather than a synthetic dataset. All reported results are recomputed and not taken from the original paper.5
• Llama-3.1-8B, Llama-3.2-3B (Grattafiori et al., 2024), Phi-3.5-mini (Abdin et al., 2024), Granite-3.0-8B (Granite Team, IBM, 2024), Mistral-7B-v0.3 (Jiang et al., 2023), and Qwen2.5-7B (Yang et al., 2024): SOTA instruct-tuned, open-source LLMs with autoregressive decoder-only architectures. We construct two types of prompts to detect entities—first by leveraging only the type names, and then by incorporating type descriptions. For details, see §A.
• GLiNER Models (Zaratiana et al., 2023): Current leading models in the zero-shot NER setting. These models employ cross-encoders with DeBERTa as backbone architectures, modeling the NER task as matching entity types with textual spans in a latent space. We evaluate both the v1 and v2.1 families across their small, medium, and large variants. The v1 family is restricted to research use as it was trained on ChatGPT annotations, whereas the v2.1 family is licensed under Apache 2.0, having been trained on Mistral annotations. Although we include these models for comparison, it is important to note that many labels were likely encountered during their training, artificially boosting their performance.

Metrics. We evaluate models with macro-averaged precision, recall, and \( F_1 \) metrics at the span level. Per-class averaging prevents frequent entity types from dominating performance.

Implementation details. All experiments were conducted on an Nvidia A100 GPU with 80GB of

4Similarity measured with sentence-transformers/all-MiniLM-L6-v2

5We identify three reasons: (1) discrepancies in entity occurrences after reconstructing the non-public SMXM-ZS datasets; (2) significantly lower performance of the available model checkpoints compared to reported metric; (3) the original training likely used different descriptions from ours, though this cannot be confirmed.

Figure 4: Iterative class sampling and synthetic instance selection during training. In each step, a random assortment of entity types is selected, and sentences of the dataset that do not contain any of these targets are filtered out for that step.
<table>
  <tr>
    <th rowspan="2">Type name</th>
    <th rowspan="2">Size</th>
    <th colspan="3">OntoNotes-ZS</th>
    <th colspan="3">MedMentions-ZS</th>
    <th colspan="3">LegalNER-ZS</th>
    <th rowspan="2">AVG</th>
  </tr>
  <tr>
    <th>R</th><th>P</th><th>F1</th>
    <th>R</th><th>P</th><th>F1</th>
    <th>R</th><th>P</th><th>F1</th>
  </tr>
  <tr>
    <td>Granite-3.0-8b-instruct</td><td>8B</td><td>0.30</td><td>0.14</td><td>0.19</td><td>0.38</td><td>0.14</td><td>0.20</td><td>0.45</td><td>0.24</td><td>0.31</td><td>0.23</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td><td>8B</td><td>0.43</td><td>0.21</td><td>0.27</td><td>0.41</td><td>0.32</td><td>0.32</td><td>0.44</td><td>0.19</td><td>0.26</td><td>0.28</td>
  </tr>
  <tr>
    <td>Llama-3.2-3B-Instruct</td><td>3B</td><td>0.34</td><td>0.10</td><td>0.15</td><td>0.41</td><td>0.26</td><td>0.27</td><td>0.44</td><td>0.19</td><td>0.26</td><td>0.23</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.3</td><td>7B</td><td>0.30</td><td>0.19</td><td>0.23</td><td>0.25</td><td>0.21</td><td>0.22</td><td>0.30</td><td>0.19</td><td>0.23</td><td>0.23</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini-instruct</td><td>3.8B</td><td>0.43</td><td>0.15</td><td>0.22</td><td>0.40</td><td>0.20</td><td>0.22</td><td>0.41</td><td>0.19</td><td>0.25</td><td>0.23</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td><td>7B</td><td>0.35</td><td>0.23</td><td>0.27</td><td>0.15</td><td>0.34</td><td>0.18</td><td>0.44</td><td>0.27</td><td>0.32</td><td>0.26</td>
  </tr>
  <tr>
    <td>GLiNER-S-v1*</td><td>166M</td><td>0.41</td><td>0.30</td><td>0.32</td><td>0.51</td><td>0.38</td><td>0.43</td><td>0.38</td><td>0.25</td><td>0.28</td><td>0.34</td>
  </tr>
  <tr>
    <td>GLiNER-M-v1*</td><td>209M</td><td>0.39</td><td>0.34</td><td>0.32</td><td>0.53</td><td>0.44</td><td>0.48</td><td>0.47</td><td>0.27</td><td>0.33</td><td>0.38</td>
  </tr>
  <tr>
    <td>GLiNER-L-v1*</td><td>459M</td><td>0.42</td><td>0.34</td><td>0.35</td><td>0.54</td><td>0.46</td><td>0.49</td><td>0.49</td><td>0.31</td><td>0.34</td><td>0.39</td>
  </tr>
  <tr>
    <td>GLiNER-S-v2.1*</td><td>166M</td><td>0.24</td><td>0.31</td><td>0.21</td><td>0.38</td><td>0.41</td><td>0.39</td><td>0.33</td><td>0.31</td><td>0.24</td><td>0.28</td>
  </tr>
  <tr>
    <td>GLiNER-M-v2.1*</td><td>209M</td><td>0.28</td><td>0.36</td><td>0.28</td><td>0.42</td><td>0.43</td><td>0.42</td><td>0.37</td><td>0.25</td><td>0.28</td><td>0.33</td>
  </tr>
  <tr>
    <td>GLiNER-L-v2.1*</td><td>459M</td><td>0.30</td><td>0.42</td><td>0.30</td><td>0.44</td><td>0.42</td><td>0.42</td><td>0.32</td><td>0.23</td><td>0.24</td><td>0.32</td>
  </tr>
  <tr>
    <td>Granite-3.0-8b-instruct</td><td>8B</td><td>0.34</td><td>0.22</td><td>0.26</td><td>0.22</td><td>0.17</td><td>0.18</td><td>0.35</td><td>0.29</td><td>0.32</td><td>0.25</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td><td>8B</td><td>0.48</td><td>0.33</td><td>0.38</td><td>0.40</td><td>0.40</td><td>0.34</td><td>0.27</td><td>0.20</td><td>0.23</td><td>0.32</td>
  </tr>
  <tr>
    <td>Llama-3.2-3B-Instruct</td><td>3B</td><td>0.55</td><td>0.24</td><td>0.33</td><td>0.47</td><td>0.24</td><td>0.28</td><td>0.42</td><td>0.25</td><td>0.31</td><td>0.31</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.3</td><td>7B</td><td>0.32</td><td>0.32</td><td>0.32</td><td>0.19</td><td>0.20</td><td>0.17</td><td>0.32</td><td>0.32</td><td>0.32</td><td>0.27</td>
  </tr>
  <tr>
    <td>Phi-3.5-mini-instruct</td><td>3.8B</td><td>0.49</td><td>0.22</td><td>0.30</td><td>0.44</td><td>0.22</td><td>0.24</td><td>0.47</td><td>0.20</td><td>0.27</td><td>0.27</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td><td>7B</td><td>0.49</td><td>0.36</td><td>0.41</td><td>0.30</td><td>0.31</td><td>0.28</td><td>0.38</td><td>0.40</td><td>0.37</td><td>0.35</td>
  </tr>
  <tr>
    <td>SMXM</td><td>345M</td><td>0.31</td><td>0.14</td><td>0.19</td><td>0.44</td><td>0.18</td><td>0.23</td><td>0.22</td><td>0.18</td><td>0.15</td><td>0.19</td>
  </tr>
  <tr>
    <td><b>ZeroNER (Ours)</b></td><td>110M</td><td><b>0.61</b></td><td><b>0.44</b></td><td><b>0.50</b></td><td><b>0.32</b></td><td><b>0.42</b></td><td><b>0.35</b></td><td><b>0.43</b></td><td><b>0.26</b></td><td><b>0.32</b></td><td><b>0.39</b></td>
  </tr>
</table>

* Lower-case entity types (threshold=0.5); see §E for title-case results. GLiNER scores are provided as a reference, as exposed to test entity type names during training.

Table 3: Test set zero-shot results. Performance of ZeroNER, LLMs, and GLiNER models across three benchmarks: OntoNotes-ZS, MedMentions-ZS, and LegalNER-ZS. Results are presented for two settings: using type names and using type descriptions. AVG represents the macro-averaged F1 score across all benchmarks. For each category, the best-performing model is highlighted in bold, and the second-best is underlined.

VRAM. The model was trained on the distillation dataset for 3 epochs. Each epoch took approximately 18 hours to complete, amounting to a total training time of 54 hours. At each epoch, we saved the best checkpoint according to the dev set of our benchmarks. We used a batch size of 8 and a constant learning rate of 2e-5, the latter selected following manual hyperparameter tuning. The entity masking probability was set to 0.3, following Aly et al. (2021). We leave 42 as the default training seed. For LLM inference, we leveraged the VLLM library6 (version 0.6.3.post1). More details about both ZeroNER training and LLM inference are discussed in §F.

5 Benchmark Contamination

To assess potential data contamination between training corpora of the considered GLiNER baselines and the benchmarks contemplated in this study, we implement a multi-tiered detection approach with increasing levels of semantic overlap tolerance. We first identify exact matches where entity type labels appear identically in both datasets. Fuzzy matches leverage Python’s difflib.SequenceMatcher to detect semantically similar labels (e.g., organization vs. organisation) using a similarity threshold of 0.9, capturing morphological variations and minor spelling differences. Word boundary matches identify cases where entity types share common semantic components (e.g., immigration judge overlapping with judge), indicating conceptual overlap. Finally, substring matches detect direct containment relationships between labels, revealing hierarchical contamination patterns. Each contamination type receives a severity weight (exact: 1.0, fuzzy: 0.8, word boundary: 0.6, substring: 0.4) to compute an overall contamination severity score, ensuring entities are categorized only once to prevent double-counting. Table 4 lists contamination metrics for both pile-mistral-v0.1 and Pile-NER-type, the training datasets of GLiNER-v2.1 and GLiNER-v1 models, respectively. Under pile-mistral-v0.1, all three benchmarks exhibit 100% entity-type contamination and high severity scores (LegalNER: 90–100%; OntoNotes: 90–93.3%; MedMentions: 76–92%), confirming that pile-mistral’s raw corpus already contains most benchmark labels. Instead, Pile-NER-type, although still showing full

6 github.com/vllm-project/vllm
<table>
  <tr>
    <th>Benchmark</th>
    <th>Split</th>
    <th># Types</th>
    <th># Contaminated</th>
    <th>Cont. Rate</th>
    <th>Severity</th>
    <th>Exact</th>
    <th>Fuzzy / Word / Sub</th>
  </tr>
  <tr>
    <th colspan="8">pile-mistral-v0.1</th>
  </tr>
  <tr>
    <td>OntoNotes-ZS</td>
    <td>Dev</td>
    <td>4</td>
    <td>4</td>
    <td>100.00%</td>
    <td>90.00%</td>
    <td>3.0</td>
    <td>0.0 / 0.6 / 0.0</td>
  </tr>
  <tr>
    <td>OntoNotes-ZS</td>
    <td>Test</td>
    <td>3</td>
    <td>3</td>
    <td>100.00%</td>
    <td>93.33%</td>
    <td>2.0</td>
    <td>0.8 / 0.0 / 0.0</td>
  </tr>
  <tr>
    <td>MedMentions-ZS</td>
    <td>Dev</td>
    <td>5</td>
    <td>5</td>
    <td>100.00%</td>
    <td>76.00%</td>
    <td>2.0</td>
    <td>0.0 / 1.8 / 0.0</td>
  </tr>
  <tr>
    <td>MedMentions-ZS</td>
    <td>Test</td>
    <td>5</td>
    <td>5</td>
    <td>100.00%</td>
    <td>92.00%</td>
    <td>4.0</td>
    <td>0.0 / 0.6 / 0.0</td>
  </tr>
  <tr>
    <td>LegalNER-ZS</td>
    <td>Dev</td>
    <td>3</td>
    <td>3</td>
    <td>100.00%</td>
    <td>100.00%</td>
    <td>3.0</td>
    <td>0.0 / 0.0 / 0.0</td>
  </tr>
  <tr>
    <td>LegalNER-ZS</td>
    <td>Test</td>
    <td>4</td>
    <td>4</td>
    <td>100.00%</td>
    <td>90.00%</td>
    <td>3.0</td>
    <td>0.0 / 0.6 / 0.0</td>
  </tr>
  <tr>
    <th colspan="8">Pile-NER-type</th>
  </tr>
  <tr>
    <td>OntoNotes-ZS</td>
    <td>Dev</td>
    <td>4</td>
    <td>4</td>
    <td>100.00%</td>
    <td>90.00%</td>
    <td>3.0</td>
    <td>0.0 / 0.6 / 0.0</td>
  </tr>
  <tr>
    <td>OntoNotes-ZS</td>
    <td>Test</td>
    <td>3</td>
    <td>3</td>
    <td>100.00%</td>
    <td>93.33%</td>
    <td>2.0</td>
    <td>0.8 / 0.0 / 0.0</td>
  </tr>
  <tr>
    <td>MedMentions-ZS</td>
    <td>Dev</td>
    <td>5</td>
    <td>5</td>
    <td>100.00%</td>
    <td>64.00%</td>
    <td>1.0</td>
    <td>0.0 / 1.8 / 0.4</td>
  </tr>
  <tr>
    <td>MedMentions-ZS</td>
    <td>Test</td>
    <td>5</td>
    <td>5</td>
    <td>100.00%</td>
    <td>80.00%</td>
    <td>2.0</td>
    <td>0.8 / 1.2 / 0.0</td>
  </tr>
  <tr>
    <td>LegalNER-ZS</td>
    <td>Dev</td>
    <td>3</td>
    <td>3</td>
    <td>100.00%</td>
    <td>93.33%</td>
    <td>2.0</td>
    <td>0.8 / 0.0 / 0.0</td>
  </tr>
  <tr>
    <td>LegalNER-ZS</td>
    <td>Test</td>
    <td>4</td>
    <td>2</td>
    <td>50.00%</td>
    <td>40.00%</td>
    <td>1.0</td>
    <td>0.0 / 0.6 / 0.0</td>
  </tr>
</table>

Table 4: GLiNER contamination metrics. Overlap analysis between the entity types in LegalNER-ZS, OntoNotes-ZS, and MedMentions-ZS (dev/test) and the GLiNER training corpora: pile-mistral-v0.1 and Pile-NER-type.

type overlap, demonstrates lower severity on MedMentions (64–80%) and markedly reduced contamination on LegalNER’s test split (only 50% of types, severity 40%).

6 Results and Discussion

Our core results are detailed in Table 3. In this section, we present the evaluation of ZERONER against various baseline models and discuss key insights. We refer the reader to §H for data cleaning ablations and §G for efficiency considerations.

ZERONER versus SMXM. Compared to SMXM (Aly et al., 2021), our method achieves substantially higher results, although it relies on a smaller BERT backbone and does not utilize in-domain supervised training. ZERONER registers a 100% increase in average performance. This gap highlights the advantage of using distilled annotations and entity descriptions, even when the training data distribution differs from the test benchmarks, as well as the strength of silver data in the absence of direct supervision.

ZERONER versus LLMs. Despite their extensive parameter space and recent advancements, LLMs continue to struggle in zero-shot NER. ZERONER consistently surpasses all LLMs of varying sizes, with performance gains ranging from 4% to 16% relative to the strongest baseline, Qwen2.5.

ZERONER versus GLiNER. Notably, ZERONER outperforms 5 out of 6 evaluated GLiNER models. It reliably achieves F1 score improvements of up to 10% across all model sizes of the v2.1 version, and also beats the base and medium configurations of the v1 version. Figure 5 provides a per-type comparison between ZERONER and the three contaminated variants of GLiNER-v2.1 on both dev and test splits. ZERONER exhibits stronger performance on broad, context-sensitive categories such as FAC, LOC, GPE LAW, and NORP, where natural language descriptions facilitate more accurate disambiguation of in-domain usage. Conversely, GLiNER yields higher scores on heavily contaminated labels—most notably ORG. This advantage is attributed to the training on pile-mistral-v0.1 corpus, which includes ~20K annotations labeled as Organization or close variants (e.g., political organization, religious organization), establishing pattern-label associations. We further observe that ZERONER’s performance on certain categories—such as Judge and Statute—can deteriorate when descriptions become overly detailed. For example, the most effective description for JUDGE is a concise formulation like “The name of the judge presiding over the current case.”

Do LLMs need descriptions? LLMs, even with their extensive pre-trained knowledge, face challenges in generalizing when relying solely on entity type names. The LLaMA family is the only evaluated model group that has limited benefit from incorporating entity descriptions. Other LLMs display clear F1 growth when such descriptions are provided, with Qwen 2.5 securing an overall increase of 8%. As noted by Aly et al. (2021), OntoNotes guidelines are highly specific and representative of the targets. This is evident in the
Figure 5: Impact of entity type contamination. Per-type F1 scores comparing ZeroNER and the contaminated variants of GLiNER-v2.1 on both development and test splits.

OntoNotes-ZS results, where all LLMs—including LLaMA—demonstrate improved performance, as plotted in Figure 6. These observations suggest that some models require a certain level of description quality to benefit from additional context, whereas models like Qwen 2.5 appear more capable of leveraging such information for zero-shot adaptation.

7 Conclusions

We present ZeroNER, a knowledge distillation framework specifically designed for hard zero-shot NER. After rigorous evaluation on heterogeneous datasets, ZeroNER proves capable of recognizing unseen entities using only a textual description of them, eliminating the need for re-training. Regardless of model scale, ZeroNER delivers stronger results than all considered LLMs, combining accuracy with computational efficiency. In comparison to GLiNER, which relies on embedding-level alignment between input sentence and class name tokens, ZeroNER retains a pronounced advantage, even in the absence of target contamination. Moreover, our findings indicate that LLMs display meaningful generalization potential when supplied with well-crafted descriptions, opening the door to future work on test-time scaling.

Figure 6: Impact of entity type descriptions on macro-F1 scores in LLMs. Comparison of LLMs on the OntoNotes-ZS test set with and without the incorporation of descriptions for the target classes in the prompt.

Limitations and Future Work

ZeroNER, though effective, comes with certain inherent limitations that warrant discussion. Compared to GLiNER, it currently lacks support for nested entities. The sensitivity of the ZeroNER to the quality of descriptions presents another con-
sideration. Poorly constructed definitions can lead to degraded results, and adapting ZERONER to specific benchmarks may require manual refinement. In addition, model performance is likely influenced by the choice of encoder. We selected BERT for fair comparison with the SMXM baseline, but stronger architectures such as RoBERTa, DeBERTa, or ModernBERT may offer further improvements—a direction we leave open for future exploration, along with multi-linguality. Looking ahead, future work may also explore techniques inspired by soft prompting (Ramnath et al., 2025) to automatically learn entity descriptions that maximize the likelihood of successful recognition by LLMs. More generally, automating the generation of high-quality type descriptions represents a promising direction for future research, as illustrated by recent work such as Picco et al. (2024). Additionally, our insights suggest designing hybrid training strategies that alternate between using the type name and the type description. Such an approach could promote more adaptive behavior, favoring minimal definitions for well-represented categories and switching to richer descriptions for ambiguous or underrepresented types. From an interpretability standpoint, it would also be valuable to investigate which tokens within a description contribute most to performance. Techniques such as input perturbation or attribution methods—already applied in other fields (Domeniconi et al., 2014a)—could be adapted to analyze token-level influence. Beyond NER, ZERONER may be extended to other low-resource tasks. Three promising directions involve applying it to cross-domain document classification (Domeniconi et al., 2014b, 2016, 2017; Moro et al., 2018), node classification and graph clustering (Lodi et al., 2010), and semantic parsing (Frisoni et al., 2021, 2022).

Acknowledgments

Research partially supported by AI-PACT project (CUP B47H22004450008, B47H22004460001); National Plan PNC-I.1 DARE initiative (PNC0000002, CUP B53C22006450001); PNRR Extended Partnership FAIR (PE00000013, Spoke 8); 2024 Scientific Research and High Technology Program, project “AI analysis for risk assessment of empty lymph nodes in endometrial cancer surgery,” the Fondazione Cassa di Risparmio in Bologna; Chips JU TRISTAN project (G.A. 101095947).

References

Marah Abdin, Jyoti Aneja, Hany Awadalla, and Ahmed Awadallah. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.

Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, et al. 2022. Large language models are few-shot clinical information extractors. In EMNLP, pages 1998–2022. ACL.

Rami Aly, Andreas Vlachos, and Ryan McDonald. 2021. Leveraging type descriptions for zero-shot named entity recognition and classification. In ACL/JCNLP, pages 1516–1528, Online. ACL.

Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoît Crabbé, et al. 2024. Nuner: Entity recognition encoder pre-training via llm-annotated data. CoRR, abs/2402.15343.

Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In KDD, pages 535–541. ACM.

Jean-Flavien Bussotti, Luca Ragazzi, Giacomo Frisoni, Gianluca Moro, and Paolo Papotti. 2024. Unknown claims: Generation of fact-checking training examples from unstructured and structured data. In EMNLP, pages 12105–12122. ACL.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, et al. 2022. Scaling instruction-finetuned language models. CoRR, abs/2210.11416.

Alessio Cocchieri, Giacomo Frisoni, Marcos Martínez Galindo, Gianluca Moro, Giuseppe Tagliavini, and Francesco Candoli. 2025. Openbioner: Lightweight open-domain biomedical named entity recognition through entity type description. In NAACL, pages 818–837. ACL.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186. ACL.

Yuyang Ding, Juntao Li, Pinzheng Wang, Zecheng Tang, et al. 2024. Rethinking negative instances for generative named entity recognition. In ACL, pages 3461–3475. ACL.

Giacomo Domeniconi, Marco Masseroli, Gianluca Moro, and Pietro Pinoli. 2014a. Discovering new gene functionalities from random perturbations of known gene ontological annotations. In KDIR, pages 107–116. SciTePress.

Giacomo Domeniconi, Gianluca Moro, Andrea Pagliarani, and Roberto Pasolini. 2017. On deep learning in cross-domain sentiment classification. In IC3K, pages 50–60. SciTePress.

Giacomo Domeniconi, Gianluca Moro, Roberto Pasolini, and Claudio Sartori. 2014b. Cross-domain text classification through iterative refining of target
categories representations. In KDIR, pages 31–42. SciTePress.

Giacomo Domeniconi, Konstantinos Semertzidis, Vanessa López, Elizabeth M. Daly, Spyros Kotoulas, and Gianluca Moro. 2016. A novel method for unsupervised and supervised conversational message thread detection. In DATA, pages 43–54. SciTePress.

Giacomo Frisoni, Gianluca Moro, and Lorenzo Balzani. 2022. Text-to-text extraction and verbalization of biomedical event graphs. In COLING, pages 2692–2710. ICCL.

Giacomo Frisoni, Gianluca Moro, and Antonella Carbonaro. 2021. A survey on event extraction for natural language understanding: Riding the biomedical literature wave. IEEE Access, 9:160721–160757.

Abbas Ghaddar and Philippe Langlais. 2018. Robust lexical features for improved neural network named-entity recognition. In COLING, pages 1896–1907. ACL.

Granite Team, IBM. 2024. Granite 3.0 language models.

Aaron Grattafiori, Abhimanyu Dubey, and Abhinav Jauhri. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. Deberta: decoding-enhanced bert with disentangled attention. In ICLR. OpenReview.net.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.

Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, et al. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In ACL (Findings), pages 8003–8017. ACL.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al. 2023. Mistral 7b. CoRR, abs/2310.06825.

Prathamesh Kalamkar, Astha Agarwal, Aman Tiwari, Smita Gupta, et al. 2022. Named entity recognition in Indian court judgments. In NLLP, pages 184–193. Abu Dhabi, United Arab Emirates (Hybrid). ACL.

Peng-Hsuan Li, Ruo-Ping Dong, Yu-Siang Wang, Ju-Chieh Chou, et al. 2017. Leveraging linguistic structures for named entity recognition with bidirectional recursive neural networks. In EMNLP, pages 2664–2669. Copenhagen, Denmark. ACL.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, et al. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.

Stefano Lodi, Gianluca Moro, and Claudio Sartori. 2010. Distributed data clustering in multi-dimensional peer-to-peer networks. In ADC, volume 104 of CRPIT, pages 171–178. Australian Computer Society.

Sunil Mohan and Donghui Li. 2019. Medmentions: A large biomedical corpus annotated with UMLS concepts. In AKBC.

Gianluca Moro, Andrea Pagliarani, Roberto Pasolini, and Claudio Sartori. 2018. Cross-domain & in-domain sentiment analysis with memory-based deep neural networks. In IC3K, pages 125–136. SciTePress.

Hoang Van Nguyen, Francesco Gelli, and Soujanya Poria. 2021. DOZEN: cross-domain zero shot named entity recognition with knowledge graph. In SIGIR, pages 1642–1646. ACM.

Rasha Obeidat, Xiaoli Fern, Hamed Shahbazi, and Prasad Tadepalli. 2019. Description-based zero-shot fine-grained entity typing. In NAACL, pages 807–814, Minneapolis, Minnesota. ACL.

OpenAI. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Gabriele Picco, Leopold Fuchs, Marcos Martínez Galindo, Alberto Purpura, et al. 2024. Description boosting for zero-shot entity and relation classification. In ACL, pages 9441–9457. ACL.

Gabriele Picco, Marcos Martínez Galindo, Alberto Purpura, Leopold Fuchs, et al. 2023. Zshot: An open-source framework for zero-shot named entity recognition and relation extraction. In ACL, pages 357–368. ACL.

Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, et al. 2013. Towards robust linguistic analysis using ontologies. In CoNLL, pages 143–152. ACL.

Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, et al. 2025. A systematic survey of automatic prompt optimization techniques. CoRR, abs/2502.16923.

Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, et al. 2024. Golli: Annotation guidelines improve zero-shot information-extraction. In ICLR. OpenReview.net.

Xiao Wang, Weikang Zhou, Can Zu, Han Xia, et al. 2023. Instructue: Multi-task instruction tuning for unified information extraction. CoRR, abs/2304.08085.

Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. 2019. Zero-shot learning - A comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell., 41(9):2251–2265.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, et al. 2024. Qwen2.5 technical report. CoRR, abs/2412.15115.

Andrew Zamai, Andrea Zugarini, Leonardo Rigutini, Marco Ernandes, et al. 2024. Show less, instruct more: Enriching prompts with definitions and guidelines for zero-shot NER. CoRR, abs/2407.01272.

Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charmois. 2023. Glner: Generalist model for named entity recognition using bidirectional transformer. CoRR, abs/2311.08526.

Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, et al. 2023. Universalner: Targeted distillation from large language models for open named entity recognition. CoRR, abs/2308.03279.

A Prompts

Figure 7 outlines the prompt used to generate annotations over the Pile. Figure 8 shows the prompt used to employ self-correction over annotations generated by the teacher LLM. Figure 9 shows the prompt used to generate multi-view descriptions. Figure 10 shows the prompt used to make inference with LLMs on OntoNotes-ZS. Table 5 documents some examples of multi-view descriptions generated by the teacher LLM.

B Benchmark Dataset Details

• OntoNotes 5.0. Multi-genre dataset designed for domain-general entity types. In particular, OntoNotes offers annotations in three genres: newswire, broadcast conversation, and web data. Various iterations of the OntoNotes dataset have found application in academic papers. Our version of OntoNotes aligns with that employed in (Li et al., 2017; Ghaddar and Langlais, 2018). Removed irrelevant entity classes: Money, Ordinal, Percent.
• MedMentions. The dataset contains not just named entities but also high-level concepts. For example, in the passage “modeling nurse-patients,” the term “modeling” is annotated with the concept Research Activity, thus adding an extra layer of complexity to the NER task. Original GitHub repository.7 Removed irrelevant entity classes: / (i.e., no trivial label identified).
• LegalNER. We considered the dataset officially released by the authors,8 which comprises only the train and dev sets for a total of 12,069 labeled sentences. Entities are sourced from both the preamble and the main body of judgments. The preamble includes structured metadata such as party names, judge and lawyer identities, dates, and court details. Removed irrelevant entity classes: Date.

Class distributions after zero-shot adaption—divided by train, dev, and test sets—are reported in Figure 11.

C LegalNER-ZS Construction

Unlike OntoNotes-ZS and MedMentions-ZS, which use the disjoint splits from Aly et al. (2021), LegalNER lacks predefined zero-shot splits. We

7github.com/chanzuckerberg/MedMentions
8github.com/Legal-NLP-EKStep/legal_NER
therefore construct LegalNER-ZS manually following the same principles. The original dataset consists of two variants (PREAMBLE and JUDGEMENT), each providing only train and dev sets. We merge these to form a unified dataset for split construction.

Our procedure involves:
1. Entity stratification: We divide entity types by frequency into disjoint sets: train (COURT-PETITIONER, RESPONDENT, LAWYER, PROVISION, OTHER_PERSON), dev (JUDGE, ORG, STATUTE), and test (GPE, PRECEDENT, CASE_NUMBER, WITNESS).
2. Trivial type removal: DATE is excluded from validation due to its trivial nature, which could distort model performance.
3. Strict partitioning: We retain only documents whose entities belong entirely to one set (train, dev, or test). Mixed-entity documents are discarded to ensure no entity type overlaps across splits.
4. Preprocessing: Annotations are converted to IOB2 format using NLTK tokenization with span alignment. We discard samples with HTML markup or without valid entities from the split-specific type set.

This results in a clean zero-shot setup where dev and test sets contain entirely unseen entity types, ensuring robust generalization assessment.

D Pile-NER-type Contamination

Table 6 shows the overlap between Pile-NER-type (pretraining dataset for GLiNER and Universal-NER) and the benchmark entity types used for zero-shot evaluation of such models. This intersection undermines the validity of the “zero-shot” claims in the original papers. It confirms that standard zero-shot NER evaluations using Pile-NER-type are heavily confounded by label contamination.

E Further GLiNER Results

While all results in the main paper use lower-case type names for GLiNER inference, the original authors also recommend testing with title-case labels, since the model can be sensitive to casing. Therefore, in Table 7 we report test set performance even in this setting. As shown, the GLiNER-v1 models are largely unaffected by casing, whereas the GLiNER-v2.1 variants register a modest increase in overall performance. Overall, ZeroNER remains the best-performing model, on par with GLiNER-L-v1.

F Hyperparameters and Implementation Details

Table 8 lists all hyperparameters examined in the inference and fine-tuning phases, highlighting the final values. The parameters of the linear classification layer \( \omega^T \) have been randomly initialized from a uniform distribution \( U(-\sqrt{b}, \sqrt{b}) \) with \( b = \frac{1}{in\_features} \). To improve the evaluation of the semantic comprehension capabilities of LLMs, we systematically converted labels that included abbreviations or specific formats into their natural language equivalents. For example, we add a short explanation to tell the model that “LOC” corresponds to “LOCATION”, while “FAC” to “FACILITY” (see Figure 10). Span-level scores are computed with the seqeval library.9

G Inference Efficiency

Table 9 reports the inference-time comparison between ZeroNER and the three GLiNER-v2.1 baselines on OntoNotes-ZS (3 types), MedMentions-ZS (5 types), and LegalNER-ZS (4 types). ZeroNER consistently achieves lower total inference time and higher throughput (samples/s), with a per-sample latency as low as 0.01 s. For instance, on OntoNotes-ZS, ZeroNER processes over 82 samples/s—more than 4× faster than GLiNER-L.

However, because ZeroNER’s cross-encoder pairs each input sentence with every type description, inference cost inevitably grows as the number of types increases (i.e., longer cumulative descriptions) and as input text length grows (i.e., more tokens to encode per pair). This effect is visible on LegalNER-ZS: although ZeroNER still achieves competitive throughput (68.51 samples/s) and matches the lowest latency (0.01 s), its total time (17.91 s) is higher than on MedMentions-ZS. LegalNER-ZS sentences are substantially longer than those in the other benchmarks, so each sentence–description pair incurs more computation. These results underscore that, while ZeroNER is very efficient in practice, inference costs scale with both the number of target types and the length of the input text.

9https://github.com/chakki-works/seqeval
H  Self-Correction

Table 10 illustrates the impact of our self-correction pipeline on model performance. By identifying and correcting noisy annotations using the model’s own predictions, we obtain a higher quality dataset that yields superior results across all benchmarks. The filtered dataset demonstrates consistent improvements, with F1 scores increasing from 0.47 to 0.50 on OntoNotes-ZS , from 0.33 to 0.35 on MedMentions-ZS, and from 0.30 to 0.32 on LegalNER-ZS . The overall macro-averaged F1 score improves from 0.37 to 0.39, confirming that annotation quality is more critical than quantity for zero-shot NER performance. These results validate our hypothesis that self-correction mechanisms can effectively identify and remediate systematic annotation errors, leading to more robust model training and better generalization to unseen entity types.

I  Manual Descriptions

Table 11, Table 12, and Table 13 enumerate all the entity-type descriptions for knowledge fine-tuning in MedMentions-ZS, OntoNotes-ZS, and LegalNER-ZS.
Prompt for Pile Entity Annotation

Annotate the given sentence with entity mentions and their corresponding labels. Then, define the most appropriate domain for the sentence.

Requirements:

Report annotations as a list of tuples in the following format: [(mention, label), ...]

Each mention must include:
- The exact string as it appears in the text.
- The corresponding entity label.

Ensure that:
- Labels do not overlap.
- Each entity is assigned to only one label.

Output following this schema:
Annotations: [list of tuples]
Domain: domain of the sentence

Now, consider the following sentence.

Sentence: { { Sentence } }

Figure 7: Prompt #1. Prompt used to annotate sentences on the Pile.

Prompt for Self-Correction

You are an expert evaluator.
Your task is to evaluate the quality of named entity annotations provided as input by the user, according to the following criteria.

Criterion 1: Completeness of annotations
3 points: All entities present in the sentence are annotated, with nothing missed.
2 points: Most entities are annotated, but one or two entities are missing.
1 point: Several entities are missing from the annotation.
0 points: The annotation is incomplete, with most entities not recognized.

Criterion 2: Correctness of entity labels
3 points: All entities are labeled with the correct entity type (e.g., PERSON, ORGANIZATION).
2 points: Most entities are labeled correctly, but there are one or two minor errors.
1 point: Some entities are correctly labeled, but there are several incorrect labels.
0 points: Entity labels are mostly incorrect, with a significant number of entities mislabeled.

Return you answer in the following output format, allowing for easy parsing:
### explanation: (your explanation for the assessment)
### score: {"completeness": a score from 0 to 3, "correctness": a score from 0 to 3}

Figure 8: Prompt #2. Prompt used to self-correct predictions in terms of correctness and completeness.
Prompt for Multi-View Descriptions

Provide a short description for the class {{Class}} for each of the following domains: {{Domain1, Domain2, ...}}. Include examples. Use a maxium of 100 words.

Output your answer in the following schema:

### descriptions:
domain 1: ...
domain 2: ...
...

Don’t add futher information.

Figure 9: Prompt #3. Prompt used to generate multi-view descriptions.

<table>
  <tr>
    <th>Class</th>
    <th>Domains</th>
    <th>Multi-View Description</th>
  </tr>
  <tr>
    <td rowspan="2">Animal</td>
    <td>Veterinary Science</td>
    <td>Animals are organisms that are not plants. Animals have nervous systems, and are capable of movement. They are multicellular organisms that are not plants. Animal is a living thing with a central nervous system and voluntary muscles. It is a member of the kingdom Animalia, a group of organisms that includes all animals, as well as some single-celled organisms such as amoebas and paramecia.</td>
  </tr>
  <tr>
    <td>Neuroscience</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Scientific Work</td>
    <td>Medicine</td>
    <td>Scientific work is a systematic process of collecting, analyzing and interpreting data to test a hypothesis and to generate new knowledge. Scientists use scientific methods to investigate the natural world. The methods are designed to be repeatable, so that the results of one investigation can be verified by other scientists. Scientists use these methods to test hypotheses, which are predictions about how nature works.</td>
  </tr>
  <tr>
    <td>Biochemistry</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Cell Line</td>
    <td>Biochemistry</td>
    <td>A cell line is a cell culture that is derived from a single cell and has undergone a number of cell divisions in vitro. Cell lines are used in a variety of ways including the study of disease, drug discovery, and the study of cellular processes. A cell line is a population of cells that have been grown in culture for an extended period of time. Cell lines are derived from primary cells, which are cells that have been isolated from a tissue sample and are still alive.</td>
  </tr>
  <tr>
    <td>Cell Biology</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Chemical</td>
    <td>Biochemistry</td>
    <td>A chemical is a substance that has the ability to react with other substances to form new substances. The reaction can be a chemical change, a physical change, or a combination of both. Chemical is a substance with a specific chemical composition and a definite chemical structure.</td>
  </tr>
  <tr>
    <td>Toxicology</td>
    <td></td>
  </tr>
</table>

Table 5: Qualitative examples of entity type multi-view descriptions.
Prompt for Zero-Shot NER

System: You are an expert entity classifier. You have to identify the entities within the given sentence that belong to one of the specified labels. There are no overlapped entities, so each word can belong to only one label. You can use only the possible labels provided. No other label is allowed.

Provide output in the following format: Return a list, marked with square brackets '[' and ']', containing string tuples. Each tuple should follow the pattern: ("entity", "label"). Prefix the entire list with '### entities:'. For example: ### entities: [("entity 1", "label of entity 1"), ("entity 2", "label of entity 2"), ...]

If no entities are found, return an empty list like this: ### entities: []
Don’t add further information.

— Standard —

User:
Possible labels: ['FAC', 'LOC', 'WORK OF ART']

Sentence: The station called me at noon and said something happened at Jingguang Bridge and that I had to go to the station immediately to research the upcoming program .

Consider that FAC corresponds to Facility and LOC corresponds to Location.

— w/ Descriptions —

User:
Possible labels:
FAC: Names of man-made structures: infrastructure (streets, bridges), buildings, monuments, etc. belong to this type. Buildings that are [...]
LOC: Names of geographical locations other than GPEs. These include [...]
WORK OF ART: Titles of books, songs, television programs and other creations. Also includes awards. These are [...]

Sentence: The station called me at noon and said something happened at Jingguang Bridge and that I had to go to the station immediately to research the upcoming program .
Figure 11: **Entity type frequency distribution in the considered benchmarks.** Class occurrences in OntoNotes-ZS, MedMentions-ZS, and LegalNER-ZS (green=train, orange=dev, red=test).
<table>
  <tr>
    <th>Dataset</th>
    <th>Overlaps (dataset type, Pile-NER type)</th>
  </tr>
  <tr>
    <td>ACE 2005</td>
    <td>(Facility, Medical facility),(Organization, Person or organization),(Weapon, Weapon system),(Geographical_social_political, Political), (Vehicle, Vehicles)</td>
  </tr>
  <tr>
    <td>Broad Tweet Corpus</td>
    <td>(Organization, Person or organization)</td>
  </tr>
  <tr>
    <td>CoNLL 2003</td>
    <td>(Organization, Person or organization)</td>
  </tr>
  <tr>
    <td>MultiNERD</td>
    <td>(Disease, Disease or medical condition),(Food, Food source),(Time, Time period),(Organization, Person or organization),(Media, Print media),(Instrument, Musical instrument),(Geographical_phenomenon, Phenomenon),(Biology, Molecular biology),(Plant, Type of plant), (Event, Sports event), (Vehicle, Vehicles)</td>
  </tr>
  <tr>
    <td>Ontonotes</td>
    <td>(Facility, Medical facility),(Quantity, Physical quantity),(Date, Date format),(Organization, Person or organization),(Ordinal, Ordinal number),(Geographical_social_political, Political),(Time, Time period),(Law, Law or bill),(Work_of_art, Work),(Percent, Percentage),(Cardinal, Cardinal number),(Location, Anatomical location),(Product, Product id),(Language, Programming language),(National_religious_political, Political),(Person, Personal information),(Event, Sports event),(Money, Amount of money)</td>
  </tr>
  <tr>
    <td>PolyglotNER</td>
    <td>(Organization, Person or organization)</td>
  </tr>
  <tr>
    <td>TweetNER7</td>
    <td>(Corporation, Type of corporation),(Group, Musicgroup), (Creative_work, Creative_work_element),(Event, Sports event), (Product, Product id)</td>
  </tr>
  <tr>
    <td>WikiANN en</td>
    <td>(Organization, Person or organization)</td>
  </tr>
  <tr>
    <td>WikiNeural</td>
    <td>(Organization, Person or organization)</td>
  </tr>
  <tr>
    <td>AnatEM</td>
    <td>(Anatomy, Brain anatomy)</td>
  </tr>
  <tr>
    <td>bc2gm</td>
    <td>(Gene, Gene/protein)</td>
  </tr>
  <tr>
    <td>bc4chemd</td>
    <td>(Chemical, Chemical compound)</td>
  </tr>
  <tr>
    <td>bc5cdr</td>
    <td>(Chemical, Chemical compound),(Disease, Disease or medical condition)</td>
  </tr>
  <tr>
    <td>CrossNER_AI</td>
    <td>(Conference, Sports conference),(University, College/university),(Researcher, Research),(Organization, Person or organization),(Task, Behavioral task),(Product, Product id),(Country, De facto independent country),(Algorithm, Query evaluation algorithm)</td>
  </tr>
  <tr>
    <td>CrossNER_literature</td>
    <td>(Magazine, Manga magazine),(Poem, Greek epic poem), (Book, Book series),(Organization, Person or organization),(Award, Award show), (Event, Sports event),(Country, De facto independent country),(Writer, Singersongwriter)</td>
  </tr>
  <tr>
    <td>CrossNER_music</td>
    <td>(Organisation, Organisations),(Album, Studio album),(Award, Award show),(Organization, Person or organization), (Band, Bandleader),(Event, Sports event),(Song, Song title),(Musical_artist, Musical), (Misc, Miscellaneous),(Person, Personal information), (Musical_instrument, Instrument),(Country, De facto independent country)</td>
  </tr>
  <tr>
    <td>CrossNER_politics</td>
    <td>(Politician, Georgian politician),(Organization, Person or organization),(Event, Sports event),(Country, De facto independent country),(Political_party, Political)</td>
  </tr>
  <tr>
    <td>CrossNER_science</td>
    <td>(Theory, Level of theory),(University, College/university),(Chemical_compound, Chemical),(Organization, Person or organization),(Award, Award show),(Scientist, Neuroscientist),(Academic_journal, Journal),(Chemical_element, Chemical),(Event, Sports event),(Discipline, Academic discipline),(Enzyme, Enzyme phase),(Protein, Gene/protein),(Country, De facto independent country)</td>
  </tr>
  <tr>
    <td>FabNER</td>
    <td>(Manufacturing_standard, Standard),(Concept_or_principle, Concept),(Machine_or_equipment, Equipment),(Enabling_technology, Technology),(Material, Construction material),(Biomedical, Medical),(Process_parameter, Parameter),(Mechanical_property, Property),(Engineering_feature, Feature),(Manufacturing_process, Process),(Application, Software application)</td>
  </tr>
  <tr>
    <td>FindVehicle</td>
    <td>(Orientation_of_vehicle, Vehicle),(Truck, Truck),(Motorcycle, Motor),(Color_of_vehicle, Vehicle),(Position_of_vehicle, Position), (Vehicle_type, Vehicle),(Vintage_car, Car), (Suv, Car),(Mpv, Car),(Recharge, Charge),(Vehicle_range, Vehicle),(Vehicle_model, Vehicle),(Sport, Sports team)</td>
  </tr>
  <tr>
    <td>GENIA_NER</td>
    <td>(Dna, Dna sequence), (Cell_line, Cell_line),(Protein, Gene/protein)</td>
  </tr>
  <tr>
    <td>HarveyNER</td>
    <td>(Exact_location, Location),(Area, Type of area)</td>
  </tr>
  <tr>
    <td>mitmovie</td>
    <td>(Title, Job title),(Average_ratings, Average),(Genre, Musicgenre),(Review, Type of review),(Song, Song title),(Year, Year founded)</td>
  </tr>
  <tr>
    <td>mitrestaurant</td>
    <td>(Amenity, Amenity),(Restaurant_name, Restaurant),(Hours, Hours),(Price, Price range)</td>
  </tr>
  <tr>
    <td>ncbi</td>
    <td>(Disease, Disease or medical condition)</td>
  </tr>
</table>

Table 6: Overlap of entity types between Pile-NER and the benchmarks used by GLiNER and UniversalNER.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="3">OntoNotes-ZS</th>
    <th colspan="3">MedMentions-ZS</th>
    <th colspan="3">LegalNER-ZS</th>
    <th rowspan="2">AVG</th>
  </tr>
  <tr>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
    <th>R</th>
    <th>P</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>GLiNER_S-v1</td>
    <td>166M</td>
    <td>0.44</td>
    <td>0.34</td>
    <td>0.35</td>
    <td>0.49</td>
    <td>0.33</td>
    <td>0.40</td>
    <td>0.47</td>
    <td>0.26</td>
    <td>0.33</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>GLiNER_M-v1</td>
    <td>209M</td>
    <td>0.41</td>
    <td>0.33</td>
    <td>0.34</td>
    <td>0.53</td>
    <td>0.41</td>
    <td>0.46</td>
    <td>0.45</td>
    <td>0.27</td>
    <td>0.33</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>GLiNER_L-v1</td>
    <td>459M</td>
    <td>0.44</td>
    <td>0.35</td>
    <td>0.37</td>
    <td>0.52</td>
    <td>0.45</td>
    <td>0.48</td>
    <td>0.42</td>
    <td>0.32</td>
    <td>0.32</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>GLiNER_S-v2.1</td>
    <td>166M</td>
    <td>0.28</td>
    <td>0.31</td>
    <td>0.24</td>
    <td>0.40</td>
    <td>0.38</td>
    <td>0.39</td>
    <td>0.34</td>
    <td>0.31</td>
    <td>0.26</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td>GLiNER_M-v2.1</td>
    <td>209M</td>
    <td>0.32</td>
    <td>0.34</td>
    <td>0.29</td>
    <td>0.43</td>
    <td>0.38</td>
    <td>0.39</td>
    <td>0.36</td>
    <td>0.25</td>
    <td>0.26</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td>GLiNER_L-v2.1</td>
    <td>459M</td>
    <td>0.30</td>
    <td>0.40</td>
    <td>0.29</td>
    <td>0.44</td>
    <td>0.43</td>
    <td>0.43</td>
    <td>0.36</td>
    <td>0.24</td>
    <td>0.27</td>
    <td>0.33</td>
  </tr>
  <tr>
    <th>ZeroNER (Ours)</th>
    <th>110M</th>
    <th>0.61</th>
    <th>0.44</th>
    <th>0.50</th>
    <th>0.32</th>
    <th>0.42</th>
    <th>0.35</th>
    <th>0.43</th>
    <th>0.26</th>
    <th>0.32</th>
    <th>0.39</th>
  </tr>
</table>

Table 7: Zero-shot performance comparison between ZeroNER and title-cased GLiNER baselines.
<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Search space</th>
  </tr>
  <tr>
    <th colspan="2">LLM inference (generation, evaluation)</th>
  </tr>
  <tr>
    <td>decoding_method</td>
    <td>"greedy"</td>
  </tr>
  <tr>
    <td>max_new_tokens</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>temperature</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>top_p</td>
    <td>1.0</td>
  </tr>
  <tr>
    <th colspan="2">ZeroNER pre-training</th>
  </tr>
  <tr>
    <td>min_sample_class</td>
    <td>25</td>
  </tr>
  <tr>
    <td>max_sample_class</td>
    <td>20</td>
  </tr>
  <tr>
    <td>max_description_length</td>
    <td>150</td>
  </tr>
  <tr>
    <td>max_sequence_length</td>
    <td>300</td>
  </tr>
  <tr>
    <td>mask_probability</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>linear_dropout</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>linear_units_symbol</td>
    <td>100</td>
  </tr>
  <tr>
    <td>adam_epsilon</td>
    <td>1e-8</td>
  </tr>
  <tr>
    <td>max_grad_norm</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>lr</td>
    <td>{2e-5*, 4e-6, 7e-6}</td>
  </tr>
  <tr>
    <td>epochs</td>
    <td>3</td>
  </tr>
  <tr>
    <td>batch</td>
    <td>8</td>
  </tr>
  <tr>
    <th colspan="2">SMXM fine-tuning</th>
  </tr>
  <tr>
    <th colspan="2">### MedMentions-ZS and LegalNER-ZS</th>
  </tr>
  <tr>
    <td>max_description_length</td>
    <td>100</td>
  </tr>
  <tr>
    <td>max_sequence_length</td>
    <td>200</td>
  </tr>
  <tr>
    <th colspan="2">### OntoNotes-ZS</th>
  </tr>
  <tr>
    <td>max_description_length</td>
    <td>150</td>
  </tr>
  <tr>
    <td>max_sequence_length</td>
    <td>300</td>
  </tr>
  <tr>
    <th colspan="2">### All</th>
  </tr>
  <tr>
    <td>mask_probability</td>
    <td>{0.3*, 0.5}</td>
  </tr>
  <tr>
    <td>linear_dropout</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>linear_units_symbol</td>
    <td>100</td>
  </tr>
  <tr>
    <td>adam_epsilon</td>
    <td>1e-8</td>
  </tr>
  <tr>
    <td>max_grad_norm</td>
    <td>1.0</td>
  </tr>
  <tr>
    <td>lr</td>
    <td>4e-6</td>
  </tr>
  <tr>
    <td>epochs</td>
    <td>3</td>
  </tr>
  <tr>
    <td>batch</td>
    <td>2</td>
  </tr>
  <tr>
    <td>val_steps</td>
    <td>1</td>
  </tr>
</table>

Table 8: Explored hyperparameters along with their empirical search grid. * marks the final picked values.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">OntoNotes-ZS (3 types)</th>
    <th rowspan="2">Latency (s)</th>
  </tr>
  <tr>
    <th>Total time (s)</th>
    <th>Samples/s</th>
  </tr>
  <tr>
    <td>gliner_large-v2.1</td>
    <td>20.27</td>
    <td>21.02</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>gliner_medium-v2.1</td>
    <td>11.09</td>
    <td>38.43</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>gliner_small-v2.1</td>
    <td>8.31</td>
    <td>51.27</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td><b>ZeroNER (ours)</b></td>
    <td>5.18</td>
    <td>82.23</td>
    <td>0.01</td>
  </tr>
  <tr>
    <th colspan="4">MedMentions-ZS (5 types)</th>
  </tr>
  <tr>
    <td>gliner_large-v2.1</td>
    <td>37.02</td>
    <td>28.31</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>gliner_medium-v2.1</td>
    <td>20.33</td>
    <td>51.55</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>gliner_small-v2.1</td>
    <td>13.90</td>
    <td>75.41</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td><b>ZeroNER (ours)</b></td>
    <td>13.60</td>
    <td>77.07</td>
    <td>0.01</td>
  </tr>
  <tr>
    <th colspan="4">LegalNER-ZS (4 types)</th>
  </tr>
  <tr>
    <td>gliner_large-v2.1</td>
    <td>41.58</td>
    <td>29.51</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>gliner_medium-v2.1</td>
    <td>23.12</td>
    <td>53.08</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>gliner_small-v2.1</td>
    <td>15.84</td>
    <td>77.46</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td><b>ZeroNER (ours)</b></td>
    <td>17.91</td>
    <td>68.51</td>
    <td>0.01</td>
  </tr>
</table>

Table 9: Inference time comparison between ZeroNER and GLiNER baselines. Test set.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Size</th>
    <th colspan="3">OntoNotes-ZS</th>
    <th colspan="3">MedMentions-ZS</th>
    <th colspan="3">LegalNER-ZS</th>
    <th rowspan="2">AVG</th>
  </tr>
  <tr>
    <th>R</th><th>P</th><th>F1</th>
    <th>R</th><th>P</th><th>F1</th>
    <th>R</th><th>P</th><th>F1</th>
  </tr>
  <tr>
    <td>ZeroNER</td>
    <td>110M</td>
    <td>0.61</td><td>0.44</td><td>0.50</td>
    <td>0.32</td><td>0.42</td><td>0.35</td>
    <td>0.43</td><td>0.26</td><td>0.32</td>
    <td>0.39</td>
  </tr>
  <tr>
    <td>ZeroNER (unfiltered)</td>
    <td>110M</td>
    <td>0.58</td><td>0.41</td><td>0.47</td>
    <td>0.29</td><td>0.39</td><td>0.33</td>
    <td>0.40</td><td>0.24</td><td>0.30</td>
    <td>0.37</td>
  </tr>
</table>

Table 10: Impact of data filtering on ZeroNER performance. Removing noisy annotations with the proposed data cleaning recipe improves F1 scores across all benchmarks.

<table>
  <tr>
    <th>Type</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>NORP</td>
    <td>This type represents adjectival forms of GPE and Location names, as well as adjectival forms of named religions, heritage and political affiliation. Also marked are head words which refer to people using the name of an entity with which they are affiliated, often a GPE or Organization. The distinction between NORP and other types is morphological. "American" and "Americans" are adjectival nationalities, while "America" and "US" are GPEs, regardless of context.</td>
  </tr>
  <tr>
    <td>PRODUCT</td>
    <td>This can be name of any product, generally a model name or model name and number. Named foods are also included. Credit cards, checking accounts, CDs, and credit plans are NOT marked. References that include manufacturer and product should be marked as two separate named entities, ORG + PRODUCT: [Apple] iPod], [Dell] [Inspiron], [Ford] [Mustang].</td>
  </tr>
  <tr>
    <td>EVENT</td>
    <td>Named hurricanes, battles, wars, sports events, attacks. Metonymic mentions (marked with a ~) of the date or location of an event, or of the organization(s) involved, are included.</td>
  </tr>
  <tr>
    <td>LAW</td>
    <td>Any document that has been made into a law, including named treaties and sections and chapters of named legal documents.</td>
  </tr>
  <tr>
    <td>FAC</td>
    <td>Names of man-made structures: infrastructure (streets, bridges), buildings, monuments, etc. belong to this type. Buildings that are referred to using the name of the company or organization that uses them should be marked as FAC when they refer to the physical structure of the building itself, usually in a locative way: "I'm reporting live from right outside [Massachusetts General Hospital]"</td>
  </tr>
  <tr>
    <td>LOC</td>
    <td>Names of geographical locations other than GPEs. These include mountain ranges, coasts, borders, planets, geo-coordinates, bodies of water. Also included in this category are named regions such as the Middle East, areas, neighborhoods, continents and regions of continents. Do NOT mark deictics or other non-proper nouns: here, there, everywhere, etc. As with GPEs, directional modifiers such as "southern" are only marked when they are part of the location name itself.</td>
  </tr>
  <tr>
    <td>WORK OF ART</td>
    <td>Titles of books, songs, television programs and other creations. Also includes awards. These are usually surrounded by quotation marks in the article (though the quotations are not included in the annotation). Newspaper headlines should only be marked if they are referential. In other words the headline of the article being annotated should not be marked but if in the body of the text here is a reference to an article, then it is markable as a work of art.</td>
  </tr>
</table>

Table 11: OntoNotes-ZS class descriptions. Source: annotation guidelines, LDC2013T19/OntoNotes-Release-5.0.
<table>
  <tr>
    <th>Type</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>CLINICAL ATTRIBUTE</td>
    <td>A clinical attribute is a measurable or observable characteristic used in clinical assessments or diagnosis. Examples include markers and biomarkers, visual acuity, [BMI], histological differentiation.</td>
  </tr>
  <tr>
    <td>BIOMEDICAL OCCUPATION OR DISCIPLINE</td>
    <td>A biomedical occupation or discipline refers to specialized fields or roles in the biomedical sciences or clinical research. Examples include Histopathology, pathology, complementary medicine, family medicine, FM, and specialty.</td>
  </tr>
  <tr>
    <td>INJURY OR POISONING</td>
    <td>An injury or poisoning refers to any form of physical harm or adverse physiological effect caused by trauma, exposure, or internal dysfunction. Examples include postoperative [AKI] (acute kidney injury) and toxicological conditions.</td>
  </tr>
  <tr>
    <td>VIRUS</td>
    <td>A virus is a microscopic infectious agent that replicates only inside the living cells of a host organism. Examples include lentiviruses used in gene therapy and pathogens like SARS-CoV-2.</td>
  </tr>
  <tr>
    <td>ORGANIZATION</td>
    <td>An organization is a structured group or institution involved in healthcare, research, or governance. Examples include government bodies supporting public health initiatives and clinical institutions conducting medical studies.</td>
  </tr>
  <tr>
    <td>BACTERIUM</td>
    <td>A bacterium refers to a type of microorganism that can exist as a single cell and may cause infections or play a role in various biological processes. Examples include species like Streptococcus pneumoniae and Streptomyces ahygroscopicus. Bacteria can be studied for their role in disease, antibiotic production, and other cellular functions.</td>
  </tr>
  <tr>
    <td>BODY SUBSTANCE</td>
    <td>A body substance is any material produced by or found within the body, such as blood, serum, saliva, sweat, or gastric acid. Specific examples include serum cytokine levels for immune responses, blood lipids for metabolic studies, and hemolymph glucose for stress responses.</td>
  </tr>
  <tr>
    <td>FOOD</td>
    <td>A food refers to any substance consumed to provide nutritional support for the body. This includes a wide range of items such as snacks, meat, dairy products, grains like wheat, and edible substances like carbohydrates, proteins, and fats. These items contribute to energy intake, nutrition, and overall diet, and can be metabolized into energy and body tissue by living organisms.</td>
  </tr>
  <tr>
    <td>BODY SYSTEM</td>
    <td>A body system consists of interconnected organs and tissues working together to carry out essential functions. Examples include the gastrointestinal tract for digestion, the nervous system for sensory and motor control, the hematological system for blood-related functions, and the endocrine system for hormone regulation.</td>
  </tr>
  <tr>
    <td>PROFESSIONAL OR OCCUPATIONAL GROUP</td>
    <td>A professional or an occupational group refers to groups of individuals who share the same profession, occupation, or role within a specific field. Examples include cardiologists, psychologists, assessors, hospice staff, and volunteers.</td>
  </tr>
</table>

Table 12: MedMentions-ZS class descriptions. Source: UMLS Metathesaurus, metathesaurus.

<table>
  <tr>
    <th>Type</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>JUDGE</td>
    <td>The name of the judges from the current case.</td>
  </tr>
  <tr>
    <td>ORG</td>
    <td>The names of organizations mentioned in the text.</td>
  </tr>
  <tr>
    <td>STATUTE</td>
    <td>The name of the act or law mentioned in the judgment.</td>
  </tr>
  <tr>
    <td>GPE</td>
    <td>Names of countries, cities, states, districts, villages, or other geopolitical locations mentioned in legal judgments. Examples include [Gujarat], [Basarh], [West Pakistan], [India], [Gaya], [Ranchi], and [England].</td>
  </tr>
  <tr>
    <td>PRECEDENT</td>
    <td>Complete case citations that include party names and legal references used as precedents in judgments. The entire citation should be marked as one entity: [United India Insurance Co. Ltd. v. Rajendra Singh], [State of Rajasthan v. Rajendra Singh, (2009) 11 SCC 106 : (AIR 1998 SC 2554)], [H.R. Sugar Factory (P) Ltd. 187 ITR 363].</td>
  </tr>
  <tr>
    <td>CASE NUMBER</td>
    <td>Standalone case numbers or legal citations mentioned without party names. Examples include [(1962) 45 ITR 210 (SC)] and [Writ Petition No. 1177 of 1974].</td>
  </tr>
  <tr>
    <td>WITNESS</td>
    <td>Names of individuals who testified as witnesses in the current legal proceedings. Examples include [Chandregowda], [Seth], [Saudan Singh], and [Rajesh Kumar Srivastava].</td>
  </tr>
</table>

Table 13: LegalNER-ZS class descriptions. Source: annotation guidelines, Legal-NLP-EkStep/legal_NER.

Reviewer paper summaries: This paper introduces a novel zero-sample named entity recognition approach. The framework aims to enable BERT-based student models to recognize any entity type without additional training by cross-focusing on the description of the entity type. The authors conducted extensive experiments and demonstrate ZERONER's strong generalization ability, allowing it to recognize entity types not seen in the training phase. It is demonstrated that ZERONER achieves state-of-the-art performance.
Summary of strengths: 1. The paper introduces a novel framework that addresses the limitations of existing zero-sample NER methods by incorporating entity type descriptions, which is a unique contribution to the field.
2. The authors provide detailed empirical results demonstrating the validity of their methodology on standardized benchmarks such as MedMentions-ST21pv, MedMentions-ST21pv, and LegalNER.
3. The fact that the entity type description is effective not only in the authors' model, but also when integrated into other existing models, suggests that it has the potential for a wider range of applications.
Summary of weaknesses: 1. Poor quality descriptions may degrade performance, thus inevitably requiring manual fine-tuning of the methodology to adapt ZERONER to a particular benchmark. However, the difficulty and effectiveness of manual fine-tuning may vary for different types of entities and datasets in different domains.
2. The paper uses data sampled from Pile Uncopyrighted to construct a synthetic BIO format dataset, how to further validate the generalizability of this dataset in different domains and scenarios?
3. ZERONER has demonstrated better performance than other models in experiments, but in real-world application scenarios, e.g., when facing massive textual data, its computational resource consumption and operation efficiency may be a limiting factor, so how will ZERONER perform in large-scale data processing scenarios?
Comments, suggestions and typos: 1. Given that ZERONER is sensitive to the quality of entity descriptions, in-depth research can be conducted to investigate methods for automatically generating high-quality descriptions, reducing the reliance on manual fine-tuning, and improving the adaptability of the model in different datasets and scenarios.
2. Including more detailed case studies or examples where the inclusion of entity type descriptions has successfully identified more entities would help illustrate the practical benefits of the approach.
Overall assessment scores: 3.5