Venue: NAACL 2025

# Handling Missing Entities in Zero-Shot Named Entity Recognition:

Integrated Recall and Retrieval Augmentation

 Ruichu Cai\({}^{1,2}\), Junhao Lu\({}^{1}\), Zhongjie Chen\({}^{1}\), Boyan Xu\({}^{1*}\), Zhifeng Hao\({}^{1,3}\)

\({}^{1}\)School of Computer Science, Guangdong University of Technology

\({}^{2}\)Peng Cheng Laboratory

\({}^{3}\)College of Science, Shantou University

{cairuchu, lujunhao.code, bariancgg, hpakyim}@gmail.com haozhifeng@stu.edu.cn

Corresponding author, hpakyjm@gmail.com

###### Abstract

Zero-shot Named Entity Recognition (ZSNER) aims to recognize entities in unseen domains without specific annotated data. A key challenge is handling missing entities while ensuring accurate type recognition, hindered by: 1) the pre-training assumption that each entity has a single type, overlooking diversity, and 2) insufficient contextual knowledge for type reasoning. To address this, we propose IRRA (Integrated Recall and Retrieval Augmentation), a novel two-stage framework leveraging large language model techniques. In the _Recall Augmented Entity Extracting_ stage, we built a perturbed dataset to induce the model to exhibit missing or erroneous extracted entities. Based on this, we trained an enhanced model to correct these errors. This approach can improve the ZS-NER's recall rate. In the _Retrieval Augmented Type Correcting_ stage, we employ Retrieval-Augmented Generation techniques to locate entity-related unannotated contexts, with the additional contextual information significantly improving the accuracy of type correcting. Extensive evaluations demonstrate the state-of-the-art performance of our IRRA, with significant improvements in zero-shot cross-domain settings validated through both auto-evaluated metrics and analysis. Our implementation will be open-sourced at [https://github.com/DMIRLAB-Group/IRRA](https://github.com/DMIRLAB-Group/IRRA).

## 1 Introduction

Zero-shot Named Entity Recognition (ZS-NER) aims to achieve efficient named entity recognition in unseen domains without requiring training on specific annotated datasets. ZS-NER allows for effective entity detection and classification in various domains[50, 49]. The existing leading ZS-NER approaches involve end-to-end training of a robust language model on large NER corpora to enhance its cross-domain generalization capabilities.

Although promising results were reported, we observe that end-to-end solutions have two notable limitations that adversely impact their generalization performance: (i) the lack of diversity in the data annotation process. Existing NER corpora typically assign only one type to each entity, whereas entities in real world may correspond to multiple types. This restricts the model's ability to learn the diversity of entities and understand the connections between different types during end-to-end training. As shown in Figure 1, given a sample similar in form to the training sample but with entities of different types, the semantic distance between the

Figure 1: The upper part shows the difference between typical dataset and perturbed dataset. The typical dataset usually assigns one single gold type to each entity, while The perturbed dataset assigns multiple types to each entity. The dashed boxes represent the scope of entity types sets corresponding to these entity types. The lower part illustrates that the entity types of the retrieved entities may be not precise enough.



types "Train institute", "Institution" and "University" could lead to "PMI" being neither extracted nor recognized, or (ii) the lack of sufficient contextual information for entity recognition. As shown in Figure 1, Unlike in the given training set sample, the context surrounding the entity "PMI" may be insufficient to determine its specific type. The key to overcome these limitations lies in decoupling the end-to-end processes to introduce specific augmentation mechanisms.

Instead of utilizing end-to-end methods, we propose the **IRRA** (**I**ntegrated **R**ecall and **R**etrieval **A**ugmentation), a two-stage ZS-NER framework that strategically leverages large language models (LLMs) techniques. Specifically, **IRRA** is featured with a _Recall Augmented Entity Extracting_ stage and _Retrieval Augmented Type Correcting_ stage. In _Recall Augmented Entity Extracting_ stage, we introduce a data perturbation method by enriching each entity's type with synonyms and hypernyms, thereby constructing a perturbed dataset which model will make inference errors on it. We then use these errors to build an enhanced dataset to train an enhanced extractor, which can recall more entities. Through the above process, as shown in Figure 1, the entity "Stanford" in the training sample matched to multiple types, also increases the probability that the test sample entity "PMI" to be extracted, due to the introduction of the types "Educational institution" and "Institution", which are more closely related to "Training institute". In _Retrieval Augmented Type Correcting_ stage, we employ Retrieval-Augmented Generation (RAG) techniques to retrieve contexts in the database related to the entities extracted in the previous stage. Then we instruct a frozen LLM to correct the entity's type based on the contextual information brought by the retrieved contexts. The main contributions of the paper are summarized as follows:

* A two-stage ZS-NER framework, **IRRA**, composed of an entity extracting stage and a type correcting stage, integrates LLMs techniques to address the issue of lack of information in end-to-end methods.
* A _Recall Augmented Entity Extracting_ stage constructs a perturbed dataset that improves type diversity. Based on this, we propose two different extraction approaches, achieving nearly a 20% improvement in recall rate compared to the base extractor.
* A _Retrieval Augmented Type Correcting_ stage incorporates RAG techniques to address the issue of performance degradation in accurately classifying entities without necessary extensive contextual information.
* We conduct extensive experiments to evaluate the effectiveness of **IRRA**, which notably outperforms the state-of-the-art LLM-based methods.

Figure 2: **The overall framework. In Stage #2, The extracted entities are used to retrieve contexts, and these retrieved entity-related contexts are used to prompt the LLM to correct the type of the entity. (“Missing” indicates that the entity is not extracted in Stage #1)



Related Work

### LLM Self-correction

The evolution of LLMs has brought the concept of self-correction into focus. The discussion around self-correction centers on whether these advanced models can recognize the accuracy of their outputs and provide refined answers [1, 25, 30]. Previous research has explored self-correction methods based on feedback [46], where feedback can come from the LLM itself or from external inputs. Internal feedback relies on the model's inherent knowledge and parameters. In contrast, external feedback involves inputs from humans, other models [78, 9], external tools, external knowledge[21, 22, 23, 24], etc. Our approach is inspired by this technology.

### Retrieval Augmented NER

The RAG method enhances the performance of LLMs by incorporating external knowledge [13]. The standard RAG approach follows the "retrieve-then-read" paradigm. First, a retrieval query is constructed, which is then used to search for relevant information from an external knowledge source such as BookCorpus [25]. The retrieved information is then provided as context to the LLM, allowing the model to generate the final response based on it. Retrieval is based on similarity scores between the query and knowledge text and is mainly divided into two categories: sparse retrievers, like BM25 [26]; and dense retrievers, such as DPR [15], Contriever [11], and BCEmbedding. Additionally, there is work involving graph-based retrieval [1].

In the NER field, there is research using RAG technique to retrieve examples from annotated corpora as few-shots to improve the model's extraction performance [23]. While in medical and biological NER tasks, there is research retrieving relevant entity descriptions by entities matching from the specific databases to correct entity type [12].

However, prior work introducing the RAG technique into the NER field has not attempted to correct entity type in open-domain NER tasks. The work on retrieving few-shot examples does not focus on introducing entity-related external contexts for the model to reference, while retrieval methods in specific domains require manually constructed and maintained databases. To the best of our knowledge, we are the first to introduce the RAG method in the open-domain NER field, and using raw contexts as the retrieval data source. **IRRA** retrieves entity-related contexts to guide the LLM in correcting entity type. This retrieval process can be directly performed by a search engine, and the knowledge base used for retrieval does not require manual annotation, which enhances the robustness and generalizability of the method.

## 3 Methodology

The overall **IRRA** framework is shown in Figure 2. The entire process of this framework is divided into two stages: _Recall Augmented Entity Extracting_ stage and _Retrieval Augmented Type Correcting_. In _Recall Augmented Entity Extracting_ stage, we first deploy an LLM for entity extraction, then construct a perturbed dataset and train an LLM on it to enhance the initial extraction results. This stage is responsible for providing entity extraction results for the next stage while also enabling multi-label recognition. In _Retrieval Augmented Type Correcting_ stage, based on the extracted entities, corresponding queries are generated to retrieve and rerank contexts from the database. These contexts are then provided as additional contextual information to the type correcting model.

### Recall Augmented Entity Extracting

We develop two specialized models to extract entities. First, a base extractor \(Extractor_{base}\) is used for initial entity extraction. Then, a perturbed dataset is constructed, and an enhanced extractor \(Extractor_{enhanced}\) is trained to identify potentially missed entities, thus improve the recall rate of this stage. Afterward, the extraction results of both extractors are merged, providing the results for stage 1, which consists of multiple entity-type pairs.

#### 3.1.1 Base Extractor

We first train a vanilla model on the original dataset \(D\) to serve as the \(Extractor_{base}\) to directly extract the initial entities \(y^{base}_{i}\):

\[y^{base}_{i}=Extractor_{base}(x_{i},T_{i}).\]

Where \(x_{i}\) denotes the sentence to be extracted; \(T_{i}\) denotes the schema which is a set of entity types 

that extraction needs to be performed based on. To facilitate the analysis of the generated results, we stipulate that \(y_{i}^{base}\) must be a parsable JSON string, where each key represents an entity type and its corresponding value is a list of entities.

#### 3.1.2 Enhanced Extractor

To obtain the \(Extractor_{enhanced}\) capable of supplementing the extraction results of the \(Extractor_{base}\), we aim to discover behaviors where the \(Extractor_{base}\) overlooks some potential entities. To achieve this, we introduce a series of perturbation measures to disrupt the training dataset of the \(Extractor_{base}\). When the \(Extractor_{base}\) performs inference on this perturbed dataset, it exhibits behaviors of extraction omission or error. The perturbation measures are as follows. Formally, given a dataset

\[D=\{(x_{i},T_{i},y_{i})|i\in[0,|D|)\}.\]

\(y_{i}=\{(e_{j},t_{j})|j\in[0,|y_{i}|)\}\) denotes the entity-type pairs for the \(i\)-th sample in \(D\). We first define a function \(\mathbf{SH}(\cdot)\) based on the LLM API, which map an entity type \(t\) to the union of the synonym set \(Syn_{t}\) and hypernym \(Hyn_{t}\) set for that entity type:

\[\mathbf{SH}(t)=Sym_{t}\cup Hyp_{t}.\]

Given the \(i\)-th sample in the original dataset \(D\), we apply the following three perturbation measures to get the perturbed dataset \(D^{\prime}\):

* **Random Syno-Hypernym Selection:** For each \(t\in T_{i}\), we randomly select a \(t^{\prime}\in\mathbf{SH}(t)\) and replace \(t\) with \(t^{\prime}\) in \(T_{i}^{p}\). This perturbation introduces variability and disrupts the Base Extractor's reliance on a fixed vocabulary for understanding entity types, helping to uncover instances where it overlooks latent entities during inference.
* **Type Order Shuffle:** We randomly shuffle \(T_{i}^{p}\). This perturbation makes it more difficult for the model to correctly extract entities, as it disrupts its reliance on memorizing task definitions based on entity type order, reflecting the reality that entity type order is not fixed in real-world scenarios and encouraging the model to understand the underlying relationships instead.
* **Syno-Hypernym Types Insertion:** This is the central aspect of our perturbation strategy. For each type \(t\in T_{i}^{p}\), we first draw a random value \(r\sim U(0,1)\) from a uniform distribution. If \(r<p\), we randomly select a \(t^{\prime}\in\mathbf{SH}(t)\) and insert it into \(T_{i}^{p}\). This process is repeated, updating the value of \(r\) each time, until \(r\geq p\). This method compels the model to learn that an entity can correspond to different types and helps the model better comprehend the relationships between various entity types, thereby enhancing its ability to generalize in diverse contexts.

```
1:Input:\(D\), \(p\)
2:\(D^{\prime}=\{\}\)
3:for\((x_{i},T_{i},y_{i})\in D\)do
4:\(T_{i}^{p}=\{\}\)
5:for\(t\in T_{i}\)do
6:\(t^{\prime}\leftarrow\) randomly sample in \(\mathbf{SH}(t)\)
7:\(T_{i}^{p}\gets T_{i}^{p}\cup\{t^{\prime}\}\)
8:endfor
9:\(T_{i}^{p}\leftarrow\) randomly shuffle \(T_{i}^{p}\)
10:\(r\sim U(0,1)\)
11:for\(t\in T_{i}^{p}\)do
12:while\(r<p\)do
13:\(t^{\prime}\leftarrow\) randomly sample in \(\mathbf{SH}(t)\)
14:\(T_{i}^{p}\gets T_{i}^{p}\cup\{t^{\prime}\}\)
15:\(r\sim U(0,1)\)
16:endwhile
17:endfor
18:\(y_{i}^{p}=\{\}\)
19:for\((e_{j},t_{j})\in y_{i}\)do
20:for\(t\in T_{i}^{p}\cap\mathbf{SH}(t_{j})\)do
21:\(y_{i}^{p}\gets y_{i}^{p}\cup\{(e_{j},t)\}\)
22:endfor
23:endfor
24:\(D^{\prime}=D^{\prime}\cup\{(x_{i},T_{i}^{p},y_{i}^{p})\}\)
25:endfor
26:Output:\(D^{\prime}\)
```

**Algorithm 1**Dataset perturbation with Noise Introduction 

\(D^{\prime}\), obtain

\[y_{i}^{\prime}=Extractor_{base}(x_{i},T_{i}^{p}),\]

Then, we place \(y_{i}^{\prime}\) into \(D^{\prime}\), resulting in the enhanced dataset \(D^{\prime\prime}\) for training \(Extractor_{enhanced}\):

\[D^{\prime\prime}=\{(x_{i},T_{i}^{p},y_{i}^{\prime},y_{i}^{p})|i\in[0,|D|)\}.\]

Then we train the \(Extractor_{base}\) on \(D^{\prime\prime}\) to obtain the \(Extractor_{enhanced}\), Then we use the \(Extractor_{enhanced}\) to expand the initial recall results. Finally, we have the result of stage 1:

\[y_{i}^{final}=Extractor_{enhanced}(x_{i},T_{i},y_{i}^{base}).\]

Similarly, \(y_{i}^{final}\) is a parsable JSON string in the same format. We use \(y_{i}^{final}\) for correction in the subsequent steps.

Utilizing these methods on the IEPile dataset, we observed a nearly 20% increase in recall. This significant improvement highlights the effectiveness of our perturbation strategy. By using the perturbed dataset, we aim to develop a more adaptable and resilient model capable of handling diverse and complex real-world scenarios.

### Retrieval Augmented Type Correcting

The previous stage can recall more entities, but there is no guarantee that the corresponding type of each entity is inferred correctly. thus after extracting entities, we need to correctly correct their types. We use LLMs to perform the type correcting task. However, since \(x_{i}\) may lack relevant contextual information to help the LLM identify the correct entity type, We believe that external contexts is needed. We pull out all entities from \(y_{i}^{final}\) and deduplicate them to obtain the set of all entities to be processed, denoted as \(E_{i}^{*}=\{e_{j}^{*}|j\in[0,|E_{i}^{*}|)\}\). For each \(e_{j}^{*}\in E_{i}^{*}\), we apply retrieval augmented type correcting approaches to correctly match the entity with its type. In the correcting stage, we retrieve a small set of entity-related external contexts to guide the correcting model.

#### 3.2.1 Vector Database Construction

The corpora used for retrieval can come from sources such as Wikipedia, domain-specific databases, websites, and other data that do not require manual annotation. In **IRRA**, the retrieval method is independent of the module, we can use search engine retrieval, vector database retrieval, or other methods. However, since using search engine APIs can be costly, we pre-collect the relevant data and use vector database retrieval in this paper. After collecting a large amount of original text, due to the length of the original text often exceeds the context window length of the LLM, we chunk the original text into multiple shorter contexts. Let the set of all contexts be \(\mathcal{D}_{all}=\{d_{l}|l\in[0,|\mathcal{D}_{all}|)\}\). Given the embedding model, we define the embedding function \(\mathbf{Emb}:\mathcal{T}\mapsto\mathbb{R}^{D}\), where \(\mathcal{T}\) represents the text space and \(\mathbb{R}^{D}\) represents the \(D\)-dimensional embedding space. For each context \(d_{l}\in\mathcal{D}_{all}\), we obtain its vector representation \(\mathbf{d_{l}}=\mathbf{Emb}(d_{l})\). Then we store all context vectors in a open-source vector database along with its domain in the corresponding metadata for subsequent retrieval.

#### 3.2.2 Contexts Retrieval

Given an entity to be processsed \(e_{j}^{*}\in E_{i}^{*}\), we construct a retrieval query \(q_{j}=\text{``What is $e_{j}$''}\), and then obtain the vector representation of the query \(\mathbf{q_{j}}=\mathbf{Emb}(q_{j})\). Subsequently, we compute the cosine similarity between the query and the context:

\[CosineScore(q_{j},d_{l})=\frac{\mathbf{q_{j}}\cdot\mathbf{d_{l}}}{||\mathbf{ q_{j}}||\cdot||\mathbf{d_{l}}||}.\]

Based on the \(CosineScore\) of all contexts, we recall the top ncontexts with the highest score, denoted as \(\hat{\mathcal{D}}=\{\hat{d}_{l}|l\in[0,n)\}\).

Due to the limitations of cosine similarity, we additionally use a reranker model to further filter the \(\hat{d}_{l}\in\hat{\mathcal{D}}\). The reranker model is based on the cross-encoder architecture. Unlike other encoders that require the query and context to be fed into separate encoder models, the cross-encoder requires the query and context to be concatenated and input into a single encoder model. This allows the encoder to better capture the relationship between the query and the context. After obtaining the encoder output, the representation of the [CLS] token is taken and passed through a fully connected layer to get the relevance score between the query and the context. Similarly, given the cross-encoder based reranker model, we define the reranker function \(\mathbf{Reranker}:\mathcal{T}\mapsto\mathbb{R}\). We directly concatenate the same query \(q_{j}\) and the retrieved context \(\hat{d}_{l}\), then obtain the score:

\[RenankScore=\mathbf{ReRanker}([q_{j};\hat{d}_{l}]).\]

We then select the top **k** (**k**\(\leq\)**n**) contexts with the highest \(RenankScore\), denoted as \(\mathcal{D}^{*}=\{d_{t}^{*}|l\in[0,k)\}\).

 



#### 3.2.3 Type Correcting

After retrieving the entity-related contexts, we can use these contexts to correctly correct the entity \(e_{j}^{*}\) to its entity type. We treat this as an ICL-based QA task. We use a frozen LLM \(Corrector_{contexts}\) as the type correcting model and construct a prompt using \(x_{i}\), \(e_{j}^{*}\), \(D^{*}\), and \(T_{i}\) to instruct the LLM in performing the type correcting task and obtain the final type \(t_{j}^{*}\) to the entity \(e_{j}^{*}\):

\[t_{j}^{*}=Corrector_{contexts}(x_{i},e_{j}^{*},\mathcal{D}^{*},T_{i}).\]

For each \(e_{j}^{*}\in E_{i}^{*}\), we apply the above process in batches to correct the type of each extracted entity, thereby obtaining the final output \(y_{i}^{final}\) of our framework.

## 4 Experiment

This section presents experiments conducted under zero-shot settings to validate the effectiveness of **IRRA**. The details of the experimental settings are described in the following parts.

### Implement of IRRA

* **Backbones:** We use Llama3-8B as the backbone model for the entity extraction stage and the type correcting model. Llama3-8B aligns well with human intent and is better at understanding and responding to user requests.
* **Training details:** We directly use the trained Llama3-8B provided by the IEPile Gui et al. (2024) as the base extractor. The IEPIle dataset collects a large number of NER datasets and includes some modifications to the original NER datasets. Training for all extractors is performed using supervised instruction-based LoRA Hu et al. (2021) fine-tuning with the LlamaFactory library Zheng et al. (2024) on an Nvidia A800 GPU.
* **Database:** We use Chroma Chroma (2024) as a vector database for retrieval. Due to the high costs associated with using search engine interfaces as a retrieval tool, we collected a large amount of raw text from sources similar to those in the CrossNER dataset to serve as the data source for our vector database. Each text is sliced into multiple smaller contexts with a maximum length of not more than **512** tokens. The end of each context is a period or new-line character to ensure that the context is not split in the middle of a sentence. All the contexts are indexed and stored in the vector database. The number of documents is shown in Table 1.
* **Retrieval:** For retrieval, we adopt BCE Embedding NetEase Youdao (2023), a state-of-the-art model for context embedding, to obtain context embeddings. We first select the top n = 20 contexts, then we use BCEeranker NetEase Youdao (2023), which is usually compatible with BCEmbedding, to score and filter the retrieved n contexts, selecting the top k = 1 contexts.

### Evaluation Metrics

We mainly focus on the zero-shot evaluation. We adopt CrossNER Liu et al. (2020) benchmark following IEPile's settings. CrossNER provides datasets that spans multiple domains, including AI, Literature, Music, Politics, and Science which is suited for evaluating zero-shot NER tasks. We mainly report the micro-F1 score as it's a widely used metric in Named Entity Recognition.

### Comparison Baselines

we use the following methods as baselines:

* **InstructUIE Wang et al. (2023b) InstructUIE improves performance by learning multiple similar tasks. We use the best settings in the corresponding paper as the comparison.
* **ChatGPT**OpenAI (2024) We use the zero-shot settings on ChatGPT directly.
* **UniNER Zhou et al. (2024) This method investigates targeted distillation combined with mission-focused instruction tuning to train student models. These models not only excel in a wide range of applications but also significantly outperform general large language models. We directly use the best scores provided in the paper for comparison.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Tokens** & **AI** & **Literature** & **Music** & **Poltics** & **Science** \\ \hline
512 & 73964 & 2353876 & 2427757 & 2444698 & 1334738 \\
1024 & 33772 & 1158747 & 1144115 & 1144115 & 624434 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The number of contexts in each domain of the database under different maximum token length processing.



* **GoLLIE**(Sainz et al., 2024) GoLLIE provided some Large Language Models trained to follow annotation guidelines. We follow all the settings in this paper completely, and for the results obtained from different models, we directly choose the highest-scoring result, even if the model used is larger than ours.
* **IEPile** This method builds a large dataset to train the model's information extraction capabilities. Since our method uses this dataset for training, its settings are close to ours.
* **OneKE**(Gui et al., 2024) OneKE uses "Schema-based Polling Instruction Construction" method in instruction fine-tuning. We directly use the best scores provided in the paper for comparison.

### Main Results

We report the micro-F1 scores for each domain in CrossNER, as well as the average micro-F1 score, as shown in Table 2. Except for ChatGPT, all methods require extra training data to train the models. Among all the generative LLM-based methods, **IRRA** achieves the highest average micro-F1 score, showing an nearly 4% improvement over the previous SOTA. Additionally, we report the performance of using three different LLMs as backbones for the IEPile method. Consistent with **IRRA**'s results, IEPile also scores lower in the AI and Literature domains, which may be due to internal biases in the extra training data. We also note that using Qwen1.5-14B as the backbone for IEPile results in a relatively higher score in the AI domain, likely due to its broader internal knowledge in AI. The SOTA scores for the AI and Literature domains come from GoLLIE. Although our method is slightly weaker than GoLLIE, the model used by GoLLIE is significantly larger than ours (34B _vs._ 8B). However, even with such a large difference in model size, **IRRA** still achieves higher scores in other domains and a higher average score compared to GoLLIE, which may be due to the compensatory effect of incorporating external knowledge.

## 5 Analytical Experiment

### Ablation Analysis

To analyze the specific parts of the **IRRA** framework, we design several different pipeline settings. For each setting, we conduct the analysis from three dimensions: recall, precision and F1 score.

We report the performance of the base extractor trained solely on IEPile (**Base**) for direct entity extraction. Additionally, we compare two methods in the type correcting stage: one based on inserting annotation guidelines to the prompt **(Guidelines)** and another based on inserting retrieved contexts to the prompt **(Contexts)**.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline

- & 60.9 \\ \hline
**IRRA**(ours) & **IEPile** & 57.5 & 59.3 & **69.4** & **74.0** & **68.3** & **65.7** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The micro-F1 scores on zero-shot cross-domain setting. All settings for baselines are selected by the highest score in the corresponding papers. InstructUIE, UniNER and GoLLIE use extra training data from different sources given in their paper. Since ChatGPT does not provide the training APIs, no extra training data is used. IEPile, OneKE and our method use IEPile’s training set as extra training data.

\begin{table}
\begin{tabular}{l l c c c} \hline \hline
**Entity Extracting** & **Type Correcting** & **Recall** & **Precision** & **F1** \\ \hline Base & ✗ & 49.1 & 54.0 & 50.8 \\ Base \& Enhanced & ✗ & **69.4** & 49.5 & 57.4 \\ \hline Base & Contexts & 53.6 & **72.8** & 61.3 \\ Base \& Enhanced & Guidelines & 59.0 & 63.8 & 61.2 \\ Base \& Enhanced & Contexts & 63.4 & 68.5 & **65.7** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance of different entity extracting methods and type correcting methods across all domains. This comparative analysis experiments are all based on Llama3-8B. All other experimental settings are consistent with the main experiments.

 

As shown in Table 3, incorporating external knowledge to correct the base model's extraction results improves performance as expected. However, applying retrieval-based correction to the enhanced model's extraction results leads to a decrease in recall. This is likely because the recall-enhanced model identifies more challenging entities, which are prone to being incorrectly modified during the correction stage. These errors result in an increase in false negatives (FN) and a corresponding decrease in true positives (TP).

### Effect of Vanilla Models

To verify that our method works with different LLMs, we use Qwen1.5-14B Bai et al. (2023) and Baichuan2-13B Baichuan (2023) as additional backbone models and type correcting model in our framework. All of these models are trained on the IEPile dataset and our perturbed dataset and used as base extractor and enhanced extractor. The results from different models using the IEPile method are shown in Table 2.

We use different combinations of vanilla models within our framework, with the results presented in Table 4. The results indicate that our method improves performance across different vanilla models, with the most significant improvement observed on Llama3-8B, while the improvement is less pronounced on the larger Baichuan2-13B model. From Table 2, we can observe that even without using our method, Baichuan2-13B is still slightly weaker than other models. Based on our observations, Baichuan2-13B is weaker in following instruction, whereas Llama3-8B shows superior performance.

### Effect of Top Documents

We conducted an ablation study by evaluating the performance of our model when using the top-1, top-2, and top-3 highest-scored documents as additional knowledge during inference. The results are shown in the table 5:

In line with human intuition, the performance of the model shows a gradual decline with the increase of **k**. This indicates to some extent that the quality and quantity of retrieved documents have an impact on the performance of the model.

### Generality of Framework

To further demonstrate the validity of our approach, we propose a 10-ways-0-shots benchmark on the Few-NERD Ding et al. (2021) dataset, a commonly used dataset for Few-shot NER that contains a large number of fine-grained entity types. Under this setting, there are 10 random fine-grained entity types in the schema of each sample for extraction, and annotated samples are not allowed to be used as prompts. At the same time, we test several different base models and our methods on this benchmark:

We followed exactly the same configuration as the main experiment, which means that we choose Llama3-8B as the vanilla model. From the results, we can see that our approach can significantly improve the performance of the vanilla model, while also outperforming other basic models. Since this dataset is not divided by domains, we choose to use Wikipedia as an additional source of knowledge, which may limits the performance of our method.

### Correction Analysis

As shown in Figure 3, we observe that when the type of the entity extracted in the previous stage is unique and correct, the performance of both methods is similar. This similarity may be due to that the correcting model is more confident with easily

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Method** & **Recall** & **Precision** & **F1** \\ \hline IEPile-Baichuan2-13B(2024) & 31.3 & 67.3 & 42.7 \\ IEPile-Qwen1.5-14B(2024) & 47.9 & 70.3 & 57.0 \\ IEPile-Llama3-8B(2024) & 29.6 & 68.3 & 41.3 \\ OneKE(2024) & 54.1 & 66.5 & 59.6 \\ IRRA(ours) & 56.0 & 68.0 & 61.4 \\ \hline \hline \end{tabular}
\end{table}
Table 6: We use the test set of supervised partition (a few-nerd default partition) for our experiments.

\begin{table}
\begin{tabular}{c l c c c} \hline \hline
**Entity Extracting** & **Type Correcting** & **Recall** & **Precision** & **F1** \\ \hline Baichuan2-13B & Baichuan2-13B & 51.9 & 52.1 & 51.9 \\ Baichuan2-13B & Llama3-8B & 58.5 & 61.8 & 60.0 & 63.4 \\ Owen1.5-14B & Owen1.5-14B & 62.0 & 65.0 & 63.4 \\ Open1.5-14B & Llama3-8B & 62.6 & 66.0 & 64.2 \\ Llama3-8B & Llama3-8B & 63.4 & 68.5 & 65.7 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of different combinations of models in two stages of **IRRA** over all domains. During the retrieval and type correcting stages, we set the maximum length of the retrieved context to **512** tokens and the number of contexts utilized by the model to **1**.

\begin{table}
\begin{tabular}{l r r r} \hline \hline
**Top Documents** & **Recall** & **Precision** & **F1** \\ \hline
3 & 62.8 & 67.2 & 64.8 \\
2 & 63.1 & 68.0 & 65.3 \\
1 & 63.4 & 68.5 & 65.7 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The number of documents retrieved is controlled by the parameter **k**.

 

judged samples. However, both methods introduce some noise, leading to recognition error, especially the one based on retrieved contexts. Consider the entity "Apple" in a text about fruit. Using retrieved contexts to correct the entity type may retrieve information from technology-related contexts, leading the model to incorrectly match "Apple" to company rather than fruit.

Similar to human intuition, when the extraction model struggles to determine the specific type or completely mismatches the type, introducing the related external knowledge can achieve better improvement. This indicates that the model may lack the necessary contexts to judge the type when extracting the entity.

### Redundant Entities

From Figure 4, we can see that while our method recalls more correct entities, it also retrieves more redundant entities. For the domain-specific types, our method recalls fewer redundant entities, indicating that it is more cautious with less common types. For the common types, the number of redundant entities retrieved by our method increases, suggesting a more lenient approach toward common types. As for the miscellaneous types, our method recalls a large number of redundant entities, which may be due to the unclear definition of redundant entities. This kind of types also exhibits annotation bias in human labeling. For zero-shot tasks, handling the miscellaneous types remains a challenge.

## 6 Conclusion

In this paper, we propose a two-stage **IRRA** framework for zero-shot Named Entity Recognition (NER), strategically leveraging large language model techniques. The core idea of IRRA is to supplement missing information in entity extraction and recognition through the _Recall Augmented Entity Extracting_ and _Retrieval Augmented Type Correcting_ stages. Experimental results show significant improvements over state-of-the-art approaches, highlighting the efficacy of multi-stage processing in large model applications.

## Limitations

This work has two main limitations:

(1) Due to the prohibitive expense of commercial search APIs, our document retrieval relies on cost-efficient alternatives (e.g., local indexes), which may limit result quality compared to industry-grade systems.

(2)The pipeline's sequential structure prevents dynamic cross-module interactions (e.g., feedback loops), potentially capping performance gains. Future work could explore interactive module designs.

## Acknowledgments

This research was supported in part by National Science and Technology Major Project (2021ZD0111501), National Science Fund for Excellent Young Scholars (62122022), Natural Science Foundation of China (62406078, 62476163, U24A20233), Guangdong Basic and Applied Basic Research Foundation (2023B1515120020).

Figure 4: Type cases in redundant recalled entities. **common** represents the type common to all domains such as country, location, organisation and person. **domain-specific** represents the type specific to a domain, and **miscellaneous** represents the type that does not belong to the other two types. We counted the types of entities that were eventually recognized by the model.

Figure 3: Comparison of correction based on annotation guidelines and correction based on retrieved contexts. **em** indicates that the extracted entity’s type exactly matches the ground truth type. **cm** indicates that among the multiple types for the extracted entity, at least one matches the ground truth type. **nm** indicates that none of the types for the extracted entity match any ground truth types. **redundant** indicates that the entity should not be extracted.

 



## References

* (1)
* Baek et al. (2023) Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. _Preprint_, arXiv:2306.04136.
* Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Qwen technical report. _arXiv preprint arXiv:2309.16609_.
* Bai et al. (2021) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drein, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Karmile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conferly, Tom Henigham, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional ai: Harmlessness from ai feedback. _Preprint_, arXiv:2212.08073.
* Baichuan (2023) Baichuan 2: Open large-scale language models. _arXiv preprint arXiv:2309.10305_.
* Chen et al. (2024) Xinyun Chen, Maxwell Lin, Nathanael Scharli, and Denny Zhou. 2024. Teaching large language models to self-debug. In _The Twelfth International Conference on Learning Representations_.
* Chroma (2024) Chroma. 2024. Chroma: An open-source ai application database. [https://www.trychroma.com](https://www.trychroma.com). Accessed: 2024-07-13.
* Ding et al. (2021) Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021. Few-nerd: A few-shot named entity recognition dataset. _Preprint_, arXiv:2105.07464.
* Gao et al. (2023) Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. RARR: Researching and revising what language models say, using language models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 16477-16508, Toronto, Canada. Association for Computational Linguistics.
* Gou et al. (2024) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. Critic: Large language models can self-correct with tool-interactive critiquing. _Preprint_, arXiv:2305.11738.
* Gui et al. (2024) Honghao Gui, Lin Yuan, Hongbin Ye, Ningyu Zhang, Mengshuh Sun, Lei Liang, and Huajun Chen. 2024. Iepile: Unearthing large-scale schema-based information extraction corpus. _CoRR_, abs/2402.14710.
* Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _Preprint_, arXiv:2106.09685.
* Huang et al. (2024) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. _Preprint_, arXiv:2310.01798.
* Izcard et al. (2022) Gautier Izcard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. _Preprint_, arXiv:2112.09118.
* Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense passage retrieval for open-domain question answering. _Preprint_, arXiv:2004.04906.
* Kim et al. (2024) Seoyeon Kim, Kwangwook Seo, Hyungjoo Chae, Jinyoung Yeo, and Dongha Lee. 2024. VerifiNER: Verification-augmented NER via knowledge grounded reasoning with large language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2441-2461, Bangkok, Thailand. Association for Computational Linguistics.
* Kobayashi (2018) Sosuke Kobayashi. 2018. Homemade bookcorpus. [https://github.com/sosek/bookcorpus](https://github.com/sosek/bookcorpus).
* Lewis et al. (2021) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. _Preprint_, arXiv:2005.11401.
* Liu et al. (2020) Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2020. Crossner: Evaluating cross-domain named entity recognition.
* Madaan et al. (2020)Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder,

Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. _Preprint_, arXiv:2303.17651.
* NetEase Youdao (2023) Inc. NetEase Youdao. 2023. Bcembedding: Bilingual and crosslingual embedding for rag. [https://github.com/netease-youdao/BCEmbedding](https://github.com/netease-youdao/BCEmbedding).
* Olausson et al. (2024) Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2024. Is self-repair a silver bullet for code generation? _Preprint_, arXiv:2306.09896.
* OpenAI (2024) OpenAI. 2024. Chatgpt. [https://www.openai.com/chatgpt](https://www.openai.com/chatgpt). Accessed: 2024-07-13.
* Pan et al. (2023) Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. _Preprint_, arXiv:2308.03188.
* Paul et al. (2024) Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2024. REFINER: Reasoning feedback on intermediate representations. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1100-1126, St. Julian's, Malta. Association for Computational Linguistics.
* Picco et al. (2023) Gabriele Picco, Marcos Martinez Galindo, Alberto Purpura, Leopold Fuchs, Vanessa Lopez, and Thanh Lam Hoang. 2023. Zshot: An open-source framework for zero-shot named entity recognition and relation extraction. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_, pages 357-368, Toronto, Canada. Association for Computational Linguistics.
* Robertson (2009) S. Robertson. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. _Foundations and Trends(r) in Information Retrieval_, 3(4):333-389.
* Sainz et al. (2024) Oscar Sainz, Iker Garcia-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2024. GoLLIE: Annotation guidelines improve zero-shot information-extraction. In _The Twelfth International Conference on Learning Representations_.
* Wang et al. (2023a) Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazeldarandi, and Asli Celikyilmaz. 2023a. Shepherd: A critic for language model generation. _Preprint_, arXiv:2308.04592.
* Wang et al. (2023b) Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023b. Instructutie: Multitask instruction tuning for unified information extraction. _arXiv preprint arXiv:2304.08085_.
* Welleck et al. (2022) Sean Welleck, Kiming Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. _Preprint_, arXiv:2211.00053.
* Xie et al. (2024a) Tingyu Xie, Qi Li, Yan Zhang, Zuzozhu Liu, and Hongwei Wang. 2024a. Self-improving for zero-shot named entity recognition with large language models. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 583-593, Mexico City, Mexico. Association for Computational Linguistics.
* Xie et al. (2024b) Tingyu Xie, Jian Zhang, Yan Zhang, Yuanyuan Liang, Qi Li, and Hongwei Wang. 2024b. Retrieval augmented instruction tuning for open ner with large language models. _Preprint_, arXiv:2406.17305.
* Zheng et al. (2024) Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. LIAMATactory: Unified efficient fine-tuning of 100+ language models. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_, Bangkok, Thailand. Association for Computational Linguistics.
* Zhou et al. (2024) Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2024. Universalner: Targeted distillation from large language models for open named entity recognition. _Preprint_, arXiv:2308.03279.

## Appendix A Appendix

### Prompts

**Syno-Hypernyms generation prompt** is specifically designed and employed to generate a comprehensive set of synonyms and hypernyms for each entity type. This process ensures that for every entity type, a robust and contextually relevant collection of alternative expressions is created.

```
Prompt\({}^{\prime}\)Synonym generation prompt
```

Complete each word's list with multiple possible synonyms and hypernyms. {

*entitytype1":["entitytype1"], 'entitytype2":["entitytype2"], ... } ```

**Extraction prompt** is utilized to guide the base extractor. It directs the models to accurately extract entities from the provided text in accordance with the predefined schema. This prompt ensures that the extraction process is aligned with the specific 

requirements of the schema, thereby improving the quality of the extracted results.

**Correcting prompt based on retrieved contexts** is employed to guide the model in correcting entities and their types based on the retrieved contexts. This prompt ensures that the model accurately corrects each entity and its corresponding type.

```
("instruction":"Please extractentitiesthatmatchtheschemadefinitionfromtheminput.Returnanemptylistiftheentitytypesdoesnotexist.pleaserespondintheformatofaJSONstring-",{"scheme":[listofentitypes],"input":"inpustentencestring"} )
```

**Component prompt** is used to guide the enhanced extractor. It directs the model to expand and complement the initial recall results.

```
("instruction":"Modifytheextractedentitiesbasedoninputandschema.Ifyouthinktheremaybesomeentitiesmissingorthydont'understandsonething,pleaselistthenasentities.","schema":[Listofentitypes],"input":"inputsentencestring","entities":("neititeslytope1":[Listofentities],"eneitytype2":[Listofentities],...) )
```

**Competent prompt** Corefing promptbasedonretrieved contexts**

```
#InstructionYouneedtoanswerthe**type** of the corresponding entity in the text according to the**Types** and**therelevant contexts retrieved**.
``` #Text{text} {ext} #Entity{entity} #Relevantcontexts{contexts} #Types|listofentitypes] #Rules Guidanceinformationneededwhenextractingentities.

```
#Answerformat{ "entity":"theentityinentity",reason":"ThereasonyoujudgeothetypechwhichfromTypes","entitytype":"typefromTypes" }
```

**Competent prompt** CorefingprompbasedOnannotation guidelines**

```
#InstructionYouneedtoanswerthe**type** of the corresponding entity in the text according to the**Types** and**guidelines**.
``` #Text{text} #Entity{entity} #GuidelinesAnnotationguidelinesaboutthedata.#Types|listofentitypes|#Rules Guidanceinformationneededwhenextractingentities.#Answerformat({entity="entity":"theentityinentity","reason":"ThereasonyojudgeethetypechwhichfromTypes","entitytype":"typefromTypes"} ) ```

**Correcting promptbasedonannotation guidelines** 



**guidelines** is employed to guide the model in correcting entities and their types based on predefined annotation guidelines. This prompt also ensures that the model accurately corrects each entity and its corresponding type.