Venue: NAACL 2024

# Self-Improving for Zero-Shot Named Entity Recognition

with Large Language Models

 Tingyu Xie\({}^{1,2}\), Qi Li\({}^{1,2}\), Yan Zhang\({}^{3*}\), Zuozhu Liu\({}^{2}\), Hongwei Wang\({}^{1,2*}\)

\({}^{1}\)College of Computer Science and Technology, Zhejiang University, China

\({}^{2}\)ZJU-UIUC Institute, Zhejiang University, China

\({}^{3}\)National University of Singapore, Singapore

{tingyuxie, hongweiwang}@zju.edu.cn, yanzhang.jlu@gmail.com

Corresponding authors.Code and data are publicly available: [https://github.com/Eman1066/Self-Improve-Zero-Shot-NER](https://github.com/Eman1066/Self-Improve-Zero-Shot-NER)

###### Abstract

Exploring the application of powerful large language models (LLMs) on the named entity recognition (NER) task has drawn much attention recently. This work pushes the performance boundary of zero-shot NER with LLMs by proposing a training-free self-improving framework, which utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs. First, we use the LLM to make predictions on the unlabeled corpus using self-consistency and obtain a self-annotated dataset. Second, we explore various strategies to select reliable annotations to form a reliable self-annotated dataset. Finally, for each test input, we retrieve demonstrations from the reliable self-annotated dataset and perform inference via in-context learning. Experiments on four benchmarks show substantial performance improvements achieved by our framework. Through comprehensive experimental analysis, we find that increasing the size of unlabeled corpus or iterations of self-improving does not guarantee further improvement, but the performance might be boosted via more advanced strategies for reliable annotation selection.1

Footnote 1: Code and data are publicly available: [https://github.com/Eman1066/Self-Improve-Zero-Shot-NER](https://github.com/Eman1066/Self-Improve-Zero-Shot-NER)

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 583–593

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

There have been many works exploring new possibilities of the named entity recognition (NER) task in the era of large language models (LLMs) [7, 13, 1] recently. These studies include designing advanced prompting methods for zero-shot prediction or few-shot in-context learning (ICL) [22, 23, 24, 25], training task-specific LLMs for NER [14, 25], and generating data with LLMs to train small specific models [11, 25, 26].

In this work, we explore the possibility of pushing the performance boundary of zero-shot NER with LLMs via self-improving. We focus on the strict zero-shot scenarios where no annotated data is available but only an unlabeled corpus is accessible, and no training resource or auxiliary models are available. We propose a totally training-free self-improving framework for NER, which utilizes an unlabeled corpus to stimulate the self-learning ability of LLMs. The framework consists of the following three steps. (1) Step 1: we use LLMs to self-annotate the unlabeled corpus using self-consistency [12]. Each annotated entity is associated with a SC score, which is used as the measure of the reliability of this annotation. (2) Step 2: we select reliable annotation to form a reliable self-annotated dataset, during which diverse annotation selection strategies are explored, including entity-level threshold filtering, sample-level threshold filtering and two-stage majority voting. (3) Step 3: for each arrived test input, we perform inference via ICL with demonstrations from the reliable self-annotated dataset. Various strategies for demonstration retrieval are explored.

Our contributions include: (1) We proposed a training-free self-improving framework for zero-shot NER with LLMs. (2) This framework achieved significant performance improvements on four benchmarks. (3) We conduct comprehensive experimental analysis, finding that increasing the size of unlabeled corpus or iterations of self-annotating does not guarantee gains, but there might be room for improvements with more advanced strategies for reliable annotation selection.

## 2 Zero-Shot NER with Self-Improving Motivation.

To push the performance boundary of zero-shot NER with LLMs, we propose a self-improving framework under a strict zero-shot and low-resource setting: No annotated data but only an

unlabeled corpus is available; No auxiliary model or training step is required. This study is orthogonal to previous prompt designing works, as any advanced prompting method can be applied to this framework. Fig. 1 shows the framework overview.

**Task Formulation**. Given an input sentence \(x\), the NER task is to recognize the structure output \(y\) from \(x\), which consists of a set of \((e,t)\) pairs. \(e\) is an entity span, which is a sequence of tokens form \(x\); \(t\) is the corresponding entity type, which belongs to a predefined entity type set.

### Step 1: Zero-Shot Self-Annotating

We assume an unlabeled corpus \({\cal U}=\{x_{i}\}_{i=1}^{n}\) is available. We use the training set without labels as the unlabeled dataset in this work. For unlabeled sample \(x_{i}\), we generate predictions with LLMs via zero-shot prompting, as shown in upper part of Fig. 1. This process is formulated as \(y_{i}=\arg\max_{y}P(y|T,x_{i})\), where \(T\) is the task instruction of NER, and \(y_{i}=\{(e_{i}^{j},t_{i}^{j})\}_{j=1}^{m}\). We apply self-consistency (SC) Wang et al. (2022) to obtain a SC score for each prediction, which will be used in step 2 for reliable annotation selection. We sample multiple answers from the model, and the vote for each predicted entity \((e_{i}^{j},t_{i}^{j})\) is the times it appeared in all the sampled answers, which we denoted as entity-level SC score \(c_{i}^{j}\). Then we get the sample-level SC score \(c_{i}\) for each input sentence \(x_{i}\) by taking the average SC score over all predicted entities in this sentence, i.e., \(c_{i}=\frac{1}{m}\sum_{j}c_{i}^{j}\). For each self-annotated sample with SC scores, we can denote it as \((x_{i},\{(e_{i}^{j},t_{i}^{j},c_{i}^{j})\}_{j=1}^{m},c_{i})\).

### Step 2: Reliable Annotation Selection

We assume that a higher SC score indicates a higher reliablity. Thus, we investigate the three following strategies for reliable annotation selection. (1) _Entity-level threshold filtering_, which drops the predicted entity \(e_{i}^{j}\) if \(c_{i}^{j}<Th\_entity\), where \(Th\_entity\) is the threshold for entity-level SC score. (2) _Sample-level threshold filtering_, which drops the sample \(x_{i}\) if \(c_{i}<Th\_sample\), where \(Th\_sample\) is the threshold for sample-level SC score. (3) _Two-stage majority voting_Xie et al. (2023), is an entity-level selection method, which first votes for the most consistent entity spans, then the most consistent types based on the voted spans.

### Step 3: Inference with Self-Annotated Demonstration

When a test input \(x^{q}\) arrives, we retrieve \(k\) demonstrations from the reliable self-annotated dataset to help the inference. 2 We investigate the following four methods for demonstration retrieval. (1) _Random retrieval_, which randomly select \(k\) demonstrations. (2) _Nearest retrieval_, which select the \(k\) nearest neighbors of \(x^{q}\). The distance of samples is measured by the cosine similarity in the representation space. (3) _Diverse nearest retrieval_, which first retrieve \(K\) nearest neighbors, where \(K>k\), then uniformly samples a random set of \(k\) samples from the \(K\) neighbors. (4) _Diverse nearest with SC

Figure 1: The overview of the proposed self-improving framework for zero-shot NER with LLM.

 

ranking_, proposed by this work to achieve a better trade-off between the similarity, diversity and reliability of self-annotated demonstrations. After retrieving \(K\) nearest neighbors, we select samples with the top-\(k\) sample-level SC scores.

Let \(S=\{x_{i},y_{i}\}^{k}_{i=1}\) denotes the self-annotated demonstrations retrieved for the test input \(x^{q}\). Finally, our framework conduct ICL by concatenating these \(k\) samples as well as the test input sentence \(x^{q}\), as shown in the below part in Fig. 1. The prediction is obtained via \(y^{q}=\operatorname*{arg\,max}_{y}P(y|T,S,x^{q})\).

## 3 Experiment

### Setup

We experiment on four widely-used NER datasets, CoNLL03 Sang and De Meulder [2003], ACE05 Walker et al. [2006], WikiGold Balasuriya et al. [2009] and GENIA Ohta et al. [2002]. We use GPT3.5 (gpt-3.5-turbo) as the LLM backbone and text-embedding-ada-002 model to get sentence representations.3 We set \(k=16\) and \(K=50\). For SC, we set temperature to 0.7 and sample 5 answers. For cost saving, we randomly sample 300 test samples twice then report the means and standard deviations, and we randomly sample 500 training samples without labels to form the unlabeled corpus \(\mathcal{U}\). The naive zero-shot prompting is our baseline, which we denote as _No-demos_. We report F1 scores throughout this paper.

Footnote 3: The results of GPT-3.5 are obtained during October and November 2023 with official API.

### Results

The main results are shown in Table 1. Results of other values for thresholds \(Th\_entity\) and \(Th\_sample\) can be found in Appendix E. (1) Without annotation selection, we only generate one answer for each unlabeled sample. The results show improvements over _No-demos_, revealing that our framework is helpful even without any carefully designed annotation selection step. (2) The performance is further improved under three annotation selection strategies respectively. (3) The proposed diverse nearest with SC ranking shows consistent improvements under various settings and achieves the best results when combined with two-stage majority voting. This confirms that this strategy

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & **CoNLL03** & **ACE05** & **WikiGold** & **GENIA** & **Avg** \\ \hline No-demos & 68.97 \({}_{0.22}\) & 27.29 \({}_{0.58}\) & 70.8 & 47.41 \({}_{0.29}\) & 53.62 \\ \hline \multicolumn{6}{c}{**ICL with self-annotated demonstrations (Zero-shot)**} \\ \hline \multicolumn{6}{c}{_Without annotation selection_} \\ Random & 71.45 \({}_{0.10}\) & 30.38 \({}_{0.93}\) & 70.51 & 48.78 \({}_{0.06}\) & 55.28 \\ Nearest & 72.07 \({}_{0.11}\) & **32.20**\({}_{0.92}\) & **71.81** & 49.54 \({}_{1.88}\) & **56.40** \\ Diverse Nearest, random & **72.15**\({}_{0.65}\) & 31.07 \({}_{1.45}\) & 70.72 & **50.01**\({}_{1.20}\) & 55.99 \\ \hline \multicolumn{6}{c}{_Entity-level threshold filtering_} \\ Random & 70.91 \({}_{0.55}\) & 30.41 \({}_{0.95}\) & 72.33 & 50.70 \({}_{1.53}\) & 56.09 \\ Nearest & 73.24 \({}_{0.53}\) & 32.22 \({}_{0.38}\) & 72.53 & 49.85 \({}_{1.20}\) & 56.96 \\ Diverse Nearest, random & 74.11 \({}_{0.12}\) & **32.29**\({}_{0.31}\) & 72.01 & 50.68 \({}_{0.14}\) & 57.27 \\ Diverse Nearest, SC ranking & **74.99**\({}_{0.20}\) & 31.65 \({}_{0.97}\) & **73.53** & **51.11**\({}_{0.28}\) & **57.82** \\ \hline \multicolumn{6}{c}{_Sample-level threshold filtering_} \\ Random & 72.41 \({}_{1.28}\) & 30.00 \({}_{1.26}\) & **73.38** & 51.61 \({}_{1.21}\) & 56.86 \\ Nearest & 72.28 \({}_{0.14}\) & **32.00**\({}_{0.08}\) & 73.27 & **52.72**\({}_{0.80}\) & **57.57** \\ Diverse Nearest, random & 77.23 \({}_{0.08}\) & 30.74 \({}_{0.06}\) & 72.09 & 52.50 \({}_{0.50}\) & 56.91 \\ Diverse Nearest, SC ranking & **73.97**\({}_{0.12}\) & 31.08 \({}_{0.54}\) & 72.80 & 51.67 \({}_{0.93}\) & 57.38 \\ \hline \multicolumn{6}{c}{_Two-stage majority voting_} \\ Random & 72.12 \({}_{0.59}\) & 31.18 \({}_{0.38}\) & 72.32 & 50.17 \({}_{0.93}\) & 56.45 \\ Nearest & 71.66 \({}_{0.37}\) & 31.45 \({}_{1.32}\) & 72.84 & 50.19 \({}_{1.59}\) & 56.53 \\ Diverse Nearest, random & 72.45 \({}_{0.41}\) & 30.84 \({}_{0.56}\) & 70.83 & 51.03 \({}_{0.73}\) & 56.28 \\ Diverse Nearest, SC ranking & **74.51**\({}_{0.03}\) & **32.27**\({}_{0.25}\) & **73.98** & **52.06**\({}_{0.09}\) & **58.20** \\ \hline \multicolumn{6}{c}{**ICL with gold labeled demonstrations**} \\ \hline Random (Gold) & 78.36 \({}_{0.31}\) & 42.12 \({}_{0.30}\) & 74.27 & 54.50 \({}_{1.14}\) & 62.31 \\ Nearest (Gold) & 84.30 \({}_{0.39}\) & 52.72 \({}_{0.44}\) & 78.20 & 54.78 \({}_{0.94}\) & 67.50 \\ \hline Random (Gold), full data & 78.35 \({}_{1.44}\) & 41.33 \({}_{0.79}\) & 78.47 & 52.77 \({}_{0.03}\) & 62.73 \\ Nearest (Gold), full data & 83.51 \({}_{0.02}\) & 55.54 \({}_{0.61}\) & 79.73 & 58.72 \({}_{1.52}\) & 69.37 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Main results. The right subscript number are standard deviations. _Gold_ indicates the method has access to the gold labeled data, thus is not comparable with the rest of methods. _Full data_ indicates the method has access to the full training set. Results of \(Th\_entity=4.0\) and \(Th\_sample=4.0\) is shown here. Texts in **bold** are the best results in each category; Text underlined are the best results among all methods. The proposed framework significantly improves the zero-shot performances. On average, two-stage majority voting combined with the proposed diverse nearest with SC ranking achieves the best results.

 

achieves a better trade-off between similarity, diversity and reliability of the demonstrations. (4) Random retrieval lags behind nearest retrieval in self-improving scenario but is not as much as in the gold label scenario, likely because of the noise contained in self-annotated labels. The model may directly copy the wrong answers in the most similar self-annotated demonstrations due to the copy mechanism of ICL Lyu et al. (2023).

### Analysis

**Increasing unlabeled data**. We expanded the size of \({\cal U}\) by 10 times and randomly sampled 5000 samples from the original training set. Results are shown in Fig. 2. Increasing the size of the unlabeled corpus does not guarantee performance improvements under the self-improving scenario. Meanwhile, increasing the size of the demonstration pool only brings marginal improvement, even under the gold label scenario. The reason may be that the small dataset already approximately captures the data distribution.

**Iterative self-improving**. We use the self-annotated data as demonstrations to guide the next iteration of self-annotating, forming a bootstrapping process. The illustration of iterative self-improving process can be found in Appendix G. We experiment up to 8 iterations. The 0-th iteration indicates the _No-demos_ setting. Results are shown in Fig. 3. Increasing iterations of self-improving cannot guarantee improvements on most datasets. This may due to the fact that error accumulation in self-annotating is difficult to be eliminated in this training-free process.

**Upper bound of reliable annotation selection.** We keep only the true predictions and discard the false predictions in all the sampled answers to evaluate the upper bound of reliable annotation selection. Results are shown in Table 2. More detailed results can be found in Appendix F. _Upper bound_ setting performs on par with the _Gold label_ setting, indicating that there might still be space to be improved for reliable annotation selection.

**SC score analysis.** We plot the kernel density estimation for entity-level SC scores in Fig. 4. Most true predictions gather in the interval of high SC scores, while most false predictions have low SC scores. This shows that SC scores effectively reflect the reliability of annotations.

**Self-verification.** Besides SC, we also explore self-verification (SV) to measure the confidence of self-annotation by asking the LLM to score its own answer about its own confidence. After the LLM outputs the recognized entities, we obtain the

Figure 4: Kernel density estimation for SC scores. Vertical axes represent density, horizontal axes represent SC scores.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & **CoNLL03** & **ACE05** & **WikiGold** & **GENIA** & **Avg** \\ \hline No-demos & 68.97 & 27.29 & 70.8 & 47.41 & 53.62 \\ \hline TSMV & 74.51 & 32.27 & 73.98 & 52.06 & 58.20 \\ Upper bound & 81.65 & 37.82 & 76.57 & 56.24 & 63.07 \\ \hline Gold label & 84.30 & 52.72 & 78.20 & 54.78 & 67.50 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results of the upper bound of reliable annotation selection. _TSMV_ represents two-stage majority voting. We display the best results for each strategy. The setting of _Upper bound_ performs on par with the setting of _Gold label_, showing that there might be space to be improved for reliable annotation selection.

Figure 3: Increasing the iterations of self-improving does not guarantee performance improvements.

Figure 2: Results of increasing the size of unlabeled dataset. Vertical axes represent F1 scores. _Ours_ refers to the combination of two-stage majority voting and diverse nearest with SC ranking. Increasing unlabeled data does not guarantee performance gains.

 

SV score by asking the LLM: "How confident are you in providing the above answers? Please give each named entity in your answer a confidence score of 0-5." The comparison results between SC and SV are in Table 3. As shown in the table, SV also achieves some improvements compared with the No-demos baseline. However, it lags behind the SC measurement. This is presumably because the LLM tends to be over-confident about its own answer, since we found that no sample gets a confidence score lower than 3 under the SV measurement in CoNLL03 benchmark. The overconfidence problem is also mentioned in Li et al. (2023).

**Evaluation on weaker LLMs.** To explore the performance of the proposed self-improving framework on weaker LLMs, we conduct experiments on the Llama2 chat 13B model Touvron et al. (2023),4 the results are shown in Table 4. Two-stage majority voting selection strategy and the nearest neighbor retrieval method are used in this experiment. With a much weaker ability in zero-shot scenarios, Llama2 13B model shows negative results under the self-improving framework. This indicates that the proposed framework is more suitable for models with a strong zero-shot capability. For the models with a relatively weaker zero-shot ability, improving the prompt designing might be a more effective strategy to boost performance.

Footnote 4: [https://huggingface.co/meta-llama/Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf)

## 4 Related Work

**Information extraction with LLM.** The research of information extraction (IE) with LLMs includes prompt designing Wei et al. (2023); Wang et al. (2023); Xie et al. (2023); Li et al. (2023), task-specific LLMs instruction-tuning Zhou et al. (2023); Sainz et al. (2023) and data augmentation Zhang et al. (2023); Ma et al. (2023); Josifoski et al. (2023). Zhang et al. (2023) use LLM to annotate data, which is used to fine-tune a specific IE model, then the fine-tuned model is used to help select the data to be annotated in the next iteration. Unlike previous works, this work propose a training-free self-improving framework to push the zero-shot boundary of LLM on NER. Different from Zhang et al. (2023), no seed labeled data, expert small model nor training resources are required in our framework. In addition, our work is **orthogonal** to previous prompt designing works. They explored various advanced prompt formats to boost performance, and did not utilize unlabeled corpus. Unlike them, this work improves zero-shot NER by using unlabeled corpus without designing any complex prompt format.

**Demonstrations in ICL.** Some works explored factors that have impacts on ICL Lyu et al. (2023); Min et al. (2022); Wei et al. (2023). Lyu et al. (2023) investigate the impact of randomly assigning labels to demonstrations in ICL. However, this random labeling method is not suitable for tasks like NER, which requires label information on the token-level instead of sentence-level. Different from them, we first use LLM to make predictions on the unlabeled corpus, then select reliable self-annotated data as demonstrations.

## 5 Conclusion

We propose a training-free self-improving framework for zero-shot NER with LLMs, which achieves significant performance improvements on four benchmarks. Comprehensive experimental analysis shows that, simply increasing the size of unlabeled corpus or the iterations of self-annotation do not guarantee further improvement, but there might still be room for improvement with more advanced strategies for reliable annotation selection.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Method** & **CoNLL03** & **WikiGold** \\ \hline No-demos & 42.24 & 28.57 \\ \hline Nearest & 23.55 & 8.94 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results on the Llama2 chat 13B. Two-stage majority voting is used here. The negative results show that the proposed framework is more suitable for models with a strong zero-shot capability. The negative effect is obvious on the first sampled test set, thus we do not continue to test on other seeds.

\begin{table}
\begin{tabular}{l l r} \hline \hline
**Method** & **SC** & **SV** \\ \hline No-demos & 68.97 \({}_{0.22}\) \\ \hline \multicolumn{3}{l}{_Entity-level threshold filtering_} \\ Random & 70.91 \({}_{0.55}\) & 70.91 \({}_{0.56}\) \\ Nearest & 73.24 \({}_{0.53}\) & 71.23 \({}_{0.01}\) \\ Diverse Nearest, random & 74.10 \({}_{0.12}\) & 71.44 \({}_{0.93}\) \\ Diverse Nearest, score ranking & **74.99**\({}_{0.20}\) & 68.09 \({}_{0.60}\) \\ \hline \multicolumn{3}{l}{_Sample-level threshold filtering_} \\ Random & 72.41 \({}_{1.28}\) & **71.00 \({}_{0.32}\)** \\ Nearest & 72.28 \({}_{0.14}\) & 70.45 \({}_{0.46}\) \\ Diverse Nearest, random & 72.32 \({}_{0.08}\) & 70.06 \({}_{1.29}\) \\ Diverse Nearest, score ranking & **73.97**\({}_{0.12}\) & 68.95 \({}_{0.35}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison between SC and SV on CoNLL03 dataset. \(Th\_entity=4.0\) and \(Th\_sample=4.0\) is used. Right subscript number are standard deviations. Texts in **bold** are the best results in each category; Text underlined are the best results among all methods.

 



### Limitations

We acknowledge the following limitations of this study.

* This work focus on exploring the zero-shot self-improving framework on NER task. The investigation of this paradigm on other IE tasks are not studied yet.
* We explored the commonly-used self-consistency and the self-verification method to obtain the confidence score for measuring the quality of self-annotated data. There might be other approaches to measure the quality of self-annotation.
* The zero-shot performance still lag behind previous state-of-the-art of fully-supervised methods.
* Although this framework achieves significant improvement on the strong LLM, GPT-3.5, it gets negative results on a much weaker LLM, Llama2 13B. Improving the zero-shot NER on the weaker and smaller LLMs remains to be explored.

## Acknowledgements

This research is supported by Zhejiang Provincial Natural Science Foundation of China (LDT23F02023F02). We would like to thank the anonymous reviewers for their insightful comments and constructive suggestions. We would also like to thank Chen Wang and Xinlong Qiao for their help at the visualization.

## References

* Balsauriya et al. (2009) Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R Curran. 2009. Named entity recognition in wikipedia. In _Proceedings of the 2009 workshop on the people's web meets NLP: Collaboratively constructed semantic resources (People's Web)_, pages 10-18.
* Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodokumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reinier Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalamayan Sankaranarayana Pilali, Marie Pellat, Aitor Leukowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways.
* Han et al. (2023) Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. 2023. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. _arXiv preprint arXiv:2305.14450_.
* Josfroski et al. (2023) Martin Josfroski, Marija Sakota, Maxime Peyrard, and Robert West. 2023. Exploiting asymmetry for synthetic training data generation: Synthie and the case of information extraction.
* Li et al. (2023a) Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023a. Evaluating chatgpt's information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. _arXiv preprint arXiv:2304.11633_.
* Li et al. (2023b) Peng Li, Tianxiang Sun, Qiong Tang, Hang Yan, Yuanbin Wu, Xuanying Huang, and Xipeng Qiu. 2023b. CodelE: Large code generation models are better few-shot information extractors. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 15339-15353, Toronto, Canada. Association for Computational Linguistics.
* Li et al. (2023c) Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023c. Towards general text embeddings with multi-stage contrastive learning. _arXiv preprint arXiv:2308.03281_.
* Luan et al. (2019) Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishizzi. 2019. A general framework for information extraction using dynamic span graphs. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3036-3046, Minneapolis, Minnesota. Association for Computational Linguistics.
* Lyu et al. (2023) Xinxi Lyu, Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannaneh Hajisihrizi. 2023. Z-ICL: Zero-shot in-context learning with pseudo-demonstrations. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2304-2317, Toronto, Canada. Association for Computational Linguistics.
* Mengyu et al. (2023) Mingyu Derek Ma, Xiaoxuan Wang, Po-Nien Kung, Jeffrey Brantingham, Nanyun Peng, and Wei Wang. 2023. Star: Improving low-resource information extraction by structure-to-text data generation with large language models

* Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanesh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 11048-11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Ohta et al. (2002) Tomoko Ohta, Yuka Tateisi, Jin-Dong Kim, Hideki Mima, and Junichi Tsujii. 2002. The genia corpus: An annotated research abstract corpus in molecular biology domain. In _Proceedings of the human language technology conference_, pages 73-77. Citeseer.
* OpenAI (2022) OpenAI. 2022. Introducing chatgpt.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
* Sainz et al. (2023) Oscar Sainz, Iler Garcia-Ferrero, Rodrigo Aggerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2023. Gollie: Annotation guidelines improve zero-shot information-extraction.
* Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. _arXiv preprint cs/0306050_.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Walker et al. (2006) Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus ldc2006t06, 2006. _URL [https://catalog.ldc.upenn.edu/LDC2006T06_](https://catalog.ldc.upenn.edu/LDC2006T06_).
* Wang et al. (2023) Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models.
* Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. _arXiv preprint arXiv:2203.11171_.
* Wei et al. (2023a) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023a. Larger language models do in-context learning differently.
* Wei et al. (2023b) Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023b. Zero-shot information extraction via chatgpt.
* Xie et al. (2023) Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023. Empirical study of zero-shot ner with chatgptt.
* Zhang et al. (2023) Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. LImaaa: Making large language models as active annotators.
* Zhang et al. (2006) Suxiang Zhang, Ying Qin, Juan Wen, and Xiaojie Wang. 2006. Word segmentation and named entity recognition for SIGHAN bakeoff3. In _Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing_, pages 158-161, Sydney, Australia. Association for Computational Linguistics.
* Zhou et al. (2023) Wenzuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2023. UniversalNER: Targeted distillation from large language models for open named entity recognition.

 

Dataset Statistics

We evaluate on four commonly-used NER English datasets, CoNLL03 Sang and De Meulder (2003), ACE05 Walker et al. (2006), WikiGold Balasuriya et al. (2009) and GENIA Ohta et al. (2002), among which CoNLL03, WikiGold and GENIA are public datasets, and ACE055 can be accessed on Linguistic Data Consortium (LDC) platform with specific license. In addition, we also evaluate on two Chinese datasets, Ontonotes 46 and MSRA Zhang et al. (2006), in Appendix B. Table 5 and 6 shows the statistics of the processed datasets used in this work. For CoNLL03, we use the processed version shared by Han et al. (2023). For ACE05, we follow Luan et al. (2019)'s processing steps.

Footnote 5: [https://catalog.ldc.upenn.edu/LDC2006T06](https://catalog.ldc.upenn.edu/LDC2006T06)

Footnote 6: [https://catalog.ldc.upenn.edu/LDC2011T03](https://catalog.ldc.upenn.edu/LDC2011T03)

## Appendix C Results on Other Embedding Models

We explore the effect of using other embedding models for retrieval, SBERT Reimers and Gurevych (2019)7 and GTE Li et al. (2023c)8. Results are in Table 8.

Footnote 7: [https://huggingface.co/sentence-transformers/all-mgnet-base-v2](https://huggingface.co/sentence-transformers/all-mgnet-base-v2)

Footnote 8: [https://huggingface.co/thenlper/gte-large](https://huggingface.co/thenlper/gte-large)

## Appendix D Results on Various Number of Demonstrations

We investigate the performance on various number of demonstrations in the input context, the results are in Table 9. As shown in the table, the quantity of examples is not always proportional to the final performance. Similar findings have also been mentioned in Min et al. (2022). We hypothesize that after the LLM learns the mapping between the input-output examples, new information gained from more examples is marginal and might be offset by the more noise introduced.

## Appendix E More Results on Threshold Filtering

Table 10 shows the results of various values of entity-level and sample-level SC thresholds.

## Appendix F Upper Bound of Reliable Annotation Selection

Table 11 summarizes the complete results of the upper bound of reliable annotation selection.

## Appendix G Illustration of Iterative Self-improving

The bootstrapping process of iterative self-improving is shown in Fig. 5.

## Appendix H Case Study

We take a closer look at the cases where the errors in predictions are corrected with self-annotated demonstrations, as shown in Fig. 6. The proposed framework makes the model reuse its own knowledge and correct its own errors, forming a process of self-improving.

## Appendix I Prompts

We show the prompts use in this work in Table 12. We take samples from ACE05 for demonstrations.

\begin{table}
\begin{tabular}{l r c} \hline \hline
**Dataset** & **Ontonotes 4** & **MSRA** \\ \hline _#Train_ & 20025 & 46364 \\ _#Test_ & 4346 & 4365 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Statistics of the processed Chinese datasets used in this work. The training set is formed by combining the original training split and development split.

\begin{table}
\begin{tabular}{l r l r} \hline \hline
**Method** & **Ontonotes 4** & **MSRA** \\ \hline No-demos & 31.71 \({}_{1.14}\) & 39.21 \({}_{0.93}\) \\ \hline
**ICL with self-annotated demonstrations** & & \\ Random & 32.45 \({}_{0.19}\) & 39.55 \({}_{0.75}\) \\ Nearest & 31.54 \({}_{1.60}\) & 36.31 \({}_{1.76}\) \\ Diverse Nearest, SC ranking & **35.57 \({}_{1.22}\)** & **40.84 \({}_{2.83}\)** \\ \hline \multicolumn{3}{l}{**ICL with gold labeled demonstrations**} \\ Random (Gold) & 49.42 \({}_{0.22}\) & 53.51 \({}_{1.38}\) \\ Nearest (Gold) & 64.16 \({}_{1.08}\) & 61.58 \({}_{1.58}\) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Results on Chinese benchmarks. Right subscript numbers are standard deviations. _Gold_ indicates access to the gold labeled data, thus is not comparable with the rest of methods. Two-stage majority voting is used here. Texts in **bold** are the best results.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline
**Dataset** & **CoNLL03** & **ACE05** & **WikiGold** & **GENIA** \\ \hline _#Train_ & 14382 & 12475 & 1422 & 16692 \\ _#Test_ & 3453 & 2050 & 274 & 1854 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Statistics of the processed English datasets used in this work. The training set is formed by combining the original training split and development split.

 



\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Datasets** & \multicolumn{4}{c}{**CoNLL2003**} & \multicolumn{4}{c}{**WikiGold**} \\ \hline Embedding Models & embed-ada & SBERT & GTE & embed-ada & SBERT & GTE \\ \hline No-demos & 68.97 \({}_{0.22}\) & 68.97 \({}_{0.22}\) & 68.97 \({}_{0.22}\) & 70.80 & 70.80 & 70.80 \\ \hline \multicolumn{5}{c}{**ICL with self-annotated demonstrations (Zero-shot)**} \\ \hline Random & 72.12 \({}_{0.59}\) & 72.12 \({}_{0.59}\) & 72.12 \({}_{0.59}\) & 72.32 & 72.32 & 72.32 \\ Nearest & 71.66 \({}_{0.37}\) & 72.07 \({}_{0.22}\) & 72.37 \({}_{1.17}\) & 72.84 & 72.39 & 72.24 \\ Diverse Nearest, SC ranking & **74.51**\({}_{0.03}\) & **72.67**\({}_{0.37}\) & **72.53**\({}_{0.96}\) & **73.98** & **76.08** & **73.60** \\ \hline \multicolumn{5}{c}{**ICL with gold labeled demonstrations**} \\ \hline Random (Gold) & 77.25 \({}_{1.39}\) & 77.25 \({}_{1.39}\) & 77.25 \({}_{1.39}\) & 75.82 & 75.82 & 75.82 \\ Nearest (Gold) & 84.71 \({}_{0.39}\) & 83.28 \({}_{1.34}\) & 83.59 \({}_{0.09}\) & 79.40 & 78.18 & 79.03 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Results on various embedding models. Right subscript numbers are standard deviations. _embed-ada_ refers to text-embedding-ada. _Gold_ indicates access to the gold labeled data, thus is not comparable with the rest of methods. Two-stage majority voting is used here. Texts in **bold** are the best results.

Figure 5: The pipeline of iterative self-improving.





\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Method** & **CoNLL03** & **ACE05** & **WikiGold** & **GENIA** & **Avg** \\ \hline No-demos & 68.97 \({}_{0.22}\) & 27.29 \({}_{0.58}\) & 70.8 & 47.41 \({}_{0.29}\) & 53.62 \\ \hline _Entity-level SC threshold = 3.0_ & & & & \\ Random & 71.17 \({}_{0.13}\) & 30.16 \({}_{0.66}\) & 71.79 & 50.41 \({}_{0.00}\) & 55.88 \\ Nearest & 71.41 \({}_{0.66}\) & 31.58 \({}_{0.76}\) & 73.16 & 51.24 \({}_{1.79}\) & 56.85 \\ Diverse Nearest, random & 72.68 \({}_{1.31}\) & 31.39 \({}_{1.62}\) & 72.01 & 50.65 \({}_{0.11}\) & 56.68 \\ Diverse Nearest, SC ranking & 73.68 \({}_{0.03}\) & 31.86 \({}_{0.13}\) & 73.36 & 51.15 \({}_{0.69}\) & 57.51 \\ \hline _Entity-level SC threshold = 4.0_ & & & & \\ Random & 70.91 \({}_{0.55}\) & 30.41 \({}_{0.95}\) & 72.33 & 50.70 \({}_{1.53}\) & 56.09 \\ Nearest & 73.24 \({}_{0.53}\) & 32.22 \({}_{0.38}\) & 72.53 & 49.85 \({}_{1.20}\) & 56.96 \\ Diverse Nearest, random & 74.11 \({}_{0.12}\) & 32.29 \({}_{0.31}\) & 72.01 & 50.68 \({}_{0.14}\) & 57.27 \\ Diverse Nearest, SC ranking & 74.99 \({}_{0.20}\) & 31.65 \({}_{0.97}\) & 73.53 & 51.11 \({}_{0.28}\) & 57.82 \\ \hline _Entity-level SC threshold = 5.0_ & & & & \\ Random & 72.53 \({}_{0.07}\) & 29.44 \({}_{0.73}\) & 72.13 & 50.65 \({}_{0.57}\) & 56.18 \\ Nearest & 74.24 \({}_{0.03}\) & 29.65 \({}_{1.30}\) & 72.45 & 48.12 \({}_{0.45}\) & 56.11 \\ Diverse Nearest, random & 73.50 \({}_{0.14}\) & 30.55 \({}_{0.27}\) & 71.34 & 49.34 \({}_{0.27}\) & 56.18 \\ Diverse Nearest, SC ranking & 72.50 \({}_{0.66}\) & 30.14 \({}_{0.35}\) & 74.01 & 49.57 \({}_{0.61}\) & 56.55 \\ \hline _Sample-level SC threshold = 3.0_ & & & & \\ Random & 70.17 \({}_{0.00}\) & 28.78 \({}_{1.71}\) & 71.81 & 50.45 \({}_{0.34}\) & 55.30 \\ Nearest & 69.48 \({}_{0.90}\) & 30.39 \({}_{0.17}\) & 70.33 & 51.76 \({}_{0.29}\) & 55.49 \\ Diverse Nearest, random & 68.98 \({}_{0.86}\) & 30.04 \({}_{0.34}\) & 69.71 & 51.71 \({}_{1.41}\) & 55.11 \\ Diverse Nearest, SC ranking & 74.32 \({}_{1.37}\) & 30.73 \({}_{0.04}\) & 74.44 & 52.31 \({}_{0.34}\) & 57.95 \\ \hline _Sample-level SC threshold = 4.0_ & & & & \\ Random & 72.41 \({}_{1.28}\) & 30.05 \({}_{1.26}\) & 73.38 & 51.61 \({}_{1.21}\) & 56.86 \\ Nearest & 72.28 \({}_{0.14}\) & 32.00 \({}_{0.08}\) & 73.27 & 52.72 \({}_{0.80}\) & 57.57 \\ Diverse Nearest, random & 72.32 \({}_{0.08}\) & 30.74 \({}_{0.06}\) & 72.09 & 52.50 \({}_{0.50}\) & 56.91 \\ Diverse Nearest, SC ranking & 73.97 \({}_{0.12}\) & 31.08 \({}_{0.54}\) & 72.80 & 51.67 \({}_{0.93}\) & 57.38 \\ \hline _Sample-level SC threshold = 5.0_ & & & & \\ Random & 73.66 \({}_{0.69}\) & 29.19 \({}_{0.26}\) & 71.92 & 51.34 \({}_{0.97}\) & 56.52 \\ Nearest & 74.19 \({}_{0.30}\) & 30.94 \({}_{0.11}\) & 74.96 & 52.01 \({}_{0.23}\) & 58.02 \\ Diverse Nearest, random & 73.16 \({}_{0.66}\) & 27.98 \({}_{0.08}\) & 74.55 & 50.64 \({}_{0.18}\) & 56.58 \\ Diverse Nearest, SC ranking & 74.53 \({}_{0.51}\) & 30.00 \({}_{0.73}\) & 73.60 & 51.02 \({}_{0.98}\) & 57.28 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Results of various entity-level SC thresholds and sample-level SC thresholds. Right subscript numbers are standard deviations.

Figure 6: Case study of self-improving. Examples from WikiGold are illustrated. The errors in predictions of _No-demos_ are corrected by our framework with self-annotated demonstrations. Texts in green are entities corrected by our method. Texts in blue are entities in demonstrations that potentially help with the error correction.





\begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Method** & **CoNLL03** & **ACE05** & **WikiGold** & **GENIA** & **Avg** \\ \hline No-demos & 68.97 \({}_{0.22}\) & 27.29 \({}_{0.58}\) & 70.8 & 47.41 \({}_{0.29}\) & 53.62 \\ \hline \multicolumn{6}{l}{_Two-stage majority voting_} \\ Random & 72.12 \({}_{0.59}\) & 31.18 \({}_{0.38}\) & 72.32 & 50.17 \({}_{0.93}\) & 56.45 \\ Nearest & 71.66 \({}_{0.37}\) & 31.45 \({}_{1.32}\) & 72.84 & 50.19 \({}_{1.59}\) & 56.53 \\ Diverse Nearest, random & 72.45 \({}_{0.41}\) & 30.84 \({}_{(0.56}\) & 70.83 & 51.03 \({}_{0.73}\) & 56.28 \\ Diverse Nearest, SC ranking & 74.51 \({}_{0.03}\) & 32.27 \({}_{0.25}\) & 73.98 & 52.06 \({}_{0.09}\) & 58.20 \\ \hline \multicolumn{6}{l}{_Upper bound_} \\ Random & 73.72 \({}_{0.41}\) & 32.71 \({}_{0.56}\) & 73.83 & 52.67 \({}_{0.09}\) & 58.23 \\ Nearest & 81.65 \({}_{0.17}\) & 37.82 \({}_{0.59}\) & 76.57 & 56.24 \({}_{0.44}\) & 63.07 \\ Diverse Nearest, random & 78.84 \({}_{1.43}\) & 35.79 \({}_{0.26}\) & 76.20 & 54.46 \({}_{0.98}\) & 61.32 \\ Diverse Nearest, SC ranking & 80.12 \({}_{0.02}\) & 35.23 \({}_{0.63}\) & 76.64 & 54.58 \({}_{0.57}\) & 61.64 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Complete results of the upper bound of reliable annotation selection. Right subscript numbers are standard deviations.

\begin{table}
\begin{tabular}{l} \hline \hline Prompts of ICL \\ \hline Given entity label set: [’Person', ’Organization’, ‘Location’, ’Facility’, ’Weapon’, \\ ’Vehicle’, ’Geo-Political Entity’]. \\ Please recognize the named entities in the given text. Based on the given entity label set, \\ Viewlele’, ’Geo-Political Entity’]. \\ Please recognize the named entities in the given text. Based on the given entity label set, \\ Viewlele’, ’Geo-Political Entity’]. \\ Please recognize the named entities in the given text. Based on the given entity label set, \\ Viewlele’, ’Geo-Political Entity’]. \\ Please recognize the named entities in the given text. Based on the given entity label set, \\ provide answer in the following JSON format: [{’Entity Name’: ’Entity Label’}]. If there \\ In one entity in the text, return the following empty list: []. \\ Text: right now we ’re also waiting to hear from the president at the white house . \\ Text: right now we ’re also waiting to hear from the president at the white house . \\ Answer: [{‘white house’: ’Location’}, {’president’: ’Person’}] \\ Text: At the Pentagon , Barbara Starr reports officials say today begins a new strategy \\ in the skies over Baghdad . \\ Answer: [{’Barbara Starr’: ’Person’}, {’Pentagon’: ’ Facility’}, {’officials’: ’Person’}, \\ {’skies’:’Location’}, {’Baghadad’: ’Geo-Political Entity’}] \\ Text: John Irvine , ITV News , Baghdad . \\ Answer: [{’John Irvine’: ’Person’}, {’ITV News’: ’Organization’}, \\ {’Baghdad’: Geo-Political Entity’}] \\ ... ... \\ Text: right now we ’re also waiting to hear from the president at the white house . \\ Answer: \\ \hline \hline \end{tabular}
\end{table}
Table 12: Prompts used in this work. A few samples from ACE05 are displayed for demonstrations.