Venue: NAACL 2025

# PICLe: Pseudo-Annotations for In-Context Learning

in Low-Resource Named Entity Detection

Sepideh Mamooler, Syrielle Montariol, Alexander Mathis , Antoine Bosselut

EPFL

{firstname.lastname}@epfl.ch

Equal SupervisionOur code can be found at [https://github.com/sMamooler/PICle](https://github.com/sMamooler/PICle).

###### Abstract

In-context learning (ICL) enables Large Language Models (LLMs) to perform tasks using few demonstrations, facilitating task adaptation when labeled examples are hard to obtain. However, ICL is sensitive to the choice of demonstrations, and it remains unclear which demonstration attributes enable in-context generalization. In this work, we conduct a perturbation study of in-context demonstrations for low-resource Named Entity Detection (NED). Our surprising finding is that in-context demonstrations with partially correct annotated entity mentions can be as effective for task transfer as fully correct demonstrations.

Based off our findings, we propose **P**seudo-annotated **I**n-**C**ontext **L**earning (PICLe), a framework for in-context learning with noisy, pseudo-annotated demonstrations. PICLE leverages LLMs to annotate many demonstrations in a zero-shot first pass. We then cluster these synthetic demonstrations, sample specific sets of in-context demonstrations from each cluster, and predict entity mentions using each set independently. Finally, we use self-verification to select the final set of entity mentions. We evaluate PICLe on five biomedical NED datasets and show that, with zero human annotation, PICLe outperforms ICL in low-resource settings where limited gold examples can be used as in-context demonstrations.1

Footnote 1: Our code can be found at [https://github.com/sMamoloclr/PICLe](https://github.com/sMamoloclr/PICLe).

## 1 Introduction

With in-context learning (ICL), Large Language Models (LLMs) can be adapted to perform many tasks using few demonstrations Brown et al. (2020); Dong et al. (2022); Srivastava et al. (2023); Ye et al. (2023). This emergent property of LLMs is particularly beneficial in tasks where limited supervision data is available for fine-tuning models, such as in specialized domains where only expensive expert annotations can be relied upon to produce quality data (_e.g._, biomedical, clinical, and legal domains, among many others), and in situations where in-house proprietary datasets must be compiled with few available experts to perform the annotation.

Despite its promise in these settings, ICL is highly sensitive to the choice of the demonstrations Wang et al. (2024); Li and Qiu (2023); Liu et al. (2021), and it remains unclear which characteristics of demonstrations are critical for successful task adaptation. Consequently, prior work has explored which demonstration characteristics lead to successful task adaptation in ICL Min et al. (2022); Yoo et al. (2022); Wei et al. (2023), but these studies have focused on scalar-output tasks with a limited, predefined label space, such as classification. As a result, demonstration characteristics that optimize performance remain unclear for tasks that require structured, open-ended prediction such as Named Entity Detection (NED).

In NED, the goal is to identify all mentions of a specific type of entity within a given query, producing a structured output, with a label space effectively bounded only by the number of domain entities. In this work, we focus on NED given its high number of use cases Navarro et al. (2023); Skylaki et al. (2020); Ushio et al. (2022), particularly in specialized domains where effective annotation is challenging, as (1) it requires considerable domain expertise, and (2) entities can change over time, introducing distribution shifts in supervised datasets.

We conduct a thorough analysis of demonstration properties that impact in-context adaptation in NED. First, we analyze the importance of the context-label correspondence of in-context demonstrations by introducing noise through various perturbations that preserve different aspects of this mapping. Second, we investigate the effects of _partial correctness_ in demonstrations. In NED, partial

correctness refers to annotations that differ from the complete list of entity mentions in the input query but still share some overlap with it. To explore this, we apply various perturbation schemes to produce demonstrations with different levels of correctness. We find that while a semantic mapping between the demonstration context and label is essential, even weak semantic mappings can provide sufficient signals for in-context task transfer. Moreover, our analysis reveals that in-context learning is surprisingly resilient to partially correct annotations, provided that the demonstration labels include a large number of entities.

Following this analysis, we introduce **P**seudo-annotated **In**-**C**ontext **L**earning (PICLe), a framework for in-context NED with pseudo-annotated demonstrations that requires no human labeling effort. First, we exploit a pool of unlabeled samples to obtain pseudo-annotations through zero-shot prediction from LLMs, followed by a self-verification step in which the model is prompted to verify the type of individual entities. Then, the (noisy) pseudo-annotated samples are clustered, and demonstration sets are sampled from each cluster individually. These cluster-specific demonstrations are used to predict the entities mentioned in the test query. Predictions from all clusters are consolidated to obtain the final set of entity mentions.

In our evaluations with multiple LLMs across \(5\) biomedical entity detection datasets Taboureau et al. (2010); Li et al. (2016); Smith et al. (2008), we show that PICLe is as effective as, and on average outperforms, standard ICL that uses gold-labeled demonstrations.

Contributions.In summary,

1. we conduct a perturbation study to identify demonstration attributes that enable ICL in low-resource NED. We find that above a surprisingly low correctness threshold, partially correct annotations are as effective for ICL as demonstrations with fully correct gold annotations.
2. we propose PICLe, a novel framework for incontexl learning that uses pseudo-annotated demonstrations as in-context examples. We show that without human-annotation effort, PICLe competes and even outperforms ICL with gold-labeled demonstrations in resource-scar settings.

## 2 Related Work

What matters in in-context learning?In-context learning is remarkably effective for performing various NLP tasks with only a few task demonstrations appended to the prompt Brown et al. (2020). However, despite a large body of work on designing novel in-context learning methods Gao et al. (2021); Sorensen et al. (2022); Mishra et al. (2022), it is not yet fully understood what makes in-context learning effective, with multiple works demonstrating surprising variables, such as the impact of the demonstration order Lu et al. (2022), the term frequencies of test examples in pretraining data Razeghi et al. (2022), and basic output calibration Zhao et al. (2021); Fei et al. (2023); Jiang et al. (2023b). Consequently, recent works explore _how_ demonstration components might be separately responsible for in-context transfer. Min et al. (2022) show that in-context demonstrations serve to show the label space of demonstrations, the distribution of their input text, and their overall format. However, Yoo et al. (2022) perform quantifiable analysis on the impact of ground-truth label demonstrations on a larger set of tasks and datasets and find that ground-truth labels have substantial impacts on ICL performance. Wei et al. (2023) continue this line of work and show that the degree to which the label mapping influences task transfer depends on the scale of the model, and that smaller models are more capable of ignoring misaligned label mappings Wang et al. (2023a) show similar results for CoT reasoning, finding that CoT is also possible without valid demonstrations, and that demonstrations that are relevant to the query and have the correct order of reasoning steps are more important for effective transfer.

However, these works focus on classification tasks, which lack the concept of partial correctness; a label is either fully correct or entirely incorrect. In token-level tasks like NED, however, the list of annotated entities can be partially correct. We show that partially correct demonstrations can perform as effectively as fully correct ones--a result not addressed by these prior works. Furthermore, contrary to Min et al. (2022)'s findings for classification tasks, we show that ICL demonstrations with fully incorrect labels are not effective in NED.

Pseudo-annotation.Pseudo-annotation is a popular semi-supervised learning method in many domains Yang et al. (2022); Ye et al. (2024). It has recently been used for various NLP tasks to generate 

demonstrations for ICL Wan et al. (2023),b) and fine-tuning LLMs Huang et al. (2023); Honovich et al. (2023); Wang et al. (2023). Demonstrations' pseudo-annotations are either random (_e.g._ Z-ICL, Lyu et al. (2023), for classification tasks) or partially correct Wan et al. (2023); Chen et al. (2023). In particular, COSP Wan et al. (2023) selects and builds a demonstration pool from an LLM's zero-shot outputs via multiple rounds of prediction with high temperature and exceeds few-shot baselines for a range of reasoning tasks. Most similar to our work is Self-ICL Chen et al. (2023), which uses zero-shot models to generate in-context demonstrations for text classification. In our work, we construct a pipeline for leveraging zero-shot predicted labels for real test examples in named entity detection, but ground our pseudo-annotation method in analysis of how demonstration noise influences downstream in-context learning performance.

Information extraction with ICLAlthough LLMs have achieved SOTA performance in many tasks, their performance in information extraction is still significantly below supervised baselines Ma et al. (2023). Recent works have designed dedicated prompting techniques to improve in-context NER for LLMs Lee et al. (2022); Shen et al. (2023); Chen et al. (2023). Prompt-NER Shen et al. (2023) provides entity definitions to the model, and prompts it for a list of potential entities with an explanation justifying the compatibility of each entity with the provided definition. Their approach outperforms vanilla prompting, but still requires human effort to annotate gold demonstrations that may not be available in many application settings. In our work, we adopt a similar task formulation as Prompt-NER, but do not require labeled demonstrations or explanations, as we use pseudo-annotation to produce in-context learning examples.

## 3 Experimental setup

The task of Named Entity Detection (NED) requires detecting all mentions of entities in a text. We formulate the task such that the language model is given a passage of text as part of a prompt and must predict the list of entities that are mentioned in the passage. Optionally, in few-shot settings (_i.e._, in-context learning), the prompt also contains several demonstrations, each including an example passage and a corresponding list of mentioned entities in the passage.

Datasets.We consider five biomedical NED datasets with rich and comprehensive collections of diverse specialized entity types (Table 1). **ChemProt** contains annotations for extracting chemical compounds (drugs) and gene and protein-related objects Taboureau et al. (2010). Originally, each sample of this dataset is a paragraph, but we split these paragraphs into sentences. We construct two datasets from ChemProt: _ChemProt-Chem_ and _ChemProt-Gene_, for detecting chemicals and genes, respectively. **BC5CDR** contains biomedical abstracts annotated for chemical and disease extraction Li et al. (2016). Similar to ChemProt, we conduct our experiments on two sub-portions, _BCS-Chem_ and _BC5-Disease_. Finally, **BC2GM** contains biomedical abstracts annotated for the extraction of genes, proteins, and related entities Smith et al. (2008).

Models.We use three LLMs in our experiments: the proprietary GPT-3.5-Turbo model, and the open-source Mistral-7b-instruct Jiang et al. (2023) and Llama-2-7b-Chat Touvron et al. (2023) models. In the remainder of the paper, we refer to them as Mistral and Llama2, respectively.

\begin{table}
\begin{tabular}{l l l c r r r} \hline \hline
**Source** & **Name** & **Entity type(s)** & **\#Train** & **\#Test** & \multicolumn{1}{c}{\begin{tabular}{c} **Avg \# words** \\ **per entity** \\ \end{tabular} } & \multicolumn{1}{c}{
\begin{tabular}{c} **Ratio null** \\ **samples (\%)** \\ \end{tabular} } \\ \hline ChemProt & ChemProt-Chem & chemical & & & 1.39 & 41.3 \\ Tagboureau et al. (2010) & ChemProt-Gene & gene/protein & & & 1.62 & 45.0 \\ \hline BC5CDR & BC5-Chem & chemical & & & 1.36 & 35.3 \\ Li et al. (2016) & BC5-Disease & disease/illness & 4,560 & 4,797 & & 1.70 & 41.7 \\ \hline BC2GM Smith et al. (2008) & gene/protein & 12,575 & 5,039 & 2.45 & 48.9 \\ \hline \hline \end{tabular}
\end{table}
Table 1: **Datasets’ description and statistic**: number of samples (sentences) in train and test splits, average number of words per entity and null samples (samples with no labeled entities) ratio in train split. We use the versions available in the HuggingFace library.2 

Metrics.Using each dataset's original IOB2 (Inside-Outside-Beginning) annotation scheme, we compute the micro-averaged precision, recall, and F1 score to measure entity mention detection performance.3 We consider an entity as correctly detected only if the model extracts the exact span: if some tokens are added or missing compared to the gold span, it is marked incorrect.

Footnote 3: We use sequeval ([https://github.com/chakki-works/seqeval/](https://github.com/chakki-works/seqeval/)) a widely-used Python library for sequence labeling evaluation.

## 4 Do we need gold demonstrations?

In this section, we analyze which components of ICL demonstrations are critical for task transfer by studying the effect of fully incorrect (Section 4.1) and partially incorrect (Section 4.2) demonstrations in the in-context prompt. In all analyses, we use \(k\)NN demonstration retrieval Liu et al. (2022).4

Footnote 4: A comparison of different retrieval methods is provided in Appendix C, Figure 8.

### Input-output correspondence of in-context demonstrations

Prior research shows that correct demonstrations are not imperative for priming models in classification tasks Lyu et al. (2023), and that incorrect demonstrations are sufficient to show desired incontark transfer behavior, including domain relevance and annotation format.

In our analysis, we investigate essential demonstration attributes for successful in-context task transfer in NED by designing various corruption schemes, each targeting specific demonstration aspects (see Table 2 for examples). We compare performance under these corruptions to zero-shot prediction (**No Demo**) and standard ICL (**Gold Label**). We compare all settings using the same test samples and instructions prepended to the prompt.

#### Random ID Label:

We replace ground-truth entity labels with random in-distribution entities. For each input sentence, every entity in the ground-truth annotation is replaced by an in-distribution (ID) entity randomly sampled from all labels in training samples of the dataset.

##### Swapped ID Labels:

We swap entity labels in the ground-truth demonstrations with the entity labels of a randomly chosen sample in the training split. Contrary to **Random ID Label** where the _number_ of entities is preserved, the number of entities in each annotation changes compared to the original ground-truth.

#### Random 00d Label:

We replace entity labels in the ground-truth demonstrations with a random out-of-distribution (OOD) English word.5

Footnote 5: OOD words are randomly sampled from the English vocabulary in the NLTK library Bird et al. (2009).

#### Random 00d Label from Text:

We replace ground-truth entity labels with words randomly selected from the sample's text that are not included in the ground-truth annotation (_i.e._, not a target entity).

#### Corrupted 00d Text:

We replace the entity mentions in the text with random OOD English words.6

Footnote 6: OOD Text and Label:

Similar to **Corrupted 00D Text**, but we replace ground-truth labels as well, such that the entities in the text and label match.

#### Corrupted and Shuffled 00d Text, Corrupted and Shuffled 00d Text and Label:

Same as their non-shuffled counterpart, but with randomly shuffling the words of the sentence.

ResultsWhen looking at Mistral's performance averaged over all datasets, we find as expected that demonstrations with gold annotations consistently improve the performanc

\begin{table}
\begin{tabular}{l l} \hline \hline Text & This pretreatment had no effect on the inhibition of GABA-T or the elevation of brain GABA levels produced by VIG . \\ Gold Labels & [GABA, GABA, VIG] \\ \hline Random ID Labels & [quinoxalines, W13, N-acetylcysteine] \\ Swapped ID Labels & [AMG] \\ Randon OOD Labels (from nltk) & [unmelorated, suddy, vista] \\ Random OOD Labels from Text & brain, pretreatment, elevation] \\ \hline Corrupted 00D Text & This pretreatment had no effect on the inhibition of **ummeliorated** or the elevation of brain **suddy** levels produced by **vista**. \\ Corrupted and Shuffled 00D Text & of had by produced on elevation no . levels or effect the **vista** of the **mism ameliorated** pretreatment brain inhibition **suddy** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Examples of different text and labels corruption schemes. Source: ChemProt-Chem.

 

(Figure 1).6

Footnote 6: For detailed results per dataset, and similar performance of GPT-3.5-Turbo, see Appendix B (Figure 5).

However, corrupting demonstrations lowers the performance, particularly for **Random ID Label**, **Swapped ID Label** and **Random OOD Label**, which are notably worse than zero-shot prediction. This observation differs from the findings of Min et al. (2022) for in-context text classification and multiple choice QA, as well as Wang et al. (2023a)'s observations for QA with chain-of-thought reasoning, likely due to the open-endedness of the NED predictions (_i.e._, predicting multiple labels from a broad label space). In these three corruption schemes, the contextual and semantic correspondence between the input sentence and gold entities is broken. As a result, the model learns spurious text-label correspondences from these demonstrations, leading to worse performance than in the zero-shot setting. **Random OOD Label from Text** outperforms **Random OOD Label**, likely because it maintains a semantic correspondence between inputs and labels, but still underperforms zero-shot prediction due to misleading contextual associations.

Interestingly, both shuffled and unshuffled text corruption schemes (**Corrupted (and Shuffled) OOD Text**) exhibit no substantial performance drops, maintaining an edge over zero-shot prompting (despite the input prompt being the same for all corruptions in Table 2). We hypothesize that the model relies less on word order in the demonstrations to adapt to NED. Similar to how previous work showed that models no longer represent local word order in long contexts Sun et al. (2021), we infer that the model does not need to represent explicit word order in exemplars to use them for transfer for a non-shuffled test sample. Moreover, despite the label corruption in **Corrupted (and Shuffled) OOD Text and Labels**, the performance only slightly decreases compared to schemes with intact labels, still outperforming the zero-shot setting. This finding suggests that the model can induce entity presence from the global context, as ICL with these demonstrations still out-performs zero-shot predictions by up to \(10\%\).

Based off these findings, we conclude that for effective in-context task transfer in NED, the demonstrations must retain a degree of semantic correspondence between the input text and the extracted entities, but that the model's ability to adapt indextext is robust to noise in the demonstrations.

### Partially Correct ICL demonstrations

To further investigate the findings above, we perform a second study where we perturb demonstrations by modifying the context-label correspondence in a controlled manner. Specifically, we vary the correctness of the gold labels by applying different heuristic perturbations to the gold entity labels according to a perturbation factor \(p\in\{0.1,0.2,...0.9\}\):

Deletion: each entity in the ground-truth annotation is deleted with probability \(p\).

Substitution: each entity in the ground-truth annotation is substituted with a random distribution entity chosen from the dataset's label space with probability \(p\).

Figure 1: **10-shot ICL performance using various demonstration corruption schemes, with Mistral and \(k\)NN demonstration retrieval. We compare to zero-shot and 10-shot with gold demonstrations, averaging over all datasets.

 



**Addition and Substitution:** for each entity in the ground-truth, an entity chosen randomly from the dataset's label space is added to the annotation with probability \(p\); additionally, each ground-truth entity is substituted with a random entity from the same label space with probability \(p\).

**Deletion and Substitution:** each entity in the ground-truth is removed with probability \(p\). The remaining entities are substituted with a random entity from the dataset with probability \(p\).

Following these perturbations, we report the precision, recall, and F1 score of the perturbed demonstrations (evaluated based off the initial gold demonstration labels) against the F1 score of downstream predictions for test samples that contain at least one entity in their gold annotations.

ResultsDemonstrations subject to different perturbations may have similar demonstration F1 scores, but result in considerably different prediction F1 scores (Figure 2). Specifically, we note that for a fixed demonstration F1 score, the perturbed demonstrations that retain a higher number of entities in the demonstration achieve much greater performance (_i.e._, **Substitution** and **Addition and Substitution**). Even with heavily perturbed demonstration labels, the prediction F1 stays above zero-shot performance and even remains close to the performance of 10-shot ICL (with gold labels) so long as some of the gold entities remain in the demonstration labels. Based off our findings, we hypothesize that **demonstrations with noisy but partially correct labels** (such as those predicted by a zero-shot model) could benefit ICL for named entity detection.

Further results about the number of entities in the demonstration and perturbation factor, along with a comparison of the precision and recall of demonstrations against predictions, can be found in Appendix Figures 6 and 7. These analyses show that for a given perturbation factor, changes that preserve or increase the total number of entities in the demonstration (such as Substitution or Addition and Substitution) cause a less pronounced performance drop.

## 5 In-context NED with pseudo-annotated demonstrations

In this section, we propose PICLe, a framework for pseudo-annotating unlabeled samples that can be leveraged for in-context learning. This framework consists of two stages (Figure 3). In the first stage, we start with a set of unlabeled samples and prompt the model in a zero-shot pass to extract entities in each sample. Then, we improve the quality of these pseudo-annotations by prompting the model to verify each predicted entity (_i.e._, self-verification; Weng et al., 2023), and filter entities that are not of the correct entity type. We use \(k\)-means clustering to group the pseudo-annotated samples into \(K\) clusters based off the embedding of their text and pseudo-annotations.7 Each cluster is used as an individual pool of demonstrations for the downstream NED task. In the second stage, we prompt the model \(K\) times, each time choosing the demonstrations from one cluster of pseudo-annotated samples (a sampling method we refer to as Sp-\(k\)-means, _i.e._, Specialized \(k\)-means). Then, for each entity in the \(K\) lists of predictions, we perform a self-verification step to verify if the entity has the correct type or not, and retain the extracted entities that have the correct entity type. In all of our experiments, we pseudo-annotate \(1000\) samples from the training set of the datasets with greedy decoding.

Footnote 7: We embed the text and entities of samples using the S-PubMedBert-MS-MARC0 sentence transformer ([https://huggingface.co/pritamdeka/S-PubMedBert-MS-MARC0](https://huggingface.co/pritamdeka/S-PubMedBert-MS-MARC0)).

Figure 2: **10-shot ICL performance with perturbed demonstrations** with different perturbation schemes and using Mistral and \(k\)NN demonstration retrieval. We report the prediction F1 as a function of the precision, recall, and F1 of the perturbed demonstration label sets (relative to the gold demonstrations) averaged over all datasets. The size of the points shows the average number of entities in the label sets of the perturbed demonstrations.



PICLe performanceWe evaluate PICLe on the same biomedical NED datasets used for our analysis in Section 4 and compare PICLe's performance with standard ICL using gold demonstrations sampled from different demonstration pool sizes, representing various degrees of annotation scarcity. For baselines that use gold annotations as in-context examples, we initially sample demonstration pools of size \(N\) from the full training set of each dataset, which range in size from \(4.5\text{K}\) to \(12.5\text{K}\) examples (Table 1). In scarce annotation settings, we then sample demonstrations from these pools for gold in-context learning using \(k\text{NN}\) (following ablation study in Figure 8 in Appendix). We experiment with \(N\in[10,50,100]\), reporting results for a diverse set of annotation budgets. We repeat all experiments with \(5\) seeds and report the average performance and standard deviation for these runs.

Our results in Figure 4 show that across most datasets (with the exception of _ChemProt-Chem_), PICLe significantly outperforms the zero-shot baseline by an average of \(10.7\%\) (\(57.1\%\) compared to \(46.4\%\)). Furthermore, PICLe also matches or outperforms in-context learning with gold demonstrations in resource-scarce settings, even beating an in-context learning baseline that has access to \(100\) human-annotated demonstrations (\(57.1\%\) vs. \(52.8\%\)). We note that the dataset with the highest performance, BC5chem (\(77.7\%\) average F\(1\) score for PICLe), contains entity annotations whose surface forms generally contain fewer tokens (see Table 1). On the contrary, the dataset with the lowest performance, BC2GM (\(50\%~{}\mathrm{F}1\) for PICLe), has entity annotations that contain longer surface forms, making it more difficult to match the exact span in a generative manner.

We also compare PICLe with a supervised baseline, fine-tuning a domain-specific language encoder, BiomedNLP-BiomedBERT-large, on various numbers of gold annotations (see Table 9 in Appendix). While the performance of fine-tuning on \(10\) gold samples is low and shows high variance between datasets, the performance with \(50\) gold samples already outperforms all LLM baselines. However, we note that the sequence labeling formulation of the task for the supervised baseline differs from the generative formulation for LLMs, providing the supervised baseline with a simpler

Figure 4: **Performance of PICLe, zero-shot, and 10-shot ICL with gold demonstrations selected from \(10,50,100\) gold examples using Mistral. The error bars show the variance across \(5\) seeds for sampling subsets of gold examples. All methods are followed by self-verification.**

Figure 3: **PICLe pipeline. Unlabeled samples are pseudo-annotated through a zero-shot prediction and self-verification pass. Subsequently, they are clustered, and cluster-specific sets of ICL demonstrations are chosen at random from each group. Each set is independently used to find entity mentions in the query, and the final set of entity mentions is obtained by aggregating these independent sets and asking the model to verify the type of each predicted entity.**



format for predicting entity spans, more adapted to our strict exact match evaluation.

Ablation studyIn Table 3, we ablate each step of the PICLe pipeline to evaluate the importance of each component (see Appendix Table 7 for a detailed version including precision and recall). For pseudo-annotation, similarly to Wan et al. [2023], we compare our method with running zero-shot prediction \(10\) times with non-zero temperature (\(T=0.7\)) and filtering the \(10\) sets of extracted entities using self-verification or merging (_i.e._, prompting the LLM to aggregate the entity lists). Both lead to a slightly lower F1 score (rows \(\#1\) and \(\#2\)) while being much more computationally expensive than a single round of zero-shot prediction. We also find that self-verification helps with improving pseudo-annotations (row \(\#3\) vs. row \(\#8\)). Additional results demonstrating the effectiveness of self-verification in the pseudo-annotation and inference steps are presented in Tables 5 and 6.

We also compare the effect of different demonstration retrieval methods: random sampling, \(k\)NN, vanilla \(k\)-means, and specialized \(k\)-means (Sp-\(k\)-means). \(k\)NN (\(k=10\)), known for being sensitive to noisy demonstrations Zhang et al. [2022], scores the lowest (row \(\#4\)). For random retrieval, we sample demonstrations using \(5\) different seeds; the predicted entity lists are merged and post-processed using self-verification (row \(\#5\)). Similarly, for \(k\)-means (row \(\#6\)), we randomly sample one demonstration per cluster, increasing the intra-run diversity. Conversely, in Sp-\(k\)-means, demonstrations in each round are all sampled from the same cluster, maximizing inter-run diversity. We sample demonstrations using \(5\) different seeds, leading to \(5\) inference runs. The predicted entity lists are merged and self-verified. The diversity of demonstrations for \(k\)-means leads to a higher recall than random (\(48.6\) vs. \(40\%\)), but not as high as Sp-\(k\)-means (\(53.5\%\)), which benefits from having separate clusters that lead to more varied predictions. Self-verification improves performance during inference (rows \(\#7\) vs. \(\#8\)), especially precision (\(+20\%\)).

Finally, to validate the importance of pseudo-annotations on the downstream performance, we run zero-shot inference \(10\) times with temperature \(0.8\), and pool all predictions before applying self-verification (rows \(\#9\)-\(\#11\) of Table 3). This baseline improves compared to vanilla zero-shot followed by self-verification, but underperforms PICLe, showing that pseudo-annotations not only provide _seeds_ for diversity but also promote task transfer.

Model GeneralizationWe measure the performance of PICLe when using different models to generate pseudo-annotations: Llama2 and GPT-3.5-Turbo. All experiments with gold samples are performed by randomly sampling demonstrations with \(5\) seeds and averaging the results. On average over all datasets, Llama2 exhibits the same behavior as Mistral, outperforming zero-shot and 10-shot with scarce gold demonstrations (Table 4, and Table 8 in Appendix). We note that for Mis

\begin{table}
\begin{tabular}{l c r} \hline \hline
**Demonstration retrieval** & **Inference** & **F1** \\ \hline Llama2 + PICLe & & 51.9 \\ Zero-shot & Llama2 & 48.3 \\
10 gold samples + \(k\)NN & & 45.5 \\ \hline Mistral + PICLe & & 57.1 \\ Zero-shot & & 46.4 \\
10 gold samples & Mistral & 42.6 \\
100 gold samples + \(k\)NN & & 52.8 \\ Full train set + \(k\)NN (oracle) & & 63.2 \\ \hline GPT-3.5-Turbo + PICLe & Mistral & 56.5 \\ \hline \hline \end{tabular}
\end{table}
Table 4: **Performance of PICLe using different LLMs for pseudo-annotation and prediction**, compared with zero-shot and \(10\)-shot ICL with gold annotations, averaged over all \(5\) datasets.

\begin{table}
\begin{tabular}{c l l l l l} \hline \hline  & \multicolumn{2}{c}{**Pseudo-annotation**} & \multicolumn{2}{c}{**Inference**} & \multirow{2}{*}{**F1**} \\ \cline{2-2} \cline{4-6}  & **Runs** & \multicolumn{1}{r}{**Post-processing**} & \multicolumn{1}{r}{**Demo retrieval**} & \multirow{2}{*}{**SV**} \\ \cline{1-1}  & & & **precessing** & & \\ \hline
1 & 10 & merging & & ✓ & 55.7 \\
2 & 10 & SV & Sp-\(k\)-means & ✓ & 55.1 \\
3 & 1 & none & & ✓ & 51.8 \\ \hline
4 & 1 & & \(k\)NN & ✓ & 42.7 \\
5 & 1 & SV & random & ✓ & 47.9 \\
6 & 1 & & \(k\)-means & ✓ & 55.1 \\ \hline
7 & 1 & \multirow{2}{*}{SV} & Sp-\(k\)-means & \(\times\) & 49.2 \\
8 & 1 & & & ✓ & **57.1** \\ \hline
9 & & & \multicolumn{1}{c}{Zero-shot} & \(\times\) & 44.6 \\
10 & NA & NA & Zero-shot & ✓ & 46.4 \\
11 & & & 10 Zero-shot & ✓ & 50.6 \\ \hline \hline \end{tabular}
\end{table}
Table 3: **Ablation of each component of PICLe**, averaged over all \(5\) datasets, using Mistral. SV refers to the use of self-verification and \(10\) Zero-shot refers to \(10\) zero-shot passes with a non-zero temperature.

 

tral, PICLe outperforms gold demonstrations in all resource-scarce settings (_i.e._, \(N=100\)). However, the performance is lower than gold demonstrations when we can sample demonstrations from the full training set (_i.e._, Full train set + \(k\)NN). We consider this performance to be an _oracle_ ICL setting given the large number of available training examples, which would not be realistic for low-resource settings, and might question the necessity of using ICL compared to fine-tuning a NED model.

Given the cost to use GPT-3.5-Turbo, we only use it for the pseudo-labeling step of PICLe. Since the pseudo-annotated samples are generated from the training set of each dataset, we measure the F1 score of samples pseudo-annotated by each model: Mistral's score is \(45.8\%\), Llama2 reaches \(48\%\), while GPT-3.5-Turbo achieves \(62.7\%\). Consequently, we experiment with using the pseudo-annotations from GPT-3.5-Turbo as a demonstration pool for performing inference with Mistral, whose inference cost is lower. Using a higher-quality demonstration pool results in higher precision (\(65.2\%\) vs. \(61.8\%\)) but comparable F1 score on average. This result is in line with our findings in Section 4.2, where the prediction F1 score curve becomes more flat with high values of demonstration F1 score (Figure 2). More specifically, our inference method Sp-\(k\)-means succeeds in leveraging the noisy demonstrations generated by Mistral, and slight improvements in the demonstration F1 score from using GPT-3.5-Turbo demonstrations do not improve results.

## 6 Conclusion

In this work, we study the demonstration attributes that enable in-context generalization for named entity detection. We find that the context-label semantic correspondence is crucial for effective inconstext NED, and without this correspondence, incorrect examples hurt performance, pushing it below zero-shot NED. However, our analysis demonstrates that partially correct demonstration label sets are just as effective as gold label sets, provided a sufficient number of correct label mappings are found in the demonstration. Based off these findings, we design an ICL framework, PICLe, for named entity detection that leverages LLMs to produce pseudo-annotated examples that can be used for in-context transfer. Our results on five biomedical NED datasets demonstrate that PICLe is more effective than zero-shot prediction and outperforms in-context learning with gold demonstrations when gold demonstrations are scarce.

## 7 Limitations

Single Task.This work introduces a method to alleviate annotation effort for named entity detection (NED) while achieving comparable performance to few-shot NED with human-labeled annotations. While this pipeline could be generalized to other tasks besides NED, the experiments presented in this paper are limited to this particular task. However, we demonstrate its effectiveness over a broad set of entity types. Similarly, further work is needed to generalize our conclusions on the partial correctness of demonstrations to all structured output tasks.

Sensitive applications.We apply our system to documents from the biomedical domain. The evaluation sets are drawn from abstracts from published articles. However, the tools we develop can be used to extract the same type of entities in sensitive documents. Our tools were not tested for these applications, and practitioners should be aware that performance on such different types of documents is not guaranteed to transfer.

Annotation bias.Annotated data can contain various forms of annotation bias, which lead trained models to make biased predictions when labeling entities based on the knowledge and beliefs of the annotators. This bias is usually alleviated following common annotation practices such as computing inter-rater agreement and having detailed annotation guidelines discussed with the annotators. However PICLe only uses models' pseudo-annotations, since we focus on domains for which expert annotation is challenging to obtain. Consequently, given the lack of interpretability and training data openness of the used LLMs, we cannot assess the reliability and fairness of the demonstrations.

## Acknowledgments

We thank Said Gurbuz for his work on the finetuning experiment (BioMedBERT) in low-resource domains, Anna Sotnikova and Beatriz Borges for providing comments on earlier drafts of this paper, and Angelika Romanou for the illustration in Figure 3. This project was supported by the EPFL Center for Imaging through a Collaborative Imaging Grant and a Boehringer Ingelheim Fonds PhD stipend (S.M.). We also gratefully acknowledge the 

support of Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29), Sony Group Corporation, and the Allen Institute for AI.

## References

* S. Bird, E. Klein, and E. Loper (2009)Natural language processing with python: analyzing text with the natural language toolkit.. Cited by: SS4.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1674-469X, Link, Document Cited by: SS1.
* J. Chen, Y. Lu, H. Lin, J. Lou, W. Jia, D. Dai, H. Wu, B. Cao, X. Han, and L. Sun (2023)Learning in-context learning for named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 13661-13675. External Links: ISBN 978-3-319-1422-3, Link, Document Cited by: SS1.
* W. Chen, C. Wu, Y. Chen, and H. Chen (2023)Self-ICL: zero-shot context learning with self-generated demonstrations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 15651-15662. External Links: ISBN 978-3-319-1422-3, Link, Document Cited by: SS1.
* Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui (2022)A survey for in-context learning. arXivabs/2301.00234. External Links: Link, 2201.00234 Cited by: SS1.
* Yu Fei, Y. Hou, Z. Chen, and A. Bosselt (2023)Mitigating label biases for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 14014-14031. External Links: Link, Document Cited by: SS1.
* T. Gao, A. Fisch, and D. Chen (2021)Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 3816-3830. External Links: Link, Document Cited by: SS1.
* O. Honovich, T. Scialom, O. Levy, and T. Schick (2023)Unnatural instructions: tuning language models with (almost) no human labor. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 14409-14428. External Links: Link, Document Cited by: SS1.
* J. Huang, S. Gu, L. Hou, Y. Wu, X. Wang, H. Yu, and J. Han (2023)Large language models can self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 1051-1068. External Links: Link, Document Cited by: SS1.
* A. Qiaochu Jiang, A. Sablayrolles, K. Mensch, C. Bamford, D. Singh Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. Renard Lavaud, M. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed (2023)MATRAT 7b. ArXivabs/2310.06825. External Links: Link, 2310.06825 Cited by: SS1.
* Z. Jiang, Y. Zhang, C. Liu, J. Zhao, and K. Liu (2023)Generative calibration for in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 2312-2333. External Links: Link, Document Cited by: SS1.
* D. Lee, A. Kadakia, K. Tan, M. Agarwal, X. Feng, T. Shibuya, R. Mitani, T. Sekiya, J. Pujara, and X. Ren (2022)Good examples make a faster learner: simple demonstration-based learning for low-resource NER. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 2687-2700. External Links: Link, Document Cited by: SS1.
* J. Li, Y. Sun, R. J. Johnson, D. Sciaky, C. Wei, R. Leaman, A. Peter Davis, C. J. Mattingly, T. C. Wiegers, and Z. Lu (2016)Biocreative V CDR task corpus: a resource for chemical disease relation extraction. Database: The Journal of Biological Databases and Curation. Cited by: SS1.
* X. Li and X. Qiu (2023)MoT: memory-of-thought enables chatgPT to self-improve. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, pp. 6354-6374. External Links: Link, Document Cited by: SS1.
* J. Liu, D. Shen, Y. Zhang, B. Dolan, Lawrence Carin, and Weizhu Chen (2021)What makes good in-context examples for gpt-3?. In Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Doep Learning Inside Out, External Links: Link, Document Cited by: SS1.
* Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp (2022)Fantastically ordered 

prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 8086-8098. External Links: Link Cited by: SS2.
* X. Lyu, S. Min, I. Beltagy, L. Zettlemoyer, and H. Hajishirzi (2023)Z-ICL: zero-shot in-context learning with pseudo-demonstrations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 2304-2317. External Links: Link Cited by: SS2.
* Y. Ma, Y. Cao, Y. Hong, and A. Sun (2023)Large language model is not a good few-shot information extractor, but a good reranker for hard samples!. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 10572-10601. External Links: Link Cited by: SS2.
* S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer (2022)Rethinking the role of demonstrations: what makes in-context learning work?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, J. Dy Dykstra, K. D'Amica, T. E. Moore, R. Gopalakrishnan, and P. Dollar (Eds.), Vol. , pp. 11048-11064. External Links: Link Cited by: SS2.
* S. Mishra, D. Khashabi, C. Baral, Y. Choi, and H. Hajishirzi (2022)Reframing instructional prompts to GPTK's language. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, pp. 589-612. External Links: Link Cited by: SS2.
* D. F. Navarro, K. Ijaz, D. Rezzaadegan, H. Rahimi-Ardabili, M. Dras, E. Coiera, and S. Berkovsky (2023)Clinical named entity recognition and relation extraction using natural language processing of medical free text: a systematic review. International Journal of Medical Informatics177, pp. 105122. External Links: Link Cited by: SS2.
* Y. Razeghi, R. L. IV, M. Gardner, and S. Singh (2022)Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, pp. 840-854. External Links: Link Cited by: SS2.
* Y. Shen, Z. Tan, S. Wu, W. Zhang, R. Zhang, Y. Xi, W. Lu, and Y. Zhuang (2023)PromptNER: prompt locating and typing for named entity recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 12492-12507. External Links: Link Cited by: SS2.
* S. Skylaki, A. Oskooei, O. Bari, N. Herger, and Z. Kriegman (2020)Named entity recognition in the legal domain using a pointer generator network. arXiv preprint arXiv:2012.09936. Cited by: SS2.
* S2. External Links: Document, ISSN 0001-47X1, Link Cited by: SS2.
* T. Sorensen, J. Robinson, C. Rytting, A. Shaw, K. Rogers, A. Delorey, M. Khalil, N. Fulda, and D. Wingate (2022)An information-theoretic approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 819-862. External Links: Link Cited by: SS2.
* A. Srivastava, A. Rastogi, A. Rao, A. V. Moeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al. (2023)Beyond the imitation game: quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. Cited by: SS2.
* S. Sun, K. Krishna, A. Mattarella-Micke, and M. Ilyyer (2021)Do long-range language models actually use long-range context?. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 807-822. External Links: Link Cited by: SS2.
* D372. External Links: Link Cited by: SS2.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023)LIMA 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: SS2.
* A. Ushio, F. Barbieri, V. Sousa, L. Neves, and J. Camacho-Collados (2022)Named entity recognition in Twitter: a dataset and analysis on short-term temporal shifts. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th 

International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 309-319, Online only. Association for Computational Linguistics.
* Wan et al. [2023a] Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Arik, and Tomas Pfister. 2023a. Better zero-shot reasoning with self-adaptive prompting. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 3493-3514, Toronto, Canada. Association for Computational Linguistics.
* Wan et al. [2023b] Xingchen Wan, Ruoxi Sun, Hootan Nakhost, Hanjun Dai, Julian Eisenschlos, Sercan Arik, and Tomas Pfister. 2023b. Universal self-adaptive prompting. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 7437-7462, Singapore. Association for Computational Linguistics.
* Wang et al. [2023a] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023a. Towards understanding chain-of-thought prompting: An empirical study of what matters. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2717-2739, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. [2024] Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2024. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. _Advances in Neural Information Processing Systems_, 36.
* Wang et al. [2023b] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b. Self-instruct: Aligning language models with self-generated instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* Wei et al. [2023] Jerry W. Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. 2023. Larger language models do in-context learning differently. _ArXiv_, abs/2303.03846.
* Weng et al. [2023] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. Large language models are better reasoners with self-verification. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 2550-2575, Singapore. Association for Computational Linguistics.
* Yang et al. [2022] Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. 2022. A survey on deep semi-supervised learning. _IEEE Transactions on Knowledge and Data Engineering_, 35(9):8934-8954.
* Ye et al. [2024] Shaokai Ye, Anastasia Filippova, Jessy Lauer, Steffen Schneider, Maxime Vidal, Tian Qiu, Alexander Mathis, and Mackenzie Weygandt Mathis. 2024. Superanimal pretrained pose estimation models for behavioral analysis. _Nature Communications_, 15(1):5165.
* Ye et al. [2023] Shaokai Ye, Jessy Lauer, Mu Zhou, Alexander Mathis, and Mackenzie W Mathis. 2023. Amadeusgpt: a natural language interface for interactive animal behavioral analysis. In _Thirty-seventth Conference on Neural Information Processing Systems_.
* Yoo et al. [2022] Kang Min Yoo, Junyeob Kim, Hyuhng Joon Kim, Hyungsoo Cho, Huyiyeol Jo, Sang-Woo Lee, Sang-goo Lee, and Taeuk Kim. 2022. Ground-truth labels matter: A deeper look into input-label demonstrations. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 2422-2437, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Zhang et al. [2022] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. In _The Eleventh International Conference on Learning Representations_.
* Zhao et al. [2021] Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. _International Conference on Machine Learning_ 

Reproducibility statement

Code.We plan to share the code for PICLe and all of our experiments. The decoding temperature is specified for each experiment in their corresponding section. For experiments with non-zero temperature, we use top_p=1, and max_tokens=\(512\) in all experiments. All models were run on a single NVIDIA A100 GPU with 80 GB Memory, each inference run taking between 5 and 20 minutes depending on the dataset.

Data.The datasets we use are publicly available on the Huggingface platform.8

Footnote 8: [https://huggingface.co/datasets/bigbio/9](https://huggingface.co/datasets/bigbio/9)

Models.As described in Section 3, we use two open-source models for our studies whose checkpoints can be found in the Huggingface model library: Mistral-7b-instruct9 and Llama-2-7b-Chat.10 We also conduct experiments using a proprietary LLM from OpenAI, gpt-3.5-turbo-0125,11 which unfortunately is subject to be updated (or removed from the API entirely) at any moment, limiting the long-term reproducibility of the results obtained with this tool. For supervised fine-tuning, we use the text encoder BiomedNLP-BiomedBERT-large.12

Footnote 9: [https://huggingface.co/mistralai/Mistral-7b-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7b-Instruct-v0.1)

Random seedsWe repeat all of our experiments that involve randomization 5 times with the following seeds: 12345, 24690, 37035, 49380, 61725.

PromptsExamples of the prompts used for the self-verification pass and NED are shown in Figures 9 and 10 respectively.

## Appendix B Additional analysis for ICL demonstration

### Corrupted random demonstrations

Figure 5 shows results per dataset for all corruption schemes with Mistral and GPT-3.5-Turbo. We observe a similar trend for both models.

### Partially correct demonstrations

Figure 6 shows the evolution of the downstream F1 score depending on the number of entities in the demonstrations and the perturbation factor. As expected, an increased perturbation factor leads to a lower demonstration F1 and a lower downstream F1 (right side of the figure). Similarly, adding or removing entities in the demonstration labels leads to a lower downstream F1. However, with the same perturbation factor, perturbations that do not decrease the number of entities in the demonstration (Substitution and Addition and Substitution) lead to a much softer rate of performance loss. Similarly, to reach the same downstream performance as zero-shot (around \(0.5\) on average), removing one entity is enough, while at least two entities need to be added. This result supports the hypothesis that a way to increase downstream performance is to give preference to a higher recall and number of entities in the demonstration set. Figure 7 compares the precision and recall of demonstrations against the precision and recall of predictions.

## Appendix C Additional results for PICLe

Ablation studyWe ablate each component of PICLe's pseudo-annotation and inference steps in Table 7. These results show the importance of the self-verification step in pseudo-annotation (row \(\#3\) vs \(\#8\)), and inference (row \(\#7\) vs \(\#8\)) for the downstream precision. Additionally, we observe that our proposed Sp-\(k\)-means leads to a higher recall and F1 compared to other demonstration sampling methods (rows \(\#4\)-\(6\) vs \(\#8\)).

Results with different modelsTable 8 evaluates PICLe with different base language models for pseudo-annotation and inference. We observe that for both Llama2 and Mistral, PICLe obtains higher recall and F1 than zero-shot and almost all 10-shot with gold demonstration methods.

Impact of demonstration retrievalHere, we compare random, \(k\)-means, and \(k\)NN demonstration retrieval methods for gold demonstrations with and without the self-verification step. Figure 8 shows that demonstration retrieval performance between methods varies across datasets. However, \(k\)NN outperforms the other two methods on three out of five datasets.

Impact of self-verificationWe evaluate PICLe's pseudo-annotations before and after self-verification and report the results per dataset in Table 5. We also report the performance of zero-shot NED before and after the self-verification pass 

in Table 6. It is observed that self-verification reduces the recall and significantly increases the precision, and on average leads to a higher F1 score, thereby improving the overall pseudo-annotation and prediction performance.

Fine-tuning resultsWe report the performance of fine-tuned BioMedBERT in Table 9. As expected with 10 gold samples fine-tuning results in low performance and high variance. However, the performance with 50 samples already outperforms all in-context learning baselines.

Efficiency analysisCompared to the \(10\)-shot baseline with gold demonstrations, PICLe has a zero-shot pass followed by the self-verification step for pseudo-annotation, \(10\) additional rounds in the Sp-\(k\)-means component, and one final self-verification step. This is while PICLe requires no human annotation which is crucial in low-resource domains.

Figure 5: **10-shot ICL performance using various demonstration corruption schemes, compared with zero-shot and ICL with gold annotations, for each dataset. Experiments performed using Mistral (top) and GPT-3.5-Turbo (bottom) and \(k\)NN demonstration retrieval. (Best viewed in color.)**



Figure 6: **10-shot ICL performance with partially correct demonstrations with different perturbation schemes using Mistral and \(k\)NN demonstration retrieval. We observe the impact on prediction F1 score of the perturbation factor and the number of entities in the demonstrations for different perturbation types, averaged over all datasets. The size of the points shows the demonstrations’ F1 score.**

Figure 7: **10-shot ICL performance with partially correct demonstrations with different perturbation schemes using Mistral and \(k\)NN demonstration retrieval. We observe the impact of the demonstration precision and recall on the downstream prediction precision and recall, respectively, averaged over all datasets. The size of the points shows the demonstrations’ number of entities in the annotations.**





\begin{table}
\begin{tabular}{c c l l r r r r} \hline \hline  & \multicolumn{2}{c}{**Pseudo-annotation**} & \multicolumn{2}{c}{**Inference**} & \multirow{2}{*}{**Precision**} & \multicolumn{1}{r}{**Recall**} & \multicolumn{1}{r}{**F1**} \\ \cline{3-8}  & **Runs** & & & **Post-processing** & **Demo retrieval** & **Self-verification** & & \\ \hline
1 & 10 & LLM-merging & Sp-\(k\)-means & ✓ & 56.7 & 55.1 & 55.7 \\
2 & 10 & self-verify & Sp-\(k\)-means & ✓ & 56.3 & 54.2 & 55.1 \\
3 & 1 & none & Sp-\(k\)-means & ✓ & 55.2 & 49.4 & 51.8 \\ \hline
4 & 1 & self-verif & \(k\)NN & ✓ & 72.5 & 32.8 & 42.7 \\
5 & 1 & self-verif & random & ✓ & 68.1 & 39.8 & 47.9 \\
6 & 1 & self-verif & \(k\)-means & ✓ & 64.8 & 48.6 & 55.1 \\ \hline
7 & 1 & self-verif & Sp-\(k\)-means & \(\times\) & 41.8 & 60.7 & 49.2 \\
8 & 1 & self-verif & Sp-\(k\)-means & ✓ & 61.8 & 53.5 & **57.1** \\ \hline \hline \end{tabular}
\end{table}
Table 7: **Ablation of each component of PICLe, averaged over all datasets, using Mistral for pseudo-annotation and inference.**

Figure 8: **10-shot NED with gold annotations with Mistral using different demonstration retrieval methods: \(k\)-means, \(k\)NN, and random.**



**user**: Given the context and definition of chemical entity, answer the following question. Please reason about your answer and include YES or NO in your response. YES if the given phrase is a chemical entity, and NO if it is not. If you are not sure, you can say I don't know.

Context: p75NTR expression in rat urinary bladder sensory neurons and spinal cord with cyclophosphamide-induced cystitiis.

Chemical definition: Chemical refers to any substance having a distinct molecular composition that is produced by or used in a chemical process. Chemicals can be elements or compounds, and they can exist in various forms--solid, liquid, or gas.

Based off this context and definition, does cyclophosphamide correspond to the name of a chemical entity?

Figure 9: Example of prompt used for **self-verification**. Dataset: BC5-Chem.





**user**: Your task is to extract all of the chemicals mentioned in a given abstract published in PubMed. Chemical refers to any substance having a distinct molecular composition that is produced by or used in a chemical process. Chemicals can be elements or compounds, and they can exist in various forms--solid, liquid, or gas. Please extract all of the entities corresponding to chemicals from the following paragraph, the same way as they are marked in the examples. Make sure to include all and only the chemicals mentioned in the text, but not the genes and proteins, and diseases. If there are no chemical entities in the text output 'None'. You will be penalized if you include an entity more or less than the number of times it appears in the text.

## Here are some examples:

- 801 binding in cortex. Chemical entities:

**assistant**:

H | yes | because it is a chemical

MK - 801 | yes | because it is a chemical

**user**: Input: The normalized reflex amplitude was significantly higher during pain, but only at faster stretches in the painful muscle. Chemical entities:

**assistant**:

...

* HT ) 2A receptors ( Ki = 180 nM ) and rat alpha1 adrenoreceptor ( Ki = 237 nM ). Chemical entities:

**assistant**:

NRA0160 | yes | because it is a chemical

dopamine | yes | because it is a chemical

serotonin | yes | because it is a chemical

5 - HT | yes | because it is a chemical

**user**: Input: p75NTR expression in rat urinary bladder sensory neurons and spinal cord with cyclophosphamide-induced cystitis. Chemical entities:

Figure 10: Example of prompt used for **NED**. Dataset: BC5-Chem.