ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models (2024)

Abstract: Although Large Language Models (LLMs) exhibit remarkable adaptability across domains, these models often fall short in structured knowledge extraction tasks such as named entity recognition (NER). This paper explores an innovative, cost-efficient strategy to harness LLMs with modest NER capabilities for producing superior NER datasets. Our approach diverges from the basic class-conditional prompts by instructing LLMs to self-reflect on the specific domain, thereby generating domainrelevant attributes (such as category and emotions for movie reviews), which are utilized for creating attribute-rich training data. Furthermore, we preemptively generate entity terms and then develop NER context data around these entities, effectively bypassing the LLMs’ challenges with complex structures. Our experiments across both general and niche domains reveal significant performance enhancements over conventional data generation methods while being more cost-effective than existing alternatives.1

heading: 1 Introduction
text: Recently, Large Language Models (LLMs) have showcased their impressive capabilities across various NLP downstream tasks, including question answering, mathematical reasoning, and code generation (Touvron et al., 2023; Anil et al., 2023; OpenAI, 2023). Beyond these tasks, there is also a growing interest in exploiting LLMs’ strong capabilities and extensive parametric knowledge for synthetic data generation. While previous research has predominantly focused on generating datasets for instruction tuning (Wang et al., 2023c; Xu et al., 2023b; Li et al., 2023b) or text classification tasks (Meng et al., 2022, 2023; Ye et al., 2022a,b; Gupta et al., 2023; Li et al., 2023c), there has been
1Our code, prompts, and generated datasets are available at https://github.com/StefanHeng/ProgGen for reproducing our results.
limited focus on producing high-quality datasets for information extraction (IE). This gap can be attributed to the deficiency of LLMs in handling the complex formats and relationships between entities (Li et al., 2023a; Wei et al., 2023).
Stemming from the limited capabilities of IE, LLMs also demonstrate inadequate performance in the domain of Named Entity Recognition (NER) (Xie et al., 2023; Zhou et al., 2024; Li et al., 2024). Achieving high performance in NER typically necessitates a supervised model trained on in-domain data, which is a requirement hard to meet in realworld scenarios. Leveraging the vast knowledge of LLMs to enhance the capabilities of small supervised models remains a challenging task. Several previous works (Ma et al., 2022; Zhou et al., 2022) augment training data through the masked language modeling (MLM) task. For example, Kim et al. (2022) employ a retrieval system to generate NER samples from open-domain QA datasets; Zhou et al. (2024) utilize LLMs to annotate a pre-existing corpus for supervised fine-tuning, a process that necessitates domain-specific corpora for application.
We introduce ProgGen, a new method that leverages the self-reflexive capabilities of LLMs to progressively synthesize NER datasets. Our method, in contrast to traditional approaches that directly generate synthetic NER datasets, operates through a sequential process involving multiple steps. To overcome the challenges posed by LLMs’ limited ability to handle complex structures, we initially focus on generating entity terms, which are then employed for constructing the NER dataset. This step-by-step generation process allows for more effective dataset creation in new domains.
In addition to progressively generating NER datasets, we follow several recent studies to leverage LLMs’ ability to self-reflect (Shinn et al., 2023; Madaan et al., 2023) from two perspectives. First, instead of employing simple classconditional prompts, we leverage LLMs to generate
15992
domain-related attributes. For each attribute, we enumerate several values, iterating through combinations of these values to potentially generate a diverse and attributed dataset. Second, we adopt the self-reflection capability to examine the correctness of the entities it generates. These methods expand our usage from solely relying on the parametric knowledge of LLMs to leveraging their cognitive abilities to generate high-quality NER data.
Our findings reveal that a small supervised model, fine-tuned on our dataset, demonstrates robust performance across both general-domain and new-domain datasets in a low-resource context. Additionally, our approach exhibits cost-efficiency advantages over traditional prompt methods or LLMs employing in-context learning. We summarize our key contributions as follows:
1. Novel Progressive Generation Method - We deploy a step-wise method starting from generating entities for subsequent NER sample generation, where we also leverage LLMs’ cognitive ability to self-flex on attributes and self-correct wrong NER samples.
2. Low-Resource Utilization - Our approach minimizes reliance on human-annotated samples, substantially smaller compared to the datasets required by prior models and are cheap in terms of inference cost, making our approach particularly advantageous in scenarios where large annotated datasets are unavailable or impractical to compile.
3. Enhanced Efficiency and Performance - Our downstream model are efficient compared to LLM inference and exhibits superior performance in specialized domains, surpassing the standard data generation baseline by 8% in F1
score in the best case.

heading: 2 Related Work
text: LLMs have opened new avenues for generating synthetic data, as highlighted by recent studies (Meng et al., 2022, 2023; Ye et al., 2022a,b; Yu et al., 2023b; Gupta et al., 2023; Li et al., 2023c), mainly for text classification tasks. However, the use of overly simplistic and generic prompts in these methods often yields suboptimal results, as the representativeness and the diversity of the synthetic data. To overcome this, some studies have adopted interactive learning strategies to enhance instance generation (Liu et al., 2022a; Zhang et al., 2022a; Yu et al., 2023a; Chung et al., 2023), albeit at the expense of increased human intervention.
Generating synthetic data for IE is challenging due to the need for detailed consideration of entities and their relationships. To address this, some studies incorporate additional rules (Zhang et al., 2022b) or knowledge graphs (Chia et al., 2022; Josifoski et al., 2023) to assist IE synthetic data generation. Xu et al. (2023a) propose a two-stage training pipeline to alternately learn from synthetic and golden data together. For the NER task studied in this work, Liu et al. (2022b); Zhou et al. (2022) used masked language modeling to augment the training data, GeNER (Kim et al., 2022) leverages an entity retrieval model to create entity labels. Recently, several works studied to generate synthetic text with LLMs, but focus on general-domain or biomedical NER with fixed entity type (Guo and Roth, 2021; Hiebel et al., 2023; Tang et al., 2023; Xu et al., 2023c). These approaches, however, do not fully accommodate the diversity of entity types encountered in broader, open-world scenar-
ios. Zhou et al. (2024) also studied creating NER instruction tuning data, but they focused on using LLMs to label the corpus to create data for supervised finetuning, which diverges significantly from the synthetic data generation focus of our study.

heading: 3 Method
text: Our workflow is succinctly depicted in Figure 1. At the outset, we engage LLMs in a sequential process to generate a variety of NER samples. Unlike traditional methods that rely on domain-specific, unlabeled corpora, our approach is more pragmatic and resource-efficient. We utilize only a handful of examples to prompt LLMs to produce sentences along with their corresponding named entity annotations. This method is exemplified in Table 1, which showcases a standard few-shot prompt de-
signed for NER sample generation2. Our procedure involves requesting LLMs to produce multiple samples within a single completion to enhance efficiency. Subsequently, we carefully select a subset of entity annotations identified by LLMs as particularly challenging. These annotations are reannotated by the same LLM, this time with the aid of additional, manually crafted annotation guidelines and a selection of representative examples specifically chosen for this task. To perform NER tasks, we utilize DeBERTa (He et al., 2021) trained on the generated dataset, demonstrating the practicality and effectiveness of our low-resource, LLMbased approach for NER sample generation.3

heading: 3.1 Problem Setup
text: Our setup is formalized as follows. We are given a brief task domain description d, target named entity classes Y with their definitions, and a few demo samples Ddemo = {(xi, yi)}ni=1 where n < 10. Each sample consists of a sentence xi paired with an entity list yi = (e, c)
ni , consisting of contiguous text spans in xi and their corresponding entity classes. Utilizing these information, we prompt LLMs in a step-wise fashion and process the LLMs’ responses into NER samples Dgen. The aggregated dataset Dgen, combined with the demo samples Ddemo, is used to train a small model for NER. The model is then evaluated on an unseen test set.

heading: 3.2 Sample Diversity
text: To enhance the diversity of our generated NER samples, we augment a standard NER-samplegeneration prompt (Simple Prompt) by incorporating explicit diversity requirements for the generated samples through configuration sampling. We propose 4 variants aiming to enrich sentence semantics and entity variety (examples in Table 1). Processed diversity requirement pool with statistics are available at Appendix E.1.

heading: 3.2.1 Diversify Sentence
text: We study imposing explicit requirements on various semantic aspects of the generated sentences (Diversify X). Inspired by several previous work to generate diverse data (Yu et al., 2023a; Chung et al., 2023), we generate attribute dimensions and values given the dataset domain in a human-AI collaboration fashion (illustrated in Figure 2).
2NER sample representation were searched over 4 choices as detailed in Appendix B.1.
3See Appendix F for additional prompts and generated datasets used in our study.
Generate Attribute Dimension We first prompt LLMs to generate diverse attribute dimensions given the dataset domain. Given that this step consumes relatively less tokens and thus less cost, we leverage the more capable GPT-4 model. We manually review and select suitable attribute dimensions for subsequent steps.
For each dataset, we also include a “sample topic/category” dimension , such as “news topic” and “movie query category”, leveraged for subsequent diversity variants (§3.2.2 & §3.2.3).
Generate Attribute Values In this step, LLMs are prompted to generate potential values for each selected attribute dimension, where we examine and filter out low-quality values. We utilize GPT3.5 for its cost-effectiveness.
Construct Diversity Requirement We sample each generated attribute dimension at a predetermined probability such that the expected number of requirements in a sample-generation prompt Rx is small (∈ [1, 2]). We then sample an attribute value for each sampled attribute dimension. The diversity requirement templated from the sampled dimension-value pairs.

heading: 3.2.2 Diversify Entities
text: We study explicit requirements on inclusion of named entities in the generated sentences (workflow illustrated in Figure 2).
Generate Entities We generate named entities by prompting GPT3.5 given dataset domain (Diversify Y). Similarly to the Diversify Sentence process, we filter out low-quality generated entities based on manual inspection.
We explore two variants, (1) vanilla where we generate M entities for each entity class, and (2)
latent where we generate M entities conditioned on each entity class and additionally on each domainspecific topic value (via the dataset-independent “topic” attribute).
Construct Diversity Requirement We sample from the generated entity pool with an expected number of exampled entity Ry. We then construct the diversity requirement from the sampled named entities via a template. We omit the entity type corresponding to each sampled entity so that LLMs have more freedom in entity type annotations4.

heading: 3.2.3 Diversify Both Sentence and Entities
text: We provide explicit requirements on both sentence semantic and entity inclusion (Diversify X + Y). We first construct the sentence requirements, then we use the sampled “topic” attribute value to sample named entities from the corresponding latent entity pool such that the diversity requirements are semantically consistent.

heading: 3.3 LLM Self-Correction
text: Creating NER datasets critically involves defining entities and providing annotation guidelines for edge cases to ensure clarity (Zhou et al., 2024). These guidelines often vary across datasets, even for the same entity types (e.g., Location in news vs in restaurant queries). Given LLMs’ ability to learn from feedback and correct their own mistakes (Shinn et al., 2023; Madaan et al., 2023), we tailor instructions and examples for each dataset and entity class, directing LLMs to refine their annotations, aiming to align more closely with datasetspecific nuances. Further, to enhance cost efficiency, only a subset of annotations that LLMs
4We found that this lead to performance gains in preliminary experiments. Additional discussion in Appendix D.2.2.
find “challenging” are selected for self-correction.

heading: 3.3.1 Annotation Uncertainty Ranking
text: We identify challenging samples using a scoring function that averages the log-probability of tokens within each entity annotation, i.e. entity span (entity type), where a lower score means higher uncertainty and thus more challenging. Preliminary experiment compares this function with averaged token-wise loss values from a downstream model trained on the generated data (Appendix B.3), and shows that self-correction with the former scoring function yields better downstream performance.

heading: 3.3.2 Prompt Construction
text: We stratify all annotations to self-correct by the entity type annotated, allowing for more precise annotation instructions given the same token budget. Conditioned on the entity class, an LLMs are given (context, entity) pairs and are instructed to optionally improve the annotation by (1) correcting the entity span boundary, (2) correcting the entity type, and (3) dropping the entity annotation (template shown in Table 2). We batch correction samples for cost efficiency.
We construct the entity-type-specific prompts manually by alternating between the following 2
steps. (1) We examine top-uncertain entity annotations for each generated dataset to identify common error cases in LLMs’ annotations. We write correction instructions including common error cases. We hand-picked diverse demo samples from the generated datasets that represent the most common failure cases. (2) In case of uncertain annotation or class definition ambiguities (especially a problem for specific-domain datasets), since we were not able to find detailed annotation schemas for the datasets we studied, we inspected the original samples in the dataset to infer the annotation guidelines. We then manually write the entity definitions and annotation instructions.
Further analysis of ProgGen design choices are available in Appendix B.

heading: 4 Experiments
text: We assess the effectiveness of our framework via downstream model performance on generated NER datasets.

heading: 4.1 Datasets
text: We examine datasets from various domains and with different number of entity classes, including:
• CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003) features news articles from Reuters with three broadly-defined entity types: person, location and organization.
• WikiGold (Balasuriya et al., 2009) comprises Wikipedia articles. It’s a small dataset with the same set of entity types as CoNLL-2003.
• MIT-Movie and MIT-Restaurant (Liu et al., 2013) contain dialog system queries related to movies and restaurants, respectively. Both datasets incorporate a wide range of entity types, such as movie titles, genres, characters, and names of restaurants and cuisines.
The dataset statistics5 are summarized in Table 5Following prior studies (Li et al., 2021; Kim et al., 2022),
3. For datasets that originally contain over 1% negative samples, i.e. samples with no entity annotations, we include a negative sample in the set of demo samples to enhance LLMs learning what’s not a relevant named entity.

heading: 4.2 Experiment Setup
text: Data Generation We use 1-shot demo as described in Ma et al. (2022), wherein we include one entity annotation for each entity class. In the attribute dimension generation step for Diversify X, we prompted the ChatGPT4 web App 3 times. All other generations are from ChatGPT3.5 through OpenAI API6. We generate ~1.5K training samples for each dataset and for each method. We drop duplicates and filter out malformed samples (details in Appendix D.1). See Appendix E and F for generated datasets samples and statistics. We select all generated entity annotations with log-probability below Pc = −2e−2 for LLM self-correction, capped at Tc = 20% of all entity annotations.
Downstream Model Training For all setups, we train a DeBERTaV3 (He et al., 2023) model for 16 epochs with batch size 24, utilizing the microsoft/deberta-v3-base checkpoint from HuggingFace. We train with the AdamW optimizer (Loshchilov and Hutter, 2019), a learning rate of 4e−5, a weight decay of 1e−4, and a linear scheduler with warmup over the first 200 steps. The
the miscellaneous (MISC) class is omitted due to its lack of a precise definition. Datasets originally containing the MISC entity class are denoted with asterisks*.
6We used the gpt-3.5-turbo-1106 checkpoint.
1-shot demo samples are integrated into training with a weight of 5. Following prior work (Ma et al., 2022; Yu et al., 2023a), we don’t use a validation set and directly evaluate the final trained model with micro-average F1. For our approaches, we also report precision and recall.
Baselines We compare our approach with the existing few-shot baselines: (1) Few-shot FT (Huang et al., 2021), which directly finetunes a pretrained language model on few-shot examples. (2) SpanNER (Wang et al., 2021), which learns from natural language supervision to identify entity classes. (3) Ent-LM (Ma et al., 2022), which leverages masked language modeling to augment the few-shot examples. We also consider the following data generation baselines: (4) GeNER (Kim et al., 2022), which uses an entity retrieval module to retrieve from unlabeled corpus to create NER training data. (5) LLM-DA (Ye et al., 2024), which prompts GPT3.5 to rewrite the context and entities of few-shot samples to augment training data. (6) Our standard data generation prompt (Simple Prompt). Besides, we provide few-shot in-context learning (ICL) on the same set of demonstration samples for reference. We use reported results from prior work.
See Appendix D for additional implementation details.

heading: 4.3 Results
text: Tables 4 and 5 display our main results. We first note that the Simple Prompt baseline is strong and consistently out-perform prior works in few-shot NER, stemmed from the strong base knowledge
of LLMs. Also, few-shot ICL have recall values higher than most data generation approaches, reflecting loss of world knowledge when distilling LLMs to generated data.
We further note that (1) our best-performing setup matches or outperforms ICL in 3 out of 4 datasets we study. In general domains (News, Wikipedia), even Simple Prompt out-performs ICL, indicating a small pretrained BERT model can learn from noisy LLM annotations. (2) Batching sampleand correction-generation achieves roughly the same cost w.r.t. ICL inference. Data generation cost is one-time (see additional cost analysis in Appendix A.4), whereas few-shot ICL incurs inference API querying cost for each new prediction sample. Combined with the fact that (3) a small, task-specific downstream model has negligible inference overhead (c.f. LLMs), we highlight the efficacy of data generation for NER tasks.

heading: 4.3.1 Sample Diversity
text: Incorporating diversity notably improves model performance, outperforming Simple Prompt across all datasets in most cases. Diversify Y variants are the best-performing, yielding at least 2% F1 gains on average and up to 8% improvement for nuanced domains. The latent variant is more robust with higher average performance and more consistent gains over Simple Prompt, which may benefit from the implicit diversity in sentence semantics rooted from the latent “topic” dimension. Conversely, Diversify X tends to lower performance, a sign that sentence diversity may not benefit NER tasks, relative to entity diversity. Further manual inspection finds that data generated from Diversify X variants have lower-quality entity annotations (Appendix A.5.3), which may be the root cause of low performance. In particular, such annotations are more frequently incorrect and the wrong annotations encompass more disparate failure types. We
hypothesize that explicit semantic requirements in Diversify X result in less-frequent sentences and smoother token distributions, thus creating room for additional LLM annotation errors. Combining sentence and entity diversity (Diversify X+Y) yields intermediate gains, likely still affected by the shortcomings of Diversify X.
Diversity have diminishing gains. Comparing the vanilla and latent variants of Diversify Y, while the performances in general domain are mixed, vanilla Diversify Y wins in specific domains. We hypothesize that specific-domain entity classes are already narrowly focused (e.g. Director in movie queries vs. Person in news) such that the vanilla setting generates sufficiently diverse datasets. The additional “topic” attribute value for the latent setting may only marginally increase diversity and at times introduce noise (e.g. generate Genre entities for movie queries about Cast and crew; Table 35; Appendix F.4.5).

heading: 4.3.2 Self-Correction
text: LLM self-correction leads to additional 1-2% F1 improvements for most cases. Manual inspection on LLM self-correction responses further show that most corrections are valid (i.e., turns originally wrong entity annotations into correct ones), indicating LLMs can learn and generalize from annotation feedback. LLM self-correction mostly preserves the relative trend among diversity variants as opposed to closing the gap, underscoring the critical role of sample diversity and the initial quality of annotations. Precision scores generally lag behind recall, suggesting a propensity for over-identification of named entities by LLMs, leading to more false positives (Wang et al., 2023a). Further, self-correction tends to enhance precision more than recall, correlating with the distribution of LLM correction kinds (Appendix A.5.3).
Below, we discuss and show examples for two
most common classes of LLMs’ systematic biases that yield false positives, which are the focus of annotation instructions in self-correction prompts. See Appendix A.6 for more error classes and representative samples.
LLMs are overly confident in what is a named entity. LLMs tend to falsely annotate generic, non-named references and categorical references as named entities.
Span: The CEO of Apple Inc. announces a new product launch event. Entity Type: person Span: global organization announces plans for climate change summit in Reykjavik. Entity Type: organization
LLMs have overly broad definitions of entity classes. We found that LLMs may associate named entities of irrelevant types into one of the entity classes in prompt, such as counting events and awards as organization entities.
Span: Zhang Yimou, a renowned Chinese film director, has won numerous awards... Entity Type: Location (Correct type: Demonym) Span: Show me a trailer for the movie that has a song performed by Elton John. Entity Type: Song (Correct type: Artist)

heading: 4.3.3 Specific Domain
text: In specific domains like Movies and Restaurants, data generation shows reduced effectiveness compared to general domains (News, Wikipedia), with most ProgGen variants lagging in F1 scores behind ICL. This shows that distilling LLMs’ knowledge via data generation is especially challenging for more nuanced domains, with presumably smaller proportion of pre-training corpus on the given domain and respective entity classes. Notably, recall values are lower for specific domains, suggesting that the generated dataset might not fully capture the requisite domain-specific knowledge. However, gains for Diversify Entity variants roughly double for specific domains compared to general domains, effectively narrowing the performance gap with ICL in these challenging areas.
We further note that, in specific-domain datasets, certain entity classes are especially hard to define and separate due to ambiguity, yielding more wrong LLM annotations (detailed in Appendices A.5.3 and A.7). Manual inspection and partial F1 scores reveal that while LLMs correctly identify many entities, inaccuracies primarily arise from imprecise span boundaries and misclassified entity types (Appendices A.1 and A.6).
See Appendix B for additional experiments.

heading: 5 Discussion
text: Scaling diversity alone plateaus Using generated dataset size as a rough proxy of sample diversity, we plot its relationship with downstream model performance in Figure 3. We compare Simple Prompt with the top-performing Diversify Y variants where we re-generate 3K valid samples. We take 6 subsets of sample sizes [300, 600, 900, 1.5K, 2.1K, 3K] where we enforce the smaller subsets are contained in the larger subsets. We then train NER models as in §4.2. We observe that substantially increasing dataset size doesn’t lead to much performance gains across the board, confirming our speculation that sample diversity alone has diminishing returns (§4.3.1).
Entity annotation accuracy is performance bottleneck We conduct an ablation experiment to evaluate the impact of sentence distribution and LLM annotation accuracy on model performance, using modified datasets described below:
• Supervised Learning. A 90/10 train/dev split on 1.5K dataset samples as an upperbound7.
• LLM Annotated. A 1.5K-sample subset of the training set containing the 1,350 training samples above, with entity annotations from ChatGPT.
7For WikiGold, we use the entire training dataset with just 1.1K samples.
• LLM Rephrased & Annotated. Sentences from the samples above, first rephrased and then annotated by ChatGPT.
• LLM Generated. The best-performing diversity setup using 1.5K LLMs-generated data for reference (from Table 4).
We then train NER models as in §4.2. For the Supervised Learning setup, we use the model with best dev loss. The results are shown in Table 6. We note that differences in sentence syntactic and semantic distributions hardly influence downstream performance, where as the largest drop in performance is attributed to the lower-quality of LLMsgenerated annotations compared to the original. On the other hand, top-performing data-generation approaches achieve similar performances, indicating that with sufficient diversity, LLM-generated sentences matches a domain-relevant unlabeled corpus. Thus, annotating accurately w.r.t entity class definitions becomes the performance bottleneck.
Step-wise sample generation Considering the limited instruction following capabilities of GPT3.5 (Ouyang et al., 2022) may influence performance, we explored generating NER samples step-wise (from sentences, entity spans, to entity types) with manually-written Chain-of-Thought (Wei et al., 2022) demos for entity annotation. We found no noticeable performance gains, despite the higher token querying costs due to re-prompting partially generated samples. See Appendix B.2 for additional discussion.

heading: 6 Conclusion
text: We present ProgGen, an innovative framework that leverages LLMs to generate diverse and accurate task-specific datasets via requirement sampling and self-reflection for low-resource NER. Experimental results on 4 datasets encompassing various domains demonstrate the advantages of our data generation approach compared to LLM few-shot in-context learning both in downstream model performance and cost-efficiency. Our research highlights the critical role of entity diversity in improving dataset quality and demonstrates the effectiveness of incorporating LLM self-correction alongside manual feedback to enhance annotation accuracy. Further analysis on the generated datasets reveals persistent biases in LLM entity classification and that annotating accurately is a major bottleneck to close the gap with traditional supervised learning methods.

heading: 7 Limitation & Future Work
text: More Comprehensive Experiments Our investigation focused on widely recognized NER benchmarks. Although we did not find signs of LLMs regurgitating training samples8 (Carlini, 2020), experiments on less-widely-used datasets such as datasets in the medical domain or datasets released after LLMs’ pretraining cutoff times would yield more reliable findings, as such datasets are less likely to be contaminated. Additionally, our work can be viewed as data augmentation which may particularly benefit lower-resource domains.
Furthermore, our study was constrained to utilizing ChatGPT3.5, primarily due to budgetary limitations. Expanding future research to a broader range of LLMs could provide more insights, as more advanced LLMs may follow instructions and annotate entities better.
Enhancing Automation Selecting representative demos for LLM Self-Correction can get laborintensive since it involves examining large number of entity annotations for each generated dataset. It’s worth assessing annotation instructions and demos shared for a given task domain. Advancing automatic feedback generation leveraging stronger LLMs such as GPT-4 can help, namely in (1) crafting annotation instructions and (2) evaluating annotations from weaker LLMs.
Further, the hyperparameter choices for data generation in this work is setup ad hoc whereas they may influence downstream performance noticeably. Future work can explore how to optimize these hyperparameters under the few-shot setup.
Better Annotations We found that LLMs may generate different/conflicting annotations on the same sentence. Such uncertainty signals can be leveraged to identify challenging instances and improve entity annotations. See Appendix C for additional discussions.

heading: Ethics Statement
text: We acknowledge the potential biases of texts generated by language models which may not be factual and appropriately represent the included entities (including persons, locations, organizations, movies, restaurants) in the generated datasets. In our studies, we did not find noticeable cases of
8For example, in CoNLL-2003, we found that LLMsgenerated news articles that are more recent, as opposed to news around 2003.
toxic language in the generated data as OpenAI API have content moderation policies in place. Nonetheless, harmful synthetic sentences may be present in the processed datasets.
Our research utilized publicly accessible NER benchmarks and was designed to be computationally efficient, with each training session on a generated dataset requiring no more than 10 minutes on a NVIDIA GeForce RTX 2080 Ti GPU.

heading: Acknowledgements
text: This work was supported in part by NSF CAREER IIS-2144338, ONR MURI N00014-17-1-2656, and research grant from ADP. We appreciate the compute resources provided by Chao Zhang’s group. We also thank Junmo Kang, Alan Ritter and the anonymous reviewers for their insightful feedback on the project.

heading: A Additional Analysis

heading: A.1 Partial F1 Evaluation
text: We note that f1 only reward exact boundary matches of identified named entities (Manning, 2006) whereas partial matches may still benefit users of NER systems. As such, we also report the partial f1 scores following prior work (Zhou et al., 2024). We use the setup as described in SeguraBedmar et al. (2013), where predicted named entities that overlaps with ground truth named entities of the same type are counted as 0.5 true positive9. We report partial F1 scores for the main experiments (§4.3) in Table 7. We observe that F1 and partial F1 scores share similar trends where we can make similar conclusions as in §4.3. We further note that (1) the gap between exact F1 and partial F1 scores are greater for specific domains (~10% vs ~3% for general domains), and thus (2) partial F1 closes the performance gap between generaldomain- and specific-domain-datasets, indicating that annotating the precise entity span boundaries is more challenging in specific domains, corroborating with manual inspection (Appendix A.6).

heading: A.2 Full Supervised Learning
text: As reference, we show standard supervised learning results using the same training pipeline as in §4.2 and our de-duplicated version of the original NER datasets (Appendix D.1) in Table 8. We report F1 on the model with best dev loss.

heading: A.3 OpenAI API Stability
text: We note that data generation via OpenAI’s Chat Completion API may yield unstable results, with minor data generation pipeline edits. Unfortunately, OpenAI API did not support generation output seeding until Nov. 2023, where even seeding yield different results sometimes. Earlier NER data generation results yield slightly different F1 scores compared to that we report in §4.3, as shown in Table 9. We attribute the discrepancy to OpenAI server changes (system_fingerprint)10. We can observe similar trends except that in earlier experiments, Diversify X works better more frequently. The final conclusion remains the same: (1) diversity improves performance, where entity diversity
9We used the implementation from the nervaluate package available at https://github.com/davidsbatista/ NER-Evaluation
10https://cookbook.openai.com/examples/ reproducible_outputs_with_the_seed_parameter
contributes the most, and (2) the latent variant of diverse entity yield most robust gains across datasets.
To further assess whether unstable OpenAI completions influence our findings, we re-run Simple Prompt 3 times with different random seeds (42, 43, 44) for prompt construction, OpenAI completion and downstream model training. We note that GPT3.5 frequently fail to generate L = 50 samples as instructed in prompt (may stop generation around 20 samples; see Appendix D.2.3), Thus, for the ease of seeding, we generated NER samples with L ∈ [3, 10]. We report the average F1 and standard deviation in Table 10. We observe that different L’s influence dataset-specific performances noticeably with fluctuations, while for both L values, the average F1 scores stays around the same w.r.t. the results we report (Tables 4 and 9). This confirms our findings that that ProgGen leads to noticeable gains over Simple Prompt.

heading: A.4 API Querying Cost
text: In this section, we show the token querying cost11
for each step of data generation for the main experiments (§4.3) in Table 11. We make the following observations: (1) Simple Prompt costs 4X-5X less than ProgGen, since it contains only NER sample generation with significantly less number of prompt tokens (Appendix D.2.3). (2) The cost for Diversify X are primarily attributed to the high price of querying GPT4. (3) For a given dataset, costs for NER sample generation and entity correction are roughly the same across all diversity variants, since the only differences are the length of the diversity requirements and number of generated entity annotations, respectively.

heading: A.5 LLM Self-Correction

heading: A.5.1 Prompt Quality
text: We found that LLMs are especially biased in certain cases. For example, LLMs frequently consider demonyms (e.g. Chinese, American) as location entities. We found that more precise categorization of wrongly-annotated terms in the annotation instruction leads to more valid LLM self-corrections (e.g. “Demonyms” vs “Adjectives”, “Ambiguous identifiers” vs “General references”). Instructing LLMs that certain terms are not named entities seems less effective than suggesting such terms are named entities of “other” type.
11We used GPT3.5 input $0.001, output $0.002 and GPT4 input $0.03, output $0.06 for 1K tokens (cost at the time of experiments).

heading: A.5.2 Performance Drops
text: In the main results (Tables 4 and 5), we observe that less frequently, self-corrections leads to performance drops or insignificant performance differences. We hypothesize the reason is unstable training, as a result of 3 scenarios: (1) Since average log-probability is imperfect in identifying wrong annotations, valid LLM corrections may not significantly influence downstream model learning if the majority of same-class annotation errors are in the remaining 80% un-selected annotations, and thus not corrected (for example, cursory manual inspection found only 6 out of 40 occurrences of directed (Director) selected); (2) LLMs may (less frequently) falsely correct previously valid entity annotations, such as introducing noise into the training data by misclassifying entities; (3) Certain
training samples have a larger influence on model learning (Thakkar et al., 2023). Changes to the annotations of these samples can lead to significant shifts in results. Appendix B.3.3 can serve as an evidence for points (2) and (3).
We notice that occasionally, LLM SelfCorrection trades improved precision with lower recalls, indicating LLMs may drop valid named entity annotations.

heading: A.5.3 Correlation with Correction Statistics
text: We make a loose assumption that the probability of LLM corrections that are invalid is about the same across diversity variants and regardless of the class
of annotation error.
Downstream Performance As such, we can correlate the distribution of LLM correction types (Appendix E.3) with downstream performance gains (Tables 4 and 5).
We note that by definition of F1 score for NER evaluation, (1) valid entity span corrections would improve true positives and reduce false positives, (2) valid entity type corrections would improve true positives, and (3) valid entity drops would reduce false positives. More true positives lead to improvements in precision and recall, whereas less false positives lead to higher recall.
We make the following observations: (1) For all datasets except MIT-Restaurant, we observe high ratio of entity drops relative to entity span and type corrections, indicating high occurrence of false positives (Tables 32 and 33), which can explain mostly higher precision gains from LLM self-correction. (2) For all diversity variants with explicit semantic requirements (Diversify X and Diversify X+Y), we observe noticeably higher ratios of annotations LLM corrected (Table 32), reflecting the low-quality annotations related to sentence diversity.
Annotation Difficulty With the same assumption, we can also get insight into which classes are more challenging for LLMs to annotate. In Table 33, we observe that the difficulty for annotating entities in specific domains is greater and more varied across classes (e.g. In MIT-Movie, LLMs corrected from 10+% to ~60% selected entities).
Specifically, based on the distribution of entity type corrections (Figure 5), we can probe into how well LLMs separate relevant entity classes. Notably, (1) in CoNLL-2003 and WikiGold, Location entities and Organization entities are more interleaved. (2) In MIT-Movie, Genre and Plot entities, and Viewers’ Rating and Review entities, are more mixed up. (3) LLMs especially struggle with identifying the correct type in MIT-Restaurant, with Amenity, Cuisine and Dish entities mixed together and many Amenity and Hours entities falsely labeled.

heading: A.6 Representative Annotation Errors
text: In this section, we select and show representative entity annotation errors for each dataset in Tables 12 and 13 to highlight systematic biases in LLMs. See the exact and illustrative annotation instructions that covers broader classes of LLM annotation
errors repository12. We note that, for general domain datasets
(CoNLL-2003 and WikiGold), the main problem in LLM annotations is false positives, including (1) annotating generic and non-name references (e.g. CEO as person entity) and (2) categorizing irrelevant named entities (e.g. mythological figures as person entities). For specific domain datasets (MIT-Movie and MIT-Restaurant), LLMs mainly suffer from (1) imprecise spans annotated (e.g. “action movie” as opposed to “action”), and (2) entity type confusion (e.g. Viewers’ Rating vs Review). In MIT-Restaurant, we also observed that GPT-3.5 may drop many valid named entities (more frequently than other datasets).

heading: A.7 Entity Class Ambiguities
text: We note that, espcially for datasets in the specific domain (MIT-Movie and MIT-Restaurant), certain entity classes are hard to define just given its label (e.g. Viewers’ Rating vs Review). Without access to entity annotation schemas, there can be multiple valid interpretations leading to different “correct” annotations. Further, inferring the annotation schemas is challenging even for humans since even original samples in the test set are noisy. We show three significant (highly-occurring) classes of ambiguities encountered in our study below. See Appendix A.6 for representative LLM annotation errors that reflect these challenges. In particular, we define the entities with our judgement from reviewing samples in the dataset (mostly determined via majority vote on similar entity contexts) and show representative ambiguous samples in the test set.

heading: A.7.1 Ambiguous Class Separation
text: MIT Movie: Viewers’ Rating vs Review We separate Viewers’ Rating entities and Review entities based on level of detail, where Viewers’ Rating entities are shorter such as “good” and Review entities are more detailed such as “incredible visual effects”. By this definition, a large fraction of Review entities in the test set should have been Viewers’ Rating entities.
MIT Restaurant: Cuisine vs Hours It’s unclear whether meal times such as “breakfast” and “lunch” are Cuisine entities or Hours entities, since (1) both types share similar query contexts for the same
12https://github.com/StefanHeng/ProgGen/ reproduce/correction
entity span and (2) the occurrence counts of the same entity span can be close to each other (e.g. 8 vs 9). Thus, we found no better solution than just declaring all such terms as Hours entities (for slightly more annotations).
We show samples in Table 14.

heading: A.7.2 Conflicting Annotation Guideline
text: We note that higher-order annotation guidelines are counter-intuitively conflicting in certain cases, where annotations need to follow opposite rules for different entity classes13.
13One reason can be that the tasks for MIT-Movie and MITRestaurant are more precisely slot filling as opposed to named entity recognition, where the relevant information to extract may differ.
MIT Movie: Trailer vs rest We note that annotating the Trailer class is not precisely NER, where vast majority of the annotated spans simply indicate querying about trailer information (A). However, other entity types don’t following the same rule, i.e. querying about other entity types (e.g. director information) are not highlighted (B).
MIT Restaurant: Rating vs Price Rating entities frequently include "rating" in the phrase, such as "4-star rating" as opposed to "4-star" (A), where as Price entities often omit trailing "price" terms (B).
We show samples in Table 15.

heading: A.7.3 Unseen Edge Cases
text: We note that LLMs-generated samples are more diverse (Appendix E.2) and LLMs may generate edge-case spans unlike any cases in the original entities annotated. For example, in WikiGold, LLMs may generate places in mythology as location entities where as the original dataset samples don’t contain sentences about mythology. In such ambiguous cases, we use our judgement to determine the correct annotation. For example, (1) in WikiGold, we consider places in mythology such as "underworld" as Location entities; (2) In MIT-Movie, we consider popular movie studios such as "Marvel" and "Disney" as Genre entities, and we ignore label interpretations (e.g. "Parental Guidance" as opposed to "PG") as MPAA Rating entities.

heading: B Additional Experiments
text: In this section, we discuss preliminary experiments that guide the ProgGen design choices.

heading: B.1 Sample Format Search
text: Cursory experiments explored 4 different formats for representing NER samples (examples in Table 16) in two directions: (1) generating complete sentences (natural language), and (2) generating token by token (token-wise):
1. Natural Pair. Generate a sentence and the list of (entity name, entity type) pairs.
2. Natural Inline. Generate a sentence and then generate the sentence again with inline entity class annotations.
3. Token List. Generate a sentence, then generate
the tokens in the sentence, and finally the BIO tags corresponding to each token.
4. Token Lines. Generate (token, BIO tag) line by line, in CoNLL-2003 style.
We note that each sample formatting are equally expressive. Namely, LLMs have the option to annotate multi-occurring text spans with potentially different types.
We conduct experiments on the CoNLL-2003 dataset and select the top-performing sample format for the rest of our experiments in this study.
We generate around 1K NER samples1415 using a prompt template like Simple Prompt (Table 1). We use the first 5 samples in the CoNLL-2003 training set as demo examples. we observed that the Token Lines format had repetition issues and gen-
14We used the GPT3.5 gpt-3.5-turbo-0613 checkpoint. 15Experiments were done between Sep. 12th - 17th, 2023.
erated exact samples in the original CoNLL-2003 dataset. Thus, we did not investigate this sample format further. A BERT (Devlin et al., 2019) model was trained on other sample formats with learning rate 2e−5. Other training hyperparameters remain the same as the main experiments (§4.2 and Appendix D). We used the bert-base-cased checkpoint from HuggingFace. We compare the best test set F1 of the BERT model from epoch-wise evaluation (results in Table 16). The model trained on datasets generated via the Natural Pair format performs the best.

heading: B.2 Step-wise Generation
text: The instruction-following capability of ChatGPT3.5 may be limited. The complexity of simultaneously generating diverse samples and accurately annotating entities in a single prompt may present a considerable challenge. Consequently,
we investigate whether separate prompts for each aspect (sentence generation & sentence annotation) leads to improved instruction following and thus improved downstream performance. In this section, we explore multiple variants of step-wise sample generation, all starting from sentence generation.
B.2.1 2-Stage Generation
Our 2-Stage NER sample generation pipeline is illustrated in Figure 4a. We first prompt LLMs to generate diverse sentences. Then, the generated sentences are annotated by LLM. In the first step (sentence generation), we add explicit diversity requirements just like in default NER sample generation (§3.2; note the diversity requirements are for the sentences). Then, for all diversity setups, we use the same annotation prompt with temperature t = 0.
We run experiments with setup just like in §4.2.
Suppose you are a news writer. Please generate 3 synthetic sentences from news stories. Cover the following entity types: [person, location, organization]. Examples: {sentence demo examples} {diversity requirement}\n
Here are 5 sentences from news stories. Identify and classify named entities of types [person, location, organization]. Examples: {NER-sample demo examples} Please annotate the following 5 sentences: 1. Time Magazine names Dr. Jane Smith as person of the year. 2. ... \n \n
1. 2. ...Time Magazine names Dr. Jane Smith as person of the year. \n
1. [Time Magazine (organization), Jane Smith (person)] 2. ... \n
Step 1: Generate Sentences
Step 2: Generate Annotations
(a) 2-Stage NER Sample Generation pipeline. We first prompt LLMs to generate sentences. The LLMs are prompted again to annotate the generated sentences.
Suppose you are a news writer. Please generate 3 synthetic sentences from news stories. Cover the following entity types: [person, location, organization]. Examples: {sentence demo examples} {diversity requirement}\n
Here are 5 sentences from news stories. Identify named entities of types [person, location, organization]. Examples: {(sentence, entities) demo examples} Please identify named entities the following 5 sentences: 1. Time Magazine names Dr. Jane Smith as person of the year. 2. ... \n \n \n
1. 2. ...Time Magazine names Dr. Jane Smith as person of the year. \n
1. [ , ] 2. ... Time Magazine Jane Smith \n
Step 1: Generate Sentences
Step 2: Generate Candidate Entity Spans
Here are 5 sentences from news stories with potential named entities identified. Classify the text spans in context of the sentence into categories [person, location, organization, other entity, not an entity]. Examples: {(sentence, entities) demo examples} Please classify text spans in the following 5 sentences: 1. Time Magazine names Dr. Jane Smith as person of the year.   Text: Time Magazine Text: Jane Smith 2. ... \n \n \n \n \n
1. Label: organization Label: person 2. ... \n \n
Step 3: Generate Candidate Span Labels
(b) 3-Stage NER Sample Generation pipeline. We first prompt LLMs to generate sentences. Then, LLMs are prompted again to identify potential named entities in the generated sentences. Finally, LLMs are prompted to classify the type of each named entity identified in context of the generated sentences, with the option to drop the named entity.
Figure 4: Step-wise NER Sample Generation pipeline.
The results are shown in Table 1716. We observe that, compared to the vanilla (1-stage) data generation (Appendix A.3 and Table 9), 2-stage doesn’t yield higher F1 scores, even at the expense of additional token querying cost (for prompting the generated sentences again), indicating simply reannotating in an isolated prompt doesn’t improve annotation accuracy.
Chain-of-Thought We study whether chain-ofthought (CoT) prompting (Wei et al., 2022) can help LLMs annotate better where we manually write annotation reasoning for each demo sample. We compare generating entity annotations (step 2) naively (vanilla) or with manually written chain-
16Experiments were done between Nov. 28th - Dec. 28th, 2023.
of-thought reasoning (CoT). We experiment on the CoNLL-2003 dataset with Simple Prompt17. An example entity annotation prompt is shown in Table 18. We found that CoT results in worse downstream performance (69.6% vs 72.0% F1). Manual examination reveals that many LLM-generated reasoning chains are not factual, as if copying the formatting from the demo samples.
B.2.2 3-Stage Generation We also experimented with 3-stage generation (pipeline illustrated in Figure 4b), further splitting the entity-generation step into candidate entity span generation (step 2) and entity span classification (step 3). In the last step, LLMs can also optionally drop the entity candidate entirely to address the common false-positive issues (Detailed discussion in Appendix A.5.3). Intuitively, since step 3 cannot add entity spans, LLMs are instructed to generate more candidate entities in step 2 where we also add non-entities as demos. In the last step, we group all candidate spans for the same sentence together
17Experiments were done between Nov. 19th - 21st, 2023.
to reduce number of prompt tokens and thus token querying costs (c.f. classifying each distinct (sentence, entity) pair).
We experimented with CoT prompting for both step 2 and 3 on CoNLL-2003 and MIT-Movie to improve annotation accuracy18. We show example prompts in in Tables 19, 20, 21 and 22. For MIT-Movie, to provide additional annotation instructions, we further added a manually crafted NER sample to showcase representative annotation errors. Performance comparisons are shown in Table 23. We found that 3-stage generation shows gains for CoNLL-2003, but makes no difference in F1 score for MIT-Movie. Manual inspection shows that the main problem for MIT-Movie is incorrect span boundaries as a result of step 2, where as step 3 can only correct type errors. Thus, we proposed LLM self-correction (§3.3) with more explicit feedback and correction mechanisms.

heading: B.3 Entity Annotation Uncertainty Ranking
text: As discussed in §3.3 and §3.3.1, we compare two scoring functions to select entity annotations for LLM Self-Correction (Appendices B.3.1 & B.3.2).

heading: B.3.1 LLM Generation Log-Prob
text: We average the log-probabilities for all tokens w.r.t the entity annotation (i.e. log-probs for span (entity type)), denoted log-prob. As such, a lower average log-prob means more uncertainty.
18Experiments were done between Dec. 27th - Jan. 15th, 2023.

heading: B.3.2 Downstream BERT Loss
text: We first train a BERT-class model on the generated dataset and compare its prediction signals with the LLM annotations (i.e. the dataset the model was trained on), where lower prediction discrepancy means more uncertainty (denoted loss). Intuitively, we can leverage feedback from the trained downstream model since it learns from a larger pool of NER samples and learns to resolve noises in the original LLM annotations in the training process. We run inference with the downstream model on the generated sentences to get token-wise cross entropy (CE) loss w.r.t the original LLM annotations19, where higher loss means higher discrepancy between BERT and LLMs’ annotations. To
19In terms of implementation, we simply use CrossEntropyLoss from PyTorch treating the LLM annotations as “ground truth”.
get the ranking score for each LLM entity annotation, we average the negated loss values for the subset of tokens in the sentence that correspond to the entity span. We further include one more token on both sides of the entity span if possible to better capture discrepancy in span boundaries. For example, given a generated NER sample with tokens [Bob, is, born, in, Athens, .] and BIO tags [B-person, O, O, O, B-location, O], to get the uncertainty score for “Athens“, we average the negated loss for the tokens [in, Athens, .].
We note that conceptually, loss ranking can be considered as a log-probability and hence comparable with log-prob on the same scale. As shown in equation 1, the negated CE loss for token position i with entity class label y is effectively the log of a probability in range [0, 1]:
−CE Loss(i, y) = log exp(logits(i)) sum(exp(logits(i)))
= log softmax(logits(i))[y]
= logP (y)
(1)

heading: B.3.3 Experiment
text: Compare the quality of entity annotations selected via log-prob and loss, we conducted preliminary experiment on data generated for MIT-Movie via Diversify Y (vanilla). The experiment setup including annotation-selection hyperparameters for
log-prob remain the same as in the main experiments (§4.2). For loss, we do not perform any filtering (Pc = 0) and simply select the top-ranking Tc = 20% for each entity class normalized by entity proportions. In addition to LLM self-correction,
one of the authors manually annotated the selected entity annotations. The results are shown in Table 24. We observe that whether entity annotations are corrected via LLMs or manually, log-prob-selected annotations noticeably yield better downstream F1.
Treating manual corrections as ground truths, we further examine the corrections as shown in Table 25. We observe that, comparing entities annotations selected via log-prob and loss, (1) the ratio of invalid annotation corrections that LLMs make is the same, and (2) loss selects more wrong entity annotations and identifies significantly more non-named-entities. Thus, we conclude that in this case, loss scoring selects entity annotations that less significantly contribute to performance gains. We hypothesize that the downstream model might identify more errors that it already resolved during the training process (Appendix A.5.2). We look forward to more comprehensive investigations on selecting influential entity annotations in future work.

heading: C Future Work
text: More Analysis Given the initial success of ProgGen, it would be interesting to explore how does sample diversity contribute to performance. Notably, we hypothesized that distilling entity knowledge is more challenging in specific domains
(§4.3.3), experiments on how scaling the number of training samples influence downstream performance can provide more insights.
Cost analysis trading downstream performance with token querying cost is also interesting to investigate. For example, how would hyperparameters such as batch size (for sample generation and correction generation) influence the quality of the generated dataset?
ProgGen Extensions Cursory manual examinations on the generated datasets found several cases where LLMs generate syntactically and semantically similar sentences (examples in Table 26). This implies that (1) the number of generated samples may not correlate linearly with sample diversity and (2) the downstream training process is biased towards the “duplicated” groups of samples. Future work can explore how to select a diverse subset of generated sentences for annotations and downstream training.
In our main experiments (§4.3), we found that Diversify Y variants work well, where the vanilla variants perform the best across almost all datasets and the latent variant leads to more robust performances. One extension can explore merging these variants via sampling to get the best of both worlds and further, generating entity pools conditioned on additional attribute dimensions from Diversify X.
Our current self-correction pipeline only handles the most common cases of entity annotation errors. More more nuanced annotation errors can be incorporated, including (ordered by occurrence from cursory inspection): (1) both wrong span and wrong type, and (2) two valid annotations. We show example wrong entity annotations of such case in Table 27. Our work mainly focused on reducing false positives since we found this is a more prevalent issue, future work can also explore looking at existing annotations given a sample and add additional annotations (reducing false negative).
Better Annotations We note (1) the strong performance of Simple Prompt (Table 4) and that (2) Simple Prompt have relative less number of generated samples in downstream training due to deduplication (Appendix E.2). Thus, we hypothesize that Simple-Prompt-generated NER samples may have more accurate annotations, despite less diverse, possibly due to an implicit self-consistencylike (Wang et al., 2023b) “check” where LLMs generate the same sentence multiple times. In particular, one of two cases will happen: (1) all gen-
erated sentences have the same annotations and thus is part of the processed dataset, or (2) some of the generated annotations are conflicting and thus such samples are dropped from the processed dataset. We also observed that despite many wrong entity annotations GPT-3.5 make, GPT-3.5 may have a higher accuracy for easier tasks such as simply checking if terms like “Nigerian President” is a named entity. Thus, despite incorporating CoT and stage-wise annotation are shown nontrivial (Appendix B.2), future directions include exploring more variants, including (1) combining reannotation (Appendix B.2) with self-consistencylike (Wang et al., 2023b) majority voting to reduce noises (in entity class, demo & input ordering) in LLM annotations and (2) least-to-most-promotinglike (Zhou et al., 2023) step-wise annotation correction to reduce bias.
Additionally, the task of selecting potentially all wrong entity annotations in the generated dataset in a single trial may be too challenging. In light of the hypothesis that LLMs may not correct enough wrong annotations of the same failure class to significantly influence training (Appendix A.5.2), future work can explore propagating “confident” LLM corrections to the non-selected pool of entity annotations, leading to better annotated datasets.

heading: D Additional Implementation Details

heading: D.1 NER Sample & Dataset Processing
text: This section details how LLMs’ free-form responses are parsed into structured NER samples for training.
NER Sample Parsing We use regular expression (re) patterns to match and extract the generated sentences and list of entity name-type pairs. We drop malformed NER generations. Primary reasons for invalid NER samples include (1) overlapping entity spans annotated, (2) unseen entity type annotated, and (3) annotated entity spans not found in the generated sentence. For the few-shot ICL baseline, to ensure a prediction is made for each sample
in the original test set, we filter out the subset of malformed entity annotations in the generated annotation list (as opposed to filtering out the test sample entirely).
Despite that LLMs are instructed to (1) annotate each multi-occurring named entities multiple times (Appendix B.1) and (2) order the annotated spans by occurrence, we found LLMs sometimes don’t follow the instructions. We show one rare case where LLMs annotate a multi-occurring span with different annotations below:
Sentence: Can you tell me about the character Harry Potter from the Harry Potter series? Named Entities: [Harry Potter (Character), Harry Potter (Title)]
Thus, we pragmatically (1) duplicate single annotations for multi-occurring entities as much as it occurs (as a result, they have the same entity type), and (2) re-order the entity annotations to match the occurrence order in the generated sentence as needed.
BIO Format Mapping For downstream model training, we tokenize the sentences and entity spans by splitting on any whitespace and punctuations (and then process NER tags into BIO format). Conversely, since the original datasets we studied are in the BIO format and we need the demo samples in a natural-language format in prompts (Appendix B.1), we detokenize the original samples via the sacremoses package.
Deduplication We observed that the original datasets may contain duplicate samples or conflicting annotations. Thus, we filter out these samples. In particular, we (1) keep only 1 copy of duplicate samples (same sentence and same annotations; duplicate), and (2) drop all conflicting samples, i.e. samples with the same sentence but different annotations (conflicting). The number of dropped samples are summarized in Table 28.

heading: D.2 Data Generation

heading: D.2.1 Diversify X
text: Generate Attribute Values Since ChatGPT generates long-form responses, the responses from attribute dimension generation also include additional attribute descriptions and example values. We may select high-quality generated descriptions and examples and include them in the attribute value generation prompt (example in Appendix F.1.1).
Construct Diversity Requirement The datasetindependent “topic” dimension is always included/sampled in the diversity requirements, for simplicity of incorporating with the Diversify X + Y variant, passing the sampled topic value to the named entity pool.
During attribute dimension sampling, we note that certain dimensions may have similar or conflicting values. For example, for MIT-Restaurant, attribute dimensions “dietary restriction” and “special offering” may both contain dietary needs. Thus, we group such “conflicting” attributes together and ensure at most one attribute dimension in the conflicting set is sampled. The conflicting sets for each dataset are shown in Table 29.
Another reason for keeping a small Rx (∈ [1, 2]; §3.2.1) is that we expect limited instructionfollowing for GPT3.5 given a larger number of diversity requirements, where too many instructions presumably lead to lower-quality samples (Ouyang et al., 2022).

heading: D.2.2 Diversify Y
text: Generate Entities We generally use a smaller M for latent Diversify Y compared to the vanilla variant. Intuitively, this is because with a more detailed requirement (conditioned on both an entity class and a domain-specific topic), there tend to be less relevant named entities.
Construct Diversity Requirement For each entity class, we sample a few entities {ei}Pc1 for each entity class c, where Pc is sampled uniformly in [0, 3] to ensure entity class balance. We then union all named entities sampled and takes a subset on the named entities as if in Bernoulli trials where we enforce the expected number of remaining entities to be Ry.
In preliminary experiments, we found that including the entity type corresponding to each sampled entity in the entity inclusion requirement consistently yield worse performance (~3%). We hypothesize that (1) LLM-generated entity pool may not be accurate, i.e. named entities sampled may not belong to the given type, leading to incorrect requirements, and (2) specifying the entity type for spans that may belong to multiple entity types based on context (e.g. Burger as a Dish or a Cuisine entity) may limit the LLMs’ capability in following the instruction. In both cases, LLMs may generate samples with wrong annotations, leading to worse downstream performance.

heading: D.2.3 NER Sample Generation
text: In our API calls, we use temperature t = 1 and parameter top p = 1.0 unless otherwise specified. For a fair comparison, for each approach, we ensure LLMs generated 1.8K samples. We filter out malformed samples LLMs generated and de-duplicate the remaining, valid samples (Appendix D.1), resulting in different number of NER samples in the NER dataset for training. For Simple Prompt, we request (L = 50) samples in each prompt (thus ~36 API calls) to further save token querying cost (since less prompt tokens). We found that GPT-3.5 frequently fails to generate 50 samples requested so we made additional API calls until 1.8K samples are found (without checking for validity and duplication). For all diversity setups, we request 3 samples in each prompt to sample more diverse configurations, making ~600 configurations total for each variant. We use E [Ry] = 4.5 for MITMovie and E [Ry] = 1.5 for other datasets in our experiments, which empirically performs well in early experiments.
Few-shot ICL experiments are greedily decoded (t = 0).

heading: D.2.4 Self-Correction
text: For LLM self-correction, each prompt includes 3 samples to correct and we use greedy decoding (t = 0). When crafting the self-correction annotation instructions and demos, we generally follow the rule below: 1) if a failure case appears more than 5 times, we include a demo correction sample, and 2) if a failure case appears more than 3 times, we summarize the error case in the annotation instruction. We provide no more than 6 error cases in the instruction and no more than 6 demos. The typical number of wrong-annotation cases in instruction and demos are about 3.
Less common failure cases are ignored to prevent overly tuning for each generated dataset. We further note that (1) ChatGPT3.5 can correct many “easier” wrong entity annotations 0-shot, i.e. without any instruction and demo, and ChatGPT3.5 is biased sometimes and its capacity and instruction following capabilities are limited, where (2) a fraction of LLM Self-Corrections are invalid and (3) additional demos may not lead to additional valid corrections.

heading: D.3 Downstream Model Training
text: The maximum token length for each sequence is 144. We use the standard training practice and cross
entropy loss in PyTorch. All models are trained with NVIDIA GeForce RTX 2080 Ti GPUs.
We use DeBERTa and learning rate of 4e−5 due to cursory hyperparameter search via supervised learning. We searched over BERT, RoBERTa and DeBERTa (Devlin et al., 2019; Liu et al., 2019; He et al., 2023) and learning rate in [1e−5, 2e−5, 3e−5, 4e−5, 5e−5, 5e−6] and found this setup consistently yield relatively high F1 scores across all 4 datasets studied.

heading: D.4 Special Handling
text: CoNLL-2003 and WikiGold both contain over 1% negative samples where as MIT-Movie and MIT-Restaurant don’t. Thus, we include a negative demo sample for CoNLL-2003 and WikiGold (§4.1). We also add an explicit instruction in the data generation prompt for these datasets in case no named entities is found in the generated sentences (as shown in Table 1). For WikiGold, we used the same train/dev/test split as in Meng et al. (2021).
For MIT-Movie and MIT-Restaurant, we lowercase all generated NER samples before (deduplication and) downstream training, since (1) the original dataset samples are all in lowercase, and (2) lowercasing leads to noticeable gains (~10%). For MIT-Movie entity pool generation, we found that GPT-3.5-generated entities don’t belong to the respective entity classes for several classes, so we additionally provided entity definitions and some demos in the prompt as necessary.

heading: E Additional Data Statistics

heading: E.1 Diversity Requirements
text: We show summary statistics of the diversity requirement configurations for Diversify X and Diversify Y respectively in Tables 29 and 30.

heading: E.2 Sample Generation
text: We show summary statistics of generated datasets (before LLM Self-Correction) for Simple Prompt and our diversity approaches in Table 31. We observe consistent trends where we make the following (unsurprising) observations: (1) Diversify X leads to longer generated sentences, indicating more tokens are needed to satisfy the semantic requirements, (2) Diversify Y shows increased number of distinct named entities, (3) Simple Prompt in general domain have less samples processed, mostly due to de-duplication, and (4) Explicit diversity requirements yield improved diversity where
pe r. loc . org .
Target
per.
loc.
org.
So ur
ce 0 23
0 19
8 21 0
5
10
15
20
(a) CoNLL2003
pe r. loc . org .
Target
per.
loc.
org.
So ur
ce
0 8
8 31
0 16 0
10
20
30
(b) WikiGold
Re st.
Na me Am en
ity
Cu isin
e Dis h Ho urs
Loc ati
on Pri ce
Ra tin
g
Target
Rest. NameAmenity Cuisine
Dish Hours Location Price
Rating
So ur
ce
2 31 0 0 0 0 0
0 61 0 118 4 2 2
8 23 29 49 0 0 0
1 4 20 5 0 0 0
0 15 4 0 0 0 0
8 63 1 0 1 0 0
0 77 13 3 1 0 8
3 37 18 4 0 0 9
0
25
50
75
100
(c) MIT-Restaurant
Tit le Vie w.
Ra t. Yea r Ge nre Dir ect or MP AA Ra t. Pl ot Ac tor Tra iler So ng Re vie w Ch ar.
Target
Title View.Rat. Year
Genre DirectorMPAARat.
Plot Actor Trailer Song Review Char.
So ur
ce
0 1 24 0 0 0 0 0 2 0 2
2 3 37 0 6 3 1 0 1 16 2
2 0 4 0 0 0 0 0 0 0 0
9 4 0 0 2 43 1 1 3 0 3
1 0 0 1 0 0 2 0 0 0 0
0 7 0 5 0 0 0 0 0 0 0
2 0 1 10 1 0 1 1 4 2 4
1 0 0 0 0 0 0 0 2 0 6
5 0 0 2 0 0 1 0 0 0 1
2 0 0 4 2 0 0 7 0 0 0
3 67 0 14 2 0 3 0 1 4 0
2 0 0 4 1 1 0 1 0 0 0
0
20
40
60
(d) MIT-Movie
Figure 5: LLM Entity Type Self-Correction counts by dataset.
Diversify X+Y yields the highest diversity as expected.

heading: E.3 LLM Self-Correction
text: For each dataset, we show summary statistics for distribution of LLM correction types, across different diversity setups and across entity classes, in Tables 32 and 33. Entity type corrections to “other” are considered as entity annotation drops. For entity type corrections to relevant entity types within the dataset, we show the count of corrections in Figure 5.

heading: F Additional Prompts and Samples
text: In this section, we show (1) prompt templates in F.1, (2) some prompt and response samples for the CoNLL-2003 dataset for space constraints in F.2, (3) a small subset of the generated diversity configurations for each dataset in F.3, and (4) selected 1-shot demo samples and a few generated NER samples for each dataset and each diversity variant in F.4. The full prompts and LLM responses for (1) diverse requirement generation, (2) NER sample generation and (3) entity correction generation, and the corresponding processed data are all available in our repository20.
20https://github.com/StefanHeng/ProgGen/ reproduce

heading: F.1 Prompt Templates

heading: F.1.1 Diversify Sentence
text: Generate Attribute Dimensions What do you think are important attributes to generate some diverse {domain sample description}? Examples: {example attributes}... Let’s think step by step.
Generate Attribute Values Suppose you are {domain description}. List {#generate} different {attribute dimension} for {domain sample description}. Some examples are {example attribute values}.

heading: F.1.2 Diversify Entities
text: Generate Entities (vanilla) Suppose you are {domain description}. Please generate {#generate} diverse named entities in {domain sample description} that can be categorized as {entity class}.
Generate Entities (latent) Suppose you are domain + latent topic description. Please generate {#generate} diverse named entities in {domain sample description} that can be categorized as {entity class}.

heading: F.1.3 Few-Shot ICL baseline
text: Suppose you are {domain description}. You are given {domain sample description}. Please identify all named entities occurred that belong to one of the following entity types: {entity types in the dataset}. Please list such named entities with the corresponding entity types on the following line, in the order of occurrence.\n Here are some example annotations for your reference. Please follow this format.\n Examples:\n {(sentence, entity annotations) demo samples}\n ---\n Please annotate the following {domain sample prefix}:\n {domain sample prefix}: "{sample}"

heading: F.2 Prompt Examples for CoNLL-2003

heading: F.2.1 Diversify Sentence Generate Attribute Dimensions

heading: Prompt to ChatGPT
text: What do you think are important attributes to generate some diverse news from Reuters? Examples: topic, writing style... Let’s think step by step.

heading: Example Response from ChatGPT
text: Creating diverse news content, especially for a global news organization like Reuters, involves several key attributes. Let’s break them down step by step:
1. **Topic Selection**: Covering a wide range of topics is crucial. This includes politics, economics, technology, health, environment, culture, sports, and human interest stories. The goal is to cater to varied interests and provide a holistic view of global events. 2. **Geographical Diversity**: Ensuring that news covers different regions of the world is important. This involves not just focusing on major powers like the US, China, or the EU, but also giving attention to less covered areas like Africa, South America, and smaller Asian and European countries. 3. **Writing Style**: Reuters is known for its straightforward, factual reporting style. However, incorporating different writing styles like in-depth analysis, opinion pieces, and feature stories can enrich the content. 4. **Source Variety**: Utilizing a wide array of sources, from government officials to local activists, experts in various fields, and ordinary citizens, can provide a more comprehensive perspective on issues. 5. **Multimedia Content**: Incorporating videos, podcasts, and interactive graphics can enhance the storytelling and appeal to a broader audience.
6. **Inclusivity and Representation**: Reflecting diverse voices in terms of ethnicity, gender, age, socio-economic status, and cultural backgrounds in both the news stories and among the journalists. 7. **Fact-Checking and Accuracy**: Maintaining high standards of journalism by rigorously fact-checking information and presenting unbiased, accurate news. 8. **Language Accessibility**: Offering news in multiple languages can reach a broader global audience. 9. **Investigative Journalism**: Going beyond surfacelevel reporting to uncover deeper truths, especially in areas like corruption, human rights abuses, and environmental issues. 10. **Audience Engagement**: Encouraging reader interaction through comments, social media, and possibly reader polls or Q&A sessions. 11. **Ethical Reporting**: Upholding journalistic ethics by respecting privacy, avoiding sensationalism, and reporting responsibly on sensitive issues. 12. **Technology and Innovation**: Leveraging new technologies like AI, VR, or AR for more immersive and interactive news experiences.
By focusing on these attributes, Reuters or any other news organization can produce diverse, engaging, and high-quality news content that appeals to a global audience.

heading: Generate Attribute Values
text: Prompt to ChatGPT
Suppose you are a news writer for Reuters. List 30 diverse news categories for news articles. Some examples are [health, local stories, international events, environment, business, sports, politics, economics, breaking news, culture, science, technology, human interest stories].
Example Response from ChatGPT 1. World News 2. Politics 3. Technology 4. Sports 5. Entertainment 6. Health 7. Business 8. Environment 9. Breaking News
10. Human Interest Stories 11. Local Stories 12. Economics 13. International Events 14. Science 15. Education
16. Travel 17. Crime 18. Fashion 19. Food 20. Religion 21. Art 22. Music 23. Film 24. Literature 25. Automotive 26. Lifestyle 27. Weather 28. Wildlife 29. Education 30. Immigrant Stories

heading: Prompt to ChatGPT
text: Suppose you are a news writer for Reuters. List 8 different writing styles for news articles. Some examples are [op-eds, opinion pieces, feature stories, in-depth analysis].
Example Response from ChatGPT
1. Hard news reporting 2. Investigative journalism 3. Informative news articles 4. Human interest stories 5. Broadcast news 6. Long-form journalism 7. Photojournalism 8. Data-driven journalism

heading: F.2.2 Diversify Entities Generate Entities (vanilla)

heading: Prompt to ChatGPT
text: Suppose you are a news writer. Please generate 50 diverse named entities in news articles that can be categorized as location.

heading: Example Response from ChatGPT
text: 1. New York City
2. Eiffel Tower 3. Sahara Desert 4. Mount Everest 5. Great Wall of China 6. Rio de Janeiro 7. Sydney Opera House 8. Taj Mahal 9. Amazon Rainforest
10. Grand Canyon 11. Tokyo 12. Moscow 13. London 14. Paris 15. Rome 16. Antarctica 17. Hawaii 18. Dubai 19. Bali 20. Barcelona 21. African Savannah 22. Las Vegas 23. Machu Picchu 24. Venice 25. Yellowstone National Park 26. Cairo 27. Galapagos Islands 28. Himalayas 29. Easter Island 30. The Bahamas 31. Mount Kilimanjaro 32. Norwegian Fjords 33. Great Barrier Reef 34. Prague 35. Istanbul 36. Vatican City 37. San Francisco 38. Cape Town 39. Athens 40. Sydney 41. Bangkok 42. Buenos Aires 43. Montreal 44. Vancouver 45. Toronto 46. New Delhi 47. Cairo 48. Singapore 49. Sydney 50. Shanghai

heading: Generate Entities (latent)

heading: Prompt to ChatGPT
text: Suppose you are a news writer for Entertainment. Please generate 15 diverse named entities in news articles that can be categorized as person.
Example Response from ChatGPT 1. Tom Hanks 2. Beyoncé 3. Brad Pitt 4. Rihanna
5. Will Smith 6. Taylor Swift 7. Leonardo DiCaprio 8. Oprah Winfrey 9. Jennifer Lopez
10. Chris Hemsworth 11. Meryl Streep 12. Dwayne "The Rock" Johnson 13. Kim Kardashian 14. Hugh Jackman 15. Lady Gaga

heading: F.2.3 Few-Shot ICL baseline
text: Suppose you are a news writer. You are given a sentence from a news story. Please identify all named entities occurred that belong to one of the following entity types: [person, location, organization]. Please list such named entities with the corresponding entity types on the following line, in the order of occurrence.\n
Here are some example annotations for your reference. Please follow this format.\n Examples:\n 1. Sentence: "BASEBALL - DODGERS WIN FIFTH STRAIGHT." Named Entities: [DODGERS (organization)]\n 2. Sentence: "With 2.5 percent gross domestic product growth expected for 1996, new job growth should slowly lower the unemployment rate over the rest of the year." Named Entities: []\n 3. Sentence: "A.de Silva not out 49" Named Entities: [A.de Silva (person)]\n 4. Sentence: "Saudi Arabia executes Pakistani man." Named Entities: [Saudi Arabia (location)]\n ---
Please annotate the following sentence:\n Sentence: "Japan began the defence of their Asian Cup title with a lucky 2-1 win against Syria in a Group C championship match on Friday."

heading: F.2.4 LLM Self-Correction
text: We show annotation instructions and demo pools for the 3 entity classes: Person, Location and Organization. These are for illustrative purposes, encompassing annotation errors in all diversity variants. Please refer to our repository for the illustrative prompts for all datasets and the exact prompts used for each generated dataset.

heading: Person
text: Here are 3 sentences from news articles. Your task is to classify whether the span of text is a named entity of type person. In particular, for each query with a span of text enclosed in double braces, please classify the enclosed span of text in context of the query into one of the following categories: - (A). The span is a named entity of type person - (B). The span contains a named person entity but the span boundary is not precise - (C). The span is a named entity but the category is not person - (D). The span is not a named entity If the label is (B), you should provide the correct span boundary.
If the label is (C), you should provide the correct entity type. Pick an entity type from the following list: [location, organization, other].\n
A named person entity must be the name of a person. Only first names, last names, and full names are considered named person entities. A named person entity should not have any starting titles such as "President", "Prime Minister", "Mayor", "Professor" and "Dr". (Governmental, political or executive) titles such as "Prime Minister of Australia", "President of Iceland" and "CEO of Google" are not relevant named entities. General reference to a person or people such as "actress", "chef", "CEO", "woman", "high school student" are not named entities.\n
Here are some example span classifications for your reference. Please follow this format.\n Examples:\n 1. Sentence: "Canadian {{Prime Minister Justin Trudeau}} visits Indigenous communities." Text Span: "Prime Minister Justin Trudeau" Label: (B). Wrong Boundary. The correct span boundary is "Justin Trudeau".\n 2. Sentence: "Newly elected {{president}} pledges to address climate change." Text Span: "president" Label: (D). Not a Named Entity.\n 3. Sentence: "{{CEO of Amazon}} steps down from role." Text Span: "CEO of Amazon" Label: (D). Not a Named Entity.\n 4. Sentence: "{{CEO}} of Dubai-based company arrested for fraud." Text Span: "CEO" Label: (D). Not a Named Entity.\n ---
Please classify the following 3 spans of text:\n 1. Sentence: "{{Cindy Gruden}} appointed as the new CEO of a major automotive company." Text Span: "Cindy Gruden"\n 2. Sentence: "{{Justin Trudeau}} meets with CEOs of major tech companies to discuss investment opportunities in Canada." Text Span: "Justin Trudeau"\n 3. Sentence: "{{Angela Merkel}} visits Silicon Valley to discuss collaboration with automotive tech companies." Text Span: "Angela Merkel"\n

heading: Location
text: Here are 3 sentences from news articles. Your task is to classify whether the span of text is a named entity of type location. In particular, for each query with a span of text enclosed in double braces, please classify the enclosed span of text in context of the query into one of the following categories: - (A). The span is a named entity of type location - (B). The span contains a named location entity but the span boundary is not precise - (C). The span is a named entity but the category is not location - (D). The span is not a named entity If the label is (B), you should provide the correct span boundary. If the label is (C), you should provide the correct entity type. Pick an entity type from the following list: [person, organization, other].\n
A named location entity must be the name of a location. Generic location references such as "city hall", "community center", "downtown", "major cities" and "bank" are not named
entities. Categorical location descriptors such as "major airports" and "arts district" are also not named entities. Demonyms such as "American", "Chinese" and "Russian" are not relevant named entities. Events like "Olympics" and "Fashion Week" are also not relevant named entities. Hurricanes and other natural disasters are not relevant named entities.\n
Here are some example span classifications for your reference. Please follow this format.\n Examples:\n 1. Sentence: "The Prime Minister of {{Japan}} meets with world leaders to discuss economic cooperation." Text Span: "Japan" Label: (A). Correct Entity Annotation.\n 2. Sentence: "The European Union imposes sanctions on {{Russian}} officials over human rights abuses." Text Span: "Russian" Label: (C). Wrong Type. The correct entity type is other.\n 3. Sentence: "German Chancellor Angela Merkel meets with {{French}} President Macron." Text Span: "French" Label: (C). Wrong Type. The correct entity type is other.\n 4. Sentence: "New restaurant to open in {{downtown area}}, bringing jobs and economic growth." Text Span: "downtown area" Label: (D). Not a Named Entity.\n 5. Sentence: "Celebrities flock to {{Paris Fashion Week}} for the latest trends." Text Span: "Paris Fashion Week" Label: (B). Wrong Boundary. The correct span boundary is "Paris".\n ---
Please classify the following 3 spans of text:\n 1. Sentence: "Gang-related violence on the rise in {{New York City}}." Text Span: "New York City"\n 2. Sentence: "The {{Tokyo}} Marathon, one of the largest and most prestigious races in the world, has been postponed due to the ongoing pandemic." Text Span: "Tokyo"\n 3. Sentence: "{{Detroit}} police arrest three suspects in connection with a series of robberies in the downtown area." Text Span: "Detroit"\n

heading: Organization
text: Here are 3 sentences from news articles. Your task is to classify whether the span of text is a named entity of type organization. In particular, for each query with a span of text enclosed in double braces, please classify the enclosed span of text in context of the query into one of the following categories: - (A). The span is a named entity of type organization - (B). The span contains a named organization entity but the span boundary is not precise - (C). The span is a named entity but the category is not organization - (D). The span is not a named entity If the label is (B), you should provide the correct span boundary. If the label is (C), you should provide the correct entity type. Pick an entity type from the following list: [person, location, other].\n
A named organization entity must be the name of an organization. Generic organization categories such as "high school", "city council", "local community", "government", "company" are not named entities. Descriptive organization types such as "non-profit organization", "foreign government" and "global technology company" are also not named entities.
Demonyms such as "Chinese", "Russian", "Indian", "Australian" and "European" are not relevant named entities. Viruses such as "COVID-19" are also not relevant named entities. Products such as "iPhone", "Windows" and "Instagram" are not relevant named entities.\n
Here are some example span classifications for your reference. Please follow this format.\n Examples:\n 1. Sentence: "Indian government bans {{TikTok}} and 58 other Chinese apps." Text Span: "TikTok" Label: (A). Correct Entity Annotation.\n 2. Sentence: "{{CEO}} of Tesla Elon Musk Denies Securities Fraud Allegations" Text Span: "CEO" Label: (D). Not a Named Entity.\n 3. Sentence: "{{Japanese}} automaker Toyota recalls millions of vehicles." Text Span: "Japanese" Label: (D). Not a Named Entity.\n 4. Sentence: "{{Russian}} cosmonauts and Chinese astronauts conduct joint space mission." Text Span: "Russian" Label: (D). Not a Named Entity.\n 5. Sentence: "Pfizer announces new vaccine efficacy data against {{COVID-19}} variants." Text Span: "COVID-19" Label: (D). Not a Named Entity.\n 6. Sentence: "Local {{non-profit organization}} provides free meals to homeless veterans." Text Span: "non-profit organization" Label: (D). Not a Named Entity.\n ---
Please classify the following 3 spans of text:\n 1. Sentence: "The pharmaceutical company Pfizer announces a breakthrough in the development of a potential {{HIV}} vaccine." Text Span: "HIV"\n 2. Sentence: "Celebrities flock to {{Cape Town Fashion Week}} for the latest trends." Text Span: "Cape Town Fashion Week"\n 3. Sentence: "{{The National Gallery of Art}} opens new exhibit featuring works by Monet and Renoir." Text Span: "The National Gallery of Art"\n

heading: F.3 Diversity Requirement Sample Values
text: In this section, we show a few of the generated diversity requirement configuration values for illustrative purposes in Tables 34, 35 and 36.

heading: F.4 Generated NER Samples
text: In this section, we show the selected demo samples and a few generated NER samples from Simple Prompt and each diversity variant. We show the raw LLM-generated samples, which may not follow the diversity instructions and demo formatting. Note that the generated samples may contain wrong annotations. For diversity approaches, we additionally show the corresponding sampled configurations in the prompts.
Variant Group Sample Values
CoNLL-2003
Diversify X News Topic World News, Politics, Technology, ...Writing Style Breaking news reports, Investigative journalism, Human interest stories, ...
Diversify Y (vanilla)
per. Joe Biden, Angela Merkel, Donald Trump, ... loc. New York City, Paris, Tokyo, ... org. United Nations, World Health Organization, European Union, ...

heading: F.4.1 Demo Samples
text: CoNLL-2003 1. Sentence: "BASEBALL - DODGERS WIN FIFTH STRAIGHT." Named Entities: [DODGERS (organization)]\n 2. Sentence: "With 2.5 percent gross domestic product growth expected for 1996, new job growth should slowly lower the unemployment rate over the rest of the year." Named Entities: []\n 3. Sentence: "A.de Silva not out 49" Named Entities: [A.de Silva (person)]\n 4. Sentence: "Saudi Arabia executes Pakistani man." Named Entities: [Saudi Arabia (location)]\n

heading: WikiGold
text: 1. Sentence: ‘His book, A Biblical Case for an Old Earth was described in a review by Law Professor David W. Opderbeck, in the American Scientific Affiliation’s Perspectives on Science and Christian Faith as "succeed[ing] admirably" in "establish[ing] that the ‘day-age’ view is a valid alternative for Christians who hold to biblical inerrancy", but as "less persuasive" at "argu[ing] for a concordist understanding of the Genesis texts and modern science."’ Named Entities: [David W. Opderbeck (person), American Scientific Affiliation (organization)] 2. Sentence: "He scored one international goal, against Slovakia in 1994." Named Entities: [Slovakia (location)]\n 3. Sentence: "It was built as a self-contained steamship dock facility." Named Entities: []\n

heading: MIT-Movie
text: 1. Query: "what type of movie genre is the perfect weapon" Named Entities: [the perfect weapon (Title)]\n 2. Query: "how many movies came out in 2004" Named Entities: [2004 (Year)]\n 3. Query: "could you show me some part of the new indiana jones movie" Named Entities: [some part (Trailer), indiana jones (Character)]\n 4. Query: "when did mark joffe direct the bounty hunter film that is rated pg" Named Entities: [mark joffe (Director), bounty hunter (Plot), pg (MPAA Rating)]\n 5. Query: "is cary grant in any historical films that are a must see" Named Entities: [cary grant (Actor), historical (Genre), must see (Viewers’ Rating)]\n 6. Query: "show me a movie with the song a whole new world" Named Entities: [a whole new world (Song)]\n 7. Query: "what movie is considered the funniest of all time" Named Entities: [funniest of all time (Review)]\n

heading: MIT-Restaurant
text: 1. Query: "is there any place around here that has a good menu for happy hour" Named Entities: [good (Rating), happy hour (Amenity)]\n 2. Query: "where can i get afghan cuisine before 9 a m" Named Entities: [afghan (Cuisine), before 9 a m (Hours)]\n 3. Query: "call mcdonalds" Named Entities: [mcdonalds (Restaurant Name)]\n 4. Query: "find us a place to eat for under 10 dollars a plate" Named Entities: [under 10 dollars a plate (Price)]\n
5. Query: "im thirsty take me to the closest slurpee" Named Entities: [closest (Location), slurpee (Dish)]\n

heading: F.4.2 Simple Prompt
text: CoNLL-2003 1. The European Union is considering imposing tariffs on American products. Named Entities: [European Union (organization), American (location)]\n 2. Elon Musk’s SpaceX successfully launches another batch of Starlink satellites. Named Entities: [Elon Musk (person), SpaceX (organization)]\n 3. The protests in Hong Kong continue to escalate as tensions rise between the government and demonstrators. Named Entities: [Hong Kong (location)]\n ...

heading: WikiGold
text: 1. The Eiffel Tower is a wrought iron lattice tower on the Champ de Mars in Paris, France. Named Entities: [Eiffel Tower (location), Champ de Mars (location), Paris (location), France (location)] 2. John F. Kennedy International Airport is a major international airport in Queens, New York City. Named Entities: [John F. Kennedy International Airport (location), Queens (location), New York City (location)] 3. The World Health Organization is a specialized agency of the United Nations responsible for international public health. Named Entities: [World Health Organization (organization), United Nations (organization)] ...

heading: MIT-Movie
text: 1. Can you tell me the plot of the movie Inception?
Named Entities: [Inception (Title)]\n
Variant Group Sample Values
MIT-Restaurant
Diversify X
Meal Category fine dining, buffet, late-night dining, ... User Demographic 25-year-old female student, 40-year-old female accountant, 55-year-old male retiree, ...
Ambiance Casual, Outdoor seating, Family-friendly, ... Price Range Affordable, Budget-friendly, Mid-range, ...
Dietary Restriction Vegetarian, Vegan, Gluten-free, ... Special Offering Chef’s specials, Private Dining, Tasting Menus, ...
Service Mode takeout, delivery, dine-in, ...
Rest. Name La Cucina Italiana, The Steakhouse, Sushi Park, ... Amenity WiFi, Outdoor seating, Parking, ... Cuisine Italian, Thai, Chinese, ...
2. What is the viewers’ rating for The Shawshank Redemption?
Named Entities: [The Shawshank Redemption (Title), viewers’ rating (Viewers’ Rating)]\n 3. Show me movies from the 1980s.
Named Entities: [1980s (Year)]\n ...

heading: MIT-Restaurant
text: 1. "Can you recommend a good sushi restaurant nearby?" Named Entities: [sushi (Cuisine), nearby (Location)]\n 2. "What Italian restaurant has the best reviews?" Named Entities: [Italian (Cuisine), best (Rating)]\n 3. "I want to go to a steakhouse for dinner tonight." Named Entities: [steakhouse (Cuisine), tonight (Hours)]\n ...

heading: F.4.3 Diversify X
text: CoNLL-2003 Prompt to ChatGPT Additionally, the generated news article sentences should follow the requirements below: 1. The news should be about Education; 2. The writing style of the news should be human interest stories.
Example Response from ChatGPT
1. "Local high school student overcomes homelessness to graduate with honors." Named Entities: [high school student (person)]\n 2. "Non-profit organization partners with school district to provide free tutoring services for low-income students." Named Entities: [non-profit organization (organization), school district (organization)]\n 3. "Former teacher starts scholarship fund to help underprivileged students pursue higher education." Named Entities: [teacher (person), scholarship fund (organization)]\n

heading: WikiGold

heading: Prompt to ChatGPT
text: Additionally, the generated Wikipedia article sentences should follow the requirements below: 1. The topic of the Wikipedia article sentences should be Mathematics; 2. The writing style of the Wikipedia article sentences should be biographical.

heading: Example Response from ChatGPT
text: 1. In 1931, mathematician Kurt Gödel published his famous incompleteness theorems, which revolutionized the study of mathematical logic. Named Entities: [Kurt Gödel (person)]\n
2. As a pioneer in the field of algebraic geometry, David Mumford was awarded the Fields Medal in 1974 for his groundbreaking work on geometric invariant theory. Named Entities: [David Mumford (person)]\n 3. The Clay Mathematics Institute offers a prize of one million dollars for a correct proof of the Riemann hypothesis, one of the most famous unsolved problems in mathematics. Named Entities: [Clay Mathematics Institute (organization), Riemann (person)]\n

heading: MIT-Movie

heading: Prompt to ChatGPT
text: Additionally, the generated spoken movie queries should follow the requirements below: 1. The query should inquire about Movie showtimes and ticket availability; 2. The user should be adventurous.

heading: Example Response from ChatGPT
text: 1. "Can I buy tickets for the latest James Bond movie?" Named Entities: [latest James Bond (Title)]\n 2. "Is there a sci-fi movie with a soundtrack by Hans Zimmer playing at the theater today?" Named Entities: [sci-fi (Genre), Hans Zimmer (Song)]\n 3. "I’m looking for a movie directed by Christopher Nolan and rated R. Do you have any suggestions?" Named Entities: [Christopher Nolan (Director), R (MPAA Rating)]\n

heading: MIT-Restaurant

heading: Prompt to ChatGPT
text: Additionally, the generated restaurant queries should follow the requirements below: 1. The meal category should be fine dining; 2. The restaurant’s price range should be affordable.

heading: Example Response from ChatGPT
text: Query: "can you recommend a fine dining restaurant with affordable prices" Named Entities: [fine dining (Amenity), affordable prices (Price)]\n Query: "what are the top-rated fine dining restaurants in the area" Named Entities: [top-rated (Rating), fine dining (Amenity)]\n Query: "where can I find a fine dining restaurant with budgetfriendly options" Named Entities: [fine dining (Amenity), budget-friendly options (Price)]\n

heading: F.4.4 Diversify Y (vanilla)
text: CoNLL-2003 Prompt to ChatGPT Additionally, in the generated sentences, include the following terms: [United Nations Development Programme, Steven Spielberg].

heading: Example Response from ChatGPT
text: 1. "Steven Spielberg to direct new film about the United Nations Development Programme." Named Entities: [Steven Spielberg (person), United Nations Development Programme (organization)]\n 2. "United Nations Development Programme launches new initiative to combat climate change in developing countries." Named Entities: [United Nations Development Programme (organization)]\n 3. "Steven Spielberg’s latest movie receives funding from the United Nations Development Programme."
Named Entities: [Steven Spielberg (person), United Nations Development Programme (organization)]\n

heading: WikiGold

heading: Prompt to ChatGPT
text: Additionally, in the generated sentences, include the following terms: [Alexander the Great, Angela Merkel].

heading: Example Response from ChatGPT
text: 1. "Alexander the Great is often considered one of the most successful military commanders in history for the remarkable achievements of his campaigns." Named Entities: [Alexander the Great (person)]\n 2. "Angela Merkel, the former Chancellor of Germany, is known for her leadership during the European financial crisis and her role in the European Union." Named Entities: [Angela Merkel (person), Germany (location), European Union (organization)]\n 3. "The Louvre, located in Paris, is the world’s largest art museum and a historic monument in France." Named Entities: [Louvre (location), Paris (location), France (location)]\n

heading: MIT-Movie

heading: Prompt to ChatGPT
text: Additionally, in the generated queries, include the following terms: [Zooey Deschanel, Cyberpunk, TV-Y, Take Me Out to the Ball Game, 1968].

heading: Example Response from ChatGPT
text: 1. "Can you tell me if Zooey Deschanel is starring in any new comedy films" Named Entities: [Zooey Deschanel (Actor), comedy (Genre)]\n 2. "I’m looking for a cyberpunk movie released in 1968" Named Entities: [cyberpunk (Genre), 1968 (Year)]\n 3. "Is there a family-friendly movie with a TV-Y rating featuring the song ’Take Me Out to the Ball Game’" Named Entities: [family-friendly (Genre), TV-Y (MPAA Rating), Take Me Out to the Ball Game (Song)]\n

heading: MIT-Restaurant

heading: Prompt to ChatGPT
text: Additionally, in the generated queries, include the following terms: [Value meal, The Firehouse Grill].

heading: Example Response from ChatGPT
text: 1. "I’m hungry, can you recommend a restaurant with a good value meal?" Named Entities: [good (Rating), Value meal (Amenity)]\n 2. "What’s the address and hours of operation for The Firehouse Grill?" Named Entities: [The Firehouse Grill (Restaurant Name), address (Location), hours of operation (Hours)]\n 3. "I’m craving Italian cuisine, do you know a restaurant nearby that serves it?" Named Entities: [Italian (Cuisine), nearby (Location)]\n

heading: F.4.5 Diversify Y (latent)
text: CoNLL-2003 Prompt to ChatGPT Additionally, in the generated sentences, include the following terms: [Pandora Media, Sony Music Entertainment, Sony Music].
Example Response from ChatGPT
1. "Pandora Media announces partnership with Sony Music Entertainment." Named Entities: [Pandora Media (organization), Sony Music Entertainment (organization)]\n 2. "Sony Music to release new album in collaboration with a popular artist." Named Entities: [Sony Music (organization)]\n 3. "Pandora Media’s CEO discusses future plans for the company at annual conference." Named Entities: [Pandora Media (organization)]\n

heading: WikiGold

heading: Prompt to ChatGPT
text: Additionally, in the generated sentences, include the following terms: [Doctors Without Borders, Centers for Disease Control and Prevention].

heading: Example Response from ChatGPT
text: 1. Doctors Without Borders provides medical aid to over 70 countries around the world. Named Entities: [Doctors Without Borders (organization)]\n 2. The Centers for Disease Control and Prevention is responsible for monitoring and preventing the spread of infectious diseases in the United States. Named Entities: [Centers for Disease Control and Prevention (organization)]\n 3. In 2018, the organization partnered with the local government to provide healthcare services in rural areas. Named Entities: []\n

heading: MIT-Movie

heading: Prompt to ChatGPT
text: Additionally, in the generated queries, include the following terms: [Taylor Swift, X, Jessica Chastain, 2011, Patty Jenkins, Sam Mendes, Standout performances].

heading: Example Response from ChatGPT
text: 1. "What 2011 movie directed by Patty Jenkins received standout performances by Jessica Chastain?"
Named Entities: [2011 (Year), Patty Jenkins (Director), Jessica Chastain (Actor), Standout performances]\n 2. "Could you recommend a film with a Taylor Swift song in the trailer?"
Named Entities: [Taylor Swift (Song), Trailer]\n 3. "Show me a Sam Mendes movie with a suspenseful plot."
Named Entities: [Sam Mendes (Director), suspenseful (Plot)]\n

heading: MIT-Restaurant

heading: Prompt to ChatGPT
text: Additionally, in the generated queries, include the following terms: [Cheeseburger, Mexican, Discounted items].

heading: Example Response from ChatGPT
text: 1. "Can you recommend a place that serves a good cheeseburger?" Named Entities: [cheeseburger (Dish)]\n 2. "I’m craving Mexican food, where can I find a Mexican restaurant nearby?" Named Entities: [Mexican (Cuisine), nearby (Location)]\n 3. "Do you know of any restaurants that offer discounted items during happy hour?" Named Entities: [discounted items (Amenity), happy hour (Amenity)]\n

heading: F.4.6 Diversify X+Y
text: CoNLL-2003 Prompt to ChatGPT Additionally, the generated news article sentences should follow the requirements below: 1. The generated news article sentences should be about Art; 2. The generated news article sentences should include the following terms: [Colosseum, Rome, Public Art Fund, Art Institute of Chicago].

heading: Example Response from ChatGPT
text: 1. The Colosseum in Rome to host a new public art exhibition. Named Entities: [Colosseum (location), Rome (location), Public Art Fund (organization)]\n 2. Art Institute of Chicago unveils new digital art collection. Named Entities: [Art Institute of Chicago (organization)]\n 3. Renowned artist to create a new mural for the Public Art Fund in Rome. Named Entities: [Public Art Fund (organization), Rome (location)]\n

heading: WikiGold

heading: Prompt to ChatGPT
text: Additionally, the generated Wikipedia article sentences should follow the requirements below: 1. The topic of the Wikipedia article sentences should be Music; 2. The generated Wikipedia article sentences should include the following terms: [Bob Dylan, Rock and Roll Hall of Fame, Berklee College of Music].

heading: Example Response from ChatGPT
text: 1. Her rendition of "Like a Rolling Stone" by Bob Dylan earned her a standing ovation from the audience. Named Entities: [Bob Dylan (person)]\n 2. The band was inducted into the Rock and Roll Hall of Fame in 2006, in recognition of their influence on the music industry. Named Entities: [Rock and Roll Hall of Fame (organization)]\n 3. Many successful musicians and artists, including Quincy Jones and Melissa Etheridge, have attended the prestigious Berklee College of Music. Named Entities: [Quincy Jones (person), Melissa Etheridge (person), Berklee College of Music (organization)]\n

heading: MIT-Movie

heading: Prompt to ChatGPT
text: Additionally, the generated spoken movie queries should follow the requirements below: 1. The query should inquire about Soundtrack and music information; 2. The user should be ambivalent; 3. The generated spoken movie queries should include the following terms: [Thrilling musical arrangements, EDM, Rock and roll legend, Captivating theme music, I Wanna Dance with Somebody, Ska].

heading: Example Response from ChatGPT
text: 1. "I’m looking for some thrilling musical arrangements in a movie. Can you recommend anything with EDM or rock and roll legend in the soundtrack?" Named Entities: [thrilling musical arrangements (Song)]\n 2. "I’m in the mood for some captivating theme music. Do you know any movies featuring ’I Wanna Dance with Somebody’ or Ska?" Named Entities: [captivating theme music (Song), I Wanna Dance with Somebody (Song), Ska (Genre)]\n
3. "I’m not sure what I’m in the mood for. Can you show me a movie with an awesome EDM soundtrack?" Named Entities: [EDM (Song)]\n

heading: MIT-Restaurant

heading: Prompt to ChatGPT
text: Additionally, the generated restaurant queries should follow the requirements below: 1. The meal category should be fine dining; 2. The restaurant’s price range should be affordable; 3. The generated restaurant queries should include the following terms: [Near me, Valet parking, Blue Hill at Stone Barns, High-end].

heading: Example Response from ChatGPT
text: Query: "Can you recommend a fine dining restaurant near me with valet parking?" Named Entities: [fine dining (Amenity), near me (Location), valet parking (Amenity)]\n Query: "I’m looking for an affordable high-end restaurant with valet parking, maybe something like Blue Hill at Stone Barns" Named Entities: [affordable (Price), high-end (Price), valet parking (Amenity), Blue Hill at Stone Barns (Restaurant Name)]\n Query: "Where can I find a fine dining restaurant around here with valet parking and a price range that’s affordable?" Named Entities: [fine dining (Amenity), around here (Location), valet parking (Amenity), affordable (Price)]\n

heading: Future Work
text: "Pandora Media’s CEO discusses future plans for the company at annual conference. " Named Entities: [Pandora Media (organization)]\n