LLMaAA: Making Large Language Models as Active Annotators
Authors: Ruoyu Zhang
Yanzeng Li
Yongliang Ma
Ming Zhou
Lei Zou
Venue: EMNLP 2023 Findings (2023)
Keywords: Large language model
psuedo labeling
active learning
TL;DR: We explore the potential of LLMs as annotators, with active data selection to enhance efficiency and automatic reweighting to promote reliabilty.

Abstract: Prevalent supervised learning methods in natural language processing (NLP) are notoriously data-hungry, which demand large amounts of high-quality annotated data. In practice, acquiring such data is a costly endeavor. Recently, the superior few-shot performance of large language models (LLMs) has propelled the development of dataset generation, where the training data are solely synthesized from LLMs. However, such an approach usually suffers from low-quality issues, and requires orders of magnitude more labeled data to achieve satisfactory performance.
To fully exploit the potential of LLMs and make use of massive unlabeled data, we propose LLMaAA, which takes LLMs as annotators and puts them into an active learning loop to determine what to annotate efficiently. To learn robustly with pseudo labels, we optimize both the annotation and training processes: (1) we draw $k$-NN examples from a small demonstration pool as in-context examples, and (2) we adopt the example reweighting technique to assign training samples with learnable weights.
Compared with previous approaches, LLMaAA features both efficiency and reliability.
We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples, which is much more cost-effective than other baselines.

LLMAAA: Making Large Language Models as Active Annotators
Ruoyu Zhang'; Yanzeng Li', Yongliang Ma', Ming Zhou', Lei Zou'
Wangxuan Institute of Computer Technology, Peking University, Beijing, China LLangboat Technology, Beijing, China
{ry_zhang, zoulei Pepku. edu. cn, liyanzengêstu. pku. edu. cn, (mayongliang,
com
Abstract
Unlabeled
Labeled Data
Data
Human Annotator
Prevalent supervised learning methods natu- ral language processing (NIP) are notoriously data-hungry, which demand large amounts of high-quality annotated data. In practice, acquir- ing such data is costly endeavor. Recently, the superior performance of large language mod- els (LLMs) has propelled development of dataset generation, where the training data solely synthesized from L. However, such an approach usually suffers from low-quality is- sues and requires orders of magnitude more la- beled data to achieve satisfactory performance. To fully exploit the potential of LLMs and make use of massive unlabeled data, we propose LL- MAAA A, which takes LLMs as annotators and puts them into active learning loop to deter- mine what to annotate efficiently. To learn ro- bustly with pseudo labels, we optimize both annotation and training processes: (1) we draw k-NN samples from a small demonstration pool in-context examples, and (2) we adopt the automatic reweighting technique assign train- ing samples with learnable weights. Compared with previous approaches, ILLMAA features both efficiency and reliability. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation traction. With LLMAAA A, task-specific models trained from LLM- -enerrete labels can outper- form their teacher I.LMss within only hundreds annotated examples, which is much more cost-effective than other baselines'.
(a) Human annotation as supervision.
In-Context
Labe led
Examples
Prompt
Data
(b) Text generation supervision.
In-Context Examples
Unlabeled
Labeled
Specific Model
PLM
Active Data
Acquisition
(c) LLMAAA: Active LLM annotation as supervision.
Figure Comparing LLMAAA with other frameworks. We actively acquire annotations from LLM for effi- ciency, requiring little human effort.
vancements comes vast potential risks in adopt- ing LLMs for widespread downstream produc- tion applications. One of the main concerns is about data privacy and security. Under the preva- lent "Laangaag--Model-ae as--eeiceee (LMaaS, Sun et al., 2022) setting, users are required to feed their own data, potentially including sensitive or private information, to third-party LLM vendors to access the service, which increases the risk of data leak- age (Lyu et al., 2020; Yu al., 2022; Li et al., 2023). Besides, LLMs usually consume abundant tokens by continuous requests to APIs, where the marginal cost and latency become substantial in large-scale or real-time applications, hindering LLMs' practi- cal deployment in cost-sensitive scenarios (Goyal et al., 2020; Cao et al., 2023).
1 Introduction
Large language models (LLMs) have exhibited re- markable few-shot performance in wide range of tasks, with only few demonstrations and well- designed prompts (Brown et al., 2020; Ding et al., 2022: Liu et al., 2023). However, with rapid ad-
On the other hand, training task-specific mod- els (TAMs) for NLP tasks necessitates extensive amounts of labeled data. Due to the superior gener- ative capacity of L.Mss, some researchers attempt to synthesize training data with text generation (Meng et al., 2022: Ye et al., 2022), as depicted
This work was done during an internship at Langboat Technology.
'Our code and data are available at https://github. LUMAAA

---

in Figure 1. However, the generated text usually struggles with low-quality issues and may exhibit domain shifts with test data (Gao et al., 2023). To exploit the abundant unlabeled corpus, an alterna- tive is to employ LLMs as annotators, which gener- ate labels in a zero-shot or few-shot manner. While this approach seems promising, it is important to acknowledge that LLM-generated labels inevitably contain noise, especially when applied to challeng- ing tasks and domain-specinn data (Agrawal et al., 2022; Kazemi al., 2023). Besides, larger models come with heavier expenses, and it is also crucial to reduce the annotation cost when the budget is restricted.
usually require large-scale synthetic training data (size varying from 10k 200k, Ye et al., 2022: Gao et al., 2023). These results confirm the potential of LLMAAA as a practical and cost-efficient solu- tion to make LLMs as good annotators. The TAMs created through our framework offer advantages in terms of task-specific performance, data privacy, and inference costs, which release the capacity of LLMs for real-world productivity.
We summarize our contributions as follows:
We propose LLMAAA, framework to employ LLMs as annotators, featuring both efficiency and reliability.
To enhance the reliability (i.e. accuracy) of TAMs' performance as well as to ensure the data efficiency annotation cost, we propose LL- MAAA, an innovative framework that integrates active learning into the LLM annotation process, i.e., making LLMs as Active Annotators. By ex- ploring different active acquisition strategies, LL- MAAA enables the LLM annotate more informa- live instances that benefit model performance more, To train TAMS reliably, we optimize both the an- notation and training processes within LLMAAA framework. Firstly, we employ prompt engineer- ing techniques to enhance LLMs performance by (I) selecting '-NN samples from a demonstration pool as in-context examples, and (2) building fine- level descriptions aligned with natural language for unnatural labels (e.g., category labels in the RE task). The valuable contextual information helps improve the quality of LLM annotations substan- tially. During training, we adopt the automatic reweighting technique (Ren et al., 2018) to assign learnable weights to the silver? training samples. This strategy allows the model to prioritize more informative and representative samples while si- multaneously reducing the impact of noisy annota- tions.
LLMAAA is capable to train TAMs that outper- form teacher LLMs within hundreds of annotated samples, on classic NLP tasks like NER and RE
LLMAAA sheds light on the practical substi- tution of LLMs, with a cost-effective, privacy- ensured, yet well werf-peefo solution.
2 Related Work
LLM and In-Context Learning Large language models (LLMs), usually pretrained on large-scale corpus to capture rich linguistic patterns and gener- ate coherent text (Brown et al., 2020; Raffel et al., 2020; Chowdhery et al., 2022; OpenAI, 2023; Tou- vron et al., 2023), have shown remarkable perfor- mance in a wide range of NLP tasks (Min et al., 2021; Zhao et al., 2023). With the proposal of in-context learning (Brown et al., 2020), prompt engineering has been extensively explored steer LLMs'' behavior for desired outcomes. These tech- niques design specific prompts or instructions to guide models' outputs (Ding et al., 2022: Liu et al., 2023), either in rulc-based (Shin et al., 2020) or learning bearningbbased based (Lesterr et al., 2021) manners. Re- cent trend focuses the strong reasoning capabili- ties of LLMs and enhances LLMs' performance complex task with chain-of-thought (CoT) prompt- ing (Wei al., 2023a). In general, prompt engineer- ing improves the controllability and performance of LLMs in sho-shot and zero-shot settings (Zhong et al., 2021), and enables LLMs to solve spe- cific tasks, e.g. information extraction (Wei et al., 2023b; Wang et al., 2023).
We evaluate LLMAAA on two practical NLP tasks: named entity recognition (NER) and relation extraction (RE). Experiments show that: (1) with small-scale gold data (~100 examples) serving for demonstration and validation, the trained TAMs can outperform their teacher LLMs within hun- dreds of silver samples via LLMAAA; (2) our approach is significantly more data efficient com- pared to prevalent data generation methods, which
Dataset Synthesis Supervised learning methods in NLP are often limited by high-quality anno- tated data. To address the bottleneck, researchers have explored techniques to synthesize training
"Wo refer gold data to ground-trunt found--ruth/human-noo labeled sam- ples, and silver data to L-bM-lboled samples.

---

Nick
Completion
PCS pesaaindatt
Large unlabeled data
LLM Annotator
oer Sew Poeer Meer "bu""U00 (28)
Rod
Small labeled data
You poel
Acquistion
Training
Task-Specific Model
Figure 2: LLMAAA puts the LLM annotator in an active learning iteration, which mainly consists of three novel components: (1) an LLM annotator optimized with prompt engineering that generates pseudo labels, (2) active acquisition mechanism for efficient data selection, and (3) automatic reweighting technique to ensure robust learning with noisy labels. The annotation training stages run iteratively and gradually produce labeled data for task-specific models.
data with LLMs, either by annotation or by gen- eration. Following the first line of research, Feng et al. (2021); Chen et al. (2023) employ LLMs unsupervised annotators to generate dialogue datasets. Recently, AnnoLLM (He et al., 2023) makes LLMs' performance par with crowd- source annotators by thain-of-thhuuht prompting and self- eeff-generated explanations. As a contempo- rary work, Bansal and Sharma (2023) use LLMs for annotation the domain transfer setting. Under the formulation of active learning, they propose a new metric, conditional informativeness, that works well with noisy labels. Among generation- based methods, Wang al. (2021) first use LLM with few-shot prompts generate training data. Schick and Schütze (2021) attempt to generate la- beled text counterparts and text pairs for seman- tie textual similarity tasks. ZEROGEN (Ye et al., 2022) and SUNGEN (Gao et al., 2023) further ex- tend this practice to zero-shot learning by train- ing small models with zero-shot L..M-generated datasets. However, these approaches still suffer the low-quality and domain-shiff issues of the synthetic data, and none of them consider the cost efficiency of data generation via LLMs.
jectives (Du et al., 2017; Yang et al., 2017; Ash et al., 2020; Margatina et al., 2021). The selected samples are annotated by human annotators and then added into the labeled dataset iteratively.
3 LLM as Active Annotator
To exploit LLMs' superior few-shot performance and leverage abundant unlabeled data, we attempt to take LLM as annotator and train task-specific models for inference. An ideal process should be both efficient and reliable: we want to learn TAMs robustly with minimal LLM-generated labels.
Concretely, our solution is to make LLMs Active Annotator. As shown in Figure 2, LL- MAAA comprises three key components: (1) LLM annotator that generates pseudo labels of given data, (2) an active acquisition mechanism for efficient data selection, and (3) an automatic reweighting technique to ensure robust learning with noisy labels. LLMAAA iterates the three stages to gradually produce stronger TAMs.
3.1 Optimizing LLM as Better Annotator
In-context lcarning (i.e. PROMPTING) enables LLM to conduct few-shot inference without fine- tuning. Given manually-designed prompt (-. demonstration set S {xl, >' and the query example Xg. PROMPTING first builds sentence T(S, xy), conditioned on which LLM then gener- ates a text sequence
Labor Efficiency and Active Learning Active learning is a technique proposed to minimize the annotation cost during the labeling process (Set- tles, 2009; Ren et al., 2021). A popular setting for active learning is the pool- pool-based paradigm, which aims to select the most beneficial samples from unlabeled data pool based on criteria including uncertainty (Lewis and Gale, 1994; Houlsby et al., 2011; Gal et al., 2017), diversity (Huang et al., 2010; Sener and Savarese, 2018), and hybrid ob-
Yg argmaax (MM,
Finally, Yy is mapped to the label space Y
Despite the decent abilities, previous studies show that the design task-specific prompts has

---

large impact on performance. varying between near state-of-the-an and random guess (Gao al., 2021; al., 2022b). Finding the best prompts for given tasks and given data points is intractable. However, there are several principles turn out to be effective, compared with plain instruction.
strategies for thorough comparisons, and illustrate with classification task for simplicity?.
Random We consider random selection as base- line, which samples uniformly from D pooo Typi- cally pool data and test data share the same distri- bution, thus the sampled batch is expected be i.idd. with test data.
k-NN Example Retrieval To select good in- context examples, Liu al. (2022) propose k-NN retrieval strategy, which first embeds the demon- stration pool Demo and query sample vector representations, and then retrieves the nearest neighbors the query to form its exemplars. The rationale behind this is that semantically similar examples may help LLM answer the query better. Following their practice, we use Sentence-BERT (Reimers and Gurevych, 2019, 2020) build the representations.
Maximum Entropy Entropy is one of the most widely used estimations of uncertainty (Settles, 2009). Data for which the model M has the highest entropy are sampled for annotation according to
argmax
(M)())x) log Par(y)x. yCy
Σ
xEDpool
Culotta and McCallum (2005) propose sort examples with the probability as- signed by M to predicted class j, which samples
Least Confidence
Label Verbalizer classification tasks, the sur- face forms of labels may induce difficulties and ambiguities. Taking relation classification for in- stance, the label parents" can indicate ei- ther "subject is the parent of object" or "object is the parent of subject", depending its defini- tion. To address this problem, we utilize a label verbalizer to transform the surface forms natural language descriptions with pre-defined templates (Sainz et al., 2021; Lu et al., 2022a), serving as fine-level guidance. The semantic templates we use are shown in Table 7.
argmax (1 (o)x)
xED
K-Means Diversity sampling intends to select batches of data that is heterogeneous in the feature space. Following Yuan et al. (2020), we apply k- means clustering to the /2-normalized embeddings of M and sample the nearest neighbors of the k cluster centers.
3.3 Robust Learning with Noisy Labels
LLM annotators inevitably produce noisy labels, especially with harder tasks and domain-specific data. To stay robust against training label bias, we adopt the automatic reweighting technique (Ren et al., 2018) to assign different weights to training examples adaptively.
3.2 Active Data Acquisition
Active learning (AL) seeks to reduce labeling ef- forts by strategically choosing which examples to annotate. We consider the standard pool-based setting, assuming that a large pool of unlabeled data Dpool is available. AL loop starts with a seed labeled set Diabeled- At each iteration, we train model M on Diabeled and then use acquisition function f(, .M) to acquire batch B consisting of b examples from Ppool We then query the LLM annotator to label B. The labeled batch then re- moved from the pool Poool and added to labeled set Diabeled, and will serve as training data for the next iteration. The process is repeated for times.
We assume that small-scale validation set D with clean labels (e.g. human annotations) is avail- able throughout learning, with |Dvail < [D pPpool]. Concisely, automatic reweighting learns sample weights w by a meta-learning objective that min- imizes validation loss w.r.t. w, and uses online approximation to eliminate the nested loop of opti- mization. The training process of TAM is shown in Algorithm 1.
Tasks
Active acquisition strategies generally maxi- mize either uncertainty or diversity. On one hand, uncertainty- methods leverage model predic- tions to select hard examples. On the other hand, diversity-basee methods exploit the heterogeneity of sampled data. We will cover some common
We instantiate LLMAAA with two tasks: named entity recognition (NER) and relation extraction
'Adaptation to other settings (e.g. sequence tagging) will introduced in "We use BERT family as M encoder, and the embeddings refer to BERT output.

---

Algorithm I: Automatic Reweighting
Model We use the same model architecture Baldini Soares al. (2019), which first encloses entity spans with special tokens [E] and [[EE], then encodes the sentence with BERT. The concatenated embedding of subject and object is fed into a linear classifier with activation for final prediction.
Input: Noisy data Dirain. clean data vall. batch size n, m, initial parameter Bo.
step S
Output: Trained parameter 05
for 8 = S do
Brain + SampleBatch D Dirrain n)
Experiments and Analysis
Bval -SaampeeBatch((aaa m)
(Vrrain) /2= < (ooward(Burinn 0g)
5.1 Setup
// build computation graph with automatic differtiation
Dataset We experiment with three different NLP datasets: Chinese OntoNotes 4.0 (Weischedel al., 2011) and English CoNLL03 (Tjong Kim Sang and De Meulder, 2003) for NER, and Re-TACRED (Stoica et al., 2021) for RE. For Re-TACRED, we select subset describing personal relationships and balance the NA relation instances to the orig- inal portion. Details of dataset statisties are de- scribed in Appendix A. We report the precision, recall, and micro for both tasks.
E + 0; brain f Ei=1 efhraain Mirain)
V0,- - Backward (Ierain 0.)
0, t 02- avo,
(Whal) 1m +- Forward (Bvll,
lall m Ei=1 C(Vinl Vial)
Ve + Backward(haat
// truncate weights to zero, and normalize to one
W + maxmaaxeee
Baselines We compare LLMAAA with the fol- lowing baselines: (1) PROMPTING. The prompt- based direct inference on test data, using the same engineering techniques LLMAAA's teacher LLMs. (2) SUPERVISED. The TAMs trained clean-labeled data vat used in LLMAAA' demoosttration/vallaatio (3) ZEROGEN (Ye al., 2022). Zero-shot data synthesis method via text generation. (4) FEWGEN. data synthesis method that enhances ZEROGEN with in-context examples uniformly sampled from the demonstration pool.
W Σ, 10+0(1 18)
Iraii + Ei=1 wiR WVr(Wirain Werai)
00,- Backward (aaandd(rrrai (ttrain 0g)
0.+1 OptimizerStep(Os, v0a)
(RE). We opt for two simple yet effective models as TAMs, and leave other choices for future study.
4.1 Named Entity Recognition
Formulation NER aims to extract entities {ei} from text x, where e; can be expressed as con- tinuous span of sequences with predefined type. We consider the flat scenario (i.e. no overlapping entities), in which NER can be reformulated as a sequence tagging problem with BIO label.
Implementation We use ChatGPT as LLM an- notator for main experiments, and adopt BERT (Devlin al., 2019; Cui al., 2021) as TAM's en- coder. We also explore with other LLM annotators, GPT-3 (Brown et al., 2020) and GPT-4 (OpenAI, 2023), in $ 6. We randomly sample 100 exam- ples from the original validation sets as gold data, reusing the same set for demonstration Ditem and validation D We use the original training sets as D Pool and randomly initialize seed labeled set Diabeled with size of 50 and acquire 50 samples per batch for 9 iterations, which generates 500 sil- ver annotated samples in total. We generate 500 and 5,000 samples via ZEROGEN and FEWGEN for comparison. TAMs under all settings are trained three times with different random seeds, and we report the mean and standard deviation in the re- sults. The training process and hyperparameters detailed in Appendix B.
To smoothly adapt uncertainty-basee active func- tions from classification task to sequence tagging, we provide three pooling options: average, sum, and max. In practice, adopt average and sum operations for better empirical performance.
Model Following Devlin et al. (2019), we lever- age BERT to convert tokens into vectorized fea- tures, and use a linear classifier with activation to predict the (class)-BIO label for each token.
4.2 Relation Extraction
Formulation Given subject entity Csub and ob- ject entity Gobj in a sentence, RE classifies their relation into a predefined set R U (NA).
hhttps:///penaa..om//l//gg//////

---

Chinese OntoNotes 4.0
English CoNLLO33
Re- Re-TacRED-subeee
Avg.
Method
#Data
R
F1
P
R
F1
P
R
F1
PROMPTING
100/-
67.72
74.02
70.73
79.18
83.59
81.33
64.21
86.68
73.77
75.28
SUPERVISED
100/-
m
75.661.44
73.00h
77.166.1
78.52as1
77.94
62.36x
91.8%
74.28
1,000
62.10
71.876m
66.621m
71.14xa
71.10mm
71.07a.
61.60mm
78.25
68.57
68.75
ZEROGEN
-/ -55000
62.00
72.84230
66.97as
74.233.32
71.78
72.992s1
41,46
94,28
66.57a.6
68,84
100/ 500
71,78434
71.06
71.35
73.062.31
69.87223
71.43g21
69 69.21
77 27.84
73. 12,.6
71.97
FEWGEN
100/5000
68.05ga1
77.1
71.43a.2
75.93267
72.93
74.40 20
68.07305
92.244,33
78.20u
LLMAA.-rradom
100 500
68, 85.35
71.632
70.21200
77.69t.11
80.75
79. 19.12
63,233,60
97.752xs
76.41648
75.26
LLMAAA condidence
100/500
72.662.1
75.49 o7
74.0thu44
82.91axs
82.78kwi
82.844
71.49476
93.28%
80..9223
79,21
Table I:
Evaluation results for LLMAAA and other baselines across three different datasets, using ChatGPT as LLM backbone. We report the mean and standard deviation of three separate runs for each method. Since we set the temperature 0 in PROMPTING, its results are deterministic and we only evaluation once. We also denote the amount of data (gold/silver) that TAM used for training.
Relation
Generated Data
Marysubj's 's father is Adamobj.
per:parents
Tomanbj mother, Maryobj. lives in New York.
Michelle Obamaaubj parents are Fraser C. Robinson III and Marian Shields Robinsonabj-
Mikesubj 's son is named Jack
per:children
Libysubiss children are Alex and Bella.bj
Sarah has a daughter named Emily
Table 2: A case study of generated data with ZEROGEN Re-TACRED. We leverage ChatGPT the text generator, and the full prompts we use can be found in Appendix B.3.
We follow consistent principles in prompt de- sign. Empirically, we find that in-context examples bring marginal benefit to RE, while label verbalizer is a technique specifically designed for the classi- fication task. Therefore, We apply k-NN example retrieval to NER and label verbalizer to RE sepa- rately. We set & to 5 for all experiments, including GEW Refer to Appendix B.3 for full prompts.
dation on unlabeled data, which benefits gener- alization.
We also notice that generation-based methods, i.e. ZEROGEN and FEWGEN, fail to establish on- par results, even with 10x more data in zero-shot setting. We argue that the text-generation abilitics of LLMs are exaggerated in complex scenarios. To demystify the illusion, we devise a case study on Re- Re-TACRED, is shown in Table 2. ZEROGEN tends to generate simple templated sentences that deviate from the news domain, i.e. the original corpus of Re- Re-TACRED. These results may induce low-quality and domain-shift issues that hamper TAMs' performance. FEW GEN's performance im- proves with in-context examples, however, it still lags far behind LLMAAA. In contrast, exploit- ing the unlabeled data effectively alleviates these problems with much higher efficiency, where only hundreds of annotated samples are sufficient for satisfactory performance.
5.2 Overall Results
Table 1 denotes our main experiment results. LL- MAAA with least confidence as acquisition func- tion outperforms all comparative baselines across all datasets, with 74.00%, 82.84% and 80.79% Fl scores on Chinese OntoNotes 4.0, English CoNLLO33 and Re-TACRED-subee respectively. Comparing with PrompTiNG (i.e. the LLM an- notator), LLMAAA shows steady improvement (4% in average score) with TAMs of much fewer parameters and lower inference latency, indicating that LLMAAA provides decent substitute for LLMs in real-world deployments. LLMAAA also surpasses SUPERVISED, where TAMs are trained on clean-labeled but smaller-scale data. This sug- gests that LLMAAA is capable of deriving rich knowledge beyond the limited demonstration/vali-
5.3 Ablations
5.3.1
Effects of Prompt Engineering
Though ChatGPT can well follow human instruc- tions in general, it still struggles with difficult tasks and domain- sppeiffc data. We compare the infer-

---

15.2
02.
2.00
00.
602
bibeed crmple
Maximum Entropy
Least Confidence
Neers
Rendom
Figure 3: LLMAAA's performance with different active acquisition strategies, shown by F1 scores. The dashed lines denote PROMPTING's 's results. For each method, we report the mean and standard deviation of three runs initialized with different random seeds.
Cotoofictus 4.0. Entrogy
Chiness OntorMotss Confidencs
K-Maars
Random
12.5
67.5
67.5
05.3
05.0
62.5
660
55.9
N7.
English CoNLLOS, Nucinum Entrayy
Englis CohLies, Leual Confidence
Englih CoNLLOS, MoMuurs
English CoNiLLS, Randum
00
70
10
Te Entropy
RetSCRiDeeuinee Confidence
Ie-AACEED-bbbet
DeeIACRDDecbbet
60
00
11-009
170
220
303
472
107
200
300
432
500
120
200
500
500
300
200
315
600
500
rewelghtinng
W/o rowclghting
Figure 4: Results for analyzing the effects of automatic reweighting. We remove the aaline-approximatee sample weights and train TAMs with standard loss objectives for ablation. The dashed lines denote PROMPTINg's performance. For each method, we report the mean and standard deviation of F1 scores within three different runs.
OntoNotes
CoNLL
Re-TacRED
LLMs' zero-shot performance is unsatisfactory. In addition, the label verbalizer can help align unnatu- ral labels with natural language descriptions, which improves the performance in RE (from 70.94% to 73.77% in F1). These findings emphasize that prompt engineering is crucial for building strong annotators, and incorporating similar and aligned contexts contributes to better inference.
Base Instruction 49.62
55.74
70.94
+4-NN Examples
70.73
81.33
+Label Verbalizer
73.77
Table 3:
Comparison results between plain instructions and optimized prompts in F1 scores.
ence performance of plain instructions with opti- mized prompts in Table 3. Without k-NN example retrieval module (i.e. in zero-shot manners), the LLM annotator is unable to extract entities well in NER task, shown by drastic drop in F1 scores (21% % on OntoNotes and 25% on CoNLL). This re- sult highlights the need for demonstrations, where
5.3.2 Accelerating with Active Learning
Figure 3 shows LLMAAA performance with dif- ferent active learning strategies across all datasets. Uncertainty- methods, i.e. maximal en- tropy and least confidence, perform significantly better than the random baseline, with faster con-

---

Backbone
Method
R
6
Analysis
PROMPTING
41.82
22.77
29.49
GPT-3
LLM-AA-confddnee
57.26
56.09
56.63
6.1
LLMAAA with Different Annotators
PROMPTING
67.72
74.02
70.73
ChatGPT
L.MAAA-confidence
72.66
75.49
74.00
To guarantee the universal effectiveness of LL- MAAA, we further investigate the performance with other LLM annotators, i.c. GPT-3 (Brown et al., 2020) and GPT-4 (OpenAI, 2023). Due to budgetary considerations. we opt to restrict our ex- periments to OntoNNtes. The precision, recall and Fl score are shown in Table 4. The results indicate that LLMAAA benefits from better annotators with continuous improvements, and more importantly, TAMs trained by LLMAAA outperform the LLM annotators consistently. The student outperforms the weak teacher by large margin (27% in F1 for GPT-3). As the teacher grows stronger, this gap narrows down. This trend meets our expectations: since student TAMs are trained with fixed budget of data (500 samples), enhancing the capabilities of teacher LLMs will gradually approach the per- formance ceiling of the students. More annotation budget and more powerful TAMs can help extend this limit, while we leave the exploration for future research.
PROMPTING
68.70
79.42
73.68
GPT4
LLMAAA-cooffddnce
73.47
76.42
74.90
Table 4: Results on Chinese OntoNotes 4.0 for PROMPT- ING and LLMAAA with different LLMs. LLMAAA least confidence the acquisition function, and annotates 500 samples for TAM training.
vergence and higher FI scores at the end of itera- tions. It is worth noting that (I) incertainty-basel methods are able to achieve on-par performance with random selection with only 30%-40% train- ing data, (2) they surpass PROMPTING consistently within 500 LLM-annotated training samples. In summary, uncertainty-bause active learning strate- gies enable ILLMAA to be more efficient and more capable.
Though k-means clustering encourages diversity in feature space, it only outperforms random sam- pling OntoNotes, while yielding similar results CoNLLO3 and Re-TacRED. This suggests that it may require more training data for finetuned BERT to learn informative representations, and such a diversity-bascl method may fail low-resource environments, e.g. at early iterations of the loop.
6.2 Why Can Students Outperform Teachers?
An interesting observation across our experiments is that student TAMs trained with generated labels can outperform teacher LLMs, i.e. LLMAAA > PROMPTING, even without sample reweighting, shown by Figure 4. Such results partially align with previous findings in knowledge distillation (Wang, 2021; Song al., 2021) and pseudo-label- learning (Lee, 2013; Sanyal et al., 2022; Min et al., 2023), which share similar yet slightly different settings with LLMAAA.
5.3.3 Reweighting Helps Robust Training
Figure 4 depicts the learning trials with and without the automatic reweighting technique. We observe that reweighting training samples consistently help improve performance across all datasets and meth- ods. This finding proves that the training process of TAMs is more noise-tolerant with automatic reweighting, even with only small-scale clean- labeled set (100 samples) serving for validation.
We attempt further explain the phenomenon in a simplified setting, where we consider a binary classification task that predicts for x ~ D(x), where D(x) is discrete as in language space. For simplicity, we let = 1 denote the correct label and = 0 otherwise. We first make the natural assumption that the teacher's performance is above chance, i.e. the accuracy p > 0.5. Querying teacher for target sample x, will generate pseudo label 3/t ~ Bernoullipp). If the student is a universal function approximator S(x; 8) that outputs a scalar as probability that 9 = 1, then minimizing the
In particular, the performance gain from auto- matic reweighting is more prominent on Onto- Notes and Re- TACRED, and diminishes on Co- NLL03. We argue that automatic reweighting plays crucial role when the LLM annotators are rela- tively poor (as in OntoNotes and Re-TACRED). In such scenarios, the online approximation of the validation set serves as an effective estimation of unbiased data distribution, and helps prevent TAMs from overfitting noisy labels.

---

cross-entropy loss
in LLMAAA and previous zero-shot generation- based methods, c.g. ZEROGEN and SUNGEN, is that we use an unlabeled data pool Pooo and annottend data D to provide ex- tra knowledge. However, we shall point out that unlabeled text is readily available in many real- world scenarios, thus it is practical to make the pool-based assumption. Additionally, in complex tasks where zero-shot inference fails (like NER in our experiments), incorporate demonstrations for LLMs. In LL- MAAA, we strive for minimizing human efforts by restricting the scale (100 samples), and exploiting the same data for demonstration and validation. Another bottle- neck is the model capacities of teacher LLMs and student TAMs. On one hand, weak teacher is unable to teach excellent students that are ready to be used for applications (c.g. GPT-3). On the other hand, TAMs are bounded depending on their architectures. When the teacher surpasses the ceil- ing, will be theoretically impossible for students to outperform teachers. Despite these cases, we are optimistic that LLMAAA is effective in most situations.
min ((x.)r log(5(x;;;)) 0))
0
- (1 - log((1 S(x;; 8))
will reach optimal with S(x; 0) Usually we predict with heuristics that yj = 1 if (x; 0) > 0.5. With the previous assumption, which means that S always predicts correctly. This toy case nonetheless explains that an ordinary teacher can raise better students. Though teacher LLMs are deterministic for specific x when the temperature is set to O, their predictions are yet statistically random in Dxx), where the same con- clusion holds.
have
1,
costly yet necessary
data to a small
We shall point out that the above discussion con- siders much-relaxee setting, where attempt account for an intuitive understanding on why students outperform teachers in the hard label dis- tillation problem. We leave the rigorous theoretical analysis for future work.
7 Conclusion
In this work, we propose LLMAAA, framework that uses LLMs as active annotators address the challenges of data scarcity in NLP tasks. With ac- tive learning strategics, LLMAAA allows LLMs to label more informative samples that promote TAMs performance efficiently. We also optimize for reli- ability within the framework, which uses prompt engincering techniques and automatic reweighting improve annotation quality and to reduce the im- pact of noisy labels, respectively. Experiments on NER and RE tasks demonstrate the effectiveness of LLMAAA. A. The evaluation results highlight the ef- ficiency and reliability of LLMAAA. Trained with just hundreds of --n-ttnnt samples, TAMs are able to outperform their teacher LLMs sub- stantially. Besides, LLMAAA is also much more efficient compared prevalent data generation methods, which usually require orders of mag- nitude more synthetic training data. These find- ings reveal that LLMAAA offers a cost-effective, privacy-ensured, yet well-performing solution to apply LLMs in practical scenarios.
We adopt the proprietary GPT family as annota- tors in experiments, which are provided by OpenAI in a black-box manner. Though powerful, this prac- tice may raise several concerns, e.g. the potential exposure to test data. Nevertheless, we believe that given the comprehensive analysis in 6.1, it does not affect the effectiveness of our method.
Ethics Statement
This work utilizes publicly available benchmark datasets, and we respect and adhere to their licenses and agreements. Our proposed method involves the use of LLMss for data annotation, as discussed in GPT3Mix (Yoo al., 2021). This paradigm still poses several challenges, such as the poten- tial biases or toxic content in the generated data. Therefore, it is crucial to exercise caution when employing our method to invoke LLMs for generat- ing data and when utilizing TAMs trained on such generated data. Applying our work to downstream tasks such as NER and RE may result in issues such as mis-extraction and false information, and may fail in some cases. When employing our method, essential to consider using debiasing (Schick et al., 2021) 1) or manual checking to mitigate these concerns.
Limitations
Although LLMAAA demonstrates success in trans- ferring and exceeding I.MM'' capabilities with cheaper TAMs, it does come with certain limita- tions. The main difference between the setting

---

Acknowledgements
the Association for Computational Linguistics: Hu- man Language Technologies, pages 52-62, Atlanta, Georgia. Association for Computational Linguistics.
We appreciate the anonymous reviewers for their valuable advice on this manuscript. We would like to thank Tianhao Wu for the insightful dis- cussion and feedback. This work was supported by NSFC under grant 61932001 and U20A20174. The corresponding author of this paper is Lei Zou (zoulei ēpku. cn).
Maximillian Chen, Alexandros Papangelis, Chenyang Tao, Seokhwan Kim, Andy Rosenbaum, Yang Liu. Zhou Yu, and Dilek Hakkami-Tur. 2023. PLACCES: Prompting language models for social conversation synthesis. In Findings of the Association for Com- putational Linguistics: EACL 2023, pages 844-868, Dubrovnik, Croatia. Association for Computational Linguistics.
References
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus. Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellster, Douglas Eck, Jeff Dean. Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways.
Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1998-20222 Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2020. Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learn- ing Representations.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2895- 2905, Florence, Italy, Association for Computational Linguisties.
Parikshit Bansal and Amit Sharma. 2023. Large lan- guage models annotators: Enhancing generaliza- tion of nip models at minimal cost.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for chinese bert. IEEF/ACM Transac- tions on Audio, Speech, and Language Processing. 00:3504-3514
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Aron Culotta and Andrew McCallum. 2005. Reducing labeling effort for structured prediction tasks. In Pro- ceedings of the 20th National Conference on Artifi- cial Intelligence- Volume 2, AAAI'0,, page 746-751. AAAI Press.
Jacob Devlin, Ming- Heig-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Qingqing Cao, Bhargavi Paranjape, and Hannaneh Ha- jishirzi. 2023. Pumee: Pruning and merging tokens for efficient vision language models.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022. OpenPrompt: An open-source framework for prompt- learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics:
Wanxiang Che, Menggiu Wang, Christopher D. Man- ning, and Ting Liu. 2013. Named entity recognition with bilingual constraints. In Proceedings of the 2013 Conference of the North American Chapter of

---

System Demonstrations, pages 105-113, Dublin, Ire- land. Association for Computational Linguistics.
Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Pro- ceedings of a meeting held 6-9 December 2010, Van- couver, British Columbia, Canada, pages 892-900. Curran Associates, Inc.
Bo Du, Zengmao Wang, Lefei Zhang, Liangpei Zhang. Wei Liu, Jialie Shen, and Dacheng Tao. 2017. Fix ploring representativeness and informativeness for active learning. IEEE Transactions on Cybernetics, 47.1):14-26.
Mehran Kazemi, Sid Mittal, and Deepak Ramachandran. 2023. Understanding finetuning for factual knowl- edge extraction from language models.
Xiachong Feng, Xiaocheng Feng, Libo Qin, Bing Qin, and Ting Liu. 2021. Language model as an annota- tor; Exploring DialoGIT for dialogue summarization. Proceedings of the 59th Annual Meeting of the A Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Procexsing (Volume 1: Long Papers), pages 1479-1491, Online. Association for Computational Linguistics.
Hoyn-Hyun La Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges rep- resentation learning, ICML, volume 3, page 896.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021.
The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. 2017. Deep Bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pages 1183-1192.
David D. Lewis and William A. Gale. 1994, se- quential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '94, page 3-12, Berlin, Heidelberg. Springer- Verlag.
PMLR.
Jiahui Gao, Renjie Pi, Lin Yong, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang. Xiaodan Liang Zhenguo Li. and Lingpeng Kong. 2023. Self- self-guided noise-free data generation for efficient zero-shot learning. In The Eleventh International Conference on Learning Representations.
Yansong Li, Zhixing Tan, and Yang Liu. 2023. Privacy- preserving prompt tuning for large language model services.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers). pages 3816 3816-3830, Online. Association for Computa- tional Linguistics.
Jiachang Liu, Dinghan Shen. Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-1 I 14, Dublin, Ireland and Online, Associa- tion for Computational Linguistics.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan T. Chakaravarthy, Yogish Sabhar- wal, and Ashish Verma. 2020. Power-bert: Accel- erating BERT inference via progressive word-vector climination. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3690-3699. PMLR.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 15(5):1-35.
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization, In International Confer- ence on Learning Representations.
Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023. Annollm: Making large language models to be better crowdsourced annotators.
Keming Lu, I-Hung Hsu, Wenxuan Zhou,
Mingyu Derek Ma, and Muhao Chen. 2022a. Summarization as indirect supervision for relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6575 6594, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning for classification and preference learning.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022b. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the
Sheng-Jun Huang, Rong Jin, and Zhi-Hua Zhou. 2010. Active learning by querying informative and represen- tative examples. In Advances in Neural Information

---

60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021. A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-40.
Lingjun Lyu, Xuanli He, and Yitong Li. 2020. Differ- entially private representation for NIP:: Formal guar- antec and an empirical study on privacy and fairness. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 2355-2365. Online. Association for Computational Linguistics.
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirr.. 2021. Label ver- balization and entailment for effective zero and few- shot relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1199-1212, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021. Active learning by ac- quiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650-663, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sunandini Sanyal, Sravanti Addepalli, and R Venkatesh Babu. 2022. Towards data-fies free model stealing in a hard label setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language mod- els: Towards zero-shot language understanding. In Advances in Neural Information Processing Systems.
Timo Schick and Hinrich Schütze. 2021. Generating datasets with pretrained language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6943- 6951, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. 2021. Re- cent advances natural language processing via large pre-trained language models: A survey. arXiv preprint 1xxi:2211.0022::
Timo Schick, Sahana Udupa, and Hinrich Schüizze. 2021. Self-diagnosis and self-debiasing: A proposal for re- ducing corpus- based bias in NLP. Transactions of the Association for Computational Linguisties, 9:1408- 1424.
Zeping Min, Qian Ge, and Cheng Tai. 2023. Why the pscudo label based semi-supervised learning algo- rithm is effective?
Ozan Sener and Silvio Savarese. 2018. Active learn- ing for convolutional neural networks: A core-set approach. In International Conference on Learning Representations.
OPenAT. 2023. Gpt-4 technical report.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:67.
Burr Settles. 2009. Active learning literature survey.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Nils Reimers and Tryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IICNILP)) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Dandan Song, Jing Xu, Jinhui Pang, and Heyan Huang. 2021. Classifer-adapttation knowledge distillation framework for relation extraction and event detec- tion with imbalanced data. Information Sciences, 573:222-238.
Nils Reimers and Tryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525, Online. Association for Computational Linguisties.
George Stoica, Emmanouil Antonios Platanios, and Barnabas Poczos. 2021. Re-tacred: Addressing shortcomings of the tacred dataset. Proceedings of the AAAI Conference on Artificial Intelligence, 131133:1384-138500
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018 Learning to reweight examples for robust deep learning. Proceedings of the 35th In- ternational Conference on Machine Learning, ICML. 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 volume 80 of Proceedings of Machine Learning Research, pages 4331 1331-4340. PMLR
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tuning for aanguuge-ootell-as---srrrr
In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 20841 20851-20855, PMLR

---

Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang- -Woo Lee, and Woomyoung Park. 2021. GPT3Mix: Lever- aging large-scale language models for text augmen- tation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 2225-2239, Punta Cana, Dominican Republic. Association for Computational Linguistics.
I anguage-indeeennde
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A. Inan, Gautam Kamath, Janardhan Kulka- mi, Yin Tat Lee, Andre Manoel, Lukas Wutschitz. Sergey Yekhanin, and Huishuai Zhang. 2022. Differ- entially private fine-tuning of language models. In The Tenth International Conference on Learning Rep- resentations, ICIR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Shuhe Wang. Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li. and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models.
Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd- Graber. 2020. Cold-start active learning through self- supervised language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7935-7948, Online. Association for Computational Linguistics.
Zi Wang- 2021. Zero-shot knowledge distillation from a decision-basel black-box model. In Proceedings of the 38th International Conference on Machine Learn- ing. ICML 2021. 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 10675-10685, PMLR.
Yuhao Zhang, Victor Zhong. Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguistics.
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021. Towards zero-label language learning.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023a. Chain-of-thought prompting elicits reasoning in large language models.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjic Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303. 18223.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023b. Zero- shot information extraction via chatting with chatgpt. arXiv preprint 23..::3322.1000..
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learn ing by meta- meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Lingnistics: EMNLP 2021, pages 2856-2878, Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Ralph Weischedel, Martha Palmer, Mitchell Marcus. Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni- anwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2011. Ontonotes 4.0. Linguistic Data Consortium LTC201
A Dataset Statistics
Lin Yang, Yizhe Zhang. Jianxu Chen, Siyuan Zhang, and Danny 7. Chen. 2017. Suggestive annotation: deep active learning framework for biomedical image segmentation. In Medical Image Computing and Computer Assisted Intervention MICCAI 2017: 20th International Conference, Quebec City, QC, Canada, September 11-13, 2017, Proceedings, Part III, page 999-407 Berlin, Heidelberg. Springer- Verlag.
In this section, describe the statistics and pre- processing of each dataset in detail.
OntoNotes 4.0 (Weischedel et al.. 2011) is a large corpus comprising various genres of text (news, web text, etc.) in three languages (English, Chinese, and Arabic) with structural information and shallow semantics, and has been widely used for NER. We use the Chinese text data and take the same data split as Che al. (2013), which uses four most common entity types, i.e. PER (person). LOC (location), ORG (organization) and GPE (geo- political entities). We truncate token length within 512 (i.e. split long input to multiple chunks) to
Jaacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. ZeroGen: Efficient zero-shot learning via dataset generation. Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11653-1 1669, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.

---

Training
Validation
Testing
Model
API
per:age
421
256
208
GPT-3
text-curie-001
per:nationality
295
222
115
ChatGPT
8pt-35-turbo
per:parents
182
69
106
GPT-4
gpt-4
per:children
275
114
55
per:siblings
211
33
66
Table 6: Azure OpenAl service API that we use.
per:spouse
271
189
73
no_relation
2,482
1,324
934
Total
4.137
2,207
1,557
B.2 Training Task-Specific Models
For all experiments that train TAMs for inference (i.e. LLM^AA, ZEROGEN, FEWGEN and SUPER- VISED), we repeat each with three random seeds, resulting in different parameter initialization and random data sampling. We report the mean and standard deviation in our results.
Table Statistics of relation types each split Re- TACRED-subsee We replace the label "per:origin" with "per:nationalie" for clarity.
fit in BERT input limit. The processed data con- tains 15,724/4.301/4,34 samples for training/vali- dation/testing, respectively.
We use bert- as TAMs' encoders with a learning rate of 5e-5 for English data (CoNLLO3 and Re-TACRED), and beeteeeeeert-bseeeeee
(Devlin et al., 2019)
The English CONLL 2003 shared task (Tjong Kim Sang and De Meulder, 2003) NER dataset that contains four entity types: PER (per- son), LOC (location), ORG (organization), and MISC (miscellancous entities), which consists of 14,041/3,250/3,45 sentences for training/vallid-- tion/testing.
(Cui et al., 2021) with learning rate of 2c-5 for Chinese data (OntoNotes 4.0). The learning rate other parameters (i.e. linear classifiers) is set to le-4. We optimize the models via AdamW (Loshchilov and Hutter, 2019), with e 1c-6, under linear warmup schedule for the first 6% steps. We train all TAMs with a batch size of 8 for 40 epochs and take the checkpoint with the highest validation performance for final prediction.
Re-TACRED (Stoiea et al., 2021) is revised version of TACRED (Zhang et al., 2017), large- scale crowdsource-annotae annotated RE dataset. orig- inally has 40 relation types. Including all these types will lead to much longer prompts, which may exceed the API length limit and receive responses with higher latency. Therefore, we opt to select subset of relations that describe personal relation- ships for study. We keep all these relation instances in Praining/validatiooneessii
B.3 Prompts
The full prompts we use for annotation are shown in Table 7. Re-TACRED, we provide prompts both with and without verbalized labels. To add demonstration, we insert each sample's text into input and label to output. The target sample is added to the last input, and the last output is left blank for prediction.
sets, and balance the NA relation instances the original portion. The statistics for each relation type is shown in Table
For all three datasets, we randomly sample 100 examples from the original validation sets and reuse the same data for demonstration Ddemo and validation D We use the full training sets as the initial 'Dp Poool from which we randomly sample active learning's 's seed labeled sets Diabeled with a size 50.
We also show the prompts for generation in Ta- ble 8. We them similarly to annotation. In the zero-shot setting, to help models generate desired outputs, we use default example to inform LLMs about the output format.
C Annotation Examples
Implementations
We show two annotation examples of correct/par- tially wrong annotations from the CoNLL 2003 NER dataset Listing The first example is exactly correct, and the second example contains hallucinations that do not exist in ground truth: "April", "March". and "Thursday".
B.1 LLM Inference APIs
We access OpenAl APIs by Azure service. The API we use for each model is depicted in Table Since ChatGPT and GPT-4 will continue to be updated, they may generate different responses as time changes, even when the temperature is O.

---

Task
Prompting
You are a highly intelligent and accurate news domain samed-entity recognition (NER) system. You take passage as input and your task is to recognize and extract specific types of classify into set following predefined entity types: [person (PER), location (LOC), organization (ORG), miscellancous entity (MISC)
Description
named entities that given passage
CoNLL
Your output format must be in json form of: ["span": span, "type": type.
Instruction
The span must exactly same the original text, including white spaces.
Input: "Input: ()"
Format
Output:
"Output: (1"
M
2
18 STARRI AN 14 (NER) bii E. BE
NER
SKTm El json #RTOR D NER #0 span 313320
Description
- [ (PER) 3A (ORG) 1 (GPE)
OntoNotes 4.0
(OC)) ]
10 st: I"span": span, "type": type,... ...]
Input:
"W (1"
Format
Output: "i ()"
Given sentence, and two entitics within the sentence, classify the relationship between the entities based the provided sentence. If no relation of interest exists, strictly return
Description
"no relation". All possible relationships are listed helow:
per:age
per:parents
per:spouse
Re- Re-TACRED
Instruction
per:siblings
(Original)
per:children
per:nationality
no_ no_relation
Input: "Sentence: ()"
Output: "Relationship: ()"
Format
Struct: "el: ()
c2: !!"
Given sentence, and two entities within sentence, classify the relationship between the two entities based on the provided sentence. If no relation of interest exists, strictly return relation". All possible relationships and explanations listed below:
Description
per:age: the age of [el] [c2]
per:parents (el)'s parent is (e2)
per:spouse: Jss spouse [c2]
Re-TACRED
Instruction
per:siblings [el) the sibling of (c2)
(Verbalized)
perschildren: {el children (c2)
permationality:: [el]'s nationality is (e2)
no_relation: {el] has no known relations to (e2)
Input: "Sentence: (1"
Output: "Relationship: ()"
Format
Struct: "cl: {}
e2: ()"
Table 7: Annotator's prompts for each task.
{
"text":"" "Celtic 's Jackie McNamara who did well with last season 's successful under-22 -21 team earns call-up to the senior squad .", "Labels": [[[" "Celticc" "type":"" " ("""))""ppaa ("span": "Jackii McNamara", "type":""P" "PER "}]
{
text": "Finland 's trade surplus rose 3.83 billion markka in April from 3. billion in March the National Customs Board ( NCB said in a statement on Thursday "Labels": "NCB" "ORR"" "ORG"), (" """span:""Filla""
" "type":"Loc" { ("span":" National Customs Board", " "type": ": "ORG", ":""p"":"""prr"""" "Aprill" "t"pe"""MISC" spap "arch"""" "type": "MISC"", (pspan" "; "Thursday" "type": "MISC"")
Listing 1: Annotation examples.

---

Task
Prompting
You are an intelligent text data generator. Generate {} high-quality and diverse sentences in news domain containing entities for the following types:
Description
[person (PER), location (LOC), organization (ORG), miscellaneous entity (MISC)
Write one sample per line. No other output.
CoNLL 03
Example: "Fxample: (
Format
Output: "Output: ()"
Default
("text": text, "entities": : [[" [["name": name. "type": type]])
Example
#0
A
lentity) (ORG) HE (GPE)
Description
(PER)
HL
(LOC) ]
OntoNotes 4.0
f |l $ KO
Example: "<s7 UF"
Format
Output: "b
()"
Default
("text":
text. "entities"; [("name": name, "type": type]])
Example
You are an intelligent text data generator. Generate { high-qualin and diverse sentences in domain containing relational triplet for the following relation types:
per:age: : the age of SUBJ OBJ
per:parents SUBJ's parent is OBJ
per:spouse: SUBJ's spouse OBJ
Description
per:siblings SUBJ is the sibling OBJ
perschildren: SUBJ's children OB.
Re-TACRED
per:attionlltt:: SUBJ's nationality
no_relation SUBJ has no known relations to OBJ
Write sample per line json format. Subject and object appear in the sentence. No
other output.
Example: "Example: ()"
Format
Output: "Output: ()"
Default
("text": text, "subject": subject, "object": object, "relation": relation)
Example
Table Generator's prompts for each task.


Reviews:
[{"review_id": "2USvWSJ5iW", "reviewer": "Reviewer_Neog", "summary": "", "questions": "a. I couldn't quite follow the description of how label verbalization is performed. Were label verbalizers from the cited works used or did you use a method from one of the cited works to create your own verbalizers?\n\nb. Are the labels provided by the LLM ever used in the set of examples drawn from for in-context learning?\n\nc. In Table 1, the amount of data used for the PROMPTING baseline is '-', however, don't the prompts include in-context learning examples? Is this just the amount of data used by the downstram model? If so, I think it would be helpful to include the amount of data used by the LLM as well.\n\nd. Similarly how many in-context examples are used for FewGen?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive and insightful comments, and they are exceedingly helpful for us to improve our paper. We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments and questions are given below.\n\n### Comments:\n\n> C1: Some of the design decisions like the number of initial ground truth examples, and the number of labeled examples seem arbitrary. Given that cost-effectiveness is a primary motivation, it would have been more satisfying to see what the performance of, e.g., LLaMAA vs. finetuning if finetuning was performed on a number of samples equivalent in cost to the additional samples from the LLM. \n\nWe totally understand your concerns. Regarding the choices of the number of gold data (ground truth) and the number of silver data (LLM-annotated), we follow the principle to minimize the two numbers separately. \n**As for # gold, we mainly go with a coverage consideration** (i.e. we want to include every type of entity for NER, each with a small amount of instances to be sampled from to decrease the randomness in sampling the demonstration set), and we aim that teacher LLM can reach a considerable performance with in-context examples provided. \n**As for # silver, the primary goal of LLMaAA is to induce a student model outperforms teacher LLM with minimal pseudo-labeled data**, and we find that 500 is sufficient for most NER and RE tasks, as shown by our results.\n\nTo compare LLMaAA with FT with a number of samples equivalent in cost to the additional samples from the LLM, we must first identify how much the cost is. According to ([Wang et al., 2021](https://aclanthology.org/2021.findings-emnlp.354/)), the cost of human annotation is roughly estimated as \\\\$2/1k tokens, while ChatGPT charges for \\\\$0.002/1k tokens. Even taking the additional cost of template into account, we believe that human labeler can be 10x to 100x more expensive than LLM. Thus, 500 silver data is equivalent to dozens of gold data at most – even less than 100, which we evaluate for FT in Table 1 (LLMaAA only use silver for training student models and does not use gold).\n\n> C2: I am not sure I understand the motivation/value behind the \"medium-data\" setting studied in this paper. The datasets contain sufficiently many (hundreds to thousands of) examples not to be few-shot, but at the same time they are not big enough / sufficiently high quality that the downstream classifier's performance is near fully supervised BERT model. \n\nWe want to clarify that we target at a semi-supervised setting, i.e. rich unsupervised data (text) with frugal gold data (demonstration/validation set): since in many real world scenarios we can see that unsupervised data is often abundant while annotation is the main bottleneck. In this case, gold data serves as the guideline for silver annotations (which requires equivalent efforts as prompt designing in PROMPTING), and unsupervised data serves as the proxy for the distillation from LLM to small models. We believe that such a setting is practical for applying LLM, and will reduce inference costs and latency for sensitive applications.\n\nAs stated before, we want to build a student model outperforms teacher LLM (whose performance is satisfied but not excellent for applications) for online deployment, as a substitution. Though the performance has a gap to fully supervised BERT model, we believe that exceeding PROMPTING is still valuable. We will modify the introduction section to convey this message more clearly.\n\n> C3: I didn't find the argument in Section 6.2 particularly convincing.\n\nWe mostly agree with your opinions. The motivation we add section 6.2 is that we find that the phenomenon (s > p) is interesting in the hard label distillation setting (no soft logits given), and as far as we concern no previous works have discussed the reason behind before. So we take an much relaxed setting to account for an intuitive understanding. We will revise this part and further add a discussion of limitations of our analyses.\n\n### Questions:\n\n> Qa: I couldn't quite follow the description of how label verbalization is performed. Were label verbalizers from the cited works used or did you use a method from one of the cited works to create your own verbalizers?\n\nThe label verbalizer we used is shown in Table 7, which is adapted from ([Lu et al., 2022](https://aclanthology.org/2022.findings-emnlp.490/)) with some modifications.\n\n> Qb: Are the labels provided by the LLM ever used in the set of examples drawn from for in-context learning?\n\nNo, the labels are not used in a bootstrapping manner. It is an interesting direction, but we concern that such an approach may require multi time of inferences on the same sample, which contradicts the label efficiency issue that we care about.\n\n> Qc: In Table 1, the amount of data used for the PROMPTING baseline is '-', however, don't the prompts include in-context learning examples? Is this just the amount of data used by the downstram model? If so, I think it would be helpful to include the amount of data used by the LLM as well.\n\nIn Table 1, the PROMPTING exactly denotes the performance of the teacher LLM in LLMaAA, and it select kNN examples from the same demonstration set (100 gold data) for NER (RE does not use in-context examples, as stated). Thanks for pointing out, and we will clarify it in the experiments section.\n\n> Qd: Similarly how many in-context examples are used for FewGen?\n\nIt is also aligned with PROMPTING & LLMaAA (NER), and the examples are uniformly drawn from the same demonstration set. We find that we forget to mention that we use k = 5 for all experiments. We will add this information in the implementation section.\n\n### Missing References, Typos Grammar Style And Presentation Improvements:\n\nThanks for your instructive comments and careful suggestions. We will add a reference to ([Gao et al. 2023](https://openreview.net/forum?id=h5OpjGd_lo6)) in line 62-64, and discuss the two contemporary work in the related work section. We will also refine the expression (especially clarify efficiency and reliability), and modify Figure 1 in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- LLMaAA successfully outperforms other learning methods in low- to medium-data settings.\n  These results are valuable to the research community for two reasons: 1. LLMaAA provides a successful illustration of how to combine active learning with LLMs, and 2. the results also show the relative ineffectiveness of data synthesis and few-shot prompting.\n- The ablation results appear to be fairly comprehensive and provide useful perspective on the relative importance of: teacher strength, different acquisition functions, example reweighting, etc.", "reasons_to_reject": "- While I understand that language models have improved considerably in the past few months, and that this has created a corresponding need for research into how these models can be used to improve models that are more efficient/cost-effective, I am not sure that I would consider the approach in this paper novel.\n  Embedding-based retrieval for in-context learning, using LLMs to generate training data, and the retrieval functions used in the active learning component of this paper all stem from prior work.\n  Although I do not consider combining these things trivial, this does make me less excited about the work.\nI also have some reservations about the setting studied in this paper:\n- Some of the design decisions like the number of initial ground truth examples, and the number of labeled examples seem arbitrary.\n  Given that cost-effectiveness is a primary motivation, it would have been more satisfying to see what the performance of, e.g., LLaMAA vs. finetuning if finetuning was performed on a number of samples equivalent in cost to the additional samples from the LLM.\n  If the authors could provide some justification for these decisions I would consider increasing my soundness score.\n- I am not sure I understand the motivation/value behind the \"medium-data\" setting studied in this paper.\n  The datasets contain sufficiently many (hundreds to thousands of) examples not to be few-shot, but at the same time they are not big enough / sufficiently high quality that the downstream classifier's performance is near fully supervised BERT model.\n  This negatively impacted my excitement score.\n- I didn't find the argument in Section 6.2 particularly convincing.\n  Sure if a model is always more likely to be right than wrong, and you can draw infinitely many labels from it for a given test instance, then a \"student\" model that returns the most likely label is always right.\n  This is an extremely idealized setting that is incongruent with the one studied in the paper:\n        a. the whole motivation of active learning in this paper is that drawing labels from teachers is expensive; in cases where you only have finite labels there is a non-zero probability the student will be wrong,\n        b. the assumption the teacher is always more likely to be right on every instance is, in my opinion, very unnatural, and\n        c. if I understand correctly, the claim that s->p only holds on examples that are being labeled by the teacher, it has no bearing on the student's performance on unseen instances at test time.\n  This section had a minor negative impact on my soundness score for the paper.", "questions_to_authors": "a. I couldn't quite follow the description of how label verbalization is performed. Were label verbalizers from the cited works used or did you use a method from one of the cited works to create your own verbalizers?\n\nb. Are the labels provided by the LLM ever used in the set of examples drawn from for in-context learning?\n\nc. In Table 1, the amount of data used for the PROMPTING baseline is '-', however, don't the prompts include in-context learning examples? Is this just the amount of data used by the downstram model? If so, I think it would be helpful to include the amount of data used by the LLM as well.\n\nd. Similarly how many in-context examples are used for FewGen?", "paper_topic_and_main_contributions": "This paper is on the topics of LLMs, automatic data labeling, and active learning.\n\nThe primary contribution of this paper is LLMaAA, an active learning framework for using LLMs to generate labels for downstream models.\nThe general idea is to replace the human labeler in the standard active learning setup with an LLM.\nLLM labels are generated using in-context learning, and in-context examples are selected from a small pool of labeled examples using kNN search.\nThe acquisition functions used to select instances for labeling are based on the outputs of the downstream model, and are all taken from prior work.\nTo validate the effectiveness of this approach, the paper also contributes experimental results for LLMaAA on named entity recognition and relation extraction.\nThe results demonstrate that LLMaAA achieves superior performance to prompting and data synthesis methods, and an ablation study also characterizes the relative effectiveness of different acquisition functions and other design decisions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- The claim in lines 62-64 should be justified with a reference.\n- This is a contemporary paper that is related:\nBansal, P., & Sharma, A. (2023). Large Language Models as Annotators: Enhancing Generalization of NLP Models at Minimal Cost. ArXiv, abs/2306.15766.\n- This paper may be worth citing to provide additional evidence of students surpassing their teachers:\n[Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://aclanthology.org/2023.findings-acl.507) (Hsieh et al., Findings 2023)", "Typos_Grammar_Style_and_Presentation_Improvements": "- In the abstract I'd recommend replacing \"the teacher\" with \"its teacher LLM\" for clarity.\n- Figure 1 seems to imply your method does not involve in-context learning and prompting LLMs, as these boxes are omitted from 1.c, however these things are actually crucial to the method.\n  Furthermore, the figure also implies that the method does not require labeled data, however a seed pool of 100 examples.\n- Efficiency and reliability---the primary purported benefits of the proposed method---are never formally defined.\n  While I could infer what is meant by these terms from the rest of the paper, (reliability = accuracy, efficiency = data efficiency), I feel it would be helpful to make these terms more clear.\n- In my opinion, some of the language in this paper is a bit sensational, e.g., the word 'deified' in line 404, and 'arcane' in 430.\n- 590 well effective"}}, {"review_id": "V8Laf3Fw6q", "reviewer": "Reviewer_P3tf", "summary": "", "questions": "Q1. While the authors evaluate their method on low-resource settings (which is reasonable) just curious about how will the performance change when increasing LLMaAA generated data. In other words, if using 5,000 data instead of 500, is the performance still competitive compared to other baselines? \n\nQ2. What do the actual examples look like that are generated by the proposed method? As shown in Table 2, it would be interesting to see the case study of what actual samples generated and annotated by LLMaAA look like.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback and valuable comments. We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments and questions are given below.\n\n### Comments & Questions:\n\n> C1 & Q1:  How the proposed method will work in a very low resource setting (10 samples) or in a large number of labeled data.\n\nAs a brief recap, we use 100 gold data (ground truth) for demonstration and validation, and we use 500 silver data (LLM-annotated) to train small models. According to the context, we assume that here you ask about the number of silver data.\n- Under very low resource setting (i.e. 10 samples), training small models even with gold data still can’t achieve satisfactory generalization performance, therefore it seems to be trivial to discuss such a small scale.\n- With a larger number of data, a more suitable analogy of LLMaAA should be SUPERVISED (i.e. training with gold data) rather than ZeroGen/FewGen, which are broadly discussed in active learning literature: the performance keeps increasing until saturation. This trend also meets our results in Figure 3. We believe that with a large number of silver data, LLMaAA performance can further improve (but not so much). Besides, in Table 1, the only performance degradation is ZeroGen on Re-TacRED, so we also concern that the degradation trend does not generally hold.\n\n> Q2:  Case study of LLMaAA annotations. \n\nSure, here is two examples of correct/partially wrong annotations on `en_conll03` (NER) task:\n\n```json\n{\"text\":\"Celtic 's Jackie McNamara , who did well with last season 's successful under-21 team , earns a call-up to the senior squad .\",\"labels\":[{\"span\":\"Celtic\",\"type\":\"ORG\"},{\"span\":\"Jackie McNamara\",\"type\":\"PER\"}]}\n{\"text\":\"Finland 's trade surplus rose to 3.83 billion markka in April from 3.43 billion in March , the National Customs Board ( NCB ) said in a statement on Thursday .\",\"labels\":[{\"span\":\"NCB\",\"type\":\"ORG\"},{\"span\":\"Finland\",\"type\":\"LOC\"},{\"span\":\"National Customs Board\",\"type\":\"ORG\"},{\"span\":\"April\",\"type\":\"MISC\"},{\"span\":\"March\",\"type\":\"MISC\"},{\"span\":\"Thursday\",\"type\":\"MISC\"}]}\n```\nThe first example is exactly correct, and the second example contains hallucinations that does not contained in ground truth: `{\"span\":\"April\",\"type\":\"MISC\"}`, `{\"span\":\"March\",\"type\":\"MISC\"}` and `{\"span\":\"Thursday\",\"type\":\"MISC\"}`.\n\n### Typos Grammar Style And Presentation Improvements:\n\nThanks for your suggestions. We will modify Figure 4 if there is not enough space in later revision.\n\nAt the very last, LLMaAA targets at general NLP tasks beyond sequence labeling (i.e. RE is a text classification task). We hope that our responses help address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is easy-to-follow and well-organized. ", "reasons_to_reject": "Is there any chance to explore how LLMaAA will work on different numbers of labeled sample settings? The authors only provide a single setting which uses 500 samples to evaluate their method. Since their baseline methods FEWGEN and ZEROGEN show performance degradation when using larger amounts of data, it is a concern that LLMaAA would show similar trends. It would be interesting to see how the proposed method will work in a very low resource setting (10 samples) or in a large number of labeled data.", "questions_to_authors": "Q1. While the authors evaluate their method on low-resource settings (which is reasonable) just curious about how will the performance change when increasing LLMaAA generated data. In other words, if using 5,000 data instead of 500, is the performance still competitive compared to other baselines? \n\nQ2. What do the actual examples look like that are generated by the proposed method? As shown in Table 2, it would be interesting to see the case study of what actual samples generated and annotated by LLMaAA look like.\n", "paper_topic_and_main_contributions": "The authors proposed a method of how samples should be annotated efficiently using large language models for sequence labeling tasks. To effectively leverage pseudo-labels, the authors first draw k-NN samples and adopt the automatic reweighting technique to assign training samples with learnable weights. They successfully incorporate LLMs-generated noisy data and achieve competitive performance on NER and RE tasks compared to strong baselines. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 4 shows can be simplified by providing the subsets of graphs. Since it is cluttered to show all graphs on different automatic reweighting methods on all datasets, it would be good to show a single dataset on all automatic reweighting methods, and the remaining can be moved to a supplementary section. "}}, {"review_id": "8Vf896dhXh", "reviewer": "Reviewer_z8rH", "summary": "", "questions": "Question 1:  No justification is given for the selection of the BERT student model.  Is it a strong starting point known to achieve near state-of-the-art performance?  Was it chosen because it satisfied some budget constraints?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments, and they are exceedingly helpful to improve our paper. We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments and questions are given below.\n\n### Comments:\n\n> C1: The paper misuses the term \"active learning\".\n\nThe term itself emphasizes the label acquisition process from an “oracle”, and human annotator is just one specific instance ([Settles, 2009](https://burrsettles.com/pub/settles.activelearning.pdf)). Therefore, we think that this term properly describes our framework, if we take LLM as the oracle in the loop. Nevertheless, thanks for your suggestions, and we realize that it will bring some misunderstanding. We will revise our usage of “active learning” in our writing and emphasize “active acquisition” more instead. \n\n> C2: There is no discussion of the source of the unlabeled data used in the experiments. This seems like it could have a huge effect on the methods.\n\nIn our experiments, the unlabeled data pool is the original training sets of `zh_ontonotes`/`en_conll03`/`re-tacred-subset`, and we do not use their ground truth labels throughout the experiments. The gold data for demonstration/validation is randomly sampled from the original validation sets, and we conduct evaluation on the original test sets. So the unlabeled data and gold data comes from the same domain, as the original training and validation sets. We will clarify the setting in our implementation section.\n\n> C3: Question on privacy: with the method presented in this paper, the labeled and unlabeled data do need to be uploaded, to prompt the LLM to generate labels.\n\nWe mainly aim to tackle the online privacy and costs issues. Using a offline finetuned model for inference avert these problems. As for the training stage, user can control what data to upload (by restricting the labeled and unlabeled sets) and use data masking techniques to further alleviate the privacy issue.\n\n> C4: Question on efficiency: the main experiments of the paper do present details about how much data of each kind each method uses, but they don't explore how the performance varies as a function of any of these (gold data, silver data, and student model). The experiments might have turned out very differently if the authors had started with more gold data, for example. Or perhaps if we used a more powerful or better pre-trained student model, we could get away with less OpenAI-based annotation. The paper would be much stronger if there was more sensitivity analysis to these choices, paying particular attention to providing useful guidance about the tradeoffs among these various costs.\n\nWe totally understand your concerns. We first leave alone the computation cost of finetuning (which is relatively small compared to the other two), and analyze #gold and #silver. Obviously, the performance is an monotonic increasing function of #gold and #silver converging to the ceiling, so the problem here reduces to discuss the tradeoff between #gold and #silver, given a fixed budget.\nHowever, we are afraid that the ratio #gold/#silver may vary among different tasks and even different budgets, therefore providing a grid search of the Pareto frontier for a specific budget is meaningless.\n\nHere we try to provide another line of analysis to this problem. \n- We first estimate the cost of gold and silver annotation for each sample. According to ([Wang et al., 2021](aclanthology.org)), the cost of human annotation is roughly estimated as \\\\$2/1k tokens, while ChatGPT charges for \\\\$0.002/1k tokens. Even taking the additional cost of template into account, we believe that human labeler can be 10x to 100x more expensive than LLM. So \\\\$gold is much higher than \\\\$silver, and should be optimized in priority.\n- Therefore, we can decouple the process by minimizing #gold first, then #silver. \nWe then propose a design principle of choosing #gold and #silver, given an expected performance $p$. Recall that Table 4 shows that the student performance is strongly correlated to its teacher with a gap, so we can set a threshold $\\tau$ and grid search #gold to let LLM reach $p - \\tau$, with a few test data for estimation. We can then assign remaining budget to #silver, or early stops until the final performance saturates. Since LLMaAA asks for silver data is in a iterative loop fashion, the early stopping thus can be performed easily.\n\nWe will add this part of discussion into the analysis section or the appendix if you find it helpful. If you have any other suggestions or concerns on this issue, please do not hesitate to leave your further comments.\n\n> C5: The use of proprietary language models as a component is a limitation that should at the very least be discussed. In particular, the authors and reviewers have no way of knowing whether ChatGPT or the other large models were exposed to the test data during their construction.\n\nThanks for the suggestion, and we will add a discussion on the use of proprietary language models in the limitation section. \n\n### Questions:\n\n> Q1: No justification is given for the selection of the BERT student model. Is it a strong starting point known to achieve near state-of-the-art performance? Was it chosen because it satisfied some budget constraints?\n\nWe want to incorporate least inductive bias in the choice of student model, so we build two simple models with BERT as encoders, which is just a common baseline, not sota. We want to show that LLMaAA works fine with such simple student models, without the need of “heavy” design heuristics or strong encoders.\n\n### Typos Grammar Style And Presentation Improvements:\n\nThanks for your instructive suggestions and comments. We will carefully refine our abstract, introduction and writing style and modify Figure 2’s caption accordingly. We will also include a brief example of label verbalizer in our revised version. As for now, you can refer to Table 7 in Appendix C to see how verbalizer works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The technical contribution of this work is essentially to assemble a collection of established techniques into a larger pipeline to experiment with distillation in a setting where I don't think it's been explored before.  Because the components are mostly (with some exceptions, see below) reasonably well motivated and explained, this is fine.  The experimental results show that, at least under the explored settings, one can improve the \"student\" model using noisily annotated data from the large LM teacher.\n\nI did appreciate the care the authors took in section 6.1, where different teachers are considered.  The approach does consistently outperform basic prompting across several strong models, though the gap is closing as we get to GPT-4 (not discussed).", "reasons_to_reject": "The paper misuses the term \"active learning.\"  Active learning uses a trained model to select examples for annotation, but employs humans to annotate them.  The method here takes the trained model's labels as given, without having any humans in the loop.  If these were being fed back into the same model iteratively, we'd have some version of bootstrapping or self-training.  As it is, they are being used to train a secondary model, making this effectively a kind of distillation.  (The paper even adopts distillation terminology -- \"teacher\" and \"student.\")\n\nThere is no discussion of the source of the unlabeled data used in the experiments.  This seems like it could have a huge effect on the methods.\n\nThe paper makes claims about efficiency and privacy that are not well supported.  The privacy question could potentially be addressed simply by making the argument explicit; it seems to be along the lines of \"if you do supervised training of your own model, you don't have to upload your data to the LLM service provider.\"  But in fact with the method presented in this paper, the labeled and unlabeled data *do* need to be uploaded, to prompt the LLM to generate labels.\n\nAs for efficiency, the paper could do a much better job of laying out the costs of the range of approaches it considers.  There's the cost of having humans annotate gold data, the cost of the LM service to do the generations, and the computational cost of finetuning BERT.  None of these are actually quantified.  The main experiments of the paper do present details about how much data of each kind each method uses, but they don't explore how the performance varies as a function of any of these.  Ultimately we have arbitrary choices about how much gold data and how much of each kind of silver data are used for each approach.  The experiments might have turned out very differently if the authors had started with more gold data, for example.  Or perhaps if we used a more powerful or better pre-trained student model, we could get away with less OpenAI-based annotation.  The paper would be much strongly if there was more sensitivity analysis to these choices, paying particular attention to providing useful guidance about the tradeoffs among these various costs.\n\nThe use of proprietary language models as a component is a limitation that should at the very least be discussed.  In particular, the authors and reviewers have no way of knowing whether ChatGPT or the other large models were exposed to the test data during their construction.", "questions_to_authors": "Question 1:  No justification is given for the selection of the BERT student model.  Is it a strong starting point known to achieve near state-of-the-art performance?  Was it chosen because it satisfied some budget constraints?", "paper_topic_and_main_contributions": "The paper is about building information extraction models (evaluations include named entity recognition and relation labeling) using language models.  The key idea is to augment a small annotated dataset with unlabeled instances plus labels produced by a \"large\" language model like ChatGPT; the combined gold and silver datasets are then used to finetune a cheaper model (here, BERT) to perform the IE task.  The experimental results show that on three datasets this approach is better than merely prompting the large model or using only the gold data to train the smaller model.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper would be improved with careful editing.  From the very beginning, there's a lot of repetition (e.g., \"Prevalent supervised learning methods in NLP are notoriously data-hungry, which demand large amounts of high-quality annotated data.  In practice, acquiring such data is a costly endeavor.\" could be as simple as \"This paper considers the cost of high-quality annotated data required by current supervised learning methods in NLP.\")  The opening discussion about language models as a service is pretty disconnected from the contributions of the paper.  I'd recommend revising the intro to focus directly on the paper's contributions.\n\nLine 090, I don't understand what this means:  \"building fine-level descriptions aligned with natural language for uninformed labels\"\n\nLine 125, the phrase \"real world miniaturization\" is misleading; this is not a paper about real-world applications or miniaturization\n\nFigure 2:  the numbered elements in the caption don't have analogues in the figure itself\n\nThe discussion of the label verbalizer (line 233) is not self-contained, essentially assuming the reader is familiar with all the details of past work."}}]

Author rebuttal:

Meta-review: This paper proposes LLMaAA using LLMs to annotate data in the active learning loop by optimizing both annotation and training processes to train task-specific models reliably. Reviewers appreciate that the method is reasonably well motivated and explained, presented with superior performance and comprehensive analysis with ablation study. They also bring up several areas for improvements: (1) The novelty is limited because a large portion of the framework directly borrows from existing techniques; (2) the lack of results on the effect dataset sizes, (3) improving the clarity and adding thorough analysis and discussions on efficiency and privacy. Overall, we strongly encourage authors to incorporate the feedback to improve the paper in the next version.
Decision: Accept-Findings