FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models
Authors: Ruixuan Xiao
Yiwen Dong
Junbo Zhao
Runze Wu
Minmin Lin
Gang Chen
Haobo Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Active learning
Large language model
Human-free zero-shot learning
TL;DR: A novel human-free active learning model by using LLMs

Abstract: Collecting high-quality labeled data for model training is notoriously time-consuming and labor-intensive for various NLP tasks. While copious solutions, such as active learning for small language models (SLMs) and prevalent in-context learning in the era of large language models (LLMs), have been proposed and alleviate the labeling burden to some extent, their performances are still subject to human intervention. It is still underexplored how to reduce the annotation cost in the LLMs era. To bridge this, we revolutionize traditional active learning and propose an innovative collaborative learning framework FreeAL to interactively distill and filter the task-specific knowledge from LLMs. During collaborative training, an LLM serves as an active annotator inculcating its coarse-grained knowledge, while a downstream SLM is incurred as a student to filter out high-quality in-context samples to feedback LLM for the subsequent label refinery. Extensive experiments on eight benchmark datasets demonstrate that FreeAL largely enhances the zero-shot performances for both SLM and LLM without any human supervision.

FreeAL: Towards Human-Free Active Learning in the Era of Large
Language Models
Ruixuan Xiao', Yiwen Dong', Junbo Zhao', Runze Wu?
Minmin Lin?, Gang Chen', Haobo Wang" 1+
'Zhejiang University, Hangzhou, China "NetEase Fuxi AI Lab, Hangzhou, China
(xiaoruixuan, dyw424, zhao, cg wanghaobo edu. cn (wurunze1, linminmind no jécorp. netease. com
Abstract
Collecting high-quality labeled data for model training is notoriously time-consuming and labor-intensive for various NLP tasks. While copious solutions, such active learning for small language models (SLMs) and prevalent in-context learning in era of large language models (LLMs), have been proposed and allevi- ate the labeling burden to some extent, their performances are still subject to human in- tervention. It is still underexplored how to reduce the annotation cost in the LLMs era. To bridge this, revolutionize traditional tive learning and propose an innovative col- laborative learning framework FreeAl. to in- teractively distill and filter the task-specific knowledge from LLMs. During collaborative training, an LLM serves as an active annota- tor inculcating its coarse-grained knowledge, while downstream SLM is incurred as a stu- dent filter out high-quality in-context sam- ples to feedback LLM for the subsequent la- bel refinery. Extensive experiments on eight benchmark datasets demonstrate that FreeAL largely enhances the zero-shot performances for both SLM and LLM without any human supervision. The code is available at https: //gi ctmp com/ Jus therozen/Freeee
94
Performance (%)
92
90
AL-Random
88
AL-Entropy
AL-CAL
Supervised line-tuning
86
FreeAL (Ours)
0
10% 20% 30% 40% 50% 60%ALL Acquired Human Annotation
Figure I: Comparisons of FreeAL with traditional active learning (AL) algorithms and supervised fine-tuning on the SST-2 dataset. FreeAL surpasses all the active learn- ing rivals and achieves near-supervised performance without human annotation.
2020; Yuan et al., 2020; Margatina al., 2021) is prominent solution that interactively queries an external expert or oracle mark the data points that the model wants to learn from. These methods alleviate the labeling burden to some ex- tent but still require human efforts the annotation or construction of the oracle to start with.
1 Introduction
The recent prevalent large language models (LLMs) (Ouyang et al., 2022: Thoppilan al., 2022: OpenAI, 2023), such as ChatGPT and PaLM (Chowdhery et al., 2022), have exhibited strong zero-shot learning ability by proper prompt design, yet becoming a new remedy for data efficiency. Even more inspiringly, LLMs emerge with the so-called in-context learning (ICL) (Brown et al., 2020) ability learn from few task-related la- beled samples for boosted performance. Despite the promise, some studies (Bang et al., 2023) find that I.LMs tend to underperform compared to fine- tuned small language models (SLMs) on challeng- ing tasks, which is also verified in our empirical
Modern machine learning models typically require a huge collection of precisely labeled data, which can be labor-intensive and time-consuming pro- cess. Even worse, it can be unrealistic in some practical scenarios that demand much expertise, such medical diagnosis and industrial applica- tions. To this end, plethora of approaches have been investigated to reduce the burden of anno- tation, including semi suneriupereieed learning (Sohn et al., 2020: Berthelot et al., 2019), learning with label noise (Han et al., 201 8; Li et al., 2020), and so on. Amongst them, active learning (Ein-Dor et al.,
*Corresponding author.

---

studies (Table 3). One possible reason is that ICL can not fully exploit supervised training samples due to limited context length. Moreover, their ex- tremely large size and limited accessibility also hinder their training and generalization on specific tasks. To date, it is still questionable how can we generalize to downstream tasks with the least hu- man annotation in the era of LLMs.
2 Related Work
2.1 Prompt-based Zero/Few-shot Learning
The emergent ability of LLMs has sparked height- ened interest in prompt-based zero-shot and few- shot learning (Ye et al., 2021; Schick and Schüizze. 2021). Instead of fine-tuning on massive down- stream data, in-context learning (ICL) (Brown et al., 2020), which suits LLMs to new tasks with few- shot input-label exemplars as demonstrations with- out training, has shown promising few-shot perfor- mance. It has been further improved by later works (Liu et al., 2022; Lu et al., 2022; SU et al., 2023). On the other hand, zero-shot learning is much more challenging without task task-specific data. Direct steering LLMs for predictions without in-context demonstrations can lead to significantly degraded performance (Gao et al., 2021). To bridge this, some methods (Wei et al., 2022; Sanh et al., 2022; Xu et al., 2022) adopt instruction tuning with a multi-tar task paradigm to further pre-train the LLMs with collection of different tasks in shared prompt- ing templates. However, these methods require cumbersome training for LLMs and the overwhelm- ing bulk of cross-tade task human annotations. Another new line of research (Ye et al., 2022a; Meng et al., 2022; Ye et al., 2022b) endeavors to ameliorate zero-shot learning merely via dataset generation, while the synthesized data commonly involves no- table portion of low-quality samples and misses the nuanced semanties present in the original data. In our work, we take inspiration from active learning with an innovative viewpoint to distill and filter the rich knowledge from LLMs for boosted zero-shot generalization performance.
In this work, we present novel collaborative learning paradigm FreeAL that revolutionizes tra- ditional active learning by interactively distilling and filtering the task-related knowledge from the LLMs. Our intuition that, while LLMs are hard to fine-tune, they are competent zero-shot learn- ers (Wei et al., 2022; Kojima et al., 2022) and can provide coarse- graised knowledge for downstream tasks. On the other hand, SLMs are effective weak learners (Li et al., 2020) that can distill valuable clean samples from noisy supervision. To integrate LLMs and SLMs synergistically as a whole, we design a collaborative training framework where LLM operates as an active annotator infusing its knowledge and the SLM acts as student to filter out the high-quality input-label pairs to feed back the LLM for subsequent label refinery. Empirically, FreeAL iteratively boosts the unsupervised perfor- mance of both SLMs and LLMs during collabo- rative training for transductive and inductive set- tings. As depicted in Figure 1, FreeAL allows us to achieve an extraordinary annotation- unnotation-performmnne performance trade-off by obtaining competitive results on par with the supervised counterparts while fully elimi- nating human annotation costs.
Overall, our main contributions can be summa- rized as follows,
To the best of our knowledge, we are among the first to overhaul traditional active learning in the era of LLMs for boosted generalization performance without any human supervision.
2.2 Active Learning
Active learning (AL) is prevailing paradigm in various NLP tasks (Yuan al., 2020; Zhao et al., 2020: Shelmanov al., 2021: Wang et al., 2022) that aims reduce labeling effort by selecting only the most useful examples to annotate. In each it- eration of active learning, a model is trained on the currently labeled data and then tasked with se- lecting the most informative yet-to-be te--to-bb-lablle data point to be labeled for boosted performance. Based on different querying strategies (Settles and Craven, 2008), existing traditional active learning methods can be categorized into uncertainty-based meth- ods (Prabhu et al., 2019; Margatina et al., 2021) and diversity based methods (Sener and Savarese, 2018; 111; Ru et al., 2020; Ash et al., 2020). While these
We propose novel collaborative learning framework called FreeAL to employ the LLMs active annotators and the SLMs as weak filters interactively distill the task- related knowledge from the LLMs.
Our proposed FreeAL largely improves the unsupervised learning performance for both the LLMs and the SLMs, even approaching the supervised counterparts in some scenarios. Our results prove the feasibility of human- active labeling in the era of LLMs.

---

Active Annotation by LLMs
LLM API
Instruction: ideniily the cmotion 'poottve' negative..
movic revlew
Stampl
Vote
Demonatration Pool
Demonstrations: (3) slick, engrossing [2] the film stricily routine: NNggliio
engrosiingeeoddeeee melodrams. Posiive
Round 1
& Round 3
Unlabeled Detaoet
slow:
to
Output:
Self-training
Construction of Demonstration Pool
for SLM
Feedback SLMs for Label Refinery
K-medodds
2.5,
for demonstration poal:
Index of samples selected
Lue polling y woxt aol..
It's wosts using the
Nogutive
Selection
Ccen
Derronstration Pool
&
Round
1,3....
moisy samples:
the nuiter dliggaie auild
tie ore yeur beer
Soutie
Veyatie
Noky Annotations
08000
refined
Notay Sutt
by LLM
Figure 2: Overview of Lr.eAL. In each collaborative training loop, the L.M serves as an active annotator imbuing its knowledge. Then the SLM is employed as filter distill the task-related knowledge with robust self- seff-training from LLM and filter out a high-quality demonstration pool Demoo to feedback the subsequent label refinery of LLM.
methods relieve the annotation burden to tent, they still count on human experts expensive supervision sources start with. To overcome this high cost, we investigate the opportunities of leveraging the rich knowledge of LLMs as a low- cost supervision source for boosting generalization performance without human effort.
during ICL,, construct prompt including test input zeest and m-shot in-context examples {anferm, ( 3/5= retrieved from D Demmo as the demonstration. The final prompt steers the LLM and the prediction obtained via,
arg max Ppey(((V T(adomo, Jdemo), .... T= glemo) T ((x
(I)
3 Background
Despite the simplicity, the success of ICL largely hinges on the demonstration pool Ddemo- which requires human efforts of careful annotation for every individual scenario and can be particularly annoying for challenging tasks. To bridge this gap. we resort our proposed plug-and-plas method FreeAL without involving any human supervision.
We consider unsupervised classification tasks with- out human annotations. Given an unlabeled train- ing dataset Dimin = {= with n samples, where x E X' is the input text and the corresponding ground-truth label C y (1....,C) is inac- cessible. Our task is to predict the true label for both the training dataset Dimin and test dataset Desst- Our framework employs pre-trained large lan- guage model (LLM) P and downstream small language model (SLM) S. For the LLM, we define a natural language template T(-) which contains additional task relatee information and a verbalizer V-)) which maps each class label in (1,...,C) C)} to pre-defined token in the prompt. For the fine- tuming of SLM S with parameters 0, we adopt the cross entropy loss 4 Ejer Vi log 5 (x,, 0) for training, where (1,,, 0) the j-th entry of SLM's output softmax probability for the input x;
FreeAL
In this section, we introduce our proposed frame- work FreeAL which investigates the opportunity for human-free active learning in the LLMs era. In contrast to traditional active learning that requests human annotation in each training loop, FreeAL employs LLMs weak annotators. In each train- ing loop, we alternate the following steps:
1. Active labeling of the to-be-labeled samples via LLMs based on the feedback from SLMs.
Training weakly supervised SLMs to distill the task-related knowledge from noisy anno- tations of LLMs and in turn feedback to them.
with the pseudo label J.d..
Few-shot In-context Learning. When super- vised data are available, we can directly em- ploy the few-shot ICI. for inference. In con- crete, given a demonstration supporting pool D Deemo = {ayleemo (.lemo) J for prompt retrieval
The overview of the FrecAL framework is dis- played in Figure and its overall pipeline also shown in Algorithm In what follows, we will elaborate on our FreeAL framework minutely.

---

4.1 Active Labeling by LLMs
Algorithm 1 Pipeline of FreeAL
Input: Unlabeled dataset P { pre- trained LLM and a downstream SLM S;
this step, we leverage the strong in-context learning ability of LLMs to assign weak labels to unsupervised training corpora. In particular, the core challenge lies in the construction of a proper prompt containing demonstration samples. To this end, we introduce two practical strategies for the different life cycles of FreeAL.
1: rownd < 1
2:
while not convergent do
3:
# For LLM: active annotation
4:
if round = 1 then
5:
# Initial self-generated demonstration
Generate { " as Eq.
Deemo D gen {x jsen);;
Initial Annotation by Self- Self-generated Demonstra- tion. At the initial round of FreeAL, we are given a purely unsupervised training set Drain- To en- able pseudo-labeling via LLMs, we may directly perform zero-shot ICL without access to a demon- stration pool Deemo However, such a strategy can largely impede the knowledge-distilingg process of SLMs due to shoddy initial annotations. To rem- edy this, we design novel self-generated demon-
else
Receive Deemo from SLM:
10: end if
11:
In-context learning as Eq.(1) for labeling;
12:
round + round + 1;
13:
# For SLM: knowledge distillation
14:
Robust self-training Eq.(3)
15:
# Construction of Ddmo
Filter out class-wise clean subset DJ
stration technique by virtual data generation. No- tably, when given some unlabeled samples and task- descriptive instructions, humans can imitate the ex- pression styles of these texts and leverage their own knowledge to generate similar label-aware samples. Motivated by this, we steer LLMs to first mimic the format of unlabeled samples from Drain which is important to ICL according to recent research (Min et al., 2022), and then generate new label-aware examples construct the initial Dacmo-
16:
clean
Adopt k-medoids on D'lean for demo
17:
Ddemo Ujer Dj Pdemo
18:
19:
DD Doois- = Durain \ (UJEYD's clean
20:
Feed Dacmo and Dosssy back to LLM
21:
round round + 1;
22:
end while
Refined Annotation in Subsequent Rounds. In the later rounds, the SLM S is trained using the weak annotation given by the L.MM P. Meanwhile, the SLM filters out high-quality demonstration pool Deemo as feedback; details are shown in Sec- tion 4.2. Then with a high-quality Ddemo- the LLM P re-annotates the remaining noisy-pron samples via few-shot ICL according to Eq. (1).
Specifically, the data-generation prompt con- tains hand-crafted task-descriptive instruction Pgen that explains the task background and Q randomly-selected unlabeled samples Cyen from D Prrain prototypes to imitate. An example of the prompt shown in Appendix B.3. The generation process can be formulated as,
((2.80, () <- P(pgen; T(Cgen))
(2)
4.2 Knowledge Distillation by SLMs
The generated samples constitute the generated dataset Dgen { (a } which then used as demonstration pool (i.e..Ddemo = Dgen) for the subsequent labeling. Next, we follow the standard ICL pipelines with demonstration selection (Liu et al., 2022). Each prompt contains m-nearest- neighbors from Daemo with the highest embedding similarity z,. The ICL process follows Eq.(1). With the demonstrations seen in the prompt, the LLM is able provide passable initial annotations ÿ of the training dataset Derain = (a;, 1 the annotations ÿ are employed as pscudo-labels for the subsequent training of SLMM.
Given the acquired weak annotations from LLM, it difficult for the LLM to distinguish its own errors due to the confirmation bias. Fortunately, previous studies (Han et al., 2018 8; Li et al., 2020) in weakly- supervised learning have shown that deep models have the potential of detecting noisy samples dur- ing the training procedure. Therefore, after receiv- ing weak labels, our intention two-fold: (i)-train strong and robust downstream SLM that maxi- mally distills task-specific knowledge; (ii)-employ the derived SLM to filter out a high-quality demon- stration pool to feedback LLM.

---

4.2.1 Robust Self-training
itself due the confirmation bias. Thanks our robust SLM, we can filter out those clean and rep- resentative samples and construct high-quality demonstration pool Deemo for the LLM to refur- bish its potentially wrong predictions in previous rounds. One may directly reuse the GMM-baser selection criterion again and take Di demonstra- tions. However, such selection procedure is too aggressive since excessively over-selecting some noisy samples may still improve the self-training procedure. To this end, we would like filter out a more curated Deemo that prioritizes representative examples with accurate labels to be included.
Motivated by the memorization effect of DNNs (Zhang al., 2017), the SLM tends to first fit easy patterns in the early stage of training. Thus, noisy samples mostly pose larger loss values. To this end, we adopt the selection-based technique (Li et al., 2020) from noisy label learning to train a robust SLM for knowledge distillation.
Formally, after few warm-up epochs with stan- dard training noisy labels, given the standard cross-entropy loss that reflects how well the model fits the sample Zi, we fit two-component GMM the loss I; to find out those clean samples. Let w; = p(g 1,) represent the probability of 2; belonging to the Gaussian component with smaller mean g. which can also be deemed as its clean probability. Then we divide the training dataset into a clean subset and a noisy subset by setting threshold on wi, which is considered as labeled set Dj and noisy set Du respectively,
The construction process mainly contains two steps in class-wise manner to cover every class and ensure diversity. For the training subset D Irain of class j. following the memory effect of DNNs (Zhang et al., 2017), we utilize the small loss eri- terion and select samples with the smallest cross- entropy loss 4 in the first R percent to construct Di = {( is) rank(4) < R%, 9; j}. In practice, we set a small R to ensure the high clean Secondly, we further adopt the
D:- [(r, (i) x4 E Durain 1h 2 T),
(3)
D. (x;) z; Durain W, < <T}
precision of D Di
simple clustering algorithm k-medoids embeddings of SLM to filter out the most repre- sentative medoids samples from D elean to construct DÍ demo When the k-medoids algorithm gets con- verged, the medoids of k clusters are collected D demo Finally the integral demonstration set is merged from each class as Duemo = Ujer D D'eemn With high quality D the great potential of LLM P can be unleashed to refine those noisy- prone samples Dosis Duaain (Ujey cleam) via few-shot ICL described in section 4.1.
To improve the robustness of training, we uti- lize consistency regularization for boosted perfor- mance, which assumes that classifier should pro- duce similar prediction on local neighbor of each data point. Given an input Zj, we adopt back- translation (Sennrich et al., 2016) to paraphrase and obtain the augmented version z;""". For the labeled and unlabeled data, the consistency regular- izations are formulated,
1/ = D
aug (k),
(4)
Experiment
Lu
(a((( ).
Duul
In this section, we provide the experimental results to verify the effectiveness of the FreeAL frame- work. More results, including visualizations and model selection, can be found in Appendix.
E,ED.
where loc and lu are standard cross entropy and KL divergence respectively. Finally, the total loss for self self-training of SLM is aggregated,
5.1 Setup
Luntal = Letean + a al'rrr L
(5)
Datasets. We evaluate the performance of FreeAL on both sequence-level and token-level tasks. For sequence-level tasks, we choose SST-2 (Socher et al., 2013), MR (Pang and Lee, 2005) dataset for sentiment classification, SUBJ (Pang and Lee, 2004) dataset for subjectivity classifica- tion and TREC (Voorhees and Tice, 2000) for topic classification. For token-level tasks, CoNLL03 (Tjong Kim Sang and De Meulder, 2003) dataset
where Leteun is the cross entropy loss Di, is the loss weight parameter. We refer readers to Appendix B.1 for more implementation details.
4.2.2 Demonstration Pool Filtering
While the SLM can filter out clean subset to enhance its performance during self- training other stubborn noisy labels are hard to correct by SLM

---

Model
Round Demons/Annos
SST-2
MR
SUBJ
TREC
CoNLL03
MA
BC5-C
BC5-D
Zero-shot
88.93
89.99
57.11
43.36
64.19
59.51
69.28
27.74
GPT-3,5-Turbo
Self-generated
92.16
91.74
86.54
70.74
70.89
59.78
81.05
47.12
Selected by Round
94.93
92.89
90.33
77.70
74.71
61.38
82.40
52.59
Annotated by Round
94.70
92.43
92.24
76.75
74.49
61.41
81.61
52.89
RoBERTa
4
Annotated by Round 3
95.49
92.64
92.85
81.59
78.79
62.15
82.81
59.25
Table 1: Comparisons of transductive performance training datasets of different tasks. BC5-C/D refers WASSDR-Chemica//DD dataset. For the token-level NER tasks (including CONLLO3, BC5-C, BC5-D) the F1-score given and For the other sequence-level tasks the test accuracy provided.
Model
Round
Demons/Annos
SST-2
MR
SUBJ
TREC
CoNLL03
MA
BCS-C
BC5-D
Zero-shot
92.47
90.05
55.65
77.20
66.47
59.71
67.85
29.60
GPT-3.5-Turbo
Self-generated
93.73
90.85
83.85
80.00
70.22
59.97
76.90
50.68
Selected by Round 2
95.91
93.10
90.27
79.80
70.80
60.93
80.77
52.70
Annotated by Round
94.29
89.35
86.80
71.82
61.91
80.55
53.38
RoBERTa
4
Annotated by Round 3
94.66
90.20
94.45
91.40
76.12
62.64
81.13
58.90
Table Comparisons of inductive performance test datasets of different tasks. BC5-C/D refers BC5CDR- Chemical/Diseee dataset. For the token-level NER tasks (including CoNLLO3, BC5-C, C, and BC5-D) the Fl-score is given and For other sequence-level tasks the test accuracy provided.
is adopted for named entity recognition (NER). To validate the feasibility of FreeAL in practical sce- narios such medical diagnosis and biochemi- cal applications that demand highly specialized domain-specific expertise, we also conduct exper- iments on BCSCDR (Li al., 2016) dataset with chemical and disease interactions as token-level NER tasks and Medical Abstract (MA) (Schopf et al., 2022) dataset describing 5 different classes of patient conditions as sequence-level classifica- tion task. More details are listed in Table 4.
Baselines. We compare FreeAL with multiple zero-shot and supervised baselines for LLMs and SLMs respectively. For LLMs, they vanilla zero-shot ICL without demonstrations (Brown al., 2020), supervised ICL with standard demon- stration retrieval (Liu al., 2022) from human- labeled training data, and supervised ICL with k- medoids to first filter a representative subset for demonstration retrieval. For SLMs, they are zero- shot distillation (Hinton et al., 2015; Smith et al., 2022) that finetunes the SLMs by using the annota- tions from zero-shot ICL of LLM ground-truths truths, and standard supervised fine-tuning that finetunes the SLM with human human-labeled labeled data. We also com- pare FreeAL with some traditional active learning basclines section 5.3.1, including (1) Random: It acquires annotation of to-be-labeled data randomly. (2) Entropy (Holub et al., 2008): It is the most commonly used uncertainty-basee baseline that ac- quires samples with the highest predictive entropy. (3) CAL (Margatina et al., 2021): It is recent active learning method that acquires contrastive examples for pre-trained language models.
Performance Evaluation. In this work, we eval- uate FreeAL from two aspects: (D-Transductive Performance: Given unsupervised training data, we evaluate the training accuracy of FreeAL which reflects how well task-specific knowledge is dis- tilled; Inductivee Generalization: utilize the derived models, including the SLM and Deemo for LLM, to further assess the generalization efficiency on the unseen test set with the inductive learning paradigm. We report the classification accuracy or the Fl score on both training and testing sets. We test the performance at different rounds. Round 0 denotes vanilla zero-shot learning of LLM. Round and round 2 denote the performance of LLM and SLM in the first training loop, while round 3 and are those of the second refinery training loop, as shown in Figure? 2. For all experiments, we rum three times and report the averaged results.
Implementation Details. We adopt OpenAl's GPT-3.5-Turbo language model, also known ChatGPT, as our LLM and use RoBERTa-Base from Huggingface Transformers (Wolf et al., 2020) as the downstream SLM. For the biomedical tasks including MA and BC5DER dataset, we utilize a

---

Model
Ablation
Human
SST-2
MR
SUBJ
TREC
CoNLL
MA
BC5-C
BC5-D
Zero-shot ICL
x
92.47
90.05
55.65
77.20
66.47
59.71
67.85
29.60
FreeAL (ours)
x
95.91
93.10
90.27
79.80
70.80
60.93
80.77
52.70
GPr--.5-Turbo
A Absolute gain
+3.44
+3.05
+34.6
+2.60
+4.33
+1.22
+12.9
+23.1
Supervised ICL (Standard)
96.06
92.85
89.30
81.50
85
61.22
82.24
68.63
Supervised ICL (R-medoids)
96.10
93.19
90.35
82.60
84.97
61.13
82.06
67.93
Zero-shot distillation
92.81
88.60)
59.25
82.80
69.71
61.22
77.05
31.98
FreeAL (ours)
X
94.66
90.20
94.45
91.40
76.12
62.64
81.13
58.90
RoBERTa
A Absolute gain
+1.85
+1.60
+35.2
+8.60
+6.41
+1.42
+4.08
+26.9
Supervised FT
94.89
91.05
95.95
96.70
88.11
63.96
87.26
75.38
Table 3: Performance comparison of FreeAL with the zero-shot and supervised counterparts on the test dataset. BC5-C/D refers to BCSCDR- Chemical/Diseas. dataset. The results of FreeAL are in bold. Supervised FT refers to supervised fine-tuning. The absolute gain indicates the improvement of FreeAl. compared to the zero-shot baseline.
Dataset
Domain
#Token
# #Train
#Test
the zero-shot ICL for the LLM by 3.44%, 3.05%, and 2.60% on the SST-2, MR, and TREC dataset re- spectively. It reaches a staggering lead of 44.6% on the SUBJ dataset where the LLM fails to adapt to on its own. In the medical diagnosis and biochemi- cal fields, FreeAL also exhibits a notable advantage of 12.9% and 23.1% % on the chemical and disease tasks of the BC5CDR dataset. FreeAL showcases similar trend of leading performance for the SLMs. Interestingly, In comparison to the supervised coun- terparts, FreeAL achieves competitive performance on par with these supervised rivals on some sim- ple tasks such as SST-2 and SUBJ datasets and greatly narrows the gap between the zero-shot and fully-supervised performances on other challeng- ing tasks. Notably, the performance can be further improved with more interaction rounds (also larger achieve satis-
SST-2
Sentiment cls
19.3
6,920
1,821
MR
Sentiment cls
21.6
8,662
2,000
SUBJ
Subjectivity cls
24.5
8,000
2,000
TREC
Topic cls
10.2
5,452
500
CoNLL03
NER
14.59
14,041
3,453
MA
Medical cls NER 25.92
205.3
11,550
2,888
BCSCDR
4,560
4,797
Table 4: A list of benchmarks used in the experiments. #Train and #Test indicate the size of the training and test dataset. #Token is the number of tokens on average for the corresponding training dataset.
Baiiedd-RBeRRaa-ba basee (Gururangan et al., 2020) that is pre-trained on the Semantic Scholar corpus SLM for boosted performance. For fair compar- isons, all the ICL processes of the LLM comprise m = 10 context examples as demonstrations ex- cept on MA dataset is adopted due to the maxi- mum context length 4,096 for GPT-3.5- Turbbo. The collaborative training process is performed on the training dataset first and then the fine-tuned SLM and the Dammo for LLM are utilized to be evalu- ated on the test dataset. More details of the robust self- self-training are put in Appendix B.2.
cost), but 4 rounds of interaction factory results empirically. These results suggest that FreeAL is able fully distill the task-related knowledge from LLMs' weak supervision. More analyses can be found in Section 5.3.2.
5.3 Analysis
5.3.1
Comparisons with Active Learning
5.2 Main Results
We also compare our FreeAL framework with some traditional active learning methods on the SST-2 and MR dataset. As shown in Table 5 and Fig- ure 1, It can be observed that FreeAL outstrips the traditional active learning basclines with 20% and 50% acquired human annotations, which fur- ther indicates that FreeAL can serve superior alternative to traditional active learning by lever- aging the rich knowledge of LLMs as a low-cost human-free supervision source.
Table and Table display the results of FreeAL at different rounds in the collaborative training progress on the training and test dataset for trans- ductive and inductive performance respectively. Ta- ble reports the comparisons of FreeAL with other zero-shot and supervised counterparts.
Based on these results, can be observed that FreeAL significantly enhances the unsupervised performance of both LLM and SLM. Free exceeds

---

Method
Human Anno
SST-2
MR
Model
Ablation
SST-2
MR
20% samples
92.42
88.10
FreeAL
94.66
90.20
Random
RoBERTa
50% samples
92.92
89.10
w/o robust self-training
89.18
88.95
20% samples
92.37
88.65
FreeAL
95,91
93.10
Entropy
50% samples
94.29
90.00
GPr-3.5-Turbo
with random selection
95.12
92.15
with entropy selection
95.65
92.67
20% samples
93.36
88.45
CAL
50% samples
94.56
89.75
Table 6: Ablation study of FreeAL for the SLM and LLM on the SST-2 dataset and MR dataset.
FreeAL (ours)
X
94.66
90.20
Table 5: Comparisons of FreeAL with traditional active learning algorithms on the SST-2 and MR dataset.
Fully Supervised ICL
FreCAL
SST-2
5.3.2 Effect of Collaborative Training
98.0
93.0
From a more nuanced perspective of the perfor- mance improvements at different rounds on the training set in Table 1, it can be noticed that FreeAL iteratively refines the noisy annotations during the collaborative training. The improvement from round 0 to 1 indicates the effectiveness of self- generated demonstrations for better initial annota- tions. The performance udvancements from rounds to 2 and rounds to 4 demonstrate the ability robust self-nraining for SLM to distill valuable knowledge from noisy annotations. Further, the per- formance boost from round 2 to round 3 verifies the efficacy of the active label refinery process. For the test dataset in Table 2, the performance changes fol- low similar trend with some fluctuations, which have been further discussed in Appendix A.
95.6
92.5
05.4
92.0
91.3
95.0
5
20
15
20
91.0
10
15
20
m
Figure 3: Ablation study of different numbers of in- context examples m on SST-2 and MR dataset,
the SST-2 and MR datasets. As shown in Figure 3, FreeAL is able to produce competitive perfor- mance to the supervised rivals over wide range of m from to 20, this further verifies the robustness of FleeAL. and we can simply adopt m = 10 for fair comparisons in our experiments.
To further validate the efficacy of collaborative training, we also conduct additional ablation exper- iments for the components of FreeAL as shown in Table 6. For the generalization performance on the SLM, we compare FreeAL with its variant that dis-
Conclusion
In this work, we overhaul the traditional active learning in the era of LLMs and propose a novel framework called FreeAL that merely relies on the knowledge of LLMs to enhance human-free gen- eralization performance. The key idea of FreeAL is to distill and filter the task-related knowledge from LLMs with collaborative framework, where the LLM is employed as an active annotator and the SLM is engaged valuable samples for label refinery. The empirical results indicate that FreeAL can largely improve unsupervised performance and reaches compara- ble performance with supervised rivals in some tasks. While our FreeAL framework operates au- tonomously without human supervision, it is flexi- ble and be easily boosted with additional lim- ited human supervision, which we leave for our future work. We hope that our work can spark heightened interest in developing new active anno- tation algorithms in the era LLMs.
cards robust self-training and adopts the standard cross entropy loss for training (including round and 4). be observed that robust self-training largely improves the performance of FreeAL. For the performance of LLM,, we ablate FreeAL with other selection strategies from traditional active learning rather than small loss selection, including random selection and entropy selection that selects samples with the lowest entropy values with the same budget small loss selection. We can that entropy selection slightly makes up for the poor performance of random selection, but still lags behind FreeAL by a notable margin.
weak learner filter out
5.3.3 Impact In-Context Examples m
Then, we show the effect of different numbers m in-context examples during the process of ICL.

---

Limitations
References
Our proposed FreeAL is a collaborative framework that aims to enhance unsupervised performance without human effort. Despite its effectiveness, there is still much potential for improvement. First, the effectiveness of FreeAL largely hinges on the strong ability of LLMs. For some domains that are extremely challenging or eccentrie, the com- monly adopted GPT-3.5- nowadays may fail to provide a qualified initial annotation, even with self- ceff-geeeated demonstrations. Our model is antic- ipated to be suitable for these circumstances with the advancement of more powerful LLMs across diverse domains. Besides, we thoroughly forgo human efforts in our FreeAL framework while in practical scenarios there may exist more or less available human support. It remains underexplored how to effectively combine the supervision from human experts and LLMs to synergize their individ- ual strengths, and we leave it for our future work.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy. John Langford, and Alekh Agarwal. 2020. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Yejin Bang, Samuel Cahyawijaya, Naycon Lee, Wen- liang Dai, Dan Su, Bryan Wilie. Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do. Yan Xu, and Pascale Fung. 2023. multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hal- lucination, and interactivity. CoRR, abs/2302.04023.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019. Mixmatch: A holistic approach to semi- supervised learning. In Advances in Neural Infor- mation Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5050-5060.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Ethics Statement
While our proposed FreeAL serves as an innova- tive way to enhance generalization performance without human intervention, the predictions and self- celf-generated demonstrations of the adopted LLM API may include bias and unfairness. Indeed, if one utilizes FreeAL with such biased annotations, it may unpleasantly yield unfair and biased pre- dictions based on characteristics like race, gender, disabilitics, LGBTQ, or political orientation. To alleviate this issue, we recommend that potential users first use bias reduction and correction tech- niques to remove biased text and predictions so as improve overall fairness and ethical standard.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-An Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lec, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, abs/2204.02311.
Acknowledgements
This work is majorly supported by the NSFC un- der Grants (No. 62206247), and in part by the National Key Research and Development Program of China (No. 2022YFB3304101). Junbo Zhao also thanks the sponsorship by the Fundamental Research Funds for the Central Universities (No. 226-2022-00028). This paper is also supported by Netease Youling Crowdsourcing Platform As the importance of data continues rising, Netease Youling Crowdsoureing Platform dedicated to utilizing various advanced algorithms to provide high-quality, low-noise labeled samples.
hhtps://fuxi https://fuxi1163..om 163.com

---

Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7949-7962, Online. Association for Computa- tional Linguistics.
Junnan Li, Richard Socher, and Steven C. H. Hoi. 2020. Dividemix: Learning with noisy labels as semi-supervised learning. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in co-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100- 1 114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Jiahui Gao, Renjie Pi. Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang. Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023, Self- guided noise-free data generation for efficient zero-shot learning. In The Eleventh International Conference on Learning Representations, ICIR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.ned
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Ferande 2020. Early-learning reg- ularization prevents memorization of noisy labels. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020. virtual.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- val Language Processing (Volume I: Long Papers), pages 38 3816-3830, Online. Association for Computa- tional Linguistics.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel. and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't i stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. 2021. Active learning by ac- quiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 650-663, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 8536-8546.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language mod- els: Towards zero-shot language understanding. In NewrIPS.
Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in neural network. CoRR, abs/1553.00253
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048 1 11064. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Alex Holub, Pietro Perona, and Michael C. Burl. 2008 Entropy-based active learning for object recognition. In IEEE Conference Computer Vision and Pattern Recognition, CVPR Workshops 2008, Anchorage, AK, USA, 23-28 June, 2008, pages 1-8. IEEE Computer Society.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In NeurlPS.
OpenAL. 2023. Gpt-4 technical report.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida. Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NewrIPS.
Jiao Li, Yucping Sun, Robin J. Johnson, Daniela Sci- aky, Chih Hiuhn Wei, Robert Leaman, Allan Peter Davis Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. Biocreative V CDR task corpus: a resource for chemical disease relation extraction. Database J. Biol. Databases Curation, 2016.

---

Bo Pang and Lillian Lee. 2004. A sentimental educa- tion: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACI.-04) pages 271 Barcelona, Spain.
Representations, ICLR 2018. Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceed- ings. OpenReview.net.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 86-96, Berlin. Germany. Association for Computational Lin- guistics.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploit- ing class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Compu-
tational Linguistics (ACL '05), pages 115-124, Ann Arbor, Michigan. Association for Computational Lin- guistics.
Burr Settles and Mark Craven. 2008. An analysis of active learning strategies for sequence labeling tasks. In Proceedings of the 2008 Conference Empiri- cal Methods in Natural Language Processing, pages 1070-1077 Honolulu, Hawaii. Association for Com- putational Linguistics.
Ameya Prabhu, Charles Dognin, and Maneesh Singh. 2019. Sampling bias in deep active classification: An empirical study. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP). pages 4058-4068, Hong Kong, China. Association for Computational Linguistics.
Artem Shelmanov, Dmitri Puzyrev. Lyubov Kupriyanova, Demis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, and Alexander Panchenko. 2021. Active learning for sequence tagging with deep pre-trained models and Bayesian uncertainty estimates. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1698 1712, Online. Association for Computational Linguistics.
Dongyu Ru, Jiangtao Feng, Lin Qiu, Hao Zhou, Mingx- uan Wang. Weinan Zhang, Yong Yu, and Lei Li. 2020. Active sentence learning by adversarial uncertainty sampling in discrete space. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pages 4908 4917, Online. Association for Computa- tional Linguisties.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, 7aid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari. Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Dehajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen. Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29. 2022. OpenReview.net.
Ryan Smith, Jason A. Fries, Braden Hancock, and Stephen H. Bach. 2022. Language models in the loop: Incorporating prompting into weak supervision. CoRR, abs/2205.02318.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference Empiri- cal Methods in Natural Language Processing, pages Seattle, Washington, USA. Association for Computational Linguistics.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin Raffel, Ekkn Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fix- match: Simplifying superviee learning with consistency and confidence. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 2020, virtual.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and Proceedings of the 16th Conference of the Faropean Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
natural language inference.
Hongjin SU. Jungo Kasai, Chen Henry Wu, Weijia Shi. Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf. Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Selective annotation makes language models better few-shot learners. In The Eleventh International Con- ference on Learning Representations.
Tim Schopf. Daniel Braun, and Florian Matthes. 2022. Evaluating unsupervised text classification: Zero-shot and similarity- baadd approaches. CoRR, abs/2211.16226
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du. YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng,
Ozan Sener and Silvio Savarese. 2018 Active learning for convolutional neural networks: A core-set ap- proach. In 6th International Conference on Learning

---

Amin Ghafouri. Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin HoffmaanJohn, Josh Lee, Lora Aroyo. Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Agüera y Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoe Le. 2022. Lamda: Language models for dialog applications. CoRR, abs/2201.082344
Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstratiins, pages 38-45, Online, Association for Computational Linguistics.
Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Wang Yanggang, Haiyu Li, and Zhilin Yang. 2022. Zero- Prompt: Scaling prompt- prompt-based pretraining to 1,000 tasks improves zero-shot generalization. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 4235-4252, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guisties.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CONLI--2003 shared task: tengpege-nddeendent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiang- tao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong, 2022a. ZeroGen: Efficient zero-shot learning via dataset generation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 11653- 11669, Abu Dhabi, United Arab Emirates, Association for Computational Lin- guistics.
Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. SIGIR 2000: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development Information Retrieval, July 24-28, 2000, Athens, Greece, pages 200-207, ACM.
Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng. Tao Yu, and Lingpeng Kong. 2022b. ProGen: Pro- gressive zero-shot dataset generation via in-context foedback. In Findings of the Association for Com- putational Linguistics: EMNLP 2022. pages 3671- 3683, Abu Dhabi, United Arab Emiirtes. Association for Computational Linguistics.
Shuohang Wang. Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want reduce la- beling cost? GPT-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195-4205. Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. CrossFit: A few-shot learning challenge for cross- task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7163-7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xudong Wang, Long Lian, and Stella X. Yu. 2022. Un- supervised selective labeling for more effective semi- supervised learning. In Computer Vision ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part XXX, vol- ume 13690 of Lecture Notes in Computer Science, pages 427-445. Springer.
Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. 2022, Active learning through a covering lens. In NewrlPS.
Yisen Wang. Xingjun Ma, Zaiyi Chen, Yuan Luo, Jin- feng Yi, and James Bailey. 2019. Symmetric cross entropy for robust learning with noisy labels. 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 322-330. IEEE.
Yue Yu, Lingkai Kong, Jieyu Zhang, Rongzhi Zhang, and Chao Zhang. 2022. AcTune: Uncertainty based active self-training for active fine-tuning of pretrained language models. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pages 1422-1436, Seattle, United States. Association for Computational Lin- guistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quae V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representu- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd- Graber. 2020. Cold-start active learning through self- supervised language modeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) pages 7935-7948, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen,
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. 2017. Understanding deep

---

learning requires rethinking generalization. In Sth International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nett
Source
SST-2
MR
SUBJ
TREC
MA
Human FreeAl.
0.11
0.11
0.11
0.11
0.55
1.2c
1.36
1.5e
8.5e
4.5e -3
Zhilu Zhang and Mert R. Sabuncu. 2018, Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems 31: Anmual Conference on Neu- ral Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 8792-8802.
Table 7: Comparisons of annotation cost(S) per example between human labeling and FreeAL.
Model
Round Annotations
SST-2
MR
Vanilla FreeAL
94.66
90.20
1 Initial 10% from LLM
87.97
81.20
RoBERTa
Yuekai Zhao, Haoran Zhang. Shuchang Zhou, and Zhi- hua Zhang. 2020. Active learning approaches to enhancing neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1796-1806, Online. Association for Computational Linguistics.
2
Another 10% from L.IM
93.69
87.75
3
Another 10% from LLM 93.76
88,95
Table 8: Results with multi-round annotation strategies.
sample, the ICL is performed at most twice as ini- tial annotations and refined annotations. It can be observed that FreeAL can serve as a much cheaper data labeler while achieving passable performance.
A Additional Experimental Results
A.1 Discussion on Model Selection
With our collaborative training paradigm, are able to interactively distill and filter task-related knowledge from LLMs. Empirically, our FreeAL method significantly enhances the zero-shot (dis- tillation) performance of both SLMs and LLMs as discussed in Section 5. One intriguing finding is that, in the majority of evaluation cases, the final SLMs outperform the LLMs. This observation can be attributed to the superior distillation ability of fine-tuning
When entrusted with a training set that is too large to label the entire dataset, the annotation cost can be further reduced by simple multi- round solution. The core idea is to rely more the seaally-supervisd--errree
capability of SLM to distill from small number of annotated labels. Specifically, for the initial annotation round of LLM, we randomly sample subset of P% sam- ples (empirically we set P 10) be annotated by LLM. After that, for robust self-training, we per- form the original training process for the labeled data Diabeled and simply extend the consistency regularization L for the noisy set Du to the origi- nally unlabeled data (i.e., D2 = D U Duntabeled For the demonstration pool filtering, the con- struction process of Ditem is the same, while for Dnoisy we randomly sample another subset of P% samples from the unlabeled samples to be anno- tated by LLM for the next iterations. The amount of iteration rounds can be larger than the original FreeAL if available to gradually distill the task- related knowledge with limited annotation cost.
SLMs during the weakly-supervise process. Consequently, we believe that SLMs re- main viable choice for practical deployment due to their impressive fine-tuned performance and low computational requirements. Furthermore, in more general scenarios, we recommend the utilization of validation set to determine the most suitable model for deployment.
A.2 FreeAL Can Reduce the Annotation Cost
As FreeAL solely depends the knowledge of LLM and not on human efforts, can naturally be leveraged as a low-cost data labeler in scenarios. In Table 7, we evaluate the cost disparity between FreeAL and human annotations. Follow- ing previous work ( (Wang et al., 2021), for human labeling, it costs SO. 11 per 50 input tokens with a minimum of $0. 1. For FreeAL, the cost per exam- pic for m-shot inference is estimated approximately (#Token x (m + 1) 100) x (2 x 10 10-9). where #T ooken is the average token numbers in Table 4, (2 10 6) is the cost for GPrro.5-Turbo per token, 100 is roughly the tokens for the task- specific deseriptions and the model reply. For each
As shown in the Table 8, such a simple remedy is able to achieve competitive results close to the original FreeAL with merely 10% of the previous cost each round, which proves the feasibility of FreeAL when we cannot afford to label the entire dataset. Notably, the process of randomly sam- pling the to-be-annotated subset on SLMs can be further improved with other advanced query strate- gies (c.g., sesertainty-basedd) which is a classic topic in traditional active learning.

---

Model
Ablation
Round
SST-2
MR
FreeAL
Round > 93.73 > 95.91
90.85 + 93.10
GPT-3.5-Turbo
FreeAL. w/ interaction
Round +
93.73 = 95.37
90.85 = 92.15
FreeAL
Round =
94.29 94.66
89.35 = 90.20
RoBERTa
FreeAL w/o interaction
Round 2 =
94.29 > 94.27
89.35 = 89.55
Table 9; Ablation results of the interaction between the LIM and SLM for FreeAl. on the SST-2 and MR dataset.
Model
SST-2
MR
SUBJ
Method
Human Anno
SST-2
MR
RoBERTa- Base
94.66
90.20
94.45
20% samples
92,92
87.95
ProbCover
RoBERTa-Large
95.83
91.15
95.80
50% samples
93.49
89.75
20% samples
93.14
88.15
Table 10: Comparisons of FreeAL with different SLMs.
BADGE
50% samples
93.97
89.90
20% samples
92.53
87.55
Region Entropy
A.3 Impact of SLM's Size
50% samples
94,03
88.75
20% samples
92.37
88.20
We also conduct experiments to reveal the impact the size of SLM. As depicted in Table 10, when the size of SLM grows larger from RoBERTa- Base to RoBERTa-Large, FreeAL displays superior performance. This observation indicates that our FreeAL is compatible with different sizes of down- stream SLM and the performance be further improved with a larger SLM.
Region CAL
50% samples
92.70
89.00
FreeAL (ours)
94.66
90.20
Table 11: Comparisons of FreeAL with some other active learning algorithms the SST-2 and MR dataset.
6.9k. Empirically, our FreeAL still outperforms these dataset- seauusttgenerttionn based methods by a no- table margin, as shown in Table 12.
A.4 Comparisons with Other AL Methods
Here we provide comparisons with some other ac- tive learning selection strategies, including Prob- Cover (Yehuda et al., 2022), BADGE (Ash et al., 2020), Region Entropy and Region CAL (Yu et al., 2022) in the Table 11. It can be observed that FreeAL exceeds all its rivals, which consistently demonstrates the superior performance of FreeAL..
A.6 Results with More Distillation Methods
We also provide the comparisons with some other robust distillation methods, including GCE (Zhang and Sabuncu, 2018), SL (Wang et al., 2019) and ELR (Liu et al., 2020) in Table 13. We can see that FreeAL largely advances the performances of all these distillation baselines. Overall, FreeAL is designed a flexible framework and choose an empirically strong self-training algorithm for distil- lation to prove the feasibility of human-free active learning. One may design more power distillation algorithms for improved results, which we leave for future work.
A.5 Comparisons with Dataset-generation-be based Methods
We further supplement comparisons with some lataset-generation-bee
methods, including Ze- roGen (Ye et al., 2022a), ProGen (Ye al., 2022b) and SunGen (Gao al., 2023). Our FreeAL is fundamentally different from them in several per- spectives. First, these kataset-generation based methods are tailored for an extreme scenario where training data is completely missing, which is un- practical in reality. Second, these methods typi- cally generate low-quality samples, because they overlook the nuances and semantics present in the original authentic data. As a result, they mostly require generating huge amount of synthetic data for decent performance. For example, on the SST-2 dataset, these methods generate 200k synthesized samples while authentic training samples are only
A.7 Effect of Interaction for LLM and SLM
To further demonstrate the importance of interac- tion between the LLM and the SLM. We provide the inductive performance for FreeAL without in- teraction. For the L.M,, it directly adopts its own predictions the training dataset at round 1 as the demonstration pool directly for testing. While the SLM employs its own predicted labels supervi- sion at round directly for testing.
As displayed Table 9, we observe that the SLM itself hard to distill from its own predic-

---

Method
SST-2
SUBJ
Method
SST-2
MR
ZeroGen
87.27
80.45
Zero-shot distillation
92.81
88.60
ProGinn
88.42
FreeAL. with GCE
93.68
88.90
SunGen
89.45
83.25
FreeAL with SI.
93.91
89.50
FreeAL (DiilBERT)
91.82
92.15
FreeAL with ELR
94.01
89.70
FreeAL (RoBERTa)
94.66
94.45
Vanilla FreeAL
94.66
90.20
Comparisons of FreeAL with dataset- generation-based methods. We adopt DistilBERT as the SLM of FreeAL for fair comparisons.
Table 12:
Table 13: Comparisons with more distillation methods.
100
tions due to the inevitable confirmation bias, c.g..
improves 0.2% compared with FreeAL's improve- ment of 0.85% on the MR dataset and even de- grades on the SST-2 dataset. For the LLM, it can
Performance
80
self-improve itself, but still underperforms our col-
60
laborative mechanism. Notably, LLM has an in-
DR
escapable upper bound on the performance, accord- ing to our empirical findings where SLM outper- forms LLM on 6 out of total of 8 datasets. Such results indicate that the interaction between LLM and SLM bring
50
SUBJ
TREC
CONILOS
40
0
2 Round
4
opportunities converge consensus result between them.
Figure 4: Performance of FreeAL. the training different rounds during collahorative training.
A.8 Additional Visualization Results
We further provide some additional visualization results, including the transductive performance on the training dataset (i.e., the accuracy of pseudo labels) at different rounds Figure and the vi- sualization of comparisons with traditional active learning methods on the MR dataset in Figure 5.
B.2 More Implementation Details
In our experiments, for the LLM API, we follow the default official settings of the GPT-3.5-Tume 0301 version. In the demonstration retrieval of ICL, adopt the unsupervised embeddings with bert- base-uncased at the initial annotation round and the embeddings of SLM for later rounds. The construc- tion and retrieval of demo are both performed in a class-wiss wise manner to compose the final demonstra- tions. For the robust self training of SLM, we adopt the hyperparameters either from previous works or fixed at moderate value empirically without care- ful tuning. We finetune the SLM on the basis of the trainer of Huggingface for 50 epochs. The batch size fixed 32 with maximum sequence length of 128. We adopt the AdamW optimizer with a learning rate selected from {3e 4, 3e - 5, 3e --3e-6) and a weight decay of 0.01. For robust self-training, the threshold r of GMM selection is fixed at 0.7 and the ratio R of demonstration selection is fixed at 20. The loss weight parameter a is linearly ramped up from 0 to to avoid overfitting false labels at the start. For evaluation of performance for LLM, as LLMs sometimes output ambiguous predictions outside the label space, these values are treated as random labels in the label space and repeated multiple times to evaluate the average performance
B Additional Implementation Details
B.1 More Details of Robust Self-training
During robust self-training, also involve mixup training strategy that interpolates the em- beddings and the corresponding pseudo labels on the clean subset D, to encourage linear behavior be- tween samples. A virtual mixed training example is generated by linearly interpolating the randomly sampled pair of examples (ma, (i) and (æy, (i)) in D; and taking convex combination of labels as the regression target,
Emb(x"") = =Emb(x)) + (1-o7 Emb(2z)
(6)
yTh = oÿj + ,)ú;
where Emb(x;) is the embedding of x; and Beta(s. s) and we simply set 4. The mixup loss is denoted as Imix- the total loss for self- training of SLM is aggregated,
Lotal Lelean + a(Ler + Lmix)

---

Step
Prompt Details
You are required to produce 100 English examples with labels for the task of text classification on the MR (Movie Review) dataset. These samples will be used as prompt examples for the GPT model. MR dataset is used in sentiment-analysis experiments and this dataset contains movie- review documents labeled with respect to their overall sentiment polarity (positive or negative). The task is to classify a movie review as positive or negative according to their overall sentiment polarity. For example, 100 of the unlabeled samples MR dataset are as follows: ["review": "enigma is well-made but it's just too dry and too placid ["review": "the weakest of the four harry potter books has been transformed into the stronger of the two films by the thinnest of margins "......
Demonstration Generation
You are helpful assistant for the task of text classification on the MR (Movie Review) dataset. You reply with brief, to-the-point answers with elaboration truthfully possible. MR (Movie Review) dataset is used in sentiment-analysis experiments and this dataset contains movie- review documents labeled with respect to their overall sentiment polarity (positive or negative). Your task is to a binary classification to classify a movie review as positive or negative according to their overall sentiment polarity. The category is divided into two types: 'positive' and 'negative'. Given a movic review: <QUERY>, How do you feel about the sentiment polarity of the given movie review, is this positive or negative? please answer in single line with 'positive' or 'negative'.
Active Annotation
Table 14: An example of prompt design the MR dataset for the step of demonstration generation and active annotations. The in-context examples are omitted for the active annotation process here.
93
selection process is performed the token level in different manner, select those tokens with high confidence and matched predictions to pseudo- labels as clean and then filter out those samples whose tokens are clean to constitute the clean subset. The consistency regularization and mixup loss are only suitable for the sequence-leved tasks and are disabled in the teken-level NER tasks.
92
Performance (%)
91
88
AL-Random
87
AL-Entropy
86
Supervised Fine-tuning
B.3 Prompt Design
FreeAL (Ours)
85
We provide our prompt design on the MR dataset for the initial demonstration generation step and tive annotation step in Table 14. Notably, we adopt the GPT-3.5-Turbo our LLM so the prompts are also in the chat style with instructions.
10% 20% 30% 40% 50% 60%ALL Acquired Human Annotation
Figure 5: Comparisons of FreeAL with traditional active learning (AL) algorithms and supervised fine-tuning on the MR dataset. FreeAL surpasses all the active learning rivals and achieves near-supervised performance with- out human annotation.
during evaluation. Then in subsequent rounds, the SLMs adopt their own previous predictions to re- place these ambiguous annotations LLMs for robust self-training. For teven-level tasks, as the


Reviews:
[{"review_id": "UX1Sp8WeJY", "reviewer": "Reviewer_7Urc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your time and effort in reviewing our paper and providing thoughtful feedback to improve our paper. We hope the following responses can clarify your confusion, and we are happy to provide additional explanations if needed.\n\n**Q1: Lacking comparisons with other baselines. There are many other AL sample-selection strategies, such as [1].**\n\nA1: Thanks for your advice. We have followed your advice to provide comparisons with some other AL sample-selection strategies, including ProbCover [1], BADGE [C1], Region Entropy and Region CAL [C2]. The results are shown in the table below. It can be observed that FreeAL exceeds all its rivals **despite they require 20% and 50% human-annotated data**, which consistently demonstrates the superior performance of FreeAL. \n\n| Method     &emsp; &emsp; &emsp;      | Human Annotation &emsp; &emsp;   | SST-2  &emsp; &emsp;     | MR   &emsp; &emsp;       |\n| -------------- | ---------------- | --------- | --------- |\n| FreeAL     | None         | **94.66** | **90.20** |\n| ProbCover      | 20% samples      | 92.92     | 87.95     |\n| ProbCover      | 50% samples      | 93.49     | 89.75     |\n| BADGE          | 20% samples      | 93.14     | 88.15     |\n| BADGE          | 50% samples      | 93.97     | 89.90     |\n| Region Entropy | 20% samples      | 92.53     | 87.55     |\n| Region Entropy | 50% samples      | 94.03     | 88.75     |\n| Region CAL     | 20% samples      | 92.37     | 88.20     |\n| Region CAL     | 50% samples      | 92.70     | 89.00     |\n\n*[C1] Ash J T, Zhang C, Krishnamurthy A, et al. Deep batch active learning by diverse, uncertain gradient lower bounds[J]. arXiv preprint arXiv:1906.03671, 2019.*\n\n*[C2] Yu Y, Kong L, Zhang J, et al. AcTune: Uncertainty-based active self-training for active fine-tuning of pretrained language models[C]//Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022: 1422-1436.*\n\n\n\n**Q2: Leveraging LLMs as training data generators has been explored before. For instance, SunGen[2] and ZeroGen[3]. Comparison with these sota methods is necessary in my view.**\n\nA2:   Thanks for the remark. First, we'd like to clarify that these dataset-generation-based methods are fundamentally different from our FreeAL. \n\n- **Formulation and motivations:** These dataset-generation-based methods [2,3] are tailored for an extreme scenario where training data is completely missing, which is unpractical in reality. In contrast, our FreeAL concentrates on how to generalize better to downstream tasks with the least human annotation given unlabeled authentic data. While data synthesizing is also encapsulated in FreeAL, it's a rather simple step for improved performance, but we believe it doesn't affect our core contribution in providing an advanced active labeling framework. \n- **Data Quality:** We find that dataset-generation-based methods typically (and inevitably) generate low-quality samples, because they overlook the nuances and semantics present in the original authentic data, but only rely on the knowledge of LLM. This can largely hinder their generalization ability to specific downstream applications where the distributions deviate. To avoid this, these methods mostly assume the label distribution is known or uniform as a prerequisite, which is unrealistic in real-world deployment.\n- **Data Quantity:** Originates from the quality issue, these methods mostly require generating a huge amount of synthetic data for decent performance. For example, on the SST-2 dataset, these methods generate **200k synthesized samples** cumbersomely while the amount of authentic training samples is only 6.9k. This can result in a huge burden of computation and storage. \n- **Performance:** Despite a huge gap in the data size, our FreeAL still outperforms these dataset-generation-based methods by a notable margin. To see this, we compare our FreeAL with ZeroGen and SunGen. We adopt DistilBERT as the SLM of FreeAL for fair comparisons. The experimental results on SST-2 and SUBJ are shown below, where we can see that FreeAL leads by a significant margin.\n\n| Method  &emsp; &emsp; &emsp; &emsp; &emsp;              | SST-2 &emsp; &emsp; &emsp;        | SUBJ   &emsp; &emsp; &emsp;   |\n| ----------------------- | --------- | --------- |\n| ZeroGen                 | 87.27     | 80.45     |\n| SunGen                  | 89.45     | 83.25     |\n| FreeAL (DistilBERT) | **91.82** | **92.15** |\n| FreeAL (RoBERTa)    | **94.66** | **94.45** |\n\nTherefore, we believe our work studies a rather different setup that can be much more practical and efficient with significantly better performance. Despite these advantages of FreeAL, we'd like to emphasize that our framework is designed to be a valuable plug-in helper that is flexible and compatible with different off-the-shelf data synthetic, distillation, and data selection algorithms. We'll add our new empirical results and discussions to the revision. \n\n\n\n\n\n\n\n**Q3: Also, the authors should consider some distillation methods as baselines as utilizing LLMs as annotators are kind of distillation.**\n\nA3: For the distillation baselines, since we regard the LLM as a black-box model like GPT-3.5/4 where only the output text is accessible, we have compared our FreeAL with the zero-shot distillation baseline that finetunes the SLM with standard cross-entropy loss using the annotations from the ICL of LLMs as ground-truths in Table 3 of our original submitted version. We further provide the comparisons with some robust distillation methods, including GCE [C1], SL [C2] and ELR [C3] in the table below. We can see that FreeAL largely advances the performances of all these distillation baselines.\n\n| Method     &emsp; &emsp; &emsp; &emsp; &emsp;             | SST-2    &emsp; &emsp; &emsp;    | MR    &emsp; &emsp; &emsp;   |\n| ---------------------- | --------- | --------- |\n| Zero-shot distillation | 92.81     | 88.60     |\n| FreeAL             | **94.66** | **90.20** |\n| FreeAL with GCE        | 93.68     | 88.90     |\n| FreeAL with SL         | 93.91     | 89.50     |\n| FreeAL with ELR        | 94.01     | 89.70     |\n\nDespite our new results within a short period of response time, we'd like to clarify that the primary goal of our work is to investigate the feasibility of human-free AL in the era of LLMs. Thus, we designed our FreeAL to be a flexible framework. Specifically, we chose an empirically strong self-training algorithm for distillation that proves the feasibility of human-free AL. One may design more power distillation algorithms for improved results, which we leave for future work. \n\n\n\n\n*[C1] Zhang Z, Sabuncu M. Generalized cross entropy loss for training deep neural networks with noisy labels[J]. Advances in neural information processing systems, 2018, 31.*\n\n*[C2] Wang Y, Ma X, Chen Z, et al. Symmetric cross entropy for robust learning with noisy labels[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 322-330.*\n\n*[C3] Liu S, Niles-Weed J, Razavian N, et al. Early-learning regularization prevents memorization of noisy labels[J]. Advances in neural information processing systems, 2020, 33: 20331-20342.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and well-organized.\n2. The evaluation is comprehensive. Experimental results outperform baseline methods with large margins.\n3. The idea is rather simple and straightforward.", "reasons_to_reject": "1. Lacking comparisons with other baselines. There are many other AL sample-selection strategies, such as [1].\n2. Leveraging LLMs as training data generators has been explored before. For instance, SunGen[2] and ZeroGen[3]. Comparison with these sota methods is necessary in my view. Also, the authors should consider some distillation methods as baselines as utilizing LLMs as annotators are kind of distillation.\n\n[1] Yehuda, Ofer, et al. \"Active learning through a covering lens.\" Advances in Neural Information Processing Systems 35 (2022): 22354-22367.\n\n[2] Ye, Jiacheng, et al. \"Zerogen: Efficient zero-shot learning via dataset generation.\" arXiv preprint arXiv:2202.07922 (2022).\n\n[3] Gao, Jiahui, et al. \"Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning.\" The Eleventh International Conference on Learning Representations. 2022.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes FreeAL, a human-free active learning framework that leverages powerful LLMs as data annotators. Specifically, FreeAL consists of active annotation, robust self-training, and filtering noisy data. Experimental results demonstrate the proposed FreeAL outperforms baseline methods with large margins.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0OUvzcrlBS", "reviewer": "Reviewer_3J9a", "summary": "", "questions": "- What is the superscript j in Algorithm 1?\n- Related to my second reason to reject, I am somewhat curious about how FreeAL could be used in situations where you cannot afford to use an LLM to label the entire pool of data?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and suggestions! We are encouraged to see that you find our methods novel and effective. Below are our detailed responses.\n\n**Q1: \"Robust self-training\" involves noisy label learning as well as backtranslation (which I expect is a somewhat expensive procedure), however I did not find evidence supporting that both of these are necessary.**\n\nA1: Thanks for your remark. The noisy label learning and the back-translation (i.e., the consistency regularization) are widely adopted in the weakly supervised learning literature and are both components of the robust self-training for SLMs in our FreeAL, which has been shown to be effective in Table 6. We further conduct additional ablation experiments for these two components separately: 1) FreeAL w/o noisy label learning which forgoes the dataset division of GMM and regards all the samples as clean; 2) FreeAL w/o consistency regularization which discards the augmentation of backtranslation. It can be observed from the table below that the variants without either of these components suffer from perceptible performance degradation, but still beat the distillation and most of the active learning baselines by a notable margin.\n\n| Ablation                              | SST-2     | MR        |\n| ------------------------------------- | --------- | --------- |\n| FreeAL                          | **94.66** | **90.20** |\n| FreeAL w/o noisy label learning       | 93.97     | 89.45     |\n| FreeAL w/o consistency regularization | 93.81     | 89.25     |\n| Zero-shot distillation                | 92.81     | 88.60     |\n\nNotably, we originally tend to design a plug-and-play framework that is compatible with different distillation and augmentation strategies for SLMs, as discussed in the response to the ```#Reviewer 7Urc.Q2```. Hence we simply adopt the fashionable back-translation for augmentation, which is achieved directly via off-the-shelf translation models. Its cost can be further alleviated by adopting other augmentation methods instead (e.g., synonym replacement, random insertion/deletion). We'll add more results and discussions in our revised version.\n\n\n\n\n\n**Q2: I am not sure whether the comparisons presented in Table 5 are fair. If I understand correctly, FreeAL is allowed access to 100% of the data while the other methods only see a subset. I think it would be fairer if FreeAL were similarly limited.**\n\nA2: We kindly note that there might be some misunderstanding on the pipelines of traditional active learning that ```they actually also receives all training sample in every selection and training loop```. Specifically, a classical AL algorithm has the following steps in each loop:\n\n(1) Compute the metrics on **all (unlabeled) training samples**, and proactively select a subset it wants to learn from by ranking the metrics.\n\n(2) Label the subset by the external source --- human experts or oracle (cannot label the whole training set).\n\n(3) Training the SLMs with given data and labels, then loop to step (1).\n\nIn summary, traditional AL is also allowed to access the whole training set like ours and the comparisons in Table 5 are indeed fair. Notably, thanks to the great power and low cost of LLMs, our FreeAL can provide much more weak labels for SLMs than traditional AL. We believe it's our core superiority that can break the constraint of involving human labelers who can only annotate few samples. \n\n\n\n**Q3:  What is the superscript j in Algorithm 1?**\n\nA3: Sorry for confusing the reviewer! The superscript j in Algorithm 1 refers to the class index as mentioned in line 343 of section 4.2.2. The construction process of the demonstration pool $D_{demo}$ is performed in a class-wise manner to ensure diversity, which filters out the demonstration sample class by class and then merges them to derive the demonstration pool. $\\mathcal{D}_{\\text{clean}}^j=\\\\{(x_i,\\tilde{y}_i) \\mid rank(l_i)\\le R\\\\%,\\tilde{y}_i=j \\\\}$ refers to the subset of clean samples with the observed labels equal $j$.\n\n\n\n\n\n**Q4:  Related to my second reason to reject, I am somewhat curious about how FreeAL could be used in situations where you cannot afford to use an LLM to label the entire pool of data?**\n\nA4:  Thanks for this remark! Firstly, we have to emphasize that the annotation cost of FreeAL is much lower than that the humans (about 1%)  on common labeling tasks, as discussed in Appendix A.2. This can be even more appealing when human annotation can be very expensive, e.g. medical analysis and industrial control. \n\nSecondly, we show this can also be reduced by a simple multi-round solution where LLMs annotate a subset of samples in each loop. The core idea is to rely more on the weakly-supervised-learning capability of the SLM to distill from a small number of annotated labels. Specifically, for the initial annotation round of LLM, we randomly sample a subset of P% samples (empirically we set P=10 here) from the training set to be annotated by LLM. After that, for robust self-training, we perform the original training process of FreeAL for the labeled data $D_{labeled}$ and simply extend the consistency regularization $L_{cr}^u$ for the noisy set $D_u$ to the originally unlabeled data (i.e., $D_u=D_u\\cup D_{unlabeled}$). For the demonstration pool filtering, the construction process of $D_{demo}$ is the same, while for $D_{noisy}$ we randomly sample another subset of P% samples from the unlabeled samples to be annotated by LLM for the next iterations. The amount of iteration rounds can be larger than the original FreeAL if available to gradually distill the task-related knowledge with limited annotation cost. \n\n| Model       | Round | Annotations                          | SST-2     | MR        |\n| ----------- | :-----: | ------------------------------------ | --------- | --------- |\n| RoBERTa | - | Vanilla FreeAL                   | **94.66** | **90.20** |\n| RoBERTa     | 1     | Initial 10% samples annotated by LLM | 87.97     | 81.20     |\n| RoBERTa     | 2     | Another 10% samples annotated by LLM | 93.69     | 87.75     |\n| RoBERTa     | 3     | Another 10% samples annotated by LLM | 93.76     | 88.95     |\n\nAs shown in the table above, such a simple remedy is able to achieve competitive results close to the original FreeAL with merely 10% of the previous cost each round, which proves the feasibility of FreeAL when we cannot afford to label the entire dataset. Notably, the process of randomly sampling the to-be-annotated subset on SLMs can be further improved with other advanced query strategies (e.g., uncertainty-based), which is a classic topic in traditional active learning.\n\n\n\n**Q5: Typos and presentation**\n\nA5: Thanks for your remark. We will double-check the typos and revise the inappropriate expressions in our revision to make them more self-contained.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Label generation/distillation using LLMs is currently recieving a lot of focus from the NLP community, and this paper provides an approach that significantly improves upon zero-shot learning without requiring any human annotations.\n- The student model performance is often competitive with full finetuning.\n- Given how complex the overall system is, I think the paper does a good job clearly explaining each component, and the experimentation (mostly) justifies each component's inclusion.", "reasons_to_reject": "While I believe this is overall a strong paper, there are a few aspects of the experiments that negatively impact my evaluation of its soundness:\n- \"Robust self-training\" involves noisy label learning as well as backtranslation (which I expect is a somewhat expensive procedure), however I did not find evidence supporting that both of these are necessary.\n- I am not sure whether the comparisons presented in Table 5 are fair. If I understand correctly, FreeAL is allowed access to 100% of the data while the other methods only see a subset. I think it would be fairer if FreeAL were similarly limited.\n", "questions_to_authors": "- What is the superscript j in Algorithm 1?\n- Related to my second reason to reject, I am somewhat curious about how FreeAL could be used in situations where you cannot afford to use an LLM to label the entire pool of data?", "paper_topic_and_main_contributions": "This paper contributes FreeAL, an annotation-free active-learning-based approach for distilling knowledge from LLMs (GPT 3.5) into a student model (RoBERTa).\nAt a high level the approach consists of iteratitively repeating the following steps: 1. generating labels using in-context learning with an LLM, 2. training the student model on labels from the LLM, and 3. using the loss scores from the student model to partition the example set into \"likely clean\" labels (used in subsequent round as in-context examples) and \"likely noisy\" labels (which are relabeled by the LLM in subsequent rounds).\nTo completely mitigate the need for human annotations, the initial set of in-context examples is constructed using self-generated demonstrations from the LLM.\nThe paper provides experimental results on 8 classification datasets.\nThe first set of results illustrate how the FreeAL's performance improves as each of the above steps are carried out.\nThe second set of results compares FreeAL to a purely zero-shot setup and fully supervised baselines.\nThe third set of results compared FreeAL to active learning using human labeled examples.\nThe last set of results provides an ablation study measuring the benefit of the different design decisions, and number of in-context training examples.\nOverall the results appear to be quite positive:\n- In all cases FreeAL improves upon labeling/distilling using zero-shot learning, sometimes by quite a large margin (+20-30% absolute improvement in accuracy/f1)\n- FreeAL is competitive with fully supervised performance on a number of tasks\n- FreeAL outperforms active learning using human annotations\n- The each step of the approach is shown to provide benefit.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Some of the language is a bit hyperbolic, e.g., saying the method \"revolutionizes traditional active learning\". I'd personally recommend toning it down.\n\n- 218 - minutely\n- 504 - tabel\n- In tables 1 and 2, the numbers in parentheses look like references to equations rather than rounds.\n"}}, {"review_id": "jjIxSnzSRj", "reviewer": "Reviewer_3RFu", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your effort in reviewing our paper and providing constructive feedback. We are glad to see that you find the problem we study practical, our method well-motivated and effective. We provide detailed responses to address the concerns below.\n\n**Q1: Missing reference and comparisons**\n\nA1:  Thanks for the remark. We would like to clarify that these dataset-generation-based methods are fundamentally different from our FreeAL. \n\n* **Formulation and motivations:**  These dataset-generation-based methods [3-5] are tailored for an extreme scenario where training data is completely missing, which is unpractical in reality. In contrast, our FreeAL concentrates on how to generalize better to downstream tasks with the least human annotation given unlabeled authentic data. While data synthesizing is also encapsulated in FreeAL, it's a rather simple step to provide passable initial annotations for boosted performance, but we believe it doesn't affect our core contribution in providing an advanced active labeling framework. \n* **Data Quality:** We find that dataset-generation-based methods typically (and inevitably) generate low-quality samples, because they overlook the nuances and semantics present in the original authentic data, but only rely on the knowledge of LLM. This can largely hinder their generalization ability to specific downstream applications where the distributions deviate. To avoid this, these methods mostly assume the label distribution is known or uniform as a prerequisite, which is unrealistic in real-world deployment.\n* **Data Quantity:** Originates from the quality issue, these methods mostly require generating a huge amount of synthetic data for decent performance. For example, on the SST-2 dataset, these methods generate **200k synthesized samples** cumbersomely while the amount of authentic training samples is only 6.9k. This can result in a huge burden of computation and storage. \n* **Performance:** Despite a huge gap in the data size, our FreeAL still outperforms these dataset-generation-based methods by a notable margin. To see this, we compare our FreeAL with ZeroGen, ProGen and SunGen. We adopt DistilBERT as the SLM of FreeAL for fair comparisons. The experimental results on SST-2 and SUBJ are shown below, where we can see that FreeAL leads by a significant margin.\n\n| Method     &emsp; &emsp; &emsp; &emsp; &emsp;&emsp; &emsp;               | SST-2  &emsp; &emsp; &emsp; &emsp; &emsp;     | SUBJ    &emsp; &emsp; &emsp; &emsp; &emsp;         |\n| ----------------------- | --------- | -------------- |\n| ZeroGen                 | 87.27     | 80.45          |\n| ProGen                  | 88.42     | (Not provided) |\n| SunGen                  | 89.45     | 83.25          |\n| FreeAL (DistilBERT) | **91.82** | **92.15**      |\n| FreeAL (RoBERTa)   | **94.66** | **94.45**      |\n\nTherefore, we believe our work studies a rather different setup that can be much more practical and efficient with significantly better performance. Despite these advantages of FreeAL, we'd like to emphasize that our framework is designed to be a valuable plug-in helper that is flexible and compatible with different off-the-shelf data synthetic, distillation, and data selection algorithms. We'll add these new discussions and the missing reference to the revised version. \n\n\n\n\n\n**Q2: One of my main concerns is that FreeAL is incomparable with active learning methods, which only use a small portion of datasets for annotation efficiency and are blind on other data in unlabeled pool. Thus, I suggest to delete the comparison with active learning (Figure 1 and Table 5) or delay to Appendix, where discuss the label cost.**\n\nA2: Thanks for the remark. We kindly note that there might be some misunderstanding on the pipelines of traditional active learning that ```they actually also receives all training sample in every selection and training loop```. Specifically, a classical AL algorithm has the following steps in each loop:\n\n(1) Compute the metrics on **all (unlabeled) training samples**, and proactively select a subset it wants to learn from by ranking the metrics.\n\n(2) Label the subset by the external source --- human experts or oracle (cannot label the whole training set).\n\n(3) Training the SLMs with given data and labels, then loop to step (1).\n\nIn summary, traditional AL is also allowed to access the whole training set like ours and the comparisons in Table 5 and Figure 1 are indeed fair. Notably, thanks to the great power and low cost of LLMs, our FreeAL can provide much more weak labels for SLMs than traditional AL. We believe it's our core superiority that can break the constraint of involving human labelers who can only annotate few samples. \n\n\n\n**Q3: I further notice that FreeAL first annotates the whole dataset (D_{train}) and then reduce to the to-be-labeled set D_{noisy} iteratively. It is important to display the size of the to-be-labeled set to understand the cost of FreeAL at each round, since the API cost of ChatGPT is not negligible, especially given a large scale of training set.**\n\nA3: Thanks for the question. The annotation cost of FreeAL is much lower than that of humans and can be further reduced. \n\nWe first provide the to-be-labeled count for the LLM at the round 1 (i.e., the size of whole dataset $D_{train}$) and the round 3 (i.e., the size of $D_{noisy}$ filtered by GMM) in the table below. As also discussed in Appendix A.2, FreeAL can serve as a low-cost labeler which is much cheaper (requires about 1% cost) than human experts. When entrusted with a training set that is too large to label the entire dataset, the annotation cost can be further reduced by a simple multi-round solution where LLMs annotate a subset of samples in each loop. Please refer to `#Reviewer 3J9a.Q4` for more details.\n\n| Round  &emsp; &emsp; &emsp; &emsp;&emsp;| SST-2 &emsp; &emsp; &emsp; &emsp;&emsp; | MR  &emsp; &emsp; &emsp; &emsp;&emsp; |\n| ------- | ----- | ---- |\n| Round 1 | 6920  | 8662 |\n| Round 3 | 5537  | 6931 |\n\n\n\n\n\n**Q4：It seems that the LLM’s boosted performance in FreeAL mainly attributes to better labels. However, as [2] suggests, even random labeled demonstration can yield similar LLM inference performance. I am not convinced that the boosted performance is reliable: it can be mainly an advantage of with text demo over without text demo (from round 0 to round 1) and authentic text (from unlabeled data) over synthetic text (from round 1 to round 3).**\n\nA4: Thanks for the remark. We kindly disagree with the reviewer that we can use random labels to replace the true labels for any tasks. Consider a simple thought experiment:\n\n1. Problem 1: We map the words 'A' to label 'Y1', 'B' to label 'Y2', 'C' to label 'Y3' ...\n2. Problem 2: We map the words 'A' to label 'Y26', 'B' to label 'Y25' ...\n\nCan we provide random labels to solve any problem? Clearly, if one model achieves very high performance on Problem 1, its performance on Problem 2 can be extremely low. Therefore, random pairing texts and labels can potentially destroy the performance. We'd also like to note that the original paper [2] has concluded that the good performance of ICL possibly attributes to several aspects, including label space/distribution, language format, and also correctly-paired input-labels.\n\nFrom the empirical standpoint, we'd like to clarify that our better labels do indeed improve the ICL performance. To display it, we conduct experiments on the label enhancement process of LLM where the demonstrations are equipped with random labels. Specifically, we have stored and adopted the same self-generated or SLM-filtered demonstration pool $D_{demo}$ to ensure the comparisons are fair. As shown in the Table below, FreeAL enhances the performance with a much larger margin than FreeAL with random label, which demonstrates that the boosted performance is reliable and can be attributed to better labels. Importantly, randomly-labeled demonstrations can **even worsen the performance** compared to zero-shot ICL from round 0 to 1 on MR dataset and from round 1 to 3 on SUBJ dataset.\n\n| Dataset | Round  | Demonstrations           &emsp; &emsp; &emsp; &emsp; &emsp;&emsp; &emsp; &emsp;                        | FreeAL &emsp; &emsp; &emsp; &emsp;          | FreeAL with random label |\n| ------- | ----- | ------------------------------------------------------ | ------------------ | ------------------------ |\n| SST-2   | 0->1  | without $D_{demo}$ ->  with self-generated $D_{demo}$  | 92.47 -> **93.73** | 92.47 -> 93.22           |\n| SST-2   | 1->3  | with self-generated $D_{demo}$ -> authentic $D_{demo}$ | 93.73 -> **95.91** | 93.73 -> 94.89           |\n| MR      | 0->1  | without $D_{demo}$ -> with self-generated $D_{demo}$   | 90.05 -> **90.85** | 90.05 -> 89.30```(-0.75)```     |\n| MR      | 1->3  | with self-generated $D_{demo}$ -> authentic $D_{demo}$ | 90.85 -> **93.10** | 90.85 -> 91.95           |\n| SUBJ    | 0->1  | without $D_{demo}$ ->  with self-generated $D_{demo}$  | 55.65 -> **83.85**     | 55.65 -> 82.70           |\n| SUBJ    | 1->3  | with self-generated $D_{demo}$ -> authentic $D_{demo}$ | 83.85 -> **90.27**     | 83.85 -> 83.65```(-0.20)```       |\n\nLastly, it should also be noted that the improvement of the LLMs is not our mere goal. In fact, through our extensive experiments, we found that the SLMs reach better performance on 6 of 8 benchmarks. This is attributed to our distillation step which trains the SLMs with better labels. Our work also proves that SLMs remain a viable choice for practical deployment due to their impressive fine-tuned performance and low computational requirements as discussed in Appendix A.1. Therefore, we believe better labels are still quite important in the collaborative distillation process of FreeAL. We'll add more results and discussions in our revised version.\n\n\n\n**Q5: A suggestion, not necessarily weakness: (1) have you experience with LLM/SLM self-improvement, without interaction?  I observe that round 2’s and round 3’s performance are almost the same, then can SLM still benefit from self-generated pseudo labels?**\n\nA5: That's a good suggestion, thank you! Below, we'd like to clarify the importance of interaction between the LLM and the SLM in our framework. The main reason is that deep networks are known to have the effect of confirmation bias, i.e., the mistakes in the pseudo-labels can accumulate and are nearly impossible to self-correct by the model itself. To alleviate this, we design a collaborative training framework with interaction to detect different labeling patterns by both LLMs (by activating their general knowledge) and SLMs (by self-training and distillation). Therefore, the interaction between LLMs and SLMs can bring new opportunities to converge to a consensus result between them.\n\nTo reveal it more intuitively, we additionally provide the inductive performance without interaction in the table below. For the LLM, it directly adopts its own predictions on the training dataset at round 1 as the demonstration pool directly for testing. While the SLM employs its own predicted labels as supervision at round 2 directly for testing. We have two conclusions:\n\n1. The SLM itself is hard to distill from its own predictions, e.g., improves 0.2 on the MR dataset compared with FreeAL’s improvement of 0.85. Even worse, SLM shows degraded performance when distilled from itself on the SST-2 dataset. It reveals that the SLM can be entangled in confirmation bias and fail to correct its own mistakes. \n2. LLM can self-improve itself, but still underperforms our collaborative mechanism. e.g. FreeAL improves the self-distillation variant by 0.54 and 0.95 on the SST-2 and MR datasets. But, LLM has an inescapable upper bound on the performance, according to our empirical findings where SLM outperforms LLM on 6 out of a total of 8 datasets.\n\n| Model         | Ablation        | Round         | SST-2                | MR                   |\n| ------------- | --------------- | ------------- | -------------------- | -------------------- |\n| GPT-3.5-Turbo | FreeAL      | round 1->2->3 | 93.73 -> **95.91**   | 90.85 -> **93.10**   |\n| GPT-3.5-Turbo | FreeAL w/o interaction | round 1->3    | 93.73 -> 95.37       | 90.85 -> 92.15       |\n| RoBERTa       | FreeAL      | round 2->3->4 | 94.29 -> **94.66**   | 89.35 -> **90.20**   |\n| RoBERTa       | FreeAL w/o interaction | round 2->4    | 94.29 -> 94.27```(-0.02)``` | 89.35 -> 89.55 |\n\n\n\n**Q6: (2) LLM can also self-determined “uncertain” samples, from token probability. It will be interesting to discuss more on this topic.**\n\nA6: We originally aim to design a general framework which we have proved to be effective under the setting where the LLM is employed as black-box service model and token probability of generation is not provided, like ChatGPT and GPT-4. We thank the reviewer for providing such an interesting suggestion and we'll explore it in our future work.\n\n\n\n**Q7: In Table 1 and Table 2, the RoBERTa performances on CoNLL03 are exactly the same. Is this a mistake or coincidence?**\n\nA7: Thanks for pointing out it! The performances of RoBERTa on CoNLL03 in Table 2 should be 71.82 and 76.12 respectively at round 2 and round 4. We'll revise it in our revision as well as double-check our full manuscript. \n\n\n\n**Q8: I do not quite catch up with when eq.5 is used and when eq.7 is used. Can you explain a little more?**\n\nA8: Sorry for the confusion!  We originally aim to enhance the representation learning of SLMs so we adopt both consistency regularization and the mix-up augmentation in robust self-training as in Eq.(7). Actually, we have to note that the mix-up strategy is not our contribution, hence we omit it in Eq.(5) and delay it to the Appendix.\n\n\n\n**Q9: Typos: Line 20 in Algorithm 1 should be D_{demo} and D_{noisy}. Remove “-“ in line 391 and line 395.**\n\nA9: Our bad! Thanks for pointing them out. Line 20 in Algorithm 1 should be $D_{demo}$ and $D_{noisy}$. We will double-check and fix them in our revision!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper provides a useful method for LLM-based unsupervised learning, which is practical in real-world scenarios. The authors also discuss the potential cost, compared to human annotations.\n2. The idea to create an initial demonstration set with self-generation is interesting and effective.\n3. The composition of several techniques (e.g. DivideMix and small loss selection) are well motivated and clearly ablated.", "reasons_to_reject": "1. One of my main concerns is that FreeAL is incomparable with active learning methods, which only use a small portion of datasets for annotation efficiency and are blind on other data in unlabeled pool. Thus, I suggest to delete the comparison with active learning (Figure 1 and Table 5) or delay to Appendix, where discuss the label cost.\n2. I further notice that FreeAL first annotates the whole dataset (D_{train}) and then reduce to the to-be-labeled set D_{noisy} iteratively. It is important to display the size of the to-be-labeled set to understand the cost of FreeAL at each round, since the API cost of ChatGPT is not negligible, especially given a large scale of training set.\n3. It seems that the LLM’s boosted performance in FreeAL mainly attributes to better labels. However, as [2] suggests, even random labeled demonstration can yield similar LLM inference performance. I am not convinced that the boosted performance is reliable: it can be mainly an advantage of with text demo over without text demo (from round 0 to round 1) and authentic text (from unlabeled data) over synthetic text (from round 1 to round 3).\n4. A suggestion, not necessarily weakness: have you experience with LLM/SLM self-improvement, without interaction? (1) I observe that round 2’s and round 3’s performance are almost the same, then can SLM still benefit from self-generated pseudo labels? (2) LLM can also self-determined “uncertain” samples, from token probability. It will be interesting to discuss more on this topic.\n\n[2] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a collaborative learning framework, FreeAL, where an LLM serves as an active annotator to provide coarse-grained annotations, and a downstream SLM is trained over the pseudo labels and filter out high-quality samples to feedback LLM for in-context learning. It is notable that FreeAL runs in the unsupervised setting (only use unlabeled data), and can enhance both LLM’s and SLM’s performance.\nSpecifically, it first generates an initial demonstration set with LLM, and iterate in loop as (1) LLM annotates the whole unlabeled dataset, then (2) SLM is then trained with DivideMix technique [1], and used to filter out clean demonstration data with class-wise selection.\n\n[1] Junnan Li, Richard Socher, and Steven C. H. Hoi. 2020. Dividemix: Learning with noisy labels as semi-supervised learning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I believe that it will be more comprehensive to include comparisons with zero-shot data generation methods (e.g. ZeroGen [3], ProGen [4] or SunGen [5]) in experiments. These methods cover a weaker setting (zero-shot), and can serve as better baselines to illustrate the novelty of FreeAL.\n\n[3] Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. ZeroGen: Efficient zero-shot learning via dataset generation.\n\n[4] Jiacheng Ye, Jiahui Gao, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2022. ProGen: Progressive zero-shot dataset generation via in-context feedback.\n\n[5] Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, Lingpeng Kong. 2022. Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learning.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 20 in Algorithm 1 should be D_{demo} and D_{noisy}?\n2. Remove “-“ in line 391 and line 395.\n3. In Table 1 and Table 2, the RoBERTa performances on CoNLL03 are exactly the same. Is this a mistake or coincidence?\n4. I do not quite catch up with when eq.5 is used and when eq.7 is used. Can you explain a little more?"}}]

Author rebuttal:

Meta-review: This paper presents an active learning based framework FreeAL to distill knowledge from LLM to SLM where LLM serves as an active annotator and SLM as the student model. The proposed framework eliminates the need for human-labeled supervision. There is a unanimous consensus among reviewers that this is a sound approach for knowledge distillation without human annotations and the experiment result is very solid and positive. The rebuttal has effectively addressed reviewers' concerns by adding more analysis and baseline comparisons. We encourage the authors to take the feedback to revise the final version.
Decision: Accept-Main