Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision
Authors: Jiaxin Zhang
Zhuohang Li
Kamalika Das
Sricharan Kumar
Venue: NeurIPS 2023 poster (2023)
Keywords: multi-fidelity optimization
cost-effective learning
exploration-exploitation query
limited annotation budgets
TL;DR: Interactive Multi-Fidelity Learning for cost-effective LLM fine-tuning

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in various tasks. However, their suitability for domain-specific tasks, is limited due to their immense scale at deployment, susceptibility to misinformation, and more importantly, high data annotation costs. We propose a novel Interactive Multi-Fidelity Learning (IMFL) framework for cost-effective development of small domain-specific LMs under limited annotation budgets. Our approach formulates the domain-specific fine-tuning process as a multi-fidelity learning problem, focusing on identifying the optimal acquisition strategy that balances between low-fidelity automatic LLM annotations and high-fidelity human annotations to maximize model performance. We further propose an exploration-exploitation query strategy that enhances annotation diversity and informativeness, incorporating two innovative designs: 1) prompt retrieval that selects in-context examples from human-annotated samples to improve LLM annotation, and 2) variable batch size that controls the order for choosing each fidelity to facilitate knowledge distillation, ultimately enhancing annotation quality. Extensive experiments on financial and medical tasks demonstrate that IMFL achieves superior performance compared with single fidelity annotations. Given a limited budget of human annotation, IMFL significantly outperforms the $\bf 3\times$ human annotation baselines in all four tasks and achieves very close performance as $\bf 5\times$ human annotation on two of the tasks. These promising results suggest that the high human annotation costs in domain-specific tasks can be significantly reduced by employing IMFL, which utilizes fewer human annotations, supplemented with cheaper and faster LLM (e.g., GPT-3.5) annotations to achieve comparable performance.

Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision
Jiaxin Zhang', Zhuohang Li², Kamalika Das',
Sricharan Kumar
IItuit Al Research {jiaxin_zharge kamalika_dar zhuohang li @vanderbilt..c.
Vaanderbilt University sricharan, (eccharannkumarriinttut com
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities in var- ious tasks. However, their suitability for domain-speciff specific tasks, limited due to their immense scale deployment, susceptibility to misinformation, and more im- portantly, high data annotation costs. We propose a novel Interactive Multi-Fidelity Learning (IMFL) framework for the cost-effective development of small domain- specific LMs under limited annotation budgets. Our approach formulates the fine-tuning process as multi-fidelity learning problem, focusing on identifying the optimal acquisition strategy that balances low-fidelity auto- matic LLM annotations and high-fidelity human annotations to maximize model performance. We further propose an exploration-exploitaiio query strategy that enhances annotation diversity and informativeness, incorporating two innovative designs: 1) prompt retrieval that selects in-context examples from human-ane aaman-annotated samples improve LLM annotation, and variable batch size that controls the order for choosing each fidelity to facilitate knowledge distillation, ultimately en- hancing annotation quality. Extensive experiments on financial and medical tasks demonstrate that IMFL achieves superior performance compared with single fidelity annotations. Given a limited budget of human annotation, IMFL significantly out- performs the x human annotation baselines in all four tasks and achieves very close performance as 5 human annotation on two of the tasks. These promising results suggest that the high human annotation costs in domain-specn significantly reduced by employing IMFL, which utilizes fewer human annotations, supplemented with cheaper and faster LLM (e.g., GPT-3.5) annotations to achieve comparable performance.
tasks can be
Introduction
Large language models (LLMs) like GPT-3/ChaGGP//GTT- 47 5 have lately attracted great interest from both academia and industry due to their impressive in-context learning (ICL) abilities. However, the current state-of-the-art LLMs have since quickly grown from hundreds of billions M to even a trillion (20] parameters. Models of this scale require specialized hardware, massive-scale training data, and extensive computational power, which are inaccessible for most product research teams. In addition, the generalizability of LLMs predominantly decided by the scope of the underlying pre-training data. In fact, LLMs do not perform well out of the box in many real-world domains where specialized knowledge beyond the standard fields of pre-training is required (i.e., domain shifts), such healthcare 26 and finance 46
As an alternative to general-purpon LLMs, practitioners oftentimes find small domain-specific language models (LMs) be more favorable they require less training data and faster to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Interactive fine- fine-tuning
IMFL vs Humen Annotator
High-fidelity
Limited annotation budget
humer--rncttted
80-
Uncertainty
data
73
Score
Human annotator
Target LM
Prompt
retrieval
LLM annotator
(e.g., GPT-3.5)
co-context learning
1k Humann
2x Honuse
Hamee
3x, Strmmma
Query
Asssign
Low-fidelity LLM-annotated data
Unannctated data pool
Acquired
Compare IMFL (1X Human 4X GPT3.5) with human annotators
samples
(a) Overview framework
Figure 1: (a) Proposed Interactive Multi-Fidelity Learning Framework (IMFL). IMFL aims at solving the best acquisition strategy that balances between low-fidelity automatic LLM annotations and high-fidelity human annotations to maximize model performance given limited annotation budgets. (b) IMFL significantly outperforms the 3 human annotation baselines in all four tasks and is very close to 5x upper bound in the Headline dataset (showed). This result indicates that the high human annotation cost in domain-specific tasks can be greatly reduced by employing IMFL, which utilizes fewer human annotations combined with cheaper GPT-3.5 annotations to achieve competitive performance.
compute, leading to faster development cycles and lower operating costs 15
A common
practice of developing such models is through the classic pre-training and then fine-tuning paradigm. Unfortunately, to achieve comparable performance as LLMs, tuning small LMs requires high-quality manual annotations on target domain data, which in many fields requires extensive human effort and expert knowledge, making supervised fine-tuning very expensive.
One promising approach to alleviate human annotation efforts is to leverage LLMs as knowledge bases for automatically annotat- ing new data A 45 Unfortunately, such an approach is susceptible to the misinfor- of LLMs through hal- 30, 52), which risks Efficiency
Table 1: A qualitative comparison of human annota- tion, LLM annotation, and IMFL.
Human
LLM
IMFL
Cust Saving Quality
Low
Very High
High
mation R 39
32
Very High
Low
High
lucination 16. 53
Low
Very High
High
generating unreliable falsified labels and will, in turn, demolish the model's 's utility for high-stakes applications like healthcare and finance, where the truth is of utmost importance. As a result, the key challenge at hand is how to effectively gather sufficient high-quality data given limited budgets human annotation, which is critical component in fine-tume finettnnig domain-specific LMs.
Performance
Very High
High/Very High
In this paper, present novel framework, named by IMFL, for achieving cost-effective development of domain-specifte LMs, as illustrated in Fig. D Our approach capitalizes on the insight that different data samples inherently exhibit different levels of hardness for learning [6] I. Therefore, it is dispensable to request human annotation for every sample. By discerning each sample's 's hardness level,
delegate the majority of the annotation tasks to automatic annotation tools such as LLMs while exclusively assigning a limited number of highly uncertain samples to human annotators, thereby reducing human effort significantly while still maintaining high annotation quality.
To improve cost efficiency, we formulate the domain-specific fine-tuning process as an interactive multi-fidelity learning problem. We deem LLMs and humans to be two sources of annotation with distinct fidelities, and aim to determine the optimal acquisition strategy that balances low-fidelity LLM-generated annotations and high-fidelity human annotations to maximize model performance under limited annotation budgets. We thus introduce an exploration -eattiionexlliitaiionn query strategy, wherein human annotations emphasize exploitation geared toward maximizing informativeness while LLM annotations concentrate on exploration to foster diversity and improve representatives.
To reduce the misinformation in L.LM-generated annotations and improve model usability and reliability the target domain, we incorporate two innovative designs. First, we utilize prompt retrieval to select in-context learning examples for each queried sample, thereby improving the

---

accuracy of LLM- -ggeerrtte annotations. Second, we implement variable batch sizes throughout the interactive annotation process, which manage the order in which each fidelity chosen; this facilitates knowledge distillation and ultimately enhances annotation quality while stabilizing the L.M annotations.
We evaluate our approach on four language understanding tasks across two specialized application domains, i.e., finance and medicine. Our results highlight that LMs tuned through the proposed IMFL framework with GPT-3.5 as an auto-annotator significantly outperform LMs tuned with 3x human annotations, and are even on par with LMs tuned with 5x human annotations in some cases. In contrast to tonglefiideltyy annotations such as only human or only LLMs, IMFL effectively addresses limitations related to cost saving, annotation quality, and efficiency (see Table D.. Furthermore, IMFL not only surpasses the performance of LLM annotators but also achieves highly competitive performance compared to human annotators, albeit at a substantially lower cost and effort.
2 Interactive Multi-fidelity Learning
We propose IMFL that builds on two key insights: (1) leveraging a substantial amount of low- fiwfiielitty annotations generated by LLMs to compensate for the insufficiency of high-fidelity human annotations during fine-tuning, and (2) utilizing high-fidelity human annotations as supervision signals to distill knowledge from LLMs while simultaneously enhancing their output annotation quality through in- context learning. Essentially, our approach, IMFL, can be regarded as synergy between fine-tuning and knowledge distillation under sparse human supervision.
2.1 Problem Formulation
Given a total annotation budget B and a computational cost C (e.g., costs for fine fine-tuning, inference, and query), we aim fine-tune small LM f(x: 8*) X' + 3 with pre-trained parameters 8" downstream task by annotating samples from an unannotated data pool = {0:VE= to constitute the annotated sample set A (lA) < B and initially A = Ø) such that its performance maximized. Note that in our multi-fidelity setting, the annotated set contains a human-annotated subset Air and an -nn.tttte subset Ag. A An U.G. Similarly, the total annotation budget is composed of human annotation budget Bи and LLM annotation budget Bg (Bμ is typically much smaller than Bc), i.c., B BH + Bg.
To solve for the best annotation strategy to maximize annotation and computation efficiency, we pose the annotation acquisition process multi fidelity learning problem with interactions allowed for R rounds. In the r-th round (1 S r S R), we query a set of instances Q" and annotate acquired samples A" from the unannotated pool to add annotation, i..., 41 u A" and fine-tune the target model on A" update g(r), The goal is minimize the empirical risk R(f) of the final LM f(: ((R)) on the downstream task, subject to preset annotation budget and computational cost constraints.
multi-fidelity
2.2 Multi-fidelity Learning Framework
Initialization. We initialize the multi multi-fidelity learning loop by randomly selecting a small set of samples 49, from the unannotated set 2/ to be annotated by human annotators. The pre-trained LM with parameters H* is then tuned on the initial annotated dataset:
((0)_ arg min 8-
Σ C(f(x))) i= 1,.,..,, (=..)))AA)
(1)
A
where L is the loss function, e.g., cross-entropy for elassification, and n., is the annotation size. This enables the uncertainty score the target LM to be initially updated on domain-specific data, which
helps to mitigate the cold-start issues 31
49
50
Interactive fine-tuning. After model initialization, we begin query samples from the unannotated pool 2 = 11' \, Ad, for either human or LLM annotation. Existing methods [54] often consider the entire unannotated pool during sampling. These approaches scale poorly to large unlabeled datasets as acquiring informative samples usually involves making inferences or executing clustering which can be time-consuming if these operations were be computed over all data samples. Thus, for any
3

---

interaction round r, we propose to randomly sub-sample from lt to obtain smaller candidate set 2 where the acquisition strategy can be efficiently computed.
In r-th round of interactive fine-tuning, we first perform the exploration-exploiaae
query (EEQ)
strategy S (described in detail in Section 2.3 to determine the human annotation set Ai and LLM annotation set Ag from the sub-sampled unannotated pool Ut Then the interactive multi-fidelity learning can be solved by minimizing the following total loss objective:
Statal
Σ
C (s(a;;00))) + LAG
1
E (
(2)
AH
Unlike the existing approaches that use simultaneous annotation with equal batch sizes for each round, we emphasize the importance of annotation order (human first and then LLM) and variable batch sizes for each query step (verified in Section 4.2) and identify the following two key designs that improve query efficiency and annotation effectiveness:
Design In-context learning with similarity-based prompt retrieval. According to the annotation budget B4 and BG we acquire 2/r and 2g instances for human and L.M annotators respectively. We first annotate acquired samples 2 by humans, obtain A'i and update the numan-annotated set Au Arr U U When using L.L to automatically generate annotations for new data, we then retrieve a few examples from the current numan-annotatee set Ah as in-context examples for improving the predicted annotation quality, see Fig. 2 Leveraging recent advances in prompt retrieval [25]. we compute embeddings from all annotated samples using Sentence-BERT [3 and find the most similar examples for each queried instance measured by cosine similarity. This design improves in-context learning by better utilizing human supervision which empirically helps to further improve the accuracy and robustness of LLM annotations (verified in Section More implementation details are provided in Appendix
Variable batch-size query. We propose a variable batch-size query strategy that puts more human budgets towards the initial steps of the learning process annotate the most uncertain instances and gradually decrease the batch sizes until the total budget is reached, as illustrated in Fig. 2 Naturally, another benefit of this design is that by acquiring more in the early stage, we can have access to a larger pool of high-fidelity samples for conducting aimilarity-based prompt retrieval, which further improves the in-context learning performance and stabilizes the LLM annotations. Inspired by infinite geometric series, we design a budget decay scheme and thus the human annotation budget for the r-th round to be B B1r and iterate until the total budget is reached, i.e.
Design 2
examples
BH
BH
B3
BH 2
BH
BH - Bи
(3)
2
23
2"
Note that the residual budget after Rt rounds will be jointly applied to the last round.
Leveraging the benefits of novel designs, we efficiently acquired larger high-quality data Ar annotated by LLMs (e.g., GPT-3.5). The next step to update the annotated sample set in the r-th round A Ar Ar and unannotated data pool U = 1 \ A4" Then we fine-tune the target model f using the annotated sample set (x,, y:) AA and update the model parameters g(r).
Termination. The multi fidtiiy learning process is stopped if either the two stopping criteria is satisfied: (1) Annotation budget B: if the annotation budget after It rounds is greater than the total budget limit, i.e., BH + BG 2 B. we terminate the interactive process. (2) Computational cost C: Compared with inference and query calculation cost, the computation cost of each fine-tuning round Cfe is typically much more expensive and we thus stop the fine-tuning process R x Cf: 2 C. Finally, we return the fine-tuned target LM (a: g(r) and annotated sample set A. Algorithm illustrates the step-by-step workflow of our IMFL framework.
2.3 Exploration- Query Strategy
Based on the multi-fidelity learning framework, we introduce an innovative query strategy. This approach harnesses human annotation for exploitation by maximizing informativeness through

---

Human annotation pool
Human amnolation LLM annotation
Algorithm IMFL framework
Arrotaaion balch size
1: Require: unannotated data pool U, target LM model f.
query strategy S, annotation budget
Prompt retr eval
Prompt revieval
2: Initialization: A = 2. 0 (( A4
Prompt retrieval
for rounds r = 1, 1.... do
3: 2 + Extract from 24 by random sub-sampling
4: [2t, 2al + Acquire [8 3% samples by query
1 round
2 round
3 round
function S on model f. data 2g"
5: Ai + Annotate acquired samples 2i by human
Figure 2:
Illustration of interactive fine- tuning. Human annotations first executed and the resulting annotated data are iteratively merged into the human annotation tool, which provides rich examples for prompt retrieval when calling LLM for annotation. The batch size for human annotation varies and gradu- ally decreases as the round progresses.
6: Ah = Ar U UAi
7:
Execute prompt retrieval from Ah
8: AG <- Annotate acquired samples Q by LLMs
A" AAi U.G
10:
U'AA"
11:
f(2,;; ((r)) Fine-tune f(z10(0)) on A"
12:
return f(x; B()),,
uncertainty sampling, and LLM annotation for exploration by enhancing representativeness through diversity sampling. The core idea is two-stage selection: executing 1) diversity sampling, e.g.. selecting cluster centers to reduce intra-iteration redundancy, and 2) uncertainty sampling, e.g.. selecting instances with the least confidence, to avoid inter-iteration redundancy.
Fig. presents the key components and steps the EEQ strategy. Specifically, we apply k-means cluster algorithm to embeddings of the sub-sampled unannotated data U"". Based on the annotation budget, we set = B /2" + Bg/ R. as the clustering parameters and identify the cluster centers (or samples closest to the cluster center) as samples, thus enforcing diversity exploration. We then calculate the uncertainty score for all selected samples and rank them from high to low. The top BH/ 22 uncertain samples assigned to the human annotator following the least confidence strategy:
F arg max 1 p(f(x;;0()) x;; ((:
(4)
which has shown to be simple and effective in variety of settings, resulting in enforcing uncertainty As discussed Section 2.2 we then update the numan-annotated pool A which enables us to retrieve a few examples as in in-context examples for the LLM annotator which can annotate Be/ samples with better quality and stability. More detailed discussions about the query strategy presented in Appendix
exploitation 28 42
Embedddings
Exploration
Exploitation
l-means elustering
Uncertainty
Update human annotated
Ranking
ICI with prompt retrieval
Unarnottted data pool
Select cluster
Low
High
Human annotator
onnotator
Figure 3: Illustration of exploration-exxloiaaiin query strategy with core components and steps.
3 Experiments
3.1 Datasets
We empirically validate the effectiveness of the proposed interactive multi-fidelity learning framework on four diverse datasets, spanning two important real-world application domains, namely, finance and medicine. A summary of the four datasets is provided in Table For the FPB, Headline, and MedQA datasets, we use the publicly available test data for evaluation. As for the PubMedQA dataset, we follow prior work [8] and use the dev./valid data for evaluation. We evaluate the methods by (average) F-1 score for financial datasets following the same setting used by BloombergGPP [46] Interested readers can find more
and accuracy for medical datasets, as used in prior work details about the datasets in Appendix D.I
5

---

Table 2: Summary of the four Cooainn-specifee datasets used in our experiments.
Domain
Name
Task
Size (train/test)
Metric
Financial
FPB
P
Sentiment Analysis
3876/969
F-1 score
Financial
Headline
40
News Classification
9130/2282
Average F-1 score
Medical
PubMedQA
18
Biomedical QA
500/500
Accuracy
Medical
MedQA
7
Medical knowledge QA
1 11450/1273
Accuracy
3.2 Experiments Setup
Fine-tuning. We adopt Dolly 2.0, the first open-source, instruction-follovinn LLM,, as the target LM for fine-tuning. It is based on the FleutherAT Pythia 3 model family, which a suite of decoder-only auto-regressive language models ranging from 70M 12B parameters. Limited by our computational budget, we choose to use dolly-v2-33 as the pre-trained LM for our main results. We also provide additional results using larger LMs (e.g., dolly-v2-77 and dolly-v2-12b) in Appendix C.2 to show the impact of pre-trained LM size on the final performance. For efficiency, we leverage low-rank adaption techniques (LoRA) memory and time cost. We execute all experiments on a GPU node with 8 NVIDIA V100 32G cores. More experiment setup details, e.g., hyperparameters, can be found in Appendix D.2.
B to optimize the fine-tuning process for reducing
Query and annotation. In the query step, we remove all labels in the training data to create pool of unannotated data. These original ground truth labels are treated as the high-fidelity annotations provided by Human Annotator and only accessed at the cost of budget consumption. For the low- fidelity annotation, we employ GPT-3. 5-turbo as the LLM Annotator to automatically generate annotation for unannotated data. We note that, in reality, even collecting large set of unannotated samples can oftentimes be non-trivial. As such, in our experiments, we limited our unannotated data pool only contain 3000 data samples (randomly sampled from the original training dataset), from which we perform our query strategy. Each experiment is repeated three times and the mean is reported as the final result to reduce noise.
Annotation and computational budget. Unless mentioned otherwise, we assume total annotation budget of 1000 for all datasets (see more discussions about the budget setting in Appendix C.4. As human annotation is far more expensive than using LLM (i.e., GPT-3.5) generate annotation, set the human annotation budget of IMFL to be 200 samples (20%) and the GPT-3.5 annotation budget provide a discussion about the trade-off between
to be 800 samples (80%). In Appendix C.5 annotation accuracy and cost expenses. Regarding the fine-tuning cost, we set the total number of interaction rounds for fine-tuning be R = 5 to reflect the computational budgets. It is worth noting that the performance can be further improved if more rounds (i.e., a higher budget) are allowed.
3.3 Main results
In this section, we compare IMFL with single fidelity annotations to validate the effectiveness of the proposed multi-fidelity paradigm. Fig. g compares IMFL with using only human annotations, where 1x Human, 3x Human, and 5x Human, represents the results obtained by fine-tuning on 200, 600, and 1000 human annotations, respectively. A detailed version of the main results are shown in Appendix C.T. Note that 5x Human (1000 human annotations) can be seen as the performance upper bound of IMFL (200 Human + 800 GPT-3.5) if all budget is human annotation. From the results, we can clearly see that IMFL significantly outperforms the 3x human annotation basclines in all four tasks. Particularly, IMFL achieves very close performance as 5x human annotation both Headline and PubMedQA datasets with only marginal differences (0.83% and 1.32% absolute loss respectively). This result indicates that the high human annotation cost in can be greatly reduced by employing IMFL, which utilizes fewer human annotations combined with cheaper and faster GPT-3.5 annotations to achieve similar performance.
tasks
Fig. 5 compares IMFL with using only GPT-3.5 annotations with the same total annotation budget (varied from 260 to 1000 samples). We have the following observations. First, our IMFL outperforms the GPT-3.5 annotation by a large margin (in terms of absolute gain) on PFB (+7.35%). Headline (+8.3%), PubMedQA (+6.89%) and MedQA (+19.95%) given the same 1000 annotation budget.

---

FPB [Financiall
Headline (financial)
PubMedOA (Medical)
MedOA (Medical)
80
23-
70
119core
70-
60
69
50
60
40-
1x Homan
Humnnn
Human
IMPI
Hommnr
Himan
Htman
3x Humman
3x Human
Human
MNF
Homan
Htmaa
Homan
BiF
Figure 4: Comparisons between our multi lilliffiyliiyy learning (200 human annotations + 800 GPT-3.5 annotations) and various sizes, i.e., 200 (1 x), 600 (3x), and 1000 (5x) of human annotations.
Second, on three out of four datasets (FPB, PubMedQA, and MedQA), models tuned using IMFL with a total annotation budget of 260 (100 Human + 160 GPT-3.5) are able to achieve better performance than using 1000 GPT-3.5 annotations. On the Headline dataset, using 1000 GPT-3.5 annotations performs slightly better than using IMFL with a total budget of 260, but still worse if the total budget increased to 470 ((100 50) Human + (160 + 160) GPT-3.5). This shows that while GPT-3.5 human-generated
demonstrates promising abilities to reproduce human generated labels (50, relying solely on low-fidelity GPT-3.5 labels is not ideal for fine-tuning LMs for domain-specine tasks. In addition, compared with using only GPT-3.5 annotations, IMFL shows more reliable results with smaller variance, which benefits from combination of human annotation and the similarity-based prompt retrieval strategy for improving the in-context learning capability of LLMs. These results verified that IMFL efficiently utilize sparse human supervision enhance GPT-3.5 annotations and consequently achieve better performance.
(Financial)
Headline |Financial)
PubModOA (Medical)
(Medical)
50
85
83
45
80
75
70
ACCuESCI
40
35
All GPT-3.5
CPT:3.5
All CPT.3.5
-+-
CFT-3-5
30
06
55
30
Annotation
otat
Annotation
Figure Comparisons between our multifidelity learning paradigm and single low-fidelity (all GPT-3.5) annotation four domain-specinf tasks given the same total 1000 annotation budget. Note that the samples for all GPT-3.5 are drawn based on the uncertainty score.
4 Analysis
4.1
Exploitation-Exploree Query vs Random Query Strategy
Table compares the proposed EEQ strategy with the random query strategy on multiple settings given a limited annotation budget. Under the multi-fide fille--fiilityy setting, our EEQ strategy outperforms the random query strategy by a large margin (5.91% absolution gain on average). Although fine-tuning with only human annotations is supposed to produce the best results as it has the highest annotation accuracy, we observe that using only human annotations with random query is generally worse than using human and GPT-3.5 annotations with EEQ query (on three out of four datasets), thereby validating the effectiveness of the proposed EEQ query strategy. We observe that one exception is the MedQA dataset, where using only human annotations with random query performs slightly better than the proposed IMFL This is because GPT-3.5 shows relatively low annotation accuracy on this dataset and consequently induces a negative impact on the fine-tuning performance by injecting noises into the annotated data. However, if using only GPT-3.5 annotations with the random query under the considered annotation budget, the performance would drop significantly.
7

---

Table A comparison of our EEQ query strategy and random query strategy.
Method
Budget
Query Strategy
Dataset
Multi/Single
Human
GPT-3.5
EEQ/Random
FPB
Headline
PubMedQA
MedQA
Human + GPT-3.5
200
800
EEQ
47.88
81.09
73.76
67.98
Human + GPT-3.5
200
800
Random
41.94
74.32
66.03
63.77
Only Human
1000
Random
43.81
75.46
68.87
70.17
Only GPT-3.5
1000
Random
38.56
71.04
65.89
47.13
4.2 Prompt Retrieval with Variable Batch Size
To evaluate the effectiveness of similarity-based prompt retrieval and variable batch size, in Table we consider several variants with random prompt retrieval and equal batch size baselines for comparison. We find that similarity-basee retrieval shows superior performance compared to the random prompt retrieval baselines (e.g.. 80.28 vs 73.77 on Headline and 72.05 vs 68. 10 on PubMedQA), and using variable batch size further boosts the effectiveness of retrieval by providing larger and more diverse of candidate examples, which is crucial in the limited budget setting.
Our interactive learning is conducted over the course of multiple rounds of interactive, with each round using single mini-batch of data for adaptive annotation and fine-tuning. Here we also compare the interactive multiple mini-batch update strategy with the full-batch strategy, where all annotations acquired in a single round and then used fine-tune the model. The full-batch strategy naturally reduces the computational cost for fine-tuning but loses the benefits of interactive improvements as shown the results. However, interestingly, we find that the similarity- based retrieval still provides good amount of improvements in the full-batch setting which can achieve competitive performance that is slightly better than using mini-batch updates with random retrieval. Therefore, we recommend that practitioners consider this alternative option, i.c.. full-batch incorporated with similarity-retrieve the computational budget of fine-tuning limited.
Table 4: Effects of prompt retrieval, variable batch size, and batch orders.
Method
Dataset
Budget
Batch
Batch size
Retrieval
FPB
Headline
PubMedQA
MedQA
1000
Mini-Batch
Variable
Similar
47.88
81.09
73.76
67.98
1000
5 Mini Batch
Equal
Similar
46.34
80.28
72.05
66.11
1000
Mini- Batch
Variable
Random
42.09
73.98
67.44
63.56
1000
Equal
Random
42.34
73.77
68.10
63.42
1000
Full-Batch
NA
Similar
43.72
75.48
68.90
63.79
1000
Full-Batch
NA
Random
39.80
72.11
65.94
57.23
4.3
Alternative Query Strategies
Besides the random query and the proposed EEQ query, we also explore several additional query strategies for interactive multi-fidelity learning. including (i) confidence-based strategies: predictive
entropy (Entropy) 35 least confidence (Least-c) 22 breaking ties (Breaking-)) 27 (ii) diversity-
based strategies: K-means [50], Diversity [ and (iii) hybrid strategy 119. combination of
confidence and diversity by a weighted sum. As shown in Table 5 our EEQ strategy outperforms all the other methods two representative tasks (Headline and MedQA). These alternative strategies simple to implement and perform better than the random baseline. Unfortunately. directly applying them our multi-fidelity paradigm does yield the most desirable performance. The one that achieves the closest performance is the hybrid strategy. However, ignores the effects of annotation orders and fidelity which are important ingredients for achieving high performance in our multi-fidelity setting.
4.4 Annotation Accuracy by Different GPT-based Annotators
The fine-tuning performance relies on the annotation accuracy of the L.LM annotator, since noisy annotations may hurt the final model performance in terms of accuracy and reliability. Here we focus

---

Table 5: comparison of various alternative query strategies on two representative tasks.
Dataset
Random
Entropy
Least-c
Breaking-t
K-means
Diversity
Hybrid
EEQ
Headline
74.32
76.42
77.55
77.34
76.59 66.44
77.61
79.23
81.09
MedQA 63.77
65.15
65.21
65.28
66.41
66.94
67.98
on evaluating the annotation accuracy of different variants of GPT (i.e., GPT-3 vs GPT-3.5) instead of fine-tuning accuracy through multiple experiments. We notice that GPT-3.5 annotation with zero shot performs much worse than few-shot and our retrieval methods in domain domain-specific tasks. This because although naive GPT-3.5 shows promising performance in doing zero/few-shot learning in-distribution scenarios, it lacks domain knowledge to make accurate predictions in the considered out-of-domain tasks. The proposed prompt retrieval leveraging human annotations from domain experts with in-context learning capabilities of LLMs substantially improves the performance of GPT-3.5 annotation.
Table 6: A comparison of annotation accuracy by GPT-3 and GPT-3.5 in zero/few-shot learning.
Dataset
GPT-3 Annotation
GPT-3.5 Annotation
retrieval
5-shot
O-shot
retrieval
5-shot
O-shot
Headline
75.59
72.51
70.25
79.40
76.15
73.31
MedQA
51.42
44.89
42.03
59.45
53.57
50.82
If the annotation budget very limited, GPT-3 is a cheaper alternative but underperforms GPT-3.5 even with prompt retrieval applied. In contrast, we have chance to utilize GPT-4 (more expensive and limited access) for annotation in our multi-fidelity paradigm. Please see additional GPT-4 annotation results in Appendix CK Recent work shows promising capabilities of GPT-4 on medical challenge problems. For example, GPT-4 achieves 81.38 (5-shot) and 78.87 (0-shot) for the MedQA task as reported by 32). Note that IMFL uses GPT-3.5 our LLM annotator but is casy extend to other LLM annotators, c.g., GPT-based models or open-source models such as LLaMA which depends the annotation budget. An exhaustive study of different LLM annotators is beyond the scope of this work.
4.5 Ablation Study of Human Annotation Ratio
Given total 1000 annotation budget, key question is how to assign the budget to human annotators or GPT-3.5 annotators. In our original setting, we use a 20/80 ratio since the human annotations are much more expensive (money cost, time cost, and training cost, specifically in domain- specific areas, e.g., finance and medicine) than GPT-3.5 annotations. We also expect to minimize the ratio of human annotations so we conduct an ablation study for 10/90 and 5/95 ratios and evaluate their effect on performance in our framework. Table / shows the performance comparisons human annotations, i.e., 0.5x and 0.25 human annotations. We can note that the performance drops obviously when less human effort is conducted. For the case of 0.5x human annotations, is lower than our original setting but still comparable 3x human annotations. However, the case of 0.25x x human annotations shows a sigmificant decrease because too few human annotations weaken the effect of in-context prompt retrieval and reduce the accuracy of initial uncertainty estimation. In short, a certain amount of human annotation is necessary for our framework even though we seek minimal human effort. We thus need to consider trade-off between accuracy and annotation budgets.
various ratios of
Table 7: Performance comparisons of various ratios of human annotations four datasets.
Method
Number of Annotations
Dataset
Human
GPT-3.5
FPB
Headline
PubMedQA
MedQA
IMFL
200 (1 x)
800
47.88 +: 0.98
81.09 + 0.58
73.76 + 0.95
67.98 +: 1.45
IMFL
100 (0.5x)
900
43.66 + 1.42
75.41 + 1.01
70.88 + 1.08
61.44 + 1.83
IMFL
50 (0.25x)
950
40.76 + 1.48
73.65 + 1.09
68.18 + 1.11
52.38 1.93

---

5 Related Work
Domain-specific LLMs. The significance of domain-specific training for encoder-only masked language models widely recognized. Two commonly adopted methods are to either train BERT models [9] from scratch using domain-specifie data or to continue pre- -re-training an existing model data. Following these approaches, several BERT- based models are built by
on new domain-spectie domain experts, e.g., BioBERT (21], ClinicalBERT [5]. etc. A recent trend is to train decoder-only models by utilizing domain-speeifie data, such as Med-Pal.M B8 BioGPT [26] and BloombergGPP
A6 These findings highlight the advantages of in-domain pre-training, especially if sufficient data available. However, an underlying challenge is how to train domain-specific LMSs when there is insufficient data for large-scale pre-training due to limited annotation budget. Our work addresses this underexplored problem by developing a cost-effective fine-tuning paradigm with limited budgets of human annotation.
Multi-fidelity learning. Our work is motivated by recent findings that suggest GPT models are capable of replicating or even outperforming human annotation as indicated by several studies However, these approaches suffer from unreliable annotations and lack confidence
14
4 56
when applied to fomain-speciffe tasks, especially in specialized high-stakes fields like finance and medicine. Our key idea originates from multi-fidelity optimization approaches [33] 23
that
optimize the objective function by utilizing varying approximations with different levels of precision and cost. Previous studies in the field of NL.P have explored "dual supervision" to train models by combining two types of labels A5 In contrast to this naive combination approach, we delve into
novel multi-fidelity framework that achieves cost-effective adaptation of domain-specific language models through fine-tuning and in-context learning.
Active learning. Active Learning (AL) is an extensively studied field concerned with improving the performance of language models with fewer labeled instances S54
31
Current works
within AL typically focus on two main scenarios: active fine-tuning
and active in-context learning DT0 41) The former involves iteratively updating model parameters but is not well-suited for directing training/fine-tunn
LLMs such as GPT-3.5 which would induce high computational costs. Conversely, the latter is efficient but the performance solely relies on the few-shot learning ability of LLMs, which is unreliable for comain--peciffi tasks that require expert knowledge beyond standard pre-training data. In contrast, the proposed IMFL, which fully utilizes few high-fidelity annotations from human annotators to guide the fine-tuning and knowledge distillation from LLMs under sparse human supervision. Our experiments demonstrate that our approach can significantly reduce human annotation efforts while achieving highly competitive performance given a limited budget for annotation and computational resources, which enables flexible and effective deployment in real-world applications.
can be regarded as synergizing the power of both
Discussion and Limitation
We compare IMFL to single fidelity annotations to evaluate the effectiveness of our proposed multi- fidelity paradigm. The extensive experimental results reveal that employing IMFL can significantly reduce the high cost of human annotation in domain-specific tasks. Furthermore, we demonstrate that IMFL efficiently uses sparse human supervision to improve GPT-3.5 annotations through prompt retrieval and in-context learning, ultimately leading to enhanced performance.
Despite the promising performance, we note there are certain limitations to our approach. First, the current IMFL framework assumes that the annotation budget is defined by the number of annota- tions, rather than reflecting the true cost which typically involves multiple complex factors (c.g., administrative cost, training cost of human annotators, time, etc.) in real-world scenarios. Second, IMFL's 's performance is limited by the size of the unannotated dataset and the diversity of examples presented in the dataset as IMFL only seeks to improve performance through annotating existing samples rather than creating new samples. Lastly, limited by budgets and the capacity of the LM to be fine-tuned, IMFL does not achieve state-of-the-arr performance some general NLP tasks, where directly adopting the latest LLMSs remains better choice. Nevertheless, we anticipate the performance of IMFL to continue to grow by incorporating stronger L.M annotators, such as GPT-4, to further improve annotation accuracy. We leave this our future work.
10

---

References
[1] Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10889,
2021.
[2] Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Love- nia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint SKKi:2332.00022. 2023.
[3] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint 031.0,330..0133,, 2023.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1991, 2020.
[5] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2333.127122 2023.
[6] Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv:1910.13422 2019.
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint urXiv:2204.033/1 2022.
[8] Debadutta Dash, Rahul Thapa, Juan M Banda, Akshay Swaminathan, Morgan Cheatham, Mehr Kashyap, Nikesh Kotecha, Jonathan H Chen, Saurabh Gombar, Lance Downing, et al. Evaluation of gpt-3.5 and gpt-4 for supporting real-world information needs in healthcare delivery. arXiv preprint arXiv:2304. 13714, 2023.
[9] Jacob Devlin, Ming-Was Ming-Wei&ee Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv: 11880.000000... 2018.
[10] Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. Active prompting with chain-of- thought for large language models. arXiv preprint 2..2v:2230..222444 2023.
[11] Daniel Grießhaber, Johannes Maucher, and Ngoc Thang Vu. Fine-tuning bert for low-resource natural language understanding via active learning. arXiv preprint arXiv:2012.02444 02222262, 2020.
[12] Cheng-Yu Yu Hsich, Chun-Liang Li, Chih-Kuar Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen- Yu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02300 2023.
[13] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint ArXiv:2106.09633 2021.
[14] Fan Huang, Haewoon Kwak, and Jisun An. Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech. arXiv preprint arXiv:2302.07736. 2023.
[15] Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:/904.055: 2019.
[16] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.

---

[17] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021.
[18] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William w Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv: 909.06146,, 2019.
[19] Seokhwan Kim, Yu Song, Kyungduk Kim, Jeong- Won Cha, and Gary Geunbae Lee. Mmr-based active machine learning for bio named entity recognition. In Proceedings of the human language technology conference of the NAACL, Companion Volume: Short Papers, pages 69-72, 2006.
[20] Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Moham- mad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. arXiv preprint arXiv:2205.05/99, 2022.
[21] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): ((44):1234-144))) 2020.
[22] David D Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pages 13-19. ACM New York, NY, USA, 1995.
[23] Shibo Li, Jeff M Phillips, Xin Yu, Robert Kirby, and Shandian Zhe. Batch multi-fidelity active learning with budget constraints. Advances in Neural Information Processing Systems, 35:995-1007, 2022.
[24] Shibo Li, Zheng Wang, Robert Kirby, and Shandian Zhe. Deep multi-fidelity active learning of high-dimensional outputs. In International Conference Artificial Intelligence and Statistics, pages 1694-1711 PMLR, 2022.
[25] Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLiO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learming Architectures, pages 100-114, 2022.
[26] Renqian Luo, Liai Sun, Yingee Xia, Tao Qin. Sheng Zhang, Hoifung Poon, and Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 2022.
[27] Tong Luo, Kurt Kramer, Dmitry B Goldgof, Lawrence O Hall, Scott Samson, Andrew Remsen, Thomas Hopkins, and David Cohn. Active learning to recognize multiple types of plankton. Journal of Machine Learning Research, 6(4), 2005.
[28] Seiji Maekawa, Dan Zhang, Hannah Kim, Sajjadur Rahman, and Estevam Hruschka. Low- resource interactive active labeling for fine-tuning language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2022, pages 3230-3242, 2022.
[29] Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius, and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65(4):782-796, 2014.
[30] Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.0m 08896. 2023.
[31] Katerina Margatina, Giorgos Vernikos, Loic Barrault, and Nikolaos Aletras. Active learning by acquiring contrastive examples. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 650 663, 2021.
[32] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capa- bilities of gpt-4 on medical challenge problems. arXiv preprint arxiv:2303. 2023.
[33] Benjamin Peherstorfer, Karen Willcox, and Max Gunzburger. Survey of multifidelity methods in uncertainty propagation, inference, and optimization. Siam Review, 50(3)550--511, 2018.
12

---

[34] Nils Reimers and Iryna Gurevych. Sentence- bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIP)), pages 3982-3992, 2019.
[35] Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. int. conf, on machine learning. 2001.
[36] Christopher Schröder, Andreas Niekler, and Martin Potthast. Revisiting incertainty-based query strategies for active learning with transformers. arXiv preprint arXiv:2107.05666 2021.
[37] Dan Shen, Jie Zhang, Jian Su, Guodong Zhou, and Chew Lim Tan. Multi-criteria-basee active learning for named entity recognition. In Proceedings of the 42nd annual meeting of the Association for Computational Linguistics (ACL-04), pages 589-596, 2004.
[38] Karan Singhal, Shekoofeh Azizi, Tao Tu, Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022.
[39] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972) 2023.
[40] Ankur Sinha and Tanmay Khandait. Impact of news on the commodity market: Dataset and results. In Advances in Information and Communication: Proceedings of the 2021 Future of Information and Communication Conference (FICC), Volume 2, pages 589-601. Springer, 2021.
[41] Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, et al. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975 2022.
[42] Alex Tamkin, Dat Pham Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. Active learning helps pretrained models learn the intended task. In Advances in Neural Information Processing Systems.
[43] Petter Törnberg. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. arXiv preprini arXiv:2304.06588 2023.
[44] Akim Tsvigun, Artem Shelmanov, Gleb Kuzmin, Leonid Sanochkin, Daniil Larionov, Gleb Gusev, Manvel Avetisian, and Leonid Zhukov. Towards computationally feasible deep active learning. arXiv preprint arXhv:2205.035: 2022.
[45] Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108. 7 33887, 2021.
[46] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: large language model for finance. arXiv preprint arXiv:2303.17566 2023.
[47] Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Bing Yin. and Xia Hu. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304. 127 2023.
[48] Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint arXiv:2309.0677 06794, 2023.
[49] Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang. Cold-start data selection for sho- language model fine-tuning: A prompt-based uncertainty propagation approach. arXiv preprint arXiv: 06995, 2022.
[50] Michelle Yuan, Hsuan-Ties Tien L Lin, and Jordan Boyd-Graber Cold-start active learning through self-supervised language modeling. arXiv preprint arX1v:2010.09535. 2020.
13

---

[51] Jiaxin Zhang. Modern monte carlo methods for efficient uncertainty quantification and prop- agation: survey. Wiley Interdisciplinary Reviews: Computational Statistics, 13(5):e1539 2021.
[52] Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Kumar Sricharan. SAC3; Reliable hallucination detection in black-box language models via semantic-awar cross-check consistency. EMNLP, 2023.
[53] Yue Zhang. Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.0/219 2023.
[54] Zhisong Zhang, Emma Strubell, and Eduard Hovy. A survey of active learning for natural language processing. arXiv preprint arXiv:2210. 1000:0099 2022.
[55] Yanwei Wang Shangdi Yu Zhen Guo, Peiqi Wang. Dr. Ilama: Improving small language models in -ppeif- qa via generative data augmentation. arXiv preprint arXiv:2305.07804 2023.
[56] Yiming Zhu, Peixian Zhang, Ehsan-UP UI Haq, Pan Hui. and Gareth Tyson. Can chatgpt reproduce suman-generaad labels? a study of social computing tasks. arXiv preprint arXiv:2304.1015, 2023.
14


Reviews:
[{"review_id": "cL3q6PRUfL", "reviewer": "Reviewer_Ltph", "summary": "This paper introduces an algorithm to fine-tune LMs for domain specific tasks under a certain budget constraint. They use a mix of human and a high-fidelity LM for annotations. They fix the annotation budget for each and introduce an algorithm to sample from the unannotated pool and distribute it between human labelers and LM annotator. They perform extensive comparisons and ablation studies and show the effectiveness of their approach.", "questions": "- The authors state \"Unfortunately, such an approach is susceptible to the misinformation of LLMs through hallucination, which risks generating unreliable or falsified labels and will, in turn, demolish the model’s utility for high-stakes applications like healthcare and finance, where the truth is of utmost importance.\" Do they have evidence (from their work or others) that this is the case?\n- Are the samples for the GPT-3.5 in Fig 5 drawn randomly or in accordance to the uncertainty score?\n- The authors do not discuss how the uncertainty score is calculated. A brief description would be useful. ", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The results section clearly demonstrate the effectiveness of their method under the conditions they considered. \n- The paper is well written and easy to follow. Claims are clearly stated, supported by empirical data.\n- The results are well explored and discussed. Good intuitions provided.", "weaknesses": "- The authors considered very limited tasks. They do not consider the impact of their approach on more generative tasks.\n- Their approach and conclusions strongly depends on the LM they used for annotations. Although this is specified in the limitation of their work, more powerful LMs will make the effect of their approach less profound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Ltph,\n\nThank you very much for the constructive comments and feedback. We are happy to hear that the reviewer found our method effective, and our results are well explored and discussed.  Below, we have tried to address all of your feedback and questions. Please take a look and let us know in case you would like additional clarification on any of these points.\n\n> **Impact of approach on more generative tasks.**\n\nOur focus in this work is to build a novel end-to-end framework for cost-effective fine-tuning of LMs on specialized domain-specific tasks that are beyond the scope of standard pre-training. Despite the rarity of such domain-specific benchmarks compared to general NLG tasks, we conducted a relatively comprehensive evaluation of our method, within our allowable computational budget, on four diverse tasks covering two of the most important application domains, i.e., finance and healthcare. We believe our proposed framework can be easily extended to more generative tasks such as open/close-book question-answering provided with proper data annotation and evaluation criteria.\n\n> **More powerful LMs will make the effect of the approach less profound.**\n\nWe agree with the reviewer that the performance of our framework is dependent on the LM used for annotation, which is verified in our experiments in Section 4.4. Generally speaking, our method benefits from more powerful LMs that provide higher annotation quality and thus could help to further reduce the cost of human annotations. This does not conflict with our objective. In fact, fine-tuning on domain-specific data is an essential step to obtaining a powerful and reliable LM for highly specialized domains. However, one common challenge faced in practice is the expensive cost of recruiting experienced human annotators in these domains, which our study aims to address. We envision that in real-world development environments, our framework can be applied iteratively on streaming data flows to continuously improve an LM product.\n\n> **Evidence for misinformation of LLMs through hallucination.**\n\nThanks for the comment. Yes, there are numerous pieces of evidence and experiments in the current literature that support this. For instance,\n\nDash et al. [1] conducted an evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery. Their experiments (see Table 2 in [1]) showed that while general-purpose LLMs are able to provide safe and credible responses, they often do not fully meet the specific information need of a given question, e.g., responses containing hallucinated references. \n\nKaran et al. [2] presented MultiMedQA, a benchmark medical dataset, and evaluated PaLM and Med-PaLM, which perform encouragingly but still reveal key gaps between human evaluation and remain inferior to clinicians. It is observed that models like PaLM and GPT may hallucinate convincing medical misinformation or incorporate biases that could exacerbate health disparities [3]. \n\nBang et al. [4] provided a comprehensive evaluation of ChatGPT using 23 datasets, showing that ChatGPT suffers from hallucination problems, i.e., generated hallucinated information beyond the given knowledge, which is supported by the experiments on misinformation detection related to COVID-19, and factuality on TruthfulQA.\n\nWe will add corresponding references in our final version.\n\n[1] Dash, Debadutta, Rahul Thapa, Juan M. Banda, Akshay Swaminathan, Morgan Cheatham, Mehr Kashyap, Nikesh Kotecha et al. \"Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery.\" arXiv preprint arXiv:2304.13714 (2023).\n\n[2] Singhal, Karan, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales et al. \"Large language models encode clinical knowledge.\" Nature (2023): 1-9. \n\n[3] Nori, Harsha, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. \"Capabilities of GPT-4 on medical challenge problems.\" arXiv preprint arXiv:2303.13375 (2023).\n\n[4] Bang, Yejin, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia et al. \"A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.\" arXiv preprint arXiv:2302.04023 (2023).\n\n> **Are the samples for the GPT-3.5 in Fig 5 drawn randomly or in accordance with the uncertainty score?**\n\nThanks for the comment. The samples for the GPT-3.5 in Fig.5 are drawn based on the uncertainty score. We will revise the caption to clarify this.\n\n> **The authors do not discuss how the uncertainty score is calculated. A brief description would be useful.**\n\nThanks for the suggestion. We've included a brief description of the calculation of the uncertainty score below and will add it to the final version. \n\nFor the tasks of natural language generation, the probability of the entire sequence, $\\mathbf s$, is the product of the conditional probabilities of new (next) tokens given past tokens, whose resulting log-probability is $\\log p( \\mathbf s | x) = \\sum_{i=1}^{n} \\log p(s_i | \\mathbf s_{<i})$, where $s_i$ is the $i$-th output token and $\\mathbf s_{<i}$ denotes the set of previous tokens. In this work, we define the uncertainty score $C$ by using the geometric mean of the token-level log-probabilities: $C = \\frac{1}{n} \\sum_{i=1}^n \\log p(s_i | \\mathbf s_{<i})$. Since the target LM is an offline model with complete white-box access to its token-level log probability, the uncertainty score can be easily calculated during model inference. \n\n"}, {"review_id": "m19hSsOEnZ", "reviewer": "Reviewer_EwLY", "summary": "This article is devoted to a new algorithm for development of small domain-speciﬁc LMs under limited annotation budgets. one of the main ideas of the algorithm is to use as data annotators a combination of a Human and LLM (Large language models). The authors also proposed two innovative developments in their algorithm: 1) prompt retrieval to improve LLM annotation, 2) variable batch size. The paper also presents a new conﬁdence-based query strategy based on applying k-means algorithm to embeddings of the sub-sampled unannotated data and using least conﬁdence for selected items. Of the strengths, I can emphasize the following: a new query strategy for selecting elements is presented, which showed an increase in comparison with other approaches, which introduces scientific novelty to the article. With the help of the tables given in the scientific work, the qualitative increase is shown by the approaches described in the article. Of the weaknesses , I can note that it is not described in detail what happens at the stage “Execute prompt retrieval from Ah” which is described in Algorithm 1. I would like to get a detailed description for \"prompt retrieval\", for those who are not familiar with this term. There are no tables for hyperparameters with which the models were trained. I would like to see additional statistics on datasets: average, minimum and maximum sentence length.\n\n==== After rebuttal  \nThank you for the reply, I have decided raise my score to 6.", "questions": "-", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The article presents a new algorithm for learning of small domain-speciﬁc LMs under limited annotation budgets the effectiveness of which is confirmed by comparison with various baselines. A new query strategy for selecting elements is presented, the effectiveness of the new strategy is confirmed by experiments. Two innovative designs are proposed to improve the learning process (prompt retrieval to improve LLM annotation, variable batch size), these approaches may be useful in other similar tasks.", "weaknesses": "* It is not clear which measure was used in clustering - cosine or euclidean distance, a comparison is needed. There is no information about which model is used to get embeddings in Exploration-Exploitation Query Strategy. Why was Sentence-BERT used to get embedding in Design 1? It seems that you need to use domain-speciﬁc models to get them in both cases. Since we use embedding of proposals, we would like to compare with strategies based on them, for example from the article “Deep Deterministic Uncertainty: A Simple Baseline”. In the article, the authors claim that they selecting cluster centers to reduce intra-iteration redundancy, but after all, at the next iteration, the selected elements in the new subset may be close to those already selected. Why do we reduce redundancy only inside the iteration?\n* There are no tables for hyperparameters of models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer EwLY,\n\nThank you for your detailed review and suggestions to improve the paper. We are glad to hear that you found that our idea is new, and effective and our designs are innovative and useful in broader tasks.  Below we address the concerns and questions raised in the review. \n\n> **A detailed description of \"prompt retrieval\" is desired for those who are not familiar with this term.**\n\nThank you for your suggestion. The goal of prompt retrieval is to retrieve annotated data instances that are semantically similar to the query as in-context learning examples to improve the LLM annotator's performance without the need to fine-tune it. Intuitively, the selected instance-annotation pairs would serve as an illustration to facilitate generating better and more accurate annotations for domain-specific applications. To operationalize this idea, we follow existing studies to retrieve annotated instances that are close to the queried instance in some embedding space that captures semantic or lexical similarities. The design of prompt retrieval is originally described in Section 2.2. Here we provide a more detailed description of the procedure below.\n\nGiven the queried instance $x$, annotated data pool $\\mathcal{A}$, sentence encoder model (Sentence-BERT), and the number of in-context examples $k$, we execute the following steps:\n\n1.\tCompute the embeddings of the queried instance and instances from the annotated pool.\n2.\tRetrieve the nearest $k$ neighbors of the queried instance $x$, denoted as $x_1, x_2, ...,x_k$, from $\\mathcal{A}$ according to a pre-defined similarity metric (e.g., cosine similarity) measured in the embedding space.\n3.\tConcatenate $k$ instances along with their annotations to form the in-context examples, which are then used to construct the prompt for the queried instance to generate annotation from the LLM annotator.\n\n> **Hyperparameters of the model.**\n\nThe hyperparameter setting regarding fine-tuning is presented in Table 3 of the supplementary material. We have summarized additional hyperparameters related to prompt retrieval and query strategy in the following table.\n\n| Hyperparameters              | Values                                    |\n|------------------------------|-------------------------------------------|\n| Sentence encoder model       | sentence-transformers/all-mpnet-base-v2   |\n| Dimension of embeddings      | 768                                       |\n| Pooling method               | mean pooling                              |\n| Similarity retrieval metric  | cosine similarity                         |\n| Number of clusters           | annotation budget at a specific iteration |\n| Number of retrieved examples | 5                                         |\n| Size of unlabeled data pool   | 3000                                      |\n\n> **Additional statistics on datasets: average, minimum and maximum sentence length.**\n\nThanks for the suggestion. We have summarized the additional statistics in the table below, with the average, minimum, and maximum sentence length (in tokens) included, and will add it to the revised version.\n\n| Sentence   length |  FPB  |   Headline   |  MedQA | PubMedQA |\n|:-----------------:|:-----:|:------------:|:------:|:--------:|\n|      Average      | 28.16 |        12.98 | 167.29 |    20    |\n|      Minimum      |   2   |       2      |   17   |     6    |\n|      Maximum      |  148  |      39      |   551  |    50    |\n\n> **The measure used in clustering and the embedding model used in Exploration-Exploitation Query Strategy.** \n\nFor k-means clustering, we used Euclidean distance which is the default measure. In the exploration-exploitation query strategy, we use the same Sentence-BERT model as used in prompt retrieval.\n\n> **Why was Sentence-BERT used to get embedding in Design 1? It seems that you need to use domain-speciﬁc models to get them in both cases.** \n\nOur proposed framework is flexible with different embedding models. We choose to use Sentence-BERT in our implementation because it is a general-purpose sentence encoding model which can be employed in different tasks from diverse domains without requiring task-specific fine-tuning. Although encoders fined-tuned on domain-specific tasks would potentially further improve the performance by making the prompt retrieval more effective, it would require more in-domain annotated data, which conflicts with our objective of designing a cost-effective adaptation framework.\n\n> **Why reduce redundancy only inside the iteration?** \n\nThe proposed EEQ strategy aims to reduce both intra-iteration and inter-iteration redundancy. Typically, uncertainty-based methods acquire similar samples within an iteration, known as intra-iteration redundancy, while diversity-based approaches acquire similar samples across iterations, known as inter-iteration redundancy. Existing hybrid methods try to avoid intra- and inter-iteration redundancies by combining diversity and uncertainty sampling but may suffer from these redundancies due to unifying the uncertainty and diversity objectives into a single query function, which tends to prioritize one objective over the other. \n\nOur EEQ strategy is an independent two-step selection by first executing the diversity sampling and then the uncertainty sampling. In the first step, we select a subset of an unlabeled data pool consisting of diverse data points in the embedding space. In the second step, we acquire high-uncertainty data points (the ones that are predicted with low confidence by the current model) from the subset.\n"}, {"review_id": "ObDII1cpzS", "reviewer": "Reviewer_YvhV", "summary": "This paper studies the problem of how to fine-tune a relatively small, domain-specific language model under the constraints of limited annotation resources. The authors propose an Interactive Multi-Fidelity Learning (IMFL) approach. This approach aims to optimize the performance of the fine-tuned language model through multiple rounds of human-model collaborative annotation and fine-tuning. In each round, the method applies an exploration-exploitation query strategy (such as diversity sampling, uncertainty sampling, etc.) to balance high-fidelity human annotations and low-fidelity model annotations. Moreover, the IMFL method also introduces prompt retrieval and variable batch sizes to make better use of the annotated samples. The authors test the proposed IMFL method on four datasets from the financial and medical domains.\n\n", "questions": "Why is the budget set only to 1000? While I understand that human annotations may be constrained in real-world applications, I think it would be meaningful to include the performance and analysis of the model under different budget settings in the paper.\n", "limitations": "Yes, the authors have explained the limitations of their study in the paper.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- In general, I think the experimental design of this paper is sound and the paper is easy to follow and understand.\n- I think the problem studied in this paper is interesting and highly meaningful. The authors offer a straightforward and practical strategy for optimizing the performance of the fine-tuned language model under limited human and computational resources. Notably, the designs of the variable batch size and the coordination of human and model annotations in each round are innovative.\n- The authors validate their method on four different datasets, consistently outperforming the baseline models in all cases. Particularly, I appreciate the extensive experiments and analyses performed in Section 4 to verify the effectiveness of each component of their framework, along with practical suggestions for future practitioners.", "weaknesses": "One major weakness of this paper is its strong assumption that we already have a good pool of unlabeled data, and all the methods are experimented on these pools. However, in many real-world settings, we do not have such an immediately available unlabeled data pool. The quality of the unlabeled data pool might be crucial to the effectiveness of the proposed method. Without relevant experiments, we don't know if the method proposed will still work in real-world settings.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer YvhV,\n\nThank you very much for your detailed and thoughtful comments. We are glad to hear that you found the paper interesting, highly meaningful, and innovative, and our experiments are sound and strong to verify the effectiveness. Below we address the constructive comments and feedback raised in the review. \n\n> **Assumption on the availability and quality of the unlabeled data pool.**\n\nThanks for the comment. As highlighted in the Discussion and Limitation section, IMFL's performance is limited by the size of the unannotated dataset and the diversity of examples presented in the dataset. This is because IMFL functions by annotating existing samples rather than creating new samples. The authors believe that the assumption of an unlabeled data pool is mild as this is the basis for applying any data-driven / deep learning techniques. Nevertheless, the authors do agree that the quality of the unlabeled data pool is closely related to the final model performance. Therefore, in practice, some basic data wrangling techniques such as data cleaning, down-selection, and filtering, could be employed before applying the proposed IMFL to further improve the final model performance. These efforts are helpful from a practical standpoint but a nuanced study on the data pipeline is beyond the scope of our research.\n\n> **Why is the budget set only to 1000? I think it would be meaningful to include the performance and analysis of the model under different budget settings in the paper.**\n\nThanks for the valuable suggestion. The authors agree that such analysis is meaningful and thus we have included experiments on the impact of different human annotation budgets and summarized the results in Table 2 in the supplementary material. Our results suggest that reducing the amount of allowed human annotation budget would result in decreased model performance (though still outperforming using all GPT-generated annotations).\n\nIn our experiments, we select a default annotation budget of 1000 based on the following considerations: (1) Computational Costs: Using a larger annotation budget would result in an increasing amount of annotation and computational costs for both our experiments and practical deployment. An annotated budget of <= 1000 is a typical setting used by many prior works [2-5] that consider the low-resource setting. (2) Dataset Annotation Constraints: The human annotation budget for our experiments is also limited by the availability of annotated domain-specific datasets. For example, PubMedQA only has 1000 expert-annotated QA instances [1]. In general, the authors believe an annotation budget of 1000 is practically meaningful for NLP tasks in specialized domains like medicine and finance and such experimental design choices should not affect the validity of our findings.\n\n[1] Jin, Qiao, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. \"PubMedQA: A Dataset for Biomedical Research Question Answering.\" In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577. 2019.\n\n[2] Grießhaber, Daniel, Johannes Maucher, and Ngoc Thang Vu. \"Fine-tuning BERT for low-resource natural language understanding via active learning.\" arXiv preprint arXiv:2012.02462 (2020).\n\n[3] Yuan, Michelle, Hsuan-Tien Lin, and Jordan Boyd-Graber. \"Cold-start Active Learning through Self-supervised Language Modeling.\" In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7935-7948. 2020.\n\n[4] Schröder, Christopher, Andreas Niekler, and Martin Potthast. \"Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers.\" In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2194-2203. 2022.\n\n[5] Maekawa, Seiji, Dan Zhang, Hannah Kim, Sajjadur Rahman, and Estevam Hruschka. \"Low-resource interactive active labeling for fine-tuning language models.\" In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3230-3242. 2022.\n"}, {"review_id": "oQvm1DWado", "reviewer": "Reviewer_ogFx", "summary": "The paper proposes the Interactive Multi-Fidelity Learning (IMFL) framework to develop small domain-specific Large Language Models (LLMs) under limited annotation budgets. Specifically, IMFL balances low-fidelity LLM annotations and high-fidelity human annotations to maximize model performance. Experiments on four domain-specific tasks demonstrate that IMFL outperforms single fidelity annotations, offering a cost-effective solution for domain-specific LLM development.", "questions": "1. Have you considered studying the effect of using GPT-3.5 and GPT-4 as annotators in your research? It would be interesting to explore how the performance and annotation quality differ when employing these different versions of the language model (with different prices/label quality) as annotators.\n\n2. Have you tried on other AL baselines such as [14-16] mentioned in your paper? This would make the results with more AL baselines more insightful.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the practical challenges of deploying large language models in domain-specific tasks, such as their scale and high annotation costs.\n\n2.  This work considers the practical setting of addressing the challenges of deploying LLMs, hence is somehow realistic. \n\n3. The IMFL framework offers an effective solution for the cost-effective development of small domain-specific LLMs by leveraging multi-fidelity learning.\n\n4. Experiments on financial and medical tasks provide empirical evidence of the superiority of IMFL over single fidelity annotations.", "weaknesses": "1. While the paper demonstrates superior performance compared to single fidelity annotations, it would be valuable to provide a more comprehensive comparison with alternative approaches or baselines to highlight the specific advantages of IMFL.\n\n2. It would be interesting if the author can provide more rationale for the selection of the dataset. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer ogFx,\n\nThank you very much for taking the time to engage with our paper thoroughly, and constructive comments. We’re happy to hear that you thought our work is practical and effective, addressing the ground challenges of deploying LLMs in domain-specific tasks. Please find our response below:\n\n> **It would be interesting if the author can provide more rationale for the selection of the dataset.**\n\nThanks for the great suggestion. Our focus in this work is to build a novel end-to-end framework for cost-effective fine-tuning of LMs on specialized domain-specific tasks that are beyond the scope of standard pre-training. Thus for evaluation, we mainly select datasets that focus on domain-specific applications with direct real-world implications, such as finance and medicine.  The finance datasets are selected following the BloombergGPT [1] paper. The medical datasets are selected following prior work such as MedPaLM [2].  To increase the diversity of datasets and tasks, we select FPB to cover sentiment analysis and Headline to cover news classification, while PubMedQA and MedQA are very commonly used QA datasets for evaluating the capabilities of LMs in the medical domain.\n\n\n[1] Wu, Shijie, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. \"Bloomberggpt: A large language model for finance.\" arXiv preprint arXiv:2303.17564 (2023).\n\n[2] Singhal, Karan, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales et al. \"Large language models encode clinical knowledge.\" Nature (2023): 1-9.\n\n> **Have you considered studying the effect of using GPT-3.5 and GPT-4 as annotators in your research? It would be interesting to explore how the performance and annotation quality differ when employing these different versions of the language model (with different prices/label quality) as annotators.**\n\nThank you for the comment. Our framework is flexible with any LMs for annotations. We have investigated the impact of different models by comparing GPT-3.5 with GPT-3 (see Table 6), as GPT-4’s API was not available at the time of writing. Our observation is that GPT-3.5 provides better annotation quality but at the cost of a (slightly) higher price. Per the reviewer’s request, we have added a comparison of the annotation quality of GPT-3.5 and GPT-4 and summarized the results in the following tables.  GPT-4 annotator shows better accuracy but the price is much higher than GPT-3.5 and GPT-3.  We will add them to the updated version.\n\n|          | GPT-3 Annotation |        |        | GPT-3.5 Annotation |        |        | GPT-4 Annotation |        |        |\n|:--------:|:----------------:|:------:|:------:|:------------------:|:------:|:------:|:----------------:|:------:|:------:|\n|          |     retrieval    | 5 shot | 0 shot |      retrieval     | 5 shot | 0 shot |     retrieval    | 5 shot | 0 shot |\n| Headline |       75.59      |  72.51 |  70.25 |        79.4        |  76.15 |  73.31 |       80.13      |  78.34 |  77.2  |\n|   MedQA  |       51.42      |  44.89 |  42.03 |        59.45       |  53.57 |  50.82 |       82.67      |  81.38 |  78.87 |\n\n> **Have you tried on other AL baselines such as [14-16] mentioned in your paper? This would make the results with more AL baselines more insightful.**\n\nOur work focuses on addressing the problem of cost-effective adaptation of LMs for domain-specific applications where existing works [14-16] are not directly applicable. Specifically, [14] studies contrastive AL and is computationally intensive, as its effectiveness relies on using a large batch size, making it unsuitable for low-resource settings. [15] and [16] mainly focus on the cold-start problem in AL which is orthogonal to our work. Advances in this line of research might be helpful to improve our framework in the cold-start scenarios, but a nuanced study is beyond our focus.\n"}]

Author rebuttal:
Dear reviewers,

Thank you all for taking the time to review our paper and we sincerely appreciate all the feedback. In particular, we feel encouraged to see that the reviewers find that

- **The topic of our paper is important, interesting, and practical**:  “This paper is interesting and highly meaningful. The authors offer a straightforward and practical strategy ... ” (reviewer YvhV); "The paper addresses the practical challenges of deploying LLMs in domain-specific tasks" (reviewer ogFx); "This work considers the practical setting of addressing the challenges of deploying LLMs, hence is somehow realistic." (reviewer ogFx). 

- **Our idea is innovative**:  “A new query strategy for selecting elements is presented, which introduces scientific novelty to the article” (reviewer EwLY); "Two innovative designs are proposed to improve the learning process (prompt retrieval to improve LLM annotation, variable batch size)" (reviewer EwLY); "Notably, the designs of the variable batch size and the coordination of human and model annotations in each round are innovative" (reviewer YvhV); 

- **Our proposed solution is effective**:  "The results section clearly demonstrates the effectiveness of their method." (reviewer Ltph);  "Particularly, I appreciate the extensive experiments and analyses performed in Section 4 to verify the effectiveness of each component of their framework, along with practical suggestions for future practitioners." (reviewer YvhV); "The IMFL framework offers an effective solution for the cost-effective development of small domain-specific LLMs by leveraging multi-fidelity learning." (reviewer ogFx). 

- **Our experiments are thorough and the results are promising**:  "The results are well explored and discussed. Good intuitions provided." (reviewer Ltph); "The authors validate their method on four different datasets, consistently outperforming the baseline models in all cases. " (reviewer YvhV); "Experiments on financial and medical tasks provide empirical evidence of the superiority of IMFL over single fidelity annotations." (reviewer ogFx)

- **And our paper is easy to follow and well-written**:  “The experimental design of this paper is sound and the paper is easy to follow and understand.”(reviewer YvhV); “The paper is well written and easy to follow. Claims are clearly stated, supported by empirical data.” (reviewer Ltph).


Based on the reviewer's feedback, we have made the following changes to further improve the clarity of the manuscript:

- Added a brief description of uncertainty score calculation as suggested by reviewer Ltph. 

- Added a detailed description of the prompt retrieval and the hyperparameters as suggested by reviewer EwLY. 

- Added a statistical analysis of datasets, including the average, minimum and maximum sentence length as suggested by reviewer EwLY. 

- Added a discussion and clarification about the exploration-exploitation query strategy, including the details about clustering, embedding model, redundancy, and hyperparameters, as suggested by reviewer EwLY

- Added additional experiments to compare the effect of GPT-3.5 and GPT-4 in terms of annotation quality and performance as suggested by reviewer ogFx

We are committed to address all comments and we welcome any further questions or discussions.

Authors of paper 2221


Meta-review: This is a timely paper with sound top-level ideas for blending in human supervision with LLM supervision for training a custom, smaller-scale model.  However, beyond that, the details follow standard practices and not particularly technically deep.  So much more can be done than what the paper proposes.  Hopefully, this paper will trigger follow up work.  
Overall all reviewers are positive, and for the strengths mentioned above I recommend accept as a poster.

A minor typo: "closets" should be "closest".
Decision: Accept (poster)