SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we compare IMFL with single fidelity annotations to validate the effectiveness of the proposed multi-fidelity paradigm. Fig. 4 compares IMFL with using only human annotations, where  $1 \times$  Human,  $3 \times$  Human, and  $5 \times$  Human, represents the results obtained by fine-tuning on 200, 600, and 1000 human annotations, respectively. A detailed version of the main results are shown in Appendix C.1. Note that  $5 \times$  Human (1000 human annotations) can be seen as the performance upper bound of IMFL (200 Human + 800 GPT-3.5) if all budget is human annotation. From the results, we can clearly see that IMFL significantly outperforms the  $3 \times$  human annotation baselines in all four tasks. Particularly, IMFL achieves very close performance as  $5 \times$  human annotation on both Headline and PubMedQA datasets with only marginal differences (0.83% and 1.32% absolute loss respectively). This result indicates that the high human annotation cost in domain-specific tasks can be greatly reduced by employing IMFL, which utilizes fewer human annotations combined with cheaper and faster GPT-3.5 annotations to achieve similar performance.
Fig. 5 compares IMFL with using only GPT-3.5 annotations with the same total annotation budget (varied from 260 to 1000 samples). We have the following observations. First, our IMFL outperforms the GPT-3.5 annotation by a large margin (in terms of absolute gain) on PFB (+7.35%), Headline (+8.3%), PubMedQA (+6.89%) and MedQA (+19.95%) given the same 1000 annotation budget.
Figure 4: Comparisons between our multi-fidelity learning (200 human annotations + 800 GPT-3.5 annotations) and various sizes, i.e.,  $200(1\times)$ ,  $600(3\times)$ , and  $1000(5\times)$  of human annotations.
Second, on three out of four datasets (FPB, PubMedQA, and MedQA), models tuned using IMFL with a total annotation budget of 260 (100 Human + 160 GPT-3.5) are able to achieve better performance than using 1000 GPT-3.5 annotations. On the Headline dataset, using 1000 GPT-3.5 annotations performs slightly better than using IMFL with a total budget of 260, but still worse if the total budget is increased to 470 ((100 + 50) Human + (160 + 160) GPT-3.5). This shows that while GPT-3.5 demonstrates promising abilities to reproduce human-generated labels [14, 56], relying solely on low-fidelity GPT-3.5 labels is not ideal for fine-tuning LMs for domain-specific tasks. In addition, compared with using only GPT-3.5 annotations, IMFL shows more reliable results with smaller variance, which benefits from a combination of human annotation and the similarity-based prompt retrieval strategy for improving the in-context learning capability of LLMs. These results verified that IMFL can efficiently utilize sparse human supervision to enhance GPT-3.5 annotations and consequently achieve better performance.
Figure 5: Comparisons between our multifidelity learning paradigm and single low-fidelity (all GPT-3.5) annotation on four domain-specific tasks given the same total 1000 annotation budget. Note that the samples for all GPT-3.5 are drawn based on the uncertainty score.
4.1 Exploitation-Exploration Query vs Random Query Strategy
Table 3 compares the proposed EEQ strategy with the random query strategy on multiple settings given a limited annotation budget. Under the multi-fidelity setting, our EEQ strategy outperforms the random query strategy by a large margin (5.91% absolution gain on average). Although fine-tuning with only human annotations is supposed to produce the best results as it has the highest annotation accuracy, we observe that using only human annotations with random query is generally worse than using human and GPT-3.5 annotations with EEQ query (on three out of four datasets), thereby validating the effectiveness of the proposed EEQ query strategy. We observe that one exception is the MedQA dataset, where using only human annotations with random query performs slightly better than the proposed IMFL. This is because GPT-3.5 shows relatively low annotation accuracy on this dataset and consequently induces a negative impact on the fine-tuning performance by injecting noises into the annotated data. However, if using only GPT-3.5 annotations with the random query under the considered annotation budget, the performance would drop significantly.
Table 3: A comparison of our EEQ query strategy and random query strategy.
Method	Budget	Query Strategy	Dataset
Multi/Single	Human	GPT-3.5	EEQ/Random	FPB	Headline	PubMedQA	MedQA
Human + GPT-3.5	200	800	EEQ	47.88	81.09	73.76	67.98
Human + GPT-3.5	200	800	Random	41.94	74.32	66.03	63.77
Only Human	1000	0	Random	43.81	75.46	68.87	70.17
Only GPT-3.5	0	1000	Random	38.56	71.04	65.89	47.13
4.2 Prompt Retrieval with Variable Batch Size
To evaluate the effectiveness of similarity-based prompt retrieval and variable batch size, in Table 4, we consider several variants with random prompt retrieval and equal batch size as baselines for comparison. We find that similarity-based retrieval shows superior performance compared to the random prompt retrieval baselines (e.g., 80.28 vs 73.77 on Headline and 72.05 vs 68.10 on PubMedQA), and using variable batch size further boosts the effectiveness of retrieval by providing a larger and more diverse set of candidate examples, which is crucial in the limited budget setting.
Our interactive learning is conducted over the course of multiple rounds of interactive, with each round using a single mini-batch of data for adaptive annotation and fine-tuning. Here we also compare the interactive multiple mini-batch update strategy with the full-batch strategy, where all annotations are acquired in a single round and then used to fine-tune the model. The full-batch strategy naturally reduces the computational cost for fine-tuning but loses the benefits of interactive improvements as shown in the results. However, interestingly, we find that the similarity-based retrieval still provides a good amount of improvements in the full-batch setting which can achieve a competitive performance that is slightly better than using mini-batch updates with random retrieval. Therefore, we recommend that practitioners consider this alternative option, i.e., full-batch incorporated with similarity-retrieval, if the computational budget of fine-tuning is limited.
Table 4: Effects of prompt retrieval, variable batch size, and batch orders.
Method	Dataset
Budget	Batch	Batch size	Retrieval	FPB	Headline	PubMedQA	MedQA
1000	5 Mini-Batch	Variable	Similar	47.88	81.09	73.76	67.98
1000	5 Mini-Batch	Equal	Similar	46.34	80.28	72.05	66.11
1000	5 Mini-Batch	Variable	Random	42.09	73.98	67.44	63.56
1000	5 Mini-Batch	Equal	Random	42.34	73.77	68.10	63.42
1000	1 Full-Batch	NA	Similar	43.72	75.48	68.90	63.79
1000	1 Full-Batch	NA	Random	39.80	72.11	65.94	57.23
4.3 Alternative Query Strategies
Besides the random query and the proposed EEQ query, we also explore several additional query strategies for interactive multi-fidelity learning, including (i) confidence-based strategies: predictive entropy (Entropy) [35], least confidence (Least-c) [22], breaking ties (Breaking-t) [27]; (ii) diversity-based strategies: K-means [50], Diversity [37]; and (iii) hybrid strategy [19], a combination of confidence and diversity by a weighted sum. As shown in Table 5, our EEQ strategy outperforms all the other methods on two representative tasks (Headline and MedQA). These alternative strategies are simple to implement and perform better than the random baseline. Unfortunately, directly applying them to our multi-fidelity paradigm does yield the most desirable performance. The one that achieves the closest performance is the hybrid strategy. However, it ignores the effects of annotation orders and fidelity which are important ingredients for achieving high performance in our multi-fidelity setting.
4.4 Annotation Accuracy by Different GPT-based Annotators
The fine-tuning performance relies on the annotation accuracy of the LLM annotator, since noisy annotations may hurt the final model performance in terms of accuracy and reliability. Here we focus
Table 5: A comparison of various alternative query strategies on two representative tasks.
Dataset	Random	Entropy	Least-c	Breaking-t	K-means	Diversity	Hybrid	EEQ
Headline	74.32	76.42	77.55	77.34	76.59	77.61	79.23	81.09
MedQA	63.77	65.15	65.21	65.28	66.44	66.41	66.94	67.98
on evaluating the annotation accuracy of different variants of GPT (i.e., GPT-3 vs GPT-3.5) instead of fine-tuning accuracy through multiple experiments. We notice that GPT-3.5 annotation with zero shot performs much worse than few-shot and our retrieval methods in domain-specific tasks. This is because although naive GPT-3.5 shows promising performance in doing zero/few-shot learning in-distribution scenarios, it lacks domain knowledge to make accurate predictions in the considered out-of-domain tasks. The proposed prompt retrieval leveraging human annotations from domain experts with in-context learning capabilities of LLMs substantially improves the performance of GPT-3.5 annotation.
Table 6: A comparison of annotation accuracy by GPT-3 and GPT-3.5 in zero/few-shot learning.
Dataset	GPT-3 Annotation	GPT-3.5 Annotation
retrieval	5-shot	0-shot	retrieval	5-shot	0-shot
Headline	75.59	72.51	70.25	79.40	76.15	73.31
MedQA	51.42	44.89	42.03	59.45	53.57	50.82
If the annotation budget is very limited, GPT-3 is a cheaper alternative but underperforms GPT-3.5 even with prompt retrieval applied. In contrast, we have a chance to utilize GPT-4 (more expensive and limited access) for annotation in our multi-fidelity paradigm. Please see additional GPT-4 annotation results in Appendix C.3. Recent work shows promising capabilities of GPT-4 on medical challenge problems. For example, GPT-4 achieves 81.38 (5-shot) and 78.87 (0-shot) for the MedQA task as reported by [32]. Note that IMFL uses GPT-3.5 as our LLM annotator but is easy to extend to other LLM annotators, e.g., GPT-based models or open-source models such as LLaMA which depends on the annotation budget. An exhaustive study of different LLM annotators is beyond the scope of this work.
4.5 Ablation Study of Human Annotation Ratio
Given a total 1000 annotation budget, a key question is how to assign the budget to human annotators or GPT-3.5 annotators. In our original setting, we use a  $20/80$  ratio since the human annotations are much more expensive (money cost, time cost, and training cost, specifically in domain-specific areas, e.g., finance and medicine) than GPT-3.5 annotations. We also expect to minimize the ratio of human annotations so we conduct an ablation study for 10/90 and 5/95 ratios and evaluate their effect on performance in our framework. Table 7 shows the performance comparisons of various ratios of human annotations, i.e.,  $0.5 \times$  and  $0.25 \times$  human annotations. We can note that the performance drops obviously when less human effort is conducted. For the case of  $0.5 \times$  human annotations, it is lower than our original setting but still comparable to  $3 \times$  human annotations. However, the case of  $0.25 \times$  human annotations shows a significant decrease because too few human annotations weaken the effect of in-context prompt retrieval and reduce the accuracy of initial uncertainty estimation. In short, a certain amount of human annotation is necessary for our framework even though we seek minimal human effort. We thus need to consider a trade-off between accuracy and annotation budgets.
Table 7: Performance comparisons of various ratios of human annotations on four datasets.
Method	Number of Annotations	Dataset
Human	GPT-3.5	FPB	Headline	PubMedQA	MedQA
IMFL	200 (1×)	800	47.88 ± 0.98	81.09 ± 0.58	73.76 ± 0.95	67.98 ± 1.45
IMFL	100 (0.5×)	900	43.66 ± 1.42	75.41 ± 1.01	70.88 ± 1.08	61.44 ± 1.83
IMFL	50 (0.25×)	950	40.76 ± 1.48	73.65 ± 1.09	68.18 ± 1.11	52.38 ± 1.93
Domain-specific LLMs. The significance of domain-specific training for encoder-only masked language models is widely recognized. Two commonly adopted methods are to either train BERT models [9] from scratch using domain-specific data or to continue pre-training an existing model on new domain-specific data. Following these approaches, several BERT-based models are built by domain experts, e.g., BioBERT [21], ClinicalBERT [15], etc. A recent trend is to train decoder-only models by utilizing domain-specific data, such as Med-PaLM [38], BioGPT [26] and BloombergGPT [46]. These findings highlight the advantages of in-domain pre-training, especially if sufficient data is available. However, an underlying challenge is how to train domain-specific LMs when there is insufficient data for large-scale pre-training due to a limited annotation budget. Our work addresses this underexplored problem by developing a cost-effective fine-tuning paradigm with limited budgets of human annotation.
Multi-fidelity learning. Our work is motivated by recent findings that suggest GPT models are capable of replicating or even outperforming human annotation as indicated by several studies [14, 43, 56]. However, these approaches suffer from unreliable annotations and lack confidence when applied to domain-specific tasks, especially in specialized high-stakes fields like finance and medicine. Our key idea originates from multi-fidelity optimization approaches [33, 23, 24] that optimize the objective function by utilizing varying approximations with different levels of precision and cost. Previous studies in the field of NLP have explored "dual supervision" to train models by combining two types of labels [45]. In contrast to this naive combination approach, we delve into a novel multi-fidelity framework that achieves cost-effective adaptation of domain-specific language models through fine-tuning and in-context learning.
Active learning. Active Learning (AL) is an extensively studied field concerned with improving the performance of language models with fewer labeled instances [54, 44, 31]. Current works within AL typically focus on two main scenarios: active fine-tuning [42, 28] and active in-context learning [10, 41]. The former involves iteratively updating model parameters but is not well-suited for directing training/fine-tuning LLMs such as GPT-3.5 which would induce high computational costs. Conversely, the latter is efficient but the performance solely relies on the few-shot learning ability of LLMs, which is unreliable for domain-specific tasks that require expert knowledge beyond standard pre-training data. In contrast, the proposed IMFL, which fully utilizes few high-fidelity annotations from human annotators to guide the LLM-annotator, can be regarded as synergizing the power of both fine-tuning and knowledge distillation from LLMs under sparse human supervision. Our experiments demonstrate that our approach can significantly reduce human annotation efforts while achieving highly competitive performance given a limited budget for annotation and computational resources, which enables flexible and effective deployment in real-world applications.
We compare IMFL to single fidelity annotations to evaluate the effectiveness of our proposed multifidelity paradigm. The extensive experimental results reveal that employing IMFL can significantly reduce the high cost of human annotation in domain-specific tasks. Furthermore, we demonstrate that IMFL efficiently uses sparse human supervision to improve GPT-3.5 annotations through prompt retrieval and in-context learning, ultimately leading to enhanced performance.
Despite the promising performance, we note there are certain limitations to our approach. First, the current IMFL framework assumes that the annotation budget is defined by the number of annotations, rather than reflecting the true cost which typically involves multiple complex factors (e.g., administrative cost, training cost of human annotators, time, etc.) in real-world scenarios. Second, IMFL's performance is limited by the size of the unannotated dataset and the diversity of examples presented in the dataset as IMFL only seeks to improve performance through annotating existing samples rather than creating new samples. Lastly, limited by budgets and the capacity of the LM to be fine-tuned, IMFL does not achieve state-of-the-art performance in some general NLP tasks, where directly adopting the latest LLMs remains a better choice. Nevertheless, we anticipate the performance of IMFL to continue to grow by incorporating stronger LLM annotators, such as GPT-4, to further improve annotation accuracy. We leave this as our future work.
[51] Jiaxin Zhang. Modern monte carlo methods for efficient uncertainty quantification and propagation: A survey. Wiley Interdisciplinary Reviews: Computational Statistics, 13(5):e1539, 2021.
[52] Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Kumar Sricharan. SAC<sup>3</sup>: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency. EMNLP, 2023.
[53] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
[54] Zhisong Zhang, Emma Strubell, and Eduard Hovy. A survey of active learning for natural language processing. arXiv preprint arXiv:2210.10109, 2022.
[55] Yanwei Wang, Shangdi Yu, Zhen Guo, Peiqi Wang. Dr. llama: Improving small language models in domain-specific qa via generative data augmentation. arXiv preprint arXiv:2305.07804, 2023.
[56] Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human-generated labels? a study of social computing tasks. arXiv preprint arXiv:2304.10145, 2023.