SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaxin Zhang $^{1}$ , Zhuohang Li $^{2}$ , Kamalika Das $^{1}$ , Sricharan Kumar $^{1}$ $^{1}$ Intuit AI Research  
 $^{2}$ Vanderbilt University
{jiaxin_zhang, kamalika_das, sricharan_kumar}@intuit.com zhuohang.li@vanderbilt.edu
Large language models (LLMs) have demonstrated remarkable capabilities in various tasks. However, their suitability for domain-specific tasks, is limited due to their immense scale at deployment, susceptibility to misinformation, and more importantly, high data annotation costs. We propose a novel Interactive Multi-Fidelity Learning (IMFL) framework for the cost-effective development of small domain-specific LMs under limited annotation budgets. Our approach formulates the domain-specific fine-tuning process as a multi-fidelity learning problem, focusing on identifying the optimal acquisition strategy that balances low-fidelity automatic LLM annotations and high-fidelity human annotations to maximize model performance. We further propose an exploration-exploitation query strategy that enhances annotation diversity and informativeness, incorporating two innovative designs: 1) prompt retrieval that selects in-context examples from human-annotated samples to improve LLM annotation, and 2) variable batch size that controls the order for choosing each fidelity to facilitate knowledge distillation, ultimately enhancing annotation quality. Extensive experiments on financial and medical tasks demonstrate that IMFL achieves superior performance compared with single fidelity annotations. Given a limited budget of human annotation, IMFL significantly outperforms the  $3 \times$  human annotation baselines in all four tasks and achieves very close performance as  $5 \times$  human annotation on two of the tasks. These promising results suggest that the high human annotation costs in domain-specific tasks can be significantly reduced by employing IMFL, which utilizes fewer human annotations, supplemented with cheaper and faster LLM (e.g., GPT-3.5) annotations to achieve comparable performance.
Large language models (LLMs) like GPT-3/ChatGPT/GPT-4 [4, 47, 5] have lately attracted great interest from both academia and industry due to their impressive in-context learning (ICL) abilities. However, the current state-of-the-art LLMs have since quickly grown from hundreds of billions [7] to even a trillion [20] parameters. Models of this scale require specialized hardware, massive-scale training data, and extensive computational power, which are inaccessible for most product or research teams. In addition, the generalizability of LLMs is predominantly decided by the scope of the underlying pre-training data. In fact, LLMs do not perform well out of the box in many real-world domains where specialized knowledge beyond the standard fields of pre-training is required (i.e., domain shifts), such as healthcare [26] and finance [46].
As an alternative to general-purpose LLMs, practitioners oftentimes find small domain-specific language models (LMs) to be more favorable as they require less training data and are faster to
Figure 1: (a) Proposed Interactive Multi-Fidelity Learning Framework (IMFL). IMFL aims at solving the best acquisition strategy that balances between low-fidelity automatic LLM annotations and high-fidelity human annotations to maximize model performance given limited annotation budgets. (b) IMFL significantly outperforms the  $3 \times$  human annotation baselines in all four tasks and is very close to  $5 \times$  upper bound in the Headline dataset (showed). This result indicates that the high human annotation cost in domain-specific tasks can be greatly reduced by employing IMFL, which utilizes fewer human annotations combined with cheaper GPT-3.5 annotations to achieve competitive performance.
compute, leading to faster development cycles and lower operating costs [55][2]. A common practice of developing such models is through the classic pre-training and then fine-tuning paradigm. Unfortunately, to achieve comparable performance as LLMs, tuning small LMs requires high-quality manual annotations on target domain data, which in many fields requires extensive human effort and expert knowledge, making supervised fine-tuning very expensive.
One promising approach to alleviate human annotation efforts is to leverage LLMs as knowledge bases for automatically annotating new data [43, 45]. Unfortunately, such an approach is susceptible to the misinformation [8, 39, 32, 2] of LLMs through hallucination [16, 53, 48, 30, 52], which risks generating unreliable or falsified labels and will, in turn, demolish the model's utility for high-stakes applications like healthcare and
Table 1: A qualitative comparison of human annotation, LLM annotation, and IMFL.
Human	LLM	IMFL
Cost Saving	Low	Very High	High
Quality	Very High	Low	High
Efficiency	Low	Very High	High
Performance	Very High	Low	High/Very High
finance, where the truth is of utmost importance. As a result, the key challenge at hand is how to effectively gather sufficient high-quality data given limited budgets on human annotation, which is a critical component in fine-tuning domain-specific LMs.
In this paper, we present a novel framework, named by IMFL, for achieving cost-effective development of domain-specific LMs, as illustrated in Fig. Our approach capitalizes on the insight that different data samples inherently exhibit different levels of hardness for learning [6]. Therefore, it is dispensable to request human annotation for every sample. By discerning each sample's hardness level, we can delegate the majority of the annotation tasks to automatic annotation tools such as LLMs while exclusively assigning a limited number of highly uncertain samples to human annotators, thereby reducing human effort significantly while still maintaining high annotation quality.
To improve cost efficiency, we formulate the domain-specific fine-tuning process as an interactive multi-fidelity learning problem. We deem LLMs and humans to be two sources of annotation with distinct fidelities, and aim to determine the optimal acquisition strategy that balances low-fidelity LLM-generated annotations and high-fidelity human annotations to maximize model performance under limited annotation budgets. We thus introduce an exploration-exploitation query strategy, wherein human annotations emphasize exploitation geared toward maximizing informativeness while LLM annotations concentrate on exploration to foster diversity and improve representatives.
To reduce the misinformation in LLM-generated annotations and improve model usability and reliability in the target domain, we incorporate two innovative designs. First, we utilize prompt retrieval to select in-context learning examples for each queried sample, thereby improving the
accuracy of LLM-generated annotations. Second, we implement variable batch sizes throughout the interactive annotation process, which manage the order in which each fidelity is chosen; this facilitates knowledge distillation and ultimately enhances annotation quality while stabilizing the LLM annotations.
We evaluate our approach on four language understanding tasks across two specialized application domains, i.e., finance and medicine. Our results highlight that LMs tuned through the proposed IMFL framework with GPT-3.5 as an auto-annotator significantly outperform LMs tuned with  $3 \times$  human annotations, and are even on par with LMs tuned with  $5 \times$  human annotations in some cases. In contrast to single-fidelity annotations such as only human or only LLMs, IMFL effectively addresses limitations related to cost saving, annotation quality, and efficiency (see Table 1). Furthermore, IMFL not only surpasses the performance of LLM annotators but also achieves highly competitive performance compared to human annotators, albeit at a substantially lower cost and effort.
2 Interactive Multi-fidelity Learning
We propose IMFL that builds on two key insights: (1) leveraging a substantial amount of low-fidelity annotations generated by LLMs to compensate for the insufficiency of high-fidelity human annotations during fine-tuning, and (2) utilizing high-fidelity human annotations as supervision signals to distill knowledge from LLMs while simultaneously enhancing their output annotation quality through in-context learning. Essentially, our approach, IMFL, can be regarded as a synergy between fine-tuning and knowledge distillation under sparse human supervision.
Given a total annotation budget  $\mathcal{B}$  and a computational cost  $\mathcal{C}$  (e.g., costs for fine-tuning, inference, and query), we aim to fine-tune a small LM  $f(\boldsymbol{x};\theta^{*}):\mathcal{X}\rightarrow \mathcal{Y}$  with pre-trained parameters  $\theta^{*}$  on a downstream task by annotating samples from an unannotated data pool  $\mathcal{U} = \{x_i\}_{i = 1}^U$  to constitute the annotated sample set  $\mathcal{A}(|\mathcal{A}|\leq \mathcal{B}$  and initially  $\mathcal{A} = \varnothing$  ) such that its performance is maximized. Note that in our multi-fidelity setting, the annotated set contains a human-annotated subset  $\mathcal{A}_H$  and an LLM-annotated subset  $\mathcal{A}_G$ , so  $\mathcal{A} = \mathcal{A}_H\cup \mathcal{A}_G$ . Similarly, the total annotation budget is composed of human annotation budget  $\mathcal{B}_H$  and LLM annotation budget  $\mathcal{B}_G$  ( $\mathcal{B}_H$  is typically much smaller than  $\mathcal{B}_G$ ), i.e.,  $\mathcal{B} = \mathcal{B}_H + \mathcal{B}_G$ .
To solve for the best annotation strategy to maximize annotation and computation efficiency, we pose the annotation acquisition process as a multi-fidelity learning problem with interactions allowed for  $R$  rounds. In the  $r$ -th round  $(1 \leq r \leq R)$ , we query a set of instances  $\mathcal{Q}^r$  and annotate acquired samples  $\mathcal{A}^r$  from the unannotated pool to add annotation, i.e.,  $\mathcal{U} = \mathcal{U} \setminus \mathcal{A}^r$  and fine-tune the target model  $f$  on  $\mathcal{A}^r$  to update  $\theta^{(r)}$ . The goal is to minimize the empirical risk  $\mathcal{R}(f)$  of the final LM  $f(\boldsymbol{x};\theta^{(R)})$  on the downstream task, subject to preset annotation budget and computational cost constraints.
2.2 Multi-fidelity Learning Framework
Initialization. We initialize the multi-fidelity learning loop by randomly selecting a small set of samples  $\mathcal{A}_H^0$  from the unannotated set  $\mathcal{U}$  to be annotated by human annotators. The pre-trained LM with parameters  $\theta^{*}$  is then tuned on the initial annotated dataset:
$$
\theta^ {(0)} = \arg \min  _ {\theta^ {*}} \frac {1}{| \mathcal {A} _ {H} ^ {0} |} \sum_ {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \in \mathcal {A} _ {H} ^ {0}} \mathcal {L} \left(f \left(\boldsymbol {x} _ {i}; \theta^ {*}\right), y _ {i}\right), \quad i = 1, \dots , n _ {s} \tag {1}
$$
where  $\mathcal{L}$  is the loss function, e.g., cross-entropy for classification, and  $n_s$  is the annotation size. This enables the uncertainty score of the target LM to be initially updated on domain-specific data, which helps to mitigate the cold-start issues [31, 49, 50].
Interactive fine-tuning. After model initialization, we begin query samples from the unannotated pool  $\mathcal{U}^0 = \mathcal{U}\setminus \mathcal{A}_H^0$  for either human or LLM annotation. Existing methods [54] often consider the entire unannotated pool during sampling. These approaches scale poorly to large unlabeled datasets as acquiring informative samples usually involves making inferences or executing clustering which can be time-consuming if these operations were to be computed over all data samples. Thus, for any
interaction round  $r$ , we propose to randomly sub-sample from  $\mathcal{U}^r$  to obtain a smaller candidate set  $\mathcal{U}_s^r$  where the acquisition strategy can be efficiently computed.
In  $r$ -th round of interactive fine-tuning, we first perform the exploration-exploitation query (EEQ) strategy  $S$  (described in detail in Section 2.3) to determine the human annotation set  $\mathcal{A}_H^r$  and LLM annotation set  $\mathcal{A}_G^r$  from the sub-sampled unannotated pool  $\mathcal{U}_s^r$ . Then the interactive multi-fidelity learning can be solved by minimizing the following total loss objective:
$$
\mathcal {L} _ {\text {t o t a l}} = \frac {1}{| \mathcal {A} _ {H} ^ {r} |} \sum_ {\left(\boldsymbol {x} _ {i}, y _ {i}\right) \in \mathcal {A} _ {H} ^ {r}} \mathcal {L} \left(f \left(\boldsymbol {x} _ {i}; \theta^ {(r)}\right), y _ {i}\right) + \frac {1}{| \mathcal {A} _ {G} ^ {r} |} \sum_ {\left(\boldsymbol {x} _ {j}, y _ {j}\right) \in \mathcal {A} _ {G} ^ {r}} \mathcal {L} \left(f \left(\boldsymbol {x} _ {j}; \theta^ {(r)}\right), y _ {j}\right) \tag {2}
$$
Unlike the existing approaches that use simultaneous annotation with equal batch sizes for each round, we emphasize the importance of annotation order (human first and then LLM) and variable batch sizes for each query step (verified in Section 4.2) and identify the following two key designs that improve query efficiency and annotation effectiveness:
Design 1 - In-context learning with similarity-based prompt retrieval. According to the annotation budget  $\mathcal{B}_H^r$  and  $\mathcal{B}_G^r$ , we acquire  $\mathcal{Q}_H^r$  and  $\mathcal{Q}_G^r$  instances for human and LLM annotators respectively. We first annotate acquired samples  $\mathcal{Q}_H^r$  by humans, obtain  $\mathcal{A}_H^r$ , and update the human-annotated set  $\mathcal{A}_H = \mathcal{A}_H \cup \mathcal{A}_H^r$ . When using LLM to automatically generate annotations for new data, we then retrieve a few examples from the current human-annotated set  $\mathcal{A}_H$  as in-context examples for improving the predicted annotation quality, see Fig. [2]. Leveraging recent advances in prompt retrieval [25], we compute embeddings from all annotated samples using Sentence-BERT [34] and find the most similar examples for each queried instance measured by cosine similarity. This design improves in-context learning by better utilizing human supervision which empirically helps to further improve the accuracy and robustness of LLM annotations (verified in Section [4.2]). More implementation details are provided in Appendix A.
Design 2 - Variable batch-size query. We propose a variable batch-size query strategy that puts more human budgets towards the initial steps of the learning process to annotate the most uncertain instances and gradually decrease the batch sizes until the total budget is reached, as illustrated in Fig. Naturally, another benefit of this design is that by acquiring more human-annotated examples in the early stage, we can have access to a larger pool of high-fidelity samples for conducting similarity-based prompt retrieval, which further improves the in-context learning performance and stabilizes the LLM annotations. Inspired by infinite geometric series, we design a budget decay scheme and thus set the human annotation budget for the  $r$ -th round to be  $\mathcal{B}_H^r = \mathcal{B}_H / 2^r$  and iterate until the total budget is reached, i.e.
$$
\frac {\mathcal {B} _ {H}}{2 ^ {1}} + \frac {\mathcal {B} _ {H}}{2 ^ {2}} + \frac {\mathcal {B} _ {H}}{2 ^ {3}} + \frac {\mathcal {B} _ {H}}{2 ^ {4}} + \dots + \frac {\mathcal {B} _ {H}}{2 ^ {r}} = \sum_ {r = 1} ^ {R} \left(\frac {1}{2}\right) ^ {r} \mathcal {B} _ {H} \rightarrow \mathcal {B} _ {H}. \tag {3}
$$
Note that the residual budget after  $R$  rounds will be jointly applied to the last round.
Leveraging the benefits of novel designs, we efficiently acquired larger high-quality data  $\mathcal{A}_G^r$  annotated by LLMs (e.g., GPT-3.5). The next step is to update the annotated sample set in the  $r$ -th round  $\mathcal{A}^r = \mathcal{A}_H^r \cup \mathcal{A}_G^r$  and unannotated data pool  $\mathcal{U} = \mathcal{U} \setminus \mathcal{A}^r$ . Then we fine-tune the target model  $f$  using the annotated sample set  $(x_i, y_i) \in \mathcal{A}^r$  and update the model parameters  $\theta^{(r)}$ .
Termination. The multi-fidelity learning process is stopped if either of the two stopping criteria is satisfied: (1) Annotation budget  $\mathcal{B}$ : if the annotation budget after  $R$  rounds is greater than the total budget limit, i.e.,  $\mathcal{B}_H + \mathcal{B}_G \geq \mathcal{B}$ , we terminate the interactive process. (2) Computational cost  $\mathcal{C}$ : Compared with inference and query calculation cost, the computation cost of each fine-tuning round  $\mathcal{C}_{ft}$  is typically much more expensive and we thus stop the fine-tuning process if  $R \times \mathcal{C}_{ft} \geq \mathcal{C}$ . Finally, we return the fine-tuned target LM  $f(\boldsymbol{x}; \theta^{(r)})$  and annotated sample set  $\mathcal{A}$ . Algorithm 1 illustrates the step-by-step workflow of our IMFL framework.
2.3 Exploration-Exploitation Query Strategy
Based on the multi-fidelity learning framework, we introduce an innovative query strategy. This approach harnesses human annotation for exploitation by maximizing informativeness through
Algorithm 1 IMFL framework
1: Require: unannotated data pool  $\mathcal{U}$ , target LM model  $f$ , query strategy  $\mathcal{S}$ , annotation budget  $\mathcal{B}$
2: Initialization:  $\mathcal{A} = \varnothing$ ,  $\theta = \theta^{(0)}$  on  $\mathcal{A}_H^0$
2: for rounds  $r = 1, \dots, R$  do
3:  $\mathcal{U}_s^r\gets$  Extract from  $\mathcal{U}$  by random sub-sampling
4:  $\left[\mathcal{Q}_H^r,\mathcal{Q}_G^r\right]\gets$  Acquire  $[\mathcal{B}_H^r,\mathcal{B}_G^r ]$  samples by query function  $s$  on model  $f$  ,data  $\mathcal{U}_s^r$
5:  $\mathcal{A}_H^r\gets$  Annotate acquired samples  $\mathcal{Q}_H^r$  by human
6:  $\mathcal{A}_H = \mathcal{A}_H\cup \mathcal{A}_H^r$
7: Execute prompt retrieval from  $\mathcal{A}_H$
8:  $\mathcal{A}_G^r\gets$  Annotate acquired samples  $\mathcal{Q}_G^r$  by LLMs
9:  $\mathcal{A}^r = \mathcal{A}_H^r\cup \mathcal{A}_G^r$
10:  $\mathcal{U} = \mathcal{U}\backslash \mathcal{A}^r$
11:  $f(\pmb{x}_i;\theta^{(r)})\gets$  Fine-tune  $f(\pmb {x}_i;\theta^{(r)})$  on  $\mathcal{A}^r$
12: return  $f(\pmb{x}; \theta^{(r)})$ ,  $\mathcal{A}$
Figure 2: Illustration of interactive fine-tuning. Human annotations are first executed and the resulting annotated data are iteratively merged into the human annotation tool, which provides rich examples for prompt retrieval when calling LLM for annotation. The batch size for human annotation varies and gradually decreases as the round progresses.
uncertainty sampling, and LLM annotation for exploration by enhancing representativeness through diversity sampling. The core idea is a two-stage selection: executing 1) diversity sampling, e.g., selecting cluster centers to reduce intra-iteration redundancy, and 2) uncertainty sampling, e.g., selecting instances with the least confidence, to avoid inter-iteration redundancy.
Fig. 3 presents the key components and steps of the EEQ strategy. Specifically, we apply  $k$ -means cluster algorithm to embeddings of the sub-sampled unannotated data  $\mathcal{U}_s^r$ . Based on the annotation budget, we set  $k = \mathcal{B}_H / 2^r + \mathcal{B}_G / R$  as the clustering parameters and identify the cluster centers (or samples closest to the cluster center) as samples, thus enforcing diversity exploration. We then calculate the uncertainty score for all selected samples and rank them from high to low. The top  $\mathcal{B}_H / 2^r$  uncertain samples are assigned to the human annotator following the least confidence strategy:
$$
\boldsymbol {x} _ {i} ^ {*} = \underset {\boldsymbol {x} _ {i}} {\arg \max } \left[ 1 - p \left(f \left(\boldsymbol {x} _ {i}; \theta^ {(r)}\right) \mid \boldsymbol {x} _ {i}; \theta^ {(r)}\right) \right], \tag {4}
$$
which has shown to be simple and effective in a variety of settings, resulting in enforcing uncertainty exploitation [28, 42]. As discussed in Section 2.2 we then update the human-annotated pool  $\mathcal{A}_H$  which enables us to retrieve a few examples as in-context examples for the LLM annotator which can annotate  $\mathcal{B}_G / R$  samples with better quality and stability. More detailed discussions about the query strategy are presented in Appendix B
Figure 3: Illustration of exploration-exploitation query strategy with core components and steps.
We empirically validate the effectiveness of the proposed interactive multi-fidelity learning framework on four diverse datasets, spanning two important real-world application domains, namely, finance and medicine. A summary of the four datasets is provided in Table 2. For the FPB, Headline, and MedQA datasets, we use the publicly available test data for evaluation. As for the PubMedQA dataset, we follow prior work [18] and use the dev./valid data for evaluation. We evaluate the methods by (average) F-1 score for financial datasets following the same setting used by BloombergGPT [46] and accuracy for medical datasets, as is used in prior work [38, 32]. Interested readers can find more details about the datasets in Appendix D.1
Table 2: Summary of the four domain-specific datasets used in our experiments.
Domain	Name	Task	Size (train/test)	Metric
Financial	FPB [29]	Sentiment Analysis	3876/969	F-1 score
Financial	Headline [40]	News Classification	9130/2282	Average F-1 score
Medical	PubMedQA [18]	Biomedical QA	500/500	Accuracy
Medical	MedQA [17]	Medical knowledge QA	11450/1273	Accuracy
3.2 Experiments Setup
Fine-tuning. We adopt Dolly 2.0, the first open-source, instruction-following LLM, as the target LM for fine-tuning. It is based on the EleutherAI Pythia [3] model family, which is a suite of decoder-only auto-regressive language models ranging from 70M to 12B parameters. Limited by our computational budget, we choose to use dolly-v2-3b as the pre-trained LM for our main results. We also provide additional results using larger LMs (e.g., dolly-v2-7b and dolly-v2-12b) in Appendix C.2 to show the impact of pre-trained LM size on the final performance. For efficiency, we leverage low-rank adaption techniques (LoRA) [13] to optimize the fine-tuning process for reducing memory and time cost. We execute all experiments on a GPU node with 8 NVIDIA V100 32G cores. More experiment setup details, e.g., hyperparameters, can be found in Appendix D.2
Query and annotation. In the query step, we remove all labels in the training data to create a pool of unannotated data. These original ground truth labels are treated as the high-fidelity annotations provided by Human Annotator and are only accessed at the cost of budget consumption. For the low-fidelity annotation, we employ GPT-3.5-turbo as the LLM Annotator to automatically generate annotation for unannotated data. We note that, in reality, even collecting a large set of unannotated samples can oftentimes be non-trivial. As such, in our experiments, we limited our unannotated data pool to only contain 3000 data samples (randomly sampled from the original training dataset), from which we perform our query strategy. Each experiment is repeated three times and the mean is reported as the final result to reduce noise.
**Annotation and computational budget.** Unless mentioned otherwise, we assume a total annotation budget of 1000 for all datasets (see more discussions about the budget setting in Appendix C.4). As human annotation is far more expensive than using LLM (i.e., GPT-3.5) to generate annotation, we set the human annotation budget of IMFL to be 200 samples  $(20\%)$  and the GPT-3.5 annotation budget to be 800 samples  $(80\%)$ . In Appendix C.5 we provide a discussion about the trade-off between annotation accuracy and cost expenses. Regarding the fine-tuning cost, we set the total number of interaction rounds for fine-tuning to be  $R = 5$  to reflect the computational budgets. It is worth noting that the performance can be further improved if more rounds (i.e., a higher budget) are allowed.
3.3 Main results