Distill or Annotate? Cost-Efficient Fine-Tuning of Compact Models
Year: 2023

Abstract:
Fine-tuning large models is highly effective, however, inference can be expensive and produces carbon emissions. Knowledge distillation has been shown to be a practical solution to reduce inference costs, but the distillation process itself requires significant computational resources. Rather than buying or renting GPUs to fine-tune, then distill a large model, an NLP practitioner might instead choose to allocate the available budget to hire annotators and manually label additional fine-tuning data. In this paper, we investigate how to most efficiently use a fixed budget to build a compact model. Through extensive experiments on six diverse tasks, we show that distilling from T5-XXL (11B) to T5-Small (60M) is almost always a cost-efficient strategy compared to annotating more data to directly train a compact model (T5-Small). We further investigate how the optimal budget allocated towards computation varies across scenarios. We will make our code, datasets, annotation cost estimates, and baseline models available as a benchmark to support further work on cost-efficient training of compact models.

heading: 1 Introduction
text: Increasing the size of pre-trained models can consistently improve performance on downstream tasks after fine-tuning, as seen in studies based on BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), BART (Lewis et al., 2019), T5 (Raffel et al., 2020), and the work on empirical scaling laws (Brown et al., 2020; Lester et al., 2021; Hernandez et al., 2021). However, using large models for inference is expensive and contributes to carbon emissions (Patterson et al., 2021). To address this, researchers have explored methods to compress large models through techniques such as knowledge distillation (Hinton et al., 2015; Sanh et al., 2019; Gou et al., 2021), which is effective in reducing inference costs (Magister et al., 2022)
and improving the generalization of smaller student models (Stanton et al., 2021). Nonetheless, the distillation process itself still requires signi - cant computational, memory, and storage resources (Xia et al., 2022).
In addition to compressing models, an alternative approach to improve performance without increasing inference costs is to simply label additional data for ne-tuning. Recent work has shown that a few hundred extra labels can sometimes lead to better performance than billions of additional model parameters (Kirstain et al., 2022). This raises the question of how to most ef ciently use a xed budget to train a compact model which supports efcient inference while maximizing performance. One option is to use an available budget to hire annotators to label additional data and directly netune a small model. Alternatively, the budget could be used to purchase or rent GPUs to ne-tune and distill a large teacher model (see Figure 1).
In this paper, we use the theory of consumer choice (Becker, 1965; Lancaster, 1966; Bai et al., 2021) to investigate the question of when distillation is a cost-ef cient strategy for model compression. Based on extensive empirical analysis,
11100
we provide recommendations on how to allocate a fixed budget for human annotation and computing resources to train a compact model. Our experiments across six NLP tasks reveal that distillation with unlabeled data is almost always a cost-efficient strategy for improving the performance of compact models when compared to annotation (see Table 2). Furthermore, our analysis shows that the optimal allocation of budget towards distillation increases as more labeled data becomes available (see §4.1 and Figure 2). For smaller budgets, it is Pareto optimal (Abdolrashidi et al., 2021; Treviso et al., 2022) to use smaller amounts of unlabeled data for distillation, while increasing the amount of labeled data, as this leads to a more knowledgeable teacher. As the budget increases, it becomes economical to distill using larger unlabeled datasets, because the teacher model outperforms the student by a significant margin. Finally, we investigate the cost efficiency of data annotation with GPT-3.5 (Ouyang et al., 2022) (Figure 6). We find that, although GPT-3.5 is cheaper than human annotators, finetuning T5-XXL and then distilling a small model is more cost-efficient than directly fine-tuning the small model with pseudo-labels from GPT-3.5.
We will make our code, datasets, annotation cost estimates, and baseline models available as a benchmark to support further work on cost-efficient training of compact models.

heading: 2 Study Design
text: In this section, we first describe how we formulate the problem for the cost-efficiency study (§2.1). We then compare two strategies (§2.2 & 2.3) for building compact models that incur different proportions of computational and human annotation costs. Finally, we explain how to estimate the annotation cost (§2.4) and computational cost (§2.5) involved in the two strategies.

heading: 2.1 Problem Formulation and Assumptions
text: The main focus of this study is to fairly evaluate the two approaches (§2.2 & §2.3) under a fixed budget. When financial constraints are in place, practitioners may be faced with weighing options of allocating money towards data or compute; we empirically investigate their trade-offs to maximize the resulting utility. To enable extensive studies, we simulate the process of labeling data using a variety of existing crowdsourced datasets, and the cloud GPU rentals that charge per hour of use.
We assume the NLP engineer’s salary is a fixed cost, so their time spent building models and/or managing a group of annotators to label data are not a factor in determining the total cost. The only costs considered are the direct costs for human data labeling and GPU computation. No task-specific labeled data is initially assumed to be available for free, but we do assume that pre-trained models such as T5 (Raffel et al., 2020), which are publicly available, have zero cost.

heading: 2.2 Strategy 1: Building a Compact Model
text: Directly with Annotations (Ann.)
This strategy directly fine-tunes a compact model (e.g., T5-Small (60M)), allocating the entire budget towards human annotation. This is considered the most straightforward approach practitioners would choose to train a compact model.
In particular, given a budget constraint, we prepare data that can be maximally annotated using the budget, and we train T5 (Raffel et al., 2020) on the data under a unified text-to-text framework for all tasks (Table 1), maximizing the likelihood of a target text Y given an input text X . The format for an input X and the corresponding target Y for each task is detailed in Appendix B.
Note that the most dominant cost associated with this strategy is the annotation cost. While the total cost of building this direct model can include the fine-tuning cost (i.e., computational cost), we
found it negligible in most cases and thus omitted it, unless otherwise noted, for the sake of simplicity.1

heading: 2.3 Strategy 2: Distilling from a Larger
text: Model (Dist.)
As an alternative to annotating more data, one could allocate part of the budget towards computation to train a larger (e.g., T5-XXL (11B)) model on a smaller amount of data. The large model can then be distilled to produce a final compact model that also supports efficient inference.
Following recent work (Xia et al., 2022; Zhou et al., 2022b), our study mostly focuses on taskspecific model compression rather than general distillation (Sanh et al., 2019),2 however we provide analysis of general vs. task-specific distillation in Appendix F. General distillation requires significant computational resources; also task-specific and general distillation can be used together in a complementary fashion (Jiao et al., 2020).
Notably, even for Strategy 2, annotated data is needed to train the large teacher model. Therefore, we assume to have a certain number (N ) of data initially annotated by spending some part of the budget, and fine-tune the larger model using this data in the same way as in §2.2. After that, a small model (i.e., student) is trained by distilling the larger model’s (i.e., teacher) knowledge (Hinton et al., 2015), in which the teacher’s probability distributions over a target sequence given a source input are used as soft labels. We adopt KL divergence loss, which compares two distributions, to make the student’s distribution PS follow the teacher’s output distribution PT with respect to task-specific unlabeled data3:
DKL(PT ||PS) = ∑
v∈V PT (v) log
PT (v) PS(v) (1)
where V is vocabulary space. Input and target tokens that are conditioned to produce probabilities are omitted above for brevity.
The total cost includes both the initial cost for N (the number of initially annotated training examples) and the computational cost for fine-tuning
1Fine-tuning T5-Small (60M) on 5K data, for example, takes less than half an hour, which costs approximately $1, based on the computational cost in §2.5.
2In general distillation, a pre-trained model is distilled before fine-tuning, such as DistillBERT.
3For example, source sentences without target paraphrased sentences for a paraphrase generation task. Refer to Appendix D for details of the unlabeled data.
a large model and then distilling it into a compact model.

heading: 2.4 Cost Estimation for Data Annotation
text: This study considers six diverse and practical NLP tasks, shown in Table 1. We estimate the annotation cost for each dataset based on mentions in the corresponding literature if available, correspondence with creators of the dataset, or prices of the Data Labeling Service from Google Cloud, following Wang et al. (2021)4. Detailed descriptions of our cost estimates for each dataset are provided in Appendix A.

heading: 2.5 Estimation of Computational Cost
text: This work assumes that computing resources are rented from Google Cloud for model training. We specifically consider NVIDIA A100 GPUs, each equipped with 40GB of VRAM, to fit a large model (e.g., 11B parameters) into them. The price of this, which includes a virtual machine and storage, is set at about $3.75 per 1 GPU hour. For extensive studies, we exploit our own resources, A40 GPUs that have been shown to be approximately 2x slower than A100 through benchmark results5 as well as our preliminary experiment that compares the training time. As a result, we estimate the computational cost as $1.875 per 1 GPU hour. This is a realistic price that practitioners would need to pay, unlike theoretical measures such as FLOPs, which do not reflect the real runtime (Xu and McAuley, 2022) and costs. An example breakdown of cost estimates for building compact models is provided in Appendix (Table 6).

heading: 3 Evaluating Annotation and Distillation under a Fixed Budget
text: In Table 2, we evaluate the two strategies under varying budgets for six different tasks. We first set N , the number of starting data annotated by spending an initial $. Given a fixed budget, we then either annotate more data for the annotation (Ann.) strategy, or use more GPU hours along with more unlabeled data for the distillation (Dist.) strategy.
We consider T5-Small (60M) as a compact model and T5-XXL (11B) as a teacher model for our main study. All models are fine-tuned based
4https://cloud.google.com/ai-platform/ data-labeling/pricing#labeling_costs
5https://lambdalabs.com/blog/ nvidia-rtx-a40-benchmarks
[performance] ⇒ student distillation (Dist.) on various NLP tasks. N indicates the number of starting data annotated with the corresponding (initial $). ( #Additional Data ) refers to the number of annotated data additional to N , and ( GPU Hours ) denotes the total GPU hours for fine-tuning the teacher model on N data, plus for the distillation into a small model using varied ( #Unlabeled Data ). N/A is used when it is not feasible to build a model given the cost.
on T5 v1.1 (Roberts et al., 2020), which was pretrained in an unsupervised way only, unlike the original T5 (Raffel et al., 2020).
In the case of FEVER and NATURAL QUESTIONS, following Lee et al. (2020) and Roberts et al. (2020) respectively, we consider a closed-book setting where models should rely solely on its parametric knowledge, and report performances on dev sets as test sets are private. To measure performances, we use accuracy for FEVER and MULTIPITId, F1 for WLP, STANCEOSAURUS, and NATURAL QUESTIONS, and BERT-iBLEU (Niu et al., 2021) (i.e., the harmonic mean of self-BLEU and BERTSCORE (Zhang et al., 2020)) for MULTIPITGen. More details about experimental settings are described in Appendix C.

heading: 3.1 Annotation vs. Distillation
text: In Table 2, we observe that interestingly, the distillation (Dist.) strategy significantly out-
performs the annotation (Ann). strategy across almost all cases for all tasks. While knowledge distillation (Hinton et al., 2015) has been proven effective for compression/generalization in previous works (Sanh et al., 2019; Kang et al., 2020; Le et al., 2022), our result that takes into account the realistic costs involved in building models is quite surprising, which highlights a new aspect: it is economically efficient. In other words, this suggests that exclusive reliance on scaling data by hiring human annotators might not be a good practice in light of cost efficiency.
Note that Dist. needs to be first fine-tuned on N labeled data that requires a considerable com-
putational cost, so if the fine-tuning cost exceeds the given budget, we denote such cases as N/A. In such scenarios, Ann. is essentially the right choice. We also notice some scenarios where Ann. is a better option with limited budgets. For example, Ann. defeats its counterpart with $100 for WLP
( N=5K ) and MULTIPITGen ( N=10K ). In these cases, the #unlabeled data used for distillation are highly limited ( 7K and 10K , respectively) as fine-tuning costs make up a substantial portion of limited budgets.

heading: 3.2 Does Distillation Work Better Simply by Making Use of Unlabeled Data?
text: In Table 2, we observe a substantial performance gap between Ann. and Dist. One notable point is that there is a big difference in the absolute number of data ( #labeled data and #unlabeled data ) used for each strategy given a fixed budget. In Table 2, for instance in WLP, given $500, 1923 more data can be annotated for Ann., whereas 111K unlabeled data can be leveraged for Dist. This not only means that annotated data is expensive, but also raises a question: is the performance gap simply because of the difference in the number of data points? To investigate this question by building a fair ground in terms of the size of data, we take a self-distillation (Self-Dist.) approach (Zhang et al., 2019) in which the architecture of a teacher and a student is the same (i.e., T5-Small).
In Table 3, we compare Dist. against Self-Dist. using the same 100K unlabeled data. We see that Self-Dist. is worse than the Dist. across all tasks by remarkable margins even though the same number of data is used. In fact, the performance of Self-Dist. is found to be bounded by its teacher (i.e., T5-Small (Ann.)), as also observed in (Zhou et al., 2022a). This analysis suggests that the performance gap between Dist. and Ann. can indeed be attributed to exploiting the large pre-trained language model’s capability, not simply making use of more data.

heading: 3.3 Comparison under Larger Budgets
text: Our experiments suggest that distillation (Dist.) is a more economical choice than relying completely on the human annotation to train a compact model, at least within scenarios presented in Table 2. However, this raises a question: could Ann. reach the performance of Dist. when investing a much larger budget? Table 4 shows the results of Dist. with budgets for 100K unlabeled data, and Ann. with much larger budgets (or upper bound by using all available #labeled data ). Interestingly, in some cases (STANCEOSAURUS & MULTIPITGen), Dist. turns out to be an astoundingly economically efficient way to train a compact model. Even though all existing annotated data ( ∼50K ) are used for MULTIPITGen training (w/ $14,469), it never outperforms Dist. (w/ only $245). For other tasks except for the aforementioned ones, we notice that Ann. can outperform Dist. with much larger budgets (e.g., $12,899 for FEVER). In practice, however, we still find that Ann. can be much more costly (e.g. 10x in the case of FEVER) to obtain similar performance.

heading: 4 Further Analyses
text: In this section, we study varied values of each variable: the initial number (N) of annotated data (§4.1), the compact model size (§4.2), and the teacher model size (§4.3), all of which are fixed in the main experiment (§3.1).

heading: 4.1 Pareto Curves
text: In Figure 2, we explore different combinations of #labeled data (L={0.1K, 0.5K, 1K, 5K, 10K}) and #unlabeled data (U={ 0 , 10K , 100K }). Note that U=0 indicates the annotation (Ann.) strategy in essence. We plot the performances of each combination and approximate the Pareto frontier
#unlabeled data (U={ 0 , 10K , 100K }). U=0 denotes the annotation (Ann.) strategy. The Pareto frontier ( ) is the set of optimal solutions that practitioners would choose from, and is approximated by interpolating the given data points. The X-axis is on a logarithmic scale.
(Abdolrashidi et al., 2021; Treviso et al., 2022) by interpolating the given data points. For all tasks, we observe that the distillation (Dist.) strategy is almost always Pareto optimal.6 In Appendix (Table 11), we also look at the low resource setting in detail.
Furthermore, we observe that using a smaller amount of unlabeled data ( U=10K ) is Pareto optimal for smaller budgets, while larger unlabeled data ( U=100K ) maximizes utility as the budget increases. This implies that in low-budget settings, the teacher’s capacity is limited, allowing the student to catch up quickly. However, once the teacher outperforms the student by a significant margin, it is more economical to allocate a larger part of the budget towards distillation.
In Figure 3, we provide an additional analysis by varying the number of initially annotated data (N) under fixed budgets to look at the impact of
6One exception is (L=0.1K, U=0) where a budget is so limited that leveraging a large model is barely feasible.
(a) $100
40
50
60
70
100 1000 10000
B E R TiB LE U
N
(b) $200 MULTIPITGen
40
50
60
70
100 1000 10000
B E R TiB LE U
N
40
50
60
70
100 1000 10000
B E R TiB LE U
N
(c) $300
(a) $100
35
45
55
100 1000 10000
F 1
N
(b) $200 (c) $300
STANCEOSAURUS
35
45
55
100 1000 10000
F 1
N
35
45
55
100 1000 10000
F 1
N
(a) $100
20
40
60
100 1000 10000
F 1
N
(b) $300 (c) $500 WLP
20
40
60
100 1000 10000
F 1
N
20
40
60
100 1000 10000
F 1
N
Figure 3: Results according to different number of starting annotated data (N ) under fixed additional budgets.
N. Expectedly, we notice that Dist. outperforms Ann. in general except for some cases with low N,
especially for MULTIPITGen as also evidenced in Appendix (Table 11). It is worth noting that there is a common trend across all tasks that the Dist. performances drop with high N. This is due to the limited budgets; high N requires a substantial finetuning cost for a large model, hence the budget to be used for distillation is limited. For instance, in the case of STANCEOSAURUS with budget=$200, if N is 1K , 82K unlabeled data can be used for distillation, whereas only 35K unlabeled data are used when N= 10K , resulting in the former outperforming the latter. This offers a lesson that unconditionally pursuing larger N is not desirable in a fixed budget scenario; it is advisable for practitioners to understand and consider the trade-off between the fine-tuning and distillation costs.

heading: 4.2 Varying the Compact Model Size
text: To consider various inference scenarios, we explore different sizes of a compact model in Figure 4. In general, the performances of all models improve as the budget increases, and Dist. outperforms Ann. given the same cost except for the low budget ( N=0.1K ) setting. Interestingly, we observe that T5-XXL⇒ T5-Base (Dist.) is better than T5-XXL ⇒ T5-Large (Dist.) in some cases ($1600 for WLP, $671 and $4010 for MULTIPITGen) although the former is smaller and more efficient. We conjecture that this is attributed to the model’s larger number of parameters that require more GPUs and thereby more cost. This result disproves the prevailing belief that larger models are always superior, at least in fixed-budget scenarios.

heading: 4.3 Varying the Teacher Model Size
text: We now investigate teacher models with different scales (Figure 5). It turns out that relatively smaller teacher models (T5-Large & T5-XL) cannot be good teachers in the low budgets scenarios. For instance, with $521 for MULTIPITGen, T5-Large ⇒ T5-Small (Dist.) and T5-XL ⇒ T5-Small (Dist.) underperform T5-Small (Ann.), whereas T5-XXL ⇒ T5-Small (Dist.) outperforms T5-Small (Ann.). In higher budget settings, it is noticeable that the largest teacher (XXL) is similar to or better than the smaller teacher (Large, XL). Taken together, this analysis suggests that when adopting distillation, the scale of the teacher model matters, and it may be safe to leverage sufficiently a larger model as a teacher regardless of any budgetary scenarios.

heading: 5 GPT-3.5 as an Annotator
text: Furthermore, we examine the cost efficiency of GPT-3.5 (Ouyang et al., 2022) annotation through an in-context few-shot learning scheme. Wang et al. (2021) has recently demonstrated that GPT-3 (Brown et al., 2020) can be used as a cheaper labeler compared to humans. We attempt to scrutinize its applicability to the tasks considered in this work, and also contextualize its result with that of Dist. ultimately. We make use of the text-davinci-003 model to generate pseudolabels by prompting with 32 training examples. In this experiment, we assign $200 each for WLP and STANCEOSAURUS for GPT-3.5 annotation. Note that OpenAI7 charges money based on the number of tokens used. The cost per label for WLP
7https://openai.com/api/pricing
is $0.046 and for STANCEOSAURUS is $0.073, if using GPT-3.5 (details in Appendix E).
In Figure 6, we compare GPT-3.5 annotation (GPT-3.5 Ann.) against the human annotation and distillation strategy. In addition to GPT-3.5 Ann., we combine it with human annotation (Human + GPT-3.5 Ann.) to enhance quality and make a comparison with Dist. The results clearly show that while GPT-3.5 could be better than human annotators as hinted in prior work (Wang et al., 2021), it significantly underperforms the distillation (Dist.) strategy given the same budget despite GPT-3.5’s larger parameters (175B) than the teacher (11B). This once again highlights the different view of knowledge distillation: cost efficiency.

heading: 6 Related Work
text: The costs associated with building models have been explored or concerned by many prior works.
Data Annotation. On one hand, researchers have attempted to tackle the problem of noisy or expensive human annotation. For example, Zhang et al. (2021) studies how to distribute annotation budgets between more examples with a single label and fewer examples with many labels. Chen et al.
(2022) investigates a redundant annotation with a majority vote vs. cleaning or relabeling the incorrect annotations. Wang et al. (2021) compares human annotations against GPT-3 (Brown et al., 2020) annotations. However, these works only focus on the annotation cost.
Knowledge Distillation. On the other hand, other lines of work address computational budgets associated with knowledge distillation. Ye et al. (2022) proposes using a larger and sparser student model than a teacher model to further reduce inference cost. Jooste et al. (2022) compares different distillation schemes for cheap, fast, and environmentally friendly translation models. Ma et al. (2022) explores an efficient interactive distillation with meta-learning. The aforementioned works, however, ignore the data budgets and/or barely consider the realistic computational costs involved in the distillation process. While knowledge distillation has been shown effective for compression or generalization in previous NLP works (Sanh et al., 2019; Kang et al., 2020; Le et al., 2022), it remains unclear whether or not it is efficient even when considering the actual cost of distillation, which is often overlooked. As concurrent works, Sun et al. (2023) presents a novel principle-driven self-alignment approach, and Hsieh et al. (2023) introduces a method that involves step-by-step distillation using chain-of-thought (Wei et al., 2022) rationales. Although the main focus is completely different from ours (i.e., cost), we believe that these works not only enhance this particular area but also have the potential to support our own findings regarding the cost-efficiency of distillation as the new methods would make the gap with annotation even bigger.
Data and Compute. Unlike most existing works that consider exclusively either annotation or computational cost, our study contextualizes the two superficially dissociated types of costs, known to be expensive (Ning et al., 2019; Hong et al., 2020; Hendrycks et al., 2021; Izsak et al., 2021; ObandoCeron and Castro, 2021; Minixhofer et al., 2022) while being obscure in how they can be comparable to each other. Kirstain et al. (2022) compares scaling parameters against adding more labeled examples, but a compact model and a realistic cost ($) are not of interest to it. Our work resembles Bai et al. (2021) in terms of study framework, which explores how to optimally assign pre-training and
annotation costs specifically for domain adaptation settings. Our focus is more on fine-tuning/distilling a compact model rather than pre-training from scratch and on exploring more general scenarios with diverse tasks.

heading: 7 Conclusion
text: In this work, we address a dilemma that practitioners often face when building a model: given a limited budget, how to invest it to train a compact model in an economically efficient manner? We provide empirical evidence that (i) only scaling data using human annotators or GPT-3.5 for annotation may not be the most economical solution, and (ii) when adopting the distillation strategy, using a smaller amount of unlabeled data leads to Pareto efficient models with a smaller budget, while it becomes more beneficial to use larger amounts of unlabeled data as the budget increases. Furthermore, (iii) we demonstrate that in budgetconstrained settings, a smaller final model could produce both better performance and more efficient inference. Given these findings, future work can explore different approaches to leveraging a large model’s capability such as pruning for cost-efficient compact models.
Limitations
This paper fundamentally considers a scenario in which practitioners rent cloud GPUs. In the case of hosting GPUs by themselves, the two strategies explored in this study would not be simply comparable. However, in practice, when training a large model (w/ 8 A100 GPUs), we conjecture that renting GPUs could be preferred in many cases as scaling compute powers is not trivial and prohibitively expensive (Izsak et al., 2021; ObandoCeron and Castro, 2021; Minixhofer et al., 2022). It is also noteworthy that in the future, computational costs may become cheaper as new hardware advances, the pricing policy by cloud platform services changes, and more optimization techniques are applied. On the other hand, human annotation cost is likely to be the same at least or even more expensive. With cost changes in such a direction, the same conclusion made by our study will hold even though the gap between the two strategies will get larger.
For a compression method, our work focuses on knowledge distillation (Hinton et al., 2015). However, it is worth noting that distillation amplifies a
societal bias in a compressed model (Hooker et al., 2020; Silva et al., 2021) due to its limited capacity (Ahn et al., 2022). Accordingly, practitioners are encouraged to additionally leverage bias mitigation techniques (Ahn et al., 2022) when adopting distillation for real-world applications. On top of our finding that the distillation scheme is more costefficient than the data annotation approach, other efficient methods such as pruning (Xia et al., 2022) may be investigated in future work to decide which one is the best efficient solution among methods that leverages a large model. We believe, however, it should be noted that retaining performances after pruning a large portion (e.g., ∼99.995%: 11B ⇒ 60M) for a compact model would not be trivial, evidenced in a prior work (Michel et al., 2019).

heading: Acknowledgments
text: We thank Fan Bai and Jonathan Zheng for their assistance in estimating data annotation costs and collecting unlabeled data for WLP and Stanceosaurus, respectively. This material is based upon work supported by the NSF (IIS-2052498) and by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200004. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Details of Annotation Cost Estimation
text: WLP (Tabassum et al., 2020) This is an annotated corpus containing wet lab protocols, and the included tasks are named entity recognition (NER) and relation extraction (RE). We refer to Bai et al. (2021) for the price per sentence (instance), which is $0.44. Since this price is measured for both tasks, and we are only interested in NER, we take the ratio of the number of labels for each (59.76%:40.24%) for the estimate of NER in isolation, yielding approximately $0.26.
STANCEOSAURUS (Zheng et al., 2022) This dataset includes sourced claims and relevant tweets along with annotated stances for stance classification. Since the labeling cost was not explicitly mentioned in the paper, we asked the authors for the details of the average number of annotations per hour (82 tweets) and the hiring cost ($15 per hour) to calculate the final price per label: $15 ÷ 82 × 2 (double-annotated) = $0.364. MULTIPIT (Dou et al., 2022) This provides Twitter-based paraphrase containing multiple topics. We specifically consider, out of variants, MULTIPITCROWD corpus, consisting of sentence pairs labeled whether each pair is paraphrased or not for paraphrase identification (MULTIPITId). The cost per pair is considered $0.2 as mentioned in the paper. For paraphrase generation (MULTIPITGen), we sample pairs annotated as paraphrased, and take the proportion of sampled ones out of the total (53.9%) to get the cost per paraphrased source-target instance: 100 ÷ 53.9 × $0.2 = $0.371. FEVER (Thorne et al., 2018) & NATURAL QUESTIONS (Kwiatkowski et al., 2019) These are fact verification and question answering datasets respectively for which we estimate the costs by leveraging the price from Google Cloud Platform. This charges $129 per 50 words for 1,000 units, and hence we get an estimate of $0.129 per label for both tasks.
B Input-Output Formats for Each Task
Our study uses T5 (Raffel et al., 2020) as our base model under the standard text-to-text framework. The input-output examples for each task are demonstrated in Table 7, and what follows is detailed explanations for each.
WLP This task can be regarded as a token-level classification problem, where the #class is 20 in
total: {Amount, Reagent, Device, Time, Speed, Action, Mention, Location, Numerical, Method, Temperature, Modifier, Concentration, Size, GenericMeasure, Seal, Measure-Type, Misc, Ph, Unit}. Given a source input (i.e., procedural sentence), the model is required to generate a target as a form of "Entity [Label] Entity [Label] ...".
STANCEOSAURUS For this task, the source is the concatenation of a claim, a relevant tweet, and context information (e.g., reply), and the target is supposed to one of {Supporting | Refuting | Irrelevant | Discussing | Querying}.
FEVER This is a fact verification task where the source is a claim (closed-book setting as discussed in §3), and the target is Supports or Refutes in a 2-way classification setting following Petroni et al. (2021).
MULTIPITId is also a binary classification task where given two sentences, targets should be Yes or No.
MULTIPITGen The source for this task is a sentence and the target is a paraphrased sentence.
NATURAL QUESTIONS As in FEVER, we also consider the closed-book setup that requires a model to rely on its implicit knowledge for this task where the question is a source and the target is directly the answer to the question.

heading: C Detailed Settings and Hyperparameters
text: As described in §3, we utilize T5 v1.1 (Roberts et al., 2020) as a base model, because the original version of T5 (Raffel et al., 2020) was pre-trained using a combination of several supervised tasks as well as an unsupervised task. Since this work assumes that no supervised datasets are available, our fine-tuning strategies build upon T5 v1.1 that was pre-trained in an unsupervised way only. For a
in Figure 1. Starting with ( 5K ) labeled examples, we compare the costs of annotating an additional +1K , or fine-tuning, then distilling T5-XXL (11B parameters). For Distillation (Dist.), the computational cost involves fine-tuning T5-XXL (the teacher) on 5K annotated data, plus distilling it into T5-Small using 100K unlabeled examples.
question answering task, we exceptionally use the checkpoint additionally pre-trained using salient span masking (SSM), an unsupervised pre-training objective known to be helpful for open-domain question answering (Guu et al., 2020), following Roberts et al. (2020).
Table 5 presents the dataset statistics and Table 8 presents the hyperparameters used for training models for each task. We did not try to specifically tune the hyperparameters for each model for each
task, taking into account the scenario considered by this study in which annotated data is highly limited. Moreover, in order to minimize factors other than the ones we consider for each setup, we fixed each parameter as much as possible unless significant problems were observed during training. Specifically, we chose the learning rate of 3e-5 (default in the Huggingface (Wolf et al., 2019) code base for question answering and seq2seq distillation), which we believe is not out of the ordinary, for all except
for NATURAL QUESTIONS where we adopt 1e-3 when training T5-Small model as we observed the phenomenon that it was not being trained at all by looking at its training loss with 3e-5. We trained all models with 50 epochs except for a T5-XXL model where fewer epochs are assumed to be enough. We used the final batch size of 32 by leveraging the gradient accumulation (e.g., batch size of {16, 8} and gradient accumulation of {2, 4}) when necessary to meet VRAM constraints. We adopt (layer-wise) model parallelism that allows us to load a large model on multiple GPUs. Our reported results are based on a single run due to the high computational cost required by our empirical study. Despite this, a significant difference in performance was observed between the two strategies being compared.

heading: D Unlabeled Data for Each Task
text: For the distillation strategy, unlabeled data is essentially required to transfer a large model’s knowledge into a small model. In this work, unlabeled data is literally referred to the data without the corresponding labels (i.e., only source inputs in Table 7). We exploit only input sources (without annotations) in the existing datasets excluding ones that models are evaluated on. Plus, we collect additional unlabeled corpora for each dataset for an extensive study as follows:
WLP This dataset requires procedural text as an input source. We utilize large-scale PROCEDURE corpus (Bai et al., 2021) that contains diverse domains. We specifically use CHEMSYN, chemical synthesis procedures in patents, for this study.
STANCEOSAURUS The input source for this dataset consists of a claim from diverse factchecking sites, a tweet relevant to the claim, and contextual information such as a reply or parent tweet if any. Following the methodology described in this work (Zheng et al., 2022), we collected claims and corresponding tweets by anonymizing user information.
FEVER Statements or claims are sufficient to be sources for this dataset. We leverage the synthetically generated claims in Schuster et al. (2021).
MULTIPIT The sources for this dataset are sentences written by Twitter users, which can be collected by following the method in Dou et al. (2022). For this work, we instead exploit sources
of MULTIPITAUTO (Dou et al., 2022) as unlabeled data, automatically collected recent datasets.
NATURAL QUESTIONS The source simply consists of a question. Therefore, we make use of queries in MS MARCO (Nguyen et al., 2016), where the queries are sampled from Bing’s search logs.

heading: E Details of GPT-3.5 Annotation
text: To annotate pseudo-labels using GPT-3.5, we make use of the strongest version, text-davinci-003 with 32 training examples. Our input prompt consists of a task-specific instruction89 and 32 incontext examples, and unlabeled input to annotate at the end. In order to reduce the high variance (Zhao et al., 2021; Min et al., 2022), we randomly sample and shuffle 32 in-context examples out of a 100 fixed training set for each annotation iteration. In Figure 7, we present the performance of GPT-3.5’s 32-shot learning to see its quality and feasibility, and we find that it can be qualified as a cheap labeler to improve performances, especially for low-budget settings, as found in Wang et al. (2021).
Note that OpenAI10 API charges based on the number of tokens for input prompt plus model output: $0.02 per 1K tokens. Therefore, the $ per label is calculated as $0.046 for WLP (2.3K tokens on average) and $0.073 for STANCEOSAURUS (3.65K tokens on average). Based on this, we annotate 4347 data for WLP and 2739 data for STANCEOSAURUS in total, using $200 assigned for each task.
8For WLP, "Classify named entities into one of the following categories: {Class 1, Class2, ...}"
9For STANCEOSAURUS, "Classify the stance of a given tweet toward a given claim into one of the following categories: {Class 1, Class2, ...}"
10https://openai.com/api/pricing

heading: F Additional Results
text: How well do off-the-shelf models perform for each task? In Table 9, we provide the results of the largest T5 model (11B) fined-tuned on full training data, along with relevant works’ results in resource-rich settings. Those reported numbers can serve as upper bounds or references for calibrating the relative results produced in this work (i.e., resource-limited settings). Note that these should not be used for direct comparison due to various combinations of factors including model architectures, size, approaches, pre-training scheme, training data, and budgets.
What about general distillation? While this work focuses on task-specific distillation, we also provide the result of general distillation (DistilBERT (Sanh et al., 2019)) in which a model is distilled during the pre-training phase to learn general language understanding capability before fine-tuning. To measure the total cost, the computational cost for distillation in the pre-training phase is assumed to be $0 (i.e., it is publicly available). In Table 10, we find that given the same bud-
get, adding general distillation leads to more costefficient than the annotation strategy without distillation. In addition to this, it is important to note that intuitively, general distillation (pre-training) and task-specific (fine-tuning) distillation can be combined for the better, evidenced in Jiao et al. (2020). This further spotlights the cost-efficient aspect of distillation methods.
(Dist.) on various NLP tasks. N indicates the number of starting data annotated with the corresponding (initial $). ( #Additional Data ) refers to the number of annotated data additional to N , and ( GPU Hours ) denotes the total GPU hours for fine-tuning the teacher model on N data, plus for the distillation into a small model using varying ( #Unlabeled Data ).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation section

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitation section

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and introduction sections
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 2
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. We used existing datasets and pre-trained models, following their licenses.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section D
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 2
C 3 Did you run computational experiments? Section 3 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 2 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.1 and Appendix C
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. Appendix C
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Given these findings, future work can explore different approaches to leveraging a large model’s capability such as pruning for cost-efficient compact models. Limitations
This paper fundamentally considers a scenario in which practitioners rent cloud GPUs. In the case of hosting GPUs by themselves, the two strategies explored in this study would not be simply comparable. However, in practice, when training a large model (w/ 8 A100 GPUs), we conjecture that renting GPUs could be preferred in many cases as scaling compute powers is not trivial and prohibitively expensive (Izsak et al. , 2021; ObandoCeron and Castro, 2021; Minixhofer et al. , 2022). It is also noteworthy that in the future, computational costs may become cheaper as new hardware advances, the pricing policy by cloud platform services changes, and more optimization techniques are applied. On the other hand, human annotation cost is likely to be the same at least or even more expensive