Cache & Distil: Optimising API Calls to Large Language Models (2024)

Abstract: Large-scale deployment of generative AI tools often depends on costly API calls to a Large Language Model (LLM) to fulfil user queries, a process that also exposes the request stream to external providers. To curtail the frequency of these calls, one can employ a local smaller language model – a student – which is continuously trained on the responses of the LLM. This student gradually gains proficiency in independently handling an increasing number of user requests, a process we term neural caching. The crucial element in neural caching is a policy that decides which requests should be processed by the student alone and which should be redirected to the LLM, subsequently aiding the student’s learning. In this study, we focus on classification tasks, and we consider a range of classic Active Learning-based selection criteria as the policy. Our experiments suggest that Margin Sampling and Query by Committee bring consistent benefits over other policies and baselines across tasks and budgets.

heading: 1 Introduction
text: Large Language Models (LLMs) offer unique capabilities in understanding and generating human-like text. They have gained widespread use in a wide range of applications, such as assistive tools and entertainment bots. However, large models are often very challenging for all but a few companies and institutions to run on their infrastructure (Schwartz et al., 2020). Meanwhile, smaller models typically under-perform in these applications, at least without additional fine-tuning on task-specific labelled data. Consequently, many applications access LLMs via commercial APIs despite the costs involved and the exposure of their entire request stream to the API providers.
To minimise the costs and data exposure associated with calling the API, we propose to locally train a smaller private language model, which we refer to as student, on the LLM’s predictions and,
as the student gets more accurate, it handles an increasing number of requests. The knowledge of the LLM gets continuously distilled into the smaller model. We refer to this scenario as neural caching (see Figure 1), as the student can be thought of as a smart cache. Note though that the student not only remembers what the LLM predicted but also generalises beyond these examples. The goal of this paper is to formalise the neural caching problem and investigate simple ways of approaching it.
The key element in the neural caching scenario is the policy determining which requests the student processes independently. A good policy should weigh the expected immediate user benefit (i.e., if the LLM is substantially more likely to make a correct prediction than the student) and the anticipated benefit for the student (i.e., whether the LLM’s prediction will aid in training the student). The latter underscores its relationship with Active Learning (AL, Settles, 2009; Zhan et al., 2022), although AL is typically associated with soliciting human annotations. In particular, there is a similarity to online AL (Cacciarelli and Kulahci,
11838
2023), where new unlabelled data points arrive in a stream and are discarded immediately or sent to an annotator. However, online AL tends to focus on maximising the accuracy of the final model (i.e. student in our terminology). In contrast, what matters in neural caching is the accuracy of the joint system (student, teacher, along with the policy) over its lifetime since this online accuracy reflects the average level of service offered to a user.
Despite the aforementioned differences with AL, evaluating the existing AL algorithms – specifically the example selection criteria – remains valuable given the maturity of the AL field and the ease of implementation of some of the AL methods. This study aims to achieve this, as well as to investigate the potential shortcomings of these methods. For instance, will the AL methods end up selecting examples that are too challenging even for the LLM? Would learning from these noisy examples be detrimental to the student? Answering these questions can inform future research on this practically significant scenario.
In this work, our focus is specifically on classification tasks, as opposed to free text generation. Many practical problems, such as routing user requests to relevant departments or answering questions about factual knowledge, can be framed as classification tasks. By confining our focus to classification, we can apply methods developed in AL without modification. This also allows us to circumvent additional challenges tied to the automatic evaluation of text generation (Celikyilmaz et al., 2020).
Our findings reveal the benefits of using ALbased policies such as Margin Sampling (Scheffer et al., 2001) and Query by Committee (Seung et al., 1992). Across datasets and budgets, these methods consistently outperform baselines, such as routing examples randomly or training the student at the very start. Our analysis also reveals that the student appears robust to the noise introduced by an LLM. We also analyse a simplified practical scenario where the student is not retrained and observe even greater improvements in online accuracy from using AL-based policies. We release our code to encourage further work on this problem.1
The key contributions of this work are:
• We formulate the neural caching problem as a powerful extension of using static caches. In neural caching, LLM calls are optimised,
1https://github.com/guillemram97/neural-caching
while the student model is periodically retrained on the labels. We believe online Knowledge Distillation could play a key role in saving calls to expensive models.
• We release a benchmark with LLM annotations for classification tasks to facilitate future research in this setup.2
• We evaluate and analyse different instance selection criteria for the neural caching setup.
• Our findings reveal that AL-based selection criteria consistently improve performance over baseline methods across various budgets and datasets.

heading: 2 Related Work
text: Active Learning. Active Learning (AL) seeks to reduce the amount of manual data annotation needed. To accomplish this, it selects the most informative examples from unannotated data. These datapoints are then presented to an annotator and the labels are subsequently used to train a model. The most common scenario for AL is pool-based, where a large unlabelled dataset is available from the start and then a subset of examples is selected for labelling. There has been extensive work on applying pool-based techniques to NLP tasks, especially for classification problems (Settles, 2009; Zhan et al., 2022; Zhang et al., 2022).
Online Active Learning. In single-pass online AL (Cacciarelli and Kulahci, 2023), access to a large unlabelled dataset is not available. Instead, we are given one unlabelled instance at a time and need to decide at that time whether to request annotation. Online AL was initially motivated by scenarios in which an instance would not be available for annotation at a later time, such as in defect detection or medical applications, where an item might get shipped or the patient becomes unavailable (Riquelme, 2017). Online AL tends to focus on the final accuracy of the model, rather than the online accuracy of the student and teacher combined, the measure more suitable for our scenario.
Knowledge Distillation of LLMs. Knowledge distillation (KD), i.e., training a smaller model to mimic a larger one, has garnered substantial attention (Bucila et al., 2006; Hinton et al., 2015). The
2https://huggingface.co/datasets/guillemram97/cache_llm
class of methods most closely related to ours is active KD, which effectively applies AL to KD (Liang et al., 2021; Xu et al., 2023; Baykal et al., 2023). Similar to AL, the emphasis is placed on the poolbased setting, as opposed to the online setting, with a particular focus on optimising the final accuracy of the student model, rather than online accuracy as needed for our use case.
Optimisation of Commercial LLM API Calls. Due to the high cost of commercial LLM APIs, several works have explored methods to reduce or otherwise optimise the cost of API calls. GPTCache (Bang, 2023) relies on a vector store of past query embeddings and retrieves their associated labels. It shares similarities with the Coreset version of our approach – which emerged as the weakest method in our experiments. FrugalGPT (Chen et al., 2023) implements a cascade of commercial LLMs, where bigger models are only called if the response from a cheaper model is deemed as too unreliable by a scorer that was trained with in-domain data. In contrast, in this work, we do not assume access to gold data to train a scorer. Zhu et al. (2023) present a method to allocate queries among multiple models, together with traditional caching, in a scenario with highly repetitive queries. Šakota et al. (2023); Shnitzer et al. (2023) optimise routing calls through models by predicting their respective performance. Our work deviates from all these as we propose to use continuous KD in a student model.
Concurrent work of Stogiannidis et al. (2023) also presents a calling strategy that leverages KD to reduce API calls to an LLM. Unlike our paper, which offers a systematic analysis of existing AL criteria, their work concentrates on a specific model design. This design resembles a hybrid of our Coreset and Prediction Entropy, which do not perform well in our experiments. Even more significantly, their method’s advantages are only shown in comparison to a scenario where no student model is used. This overlooks trivial baselines of frontloading and random allocation, both of which have shown hard to beat in our experiments. They also use very simple student models (kNN or a nonpretrained MLP versus our smaller pretrained LM), whose relative success may be attributed primarily to the simplicity of the two datasets in their study, which do not necessitate generalisation beyond the teacher model’s predictions.

heading: 3 The Neural Caching Problem
text: The objective of neural caching is to optimise the usage of an LLM in a scenario where labels need to be generated for a stream of inputs. As we get more predictions from the LLM, a student model is trained on them. Our goal is to achieve the highest level of service possible within a set budget of LLM calls; hence, calling the LLM serves both to attain high accuracy for the incoming input as well as to train a student model.
To put it formally, our goal is to establish a mapping between elements in the input space X and the corresponding labels in the space Y . We start with a student model S0, and we can access a teacher model T on demand. Our task is to predict labels for a sequence of n examples (x1, . . . , xn)
iid∼ X . We retrain the student model on the labels obtained from the LLM every f processed requests. This simulates the situation where the number of requests is uniform in time, and there is a set time to retrain the model, e.g. at night. For simplicity and to follow the convention in AL to retrain the model from scratch (Ren et al., 2022), every time we retrain the student model, we reset it to the original pre-trained model and then use parameter-efficient fine-tuning. Although continual learning methods could be employed (Biesialska et al., 2020; Zhou and Cao, 2021), we believe this is largely orthogonal to our primary focus on policies and resetting enhances the reproducibility of our analysis. Importantly, we do not assume access to ground truth (or human annotation) at any point in learning to simulate a fully automatic scenario.
For every new input xi, we use the student model Si/f to obtain the predicted label ŷSi . Then, we have the option to request the label ŷTi from the teacher model (LLM), which incurs a cost of c(xi). Finally, we return the label ŷi for xi: the teacher’s label if requested or the student’s otherwise.
The processing of the n examples is subject to a budget constraint, where the total cost must not exceed a fixed budget b. We assess the effectiveness of our querying strategy based on the accuracy of our predicted label ŷi compared to the actual label yi (online accuracy) on the online examples. Additionally, we measure the accuracy of the final student model Sn/f on a test dataset (final accuracy). Algorithm 1 describes the process.
Algorithm 1: Pseudo-code for the neural caching algorithm with budget b, retraining frequency f , cost per query c, data from the LLM DLLM and an initial student S0 Donline = ∅ for xi in Xonline do
if i mod f == 0 then Si/f = Train(DLLM) end ŷi = Si/f (xi) if Call_LLM(b, xi, ŷi) and b ≥ c(xi)
then ŷi = LLM(xi) b = b − c(xi) DLLM = DLLM ∪ {⟨xi, ŷi⟩}
end Donline = Donline ∪ {⟨xi, ŷi⟩}
end Dtest = {⟨xj , Si/f (xj)⟩ | xj ∈ Xtest} Acconline = Evaluate(Donline) Accfinal = Evaluate(Dtest)

heading: 3.1 Instance Selection Criteria
text: We use classical instance selection criteria from AL for the neural caching problem. We use the term selecting an instance to denote using the LLM to annotate that example.
Front-loading (FR) This simple approach involves using the entire budget initially by selecting all instances for LLM annotation. Once the budget is used up, subsequent requests are handled by the student model alone. As the examples are i.i.d. in our experiments, this strategy has the same expected final accuracy as random selection.
Margin Sampling (MS) MS (Scheffer et al., 2001; Luo et al., 2004) selects examples with high margin between the top two predictions made by the student model
Margin(xi) = log P (yi = k∗1 | xi) − log P (yi = k∗2 | xi)
(1)
where k∗1 and k ∗ 2 are the first and second most likely labels, respectively, according to the distribution P (yi | xi) computed by the student model. This is a popular selection criterion for AL (Roth and Small, 2006; Balcan et al., 2007). Schröder et al. (2022) evaluated different uncertainty-based strategies with Transformer models (Devlin et al., 2019) and found MS to be the best-performing one in
an offline, pool-based setting. To adapt MS – as well as the other criteria – to an online setting as a selection policy, we define a threshold, and only examples with a margin above this threshold are selected until the budget is exhausted. We refer to Appendix A.1 for more details.
Prediction Entropy (PE) In PE (Schohn and Cohn, 2000; Roy and McCallum, 2001), we select instances with high entropy of the output distribution:
Entropy(xi) = − ∑
j
P (yi = k∗j | xi) log P (yi = k∗j | xi) (2)
Query by Committee (QBC) In QBC (Seung et al., 1992; Burbidge et al., 2007), we select instances relying on the disagreement among a committee of models. Our committee is the set of d = 4 previous student models plus the current – presumably best – student. The disagreement is quantified by computing the proportion of committee members contradicting the current student.
Coreset (CS) CS (Sener and Savarese, 2018) uses an encoder to obtain the embedding representation of the new instance. Then, it calculates the cosine similarity between the embedding of the new input and the embeddings of past examples. If the similarity with respect to the most similar past instance xi annotated by the LLM is below a certain threshold s, then it requests further annotation from the LLM. To obtain the embeddings, we average the encoder representation across tokens, as this has been proven effective in sentence embedding benchmarks (Ni et al., 2022). Similarity with previous examples has been employed in AL to encourage diversity and coverage (Kim et al., 2006; Zeng et al., 2019). GPTCache (Bang, 2023) also uses the embedding representations to decide whether an incoming instance should be labelled.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: We study the proposed setup on four classification tasks. The first two tasks have been commonly studied in AL for NLP: ISEAR (Shao et al., 2015) and RT-Polarity (Pang and Lee, 2005). The remaining two tasks showcase harder problems where factual knowledge acquired during pre-training of an LLM could be highly beneficial: the factchecking dataset FEVER (Thorne et al., 2018)
and the question-answering dataset Openbook (Mihaylov et al., 2018). We split all datasets into online and test portions (80%-20%, except for Openbook, as it has fewer samples). The datasets are balanced.
ISEAR (Shao et al., 2015) annotates personal reports for emotion (classes: joy, fear, shame, sadness, guilt, disgust, anger; 7666 examples).
RT-Polarity (Pang and Lee, 2005) provides sentiment polarity labels for movie reviews (classes: positive, negative; 10662 examples).
FEVER (Thorne et al., 2018) is a fact-checking dataset (classes: true, false; 6612 examples) with claims that can be checked with 1-3 sentences from Wikipedia.
Openbook (Mihaylov et al., 2018) is a challenging question-answering dataset modelled after open book exams for assessing human understanding of a subject. Each instance consists of a multiple choice question (classes: A, B, C, D) and includes one fact that can help answer it. The full dataset consists of 5957 data points; we selected 5457 for the online set and 500 for testing.

heading: 4.2 Annotation by LLM
text: While we are interested in the online caching scenario, to facilitate comparisons between our methods and ensure replicability in future work, we create a dataset in which we obtain LLM predictions for all data points; this dataset is then used to simulate the online setup.
We generate soft labels using OpenAI’s text-davinci-003, an InstructGPT-based model (Zhan et al., 2022). For each task, we design a prompt that describes the task and the possible classes. Our prompts do not contain any in-context examples (zero-shot), but we use a small part of the dataset (up to 10 examples) for prompt engineering.
On all datasets, we observe that the LLM achieves better accuracy than the smaller model trained on 5000 gold labels, suggesting that KD would be useful in these datasets (Table 1). In our benchmark, we store the log-probabilities of the labels. We note that the average margin for the generated labels is substantially lower when the predicted label is wrong; we observe with additional experiments that the LLM annotations are well calibrated (Figure 4). We release our benchmark with the generated labels to encourage further work on the neural caching problem.

heading: 4.3 Experiment Details
text: We run all our experiments with three random seeds, which also determine the ordering of examples; we present the average scores. For simplicity, we use a retraining frequency f = 1000 and a constant cost per query c(xi) = 1. To avoid a cold-start, we train the initial student model S0 with N = 100 (ISEAR, RT-Polarity) or N = 1000 (FEVER, Openbook) data points from the LLM; we choose N so that S0 is better than random choice. For the student model, we use T5base (Raffel et al., 2020) as the backbone model; we freeze the model weights and add LoRA adapter layers for a parameter-efficient fine-tuning (Hu et al., 2022).
We fine-tune the student model with the crossentropy loss using the log-probabilities assigned by the teacher to each class. Using hard labels seems to work almost as well (Table 9). We split the accumulated data from the LLM into training and validation sets, and train each student from scratch for 30 epochs with early stopping with patience of five epochs. The rest of the hyperparameters can be found in Appendix A.

heading: 5 Experiments
text: We first present our results and then their analysis. To report accuracy across budgets, we use the corresponding Area Under the Curve (AUC) divided
by the budget range, thus obtaining an average accuracy.

heading: 5.1 Neural Caching without Student Retraining
text: We first study a simplified version of neural caching, where the student model is not retrained on new data points. This is a practical scenario, as retraining creates extra overhead for the application provider (e.g., consider a setting where the student is run on a portable device, which is not powerful enough to support retraining).
We adapt the AL instance selection criteria in the following way. Given a criterion C, we calculate the respective values from the previous outputs of the student and call this list the history Ĉ. If we have a remaining budget b and n remaining online instances, we use as a threshold for an incoming instance the bn -th percentile of the history Ĉ. The best possible scenario would imply having oracle threshold values for each budget (i.e. as if we had access to the full dataset offline). However, in additional experiments, we found that the above rule yields very similar scores.
To use QBC in this setup, we simulate that we have four previous students trained on subsets of the data. For example, if the student is trained on N = 1000 examples, the previous students are trained on 900, 800, 700, and 600 data points, respectively. We find that MS yields results very similar to PE and that Coreset is similar to Random. To ease visualising the results, here we omit PE and Coreset.
Table 2 and Figure 2 contain the results when we train the initial student with N = 1000 datapoints annotated by the LLM. Our experiments with different initial budgets N yield similar results (Table 8 in Appendix), and Coreset performs poorly even with different encoders.
We find that MS and PE are the best-performing methods on all datasets and across all the initial student models, followed by QBC, which outperforms the baseline of random selection. Given the simplicity of these methods, these results make a
strong case for using AL-based selection methods, especially MS. Unlike QBC, MS does not require storing multiple models and performing inference with each of them.

heading: 5.2 Neural Caching with Student Retraining
text: We now turn to the complete setup proposed in Section 3, in which the selected instances are used to retrain a student model with some periodicity. This creates the incentive to spend the budget early to get a more proficient student model as soon as possible. To observe this effect, we include a random baseline with a uniform sampling rate. This suggests waiting longer for informative examples to arrive counterweights the benefits of getting a strong student as quickly as possible. We select thresholds to encourage spending more of the budget early on (see Appendix A.1).
We show the results averaged across all datasets in Figure 3 and per-dataset in Figure 5. We observe that both MS and QBC substantially outperform the other methods. Coreset (embedding-based) does badly in all the studied setups and encoders (Table 12). Table 3 summarises the results.

heading: 5.3 Analysis
text: Hard examples with noisy labels. We have observed in our experiments that prioritising harder instances for teacher annotation leads to clear gains in online accuracy. However, as discussed in the introduction, LLM accuracy may be significantly affected by the increased ‘complexity’ of an example, which can inflate the proportion of noisy annotations in the data on which the student is trained (see Figure 4). This problem is known in KD as confirmation bias (Arazo et al., 2020; Liu and Tan, 2021). Previous results from offline KD suggest that this type of confirmation bias can be mitigated by avoiding the hardest instances (Baykal et al., 2023), improving the chances that the teacher model makes a correct prediction. However, we observe that the most significant advantage of the LLM with respect to the student in terms of accuracy lies in these samples that are deemed hard by the student (leftmost part of the plot in Figure 4); since we are optimising the online accuracy, the trade-off between providing hard or correct labels may be different in our online case than in the offline scenario. Given the above, we hypothesise that MS and QBC would be more negatively affected by the confirmation bias than front-loading, which does not prioritise hard examples. To test
this hypothesis, we designed an experiment to put an upper bound on the effect of wrong LLM annotations. For each strategy, we only retrain the student model on correct labels, simulating an oracle that discards incorrect examples. Table 5 shows the absolute improvements in the online and final accuracy with respect to the values obtained without the oracle (Table 3 and 4). We observe moderate absolute improvements, but surprisingly MS and QBC do not seem to improve more than front-loading, suggesting that the hypothesis is wrong and that the impact of confirmation bias is somewhat limited and - what is surprising - similar across strategies.
As an additional test, we analyse the subset of test examples where the teacher is incorrect. If confirmation bias is a major issue for MS and QBC than for front-loading, we would expect that they are more prone to reproducing the teacher’s errors. Again, we do not find any substantial differences between these two strategies vs front-loading (Table 11).
Online accuracy vs. final accuracy. Taking a look at the accuracy of the final student (Table 4), we observe that it is generally consistent with the online accuracy (Table 3). However, MS has a low final accuracy on FEVER while having the best online accuracy on that dataset, confirming that in some tasks, calling the LLM to obtain labels for hard examples may improve the online accuracy while not necessarily improving the student. This result emphasises the differences between our set-
up and the setting normally studied in AL.

heading: 5.4 Robustness of the Findings
text: Vary initial training (S0). We study the effect of the quantity of LLM-annotated data on which the first student model is trained, focusing on the setup with retraining (Table 6). We consider the two more challenging tasks, FEVER and Openbook. We find that QBC performs best overall, and the performance of MS is more sensitive to the initial budget. This observation suggests that better decision criteria for transitioning from a front-loading regime to MS can be beneficial; we leave this for future exploration.
Higher retraining frequency f . We repeat neural caching experiments, setting this time a higher frequency of retraining f = 100; this results in much longer runs as the student model has been retrained an order of magnitude more times. Table 7 shows the results. We observe that results are consistent and very similar to those with a lower frequency of retraining (Table 3).
Alternative teacher models Are our findings specific to the teacher model we used? We test this by repeating the neural caching experiments with Mixtral 8x7b (Jiang et al., 2024) as the teacher model. Our results (Table 10) validate Margin Sampling and Query by Committee as the best strategies.
Experiments on additional datasets We further experiment with two sentiment analysis datasets: Customer Reviews (CR) (Ni et al., 2019) and SST2 (Socher et al., 2013). Results can be found in Appendix B.6. The trends in relative performance are similar to other tasks in Table 3.

heading: 6 Conclusions
text: In this work, we have studied how instance selection criteria from AL behave when they are used to decide in real time whether we should perform an LLM call or use a student model that has been trained on previous LLM predictions. In the scenario where we are not retraining the student model, Margin Sampling performs the best, across different datasets. In the scenario where we retrain the student model with some time periodicity, Query by Committee is the most robust option. While Margin Sampling outperforms the front-loading baseline on harder tasks, it is more sensitive to the initial budget spent to train the student model S0.
We find that the embedding-based strategy (Coreset) consistently performs poorly across different encoders; to our knowledge, it is the only LLM caching approach that has been adopted by practitioners to date (e.g., GPTCache (Bang, 2023)). We believe these types of strategies could be useful in
certain contexts, e.g. multiple near-identical calls to an LLM, a scenario which has not been the focus of this work.3
Our results suggest that (i) there is room for smart LLM query allocation in the context of continuously distilling an LLM into a student model and (ii) previous literature in Active Learning can transfer well to this setup. We believe that online Knowledge Distillation could play a key role in caching LLMs, saving unnecessary calls to expensive models, and training smaller private models.
Limitations
Our experiments assume that there is no development set available for each task, which could help improve the results of non-baseline methods by introducing a task-specific hyperparameter for thresholds.
We leave for future work configurations with other student models. We also leave for future work experiments on text generation, which would require using text-based AL criteria.
In this work, we focused on a stationary (i.i.d.) stream of requests. In practice, the distribution of requests is likely to change over time (Cacciarelli and Kulahci, 2023). As suggested by the online AL literature (Bifet and Gavaldà, 2007), this should further increase the gap between the AL-based approaches and static strategies, e.g., front-loading. In those cases, we would expect improvements in both online and final accuracy.
Ethics statement We anticipate that our proposed approach will be advantageous for smaller companies and will enhance user privacy by limiting the amount of data shared with API providers. However, we recognise the potential for misuse of this technology. For instance, it might contravene the service policies of API providers and potentially could decrease the revenue of creators if they are compensated for the usage of their content by the API provider. Furthermore, while the original model provided through the API may have been fine-tuned to diminish harmful biases and tailored for fairness across diverse user groups, there is a possibility that the student model derived through our process may not inherit these qualities.
3FEVER does contain paraphrases or statements entailing each other but these constitute only a small fraction of the dataset.

heading: Acknowledgements
text: We thank Pasquale Minervini for the discussions and comments. GR and ML are supported by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1), the University of Edinburgh, School of Informatics and School of Philosophy, Psychology & Language Sciences. ML also acknowledge funding from Huawei Technologies. IT acknowledge support by the Dutch National Science Foundation (NWO Vici VI.C.212.053). AB has been supported by the European Union’s Horizon Europe Research and Innovation programme under Grant Agreement No 101070631 (UTTER). The computations described in this research were performed using the Baskerville HPC service, funded by the EPSRC and UKRI (grant EP/T022221/1) and the Digital Research Infrastructure programme (EP/W032244/1). In addition, this work also used the Cirrus UK National Tier-2 HPC Service at EPCC, funded by the University of Edinburgh and EPSRC (EP/P020267/1). The project that gave rise to these results received the support of a fellowship from ”la Caixa” Foundation (ID 100010434, grant code LCF/BQ/EU22/11930079).

heading: A Experimental details and hyperparameters
text: Student model We use the T5 implementation from Huggingface’s transformers library. We use LoRA adapters (Hu et al., 2022), as they have been considered one of the most parameter-efficient architectures in few-shot settings (Liu et al., 2022). Following Ponti et al. (2023), we add a LoRA adapter to the query, key, value and output weights in each self-attention layer of T5. We set the LoRA rank to r = 16, and the scaling to α = 0.25. We use learning rate η = 5 · 10−4, training batch size m = 16 and weight decay λ = 0.01. We validate this hyperparameter choice based on experiments using the soft labels from the teacher.
Adaptation of strategies For Entropy, we normalise before computing it by applying a softmax over the classes.
Reporting of results In order to report accuracy across budgets, we use the corresponding Area Under the Curve (AUC) divided by the budget range. By budget range, we refer to the biggest budget minus the smallest one for that task. Intuitively, this gives us an average accuracy across budgets.
Normalisation, pre-processing and evaluation We do not apply normalisation or pre-processing before using the T5 tokeniser. This can be consulted in our code.
A.1 Threshold values To encourage an early expense of the budgets in the setting with student retraining, we have selected threshold values to ensure initially a higher proportion of calls for LLM annotation (PE=0.5, MS=5, QBC=4, CS=0.9); we have selected these values so that the first student model selects at least 50% of instances for LLM annotation on RT-Polarity. However, we observe very similar results when we use the empirical threshold from Section 5.1.
A.2 Labels from the LLM We use a budget for LLM annotation of $200. All the labels are obtained during May 2023. Since the OpenAI API can only return up to the five most likely tokens, we add a bias b = 100 to the tokens that represent each class:
• ISEAR: ’ joy’, ’ fear’, ’ anger’, ’ sadness’, ’ disgust’, ’ shame’, ’ guilt’
• RT-POLARITY: ’ positive’, ’ negative’
• FEVER: ’ true’, ’ false’
• OPENBOOK: ’ A’, ’ B’, ’ C’, ’ D’
If a class is not among the five most likely tokens, it gets assigned in our experiments a log probability of -100.
A.3 Computational resources T5base has 220 million parameters. We additionally added LoRA modules, which comprise 3.5 million parameters. Only LoRA modules are trained, making it a lightweight student overall. For our experiments, we used clusters with NVIDIA Tesla V100-SXM2-16GB and NVIDIA A100-SXM440GB. Runs with frequency f=100 take between 1 and 4 hours to run.

heading: B Additional results
text: B.1 Neural caching with no student retraining We observe that Margin Sampling is the bestperforming method on all datasets and across all the initial student models, followed by Query by Committee and outperforming the baseline of random selection (Table 8). The gap between Margin Sampling and the baseline widens as we have a better initial student.
B.2 Neural caching with retraining Figure 5 shows the online accuracy per-dataset in the setup with retraining of the student.
B.3 Soft labels We conduct experiments to study the effect of using soft labels (using the logprobabilities for each class from the LLM) or hard labels (only using the first class from the LLM). To do this, we train a student model on multiple budgets and obtain the final accuracy. We observe this has some gains in FEVER (Table 9).
B.4 Effect of confirmation bias in neural caching with retraining
To study the confirmation bias, we select the samples from the test dataset where the LLM produces a wrong answer. If the model performance is affected by the noise of the labels it was trained on, it is expected it will reproduce the mistakes of the LLM; therefore, we would expect that it will have a lower score in this subset of the test dataset. We do not find that Margin Sampling and Query by Committee have lower performance
than front-loading in this subset of the dataset (Table 11).
B.5 Experiments with different encoders
To test the validity of results for Coreset, we repeat experiments from Section 5.2 with encoders SimCSE (Gao et al., 2021) and MPNet (Song et al., 2020). For SimCSE, we use sup-simcse-bert-base-uncased from the project repository.4 For MPNet, we use sentence-transformers/all-mpnet-base-v2 from Huggingface. We use threshold = 0.9 for all the methods and retraining frequency f = 100.
We show our results in Table 12. We observe that front-loading outperforms Coreset with the three encoders.
B.6 Experiments with additional tasks
Table 13 contains the results for sentiment analysis tasks for SST-2 and CR. As expected, we find results similar to those in the sentiment analysis task RT-Polarity (Table 3).
4https://github.com/princeton-nlp/SimCSE
B.7 Experiments with an alternative teacher model
We repeat the evaluation of the main results (Table 3) using Mixtral 8x7b (Jiang et al., 2024) as the teacher model. Table 10 contains the results. Our results validate Margin Sampling and Query by Committee as the best strategies.

heading: C Prompts used
text: The following are the prompts we used when calling the LLM. We have marked in blue one example, and in red the expected answer.
• ISEAR: This is an emotion classification task. Only answer one of: ’joy’, ’fear’, ’anger’, ’sadness’, ’disgust’, ’shame’, ’guilt’. INPUT: During the period of falling in love, each time that we met and especially when we had not met for a long time. OUTPUT: joy
• RT-Polarity: This is a sentiment classification task for movie reviews. Only answer either ’positive’ or ’negative’. INPUT: if you sometimes like to go to the
movies to have fun , wasabi is a good place to start . OUTPUT: positive
• FEVER: This is a fact-checking task. Only answer either ’true’ or ’false’. INPUT: On June 2017, the following claim was made: Jeb Bush is former President George H. W. Bush’s daughter. Q: Was this claim true or false? OUTPUT: false
• Openbook: This is a multiple-choice test. You are presented a fact and a question. Only answer one letter, producing no more output. FACT: the sun is the source of energy for physical cycles on Earth QUESTION: The sun is responsible for
A: puppies learning new tricks B: children growing up and getting old C: flowers wilting in a vase D: plants sprouting, blooming and wilting OUTPUT: D

heading: D Additional information about datasets
text: Datasets used RT-Polarity, FEVER and Openbook were relased as NLP benchmarks for classification. While ISEAR was originally released as part of a psychological study on emotion across cultures, it has been used as an Active Learning benchmark in the past (Ein-Dor et al., 2020; Bastos and Kaul, 2021).
We did not check if these datasets contain any information that names or uniquely identifies individual people or offensive content because the data we use comes from established classification/multiple choice benchmarks. A custom filtering or modification of the data would hamper comparability with other works using these benchmarks.
Dataset generated We release the soft labels from the LLM under the CC BY 4.0 DEED license. We refer to the original data sources for documentation such as coverage of domains, languages or linguistic phenomena.
Front-loading 0.734 0.731 Coreset (SimCSE) 0.707 0.726 Coreset (MPNet) 0.716 0.724 Coreset (T5) 0.715 0.726
Table 12: Online accuracy (AUC) for neural caching with student retraining.
CR SST-2
Random 0.911 0.893 Front-loading 0.913 0.899 Margin Sampling 0.917 0.907 Query by Committee 0.914 0.904
Table 13: Online accuracy (AUC) for neural caching with student retraining for additional tasks. We observe that results are similar to those for RT-Polarity (Table 3).

heading: Future Work
text: We leave for future work configurations with other student models. We also leave for future work experiments on text generation, which would require using text-based AL criteria. In this work, we focused on a stationary (i. i. d. ) stream of requests. In practice, the distribution of requests is likely to change over time (Cacciarelli and Kulahci, 2023). As suggested by the online AL literature (Bifet and Gavaldà, 2007), this should further increase the gap between the AL-based approaches and static strategies, e. g. , front-loading. In those cases, we would expect improvements in both online and final accuracy