Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models
Authors: Ilias Stogiannidis
Stavros Vassos
Prodromos Malakasiotis
Ion Androutsopoulos
Venue: EMNLP 2023 Findings (2023)
Keywords: Online Learning
Cost-Efficient
LLMs Applications
Caching
Teacher-Student
TL;DR: Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models

Abstract: Prompting Large Language Models (LLMs) performs impressively in zero- and few-shot settings. Hence, small and medium-sized enterprises (SMEs) that cannot afford the cost of creating large task-specific training datasets, but also the cost of pretraining their own LLMs, are increasingly turning to third-party services that allow them to prompt LLMs. However, such services currently require a payment per call, which becomes a significant operating expense (OpEx). Furthermore, customer inputs are often very similar over time, hence SMEs end-up prompting LLMs with very similar instances. We propose a framework that allows reducing the calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. The framework includes criteria for deciding when to trust the local model or call the LLM, and a methodology to tune the criteria and measure the tradeoff between performance and cost. For experimental purposes, we instantiate our framework with two LLMs, GPT-3.5 or GPT-4, and two inexpensive students, a $k$-NN classifier or a Multi-Layer Perceptron, using two common business tasks, intent recognition and sentiment analysis. 
Experimental results indicate that significant OpEx savings can be obtained with only slightly lower performance.

Cache me if you Can: an Online Cost-aware Teacher-Student Framework to Reduce the Calls to Large Language Models
Ilias Stogiannidis Stavros Vassos Prodromos Malakasiotis Ion Androutsopoulos 'Department of Informatics, Athens University of Economics and Business, Greece HHelvia.ai
Archimedes Unit, Athena Research Center, Greece
Workable
(stoyianel, rulller, ion) @aueb.gr, stavtos @ helvia.ai
Abstract
Teacher response
Teacher
Prompting Large Language Models (LLMs) performs impressively in zero- and few-shot settings. Hence, small and medium-size en- terprises (SMFs) that cannot afford the cost of creating large task task-specific training datasets, but also the cost of pretraining their own LLMss are increasingly turning to third-party services that allow them to prompt LLMs. However, such services currently require payment per call, which becomes significant operating ex- pense (OpEx). Furthermore, customer inputs are often very similar over time, hence SMEs end-up prompting LLMs with very similar in- stances. We propose a framework that allows reducing the calls to LLMs by caching previ- LLM responses and using them train a local inexpensive model on the SME side. The framework includes criteria for deciding when to trust the local model or call the LLM,, and a methodology to tune the criteria and mea- sure the tradeoff between performance and cost. For experimental purposes, we instantiate our framework with two LLMs, GPT-3.5 or GPT-4 and two inexpensive students, k--N classifier or Multi-Layee Perceptron, using two com- mon business tasks, intent recognition and sen- timent analysis. Experimental results indicate that significant OpEx savings can be obtained with only slightly lower performance.
Incoming instance
Teacher response
Cache
Customer
Retraining
Incoming
instance
Student
Incoming instance
Teacher
Incoming Instance
Student response
Reliability indicators
Criteria
student
Student response
response
Figure OCaTS architecture.
and drive the chatbot-customer interaction (Ham et al.. 2020). The best LLMs, however, currently require a payment per prompting call, and these payments become significant operating expense (OpEx) for SMEs. Furthermore, customer inputs (e.g., dialog turns) are often very similar over time, hence SMEs end up calling LLMs handle inputs that may very similar to inputs already handled by the LLMs in previous (already paid) calls.
We introduce the Online Cost-aware Teacher- Student (OCaTS) framework that allows reducing the calls to a commercial LLM, treated as teacher model, by caching its previous responses and using them train local inexpensive student model. OCaTS includes criteria for deciding when to trust the student or call the teacher, and a methodology to tune the criteria and measure the tradeoff between performance and cost. Unlike common teacher- student training for knowledge distillation (Hinton et al., 2015; Gou et al., 2021), here the teacher does not train the student all the available in- stances (in our case, all the incoming customer inputs). Also, unlike teacher-student approaches to self-training (Mi al., 2021; Li et al., 2021). 1), the teacher is already reasonably effective (but ex- pensive). In that sense, our work is closer to ac-
1 Introduction
Prompting pre-trained Large Language Models (LLMs) aligned to follow instructions (Ouyang et al., 2022; Köpf et al., 2023) performs impres- sively well in zero- and few-shot settings. Hence, small and medium-sized enterprises (SMEs) that cannot afford the cost of creating large task-specific training datasets for model fine-tuning, but also the cost of pretraining their own LLMs, are in- creasingly turning third-party services that allow them to prompt L.Mss. For example, SMEs that provide customer support chatbots prompt LLMs like GPT-4 (OpenAI, 2023) to detect user intents

---

tive learning (Settles, 2012: Monarch, 2021), but OCaTS trains the student on labels provided by a teacher LLM, not humans, and there is initially no large pool of unlabeled instances (customer inputs) to select from, as instances arrive online.
fering premium results; student, cost-effective model that is typically much smaller and simpler than the teacher; a cache, repository of incom- ing instances (e.g., customer requests) that have already been processed by the teacher. We assume that the framework is employed to handle task for which there is no available large dataset for super- vised training, apart from a few incoming instances (possibly handful per class) annotated with the ground truth (e.g., correct labels). This is very common case for SMEs that cannot afford the cost of creating large task-specific training datasets, but can easily construct small numbers of demonstra- tion instances. The teacher-student setting is online, as every incoming instance is handled at inference time as follows. First, the student is called to handle the instance. Then some student- and task-speciff criteria, which assess the reliability of the student's s output, indicate if the student's output (e.g.. label) should used or if the teacher should be consulted. If the student's 's output is selected, it is returned as the response to the incoming instance. Otherwise, the teacher is called to handle the instance. In the latter case, the instance along with the teacher's ss result are stored in the cache. Depending on the type of student, periodic re-training takes place, update the student with the cached instances.
OCaTS can be used with any service that allows prompting LLMs, and any kind of local student model. For experimental purposes, we instanti- ate OCaTS with GPT-3.5 or GPT-4 as the teacher, and a k--N or Multi-Layer Perceptron (MLP) clas- sifier as the student, using an intent recognition dataset from the banking domain or a sentiment analysis dataset. Experimental results indicate that significant OpEx savings can be obtained with only slightly lower performance. For example, the k--N student can handle approximately two-thirds of the incoming instances (customer inputs) of the intent recognition task without calling the GPT-4 teacher (Fig. 2, left, red line) for a decrease of less than 0.5 percentage points in accuracy (Fig. 2, middle, red and black lines). OCaTS introduces discounted versions of common evaluation measures (e.g.. ac- curacy) that allow an SME to quantify how much it prefers to lean towards fewer calls less user frustration (different À values in Fig. 2).
Our main contributions are: (i) We introduce a general teacher-student framework that helps SMEs reduce the prompting calls to commercial LLMs and the corresponding OpEx costs by caching the responses of the LLMs and training inexpensive local student models. (ii) We introduce discounted versions of common evaluation measures that al- low the SMEs to quantify how much they prefer fewer L.M calls vs. increased user frustration (e.g., caused by lower accuracy) and tune the frame- work's 's criteria that decide when to trust the local student model or call the LLM teacher accordingly. (iii) We instantiate the framework with GPT-3.5 or GPT-4 as teachers, and K-NN MLP classifier as students. (iv) We perform experiments on two well-known tasks for SMES, intent recognition and sentiment analysis, and show that significant cost savings can be obtained with only slightly lower performance. This is a first step towards exploring the benefits of the proposed framework with more datasets, models, and business scenarios.
Instantiations: In the experiments of this paper, we instantiate OCaTS with a GPT-3.5 or GPT-4 teacher, a distance-weighted k-NN or MLP clas- sifier as the student, for a single-label classifica- tion task (intent recognition or sentiment analysis). In all cases, we represent each incoming instance (customer request) by its MPNet- based (Song et al., 2020) vector representation (text embedding) and two criteria (Fig. 1) to decide when to use the student's response or invoke the teacher: (i) the entropy of the probability distribution (over the la- bel set) produced by the student (k-NN or MLP) for the incoming instance, and (ii) the distance of the vector representation of the incoming instance from the centroid of the vector representations of the k most similar cached instances. Consult Nguyen et al. (2022) for other possible criteria. We leave other instantiations of OCaTS (other teachers, stu- dents, tasks, representations) for future work.
2 Framework
Discounted evaluation measures:
The main goal of the proposed architecture is to reduce the number of calls to the expensive teacher model by caching previous teacher responses and using them to train a local inexpensive student model on the SME side.
Architecture:
The proposed framework (OCaTS) consists of three main components (Fig. I): a teacher, typically a aasource-intensing model of-

---

3000
80
2500
Discounted Accuracy (N)
75
80
the
2000
78
Accuracy
1500
65
1000
74
80
500
72
55
1000
2000
3000
1000
2000
3000
1000
2000
3000
incoming instances
incoming instances
incoming instances
OCaTS, A=0.05 GPT-4.2-0.05
OCaTS, A=0.1
OCaTS, A=02
OCaTS, A=
GPT-4
4, A-D.1
GPT-4, A-02
GPT A-0.3
Figure 2: Number of calls to the teacher (left), accuracy (middle), discounted accuracy (right), using a GPT-4 teacher and k-NN student, for various values, on Banking77 data. The larger the the more the SME prefers fewer calls at the expense of increased user frustration. Dashed lines show the discounted accuracy when calling GPT-4 for all incoming instances. OCaTS has a hetter discounted accuracy than always calling the GPT-4 teacher.
This introduces a tradeoff between the OpEx cost of calling the teacher and the frustration of the end- users when the less accurate student model is used instead. To quantify this tradeoff, we introduce discounted variant é of any common evaluation measure @ (e.g., accuracy, FI), as follows:
Main experiments
Here we discuss the experiments we conducted with the GPT-4 teacher, the k-NN student, and the banking intent recognition dataset. In the Ap- pendix, we report two additional sets of experi- ments, one where we replaced the k--N student by an MLP (Appendix E) keeping the rest of the sctup unchanged, and one where we replaced the task/dataset (by sentiment analysis) and the teacher (by the cheaper GPT-3.5) otherwise keeping the setup of the initial experiments (Appendix F). The additional experiments verify the conclusions of the experiments of this section.
M
(1) where is the number of incoming instances that have been processed (on which e is measured), M is the number of calls made to the teacher while processing the N instances, percentage of the incoming instances call the teacher, and is scalar specifying how intensively the measure should discounted. Assume, for example, that the accuracy of the teacher-student combination is Φ 0.8, but that this accuracy is achieved with p 3. If the SME considers this P value (which would translate, c.g., to monthly cost) as costly as a loss of five percentage points of accuracy, then & 0.75, and Eq. becomes 0.75 0.8 X 3. from which obtain À 0.15. Larger (or smaller) values correspond to cases where the SME considers the same value more (or less) costly in terms of loss of accuracy points. We can also reformulate Eq. 1 as 5 X.p. A. where = Φ - shows how much & gets discounted to account for the cost of Then À can intuitively be thought of as currency exchange rate, showing how expensive p is in terms of 8 (e.g., loss of accuracy in percentage points).
N = A.p,
M N shows for what
Intent recognition dataset: In this section, we use Banking77 (Casanueva et al., 2020), an intent recognition dataset from the banking customer vice domain. It includes 13.083 customer messages, The ground truth assigns each message single label (intent) from the 77 available. The dataset is divided into training (10,003 instances) and test (3, (8080) subsets. Appendix A shows more statistics.
Few-shot training and development sets: Assum- ing that an SME can only afford to construct a small number of training instances per class, we use only 3 x 77 - 231 instances from the original training set of Banking77, three per class, as a few- shot version of the training set. The 231 instances were manually selected to avoid unclear cases, e.g.. similar instances with different ground truth labels. Similarly, we created a few-shot development set of 13 x 77 1, 001 instances from the original training set, for hyperparameter tuning.
We implicitly assume that the exchange rate is constant for all the values of 5 and In practice, it may be different for different ranges of and but we leave this for future work.

---

Incoming instances and evaluation measure: We use the original test set of Banking77 as the in- coming instances. We repeat each experiment with five random shufflings of the test set (to obtain five different streams of input instances) and report aver- age scores over the shufflings. We set & to accuracy, since the test set is balanced (Appendix A).
We define the probability Pe of each c C as:
exp(W.)
Pc Eeec exp(We)
The entropy H of the probabilities pe of the labels of the neighbors is:
Teacher: In this section, we use GPT-4 (OpenAI, 2023) as the teacher, the most capable LLM for few-shot in-context learning tasks at the time. Each prompt includes instructions, demonstrators (in- context few-shot examples), and the incoming in- stance to be classified; see Appendix B for details.
H = Pe log Pe-
Σ dEC
The second criterion requires Hw to be less than a threshold ty. Intuitively, it requires the neighbors to agree on the label of the incoming instance. Hyperparameter tuning: parameters here, the number of neighbors k., and practical choice considering that there are 3 examples per class initially. For each indicative À value (0.05, 0.1, 0.2, 0.3), we employ Bayesian optimization on the few-shot development set (Section 3) to de- termine the optimal combination of the two thresh- olds that maximize & (discounted accuracy). We let le range in [0, 2). and (H in [0, 1.31).3 We use Optuna's 's (Akiba al., 2019) implementation of the Tree-Structured Parzen Estimator (TSPE) algo- rithm (Bergstra al., 2011) after first performing a 10x 10 grid search on the range of values of the two thresholds as a head start. The resulting contour maps and the optimal values of the two thresholds per À value can be found in Appendix D.
There are three hyper-
Student: In this section, a distance-weighted k-NN classifier is used as the student. Vector representa- tions of the incoming instances are generated with a TransforrTT (Reimers and Gurevych, 2019) variation of MPNet (Song et al., 2020).² Appendix C provides more information the dis- tance weighting used. It also shows (Fig. 7) that in a more conventional setting, where a large man- ually labeled training set is available, the k-NN classifier clearly outperforms GPT-4 in accuracy (92% vs. 82%). Note that for the k-NN student, no retraining (Fig. 1) is necessary, since the cache coincides with the memory of the k-NN classifier. The cache is initialized with the 3-shot training examples of the classes (231 instances in total).
the thresholds fc. Iz- We fix k
Criteria: We instantiate the criteria of Fig. I with two conditions. Both have to be satisfied for the student's ' response to be used; otherwise, we call the teacher. The first condition is that the cosine dis- tance between the (MPNet-based) vector represen- tation of the incoming message and the weighted centroid vector of the l nearest neighbors should be less than a threshold tc. Here c Σ iū Vi, and vē;- - w:/ Σ Wyj where w; is the weight assigned by distance weighting (Appendix C) to the i-th neighbour, and V. is the (MPNet-based) vector representation of the neighbour. Intuitively. this condition ensures that the incoming instance is sufficiently close to cached instances.
Results: We evaluate OCaTS for each of the four indicative À values, using the same incoming in- stances (original test set of Banking 77), and the X-specific tuned thresholds fc, tt- As illustrated in Fig. 2, OCaTS succeeds in managing the tradeoff between calls the teacher vs. accuracy. Figure 2 (left) shows that as the discount factor increases, fewer calls to the teacher are made. In Fig. (mid- dle), we see how much accuracy is sacrificed for this OpEx relief. In particular, for A = 0.05 the accuracy of OCaTS is very close to the accuracy of the GPT-4 teacher, within margin of 0.37 percent- age points (83.05% vs. 82.68% for the entire test set), while calling the teacher for only 1/3 of the incoming instances (1050 out of 3080). For higher values of X, we see the intended drop in accuracy to achieve an increasingly smaller number of calls to the teacher. Figure 2 (right) shows that the dis- counted accuracy O of OCaTS (solid lines, one per
To define the second condition, let C'be the set of the labels (classes) of the k nearest neighbors (here- after simply neighbors). Let wie be the weight (as- signed by distance weighting) to the i-th neighbour belonging in class e, and let Wa be the sum of all weights of neighbors of class c, i... W. = E, Wi.e-
"The maximum value of H with 77 classes is 4.34, when using natural logarithms. The upper bound of te was chosen hased on initial experiments on development data.
"Wo used gpt-4-0314 and all 111-mpnet-- sannet-bse-v2 in par- ticular, for the teacher and student, respectivoly.

---

À value) is always clearly higher than the corre- sponding discounted accuracy of always calling the GPT-4 teacher (dashed lines). Hence, OCaTS is clearly better than always calling the teacher, OpEx costs are taken into account. The difference increases (in favor of OCaTS) as À increases, i.c., reducing OpEx costs becomes more important.
MLP) in two tasks (intent recognition. sentiment analysis), leaving further instantiations for future work. Although LLMs like GPT-4 and GPT-3.5 can in principle be used for zero-shot inference, we considered in-context learning with a few demon- strator examples per class. These examples where manually selected to be diverse and indicative of the corresponding classes. This is realistic to some extent; SMEs often request small number of ex- amples from their customers, but the quality of these examples is not always guaranteed. In addi- tion, the test sets we used (from Banking77 and LMR) were balanced and thus not entirely realistic. However, we shuffle the stream of incoming (test) instances which, hence, do not arrive in uniform way with the respect to their classes. Also, to tune te and fHg used development set, extracted from the original training data. Such a develop- ment is not always available in practice, but we used for the sake of the analysis. Interested SMES can use our analysis, as a starting point for their applications and reduce the number of trials needed to find suitable values for te and tH.
4 Conclusions
We introduced an Online Cost-aware Teacher- Student framework (OCaTS) to help SMEs reduce OpEx costs by caching the responses of commer- cial LLMs and training inexpensive local students. We also introduced discounted versions of common evaluation measures, allowing SMEs to quantify the trade-off between LLM calls and user frustra- tion. By instantiating OCaTS with a GPT-4 teacher and a k-NN student and experimenting with an intent recognition dataset from the banking do- main (Banking77), we showed that the calls to the teacher can be significantly reduced (by 1/3) with only a slight performance drop (0.37 percentage points). Additional experiments with an MLP stu- dent on the same dataset led to the same findings (Appendix E). Further experiments with a GPT-3.5 teacher, the initial k-NN student, and sentiment analysis dataset (LMR) also confirmed the conclu- sions of the previous experiments (Appendix F).
Another limitation is that á takes into considera- tion only the cost to call the teacher (p), and indi- rectly the frustration of the user, implied by the performance drop. A more detailed analysis would also incorporate the student cost and other financial metrics possibly with different weights; OCaTS can be easily extended in that direction. Finally, we did not compare against existing caching libraries, e.g., GPTCache.* These libraries are quite sim- plistic and less flexible than OCaTS, which can be used with a variety of teacher- settings.
In future work, we plan to experiment with more datasets and tasks (e.g., question answering), and suggest adaptive policies for À to allow higher OpEx costs (more frequent calls to the teacher) when the cache is cold and be more selective (call- ing the teacher less frequently) later on. We also plan to enhance OCaTS with indicators of how much we can trust the teacher responses (e.g.. confidence of the teacher). Finally, we intend to incorporate more financial metrics (e.g., student costs) in the discounted versions of the evaluation measures and study more complex strategies (e.g., game-the,retic, reinforcement learning) to select the thresholds that determine when to trust the stu- dent or call the teacher.
6 Ethics statement
Constantly querying LLMs to solve everyday tasks is not only costly; it has large energy footprint as well. Our framework aims to alleviate both phenomena. Nonetheless, nificant amount of resources. We believe, however, that by making the framework and the analysis publicly available, we can pave the way towards reducing the resources required by SMEs to handle their day-to-day tasks in the long run.
study required a sig-
5 Limitations
The main scope of this work was to propose flex- ible framework (OCaTS) that will allow SMEs to reduce the OpEx costs when incorporating com- mercial LLMSs in their solutions. We considered only two instantiations of the teacher (GPT-4, GPT- 3.5) and two instantiations of the student (k-NN,
A Acknowledgements
This work was supported by Google's TPU Re- search Cloud (TRC) program.5
htttps://github.. com/ wtt.///illtteeeeeetcaaaaaoooo htttps:/ssites. research. /itiarccc..oglle/trcaboott//////

---

References
Improves Pre-training for Few-shot Learning in Task- oriented Dialog Systems. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing, pages 1887-1898, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next- generation Hyperparameter Optimization Framework. In Proceedings of the 25th ACM SIGKDD Interna- tional Conference Knowledge Discovery and Data Mining.
Robert Monarch. 2021. Human-in in-the----teeeee Machine Learning. Manning Publications.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameetr Optimization. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc.
Vu-Linh Nguyen, Mohammad Hossein Shaker, and Eyke Hüllermeier. 2022. How to Mcasure Uncer- tainty in Uncertainty Sampling for Active Learning. Mach. Learn., 11 111(1)89--122.
lñigo Casanueva, Tadas Temčinas, Danicla Gerz, Matthew Henderson, and Ivan Vulié. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38 45, On- line. Association for Computational Linguistics.
OpenAL. 2023. GPT-4 technical report. ArXiv, abs/2303.08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc.
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and
Dacheng Tao. 2021. Knowledge distillation: A sur- vey. Int. J. Comput. Vision, 1799-61878-1819
Donghoon Ham, teong-Gwain Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In Proceedings of the 58th Anmual Meeting of the Associ- ation for Computational Linguistics, pages 583-592, Online. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP)) pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the Knowledge in Neural Network. ArXiv, abs/1503.02553
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan- der Mattick. 2023. OpenAssistant Conversations Democratizing Large Language Model Alignment.
Burr Settles, 2012. Active Learning. Morgan & Clay- pool Publishers.
Kaitao Song, Xu Tan. Tao Qin, Jianfeng Lu, and Tie- Yan Liu, 2020. Mpnet: Masked and permuted pre- training for language understanding. In NeurlPS 2020. ACM.
Shiyang Li. Semih Yavuz. Wenhu Chen, and Xifeng Yan. 2021. Task-adaptive Pre-training and Self-training are Complementary for Natural Language Under- standing. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 1006- 1015, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Appendix
A Statistics of the Banking77 dataset
Figure 3 shows the label distribution of the original training and test subsets of the Banking77 intent recognition dataset. The training subset exhibits a significant class imbalance (Fig. 3, left), whereas the test subset is balanced (right). In Table 1, we provide further statistics which, along with the la- bel distribution, support the selection of the dataset as a realistic case to perform our experiments.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
B More details about the LLM teachers
Fei Mi, Wanhao Zhou, Lingjing Kong. Fengyu Cai, Minlie Huang, and Boi Faltings. 2021. Self-training
To prompt the LLMs, we used the chat completion API provided by OpenAI, which takes as input a

---

Label distribution for train subset
Label distribution test subset
175
150
125
Count
Count
100
20
10
25
10
20
30
40
50
60
70
77
Label index
Label index
Figure 3: Label distribution of the original train (left) and test (right) subsets of Banking77.
Statisties
Train
Test
My new card is here, what's the process for activating it?
User:
Number examples
10,003
3,080
Minimum length in characters
13
13
Assistant: activate_my_cam
Average length in characters
59.5
54.2
Maximum length in characters
433
368
User: unable activate my card, it won't me.
Minimum length in words
2
Average length in words
11.9
10.9
Assistant: activate_my_cam
Maximum length in words
79
69
Number of intents
77
77
User: Can you help me activate my card
Assistant: activate_my_card
Table I: Statistics of Banking77.
User: What is the youngest for account?
You expert assistant the field of customer service. Your task is help workers the customer service department of company. Your task is to classify the customer's question in order to help the customer service worker to answer the question. In order to help the worker MUST respond with the number and the name of of the following classes you know. you reply with something else, you will
Assistant: age_limit
User: What is the appropriate age for my child
be able to open an account?
Assistant: age_limit
User: How I set up an account for my children? Assistant: age_limit
Figure S: Demonstrators (in-context few-shot examples)
used in the GPT-4 teacher as conversation history.
penalized.
The classes are:
The weight inversely proportional to the square of the cosine distance between the vectors the incoming instance and the neighbor, = -²22 The class with the largest sum of neighbor weights is then assigned to the incoming instance.
eettvaat__y_caat
age_limit
Figure 4: System message used in the GPT-4 teacher.
system message (instructions) along with history of user and assistant messages, and generates an assistant message as output. The system message specifics the model's behavior as a chat assistant (Fig. 4). For our few-shot setting, we add to the sys- tem message few pairs of user-assistant messages from the training set as demonstrators (Fig. 5). The incoming instance to be classified is added as the last user message in the history.
Figure shows the learning curve of a distance- weighted h--N classifier that trained on the orig- inal training set of Banking77 and evaluated on the few-shot development set (Section 3). We observe that with sufficient training data (approx. 1000 in- stances) the k-NN classifier clearly outperforms GPT-4 in accuracy (92% vs. 82%), which justifies our choice to use student in our experiments.
D Hyperparameter tuning
Distance weighting in the k-NN student
We tuned the thresholds fe and byH anew for each value. For each value, we first performed a
The N-NN classifier assigns a weight to cach one the nearest neighbors of an incoming instance.
10 x 10 grid search on the ranges values

---

Diecounted acauracy
Clecounted accuracy, A 0.1
0.05
Enirapy Threshold
Entropy Threshold
069
Distance Threshold
Distance Threshold
Discounted
Discounted accuracy,
-
Entropy Thresholo
Entroyy Thrishoke
135
Distance Threshald
Distance Threshold
Figure 6: Contour plots (discounted accuracy) obtained during threshold tuning in the main experiments (GPT-4 teacher, N--N student, Banking 77 data), for various values.
System message: Analyze the sentiment of the following reviews and classify them as either 'positive' or 'negative'.
Leaming Curve
I-NN
0.50
User: When started watching this movie saw the dude from Buffy, Xander, and figured ah how nice that he's still making a living acting in movies. Now weird movie can stand, given that it's a good dose of weird like for example David Lynch mavies, twin peaks, last highway etc. [...] It wasn't t his acting though, that was alright, but the script just didn't make any sense. Sorry. Assistant: negative
0.85
Accuracy
0.70
As part of the celebration of the release of Casino Royale, this film with the new Bond starring in it was shown, from director Roger Michell (Notting Hill). [...] Also starring Peter Vaughan as Toots, Danira Govich as Au Harry Michell Harry, Rosie Michell Rosie and Johnny English's Oliver Ford Davies as Bruce. Very good! Assistant: positive
User:
2000
4800
ed00
abbe
Training Size
Figure 7: Learning curve of the distance- disthnee-weighted k-- NN student (solid line) when trained the original training set of Banking77 and evaluated (accuracy) on the few-shot development set. Accuracy of GPT-4 on the few-shot development also shown (dashed line).
Figure 8: System message and demonstrators when using GPT-3.5 in the sentiment analysis task.
the two thresholds, to evaluate indicative combina- tions for the thresholds. These are considered as starting points for the Bayesian optimization (Sec- tion 3). Figure 6 illustrates the presence of several good points adjacent to the best point the main experiments (Section 3), all of which maximize the discounted metric to a significant extent. This
shows that several threshold combinations may be considered optimal. Moreover there are large areas often with wide range of values for one thresh- old that are comparable in terms of maximizing the discounted accuracy measure. Table 2 provides the optimal threshold combination as selected by

---

the optimization process for each À value. We can observe that the tuned value for te decreases from
MLP student are very similar to those of the main experiments (cf. Fig. 7).
À 0.2 to = 0.3, instead of increasing, which can be accounted for by the aforementioned obser- vation that multiple points maximize ø. However, wc also notice the general increasing trend for both thresholds, which leads to fewer calls to the GPT-4 teacher, one would expect, since higher À values mean that calls to the teacher more costly.
Hyper-parameter
Range
Value
learning rate hidden layer neurons
10 0.1 (256,
1.6 10
1024)
1024
drapout rate
[0.1,0.5) 0.5
0.22
Table 3: Tuned periments with the MLP student and GPT-4 teacher.
in the Banking77 ex-
A
1x
Lc
0.05
0.8359
0.2269
0.05
0.44
0.896
0.1
1.324
0,5656
0.1
0.479
1.553
0.2
1,558
0.76
0.2
0.583
1,464
0.3
2.336
0.4993
0.3
0.646
1.882
Table 2: Tuned thresholds per in the main experiments
Table 4: Tuned thresholds per in the Banking77 ex- periments with the MLP student and GPT-4 teacher.
(GPT-4 teacher, k-NN student, Banking77 dataset).
Sentiment analysis experiments
E MLP student instead of k-NN student
As further step to confirm the conclusions of the previous experiments (Section 3, Appendix E), we conducted an additional set of experiments, now using sentiment analysis task. We used the Large Movie Review (LMR) dataset (Maas et al.. 2011), which consists of 50,000 IMDB reviews, and two labels (positive, negative review). Table shows more statistics for this dataset.
As a first step to test the generality of the con- clusions of our main experiments (Section 3), we repeated the main experiments this time using Multi-layer Perceptron (MLP) student, instead of the k-NN student. The MLP has one hidden layer with a ReLU activation function, followed by a dropout layer, then an output layer with 77 neu- rons (number of classes) and softmax. We use cross-entropy loss. Again, we initially filled the cache with 3 few-shot training instances per class (3 77 231 in total) from the original training set (the same ones in the main experiments), and the MLP was initially trained on them. We retrain the MLP every 100 calls to the teacher, i.e., when- ever 100 new training instances have been added to the cache. We used multi- obbletiveeeeiive Bayesian opti- mization on the few-shot development set (treated an incoming stream of user requests), optimizing for both loss and accuracy (no cost factors), to find the optimal hyperparameters for the MLP architec- ture. Subsequently, we tuned the threshold values (fc, tx) for each lambda again using Bayesian opti- mization, as we did in the main experiments. The tuned hyperparameters and thresholds are shown in Table 3 and Table 4, respectively. For every incom- ing test instance, the entropy 7. (Section 3) is now computed using the output probabilities of the MLP.
Again, we assume that an SME can afford to create only small number of training instances per class. To simulate this limited data setting, we created few-shot versions of the training and devel- opment sets of LMR, much as in Section 3. For the few-shot training set, we randomly selected 10 instances from the original training set, for each sentiment, as follows. We employed the GPT-3.5 tokenizer from OpenA1 and computed the token size for each review. Subsequently, we randomly chose reviews with token sizes close to the median. Similarly. for the few-shot development set, ran- domly selected 1,000 instances from the original training set, 500 for each sentiment. Finally, due to limited resources, we used only 5,000 instances
hhttps://github comm/ppenai/tkktoke
Statistics
Train
Test
Number of examples
25,000
25,000
Minimum length in characters
52
32
The other criterion (Fig. 1) remains unchanged, i.c., it requires the cosine distance of the MPNet-based vector of the incoming instance the centroid of the & most similar eached instances to be less than te: we use = 5, as in the main experiments. As shown in Fig. 9, the experimental results with the
Average length in characters
1325.06
1293.7
Maximum length characters
13704
12988
Minimum length in words
Average length in words Maximum length words
233.7
228.5
2470
2278
Number of sentiments
2
2
Table 5: Statisties of Large Movie Review (LMR).

---

3000
86
80
2500
Discounted Accuracy (%)
75
2000
Accuracy (96
70
1500
80
1000
78
500
55
50
1000
2000
3000
Numbee of incoming instances
Number of incominn instances
Number of incoming instances
OCaTS, 4, A=0.1
OCaTS, A-02
OCaTs, A=0.3
GPT4
GPT A=0.05
GPT A=02
A=0.3
Figure 9: Number of calls the teacher (left), accuracy (middle), and discounted accuracy (right), using GPT-4 teacher and MLP student, for various À values, Banking77 data. In the left sub-figure the green line (OCaTS, À = 0.2) not visible, because it overlaps with the purple one (OCaTS, À = 0.3). The results are very similar to those of the main experiments (ef. Fig. 7). Again, OCaTS (right, solid lines) has better discounted accuracy than always calling the teacher (right, dashed lines) for all four indicative À values. The larger the X, the fewer the calls to the teacher (left), at expense of reduced accuracy (middle).
94
4000
85
Discounted Accuracy
Accuracy (56
80
3000
90
75
2000
88
Number
70
1000
65
84
3000
1000
4000
5000
5000
Number of incoming instances
Number of incoming instances
Number of incoming instances
OCa15, GPT 3.5,
OCaTS, A-0.1 3.S,
OCa15,
A-0.3
GPT-3.5
A-0.3
Figure 10: Number of calls to the teacher (left), accuracy (middle), and discounted accuracy (right), using GPT-3.5 teacher and a k-NN student, for various values, on sentiment analysis (LMR) data. The results are very similar to those of the previous experiments (cf. Figures 2 and 9).
of the original test set as the stream of incoming instances. As in the previous experiments, we re- peat each experiment with five random shufflings of the incoming stream, and report the average scores. We the distance weighted k-NN clas- sifier, as in Section Again, we & 5, and we employ Bayesian optimization on the few-shot development set to determine the optimal combi- nation of the two thresholds that maximize J. We let le range in [0, 2]. and ly in [0, 0.7). For the teacher, we used the cheaper GPT-3.5 in these ex-
periments. 8 We prompted GPT-3.5 using the same in-context learning approach outlined in Section 3; we provided the few-shot training set we created demonstrators and asked GPT-3.5 to classify the review either positive negative (Figure 8). Fig- ure 10 shows that the results were very similar those of Section and Appendix E.
'The maximum value of 7l with 2 classes is 0.69 when using the natural logarithm.
"We used version gpt-3.5-tuu S-turbo-0300


Reviews:
[{"review_id": "Um8ugFyrrq", "reviewer": "Reviewer_5m6Q", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. \n\nIndeed it is a limitation that we use a single dataset for evaluation, and we will update the limitations section accordingly. Having said that, we would like to point out that this is a short paper, and in the limited space available we present a new problem (reducing the calls to costly LLMs by using a local student), a general framework for addressing it (OCaTS), and a first incarnation/evaluation of the framework in a particular task (intent recognition). We believe that making this work publicly available will foster research on a timely (as noted by the reviewer) problem and will allow others to experiment with our framework in a variety of other tasks, significantly reducing costs for enterprises (as also noted by the reviewer). \n\nAbout using open source models, we would like to point out that deploying LLMs to support commercial services in production requires significant operational costs, including expensive infrastructure and a DevOPs/MLOPs team to coordinate resources in order to handle a high throughput (e.g., of hundreds of users per minute). Therefore using open source LLMs is not necessarily more cost-effective in commercial settings, and requires further analysis that we leave for future work. We will update the manuscript accordingly to clarify this aspect.\n\nAbout selecting GPT-4 as a teacher, we have explored less expensive commercial LLMs, including GPT-3.5 and Claude, and they significantly lag behind in micro-F1 (7.9-9.3 p.p.).  We will clarify this in the manuscript accordingly.\n\nAbout flushing the cache, retraining the student (which includes selecting which accumulated data to use/discard) is included in the general framework (see Fig. 1 and Section 2), but further analysis of this aspect is outside the scope of this short paper that aims to present the problem, the general framework for addressing it, and a first incarnation/evaluation of the framework. We will more clearly acknowledge this point in the limitations section.\n\nAbout reproducibility, we do use a public dataset (BANKING77) and provide in the appendix all the hyper-parameters and prompts required to repeat our experiments. We also plan to release the code upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n+ Investigation of a timely problem.\n+ Reduces cost for enterprises.\n", "reasons_to_reject": "- Weak evaluation, only a single dataset, and limited NLP tasks (intent recognition). The paper would improve if additional NLP tasks (sentiment classification, and named-entity recognition, among others) were considered.\n- Uses proprietary GPT-4 as a teacher, which may not be cost-effective. Open-source variants make more sense if the goal is to save costs.\n- No clear description of when to flush and update the cache. For instance, the cached version may contain older data. How often will the model fetch? What are the performance trade-offs?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes techniques to cache the queries to reduce operational costs at the expense of reduced performance.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6RqtUVCDfv", "reviewer": "Reviewer_7Vyf", "summary": "", "questions": "You write that:\n> λ can intuitively be thought of as a currency exchange rate, showing how expensive ρ is in terms of δ (e.g., loss of accuracy in percentage points).\n\nI think that this is an interesting point, but it would be even more interesting if it were grounded in terms of real costs.\nI have a few questions about this:\n\nd. In your setting would it be possible to back this out to an estimate of the dollar cost of an additional point of accuracy?\n\ne. How would this compare to the dollar cost of an additional point of accuracy from using human-annotated labels instead of LLM generated labels?\n\nI understand that the second question may be difficult to answer without knowing the cost of collecting Banking77 (it looks like the experimental portion is covered by the results in Figure 3), but if you were able to quantify whether/when an additional point of accuracy is using LLMs in your framework, I think this paper would be much more exciting.\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. Please find below our response to your points (b)-(e).\n\n(b) We thank the reviewer for pointing us to abstaining and delegating classifiers and early exit. Unlike abstaining classifiers, which do not provide classification decisions when they are not confident enough, our Teacher-Student model always provides decisions. In that sense, our work is closer to delegating classifiers. In particular, our Student is similar to the cautious first-level classifier of two-level delegating classifiers. However, our Teacher is not trained, unlike the second-level classifiers of delegating classifiers. More generally, delegating classifiers were intended to be more efficient alternatives to classical ensembles (e.g., bagging or boosting), sharing with them the main goal of improving classification performance (e.g., accuracy). In our case, the goal is to save money, by calling less frequently an already high-performance, but costly and not further trained Teacher LLM, and accepting some performance loss (due to the simpler, but cheaper Student). In that sense, our work is related to early-exit approaches, which use a light/cheap first model to filter out easy instances that do not need to go through a more elaborate and expensive second model. However, we are not aware of any previous early-exit work that aims to reduce the calls to expensive LLMs. As in delegating and some early-exit classifiers, we tune the delegation/exit threshold (actually two thresholds in our k-NN Student, based on the distance and entropy of the neighbors), but to minimize discounted-accuracy, which considers both the accuracy of the overall Teacher-Student system and the cost of invoking the Teacher. So, the goal, setup, and objective function that we use are quite different compared to abstaining, delegating, and previous early-exit classifiers. These points can be easily added to the paper, along with the suggested references.\n\n(c) We would like to point out that this is a short paper, and in the limited space available we present a new timely problem (reducing the calls to costly LLMs by using a local student), a general framework for addressing it (OCaTS), and a first incarnation/evaluation of the framework in a particular task (intent recognition). Please see also our related response to Reviewer 5m6Q.\n\n(d) We have used a percentage of calls to the LLM because the cost depends on the actual rate of user messages per day or per month. For example, for 10k user messages to be classified, using the tuned hyperparameters we can project that 2/3 of calls to the LLM will be omitted, saving essentially 2/3 of \\$ 2,400 per day (see note below), i.e., \\$ 1,600 per day for a negligible drop in accuracy of 0.37 p.p. (Note that a 3-shot approach with 77 classes results in a context of approx. 8k tokens per classification call to GPT-4, costing approx. 8 x \\$ 0.03 = \\$ 0.24 per classification call). This example shows that we can retain the LLM accuracy while saving significant operational costs. Similar discussion of indicative cost savings in dollars will be added to the paper and we thank the reviewer for the suggestion. \n\n(e) We would like to stress that OCaTS aims to address an online setting, where there is no prior comprehensive dataset and instances come from the users at the time the instances need to be handled. (Such an online setting is common in business practice, when solutions for new clients need to be developed quickly.) Although we could estimate the costs of a human annotator, e.g., with a naive approach by means of their salary, we note that employing a human annotator is not a viable alternative for a commercial online setting, where each user message needs to be handled within milliseconds. Alternative setups can be devised where a human annotator is periodically employed to annotate messages that the teacher model has already handled, but this is out of scope for this short paper aiming at proposing OCaTS as a online cost-aware teacher-student framework. We will, nevertheless, add related discussion to clarify this point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "a. Performing inference using proprietary LLMs is becoming a substantial operating expense for businesses as well as NLP research labs. The proposed method could be helpful in reducing this cost while maintaining comparable levels of accuracy.", "reasons_to_reject": "b. Novelty. The idea of having the preliminary classifier abstain from making predictions (i.e., delegate making predictions to the LLM) resembles a large body of existing work on abstaining/delegating classifiers and early exiting.\n   I believe that the novel aspect of this work is that an LLM is being used to generate the labels instead of a human labeler, which, given the current popularity of LLMs, is in my opinion a relatively straightforward modification.\n   This accounts for the majority of my excitement score; if there is a novel aspect of the paper that I am missing I would be willing to revise this score upwards.\nc. Limited Scope. Experimental results are only reported for a single task, thus it is hard to generalize how well this method may work in other settings. This negatively impacts my evaluation of the paper's soundness.", "questions_to_authors": "You write that:\n> λ can intuitively be thought of as a currency exchange rate, showing how expensive ρ is in terms of δ (e.g., loss of accuracy in percentage points).\n\nI think that this is an interesting point, but it would be even more interesting if it were grounded in terms of real costs.\nI have a few questions about this:\n\nd. In your setting would it be possible to back this out to an estimate of the dollar cost of an additional point of accuracy?\n\ne. How would this compare to the dollar cost of an additional point of accuracy from using human-annotated labels instead of LLM generated labels?\n\nI understand that the second question may be difficult to answer without knowing the cost of collecting Banking77 (it looks like the experimental portion is covered by the results in Figure 3), but if you were able to quantify whether/when an additional point of accuracy is using LLMs in your framework, I think this paper would be much more exciting.\n", "paper_topic_and_main_contributions": "This paper studies the topic of mitigating the cost of LLM inference by delegating simple examples to an inexpensive preliminary classifier.\nThe primary contribution is an approach for using the LLM predictions to improve the preliminary classifier in an online setting.\nThe approach employs an embedding-based kNN classifier over past LLM predictions as the preliminary classifier.\nThe decision of whether to delegate to the LLM is based on whether the distance between the input embedding and the embeddings of the nearest neighbors exceed a learned threshold.\nThe decision threshold is learned to optimize a discounted objective function (e.g., accuracy) where the discount factor can be tuned to weigh the relative cost of performing inference with the LLM vs. making a mistake.\nThe paper also contributes experimental results applying the method to the Banking77 intent classification datasets.\nThe experimental results demonstrate that the proposed system can achieve accuracy within 0.5% (absolute) of the LLM, while only using the LLM to make predictions on 1/3 the number of examples.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2 - Discounged"}, "extra_text": {"Missing_References": "These are a few papers that study the related problems of delegating and abstaining classifiers as well as other cost-reducing approaches like early exiting:\n- César Ferri, Peter Flach, and José Hernández-Orallo. 2004. Delegating classifiers. In Proceedings of the twenty-first international conference on Machine learning (ICML '04). Association for Computing Machinery, New York, NY, USA, 37. https://doi.org/10.1145/1015330.1015395\n- Tadeusz Pietraszek. 2005. Optimizing abstaining classifiers using ROC analysis. In Proceedings of the 22nd international conference on Machine learning (ICML '05). Association for Computing Machinery, New York, NY, USA, 665–672. https://doi.org/10.1145/1102351.1102435\n- B. Barla Cambazoglu, Hugo Zaragoza, Olivier Chapelle, Jiang Chen, Ciya Liao, Zhaohui Zheng, and Jon Degenhardt. 2010. Early exit optimizations for additive machine learned ranking systems. In Proceedings of the third ACM international conference on Web search and data mining (WSDM '10). Association for Computing Machinery, New York, NY, USA, 411–420. https://doi.org/10.1145/1718487.1718538\n\nI think these would be good starting points for a related work section."}}, {"review_id": "zXGl2hk4EU", "reviewer": "Reviewer_4DeH", "summary": "", "questions": "* Could you add stats around what % of request both in train and test have same responses. \n* What is the rationale behind using a discounting factor in the eval metric?\n* It is not clear how the current algorithm handles a new label. Could you elaborate on that in the paper?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\nIndeed, in more complex compositional or complex queries one may need to use only parts of cached instances and query the LLM for other parts. We would like to point out, however, that this is a short paper and in the limited available space we present a new  problem (reducing the calls to costly LLMs by using a local student), a general framework for addressing it (OCaTS), and a first incarnation/evaluation of the framework in a particular task (intent recognition). We plan to explore more complex tasks, including compound queries, in the future, and we also expect that others will want to explore and extend our framework in a variety of tasks. Please see also our related response to Reviewer 5m6Q. We will update the manuscript accordingly to acknowledge these points.\n\nAbout handling new labels, we would like to note that this is a first instantiation of OCaTS and we opted for a simple setting to evaluate the framework. Handling cases where the label set changes (e.g., by adding new labels or deleting existing ones) is a very practical problem in commercial settings and a line of research on its own;  we plan to address this interesting direction in future work and we will update the manuscript accordingly to acknowledge this point. \n\nAbout the percentage of requests that have the same response, note that we experiment with a single-label intent classification task with 77 intents. Given a user message, the response is the intent of the user. From the training data, we use only 231 examples (exactly 3 per class) that are used both for the teacher to perform in-context learning (as demonstrators in the prompt) and as the initial cache of the student (warm up). Consequently, for each incoming test instance there will be exactly 3 instances of the test intent in the context of the teacher LLM, and at least 3 instances of the test intent in the student’s cache; the cache starts at a balanced state, but it may evolve to less balanced states if the LLM happens to be queried more frequently for some classes (intents). We can include figures showing the evolution of the class distribution in the camera-ready.\n\nRegarding the discounting factor, increasing the accuracy usually comes at a cost, i.e., making a paid call to the teacher model. On the other hand, using the cost-effective student model leads to lower accuracy. The discounted evaluation measure is a way to quantify the trade-off in a single value, allowing to compare different instantiations of the teacher-student framework. We discuss this in Section 2 (lines 154-162), but we will make the discussion clearer in the final manuscript, also adding examples of cost savings in dollars (as suggested by reviewer 7Vyf) to better ground the discussion.\n\nAbout reproducibility, we do use a public dataset (BANKING77) and provide in the appendix all the hyper-parameters and prompts required to repeat our experiments. We also plan to release the code upon acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed teacher-student framework is a simple yet effective solution for teams trying to optimize for cost and latency in using expensive LLM calls.", "reasons_to_reject": "* The current framework seems it cannot handle compositional or compound queries where a correct response to a subtask is present in the cache but the other requires LLM call.\n* It is not clear how the current algorithm handles a new label that was not part of training data.", "questions_to_authors": "* Could you add stats around what % of request both in train and test have same responses. \n* What is the rationale behind using a discounting factor in the eval metric?\n* It is not clear how the current algorithm handles a new label. Could you elaborate on that in the paper?", "paper_topic_and_main_contributions": "Authors propose a simple yet effective approach to decide when to make an expensive call LLM call vs when can past knowledge be used by caching and avoid the LLM call. Authors propose a teacher-student framework based on kNN classifier. Authors also introduce a discounted evaluation measure. Authors evaluate their approach and provide initial baselines on employ Banking77 dataset.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses a new problem of high practical relevance in education, namely how the costs of calls to LLMs can be reduced by using previous results of LLM calls in a knn fashion. The paper proposes a simple yet effective solution to a timely problem which seems appropriate for a short paper, although reviewers criticize some technical details (how to update the cache exactly, how to handle new labels) and a somewhat limited evaluation on a single dataset only.
Decision: Accept-Findings