Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection
Authors: Guillem Ramírez
Alexandra Birch
Ivan Titov
Venue: COLM (2024)
Keywords: API calls optimisation, uncertainty, calibration, cascading
TL;DR: We optimise calls on a scheme of two LLMs of different size by looking at the uncertainty of the generations from the smaller one.

Abstract: Researchers and practitioners operating on a limited budget face the well-known cost-performance trade-off dilemma. The challenging decision often centers on whether to use a large LLM with better performance or a smaller one with reduced costs. This has motivated recent research in the optimisation of LLM calls. Either a cascading strategy is used, where a smaller LLM or both are called causally, or a routing strategy is used, where only one model is ever called. This is dependent on a decision criterion in both scenarios which is typically an auxiliary neural model. In this work, we propose a cost-effective solution; we use only the uncertainty of the generations of the small LLM as the decision criterion. We compare our approach with both cascading and routing strategies using three different pairs of pre-trained small and large LLMs, on nine different tasks and against approaches that require an additional neural model. Our experiments reveal this simple solution optimally balances cost and performance, outperforming existing methods on 25 out of 27 experimental setups.

Published as a conference paper at COLM 2024
Optimising Calls to Large Language Models with Uncertainty- Based Two-Tier Selection
Guillem Ramírez University of Edinburgh gramir rezêed. aateeddacc.ur ac. uk
Alexandra Birch University of Edinburgh Birch@ed. ac.uk uk
Ivan Titov
University of Edinburgh University of Amsterdam iti ititove ed.ac. uk
Abstract
Researchers and practitioners operating a limited budget face the cost- performance trade-off dilemma. The challenging decision often centers on whether to large LLM with better performance reduced costs. This has motivated recent research in the optimisation of LLM calls. Either a cascading strategy is used, where smaller LLM or both are called sequentially, or roufing strategy is used, where only one model is ever called. Both scenarios are dependent on decision criterion which typically implemented by an extra neural model. In this work, we propose a simpler solution; we use only the uncertainty of the generations of the small LLM as the decision criterion. We compare our approach with both cascading and routing strategies using three different pairs pre-trained small and large LLMs, nine different tasks and against approaches that require an additional neural model. Our experiments reveal this simple solution optimally balances cost and performance, outperforming existing methods on 25 out of 27 experimental setups.
smaller one with
Introduction
Large Language Models (LLMs) offer a high performance for wide range of text tasks. Their widespread popularity both in research and industrial applications necessitates an understanding on how to optimally use them. Bigger models tend have better perfor- mance, while smaller models are faster and cheaper to run. Deciding which model to use common dilemma for many researchers and practitioners with limited budgets, time-constraints environmental concerns.
Recent works attempt to optimise calls to set of LLMs. In this work we consider the set-up with two LLMs, where one is more expensive with greater performance than the other. In this scenario, there are two main strategies (Figure I]: routing, where a query from is directed to only one model based on decision criterion; cascading, where the query always goes to the cheaper model and may subsequently go to the more expensive model depending on the cheaper model's 's output. These previous studies use one of these calling strategies, and involve either using an auxiliary model to score an LLM output (Chen et al.,
2023 Sakota et al. 2023 Ding et al. 2024 Madaan al. 2023) or using repeated calls to the small cheaper LLM Yue et al. 2024 Madaan et al. 2023
Studies that use an auxiliary model introduce further complexity in the optimisation ap- proach, and it remains unclear when practitioners should rely on these auxiliary models. Not only do they require additional training, but they also usually require specific training data and the auxiliary models may not generalise to other tasks. For studies that rely repeated calls to the small LLM, this approach can become expensive, undermining the original practical motivation for its use. is perhaps for these reasons, that neither approach has gained traction among practitioners.
However, we question whether these additional models the repeated calls are required to optimise LLM calls. We hypothesise that they may unnecessary in many use cases, as we can extract confidence measures from the generations of the small model. To investigate this,
1

---

Published as a conference paper at COLM 2024
Peguast
Regurssstruur
Who wrotte If There Is
Who wrate If There In Hoven't Found Yer?
Haven' Found it Yat??
Ask the lar LLMP
Small LLM
AnAe Funder
Yea
Ask the large LLMP
Small LLM
Yes gom
Payne
Large L 70b
Nick Payne
Figure 1: Routing (left) attempts to select the LLM with the best cost-accuracy trade-off given an incoming query. In cascading (right), all queries are passed through the small model, and depending on its output, the large LLM is consulted. We propose using cascading approach that uses the margin of the generations to score outputs from the small LLM.
propose cascading policy that uses simple measure of confidence of the small LLM to decide whether the large LLM needs to called. We evaluate our policy on nine tasks, and for three different pairs of pre-trained small and large LLMs. We focus specifically on short- generation tasks, which are widely used and versatile, and do not include gongggenraation tasks in this study due to the additional complexity they introduce for evaluation. In our experiments on classification, multiple-choice and policy outperforms methods that require additional training.
(QA) tasks, this
We believe our findings suggest that the optimisation of LLM calls should gravitate to- wards understanding their existing available signals, as opposed to training and running further auxiliary models. Moreover, simple and cheap policies have a lower entry cost for researchers and practitioners, which has an impact on their adoption
The key contributions of this work are as follows:
We propose using the margin of the generations of LLMs to optimise LLM calls. The approach is simple, non-parametric, and does not require data or running multiple calls to the small model.
We test our policy on nine tasks with three different pairs of small and large LLMs, and against relevant neural methods that use auxiliary models. We find that Mar- gin Sampling outperforms the other methods, despite using fewer resources. In addition, we further test our policy in a multi-task set-up and obtain similar results.
Our results underscore the importance of understanding signals from LLMs in the context of optimising calls to them, in contrast with previous work, which require additional training of auxiliary models.
2
Related work
The margin between the two most likely classes has been widely re- garded as an uncertainty measure adopted from early Active Learning literature (Scheffer ee al.) 2001 Luo et al., 2004) and considered more recently in the context of Knowledge
LLM uncertainty
Distillation from LLMs (Baykal et al. 2023 Ramírez al. 2023). Other measures of un-
certainty exist for LLMs (Baan et al.) 2023: Huang et al. 2023). The primary challenge in
We make code publicly available: https:/ http://ggtuuuucoo//gllleeaaaaaa/ haaaaammmmmmmmm/nnmn// llms
2

---

Published as a conference paper at COLM 2024
text generation lies in the difficulty of distinguishing between various forms of uncertainty, Specifically, when evaluating the confidence level of a text generator, we would ideally want to concentrate uncertainties related to the change in meaning. This entails distinguishing uncertainties that affect the intended meaning from generated text. In this study, we address this challenge by concentrating on tasks that require generating very short sequences of tokens. We find that even basic approach to uncertainty can be advantageous. Additionally, while it might seem logical to assess uncertainty through multiple text outputs from a model, this would lead to significant computational costs, which are impractical for our purposes. Therefore, we employ straightforward method that does not necessitate generating multiple text samples.
variations in the
Optimisation of inference costs Several methods have been proposed to improve the latency of LLMs, such as speculative decoding (Leviathan et al. 2023), knowledge distil- lation (Bucila et al. 2006 Hinton et al., 2015) and model quantisation (Jacob et al. 2018). However, operational costs rather than latency are the focus of this work. Our method only requires access to the logits of small pre-trained LLM with no need for its parameters.
Optimisation of LLM API Calls Recent work deals with the problem of optimising calls to a pool of LLMs (Wang et al. 2024). Sakota et al. (2023) and Lu et al. (2023) propose to train an auxiliary model that predicts the success calling each LLM. Similarly, Ding et al. (2024)
developed model that predicts the quantitative benefit of utilizing smaller language model over a larger one.
Chen et al.
2023 proposed cascading: using an auxiliary model to predict the accuracy
of the small LLM's 's output. Madaan et al. (2023) and Zhhng et al. (2023) used cascading in conjunction with multiple calls to the small model. Finally, Yue et al. (2024) propose cascading approach that requires repeated calls of the small LLM for reasoning tasks; we
show that this can be simplified to just looking at the most likely tokens for short-generation
tasks.
Ramírez et al. 2023) showed that the margin on a knowledge-distiled model could optimise calls to the larger LLM. This work was limited to one pre-trained large LLM and a smaller fine-tuned local model. However, many practitioners and researchers may not be able to fine-tune their own models due to budget constraints. We extend these findings to pairs of both small and large pre-trained LLMs, as well as to other generation tasks including multi-task setup.
Concurrent work (Gupta et al. 2024) showed that the uncertainty at the token level can be used to effectively leverage a smaller LLM in cascade setup. Their focus is on the combination of the multiple tokens involved in longer generation, and propose supervised method; we propose using the margin, a simple rule that optimises short-generation tasks without needing additional data. In addition, we show its applicability in the multi-task setup.
Our proposed method differs from previous studies as it does not require previously annotated data for the task and does not require repeated calls to the small LLM. Finally, it does not require training and deploying an auxiliary model.
3 Optimisation of LLM calls
3.1 Problem definition
In this work, we predict mappings between elements in the input space, X', and the corre- sponding labels in the output space, y. We have access the small and the large LLMs that we can prompt to become predictors fi X + A()),, where A())) denotes the class of probability distributions over y. We simulate the online setting, where users send queries
jid sequentially. We have queries (x),..., ~ X and we predict the corresponding labels (y1,..., Vq)). For each incoming query X;, we decide whether to call LLM based on calling strategy (see below), and incur given cost cs (xxx; or (x;; respectively. The average
3

---

Published as a conference paper at COLM 2024
cost of the queries by both models is then given by ć (x))) & [x(() We assume that & < 81.
3.2 LLM Calling Strategies
For strategies that require training additional model, we use train dataset Xirain, Yuain, and use either the corresponding labels from the small LLM only, fxttrran)) or from both LLMs, f&( (Xaan and fi trrann), depending on the strategy. For the training of the auxiliary models, we follow the original papers as much as possible and perform hyperparameter search where values are omitted. See Appendix A for a detailed explanation of the training process.
3.2.1 Routing Strategies
Since only the small or the large LLM is called in routing strategies, then for given target average cost per query (cs I c C G), we call the large LLM with a probability pr calculated from the re-arranged form of Equation 1
c= (1-) + prå
(1)
Random routing For every incoming query we call the large LLM with the probability pr-
We train a meta-model to predict the per- formance of the small LLM only, given an incoming query. If this prediction is below threshold value related to the probability pr, it indicates the small LLM's performance is insufficient and thus must call the large LLM.
Routing (Sakota et al. 2023
al.
2023
HybridLLM (Ding et al. 2024 The performance of both the small and large LLMs are modelled in this strategy. We train meta-model to predict an incoming query likely to be better solved by the small LLM than by the large LLM. As in Routing above, if this prediction below threshold value related to the probability pt, we call the large LLM, otherwise the small LLM.
3.2.2 Cascading
Since the small LLM is always called, then for a given target average cost per query, c, we call the large LLM with probability Pe calculated from the re-arranged form of Equation 2
== & + Pec
(2)
FrugaIGPT (Chen et al. 2023) We train a model that, given a query and a candidate answer, predicts if the latter is correct. If this prediction is below threshold value related to the probability Pe, we call the large LLM.
Margin Sampling (ours) We suggest using the uncertainty of the output, namely the margin (Scheffer et al. 2001 Luo et al., 2004), defined by:
Margin, (x;) (y, = k-1 x;) Pj (1i k x;)
(3)
where k/=l and k/=l are the first and second most likely tokens, respectively, according to the distribution of for the first predicted token position, f = 1. One advantage of this approach is that it does not require generating full sequence to able to compute uncertainty. Moreover, for the tasks consider there is generally more uncertainty in the first token. If the margin is below a threshold value related to the probability Pe, we call the large L.M.M

---

Published as a conference paper at COLM 2024
3.3 Dynamic threshold
All the investigated strategies require setting threshold for the decision criterion, and we select a dynamic threshold in this work. An initial threshold is calculated using the first 20 queries. We do not evaluate whether to call the large LLM for these 20 queries, we only obtain outputs from the auxiliary models, or the margin value for Margin Sampling. This may require calling the small LLM depending on the strategy. We then use this distribution to calculate an initial pr or pc-th percentile value. For all subsequent queries, the decision to call the large LLM is made, and the threshold is dynamically updated based on all past queries. We test the effect of the dynamic threshold in additional experiments (Table 10 and conclude it does not result in major performance inefficiencies.
4 Experimental setup
4.1 LLMs
We study three pairs of small and large LLMs in our experiments: Mistral 7B (Jiang et al. 2023) and Mixtral 8x7B (Jiang et al., 2024); Llama-2 of size 13B and Llama-2 of size 70B (Touvron et al.) 2023); ; GPT-3 (Brown et al. 2020) and GPT-4 (OpenAl, 2023). We selected these pre-trained LLMs because of their popularity among practitioners, well as to show robustness across different scales. We have arranged the pairs this way to keep the family of LLMs similar; however, this is not requirement for any of the calling strategies.
For the open-source families (Mistral, Llama-2), all our experiments are done locally in one NVIDIA A100 GPU (80 GB), after applying 4-bit quantisation. Section C contains more details about the LLMs used.
4.2 Datasets
We draw inspiration from Liang et al. (2022) and Ramírez et al. (2023) and choose wide range of tasks, showcasing different difficulties. The sizes of the datasets, along with the label distribution and accuracy of the LLMs, can be found in Appendix (Section Section C
For emotion classification we use ISEAR (Shao et al. 2015); ; for fact- checking we use FEVER (Thorne et al. 2018); for sentiment analysis we use RT-Polarity
Classification tasks
(Pang & Lee 2005 are balanced.
CR Ni al. 2019 and SST-2 (Socher et al. 2013 All of these datasets
Multiple-choice
We use Openbook (Mihaylov et al. 2018), popular multiple-choice dataset that involves common knowledge of the world.
QA short generation We use NaturalQuestions (Kwiatkosski et al. 2019), that contains real questions from human users; we use Wikifact (Petroni et 2019 Goodrich et al. 2019), which consists of knowledge base completion problem to test lactual knowledge; use bAbl (Weston et al., 2016), that tests language understanding and reasoning.
4.3 Experiment details
For each dataset we set aside 1,000 data-points for a train dataset that used only to train the auxiliary models. The remainder is used for the online test set. We use n = 500 data-points from the train dataset the Routing, HybridLLM and FrugalGPT strategies when training the auxiliary model, unless stated otherwise. For the auxiliary models, we fine-tune DistilBERT (Sanh et al. 2019), as per Sakota et al. ) Ding et al. (2024) Chen et al. (2023) ; Madaan et al. (2023) Ding et al. (2024). We further split the training data into 80% train and 20% validation, and fine-tune for 100 epochs with early stopping and patience of 20 epochs.
Unless stated otherwise, our results assume a simple cost scheme with e, (x;) 1 and (x;;) 10, consistent with the pricing of commercial APIs (Wang et al. 2024) and similar
5

---

Published as a conference paper at COLM 2024
to the cost schemes of related work (Madaan et al. 2023 Chen et al. 2023 Lu et al. 2023
Sakota et al.) 2023). We do not take into account in our experiments the latency of running DistilBERT, which deem negligible compared to running the LLMs. To evaluate accuracy across budgets, we report Area Under the Curve (AUC) of the accuracy divided by & čş. Bolded results mark best performance, and underlined results mark second-best We run our experiments with three random seeds, and report average results.
Mistral 7B & Mistral 8x7B
Llama-2 13B & Llama-2 70B
0.72
0.68
0.70
Accuracy
0.66
0.68
Margin Sampling
HybridLLM
Routing
0.62
FrugalGPT
10
Average cost per query
Average cost per query
GPT-3 GPT-4
0.78
0.76
Accuracy
0.74
0.72
0.70
2
4
6
10
Average
query
Figure 2: Accuracy curve with respect to budgets. We have averaged results for all the tasks.
Results
5.1
Comparison of calling strategies
Table showws the AUC for all five evaluated calling strategies, across the nine tasks and for the three pairs of LLMs. Averaged across all nine tasks (the final column), we see that Margin Sampling outperforms all strategies for all LLM pairs. Across individual tasks, we see that it performs best or second best for seven of the nine tasks consistently across all three LLM pairs. Of all the nine tasks and three LLM pairs, it does not achieve best or second-best performance for only two of the 27 combinations: for ISEAR and for NaturalQuestions for the Mistral 7B and Mixtral 8x7B LLM pair. This is likely due to the poor accuracy of the small LLM Mistral 7B on these tasks (Appendix C). Figure 2 shows how performance changes with budget. We see that Margin Sampling dominates across all budgets, despite not using any training data.
showss
Indeed, the performance of Margin Sampling seems to improve as the performance of the cheaper LLM improves, which is to be expected; it achieves its best results when applied top of GPT-3.
FrugalGPT is on average the second-best performing strategy. This is due to its good performance on classification tasks, which is expected as uses a classifier trained on the task as the auxiliary model. However, performs worse than the random baseline on the more

---

Published as a conference paper at COLM 2024
ISEAR RT-Pol FEVER CR SST-2 Openbook Wikifact bAbl NaturalQ Average
Mistral 7B Mixtral 8x7B
Random
0.606
0.876
0.773
0.923 0.880
0.843
0.443
0.597
0.193
0.681
Routing
0.618
0.876
0.777
0.924 0.890
0.844
0.492
0.606
0.177
0.689
HybridLLM
0.618
0.876
0.776
0.924 0.886
0.849
0.454
0.612
0.199
0.688
FrugalGPT
0.632
0.887
0.777
0.931 0.901
0.835
0.477
0.596
0.172
0.690
Margin Sampling
0.617
0.885
0.777
0.933 0.899
0.868
0.499
0.606
0.187
0.697
Llama-2
13B Llama-2
70B
Random
0.630
0.809
0.653
0.885 0.873
0.617
0.505
0.600
0.259
0.648
Routing
0.639
0.836
0.662
0.909 0.883
0.621
0.514
0.593
0.254
0.657
HybridLLM
0.641
0.844
0.681
0.899 0.874
0.626
0.514
0.608
0.264
0.661
FrugalGPT
0.662
0.856
0.668
0.918 0.899
0.598
0.507
0.602
0.258
0.663
Margin Sampling
0.645
0.853
0.691
0.912 0.893
0.640
0.516
0.609
0.270
0.670
GPT-3 GPT-4
Random
0.747
0.914
0.816
0.931
0.898
0.877
0.552
0.574
0.281
0.732
Routing 0.769
0.915
0.815
0.936 0.898
0.878
0.558
0.584
0.277
0.737
HybridLLM
0.744
0.914
0.821
0.932 0.899
0.882
0.556
0.558
0.278
0.732
FrugalGPT
0.767
0.922
0.819
0.940 0.903
0.876
0.564
0.572
0.283
0.738
Margin Sampling 0.771
0.925
0.826
0.940 0.899
0.918
0.584
0.598
0.294
0.751
Table 1: Accuracy (AUC) for the three LLM model pairs across the five classification tasks
(columns 2-6), the multiple-choice task (column 7) and the three generation tasks (columns 8-10).
challenging multiple-choice task, Openbook. FrugalGPT also performs inconsistently for QA tasks; we conclude that FrugalGPT may be satisfactory relatively easy classification tasks and struggle with harder generation tasks.
Finally, Routing and HybridLLM seem to have good performance in QA tasks while having a worse performance in classification tasks. We note that HybridLLM on average has the same performance random for the OpenAl models, which is a surprising finding.
We have not shown comparisons to approaches that require repeated calls to a small LLM throughout this work, in preliminary experiments found they do not perform well (see Section D.2)
5.2 Multi-task setting
LLMs are often used to handle various tasks simultaneously. To simulate this scenario, we create an artificial multi-task setting by merging the datasets from all nine tasks. We then sample 10,000 data-points. We split this dataset into a 10% train set (n = 1, 1,000 data-points) and 90% online test set.
Figure 3 and Table show the results for our multi-task experiments. We see again that Margin Sampling has the best performance. This shows the versatility of this method, that it can be applied across tasks with ease. In contrast, HybridLLM has poor performance both for Llama-2 and OpenAl models. We found these results still hold when using 5,000 data-points as training data (Appendix D.1.
5.3 Robustness of experiments
5.3.1
Investigating the effect of training data
To ensure that auxiliary models not being unfairly handicapped by low-data setting, we train them with double the data points as before (n = 1,00),, in spite of this being possibly infeasible amount of data for many researchers and practitioners.
7

---

Published as a conference paper at COLM 2024
Mistral
Llama-2
OpenAl
Random
0.718
0.681
0.775
Router
0.731
0.696
0.786
HybridLLM
0.726
0.676
0.773
FrugalGPT
0.733
0.694
0.781
Margin Sampling
0.736
0.704
0.792
Table Accuracy (AUC) in the multi-task setting. Methods Router, HybridLLM and FrugalGPT have been trained with 1, 000 data-points.
Mistral 7B Mixtral 8x7B
Liama-2 Llama-2 70B
0.74
0.70
Accuracy
0.73
0.72
Random
0.68
Margin Sampling
0.70
HybridLLM
0.66
Routing
0.69
FrugalGPT
2
8
10
6
10
Average cost per query
Average cost per query
GPT-3 & GPT-4
0.80
Accuracy
0.78
0.76
10
Average cost per query
Figure Accuracy curve with respect budgets, in the multi-task setting.
Table 3 shows the averaged AUC results across tasks for the three LLM pairs. The addi- tional data generally improves the performance of the auxiliary models, however, Margin Sampling still performs competitively, achieving the best performance for two of the three pairs. Table shows that the gap with FrugalGPT closes and that HybridLLM has again a limited performance on the OpenAl models. Additionally, we run experiments in settings with more data (n 5, 000), and confirm that Margin Sampling still performs competitively (Table 11
5.3.2 Investigated the effect of cost
While previous studies typically only study single cost setting SSkktaa al. 2023 Chen et al. 2023 Madaan et al. 2023 Ding et al. 2024), this could influence findings. We carry out further experiments with alternative cost schemes for the three LLM pairs across all tasks. The cost ratio Fcust = c/c, cc is sufficient to parameterise the effect of cost. We need only vary C and maintain C₂ 1. We study values of cj 2, and 20.
We note that, at the time of writing, the pricing ratio for the OpenAl models that we used (GPT-3 and GPT-4) is $30 330mmilion tokens 52/ million tokens 15
8

---

Published as a conference paper at COLM 2024
Mistral
Llama-2
OpenAl
Random
0.681
0.648
0.732
Router
0.694
0.661
0.741
HybridLLM
0.691
0.664
0.734
FrugalGPT
0.695
0.672
0.743
Margin Sampling
0.697
0.670
0.751
Table 3: Accuracy (AUC, averaged across tasks) when Router, HybridLLM and FrugalGPT have been trained with 1,000 data-points.
Mistral
Llama-2
OpenAI
G=2
G=5
G=20
G=2
q=5
G=20
G==
q=5
G=20
Random
0.681
0.681
0.681
0.648
0.648
0.648
0.732
0.732
0.732
Router
0.690
0.690
0.689
0.657
0.657
0.657
0.737
0.737
0.736
HybridLLM
0.689
0.688
0.688
0.661
0.661
0.661
0.732
0.732
0.731
FrugalGPT
0,677
0.687
0.691
0.650
0.660
0.664
0.721
0.735
0.740
Margin Sampling
0.683
0.694
0.698
0.654
0.667
0.671
0.734
0.747
0.752
Table 4: Accuracy (AUC, averaged across datasets) under cost schemes c3
and varying
q.
Table shows the results of these experiments. Independently the cost, Margin Sampling appears the best cascading strategy (against FrugalGPT). In addition, we observe that Margin Sampling is the best overall strategy for Tcost 2 5; for fcost = 2, routing strategies could be preferred. Intuitively, cascading needs the cost of the small LLM to be relatively cheap enough to not sacrifice too many calls to the large LLM (pe < pr).
5.3.3 Mixing different families of LLMs
Margin Sampling can also be applied when the two LLMs are from different families; we further test its robustness with pairs Llama-2 13B Mixtral 8x7B, Mistral 7B Llama-2 70B. Our findings confirm the effectiveness of Margin Sampling 5
Discussion
In this paper, we have used a simple measure of confidence of the small LLM, known as Margin Sampling, estimate the uncertainty of LLM output in short-generation tasks, and we leave for future work the generalisation of this approach to long- gong-generation tasks. Our experiments show that Margin Sampling performs consistently well range of short-generation tasks relevant to researchers and practitioners, such as QA and multiple- choice/ classification tasks.
Existing approaches on the optimisation of LLM calls require training an auxiliary model. We hypothesised that these approaches introduce additional complexity that obfuscates their
Llama-2 13B
Mixtral 8x7B
Mistral 7B Llama-2 70B
Random
0.664
0.666
Router
0.673
0.688
HybridLLM
0.672
0.687
FrugalGPT
0,678
0.688
Margin Sampling 0.685
0.690
Table 5: Accuracy (AUC, averaged across datasets) when the LLMs are from different families.

---

Published as a conference paper at COLM 2024
understanding, and our findings that Lyb performs poorly with the OpenAl models lends credibility to this hypothesis. In contrast, we propose moving towards research that leverages information within the LLM itself. Classic notions of the uncertainty of the LLM's generation, such as perplexity, margin or entropy, could be relevant signals to help optimise LLM calls. A natural extension of our work is to generalise it for arbitrary task length, for which it may be that global notion of uncertainty also heavily depends on the first token. Our preliminary results with Machine Translation (Appendix D.5) indicate that some further adaptation of our method may be required for tasks where the first token may not be directly or partially the answer.
It was beyond the scope of this work to investigate cascading and routing strategies of three or more LLMs. However, we hypothesise our approach may still perform well, as Margin Sampling has shown to be robust to different-sized LLMs.
7 Conclusions
We have proposed a method for LLM call optimisation that achieves superior performance without the need of auxiliary model. To the best of our knowledge, the field of LLM call optimisation has not yet gained widespread adoption among practitioners. This may be due to the limited versatility and increased complexity of previous solutions. In contrast, our proposed simple approach can be easily and quickly implemented with most commercial LLMs. We believe that our findings could encourage a new direction in the research of LLM call optimisations.
Acknowledgements
We acknowledge Yumnah Mohamied for her help and discussions while writing this paper. GR is supported by the UKRI Centre for Doctoral Training in Natural Language Process- ing, funded by the UKRT (grant EP /SOS222481 /1), the University of Edinburgh, School of Informatics and School of Philosophy, Psychology & Language Sciences. IT acknowledges support by the Dutch National Science Foundation (NWO Vici VI.C.212.053). AB has been supported by the European Union's Horizon Europe Research and Innovation programme under Grant Agreement No 101070631 (UTTER). The computations described in this re- search were performed using the Baskerville HPC service, funded by the EPSRC and UKRI (grant SPP/T02221//1 and the Digital Research Infrastructure programme HBP/WW32244/11 This work also used the Cirrus UK National Tier-2 HPC Service at EPCC, funded by the University of Edinburgh and EPSRC (EP/ JEP/PO2000 PP202267/////////1 /1). The project that gave rise to these results received the support of a fellowship from "la Caixa" Foundation (ID 100010434, grant code LCF/BCL SDQ//U222/11999
References
Joris Baan, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. Uncertainty in natural language generation: From theory to applications. CoRR, abs/ abs/2307.15703. 2023. doi: 10.48550 ARXIV:2307.1500000 URL URI/htps:://ooi.rr//ee. 4ttp://110..55///iiiiixi 2307. 15703
10.4550/ARRXI.2207..555
Cenk Baykal, Khoa Trinh, Fotis Iliopoulos, Gaurav Menghani, and Erik Vee. Robust active distillation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL http:/ts:://eeeeeeiie.eteteeeeee
id=ALDM5SN2+7M
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models
10

---

Published as a conference paper at COLM 2024
are shot learners. In Hugo Larochelle, Marc'Aureliy Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tier Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL htthhttss: http:tttt///rroceeingggs neur ips. ccppaper/2820/ /ttp////000000004777777111111122
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Tina Eliassi-Rad, Lyle H. Ungar, Mark Craven, and Dimitrios Gunopulos (eds.), Proceedings of the Tweelfh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, Pp. 535-541. ACM, 2006. doi: 10.1145/ 1150402. 1150464. URL httpp:://di..rrg/// 11 145/1 1101/2. 1150464
Lingjaa Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving doi: 10.48550 0A00://AAXXX223005522222
CoRR, abt.ab//....51122/ 2023.
URL/ittts://b0i.orr///1 488500//4500/// http://awwwv.........
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024. URL
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Paylova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang,
11

---

Published as a conference paper at COLM 2024
Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching- Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Isimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Ilaaa 3 herd of models, 2024. //attps www.abs../bb.o. 21783
12

---

Published as a conference paper at COLM 2024
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the output of large language models via token-level uncertainty quantification. CoRR, abs/ 2400.04696 2024. doi: 10.48550/ ARXTV2440.00669 URL httphhtts//:/oioorrrr www.://///.....////.. arvvv 2403. 04696
Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowlledge Discovery & Data Mining, KDD '19, pp. 166-175, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362016.
doi: 10.1145/3292500
3330955, URL/http:://oo..rr/11... 1145///9..111/392250005 3330955
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Kr- ishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty URL
and beyond. CoRR, abs 1014, 2024. doi: 10.48550/AM ARX1V.2404.00136 104110///////
https://doi.oon org/10 10. 48550/ arviv. 1404.10166
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http:
//arxiv.org/abs/1555..00 02531
Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models. CoRR, abs/ 22tb.//230..mmm 10236, 2023, doi: 10.48550/ 10.48550/AKXI.2237.... ARX1V22300202222266 URL https:: htttps:/doi..org org/10 48550/ 2307. 10236
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew G. Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neu- ral networks for efficient integer-arithmetic-on
inference. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 2704-2713. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1109/CVPR 8.0008666. URL htttp://openacces: twwccf. cttpf/..co//oooetteetteeee cvpr 2018/ html/Jacob Quant html//aaob._uaniizztiiooo and Tra.nrrnngncCPRR2211100 Batr88ppppprtt_tt html
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/ /2 2023. doi: 10.48550/ 48550/arXiv.. 2310.06825
0044850/AARIIV231100 ARXIV.2310.06825 URL pettps:/ddoi.org
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, 0/////2410000000 2024. doi: 10.48550/ AR 004088 URL htLLhhtps:///dd or org/ 18550/arXiv 2401.04088
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pp. 79-86, Phuket, Thailand, September 13-15
2005. URL htRl:httts://ccaattthoo org/20000
2005. www.mmitsttttmmmmmmmmmm wappumittttpppstttt1mm 11.
Tom Kwiatkowski, Jennimmrrrr Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Lee, and Slav Petrov. Natural questions: benchmark for question answering research. Thans. Assoc. Comput. Linguistics, 7:452-466, 2019. doi: (A116//TACL_A URL/httss::/ooi.orgg// /ttp///..1162/tcccaa00222200
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara
13

---

Published as a conference paper at COLM 2024
Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 19274-19286. PMLR, 2023. URL https://proceeddnngs mlr
press//202/Leviathan222
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Ni- ladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. CoRR, abs 109110, 2022. doi: URL http:hhtt/iooo.ooo org/10....//e www.oo/////.///////...///// 48550//rXixv 2211 09110
1148550/ARXIV.22 1.09110.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou, Routing to the expert: Efficient reward-guided ensemble of large language models. CoRR, abs/ 2311 08699, 2023. doi: 10.48550/ T.XTV.2311.08692 URL. 48550/arXiv. 2311. 08692
Tong Luo, K. Kramer, S. Samson, A. Remsen, D.B. Goldgof, L.O. Hall, and T. Hopkins. Active learning to recognize multiple types of plankton. In Proceedings of the 17th International Conference on Pattern Recognition,, 2004. ICPR 2004., volume 3, pp. 178-481 Vol.3, August 2004. doi: 10.1109 /1t/://CCR.2204134555555 ISSN: 1051-4651.
Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang. Shyam Upadhyay, Mausam, and Manaal Faruqui. Automix: Automatically mixing language models. CoRR, abs/2310.12966 2023. doi: 215500/ARXIV2310122666 URL https://doi org//0../110 48550/ 2310. 12963
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, Brussels, Belgium, Oclober 31 November 4, 2018, pp. 2381-2391 Association for Computational Linguistics, 2018. doi: 10.18653 v1/ 126-1260 URL htttps:://oot org/10. 18653/v1/d18 1260
Jianmo Ni, Jiacheng Li, and Julian McAuley: Justifying recommendations using distantly- labeled reviews and fine fine-grained aspects. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pp. 188-197, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: (0.18653/v1/D19-100 org/D19-1018
URL httphtt/cccaantholllog
OpenAL. GPT-4 technical report. CoRR, abs/2303.087744 2023. doi: 10.4855//ARIIV.200.. 08774. URLhttps:://aa http:////6ooorr//...565/////////
333550/arXiv.. wtw.XXii...33.88744.
Bo Pang and Lillian Lee. Seeing Stars:
Exploiting Class Relationships for Sentiment Cate- gorization with Respect to Rating Scales. In Proceedings of the 43rd Annual Meeting of the
Association for Computational Linguistics (ACL (05),, pp. 115-124, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. doi: 223115/121944012199568 URL
https://aclanthoolog.. www.ooyyyyog//011011111
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. URL http:ttpss///cclattooooooo.../.p
ACI,, 2002. doi:
14

---

Published as a conference paper at COLM 2024
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNIIPIJJCNIP 2019, Hong Kong, China, November 3-7, 2019, Pp. 2463-2473, Association for Computational Linguistics, 2019. doi: 2.186533V11D19-122
URL https::///i..org///////// 11t.0o0/01111111111111111
Guillem Ramírez, Matthias Lindemann, Alexandra Birch, and Ivan Titov. Cache & distil: Optimising API calls to large language models. CoRR, 2bbb/223..1361 2023. doi: 10. 48550/ 02500/RRXV.2310.135552 URL httphhhtttpsooooo/ org/1 ww8.848555iiiii. 2310. 13561
Marija Sakota, Maxime Peyrard, and Robert West.
Fly-swat cannon? cost-effective language model choice via meta-modeling, CoRR, abs/ 2308.06077, 2023, doi: 10.48550/
ARXTV.2308.00777. /tt.httss://do.oooo org/ 48550/ 2308.06077
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.0110s 2019. URL
01108
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active Hidden Markov Models for Information Extraction. In Frank Hoffmann, David J. Hand, Niall Adams, Douglas Fisher, and Gabriela Guimaraes (eds.), Advances in Intelligent Data Analysis, Lecture Notes in Computer Science, pp. 309-318, Berlin, Heidelberg, 2001. Springer. ISBN 978-3-540- 44816-7. doi: 10.100773-5400100 00177/354-44481--.111
Bo Shao, Lorna Doucet, and David R. Caruso. Universality Versus Cultural Specificity of Three Emotion Domains: Some Evidence Based on the Cascading Model of Emo- tional Intelligence. Journal of Cross-Cultural Psychology, 46(2):229-25,, February 2015. ISSN 0022-0221. doi: 10.1177/0022000 /00220221145574799 URL https://doi.orr//e... 1177/ Publisher: SAGE Publications Inc.
0022022114557479
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, An- drew Y. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGD Special Interest Group of the ACL, pp. 1631-1642. org/D13-110/
ACL, 2013. URL
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: large-scale dataset for fact extraction and verification. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pp. 809-819, Association for Computational Linguistics, 2018 doi: 00.1863/v11/18-1100 https://doi.org: 10. 18653/v1/n18-100 1074
URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat 2307009288 2023. doi: 10.48550/ 00550/ARXIV.2300 ARXIV22307 .92888. URL https! //doi. www.lorg/1...m 48550/ xivv 2307 09288
Jeremmy
models. CORR, abs/ /
15

---

Published as a conference paper at COLM 2024
Can Wang, Bolin Zhang, Dianbo Sui, Zhiying Tu, Xiaoyu Liu, and Jiabao Kang. A survey effective invocation methods of massive LLM services. CoRR, abs/ /24020030000 2024. doi: AA0S550//AAXXV.402000000000
48550/arXiv. 2402. 03408
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. Towards ai-complete question answering: A set of prerequisite toy tasks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL httt::ttpp::iii http/aawwwiv.orrgr/////a//// 05698
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfth International Conference Learning Representations, 2024. /ttphtts://opperreiiee net/ffrrumiddsooaafFAhhhh
Jieyu Zhang, Ranjay Krishna, Ahmed Hassan Awadallah, and Chi Wang. Ecoassistant: Using LLM assistant more affordably and accurately. CoRR, abs 2310.03046, 2023, doi: URL/httss //doi org/10. 48550//rXiiv 2310. 03846
A
Implementation details
For methods Routing, HybridLLM and FrugalGPT we use Hug- gingface's auttooodelforssguuueeeesssssieee For methods Routing, HybridLLM we the target number of classes to two; for FrugalGPT,, we set it to either the number of target classes for classification, /multiple-choic problems or two for QA problems. We perform hyperparameter search (grid search) on a validation set of 500 examples of Openbook and Wikifact, and we find that the different methods have a similar convergence. We decide using learning rate = 5 x 10-4, training batch size m = 16 and weight decay = 0.01, which is consistent with the reported values of Saaota et al. (2023) and seems to generalise well across tasks. Ding et al. (2024), Chen et al. (2023) and Lu et al. (2023) do not report hyperparameter values.
and the model
ceilbber/ddistibee -bbeer--bae-unncseee
A.1 Training
Routing We generate answer using the small LLM. Then, compare it to the gold label. The target class is 1 or depending on the correctness of the answer from the small LLM.
Input: 'Who wrote 'If There Is Haven't Found It Yet?"
Output: '0'
Explanation: Llama-2 13B produces an incorrect answer.
HybridLLM We generate an answer using the small and the large LLMs. Then, we use the first token 1/1 of the gold answer V,, obtain the quality gap H(x) Psmall (1/1 x) Pragge (y1 x). Following Ding et al. (2024), we subtract the median of H(x); then we
map to 1 or 0 depending on H(x) Hmedian thus leaving binary classification problem of uniform classes
Input: 'Who wrote 'If There Haven't Found It Yet?'
Output: '0'
Explanation: Llama-2 13B is less likely than Llama-2 70B to produce the right answer.
FrugalGPT (classification and multiple-choice) We train DistilBERT with gold data. During inference, we generate an answer with the small LLM. The score is the probability DistilBERT associates to this class.
16

---

Published as a conference paper at COLM 2024
ISEAR RT-Polarity FEVER
CR
SST-2 Openbook Wikifact bAbl NaturalQ
Total datapoints
6132
8529
5289
3393
7000
5457
5733
3000
2876
Number of classes
2
2
2
4
QA
QA
QA
Table 6: Datasets used.
ISEAR
RT-Polarity
FEVER
CR
SST-2 Openbook Wikifact bAbI NaturalQ
Mistral 7B
0.557
0,862
0.770
0.911
0.854
0,813
0.359
0,560
0.125
Mixtral 8x7B
0.655
0.889
0.779
0.936
0.906
0.875
0.530
0.634
0.260
Llama-13b
0.599
0.798
0.613
0,902
0.867
0.556
0.416
0.525
0.212
Llama-70b
0.661
0.820
0.691
0.871
0.882
0.681
0.590
0.676
0.307
GPT-3
0.699
0.900
0.777
0.921
0.897
0.798
0.486
0.462
0.251
GPT-4
0.796
0.929
0.852
0.943
0.899
0.956
0.622
0.695
0.324
Table 7: Accuracy of the LLMs the studied tasks.
FrugalGPT (QA) We train a binary classifier that predicts if answer is correct. To do so, we use as the positive class the gold labels. We generate answers with Llama-2 13B and tag them as either positive or negative class depending on whether they match the gold labels.
Input: 'Who wrote 'If There Is Haven't Found It Yet? ANSWER: Anna Funder'
Output: '0''
Explanation: Anna Funder is an incorrect answer.
B
Datasets
Table 6 contains some statistics the datasets used. All the classification datasets uniformly distributed. For our experiments, we have reserved 1,000 datapoints for training the scorers (Router, HybridLLM and FrugalGPT) and used the remaining of the datasets for online inference.
C LLMs used
We load the open-source LLMs with Huggingface's
AutoMode1ForCaussilMM from.pretrained, activating load in_4bit. models reta-llama/LLamm221133 BustralliiMisttral----nnstsn
We
meta- meta-1llam/LLamm-22700-hh1
and mistralliiMMttrll-e//ennnnnttttt We set the temperature to 0 and look at the most likely token.
For ISEAR, RT-Polarity, FEVER, OpenbookQA, SST-2 and CR, we use the prompts from Ramirez et al. (023) (0-shot). For Wikifact, NaturalQuestions and bAbI we use the prompts from the HELM benchmark (Liang et al. 2022).
For the OpenAl models, has a cost of around $180.
davinci 002 (GPT-3) and gpt-4. Annotating all the datasets
Table 7 contains shows the accuracy of the LLMs across the different tasks.
17

---

Published as a conference paper at COLM 2024
Mistral
Llama-2
OpenAl
Random
0.718
0.682
0.777
Router
0.734
0.696
0.790
HybridLLM
0.725
0.688
0.774
FrugalGPT
0.739
0.706
0.791
Margin Sampling
0.739
0.705
0.794
Table 8: Accuracy (AUC) in the multiple-task setting. Methods Router, HybridLLM and FrugalGPT have been trained with 50000 datapoints.
ISEAR RT-Polarity
FEVER
CR
SST-2
Openbook
Wikifact bAbl NaturalQ
Random
0.630
0.809
0.653
0.885
0.873
0.617
0.505
0.600
0.259
Committee
0.620
0.797
0.591
0.886
0.863
0.587
0.473
0.538
0.234
Table 9: Accuracy (AUC) for the Committee method Llama-2 70B.
Yuu et al. 2024 for Llama-2 13B and
D Additional results
D.1 Multi-task setup
We experiment to see how far we can get with Routing, HybridLLM and FrugalGPT with more data. We train these methods with 5, 000 datapoints. We find that Margin Sampling still outperforms them in this setup (Table 8
D.2 Multiple calls the LLM
We experiment with the method from Yue et al. (2024), which estimates the uncertainty of the generation by doing multiple calls to the small LLM. We make calls, sampling with temperature T = 1. Our results (Table reveal this method does badly in our setup. The reason for this relies that doing the multiple calls to the small LLM is relatively expensive in our setup with CSmall CLarg
10.
D.3 Investigating the effect of the dynamic threshold
To investigate the effect of the dynamic threshold used our main experiments, we simulate offline setup, where can obtain the values for the different methods, order them and find the best threshold value. Table 10 reveals that these gold threshold values only lead to slim improvement; we conclude that dynamic threshold does not result in major performance inefficiencies.
Mistral
Llama-2
OpenAl
Random
0.681
0.648
0.732
Router
0.690
0.657
0.737
HybridLLM
0.689
0.661
0.732
FrugaIGPT
0.690
0.664
0.739
Margin Sampling
0.697
0.670
0.751
Table 10: Accuracy (AUC, averaged across tasks) for the offline setup. The slim differences to the online setup (Table 1 suggest that the dynamic threshold does not lead to inefficiencies.
18

---

Published as a conference paper at COLM 2024
Openbook
Wikifact
Mistral 7B Mixtral 8x7B
Random
0.848
0.453
Router
0.850
0.538
HybridLLM
0.851
0.491
FrugalGPT
0.846
0.498
Margin Sampling
0.865
0.497
Llama-2 13B
Llama-2 70B
Random
0.607
0.506
Router
0.613
0.532
HybridLLM
0.602
0.529
FrugalGPT
0.617
0.515
Margin Sampling
0.632
0.518
GPT-3
GPT-4
Random
0.872
0.556
Router
0.875
0.604
HybridLLM
0.883
0.559
FrugalGPT
0.876
0.577
Margin Sampling
0.914
0.588
Table 11: Accuracy (AUC). Methods Router, HybridLLM and FrugalGPT have been trained with 5, 000 data-points.
D.4 Experiments with additional data
We test the limits of our results when additional data is available for a particular task. We experiments OpenbookQA and Wikifact, which are two of the harder tasks, and train the supervised methods with 5,000 datapoints. T7 the results. The trend, as expected, is that supervised methods improve the performance, but Margin Sampling still has a relevant performance.
D.5 Longer generation (Machine Translation)
To test the effectiveness of Margin Sampling in a task that requires longer text generation, we run experiments on the Europarl Parallel Corpus (Koehn 2005), specifically the German- French direction. We use Llama-3.1 of size 8B and Llama-3.1 of size 70B (Dubey et al. 2024)). For this experiment setup, we find that Margin Sampling achieves 21.9 BLEU score (Papineni et al.] 2002), underperforming the random baseline (22.2 BLEU). We conclude that some further adaptation of our method required for tasks where the first token may not be directly or partially the answer. Our previous results with tasks generating multiple tokens, ie. Wikifact and NaturalQuestions, did show that Margin Sampling improves over the random baseline where significant portion of the answers started with stop words such as articles. Therefore, we hypothesise that individual token uncertainty -not constrained to the first token- may be suitable metric for the optimisation of API calls. This is validated by the results from Fadeeva et al. (2024), which found that token uncertainty may be used for hallucination detection, and by the work from Gupta et al. (2024), who trained neural model to aggregate the individual token uncertainties.
19


Reviews:
[{"review_id": "6Wcje6Ulj4", "reviewer": "Reviewer_Q76R", "summary": "This work aims at optimizing the LLM inference cost by running user queries through small models and switching to larger models when the performance is not satisfying. \nIn contrast to other work that trains router models to predict the poor small model performance, the paper proposes to use the margin token prediction as a metric to decide when to switch between small and large models.\nThe proposed method is compared to several supervised router approaches including Routing, FrugalGPT, Hybrid LLM, and random. The proposed margin sampling approach outperforms the other methods on most tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "In the multi-task setup, we observe that the same conclusions hold when the supervised methods are trained with 5,000 datapoints (Table 7 in the Appendix). \n\nIn addition, we have run experiments on OpenbookQA and Wikifact, where the supervised methods have been trained with 5,000 datapoints. The trend, as expected, is that supervised methods improve the performance, but Margin Sampling still has a relevant performance. \n\nMistral 7B - Mixtral 8x7B\n|  | openbook | wikifact |\n|---|---:|---:|\n| Random | 0.848 | 0.453 |\n| Router | 0.850 | **0.538** |\n| HybridLLM | 0.851 | 0.491 |\n| FrugalGPT | 0.846 | 0.498 |\n| Margin Sampling | **0.865** | 0.497 |\n\nLlama-2 13B - Llama-2 70B\n\n|  | openbook | wikifact |\n|---|---:|---:|\n| Random | 0.607 | 0.506 |\n| Router | 0.613 | **0.532** |\n| HybridLLM | 0.602 | 0.529 |\n| FrugalGPT | 0.617 | 0.515 |\n| Margin Sampling | **0.632** | 0.518 |\n\nGPT-3 - GPT-4\n|  | openbook | wikifact |\n|---|---:|---:|\n| Random | 0.872 | 0.556 |\n| Router | 0.875 | **0.604** |\n| HybridLLM | 0.883 | 0.559 |\n| FrugalGPT | 0.876 | 0.577 |\n| Margin Sampling | **0.914** | 0.588 |\n\n> What is the maximum number of examples that can be used for training the rankers? \n\nThere is no limit to the number of examples that can be used for training the rankers. However, our setup implicitly assumes a low-resource setting, i.e. if a practitioner has enough data, then they may directly prefer using a task-specific model like a classifier rather than using an expensive off-the-shelf LLM. In this spirit, the original papers that introduced the supervised models trained them on limited data [1-3]. For instance, [1] assumes less than 1,000 data points are available per task. \n\nWhile we have not explicitly studied the effects of context length, it must be noted that some tasks are generally longer. For instance, bAbI has around 250 input tokens per datapoint. The cost of Margin Sampling is independent of context length, so it would not be more expensive for longer contexts. \n\n[1] https://arxiv.org/abs/2308.06077\n\n[2] https://openreview.net/forum?id=02f3mUtqnM\n\n[3] https://arxiv.org/abs/2305.05176", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method does not require tuning a model to decide when to switch the prediction. \n- The results show that the method is efficient compared to the rest of the evaluated examples. \n- The method has been tested on several model families including Mistral, Llama, and OpenAI models through the API.", "reasons_to_reject": "- The supervised models trained for comparison seem rather weak. Only 500-100 examples are used for training which does not sound optimal. Moreover, the various models are trained on top of distilbert which is a very small model. Since the training cost is negligible, it makes sense to train the compared models on more data and with a much larger model. Running BERT or Roberta for classification would be much smaller than using the actual model. \n- The evaluation uses only short context tasks and it is not clear how the proposed approach would work on a longer context. One concern is that the margin sampling might not be super efficient in larger contexts.", "questions_to_authors": "What is the maximum number of examples that can be  used for training the rankers?", "extra_scores": {}}, {"review_id": "TMfMrbN1Cn", "reviewer": "Reviewer_VLt4", "summary": "The paper proposes a strategy for optimisation of LLM calls - a practical problem when making multiple API calls for solving a problem. There are two main strategies - a cascading strategy and a routing strategy. The strategy choice depends on a decision criterion in an auxiliary neural model.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">The strategies can be viewed as descriptions of practices that worked for a set of pairs and might not be sufficient for a research paper.\n\nWe disagree with the statement that the strategies can be viewed as descriptions of practices that worked for a set of pairs. We compared Margin Sampling against Šakota et al. (WSDM 2024), Ding et al. (ICLR 2024) and Chen et al. (ES-FoMo at ICML 2024). In addition, we compare against Yue et al (ICLR 2024) in Table 8. For the training of the auxiliary models, we follow the original papers as much as possible and perform a hyperparameter search where values are omitted. \n\n>Both small and large models contain billions of parameters. What would constitute the threshold for categorising a model as small or large?\n\nRegarding what constitutes a ‘small’ or ‘large’ model: model size has been dramatically increasing in recent years, deeming absolute thresholds as susceptible to becoming outdated. Instead, we opt to refer to ‘small’ or ‘large’ in purely relative terms, as implied in 3.1 Problem Definition. The practitioner would have access to two models of different costs, and in this paper we use ‘small’ as a synonym for ‘cheaper’. Since we don’t want to assume any size, we study a variety of parameter sizes for models (7b - Mistral, 13b - Llama, and GPT-3, rumoured to be 175b) and do ablations with multiple relative cost settings (Section 5.3.2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strategies might have the potential to be useful for someone started working with LLM API calls.", "reasons_to_reject": "The strategies can be viewed as descriptions of practices that worked for a set of pairs and might not be sufficient for a research paper.", "questions_to_authors": "Both small and large models contain billions of parameters. What would constitute the threshold for categorising a model as small or large?", "extra_scores": {}}, {"review_id": "OOpaZIhEgW", "reviewer": "Reviewer_KTCU", "summary": "This paper tackles the challenge of optimizing LLM calls to two LLMs that gap in parameter size with margin sampling for a better cost-performance trade-off. Specifically, the proposed method leverages uncertainty, defined as the probability difference of the most and the second most likely tokens, of the small LLM's first predicted token position with a statistically obtained dynamic threshold to decide if a secondary call of the large LLM is necessary. The evaluation conducted on a comprehensive set of short-generation tasks and the comparison with a comprehensive set of strong baseline methods warrant the effectiveness of a margin sampling based calling strategy.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors only conducted experiments on short-generation tasks citing the evaluation complexity of long-generation tasks, which leaves a common use-case of LLMs unevaluated.\n\nShort generation is a common use case of LLMs in research and practical settings, and we show that Margin Sampling is effective. We do not expect that it would work for longer text generation, where a more complex and expensive technique may be required. We will incorporate additional results and a discussion on long-form generation in the final version of the paper.\n\n> The paper also lacks adequate analysis and assessment of the dynamic thresholding strategy [...] How would the order of samples affect the effectiveness of dynamic thresholding?\n\nThe usage of a dynamic threshold is an artefact of the online nature of the setup. In preliminary experiments, we did not observe a substantial difference between the dynamic threshold and doing an offline evaluation (i.e., assuming we have the whole dataset, therefore we can order the values and find the ‘best’ threshold values). We present here these results, which are averaged across tasks. The following should be compared with the average column in Table 1. We observe that these gold threshold values only lead to slim improvements, and conclude that dynamic threshold does not result in major performance inefficiencies.\n\n|  | Mistral | Llama-2 | OpenAI |\n|---|---:|---:|---:|\n| Random | 0.681 | 0.648 | 0.732 |\n| Router | 0.690 | 0.657 | 0.737 |\n| HybridLLM | 0.689 | 0.661 | 0.732 |\n| FrugalGPT | 0.690 | 0.664 | 0.739 |\n| Margin Sampling | **0.697** | **0.670** | **0.751** |\n\nRegarding how the order of samples affects dynamic thresholding, this is accounted in our experiments (by running on three seeds).\n\n> Could you provide the variance or standard deviation for the reported results? \n\nThe values of the standard deviation of our experiments in general are much lower than the precision of the results that we report. The reason for this is that we report Area Under the Curve (AUC) across 100 budgets. You can find the values of the standard deviation for Table 1 at https://shorturl.at/W8aty. \n\n> This paper is a perfect fit for an industry track, but it might not be of very significant interest to the research community [...]\n\nWe want to note that several of the works we compare against have been published in conferences such as ICLR and WSDM; there has been a growing scientific interest in the problem.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is simple yet effective on short-generation tasks, and it doesn't rely on auxiliary models or additional data\n\n- Comprehensive experiments on short-generation tasks: benchmarked recent works on LLM call optimization research, conducted multi-task and robustness assessments, and reported the average three-run results", "reasons_to_reject": "- The authors only conducted experiments on short-generation tasks citing the evaluation complexity of long-generation tasks, which leaves a common use-case of LLMs unevaluated. \n\n- It seems that the threshold is a key factor of the performance, but this paper only conducted experiments with the dynamic thresholding strategy that relies on stats of seen samples. The paper also lacks adequate analysis and assessment of the dynamic thresholding strategy.\n\n- This paper is a perfect fit for an industry track, but it might not be of very significant interest to the research community as evidenced by the rejection reason of the FrugalGPT paper on OpenReview. I'm leaning towards an open mind on research topic but would like to keep a note here for the meta-reviewer's reference.", "questions_to_authors": "- Could you provide the variance or standard deviation for the reported results?\n\n- How would the order of samples affect the effectiveness of dynamic thresholding?", "extra_scores": {}}, {"review_id": "XhMMPczA58", "reviewer": "Reviewer_Zkq8", "summary": "The paper addresses the optimization of calls to Language Model (LLMs) within the context of having access to both a smaller and larger LLM. The primary research objective is to determine when to utilize the larger LLM for queries that cannot be adequately resolved using the smaller one.\n\nMain contribution of this work is the validation of Margin Sampling (Scheffer et al. 2001, Luo et al. 2004, Ramirez et al. 2023) as a criterion for selecting the larger LLM. Margin Sampling operates by assessing the difference in probabilities between the most likely and second most likely tokens according to the probability distribution of the smaller LLM for the first predicted token position. The paper further refines this threshold on a per-dataset basis with minimal examples.\n\nThe experimental evaluation focuses on various tasks including classification, multiple-choice selection, and short-text generation. Results from these experiments demonstrate the efficacy of margin sampling compared to several baseline approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Speculative decoding is not directly relevant to operational costs-saving approaches as it involves calling the target (large) model for each query in the verification step, which results in the same performance as the random baseline. \n\nWe will incorporate additional results and a discussion on long-form generation in the final version of the paper.\n\nRamirez et al. 2023 proposed using Margin Sampling in the context of a locally finetuned model. We show that Margin Sampling is a good router across off-the-shelf LLMs and outperforms recent works that use neural routers. Moreover, we extend results for short-form generation and a multi-task setup. \n\nLLMs of different families\n\nSince Margin Sampling only uses the probability distribution of the small model, we hypothesise that it should still be a reliable estimator for routing when using LLMs of different families. To validate this, we conduct additional experiments. The following results are in the same conditions as Table 1, and we show the average across tasks.\n\n| METHOD | Small: Llama-13b Large: Mixtral 8x7b | Small: Mistral 7b Large: Llama-70b |\n|---|---|---|\n| Random | 0.664 | 0.666 |\n| Router | 0.673 | 0.688 |\n| HybridLLM | 0.672 | 0.687 |\n| FrugalGPT | 0.678 | 0.688 |\n| Margin Sampling | **0.685** | **0.690** |\n\nHow crucial is the threshold?\n\nThe usage of a dynamic threshold is an artefact of the online nature of the setup. In preliminary experiments, we did not observe a substantial difference between the dynamic threshold and doing an offline evaluation (i.e., assuming we have the whole dataset, therefore we can order the values and find the ‘best’ threshold values). We present here these results, which are averaged across tasks. The following should be compared with the average column in Table 1. We observe that these gold threshold values only lead to slim improvements, and conclude that dynamic threshold does not result in major performance inefficiencies. \n\n|  | Mistral | Llama-2 | OpenAI |\n|---|---:|---:|---:|\n| Random | 0.681 | 0.648 | 0.732 |\n| Router | 0.690 | 0.657 | 0.737 |\n| HybridLLM | 0.689 | 0.661 | 0.732 |\n| FrugalGPT | 0.690 | 0.664 | 0.739 |\n| Margin Sampling | **0.697** | **0.670** | **0.751** |\n\nWhat is the standard size of dataset used to train the routing module?\n\nFrugalGPT do not report how many examples are used for training [1]. In [2], they assume less than 1,000 data points are available per task. \n\n[1] https://arxiv.org/abs/2305.05176\n\n[2] https://arxiv.org/abs/2308.06077", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The simplicity of the approach and its effectiveness in reducing average cost is significant. Because the method relies on the logits of the first predicted position, the cost of routing is small.\n\n2. Experimental validity on short-form generation task shows the usefulness of the proposed approach. Multi-task finetuning also shows improved results over the baselines which is significant.", "reasons_to_reject": "1. Lack of comparison with the related baseline such as speculative decoding (which also requires access to the token probability distribution as current method).\n\n2. All the evaluation is done for short-form generation tasks, what happens when the method is used for long-form generation such as paraphrase, translation, etc. Even if the results are negative, the discussion on long-form generation is crucial to understand the limitation of proposed approach.\n\n3. Limited novelty as the idea of using Margin sampling is proposed in the similar context by Ramirez et al. 2023.", "questions_to_authors": "1. How does the method perform on long-form tasks? A study on this limitation would be appreciated.\n\n2. What happens when LLMs of different families are used for margin sampling? Is it still a reliable estimator for routing?\n\n3. Since the threshold for routing is dynamic, what is the trend of accurate predictions in the beginning and the end? Does it improve with better thresholds from the previous examples? How crucial is the threshold?\n\n4. What is the standard size of dataset used to train the routing module as used in FrugalGPT and other baselines?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Pros:

* The proposed method aims at optimizing LLM inference calls by using a novel approach to cascading
* The paper does a thorough evaluation on several models and the results show that the method is an improvement over existing approaches.

Cons: 
* Adding an analysis of long form generation would significantly improve the paper

[At least one review was discounted during the decision process due to quality]
Decision: Accept