Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing
Authors: Dujian Ding
Ankur Mallick
Chi Wang
Robert Sim
Subhabrata Mukherjee
Victor Rühle
Laks V. S. Lakshmanan
Ahmed Hassan Awadallah
Venue: ICLR 2024 poster (2024)
Keywords: Large language models
Efficient ML
Query Routing
Primary area: general machine learning (i.e., none of the above)

Abstract: Large language models (LLMs) excel in most NLP tasks but also require expensive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of response quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths to save cost and maintain quality. Our approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade  quality for cost as per the scenario requirements. In experiments our approach allows us to make up to 40% fewer calls to the large model, with no drop in response quality.

Published as a conference paper at ICLR 2024
HYBRID LLM: COST-EFFICIENT AND QUALITY- AWARE QUERY ROUTING
Dujian Ding': Ankur Mallick2, Chi Wang", Robert Sim², Subhabrata Mukherjee Victor Ruhle", Laks S. Lakshmanan', Ahmed Awadallah
University of British Columbia Microsoff du j waijiannc.. df @g@ggmai i com
Hippocratic Al
ankurma 1 1 ck @mi crosoft.com
ABSTRACT
Large language models (LLMs) excel in most NLP tasks but also require expen- sive cloud servers for deployment due to their size, while smaller models that can be deployed on lower cost (e.g., edge) devices, tend to lag behind in terms of re- sponse quality. Therefore in this work we propose a hybrid inference approach which combines their respective strengths approach uses a router that assigns queries to the small or large model based on the predicted query difficulty and the desired quality level. The desired quality level can be tuned dynamically at test time to seamlessly trade quality for cost as per the scenario requirements. In experiments our approach allows make up to 40% fewer calls to the large model, with no drop in response quality.
cost and maintain quality. Our
INTRODUCTION
Large language models (LLMs) have become the dominant force in natural language processing in recent years (Zhao et al. 2023). Their impact has been especially striking in generative ap- plications where has extended beyond standard language understanding and guestion-answering
benchmarks like (Hendrycks et al. 2020 SSrivattvaa et al.) 2022) to several successful real-world deployments. These include the wildly popular ChatGPT (OpenAl, (b) and several other chatbots (Zheng et (003) powered by different LLMs (Taori et al. 2023 Touvron et al. 2023 OpenAl, 2023]. which allow users to engage in natural language conversations and obtain informative re- sponses range of practically useful tasks like creative writing, translation, code completion, etc. An important added attraction of these models is their accessibility. Users can input queries and receive responses in natural language, without any specialized data code, and this is what has created such a widespread demand for their services across regions, professions, and disciplines.
The best performing LLMs based the transformer architecture of (Vaswani al., 2017) and generally have tens of billions of parameters. E.g.. Alpaca (Taori et al. 2023) has 13 billion param- eters, the best version of Llama-2 (Touvron al. 2023) has 70 billion parameters, while OpenAl's GPT-3.5 (OpenAl and GPT-4 (OpenAl) 2023 are rumored to be much larger. Their enormous size and the autoregressive nature of text generation in their transformer architectures means that these models typically have a high compute and memory requirement that can only be met by ex- pensive cloud servers (Yu et al. 2022). This can potentially impose an enormous cost on developers and users as more LLMe- services are introduced. In response to this there has been surge of interest in designing smaller, cost-effective LLMs e.g.. (Touvron et al. (003)) provides multiple versions of Llama-2, with the smallest having only 7 billion parameters, small enough to run on
a Iaptor[] while the smallest offering of Google's ' Palm-2 model can even run on mobile d
However empirical evaluations in (Chung et al. 2022 Touvron et al. 2023) well as own evaluation in Figure Ia show that smaller models generally lag behind in terms of response quality.
"work performed during internship at Microsoft
1 work performed while Microsoft
'https://githubb.. http:/hhww..ccmmmmmooooooaaaaaaa

---

Published as a conference paper at ICLR 2024
Examples PP(M(X) -]]
llama-2 (13bb L: gpt-3.5-ttubbo
liama-2 (13b)
L: gpt-3.5-turbe
(T)
-3.0
100
BART Score (1)
2.90
All allat-small
-300
-35-
60-
-330
Random
Routing
40
3.20
fan-t5 (300m)
Ilama-2
lama-2
gpt-3.5
-2
-1
0.0
0.2
0.4
0.6
0.8
1.0
(7b)
(13b)
-turbo
Quality Gaps: H(0)-t
Cost Advantage
(a) Accuracy v/s size of LLM
(b) Tail of accuracy difference
(c) Results with routing
Figure 1: We use a dataset of natural language queries from a range of tasks like question answering,
summarization, information extraction, etc. (See Section 4 for details). We observe that (a) smaller
models generally give poorer response quality or lower BART score (Yuan et al.) [021], }) (b) LJama-2 (13b) outperforms GPT-3.5-turbo on around 20% examples, and (c) our router can make 22% fewer calls to GPT-3.5-turbo (cost advantage) with 1% drop in response quality (BART score).
Faced with this tradcoff between response quality and inference cost, we propose a hybrid inference approach which provides the best of both worlds. Our approach is motivated by the observation that most tasks for which LLMs are useful, like creative writing, translation, code completion, etc., include range of queries of different diffieulty levels and there is always a subset of "easy" queries for which responses of a small (inexpensive and weak) model may be comparable to, and sometimes even better than those of a large (expensive and powerful) model. This is also illustrated in Figure where we plot the tail of the quality gap (defined in Section 3) between the 13 billion parameter version of Llama-2 and OpenAl's GPT-3.5-uurb, the model that powers ChatGPT. Quality gap is non-negative for examples where the response quality of Llama-2 is comparable to or better than that of GGPT-3.5-turb which is the case for around 20% queries in our dataset (described in Section (
We leverage this insight to train a router that takes large model and a small model as input, and learns to identify these easy queries as a function of the desired level of response quality, while taking into account the generative nature of tasks, inherent randomness in LLM responses, and response quality disparity between the two models. At test time, the router seamlessly adjusts to different response quality requirements and assigns the corresponding "casy" queries to the small model, leading to significant inference cost reduction with minimal drop in response quality. In Figure Ic our router assigns 22% of queries to Llama-2 (13b) I with less than 1% drop in response quality measured in BART scores (Yuan et al., 2021). The gains are even higher for pairs where the small model is closer in terms of response quality the large model (see Section 4
With the explosion in the complexity and costs of LLM deployments, small companies and individ- ual consumers, have started to rely on the pre-existing LLMMs hosted on platforms like HuggingFace (HuggingFace) and OpenAl (OpenAl, c). In this context, our hybrid inference approach can reduce the costs incurred by both consumers and platform owners because a) consumers can use it to route easy queries to small models hosted on their edge devices (laptops/smartphones) and only call the API for the more complex queries (illustrated in Figure and b) platform owners can automatically route queries to lower cost models at the backend without affecting the user experience, as long as the response quality levels are maintained. Thus our hybrid inference approach offers a flexible and cost-effective solution for harnessing the full potential of LLMs while accommodating diverse cost budgets and quality requirements.
The main technical contributions of this work are: a) we the first to explore cost-effective and quality-aware hybrid LLM inference, b) we design a novel query router which routes queries based estimate of the response quality gap between models (Section 3.I.. c) we incorporate un- certainty due to randomness in LLM responses in our router design to improve performance (Sec- tion 3.2). d) identify challenges for our router when the small model is significantly weaker than the large model and introduce novel data transformation to address this issue (Section [33)), and e) we provide extensive experimental results (Section 4 on a large benchmark dataset of real world natural language queries and responses (Jiang et al. 2023) thereby demonstrating the value of the
We term the fraction of queries routed to the small model as the cost advantage (sec 2.3
2

---

Published as a conference paper at ICLR 2024
Query
Query
All-at- cloud:
Output
Output
Query
All-at-edge:
Outpuc
Output
Query
Routing:
Router
Output
Hard Queries
Easy Queries
Figure 2: Routing between edge and cloud.
approach and its superiority over baseline approaches, enabling LLM providers and consumers to cost-efficiently enable LLM-backed experiences.
PROBLEM FORMULATION
2.1 RELATED WORK
Large Language Models (LLMs).. The advent of LLMs has led to paradigm shift in the study of natural language processing (NLP), computer vision, information retrieval, and other do- mains(Menghhmi] 2023 Chen al., 2023 Jiang et al. 2023). The impressive effectiveness and gen- cralizability of LLMs has come at the price of a drastic increase in LLM sizes (Treviso et al. 2023) and consequent challenges, including huge amounts of computational resources and data required to train, and prohibitive expenses at both training and deployment stages (Bender et al. 2021).
Efficient Machine Learning (ML) Inference. LLMs belong to a class of models called foundation models (Bommasani et al., 2021) models that are trained once and can then be used to serve wide variety tasks. As such, we expect inference cost to dominate the overall cost of such models and hence focus on works that reduce the cost of MI. inference (Menghani 2023). Common techniques for efficient ML inference include model pruning (LeCun et al. 1989), quantization (Jacob et al, 2018), knowledge distillation (Hinton et al., 2015), and Neural Architecture Search (Elsken et al, 2019). Such static efficiency optimizations typically produce a fixed model with lower inference cost and lower accuracy compared to the large model which may not suffice for foundation models like LLMs, whose core premise is that the same model will serve a range of tasks, each with its own accuracy/cost constraints. This is already manifesting in inference platforms described in Section which need more dynamic optimizations to meet the demands of all users.
Hybrid ML Inference. Recent works (Kag et al., 2022 Ding al. 2022) have introduced new inference paradigm called hybrid inference which uses two models of different sizes instead of single model for inference. The smaller model (e.g. Llama-2 (Touvron et al. 2023 generally has lower inference cost but also lower accuracy than the larger model (e.g. GPT-4 OpenAl 2023). ). The key idea is to identify and route easy queries to the small model so that inference cost can be reduced while maintaining response quality. By tuning a threshold on query difficulty we can dynamically trade off quality and cost for the same inference setup. (Kag et al., 2022) study this setup for image classification and propose to train the small model, large model, and router from scratch. However LLM training is expensive and retraining LLMs from scratch for every scenario goes against the very premise of inference with pre-trained foundation models. Moreover text generation (Igbal & Qureshi 2022) is often more ambiguous and challenging than image classification due to which novel techniques are required for effective hybrid LLM inference for text generation.
Inference with Multiple LLMs. Some recent works use multiple LLMs for inference but these approaches typically call more than one L.M for single query that incur significant compu-
3

---

Published as a conference paper at ICLR 2024
tational overheads. Specifically (Jiang et al. 2023) calls ensemble of LLMs at inference time due to which the inference cost will be proportional to the number of models in the system. (Chen et al. 2023) performs inference by sequentially calling LLMs until one has high confidence score exceeding the predefined threshold. Our work provides high quality responses while always call- ing single L.LM for all queries and will thus incur much lower costs than both of these works on average. Speculative decoding (Leviathan et al. 2023] Kim et al. (023) speeds up decoding of expensive LLMs by invoking small decoders on the "easy" decoding steps. Instead, in our work we are interested in query routing which assigns "easy" queries to small models to reduce overall infer- ence costs while maintaining high performance. While these two approaches have different goals, an interesting line of future work would be to combine these to achieve further cost reduction.
2.2 PROBLEM SETTING
We extend the hybrid ML paradigm to LLM inference by routing queries between two models with different inference costs and accuracy. We use X' and E to denote the input query space and the set of all possible output responses respectively. Let L: X' - Z denote the large model and S : X' - 2 denote the small model. Formally, the objective in our paradigm is to learn router r X - {0, 1 such that each user query x E X is routed to the small model S(x) if r(x) = 0, and to the large model L(x). otherwise. Note that always route each query to single LLM at inference time as opposed to using an ensemble (Jiang et al.) 2023) or a cascade (Chen et al. 2023) of LLMs, which may call multiple LLMs to resolve single query and ineur significant computational overheads.
2.3 EVALUATION METRIC
Response Quality Automatic evaluation for text generation is a challenging and widely studied problem. Traditional metrics, such as BLEU and ROUGE, initially designed for machine translation and summarization, have been found to be of limited concordance with human judgment and re- stricted applicability across diverse NLP tasks (Blagec al., 2022). Significant research efforts have been devoted to implementing task agnostic evaluation metrics with neural networks. GPT-rrnking (aing et al. 2023), as a representative example, employs GPT models (c.g.. GPT-4 (OpenAl) 2023) )
to provide relative rankings between pairs of generated outputs. In spite of the high correlation with human perception GPT-ranking suffers from high computational costs and inability to distinguish between examples with the same ranking. Instead, we use the BART score (Yuan et al.) 2021 to
evaluate response quality of different models since (1) it is inexpensive to compute in comparison to LLM-based metrics such GPT-ranking, and (2) it has been shown in prior work (Jiang al. 2023) that this metric correlates well with the ground truth. We also provide a case study in Appendix C.2 to empirically justify using BART score as the response quality metric. We use (2).. 9 2 - l to denote the BART score (response quality) of model responses = E Z.
Cost Advantage The absolute costs of running model may not be known a priori, and may be expressed using a variety of metrics, including latency, FLOPs, energy consumption, etc. In LLM inference, however, each of these metries is affected by several underlying confounders such as different prompt templates, hardware capability, network connectivity, etc. Moreover different plat- forms/users may be interested in different metrics. However the common underlying assumption in this and previous works on efficient ML inference is that smaller models are more efficient than larger models and therefore we expect to obtain an improvement in all the metries by routing more queries to the smaller model. Hence we define cost advantage as the percentage of queries routed to the smaller model. Note that the notion cost advantage has been used as a generic efficiency metric in previous hybrid ML work (Kag et al. 2022), where it is termed as coverage.
HYBRID LLM INFERENCE
Easy Queries. We refer to queries for which the response quality of the small model is close to the response quality of the large model as "easy" queries. The goal of our hybrid inference framework is to identify the easy queries and route them to the small model thereby ensuring significant inference cost reduction without much drop in response quality. Note that the easy queries as defined here, need not necessarily be queries that are easylinexpensive respond to, they are just queries for
4

---

Published as a conference paper at ICLR 2024
which the small model can match up the large model. Examples of easy and hard queries as per this definition are provided in Appendix C.I.
Quality Gap. We define quality gap of a query as (x) - i.. the difference in quality the small model's response S(x) and the large model's response L(x). The quality gap is a random variable since LLM responses are typically non-deterministic. This is illustrated in Figure 3 below where the blue and orange plots correspond to the distribution of responses from FLAN-1 (800m) Chung et al. 2022) and Llama-2 (13b) (Tourron al. 2023 for single query.
Proposed Orchestration Framework. Queries are routed using BERT-style encoder model (e.g.. DeBERTa (He et al., 2020) which is trained on dataset of representa- tive queries and learns to predict a score. Since the router is an encoder model, single pass of the query through it sufficient to generate the score and we assume that the cost of this step is negligible compared to the cost of run- ning autoregressive decoding using the large model L(x) (Snn et al., 2019). Thus, we expect that using the router to route queries to the small model will not detract signifi- cantly from the realizable cost advantage.
flan-t5 (800m)
llama-2 (13bb
Ilama-2 (1.3b)
transformed w/ t=1.3
PDF
BART scores (T)
Router Score. We design the router score to be large for estimate of MV VI 0] is a suitable candidate since a large value of Pr(g(S(z)) > q([()))] corresponds queries for which there is a high likelihood that the response quality of the small model will be at least as high as that of the large model. However we show below that in scenarios where the large model is significantly more powerful than
easy queries as defined above. Intuitively,
Figure 3: Response quality distribu-
Pr(H(x)
tion for FLAN-15 (800m) and Llama- 2 (13b) on the query "How to iden- tiffy the index of median?" measured in BART scores. Llama-2 (13b) with transformation significantly overlaps with FLAN-15 (800m).
Pr(H(()) 0]
the small model i.e. (S)))) << ([[(x)) in general, one can train more effective routers by relaxing the definition of
easy queries to (x] 2 -t] Pr[g(S(x)) 2 q(L(x)) t] for an appropriate t > 0. At test time we achieve the desired performance accuracy tradeoff by tuning a threshold on the score and routing queries with score above the threshold the small model. For router with parameters w, we denote router score by Pw Pw : X + [0, 1]. We discuss different router score designs in the rest of this section assuming training set of N queries Ti,:..,
3.1 DETERMINISTIC ROUTER
Previous work on hybrid ML (Ding et al. 2022:
Kag et al., 2022) assumes that neural models are deterministic functions mapping input features to a single point in the output space. To realize this for LLMs, we sample a single response per query from each model. We assign boolean labels
Vaae 1 (()())))) 2 (Lxx))) to each training query x; with the BART score as the quality function
(()) The router trained by minimizing the binary cross-entropy loss (Ruby & Yendapalli 2020).
1 ((w) = N
Σ
ys
log(pno (x:)) + (1 yl log(1 - p (zx))
(1)
Observe that the assigned labels v can be viewed an estimate for Pr (x;) 2 given single response per query from each model and thus minimizing the above loss encourages the router score Pw (x) be close to Pr[H (x) 2 0] for test queries. We refer this deterministic router as Idet-
3.2 PROBABILISTIC ROUTER
The determinism assumption can be justified for tasks where the ground truth labels are often explicit and unique such as image classification (Masana et al. 2022) and video segmentation (Yao et al. 2020). When it comes to NLP tasks, however, there usually no single best answer due to the intrinsic ambiguity and complexity of natural languages. LLMs are widely used as non-deterministic generators to capture the intrinsic uncertainty of NLP tasks, as shown in Figure 3 (ignore the dashed
We use the FL-AN-t5-large model from https///ttgiingaecccogggo///n//
5

---

Published as a conference paper at ICLR 2024
flan- (800m) L: Ilama-2 (13b)
flan-t5 (800m) L: llama-2 (13b)
tlan-t5 [800ml
L: Ilama-2 (1.36)
100
90
Refore Transformaion
Mean Years Diff.
0.-
80
After Iranstormation
Piamples
Examples
0.1
20
1=0
1:1.3
0.1
02.2
1.44
0.
0.8
153
110
0.2
9.0
0.5
1.
Pr[H(X
D]
Pr(H[X]
-t].
1.3
(a) Before transformation.
(b) Grid search for the best t.
(c) After transformation.
Figure 4: Effect of data transformation on labels for training the router.
curve for now). The non-determinism mainly comes from the randomness in the decoding phase. Users typically control the level of uncertainty by choosing different decoding strategies such as nucleus sampling (Holtzman et al. 2019). well as the values of the hyper-paranen Intuitively, higher temperature values result in a higher level of randomness and diversity among the generated responses. For black-box LLM APIs such GPT-4 (OpenAl 2023), it has been observed that even upon setting temperature to the minimum value O, it can still provide different responses for the same input queries. The underlying mechanism is still an open problem while recent study hints at the instability of the MoE backbone (Skyward (023).
temperature.
We propose to incorporate the uncertainty into the router training loss by relaxing the hard la- bels yfta E {0, 1} to the soft labels y (r; 2 0] Pr[g(S(z,)) ((L(z)))] E[][g(((xz)) 2 q( (((x)))]] where E denotes the expectation. In practice, we estimate expecta- tion by sampling 10 responses from each model and computing the sample average of the corre- sponding indicator function values. Observe that the hard label y is a sigher-variance estimate of [1 (xx() 2 ((((x)))]] (since it is obtained from single sample) and hence we expect im- proved performance of the probabilistic router with the following training loss, referred to as r Troob-
((w)
1 N
mmah log(pud (x:)) + (1 2. )og" log(1 pm(
3.3 PROBABILISTIC ROUTER WITH DATA TRANSFORMATION
While so far we have designed scores that try to estimate Pr(H(x) 2 0]. we observe that the empirical estimate of Pr[ 2 0J ([(()))(z))) > ((((x.))]] tends to be extremely small when the large model is significantly more powerful than the small model (0 for almost 90% of the queries in Figure 4a with Flan-t5 (800m) as the small model and Llama-2 (13b) the large model). Because (z)) << q (((()) for most queries in this case, it provides an extremely weak signal and as shown in Section 4 both Tdet and rprob fail to provide much improvement over random query assignment in this case.
for training using Equation
Traditional approaches for learning with imbalanced data have their shortcomings (Krawezyk, 2016). Moreover our goal is to only design a router that can reduce inference cost while maintaining particular definition of class labels
response quality as much as possible and so we are not tied to achieve this. We leverage this flexibility to introduce new labels y (t) == (r[H)(x;) 2 -I] Pr(g(S(x:)) > (zx)) t] for some O. Since -t < O, (x) 2 -t] 2 Pr(H(x) 2 0 by definition of the tail distribution and so we expect this relaxation to provide stronger signal for router training while still allowing us identify the easy queries i.c. those queries for which
q(S(z)) has a high likelihood of being close to q( (T((x))) (q(S(=)) > (([xx)) t). Visually, this corresponds to comparing the distribution of the small model's response with shifted distribution of the large model's 's' response to a query (dotted curve in Figure] 3)
Now the question is how to choose the best relaxation t? Given that tail probability Pr[H(() 2 -f lies in [0. 1], we choose by maximizing the average pairwise differences between the transformed labels to push them as far apart as possible and provide a strong signal for training. Thus we set,
arg max
yi
(t)
sr0n
(t)
(3)

---

Published as a conference paper at ICLR 2024
Table 1: Cost advantage v.s. performance drop for model pairs of different performance gaps. Performance drops are computed w.r.t. the all-at-large baseline.
Response Quality (BART Score) Drop w.r.t all-at-large (%)
Cost Advantage (%)
S: Llama-2 (7b)
S: -lama-2 (13b)
S: FLAN-15 (800m)
L: Llama-2 2 (13b)
L:: GPT--35-turkh
L: Llama-2 (13b)
Tdet
Tprob
Ttrans
Fdet
Fprob
Ttrona
Tdet
Pprob
Ttrans
10
0.1
-0.1
0.1
0.1
-0.1
0.2
2.3
2.2
2.1
20
0.1
0.0
0.0
1.0
0.8
0.8
5.8
5.8
4.7
40
0.2
0.1
0.0
3,5
3.4
2.9
13.8
13.1
10.3
We currently solve the above optimization problem via grid-search and leave more sophisticated approaches for future work. We plot the optimization objective for different values of for our training dataset in Section and show the distribution of transformed labels y (t*) in Figure As see, the distribution significantly more balanced now and expect the resulting router be much more effective. We train the router by minimizing the loss, referred to as Ptrans-
((w)
Σ (y 1Tm (1") log(Pio (x:)) + (1 Bi 1rm (1")) log(1 pw(())))
(4)
N
EVALUATION
4.1 EVALUATION SETUP
Dataset. We use the Mixinstruct dataset from (Jiang et al. 2023) to evaluate the effectiveness of different routing strategies. Mixinstruct consists of a wide range of tasks (e.g., question answering. summarization, information extraction) and enables us to train a generic router that will be effective
across different scenarios. We present additional information about this dataset in Appendix
Router Model. We use DeBERTa-v3- routers. We train each router with the corresponding loss from Section 3 for 5 epochs and use the validation set to choose the best checkpoints for final evaluation. All experiments ducted with NVIDIA A100 GPU of 80GB GPU RAM. We have made our source code available http://tthub..oom/33555...oo borr//////booorddddodoo Ilm routing.
(He et al. 2020) (300M) as the backbone to train our
Evaluation Measures. We use BART score Yuuan et al. 2021 as the quality metric and use fraction of queries routed to the small model (cost advantage) the efficiency metric (see Section 2.3
Baselines. We consider three baselines: all-at-large, all-at-small, and random. All-at-large routes all queries to the large model, while all-ar- small routes all queries to the small model. Random gen- erates random number in [0. and selects the large model if it is below the probability threshold.
Experiments. We investigate all three routers: Fdct, T Pprob. and Ptrans- We select candidate model pairs from FLAN-TS (800m). FLAN-T5 (11b), Llama-2 (7b), Llama-2 (13b), and GPT-3.5-turbo for our experiments. At test time the trained router (F'det. Fprob. or Ttrans) takes a threshold value as input and routes all queries with router score higher than the threshold to the small model as these are the easy queries. We evaluate the router performance in Section 4.2 in terms of both BART score and cost advantage (Figure [S] and Table [ validate that the router is indeed routing easy queries to the small model in Section 4.3 demonstrate that our routers are of negligible compute overhead in Appendix A.1 show how to choose routing thresholds in practice in Appendix A.2 evaluate the effectiveness our routers using a response quality metric other than the BART score in Appendix A.3 and test the generalizability of routers across model pairs in Appendix A.4
4.2 ROUTER PERFORMANCE RESULTS
Small performance gap. LLMs of the same architectures are observed to of small performance gap such as 1lama-2 (7b) Llama-2 (13b), seen in Figure 5a In this case, by trading little to
7

---

Published as a conference paper at ICLR 2024
liama-2 (7b)
llama-2 (13bb
flan-15 (600m) (13b)
gpt-3.5-turbb
Examples %r: PM[H())
100
Examples %%: PX[H(X)
100
Examples %). Pr(H(x)
60-
80
80
40
20
20
20
-2
Quality Gaps: H(X)=t
Quality Gaps: H(x)=t
Quality Gaps: HDX=t
S: 1ame-2 (7bb
S: llama 2 (13b)
S: flan-t5 (800m)
Ilama-z (13b)
-ama-2 (1.35)
BART Score (f)
BART Score [T]
BART Score (?)
-..00
-2.32
Alaalsamal
-3.00
All-a--mm
Nial--mall
Allat-large
-3.10
Rondom
-.00
-3.34
Edet
-3.70
Edee
Loet T. Larrb
$5.50
Ctrans
0.0
0.0
0.
1.0
Cost Advontage
Cost Advantage
Cost Advantage
(a) Small performance gap
(b) Medium performance gap
(c) Large performance gap
Figure 5: Error-cost tradeoffss achieved by Idet. Tprob. and Ttrans for different performance gaps.
no performance drop, we show that (1) the deterministic router Idet can achieve good cost advan- tages, (2) Fprob consistently improves ddet, and (3) Ttrans is able match or slightly improve the performance of Tprob- Numerical comparison results are summarized in Table [ Tdtt routes 20% (40%) queries the small model i.e. Llama-2 (7b) with only 0.1% % (0.20 (0.2%) drop response quality w.r.t. the all-at-large baseline. Impressively Tprob and Itrans achieve 20% cost advantages without any quality drop, and Ftrans is able achieve even 20% cost advantage without quality drop, which can be attributed to these methods capturing the non-deterministic nature of LLMs.
Medium performance gap. Often there is only a moderate performance gap between leading open- source LLMs like Llama-2 (13b) and tfate-of-the-ar commodified LLMs, such as GPT-3.5 -trrbo (Figure/5b) In this case, all our routers deliver reasonable cost advantages with acceptable quality drop. The effectiveness order between l'det, Tprrb, and Ptrans resembles that in the small quality gap case. All routers achieve 20% (40%) cost advantage with S 1% (S 4%) quality drop (Table [ In the 40% cost advantage regime, Tprob slightly outperforms rdet and Itrona improves Fprob by 0.5% in terms of quality drop.
Large performance gap. In the edge-cloud routing scenarios, edge devices often have very limited resources and can only support small models of limited quality, which can be significantly outper- formed by large models deployed on the cloud. We investigate how to effectively route queries with LLM pairs of large performance gaps, such as FLAN-15 (800m) and Llama-2 (13b) (Figure (5c). Non-trivial routing is challenging in this situation since the large model dominates for majority of examples. Both Tdet and rprob perform marginally better than the random routing baseline. In contrast, Ttrans can still effectively distinguish relatively easy queries from the harder ones. Ttrans achieves 40% cost advantages with 10.3% quality drop, which is 3.5% and 2.8% lower than Tdet and pross respectively [Table[]:
In the course of these experiments we made two interesting observations. Firstly, when the cost advantage is modest (e.g., 10%) and the LLM performance gaps are not large (e.g.. Llama-2 (7b) v.s. Llama-2 (13b)) prob is able to achieve even better performance than all-at-large which leads to the "negative quality drops" in Table This because, as seen from the large value of (r(H(()) 2 0) in the tail distribution in Figure 5 the response quality of the small model may be higher than that of the large model for several queries and by routing these queries to the small model, the router is able to even beat all-at-large. Secondly, for lower cost ad advantages (S 10%) and small or moderate LLM performance gaps Ttrans can be slightly outperformed by Tdet or Tprob- This might be due noise in the estimation of the relaxation parameter from sample averages instead of expectation in Equation (3) and from the grid search process leading to suboptimal settings of Ptrans. However we clearly see that in more challenging routing scenarios with high cost advantage targets large I.I.M
8

---

Published as a conference paper at ICLR 2024
liama-2
fian-tt
Avr. Quality Gap Diff.
Namm-7 (13h)
Avg. Qua ity Cap Diff.
L: gpt-1.--turbe
lama-2 (1.3b)
0.75
Random Ituulin
Random Pouring
2,00
Aug. Qua ity Gap
0.50
2.00
0.50
0.00
0.00
0.00
Rondom
-0.25
0.50
-.00
0.0
02
0.6 Cost Advantage
0.4
0.0
1.0
0.0
0.2
0.4
0.6
0.6
1.0
0.0
0.2
6.4
0.
0.0
1.0
Cost Advantage
Cost Advantage
(a) Small performance diff.
(b) Medium quality gaps.
(c) Large quality gaps.
Figure 6: Difference between average quality gap of queries routed to the small and large models with different performance gaps.
performance gaps, both Tdet and Fprob have difficulty in correctly routing queries, and Ttrons starts to dominate due to the benefits of the data transformation.
4.3 ROUTER VALIDATION RESULTS
We also validate that the router is functioning intended, that is, routing easy queries to the small model and hard queries to the large model. To see this, in Figure 6 we plot the difference between the average quality gaps of queries routed to the small model and those routed to the large model for our router and the random baseline v/s different values of cost advantages (i.c., the fraction of queries routed to the small model). Since the random baseline randomly assigns queries the average difference is nearly always zero. However our router routes easy queries i.e. queries with large quality gap (1()(::) q( (L)))) to the small model and queries with small quality gap to the large model. Hence the difference between the average quality gaps always has a significant positive value indicating that more easy queries are routed to the small model than to the large model in our approach as compared to the random assignment approach at all cost advantages.
DISCUSSION AND CONCLUSION
Motivated by the need to optimize the trade-off between LLM inference costs and response quality, we have presented hybrid inference approach based on quality- awarre query routing. We train router to discriminate between "hard" and "easy" queries, enabling the LLM provider to make cost- efficient decisions about which model should serve given query. Our experimental results on a variety of sate--o--the-at art LLMs of varying sizes show that such optimization is possible and that we can realize cost advantages of up to 40% with no significant drop in response quality.
To the best of our knowledge, this is the first work exploring the possibilities of cost-effective and quality-aware query routing between LLMs. We identify several important extensions for future work: (1) Task-aware routing. Our current routers make routing decisions purely based query inputs. To improve routing effectiveness, we can provide more informative signals which help routers distinguish easy queries from the hard ones, such as task labels for query examples and can also identify tasks which may be more suited to routing for a given pair of LLMs. (2) Generalizing to N-model routing. Modern MLaaS platforms typically host a large number of LLM instances of the same or different configurations to efficiently serve users in different scenarios. This natu- rally forms a more challenging routing problem with richer optimization opportunities (e.g.. load balancing) (3) Out-of-distribation (OOD) generalization. In this work, the model pair and data distribution is fixed across training and testing. In the real-world it may be cumbersome/infeasiblle to train new router for every new model pair and for every new data distribution. Therefore there is need for techniques to generalize our approach to changes in the model pair and/or data distribution at test time. (4) Novel evaluation metrics. Effective evaluation metrics critical to train high- quality routers. It is intriguing to see how to develop metrics higher tuman-judgman correlation and to which extent will improve the routing performance.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors would like to thank Daniel Madrigal Diaz, Mirian del Carmen Hipolito Garcia, Chen Dun, Guoqing Zheng, Menglin Xia, Wen Xiao, and Jieyu Zhang for helpful discussions.
REFERENCES
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majon and Shmargaret Shmitchell. On the dangers of stochastic parrots : can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Trunsparency, pp. 610-623, Virtual Event Canada, March 2021. ACM. ISBN 978-1-4503-8339-7, doi: 10. 00.114/344418883459922 URL
https:///l acm. wwt/ 1tt.:/1...111434444 3445922
Kathrin Blagec, Georg Dorffner, Milad Moradi, Simon Ott, and Matthias Samwald. A global anal- ysis of metrics used for measuring performance in natural language processing. arXiv preprint arXiv:2204.115777 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2108.07258. 2021.
Harrison Chase. LangChain, October 2022. URL Langchain
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05//00 2023.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling iustruction-fintuune language mod- els. arXiv preprint arXiv:2210. /4416 2022.
Dujian Ding, Sihem Amer- Yahia, and Laks VS Lakshmanan. On efficient approximate queries over machine learning models. arXiv preprins arXiv:2206.02845, 2022.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, The Journal of Machine Learning Research, 1991)11977-2017, 2019.
Pengcheng He, Xiaodong I. Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 00:::000090033000 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint 1503.0050 2(7), 2015.
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. arXiv preprint 19Xi:1194.00951,, 2019.
HuggingFace. Hugging face inference api. hatps:///oogiiniioe...e////eeeee
Touseef Iqbal and Shaima Qureshi. The survey: Text generation models in deep learning. Journal of King Saud Ciniversity-Comuutee and Information Sciences, 34(6):2515-2528 2022.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-riitmmeicconnty inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704-2713, 2018
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Alm-blender Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023.
10

---

Published as a conference paper at ICLR 2024
Anil Kag, Igor Fedorov, Aditya Gangrade, Paul Whatmough, and Venkatesh Saligrama. Efficient edge inference by selective query. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Sehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik, Michael W Mahoney, Amir Gholami, and Kurt Keutzer. Speculative decoding with big little decoder. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
Bartosz Krawezyk. Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5(4):221-232, 2016.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, G-eval: Nlg evaluation using gpt-4 with better human alignment, may 2023. arXiv preprint arXiv:2303. 2023.
Mare Masana, Xialci Liu, Bartlomicj Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation image clas- sification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Gaurav Menghani. Efficient deep learning: A survey on making deep learning models smaller, faster, and better. ACM Computing Surveys, 55(12):1-37,,02 2023.
OpenAL Openai gpt-3.5-turbo.
ggt- 3-5,a.
OpenAL. Chatgpt. htttps://chat.openaai b.
OpenAl. Openai platform.
OpenAl. Gpt-4 technical report, 2023.
Usha Ruby and Vamsidhar Yendapalli. Binary cross entropy with deep learning technique for image classification. Int. Ady. Trends Comput. Sci. Eng. 9(1)), 2020.
Luke son
Skyward. why,
Gpt-4 Aug 2023.
non-deterministic and moe is the likely rea- URL https://ai.pp) pen.nngginnnngne..om ish io/
gpt 4- is non- weteeeiiisstii--eee. 2
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint Ttl.:2206604615.. 2022.
Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. Fast structured decoding for sequence models. Advances in Neural Information Processing Systems, 32, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An -ostuuctionfoolooiing 1lama model. https::/githinn com/ tatsttttuuuute caatsu-lab/ae st an 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint (iXii:3307.09288 2023.
Marcos Treviso. Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, et al. Efficient methods for natural language processing: A survey. Transactions of the Association for Computational Linguistics, 11:826- 860, 2023.
11

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Rui Yao, Guosheng Lin, Shixiong Xia, Jiaqi Zhao, and Yong Zhou. Video object segmentation and tracking: A survey. ACM Trunsactions on Intelligent Systems and Technology (TIST), (14):1447, 2020.
Gy Yu, Joo Seong Jeong, Goo Kim, Soojeong Kim, and Byung- GoungGGnnnn Gon Chun. Orea: A distributed serving system for (Transformer-Basel generative models. In 16th USENIX Sympo- sium on Operating Systems Design and Implementation (OSDI 22), pp. 521-538, 2022.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text gener- ation. Advances in Neural Information Processing Systems, 34:27263-2723 2021.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt- at-bench and chatbot arena. arXiv preprint arXiv:2306. 05685, 2023.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENTS
A.1 ROUTER LATENCY
We measure the latency of our router and compare the latency of the different LLMs Flan-t5 (800m), 2 (7b), and Llama-2 (13b) that we use in our experiments for generating responses to user queries. Note that the lateney of the routers Tdde, Tprab, and Ttrans will be the same since they use the same model (DeBERTaav3-large (He et al., (020)) and are just trained differently. Also, we do not measure the latency of GPT-3.5-turbo since its responses are generated by querying the OpenAI API (OpenAl C) as the model weights not publicly available due which it is not possible to disentangle the inference latency from the network latency, queueing delay, latency of the API call, etc. However we note that the inference latency of all other LLMs we consider is significantly larger than that of the router (see T and therefore we expect the same to hold for GPT-3.5 -.trrb--turbo as well.
The latency results are reported in 2 where we measure the average latency per query averaged over 200 randomly chosen queries from our dataset (confidence bounds correspond to one standard error). As expected the router processes queries significantly faster than all the LLMs (nearly 10x faster than the fastest LLM FLAN-15(800m). This is both due its smaller size (300m param- eters) and the fact that it performs a single forward pass over the query to generate the score while the LLMs generate the response coken-by-toke in an autoregressive fashion due to which the infer- ence latency is proportional to the response length. Thus the router adds minimal overhead to the inference cost due to its small size and extremely low latency.
Table 2: Latency Values for Different Models.
Model
Latency (seconds)
Router
0.036 0.002
FLAN-t5 (800m)
0.46 + 0.039
Ilama-2 (7b)
7.99 0.15
Ilama-2 (13b)
14.61 + 0.27
A.2 EMPIRICAL DETERMINATION OF ROUTING THRESHOLD
Recall that at test time the model owner is required set threshold on the router score which serves separate the easy queries from the hard ones (see Section 3). All queries with router score higher than the threshold will be routed to the small model. Thus the threshold is a user-defined trade-off, best serve the interests of
parameter controlling the achieved different users. In this section we show how to empirically choose thresholds on router scores to achieve cost reduction with little to no performance drops. For this, we use small calibration set to recommend default thresholds to users. We investigate all three routers Tdets Tprob, and Itrans with different LLM pairs that we use in our experiments. For each LLM pair, we randomly draw 500 samples from the validation set and use grid search to determine the threshold that delivers the highest cost advantages i.e., cost savings on the validation set while keeping the performance drop (reduction in BART score) less than 1%. The limit on performance drop can be adjusted as per user requirements. With the selected thresholds, we report the achieved performance drops and cost advantages on the test sets, as summarized in Table 3
As seen from the table the performance and the cost advantage obtained on the test sets closely follows that on the validation sets for all categories of LLM pairs. This clearly illustrates that threshold chosen on the validation set generalizes well to the test set. We note that there is a slight increase in the performance drop from the validation to the test set for the LLama-2 (7b) and Llama- 2 (13b) pair, i. the LLM pair with small performance gap as per the categorization in Section 4 However this is also the pair with the highest cost advantage cost savings (> 96% for all routers) and thus the issue can be addressed by just using a more conservative limit on the performance drop while choosing the threshold which would still lead to very significant cost savings.
13

---

Published as a conference paper at ICLR 2024
Table 3: Test performance drops v.s, cost advantages achieved by thresholds chosen from 500 validation samples with I 1% sampled performance drops.
S: Llama-2 (7b)
S: Llama-2 (13b)
S: FLAN-15 (800m)
Router
L: Llama-2 (13b)
L: GPT-3.5-turbo
L: Llama-2 (13b)
Perf. Drop
Cost Adv.
Perf. Drop
Cost Adv.
Perf. Drop
Cost Adv.
Val.
0.99%
98.20%
0.97%
15.20%
0.77%
5.40%
Tdet
Test
1.60%
98.56%
0.55%
15.15%
0.69%
4.89%
Val.
0.92%
97.60%
0.56%
8.60%
0.70%
5.00%
Fprob
Test
1.42%
96.80%
0.11%
8.38%
0.57%
4.44%
Val.
0.79%
96.00%
0.77%
17.00%
0.92%
4.00%
Ttrans
Test
1.39%
96.45%
0.49%
15.68%
1.02%
5.05%
Ilama-2
liama-2 (13b)
flan-t5 (800m)
gpt-3.5-turbo
llama-Z (13b)
9.00
GPT4 Score (1)
small All-ab-large
5.00
Al-at-small
GPT-4 Score
8.00
Random
GPT-4 Score
Al Al-atsmali
Allat-large
proo
ddett
4.00
Random
All Al-ab-large
5.20
dede
Eproo
r.trans
3.00
00
0.2
0.6
0.5
3.0
0.2
0.4
0.6
1.0
0.0
0.2
0.6
0.
Advantage
Cost Advantage
Advantage
(a) High correlation (r 0.46, 0.44).
(b) Medium correlation (r
(c) Low correlation (r 0.26, 0.27).
0.38, 0.38)
Figure 7: Routing performance evaluated with GPT-4 scores. Pearson (r) and spearman (p) correla- tion coefficients between quality gaps measured by BART score and GPT-4 score are computed for each LLM pair.
A.3 ALTERNATE EVALUATION METRICS
To provide more comprehensive evaluation of our routers, we test the routing performance with metries in addition BART score (Yuan et al. 202T). GPT-4-based evaluators have been found
to be well correlated with human assessments (Liu al. 2023: Chase 2022). We generate GPT-4 evaluation scores (integer ratings from to 10) for test responses from Flan-t5 (800m), Llama-2 (7b), Llama-2 (13b), and GPT-3.5-turbo that investigate in experiments, using LangChain scoring evaluator (Chase 2022). Recall that our routers are trained with BART score due to efficiency and effectiveness reasons discussed in Section 2.3 Intuitively, if the quality gaps measured by BART score and GPT-4 score are highly correlated, we could expect good routing performance even under the GPT-4 score we have seen Section 4.2 We compute the correlation between quality gaps measured by BART score and GPT-4 score, and report it along with routing performance evaluated with GPT-4 score, as shown Figure
Aligned with our intuition, when the two metrics are well correlated (Figure 7ap. our routers trained with BART score are still effective even when evaluated against GPT-4 score. Typically, Tdets Pprobs and Ptrans are able to achieve 20% cost advantage with up to 1% performance drop, and 40% cost advantage with up to 2.1% performance drop. As the correlation gets weaker, the router performance gradually decays, strategy of using BART score practice to save labelling costs while maintaining routing perfor- mance. We first compute the correlation between BART score and the target metrics (e.g., human assessments) using a small sample and use BART score as training labels whenever there is strong positive correlation with target metrics.
shown in Figure 7b and 7c This observation suggests a simple-yet-effectie
14

---

Published as a conference paper at ICLR 2024
Train: S: lama-2 (135) laama-2 5pt-3.--turbo
Train: S: flan-t5 (800m) L: llama-2 (13b6)
Train: llama-2 llama-2 (136)
flan-t5 (11b) gpt-3.5-turme
flan-t5 (800m) flan-t5 (11b)
BART Score (f)
BART Score (h)
33.00
BART Score (f)
All-at small
3,00
Allat small
All samall
-4.00
All large
Tandom
Allat-large
Allabblarge
4.62
Coet
3.20
Edet
-0.00
p.prob
Lo900
Lpeob
t Trams
-.50
tramm
022
0.4
0.8
0.0
0.2
0.6
10
0.2
0.4
0.8
Cost Advantage
Cost Advantage
Cost Advantage
(a) High correlation (r 0.76, p 0.71).
(b) Medium correlation 0.55, p = 0.50).
(c) Low correlation (r 0.06, p 0.02).
Figure 8: Routing performance the testing LLM pairs that are different than the pairs routers were trained with. Pearson (r) and spearman (p) correlation coefficients between quality gaps of the training and testing LLM pairs computed for each setting.
A.4 GENERALIZING To DIFFERENT MODEL PAIRS
We evaluate the generalizability of our routers by testing their routing performance on LLM pairs different than the pairs they were trained with. We compute the correlation between quality gaps of training and testing LLM pairs, and report along with routing performance, as shown Figure B
Similar to our observation in Section A.3 our routers can generalize well if the quality gaps of testing L.LM pairs exhibit strong positive correlation with the quality gaps of the training pairs. In Figure Sa both pearson and spearman correlation coefficients exceed 0.7, and all three routers are able to achieve 20% cost advantage with up to 1.6% performance drop, and 40% cost advantage with up to 4.1% performance drop. As the correlation becomes weaker, the generalizability of our router gets restricted and routing performance decays, as shown in Figure 8b) and 8c This observation sheds light on using the quality gap correlation an effective indicator to decide if our routers can be applied to new LLM pairs in the early stage. Given a pair of LLMss (source pair) and a router trained on this pair we can measure the correlation between the quality gap of the source pair and the quality gap of any new target pair of LLMs to decide if the router will be effective on the target pair.
A.5 MORE ROUTER PERFORMANCE RESULTS
In this section, we provide more routing evaluation results 4 LLM pairs. Typically, shown in Figure 9 the FLAN-15 (800m) v.s. FLAN-15 (11b) pair is another example of small perfor- mance gaps, the Llama-2 (7b) v.s. GPT-3.5-turbo pair be characterized
being of medium performance gaps, while the routing between GPT-3.5-turbo and two FLAN-15 models is of large performance gaps. Qualitative comparison results are summarized in Table 4 which resemble our analysis in Section 4.2 In general, dedt is able to achieve noticeable cost advantages with little to no performance drop when the the cost advantage targets are low or the performance gaps are small. As the routing becomes challenging, the improvements over Tdst by having "roab become considerable and Itrons starts to dominate the competition.
We also provide the quality gaps difference for each routing scenario, as shown in Figure 10 where we consistently observe that our routing strategy correctly identifies easy queries and route them to the small model.
B DATASET STATISTICS
We uniformly sample 10k training examples from the training split of Mixinstruct, for each of which generate 10 responses from all investigated LLMs. Our validation and test splits are the same as the Mixinstruct dataset, which consist of 5k instruction examples separately. Our dataset in total consists of 20k instruction examples, shown in Table 5
15

---

Published as a conference paper at ICLR 2024
S: flan-t5 (800m)
S: llama-2 (7b) gpt-3.5-turbo
L: flan-t5 (11b)
100
100
Examples %
80
Examples %
80
60
40
40
20
20
0-
-3
-2
-2
-1
Quality Gaps
Quality Gaps
S: flan-tt (800m)
S: llama-2 (7b)
L: flan-t5 (11b)
L: gpt-3.5-turbo
BART Score (T)
-4.8
BART Score (1)
33.00
All-at-small
All-t-small
-4.00
All-at-large
All-at-large
Random
Random
-4.62
r_det
33.00
r_det
r_prob
r_rrob
r_trans
r_trans
0.2
0.8
1.0
0.0
0.2
0.4
0.6
Cost Advantage
Cost Advantage
S: flan-t5 (800m)
S: flan-t5 (11b)
L: gpt-3.5-turbo
L: gpt-3.5-tturbo
Examples %
Examples %
80
60
60
40
40
20
20
-2
2
-3
-2
Quality Gaps
Quality Gaps
S: flan-t5 (800m)
flan-t5 (11b)
L: gpt-3.5-turbo
gpt-3.5-turbo
BART Score (T)
-3.00
BART Score (t)
-3.00
All-at-small
All-at-smal
-3.50
All-at-large
-3.50
All-at-large
Random
Random
-4.00
dedet
-4.00
det
[_rrbb
[_rrbb
4.50
rtraas
4..0
r_trans
0.2
1.0
0.2
0.4
Cost Advantage
Cost Advantage
Figure 9: Error-cost tradcoff's achieved by rdet, Fprob, and Ptrans when the small and large models are of different performance gaps.
16

---

Published as a conference paper at ICLR 2024
Table 4: Cost advantage v.s, Performance drop.
Performance Drop (%)
Cost Advantage (%)
S: FLAN-15 (800m)
S: Llama-2 (7b)
S: FLAN-1 (800m)
FLAN-15 (11b)
FLAN-5 (11b)
GPT-3.5-turbo
1.: GPT-3.5-turbo
L: GPT-3355tture
ddet
Pprob
Ttrons
Tdet
Pprob
Ttrans
Tdet
Tprob
Ttrons
Tdet
Pprob
Ttrans
10
-0.2
-0.3
-0.2
0.3
0.3
0.6
3.6
3.4
3.6
3.8
3.6
3.3
20
-0.2
-0.2
-0.2
1,5
1.3
1.2
8.7
8.7
7.9
9.0
8.9
7.3
40
0.0
-0.1
0.0
4.1
4.1
3.6
19.3
19.8
17.4
19.2
19.4
16.5
5: flan-t5 (800m)
S: llama-2 (7b)
flan-t5 (11b)
L: gpt-3.5-turbo
Avg. Quality Gap Diff.
Avg. Quality Gap Diff.
0.40
0.50
0.20
0.00
0.00
Random Routing
Random Routing
-0.50
0.20
0.6
0.8
1.0
0.0
0.2
0.6
Cost Advantage
Cost Advantage
S: flan-t5 (800m)
S: flan-t5 (11b)
L: gpt-3.--turbo
L: gpt-3.5-turbo
Avg. Quality Gap Diff.
Avg. Quality Gap Diff.
1.00
1.00
0.00
0.00
-1.00
Random Routing
-1.00
Random Routing
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
Cost Advantage
Cost Advantage
Figure 10: Average quality gap difference between queries routed to the small and large models with different performance gaps.
C QUERY HARDNESS AND BART SCORE
C.I QUERY HARDNESS: A CASE STUDY
In this section, we demonstrate the query hardness with real-world examples from our dataset. We choose the FLAN-1 (1Ib) v.s. GPT-3.5-turbo routing pair for illustration purpose.
C.I.1 EASY QUERY EXAMPLE
Input Query: "Rewrite the sentence so that it's in the present tense. She had worked at the company for the past years"
LLM Responses:
FLAN-1 (11b): "She works at the company for the past years."
GPT-3.5-turbo: "She works at the company for the past 3 years."
17

---

Published as a conference paper at ICLR 2024
Table Dataset Statistics.
Sources
#Examples
Source of Responses
Alpace-GPT4
4,179
GPT-4
Dolly-15K
1,381
Human
GPT4AÍI-LAION
13,547
chatGPT
ShareGPT
567
chatGPT
Total
20K
Mix of Above
In this example, the small model (i.e., FLAN-15 (11b)) is able to generate the same high quality response as the large model (i.e., GPT-3.5-turbo), which makes easy query by definition.
C.1.2 HARD QUERY EXAMPLE
Input Query: "What are the benefits having a dog in the family?"
LLM Responses:
FLAN-5 (11b): "A dog is good for physical activity."
GPT-3.5-turbo:
"Having dog in the family can have numerous benefits, including:
1. Companionship: Dogs provide unconditional love and companionship. offering constant emotional support and sense of belonging.
2. Improved mental health: Being around dogs has been proven to reduce stress, anxiety, and depression. They can also help alleviate loneliness and provide a sense of purpose.
3. Increased physical activity: Dogs require regular exercise, which encourages family members to engage in physical activities such walking, running, and playing. This promotes healthier lifestyle for both the dog and the family.
4. Teaching responsibility: Caring for dog requires feeding, grooming, training, and regular vet visits. Having dog the family can teach children valuable life skills and instill a sense of responsibility.
5. Socialization: Dogs known be social creatures. Walking dog visiting dog parks can provide opportunities for social interaction and help family members connect with other dog owners in the community.
6. Teaching empathy and compassion: Nurturing and caring for dog teaches children empathy, compassion, and the importance of kindness towards animals.
7. Home security: Dogs are natural protectors and can as an early warning system, alerting the family to potential intruders or dangers.
8. Health benefits: Interacting with dogs, such petting or playing, has been shown to lower blood pressure, decrease the risk of allergies and asthma in children, and boost the immune system.
Improved family dynamics: The presence of dog can strengthen family bonds by providing shared experiences, fostering teamwork, and creating a common focus of love and care.
It is important to note that owning dog requires time, effort, and financial responsibil- ity. Families should carefully consider their lifestyle, resources, and commitment before deciding to bring dog into their home."
In this example, the response from the small model (i.e., FLAN-15 (11b)) is clearly uninformative and unpolished, while the large model (i.e., GPT-335-uurbo) able provide comprehensive and well-construutted arguments. We identify this example as a hard query.
C.2 BART SCORE: A CASE STUDY
In this section, we empirically examine the correlation between BART scores and human judgments. We take the query "Where is the Aéronef A-1 Skyraider built?" as example which the expected
18

---

Published as a conference paper at ICLR 2024
answer is "The Aéronef A-1 Skyraider primarily built by the Douglas Aircraft Company in Segundo, California." We report the random responses from Llama-2 (13b) appended with the corresponding BART scores, shown below. Responses from 2 (13b):
1. Where is the Aéronef A-1 Skyraider built? The Aéronef A-1 Skyraider was built by the Douglas Aircraft Company in El Segundo, California, USA. (-0.927)
2. Where is the Aéronef A-1 Skyraider built? A-1 Skyraider is built by Douglas Aireraft Company, located in El Segundo, California, USA. (-1.12)
3. Where is the Aéronef A-1 Skyraider built? The Aéronef A-1 Skyraider built in the United States by Douglas Aircraft Company. (-1.63)
4. Where is the Aéronef A-1 Skyraider built? The Aéronef A-1 Skyraider was built in the United States. (-2.25)
5. Where is the Aéronef A-1 Skyraider built? The Aéronef A-1 Skyraider is a fictional aircraft, is not built anywhere does not exist in reality. (-2.93)
As shown above, the best response of the highest BART score -0.927 provides all necessary in- formation such as the company name and the city name. As the BART scores decrease, it can be observed that the quality of corresponding responses starts degrading well. For example, the 3-th response does not include the city information, the 4-th response further misses the company name, while the last
completely wrong. This example empirically justifies the effectiveness of using BART scores as the response quality metric.
19


Reviews:
[{"review_id": "01SJ6h5IHh", "reviewer": "Reviewer_9pHQ", "summary": "The authors introduce a router that assigns queries to differently sized models. Their method results in 40% fewer calls to the large model, with no drop in response quality. They introduce two main techniques to improve performance:\n\n- using soft probabilities instead of hard probabilities\n- using a data transformation with a relaxation $t$ to provide stronger training signal.", "questions": "- In this method, we are able to reduce cost and latency, but not as much latency reduction as methods such as speculative decoding (https://arxiv.org/abs/2211.17192). While there is added cost with speculative decoding, do you think there's any possibility of closing this gap?\n- Do you think this might be because of scoring query wise vs token-wise? Why not use this method token wise?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Paper is well written, and authors do a good job of building upon concepts used in final technique.\n- Ablations and analysis are extensive and well-thought out, giving researchers ample inspiration to build upon this technique.\n- The analysis of performance on different model size pairs is interesting to me.", "weaknesses": "Please cite these works:\n- https://arxiv.org/abs/2305.05176 - routing on a query level\n- https://arxiv.org/abs/2211.17192, https://arxiv.org/abs/2302.07863 - latency reduction using small and big models\n\nI believe writing a discussion of the tradeoffs of these approaches would improve the current draft.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U4s76jN4Rg", "reviewer": "Reviewer_uyT6", "summary": "The paper introduces a hybrid inference strategy designed to minimize the computational expense by limiting the number of queries to the larger model and utilizing smaller models to function as decision-making routers.\nInitially, the approach assesses if the user's input query is easy or hard by evaluating the anticipated response quality from both the small and large models.\nTo evaluate the complexity of a query, the paper describes three distinct methodologies, each utilizing the same classification model but differing in their training and inference schemes.", "questions": "Same as weakness part.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Paper presents a novel hybrid inference strategy designed to minimize the computational expense by limiting the number of queries to the larger model and utilizing smaller models to function as decision-making routers.\nMoreover, paper presents multiple different approaches to training the decision making routers and its effectiveness.", "weaknesses": "I have following major concerns.\n\n1. **Reliability of BART scores for routing**\nI am uncertain about the efficiency of training the router model to decide whether the BART scores of the smaller model is similar to those of the larger one. BARTScore has demonstrated strong performance in extractive QA; however, its correlation may diminish in abstractive QA contexts [1], suggesting that the metric might not be suitable for assessing open-ended generation tasks. Establishing a correlation between evaluations of routing using BARTScore and human assessments would be beneficial to verify the reliability of BARTScore for routing evaluation purposes.\n\n2. **The Impact of Training Data Versus Model Size**\nI am of the opinion that the size of the model is not as critical as the differences in the training data used for each model in determining quality. For instance, consider evaluating the performance disparities between models like (Llama-2 7B and Llama-2 13B) versus those between (Llama-2-7B and the more recent Zephyr-7B [2]). Would the performance gap trend similar to the reported trend in Figure6?\n\n[1] G-EVAL: NLG Evaluation using GPT-4 with Better Human Alignment., Liu et al., 2023 \\\n[2] https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MzQ8XSRmUG", "reviewer": "Reviewer_VA7E", "summary": "The paper introduces a novel inference paradigm called hybrid inference, which utilizes two models of different sizes to handle queries. This approach aims to balance infference cost and response quality by routing easy queries to a smaller model while directing more complex queries to a larger model. The authors propose an orchestration framework that involves a router trained on a dataset of representative queries. The router dynamically routes queries to the appropriate model, thus reducing overall costs while maintaining response quality. They present three variations of the router: a deterministic router, a probabilistic router, and a probabilistic router with data transformation.", "questions": "The approach might encounter problems in accurately distinguishing between easy and hard queries, especially when dealing with a large performance gap between different models. How do you elaborate on this?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper sets the problem in the context of LLM inference and focuses on the evaluation of response quality and cost advantage. It defines metrics for measuring the effectiveness of the routing strategy, considering the intrinsic uncertainties in natural language processing tasks. The evaluation is conducted on the MixInstruct dataset, which comprises a diverse range of tasks such as question answering, summarization, and information extraction. The experimntal results demonstrate the efficacy of the proposed routing strategies, especially in scenarios where the performance gap between the small and large models is minimal. The deterministic router achieves good cost advantages with negligible drops in response quality, while the probabilistic router further improves the cost advantage without compromising response quality. The probabilistic router with data transformation exhibits even more promising results, achieving significant cost advantages with no quality drop.", "weaknesses": "The main limitation of the paper seems to be its reliance on the assumptions about the quality gaps and the routiing mechanisms. These assumptions could potentially affect the overall effectiveness and efficiency of the routing process. Additionally, the reliance on specific models and the need for manual intervention in setting the threshold for routing may limit the scalability and generalizability of the proposed framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RbpwVJ5FMO", "reviewer": "Reviewer_ZnPu", "summary": "Deployment of large language models is costly whereas smaller models can be deployed on edge devices but tend to lag behind in response quality. This work proposes a hybrid inference approach to save cost and maintain response quality. A query router is employed for assigning queries to large or small language model depending upon the predicted query difficulty and the desired quality level. This desired quality level is dynamically tunable at test time for trading quality for cost. The proposed design achieves 40% fewer calls to large model with no drop in response quality.", "questions": "This Paper states that “we expect that using the router to route queries to the small model will not detract significantly from the realizable cost advantage.” Is this an assumption or empirically verified conclusion?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper addresses and interesting problem considering that currently available smaller language models can fairly perform well. Depending upon the predicted difficulty level of the query, it is interesting to use a router to pass the relatively easier queries to smaller model. This approach can be cost effective.\n\nMultiple router score designs are proposed.\n\nThere is thorough empirical analysis with good discussion.", "weaknesses": "The proposed design requires that for each LLM pair, a router is required to be trained which might be a costly undertaking in a production environment. \n\nThis paper discusses the cost/quality analysis in context of a language model pair. In real world scenarios, there might be multiple LLMs available and several competing factors to be optimized or traded-off. \n\nFigure 1 is not properly aligned and some inconsistent border is visible (Fig 1 (C)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)