RouteLLM: Learning to Route LLMs from Preference Data
Authors: Isaac Ong
Amjad Almahairi
Vincent Wu
Wei-Lin Chiang
Tianhao Wu
Joseph E. Gonzalez
M Waleed Kadous
Ion Stoica
Venue: ICLR 2025 Poster (2025)
Keywords: Large language models
query routing
Primary area: generative models

Abstract: Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. Powerful models offer better results but are expensive, while smaller models are more cost-effective but less capable. To address this trade-off, we introduce a training framework for learning efficient router models that dynamically select between a stronger and weaker LLM during inference. Our framework leverages human preference data and employs data augmentation techniques to enhance performance. Evaluations on public benchmarks show that our approach can reduce costs by over 2 times without sacrificing response quality. Moreover, our routers exhibit strong generalization capabilities, maintaining performance even when routing between LLMs not included in training. This highlights the potential of our framework to deliver cost-effective, high-performance LLM solutions.

Published as a conference paper at ICLR 2025
ROUTELLM: LEARNING TO ROUTE LLMs WITH PREFERENCE DATA
Isaac Ong"1 Amjad Almahairi
Vincent Wu² Wei-Lin Chiang
Tianhao Wu²
Joseph E. Gonzalez'
M Waleed Kadous
lon Stoica
UC Berkeley 2 Anyscale
*Canva
ABSTRACT
Large language models (LLMs) excel at wide range of tasks, but choosing the right model often involves balancing performance and cost. Powerful models offer better results but expensive, while smaller models are more cost-effective but less capable. To address this trade-off, we introduce a training framework for learning efficient router models that dynamically select between stronger and weaker LLM during inference. Our framework leverages human preference data and employs data augmentation techniques to enhance performance. Evaluations on public benchmarks show that our approach can reduce costs by over 2 times without sacrificing response quality. Moreover, our routers exhibit strong general- ization capabilities, maintaining performance even when routing between LLMs not included in training. This highlights the potential of our framework to deliver cost-effective, high-performance LLM solutions.
INTRODUCTION
Recent advances in large language models (LLMs) have demonstrated remarkable capabilities across wide range of natural language tasks. From open-ended conversation and question answering to text summarization and code generation, LLMs have demonstrated an impressive level of fluency and understanding (Achiam al., 2023 Bubeck et al.. 2023). This rapid progress has been enabled by
combination of architectural innovations, such as the Transformer architecture (Vaswani et al. (017). as well as scaling up data and training infrastructure (Brown et al., 2020 Radford et al. 2019).
However, not all LLMs are created squal-there exists wide variation in the sizes of different LLMs. which in turn affects the resources required to serve them. LLMs also differ terms of the data on which they are trained, which in turn leads to variations in the strengths, weaknesses, and capabilities of different models. Broadly speaking. larger models tend to be more capable but come at a higher cost, while smaller models tend to be less capable but cheaper to serve.
This hetcrogeneous landscape presents dilemma in the practical deployment of LLMs. Although routing all user queries to the largest and most capable model ensures high-quality results, it is prohibitively expensive. Conversely, routing queries to smaller models can save costs-by by more than 50x (e.g., Claude-3 Haiku vs. OpusTp but may result in lower quality responses, the smaller model may not handle complex queries effectively.
LLM routing (Ding et al. 2024 Hu et al. 2024) offers effective solution by first processing each user query through a router, which then determines the most suitable LLM to handle the query. The router can direct simpler queries smaller models and more complex ones to larger models, thereby balancing response quality with cost efficiency.
Achieving optimal LLM maadiin-mxximiziing quality within cost constraint or minimizing cost for target quality- is challenging. An ideal LLM router must (1) optimize response quality while invoking a single LLM per query. minimizing cost and latency as compared multi-LLM approaches; (2) generalize to out-of-domain queries without needing separate routers for different domains; and (3) work across broad range of L.Mss without retraining, ensuring flexibility as the L.M landscape evolves.
"Equal contribution. Correspondence to .saacongbberkelley edeieeeeeeeeeeuuueeeee.e ann@anyscale.com
Per million output tokens: Haiku ($1.25) vs. Opus ($75)

---

Published as a conference paper at ICLR 2025
MHLU
GPT-4
(P1-4
(PT1305)
Causal LLM
Random Matrix Fartor swlton (A)
APGR
Randem
BERT
SW Ranking
(A)
Mixtral
40
80
80
200
20
40
60
80
100
100
1/ Calls UPM-4
Calls UP1-4
%
Figure 1: Routing performance/cost trade-off between GPT-4 and Mixtral-8x7B. (left) We demonstrate several routers that outperform the random baseline on OOD eval GSM8K. (center) We demonstrate improvement in router performance through data augmentation, denoted by (A), on MT Bench. (right) We display the main metrics we consider: call- pel-orerformence threshold (CPT, denoted in green) and average performance gain recovered (APGR, denoted by the blue shaded region).
In this work, introduce principled framework for learning LLM routers from preference data. Our approach involves routing between two classes of models: (1) strong models, which provide high-quality responses high cost (e.g., GPT-4), and (2) weak models, which offer lower-quality responses at a reduced cost (e.g., Mixtral-8x7B). The objective is to minimize costs while achieving specific performance target, e.g., 90% of the strong model, by intelligently routing simpler queries to weak model and reserving more complex queries for the strong model. We use our framework to train several router models and evaluate them on widely recognized benchmarks such as MMLJU (Hendrycks et al. (2020) and MT Bench (Zheng et al. (223).. We demonstrate that our router models significantly reduce costs- by over 2x- wwthoou substantially compromising quality, Moreover, they show strong performance across multiple strong weak model pairs without requiring retraining.
To summarize, we make the following contributions:
We propose learning framework for routers that leverages human preference data and data augmentation techniques, achieving over 2x cost savings popular benchmarks with minimal impact on response quality.
We demonstrate that our approach enables routers to generalize to unseen data while maintaining strong performance across multiple LLMs, allowing single trained router to be effective across wide range of use cases.
We open source our framework for training, serving, and evaluating LLM routers, allowing users to easily train their own routers and compare router performance across benchmarks.
2 RELATED WORK
key distinction exists between reward modeling (Ouyang et al. 2022) and LLM routing. Reward modeling assesses response quality after an LLM generates it, whereas routing involves selecting the appropriate L.LM beforehand. This requires a deep understanding of the query's 's complexity and the specific capabilities of available models.
Several recent works have also examined the cost perfforrfomeec trade-offs in routing between different LLMs.. L.LM-Blender (Jiang et al.) 2023) uses ensemble framework that queries multiple LLMs during inference and selects best response. Frugal-GPT (Chen et al. 2023) follows a cascading approach, sequentially querying LLMs until reliable response is obtained. AutoMix (Aggarwal ct al. 2024) uses a smaller model to self-verify its response before potentially routing the query to larger model. These methods rely multiple LLM queries, which increase latency. In contrast, our approach routes each query to a single LLM, addressing the latency constraints of an ideal LLM router.
Hybrid-LLM (Ding et al. 2024) shares some similarities with our framework but differs in key aspects: it uses synthetic preference labels from the Iixtnstr dataset (Jang al. 2023) based
2

---

Published as a conference paper at ICLR 2025
on BARTScore Yuaan al., 2021) and relies on single BERT-based router. In contrast, leverage human preference labels from Chatbot Arena (Chiang et al. 2024) and explore multiple router architectures, showing that data augmentation significantly boosts performance across all architectures. Additionally, Hybrid-IILM evaluates on the MixInstruct test split and lacks evidence of out-of-domain generalization, whereas aim to demonstrate this by evaluating on several decontaminated public benchmarks.
Finally, Zooter (Lu al., 2023) uses routing labels from QwenRM reward models (Bai et al. 2023 which can inherit biases from their training data, affecting the reliability of the routing decisions. In contrast, our approach relies mainly on human preference data. Like Hybrid-LLAn LLM, Zooter explores only BERT-style router. Additionally, their training signal relies on a fixed set of LLMs, limiting its adaptability to other L.LMs. In contrast, we show that our approach maintains strong performance even with LLMS not included in the training data.
3
LLM ROUTING
3.1 PROBLEM FORMULATION
Consider a set of LLM models M, where each model M : Q + A can be viewed as a function that maps a query E Q to an answer a = (q) E A. In this work, we focus on routing between two classes of models: (1) strong models (Msnong) which are capable of producing high-quality responses but come at a high cost, such as advanced proprietary models like GPT-4 (OpenAl 2023). and (2) weak models (Mwesk), which offer lower-quality responses but at a reduced cost, such as models like Mixtral-8x7B (Jiang et al.] 2024). This binary routing problem is common in practice, especially as users seek to optimize the trade-off between quality and cost by transitioning from closed-source to open-soure models. Additionally, solving the binary routing challenge provides foundation for extending to a more complex N -wway routing scenario.
Assume we have access to preference data: Dpret {(g, (,,w) 9 E Q, E L}. where (s represents the outcome of comparing the responses from a strong model M, E Masron and a weak model M.r .MWeak for given query q, and takes values from the set L = [win,, tie, win We introduce a principled framework for learning binary routing function 8 : 2 - {MMwaak Matong] from preference data. Our approach defines t using two key components:
1) Win Prediction Model: This model estimates the probability that a strong model in Manrng will outperform weak model in Mweak for given query q. This probability is denoted by Pe Pa(winglg) q). where 0 represents the model parameters. These parameters are learned by maximizing the likelihood of the observed preference data:
max 0
Σ
log (l q).
(ED
By optimizing this likelihood, the model captures the comparative strengths and weaknesses of the two model classes across different query types. In Section 4.2) we discuss several approaches for parameterizing this win prediction model.
2) Cost Threshold a E [0, 1]: This threshold translates the predicted winning probability into routing decision between Mweak and Matrong- Given query q, the routing decision defined as:
(weank if P(win, 4) ar
1e (q)
Mastong
if P(win, 1)) Z a.
The threshold a controls the trade-off between quality and cost: higher value of a enforces stricter cost constraints by favoring weak models more often, while lower a biases toward higher-quality (but more expensive) strong models.
Finally, with the routing function R. and two models, M, E MMsrron and Mw &weakk, we define router model MR Q X Masrong Mwsak - A, which responds query g as follows 2
(M) if Rd (q) -Mstrong
MRo (q. Ma, Mnn)
(g) if Rd (q) M
Forr brevity, denote this Mge (g).
3

---

Published as a conference paper at ICLR 2025
3.2 METRICS
In this section, we define evaluation metrics to capture the trade-off between cost and performance in the LLM routing problem. We begin with metrics that independently assess the cost efficiency and performance of a router model Mą= routing between two models M, E Marrong and Mi E Mweak- and then introduce two compounded metries used our experimental evaluations.
We measure the cost efficiency of MR- by calculating the percentage of calls to strong models:
c(M = |2] P
1
Σ [ (4) = -Msttog) )
since Marong models are significantly more costly than Mweak models.
For performance, we calculate the average response quality:
r r(MR
1
Σ QEQ
((MR-(4))
(5)
Q
where R-MR= (q)) represents an individual response quality score, such as correctness on golden- labeled datasets (e.g.. MMLU) or numerical rating (e.g. 1-5 or 1-10), with higher values indicating better quality. Similarly, r (M,) and r(Mw) can be defined for the strong and weak model respectively.
Since the router model's performance falls between that of the weak and strong models, we quantify its performance relative to the gap between them. We define the router's overall performance improvement using the performance gap recovered (PGR):
PGR(M = r(MR=) r (MM,)
(6)
r(M,) r(M,,)
This metric captures how much of the performance difference between the weak and strong models is recovered by the router model.
Neither of the above metrics alone is sufficient to capture the quality-cost trade-off in routing. For example, trivial router that always sends queries to the strong model achieves a perfect PGR but with no cost savings. To address this, we compute a call call-performanee graph for router Mr= by varying the threshold values a. We then define the average performance gap recovered (APGR) as overall measure of the router's ability to recover the performance gap under different cost constraints:
APGR(Mn=>) = PGR(MR=) (e(Mn=))
In Figure (right). APGR corresponds to the area between the router's performance curve and the weak model's performance curve. Empirically, we discretize the percentage of calls over the interval [0%, 100%] into (ehee For each c; we determine the threshold a; that satisfies the cost constraint. We approximate APGR as:
10
APGR(Mre) 10
Σ PGR(M
(8)
In many real-world applications, it is essential to quantify the cost required to achieve a certain level of performance. To address this, introduce second metric called call-performance threshold (CPT). Given a desired router performance, i.e., achieving PGR 2%, the CPT(+%) represents the minimum percentage of calls to the strong model needed reach the desired PGR. In Figure (right), the dotted green line illustrates CPT(50%), indicating the percentage of calls to GPT-4 needed achieve PGR of 50%. In this figure, C57())%) 37%.
METHODOLOGY
4.1 CHATBOT ARENA DATA
Our primary source for preference data is the 80k battles from the online Chatbot Arena platform (Chiang al. 2024), where users submit prompts and receive responses from two anonymous models.
4

---

Published as a conference paper at ICLR 2025
After reviewing the responses, users vote for winner or declare tie. This generates a dataset, denoted as Darena, which contains user queries, model responses, and pairwise comparison labels based on human judgment.
A key challenge with the raw Chatbot Arena data label sparsity. On average, the percentage of comparison labels between any models is less than 0.1%. To address this, we derive the preference data by clustering the models in the data into 10 tiers (see Appendix A based on their scores on the Chatbot Arena leaderboard and minimize intra-tier variation using dynamic programming. We choose models in the top two tiers to represent the Matrong class, and models in the third tier represent the Mweak class. Crucially, we exclude model responses and retain only the winner identities in training. The resulting dataset is defined as D Dureaa = {(q. (s,w) q E 1,, E L}.
4.1.1 DATA AUGMENTATION
Despite classifying models into tiers, the human preference signal remains sparse across different model classes. As discussed in Sec [.T] this sparsity limit generalization, particularly for parameter-heavy router models. To address this, we explore two data augmentation methods:
Golden-labeled datasets: We augment our training data with labeled datasets of the form Danid {(g, (g,,4, q E 2, lg E 12.0 E L). where golden label lg is the known correct answer, e.g. in multiple-choice questions. Specifically, we use the validation split of the MMLU multiple choice benchmark (Hendryeks et al. (020) containing approximately 1500 questions and derive comparison labels 1 simply by comparing the responses from M, and Ma, to the golden label.
LLM-judge-labeled datasets: We explore obtaining preference labels on ppen-ended purpose chat domains using LLM judge (Zheng et al.] 2023). as it has demonstrated a high correlation with human judgment (Dubois et al.| 2002 Jiang et al. 2023). Given collection of user queries, we start by generating responses from both a strong model M, E Matrong and a weak model M,,, E M then producing pairwise comparison labels using GPT-4 as judge. The primary challenge with this method is the high cost of collecting responses and pairwise comparisons from GPT-4 in large quantities. Fortunately, the Nectar dataset (Zhu et al. 2023) offers wide variety of queries with corresponding model responses. We significantly reduce our costs by selecting queries with GPT-4 responses (as Ms), on which we generate responses from Mixtral-8x7B (as Mw). Finally, obtain pairwise comparison labels using the GPT-4 judge 4 Overall, we collect a preference dataset Djudge of approximately 120K samples costing around $700 USD in total.
4.2 ROUTING APPROACHES
We now discuss several methods to define the win prediction model Po(win,lg) introduced in Eq
wieilhrity-weightedd (SW) ranking We adopt a Bradley-Tery (BT) model (Bradley & Terry 1952) similar to Chiang et al. (2024). Given user query g, compute its cosine similarity each query o' in the train set, scaled according to the maximum cosine similarity for g' in the dataset:
S(q.4))
|e|l|e'||| max S PPP
e'.ty |
where e and e' denote text embeddings for q and g' respectively. This similarity score is used to We learn BT coefficients &s. &w
compute a weight scalar for each training query w' y (+ for the strong and weak models by solving:
W (1.,0 + ct Ex )
argmin Eo
Σ
(10)
where l is a binary cross-entropy loss. These coefficients allow us estimate the win probability as: Po Po(winn - In this approach, no training is soluired-solvingee is performed at inference time.
https//leaderboarr. Imsys.org
We employ best practices recommended in /heng et al. 2023 to de-bias GPT-4 judgements We find that exponential scale works best in practice and choose 7 10.
5

---

Published as a conference paper at ICLR 2025
Matrix factorization Drawing inspiration from matrix factorization models used in recommenda- tion systems to capture low-rank structures in user-item interactions (Koren et al. 2009 Töscher et al.) 2009), apply this approach to learn from preference data. The goal is to uncover a hidden scoring function ò .M x Q - R. where 5(M, q) represents the quality of the model M's response to query q. If M is better than M' query q. then ((,,q) 8(M', q). We enforce this by modeling the win probability using sigmoid function d:
(wwinglgg) = a (8(M, .) - (M',9)).
(11)
which we optimize on the preference data. The scoring function 6 is modelled bilinear function of the model and query embeddings. We embed the model M into dm dimenssoonl vector U/m and the query 9 into d, dimensional vector U/:
8(M, g) w/ (um ④ (WT 1ig b)):
(12)
where @ denotes the Hadamard product. W1 E R2d, Xdm and b E RR are parameters of a projection layer to align the dimension of vig with um W2 E Rdd is the linear regression layer to produce the final scalar. This method is essentially learning a matrix factorization of the score matrix on the set Q x M. We train the model on 8GB GPU for 10 epochs, using batch size 64 and the Adam optimizer (Kingma & Ba 2017 7) with learning rate 3 x 10-1 and weight decay x 10-5.
We explore using standard text classification method with higher number of parameters compared to previous methods. We use a BERTRASE architecture (Devlin et al. 2018). to give contextualized embedding of the user query, and define win probability as:
BERT classifier
Po (win, (q) = o oo ficss + b),
(13)
where heLL is an embedding corresponding to the special classification token (CLS) summarizing the input query q. W and b are parameters of the logistic regression head, while is the sigmoid function. We perform full-parameter fine-tuning on Prett- We train the model on 2x1 24GB GPUs for ~ 2000 steps using a batch size of 16, maximum sequence length of 512, learning rate of 1 x 10-5 and a weight decay of 0.01.
Causal LLM classifier We finally expand the capacity of our router by parameterizing it with Llama 3 8B (AT@ Meta 2024b). We use instruction-followinn paradigm (Wei et al.) (021). i.e. provide as input an instruction prompt containing the user query and output the win probability in next-token prediction fashion, instead of using a separate classification head. Notably, we append the comparison labels as additional tokens to the vocabulary, and compute the win probability as softmax over the label classes L. We train the model on 8xA100 80GB GPUs for 2000 steps using batch size of 8, maximum sequence length of 2048, and a learning rate of 1 10-6,
5 EXPERIMENTS
As mentioned in Sec. 4.1 we primarily use the 80K Chatbot Arena data Darem for training our models, but hold out 5k samples for validation. We prune all prompt samples shorter than 16 characters, resulting in 65k pairwise comparisons between 64 different models. These consist of conversations from over 100 languages, with the bulk of the conversations (81%) in English, followed by Chinese (3.1%) 1),, and Russian (2.2%). We assign models to 10 classes to reduce sparsity of comparison labels. As discussed in Sec. 1.1. we further augment our training data with with either: 1) D goota golden-labeles data created from the MMLU validation split, or 2) Djudge, GPT-4-as-a-jjue labeled data.
Training data:
Evaluation benchmarks: We evaluate our routers on three widely-used academic benchmarks: MMLU (Hendrycks et al., (220)) consisting of 14.042 questions across 57 subjects, MT Bench (Zheng al. 2023) with 160 open-ended questions using LLM-as-a-judde and GSM8K (Cobbe et al. 2021 with over 1,000 grade school math problems. Additionally, we conduct a cross-contamination check between evaluation and training datasets, and report uncontaminated results below. We present results on public benchmarks to understand the out-of -fo--nn generalization of our routers.
OpenAl's embedding model tex -exe-embbdding-- -nnngg--mmmmll to embed the input query. We perform full- parameter finetuning both BERT and Causal LLM,, and use the validation set for model selection.
Routers: For both the matrix factorization router and the SW ranking router,

---

Published as a conference paper at ICLR 2025
We opt 4pt----116--reviiie-e-- (OpenAl 2023 M, E Masrong and Mixtral 8x7B JTang et al. 2024) as My E Mweak to concretely evaluate router performance. We use random router that routes queries randomly under cost constraint the baseline.
5.1
RESULTS
Training data
Method
C5T((0%)
CPT()0%)
APGR
Improvement
Random (95% CT)
49.03(++4)%
78.08(+3))%
0.500(+0.02)
(+0%)
D Darea
BERT
78.09%
87.64%
0.391
(-21.8%)
Causal LLM
28.82%
77.53%
0.573
(+14.6%)
Matrix Factorization
25.32 %
74.26%
0.580
(+16%)
SW Ranking
37.85%
58.99 %
0.610
(+22.1%)
Darens + Dysdge
BERT
19.58%
34.02%
0.751
(+50.2%)
Causal L.M
31.50%
48.75%
0.679
(+35.8%)
Matrix Factorization
13.40%
31.31 %
0.802
(+60.4%)
Ranking
23.21
36.04%
0.759
(+51.8%)
Table 1: MT Bench results. Note that the MT Bench score at CPT(50%), 8.8, is 95% that of GPT-4's score (9.3). Our routers exhibit strong performance on MT Bench when trained on Darena- with further improvement when the dataset is augmented with Dusde, reducing costs by up to 75% compared to the random router.
Table displays our router performance on MT Bench. For routers trained on the Arena dataset, we observe strong performance for both matrix factorization and
ranking, with both routers performing significantly better than the random router across all metrics. Notably, matrix factorization requires half the number of GPT-4 calls as compared random achieve PGR of 50%. However, our BERT and causal LLM classifiers perform close to random when trained on the Arena dataset, which we attribute high capacity approaches performing worse low-data regime.
Augmenting the preference data using GPT-4 judge leads to notable improvements across all routers. The BERT and causal LLM routers now perform much better than the random baseline, with the BERT classifier achieving an APGR improvement of over 50% compared to random. When trained on this augmented dataset, matrix factorization is the best-performing router with its CPT(80%) nearly halved and requiring 50% less GPT-4 calls as compared to random.
We also compare the MT Bench performance of our routers against existing commercial routing systems in Appendix demonstrating how our routers achieve substantial improvements over other available systems.
Training data
Method
CPT((5%)
CPT(80%)
APGR
Improvement
Random (95% CT)
50.07(=009%
79.93(=0)9
0.500(=0)
(+0%)
Darena
BERT
49.43%
77.80%
0,502
(+0.5%)
Causal LLM
48. 48.88
77.93%
0.499
(-0.2%)
Matrix Factorization
45,00 %
76.86% %
0,524
(44.9%)
SW Ranking
55.82%
80.25%
0.473
(-5.4%)
Denna + Deoid
BERT
41.30%
72.20%
0.572
(+ (+14.4%)
Causal LLM
35.49%
70.31 %
0,600
(+19.9%)
Matrix Factorization
35.46%
71.40%
0.597
(+19.5%)
SW Ranking
35.40%
71.55%
0.603
(+20.7%)
Table 2: 5-shot MMLU results for our routers. Note that the MMLU score at CPT(50%) 75, is 92% that of GPT-4's score (81). Routers trained only on arema perform poorly due to most questions being out-of-distribution but dataset augmentation with gold is highly effective, leading to significant improvement router performance even with a small number samples.
7

---

Published as a conference paper at ICLR 2025
all routers perform poorly at the level of the random router when trained only on Arena dataset, which we attribute to most MMLU questions being out-oo-distribution (see Section 5.3 However, augmenting the training dataset with golden-label data from the MMLU validation split leads to significant performance improvements on MMLU across all routers, with all routers requiring approximately 20% less GPT-4 calls than random for CPT(50%) Importantly, this is despite the fact that the additional golden-labeled dataset of approximately 1500 samples represents less than 2% of the overall training data, demonstrating the effectiveness of dataset augmentation even when the number of samples is small.
On MMLU (Table 2)
Training data
Method
(P7())0))
CPT(80%)
APGR
Improvement
Random (95% CT)
50.00(+-2)%
80.08(+1)%%
0.497(+0.00)
(+0%)
D Darna
BERT
58.78%
83.84%
0.438
(-11.8%)
Causal LLM
56.09%
83.56%
0.461
(-7.3%)
Matrix Factorization
53.59%
85.24%
0.4746
(-4,5%)
SW Ranking
54.43%
82.11 %
0.4753
(-4.3%)
D + Dastge
BERT
44.76%
79.09%
0.531
(+6,9%)
Causal LLM
33.64%
63.26
0.622
(+25.3%)
Matrix Factorization
38.82%
72.62%
0.565
(+13.8%)
SW Ranking
41.21%
72.20%
0.568
(+14.3%)
Table 3: 8-shot GSM8K results. Note that the GSM8K score CPT(50%). 75, is 87% that of GPT-4's score (86). Routers trained only on Darna again perform poorly due to questions being out-of-distribution, but augmentation with Judge substantially improves router performance.
Finally, on GSM8K (Table 3), trained only on the Arena dataset is close to random. However, training our routers on the dataset. augmented with synthetic data from LLM judge substantially improves performance, with all routers going from an APGR worse than random to an APGR greater than random. When trained on this augmented dataset, the causal LLM classifier performs the best out of all routers, requiring 17% less GPT-4 calls than random to achieve CPT(50%) and CPT(80%).
we observe that similar to MMLU, the performance of all routers
5.2 ADAPTABILITY ACROSS MODELS
We picked gpt 4-1106----- -are6--prevewww and Mixtral 8x7B as M, and Mir respectively for the above experiments. However, to demonstrate the adaptability of our routers to new LLMs, we report in Table the performance of our routers on MT Bench when they are used route between two new model pairs: (1) M, Claude 3 Opus. M Claude 3 Sonnet (Anthropic) 2024) and (2) M, Llama 3.1 70B, Mn, = Llama 3.1 8B (AI@ Meta, 2024a). Importantly, we use the same routers as before without any retraining, and only replace the strong and weak model routed to. These LLMs are also not present in the training data.
Model Pair (M, Mw)
Method
CPT(50%)
CPT(80%)
APGR
Improvement
Claude 3 Opus/
Random (95% CI)
49.89 (133%%
72.27 (.14)%
0.493 (L10.033)
(+0%)
Claude 3 Sonnet
BERT
34.85%
39. 99.04%
0.682
(+38.3%)
Causal LLM
28.12%
50.00%
0.656
(+33.1%%
Matrix Factorization
31.86%
36.43% %
0.762
(+54.6%)
SW Ranking
23.27 %
51.85%
0.772
(+56.6%)
Llama 3.1 70B / Llama 3.1 8B
Random (95% CI)
47.52
76.26(
0.512 (4-0.017)
(+0%)
BERT
30.15%
38.91% %
0.673
(+31.4%)
Causal LLM
34.05%
45.96%
0.689
(+34.6%)
Matrix Factorization
25.83%
37.30%
0.738
(+44.1%)
SW Ranking
21.18%
29.39%
0.767
(+49.8%)
Table MT Bench results for our routers when used to route between different model pairs. We use the exact same routers before trained on D Darena + Djudge Our routers generalize very well across different model pairs without any retraining.
8

---

Published as a conference paper at ICLR 2025
We observe strong results across all existing routers MT Bench when the model pair is replaced, with performance comparable to that of the original model pair. The results continue to be significantly stronger than random, with our best routers requiring approximately half the GPT-4 calls of the random router to achieve CPT) when routing between both the Claude 3 and Llama 3.1 family of models. These results suggest that routers have learned common characteristics of queries that allow them to distinguish between strong and weak models, generalizing to new models inference time without additional training.
5.3 QUANTIFYING DATASET AND BENCHMARK SIMILARITY
We attribute the difference in the performance of routers trained on the same dataset across different benchmarks to the differing distributions of evaluation data and training data. For each benchmark- dataset pair, we compute a benchmark datasee similarity score in Table 5 indicating how well-
represented evaluation data in the training data, detailed in Appendix
Arena
Arena augmented with with Djudge
Arena augmented with D guuld
MT Bench
0.6078
0.6525
MMLU 0.4823
0.5678
GSM8K
0.4926
0.5335
Table 5: Benchmark- similarity scores demonstrate strong correlation between these scores and the performance of routers on the corresponding benchmarks, providing a way of quantitatively improving router performance.
higher benchmark-datten similarity score is correlated with stronger performance on that bench- mark for routers trained using the corresponding dataset, as shown in Section 5.1 Dataset augmen- tation, be using golden-labeled or L.LM-judge-labele the preference data to be more in line with the benchmarks and increases the benchmark-datasse similarity score, which translates into improved performance. This similarity score is also useful for understanding the relative performance of routers across different benchmarks: on Daremas the similar- ity score between MT Bench and all datasets noticeably greater than other benchmarks, which we believe explains the relatively stronger router performance on MT Bench as compared GSM8K and MMLU. Benchmark-datase similarity scores are promising direction for systematically improving router performance real-world use cases, given knowledge about the query distribution.
datasets, shifts the overall distribution of
5.4
COST ANALYSIS
CPT(50%)
CPT(80%)
MT Bench
3,66 (95% GPT-4 quality)
2.49
MMLU
1.41 (92% GPT-4 quality)
1.14
GSM8K
1.49 (87% GPT-4 quality)
1.27
Table 6: Cost saving ratio of our best performing routers over GPT-4. Our routers are able to achieve significant cost savings while maintaining quality.
We estimate the average cost of using GPT-4 and Mixtral 8x7B to be $24.7 per million tokens and S0.24 per million tokens respectively (detailed in Appendix D). Based on this, in Table the cost savings achieved by our approach. To do so, we calculate the inverse of the ratio of GPT-4 calls made by our ror-performing router relative to the random baseline because the cost of GPT-4 is the dominant factor in our analysis. The results show that our routers achieve cost savings of up to 3.66x, demonstrating that routing significantly reduce cost while maintaining response quality.
quantify

---

Published as a conference paper at ICLR 2025
5.5
ROUTING OVERHEAD
Cost / million requests
Requests / second
Hourly cost of VM
SW Ranking
$39.26
2.9
S0.39
Matrix Factorization
$3.32
155.16
$0.8
BERT
$3.19
69.62
$0.8
Causal LLM
$5.23
42,46
$0.8
Table 7: Cost and inference overhead of different routers. As compared the cost of LLM generation, the cost of deploying a router is small while also able being able to support real-world workloads.
A concern with LLM routing is the overhead of routing compared to using a single model. Therefore, we measure and report the overhead of our routers in Table 7 Jusin randomly-sampled conversations from Chatbot Arena. For each router, we first profile the rate at which it can process requests, then use the VM's hourly cost to calculate the cost overhead. For routers that require embeddings, also include the cost of embedding generation based on the average input length of the training set. For routers that use GPUs, namely matrix factorization and the classifier methods, we utilize Google Cloud's g2 g2-stannd r VM containing single NVIDIA 144 GPU. For similarity- weighted ranking, we use Google Cloud's CPU-only n2 a22-standrdd- VM.
Our GPU-based routers are currently much more efficient that our CPU-based routers, but we note that there is still much room for improvement in optimizing these routers. Based on the results, our most expensive router, SW ranking, currently adds an extra cost of more than 0.4% as compared to GPT-4 generation (detailed in Appendix D). demonstrating the cost-effectiveness of these routers.
6 CONCLUSION
We demonstrate strong performance by our routers across a variety of benchmarks from open-ender question answering to humanities and math problems. By intelligently routing queries between strong and weak model, our routers achieve significant cost savings and high response quality without excessive cost or latency overhead. We also show that our routers maintain their performance across multiple strong weak model pairs without retraining-an important capability that if absent, would greatly limit usefulness.
Our results highlight the effectiveness of dataset augmentation in improving router performance. While training routers solely on Darena results in poor performance on MMLU and GSM8K, augment- ing the training data with an LLM judge or in-domain data enables our routers to outperform the random baseline across all benchmarks. The greatest performance gains occur when the training data closely resembles the evaluation data, indicated by the senchmark-datass similarity score. We believe that this framework provides a clear path towards improving routing performance for specific use cases.
While our work demonstrates strong results, there are a few limitations. First, although we evaluate on a diverse set of benchmarks, real-world applications may have distributions that differ substantially from these benchmarks. To this end, we show that users can collect a small amount of in in-domain data to improve performance for their specific use cases via dataset augmentation. Next, while we focus on the two-model routing setting in this work, promising future direction would be to extend this approach to multiple models. Finally, rather than there being single best router for all queries, the decision of which router to use should be based holistically on latency and cost requirements, as well as the types of queries handled. In our experiments, we observe that performance between different routers trained on the same dataset can vary widely on the same benchmark without clear explanation we leave further investigation into this for future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
We are grateful to Kourosh Hakhamaneshi, Goku Mohandas, Arman Zharmagambetov and Anastasiia Razdaibiedina for their valuable discussions and feedback on this work. This work in part supported by gifts from Accenture, AMD, Anyscale, Google, IBM, Intel, Microsoft, Mohamed Bin Zayed University of Artificial Intelligence, Samsung SDS, SAP, Uber, and VMware.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint wrXiv:2303.08774 2023.
Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui, and Mausam. Automix: Automatically mixing language models, 2024. URL
https: araiv.ii......v.. http//bs/223000 12963
AI@Meta. Llama 3.1 model card, 2024a. URL https: htw..tttbbcoo.ome.. www..//ooo///...ooooom... 1amaa /
Llamaammmeesssallld////m//leeeed 09-29.
1/MODEL CARD. md Accessed: 2024-
AI@ Meta. Introducing meta llama 3: The most capable openly available Ilm to date, 2024b. URL
https: www.m..e.aa....mmmmm.mmmm.mme... www www.o//w/o///.oooooooooooooooeeo meeaa 1lama- 3/
Accessed: 2024-05-21.
Anthropic. "introducing the next generation claude", 2024. URL
com/news/cle hwtcc/aaaeeeeeee Family/ Accessed: 2024-05-22.
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL https: http://aaxxvv.o.//a... www./bsg./oooooooo 23.923309200 16609
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324-34 3454443451111 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rXiv:2303.122122 2023.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.051 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.
11

---

Published as a conference paper at ICLR 2025
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024. URL
httpss://ppeneviie wtt.:/rrt//fftttt/ee/ee
net forum? id fmmt
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang. Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system, 2024. URL httt:s::/aaxxv.orgg//e/
12031
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04003 2024.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https::/arxxie org/ abs 1412. 6980
Ychuda Koren, Robert Bell, and Chris Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, 2009.
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing the expert: Efficient reward-guided ensemble of large language models, 2023. URL
btttps://arivv http:///wwrr//a//21....8888
Martian. Martian router, 2024. UtL/https::/itttmrrtttt com/
Accessed: 2024-06-30.
OpenAl. Gpt-4 technical report. arXiv preprint hrtppimmmaaxi::2333.088774...2
OpenAl. Openai pricing, 2024. URL https: com///aaiccmcaaaoooo httm//appprrinng 2024-06-30.
Accessed:
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019.
Together.AI. Together.ai pricing, 2024. URL cessed: 2024-06-30.
Ac-
Andreas Töscher, Michael Jahrer, and Robert M Bell. The bigchaos solution to the netflix grand prize. Netflix prize documentation, pp. 1-52, 2009.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arX.v:2307.00288. 2023.
UnifyAL Unifyai, 2024. URL http:::tt:///
Accessed: 2024-06-30.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017
12

---

Published as a conference paper at ICLR 2025
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 219.022. 2021.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Truck, 2023. URL https
//openreview.o net/ wtwpre/wwwwe.nneeooommeeee.e... dorumiiaoue
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023.
13

---

Published as a conference paper at ICLR 2025
A
ARENA MODEL TIERS
Tier
Models
Tier
spt-4-0125-preview, 1p1-4-110--prrviee
Tier
gpt-4-0314, gpt-4-0613, mistral-medium, claude-,,
Tier
claude-2.0, mixxaal--8777---tttt-000 http://wwo.....aa..e.eeeeeeiiiio
gpt-3.5-turbo-
0314, gpt-3.5-turbo-06113, gemini-pro, -0-....5.tub----222,ccacoddoo chaud-iinntssccuudd--sssat-tttt yi-34b- chat, www.rttng-m-----aapna wizardlm-70b, vicuna-33b, tulu-2-dpo-70b, nous-hermes-2-
mixtral-8x7b-dpo, 1lama-2-70b-chat, oopnc22-
Tier 3
llama2-70b-sterlln-chh,, ppix-70b-online, dolphin-2.2.1-mistrall gpt-3.5-turbo-
1106, ddepseeck-llm-6b-chhe. openhermes-2.55 2.5tmmes-225-mmmtra--755 openchat-3.5-0106
wizardlm-13b, mistral-7b-instuutt-v0..2 solar- 1aaa-10007--itttucttl.... zephyr-7b-beta,
zephyr-7b-alpha, codellama-33b-instuee. mpt-30b-chat, 1lama-2-13b-chat, vicuna-13b,
qwenl.5-7b-chat, 7p-o-in-,,,,,,, falcon-180b-che llama-2-7b-chat, guanaco-33b,
qwen-14b-chat
Tier
stripedhyena--ous--20 7b, mistral- diistral-bb-nnsse ib-instuct, vicuna-7b, wwwel.p.ww..l..he..... palm-2
Tier
koala-13b, chatglm3-6b,
Tier
mpt-7h-chat, RWK KWKK-4-Raven- 14B, chatglm2-6b, alpaca- oasst-pythia- 12h
Tier
fastchat-15-3b chatgim-6b
Tier 8
dolly-v2-12b, stablein-tunncd-alpha-a
Tier
llama-13b
B DATA CONTAMINATION
We check for cross-contamination between our evaluation dataset and the preference data used for training using embedding similarity search. Embeddings are generated for the evaluation and training data using OpenAl's text -etteddinn------eeeee 3mmag-33-aa11 model. For each evaluation example, we perform similarity search across all training data with threshold of 0.95, returning a list of contaminated examples. We discard these evaluation examples and report results on uncontaminated scores.
C BENCHMARK-DATASER SIMILARITY
Let Let (b1, bg..... byr be the embeddings the prompts for a given benchmark and eD {d1.d2.... .2..... dm) }o be the embeddings of specific preference dataset Ppect where and m are the total number of evaluation and preference data samples respectively. We define the benchmark-datt similarity score S(B, Dpedl for each benchmark B as the average maximum similarity for each evaluation prompt across all dataset samples:
by d, | dlldd
S(B, preet)
max Sm
(14)
71
We opt to use only the maximum similarity score because having small number of samples of preference data that are very similar to the user's query is most valuable for efficient query routing, as opposed to having many samples that are less similar to the user prompt.
D COST CALCULATION
Since our evaluations are performed with the gpt-4-110 endpoint, we use its pricing (S10 per million input tokens and $30 per 1 million output tokens) in our analysis. For the sake of simplicity. we assume the routers will be mostly handling short prompts in a single turn setting. We find the average input prompt in the training set to be 95 tokens long, and the average output responses to be 264 tokens long. This means the input/output tokens ratio is roughly 264
Using these information,
96
0100000000
estimate the average cost of using GPT-4 be:
24.7 USD per
96-4 064
14

---

Published as a conference paper at ICLR 2025
million tokens. For Mixtral 8x7B, assume the same price for both input and output tokens, which makes the average cost $0.24 USD per million tokens.
E INDEPENDENT BENCHMARKS
MT Bench
MT Bench
GPT-4
GPT-4
Hartaag
Unils
Rendom
Random Matri (A)
Matrix Factorization (A)
Llama 70B
Causal
Mixtral
Causel LUM (AI
0
40
60
80
100
20
80
100
Calls GPT-4
Calls GPT-4
Figure 2: Performance of our routers as compared to other routing systems on MT Bench. Our routers demonstrate competitive performance, achieving stronger performance than existing routers for the same cost.
In Figure we present the performance of our best-performing routers MT Bench compared to Unify AI (UnifyAl 2024) and Martian (Martian 2024), two existing commercial offerings for LLM routing.
Here, we route between gpt-4-turbo 2024 04-09 (OpenAl 2023) as Mg. and either mi Set_ral-xx7-iint.eee
(Jiang al. 2024 1 lama-22 - 2-72-07 70b-chat (Touvron et al.) 2023) as Ma depending on which model each system supports. For Unify AI, we select the best- performing router configuration the user dashboard and use for benchmarking. For Martian, optimize for performance and specify the maximum cost per million tokens as $10.45, approximating this value using public inference costs (OpenAl 2024 Together.Al 2024) based 1:1 input:output token ratio so that 50% of calls are routed to GPT-4.
Both the matrix factorization router and causal LLM routers perform very competitively when trained on Darena + Djudge. outperforming the commercial routing systems by achieving the same performance with up to 40% fewer calls routed to GPT-4.
ADDITIONAL PLOTS
We include additional plots for the results presented in Section 5.1
15

---

Published as a conference paper at ICLR 2025
MT Bench
MT Bench
Ra idom
GPT-4
SW Ranking
Fectorization
BERT
Jsal L.M
Rancom
Ranking
Matri Facto 12ation (9)
BERT (A)
Mixtral
Mixtral
0
20
40
60
80
100
20
40
60
80
100
Calls to GPT-4
Calls to GPT-4
Figure 3: MT Bench performance for all routers.
MMLU
MMLU
GPT-4
Ra Random
GPT-4
Ranking
Matrix Factorization
BERT
SW Ranking (4)
Matrix Facto aettoization (A)
Mixtral
Mixtral
Causal LUM (AI
100
80
% Calls to GPT-4
Calls to GPT-4
Figure 4: 5-shot MMLU performance for all routers.
GSMBK
GSMBK
GPT-4
Random
GPT-4
SW Ranking
Matrix Fectorization
Ca Jsel LLM
Rancom
SW Ranking
Matrix Facto
BERT (A)
Mixtral
Mixtral
Causal LLM (AI
0
20
40
60
80
100
20
40
60
80
100
% Calls to GPT-4
Calls to GPT-4
Figure 5: 8-shot GSM8K performance for all routers.
16


Reviews:
[{"review_id": "Cndn7h17jH", "reviewer": "Reviewer_V25N", "summary": "This paper introduces RouteLLM, a framework for training router models that direct queries between stronger and weaker LLMs to optimise cost-performance tradeoffs. It employs preference data from Chatbot Arena. Empirical results show that data augmentation is important to letting the routers outperform a random baseline on MMLU and GSM8K. It also demonstrates that the routers generalise across domains, and do not need to be retrained.", "questions": "1. How well do the routers do at separating models which are much closer in ability e.g. two different 7B models?\n2. How well do the routers do if the two models are not strictly \"stronger\" or \"weaker\", but rather have been finetuned to do different tasks?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper presents a novel framework of directly using human preference data, as opposed to reward models, to route between a pair of LLMs.\n2. The empirical evaluation is comprehensive, touching multiple architectures, LLMs and common benchmarks.\n3. Both metrics of \"average performance gap recovered\" and \"call-performance threshold\" are clear reflections of real world considerations: the general ability for the router to close the performance gap between the better and worse model, as well as the cost to doing so for a minimum quality bar.\n4. The increasing variation in LLM quality and cost makes it more important to be able to efficiently tradeoff between cost and performance. On MT Bench, RouteLLM is able to achieve comparable performance to GPT-4 with a cost saving of ~3.7x.", "weaknesses": "1. The paper focuses on a binary routing of \"strong\" versus \"weak\" model, but doesn't consider other binary differences between models.\n2. The paper does not discuss in depth why certain architectures perform better or worse, especially with respect to the improvement in the causal LLM's performance when faced with data augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R4qducnFBL", "reviewer": "Reviewer_GuJg", "summary": "Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. This paper proposes a routing approach, RouteLLM, to dynamically select between a stronger and weaker LLM during inference. Experiments on three real-world benchmarks demonstrate the effectiveness of RouteLLM.", "questions": "Q1. In Sec 3.1, authors introduced the cost threshold \\lambda. It is unclear that how to choose the right cost threshold when user have specific cost budgets.\n\nQ2. Some details in overhead analysis are unclear. Both SW ranking and matrix factorization rely on embeddings generated by text-embedding-3-small. Are the costs incurred by embedding extraction included in the overhead analysis? Also, given that the model size ratio between BERT-base (110M) and causal LLM (8B) is 110M / 8B ~= 1%, it is surprising to see the cost overhead of BERT-base is ~60% of the causal LLM, and the achieved throughput is only 60% higher, according to Table 7. More details on how the overheads are estimated could be very helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1. The proposed approach, RouteLLM, is able to achieve over 2x cost savings on popular benchmarks with\nminimal impact on response quality.\n\nS2. Authors demonstrate that RouteLLM enables routers to generalize to unseen data while maintaining strong performance across multiple LLMs.", "weaknesses": "W1. Unclear novelty and limited technical contribution. Training a router to harness the respective strengths of different LLMs has been widely studied [1,2,3,4]. Specifically, how to generalize LLM routing to OOD data has been studied in [5], how to use LLMs to generate more training data to help improve routing performance has been explored in [6], which authors did not compare to. Moreover, some technology (SW ranking) proposed in this paper shares unignorable similarity to prior work [7]. \n\nW2. Weak baselines. Provided the rich literature on this topic as aforementioned, considering a random router as the only baseline is insufficient in this work. The effectiveness of RouteLLM could be further demonstrated if authors could compare it to more advanced baselines (e.g., a subset from [1-6]).\n\nW3. In Sec 5.5, authors provided the overhead analysis. Notably, SW ranking is both expensive ($37 / 1M requests) and slow (2.9 requests / second), which makes it hard to use in practice.\n\n\nReferences:  \n[1] Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models, https://arxiv.org/pdf/2311.08692  \n[2] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, https://arxiv.org/abs/2404.14618  \n[3] Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling, https://arxiv.org/pdf/2308.06077  \n[4] ROUTERBENCH: A Benchmark for Multi-LLM Routing System, https://arxiv.org/pdf/2403.12031  \n[5] Large Language Model Routing with Benchmark Datasets, https://arxiv.org/pdf/2309.15789  \n[6] Routoo: Learning to Route to Large Language Models Effectively, https://arxiv.org/abs/2401.13979  \n[7] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, https://arxiv.org/abs/2403.04132", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m34snPRzNN", "reviewer": "Reviewer_7nQc", "summary": "The paper presents a framework for training router models using human preference data and data augmentation, achieving over 2x cost savings on popular benchmarks with minimal impact on response quality. The authors employ a binary routing approach to direct simple queries to a cost-effective model (e.g., Mixtral-8x7B) and complex queries to a stronger model (e.g., GPT-4). They demonstrate generalization across unseen data and new LLMs without retraining, providing a single trained router adaptable to multiple use cases.", "questions": "- Does the paper provide guidance on selecting the most suitable win prediction method across various scenarios?\n- Could insights be provided on optimal $\\alpha$ values for different query types, including a breakdown of routing decisions under varying $\\alpha$ thresholds?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Demonstrates significant cost savings (over 2x) without compromising response quality, verified on MMLU, MT Bench, and GSM8K.\n- Introduces APGR to quantify performance relative to cost, effectively balancing quality and cost constraints (Eq. 7).\n- Implements diverse methods for defining the win prediction model.\n- Demonstrates robustness and adaptability, as the router generalizes effectively to unseen LLM pairs and domains without retraining.", "weaknesses": "- Lacks detailed analysis of routing patterns under different $\\alpha$ values, such as which query types tend to be routed to strong vs. weak models, making it unclear how to set optimal $\\alpha$ values for specific use cases (Sec. 5.1).\n- Insufficient exploration of the router's decision-making robustness, especially regarding handling ambiguous queries where strong and weak models may perform similarly.\n- Performance still heavily depends on data augmentation with high-cost LLM-judged labels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)