RouteLLM: Learning to Route LLMs from Preference Data

Authors: Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, Ion Stoica
Primary area: generative models
Keywords: Large language models, query routing

Abstract:
Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. Powerful models offer better results but are expensive, while smaller models are more cost-effective but less capable. To address this trade-off, we introduce a training framework for learning efficient router models that dynamically select between a stronger and weaker LLM during inference. Our framework leverages human preference data and employs data augmentation techniques to enhance performance. Evaluations on public benchmarks show that our approach can reduce costs by over 2 times without sacrificing response quality. Moreover, our routers exhibit strong generalization capabilities, maintaining performance even when routing between LLMs not included in training. This highlights the potential of our framework to deliver cost-effective, high-performance LLM solutions.

Decision: accept (poster)
Reviewer scores: 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730677608587

thread_type:
official_review

<thread_object>
creator: reviewer_7nqc

<content>
summary:
The paper presents a framework for training router models using human preference data and data augmentation, achieving over 2x cost savings on popular benchmarks with minimal impact on response quality. The authors employ a binary routing approach to direct simple queries to a cost-effective model (e.g., Mixtral-8x7B) and complex queries to a stronger model (e.g., GPT-4). They demonstrate generalization across unseen data and new LLMs without retraining, providing a single trained router adaptable to multiple use cases.

strengths:
- Demonstrates significant cost savings (over 2x) without compromising response quality, verified on MMLU, MT Bench, and GSM8K.
- Introduces APGR to quantify performance relative to cost, effectively balancing quality and cost constraints (Eq. 7).
- Implements diverse methods for defining the win prediction model.
- Demonstrates robustness and adaptability, as the router generalizes effectively to unseen LLM pairs and domains without retraining.

weaknesses:
- Lacks detailed analysis of routing patterns under different $\alpha$ values, such as which query types tend to be routed to strong vs. weak models, making it unclear how to set optimal $\alpha$ values for specific use cases (Sec. 5.1).
- Insufficient exploration of the router's decision-making robustness, especially regarding handling ambiguous queries where strong and weak models may perform similarly.
- Performance still heavily depends on data augmentation with high-cost LLM-judged labels.

questions:
- Does the paper provide guidance on selecting the most suitable win prediction method across various scenarios?
- Could insights be provided on optimal $\alpha$ values for different query types, including a breakdown of routing decisions under varying $\alpha$ thresholds?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679883997

thread_type:
official_review

<thread_object>
creator: reviewer_gujg

<content>
summary:
Large language models (LLMs) excel at a wide range of tasks, but choosing the right model often involves balancing performance and cost. This paper proposes a routing approach, RouteLLM, to dynamically select between a stronger and weaker LLM during inference. Experiments on three real-world benchmarks demonstrate the effectiveness of RouteLLM.

strengths:
S1. The proposed approach, RouteLLM, is able to achieve over 2x cost savings on popular benchmarks with
minimal impact on response quality.

S2. Authors demonstrate that RouteLLM enables routers to generalize to unseen data while maintaining strong performance across multiple LLMs.

weaknesses:
W1. Unclear novelty and limited technical contribution. Training a router to harness the respective strengths of different LLMs has been widely studied [1,2,3,4]. Specifically, how to generalize LLM routing to OOD data has been studied in [5], how to use LLMs to generate more training data to help improve routing performance has been explored in [6], which authors did not compare to. Moreover, some technology (SW ranking) proposed in this paper shares unignorable similarity to prior work [7]. 

W2. Weak baselines. Provided the rich literature on this topic as aforementioned, considering a random router as the only baseline is insufficient in this work. The effectiveness of RouteLLM could be further demonstrated if authors could compare it to more advanced baselines (e.g., a subset from [1-6]).

W3. In Sec 5.5, authors provided the overhead analysis. Notably, SW ranking is both expensive ($37 / 1M requests) and slow (2.9 requests / second), which makes it hard to use in practice.


References:  
[1] Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models, https://arxiv.org/pdf/2311.08692  
[2] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, https://arxiv.org/abs/2404.14618  
[3] Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling, https://arxiv.org/pdf/2308.06077  
[4] ROUTERBENCH: A Benchmark for Multi-LLM Routing System, https://arxiv.org/pdf/2403.12031  
[5] Large Language Model Routing with Benchmark Datasets, https://arxiv.org/pdf/2309.15789  
[6] Routoo: Learning to Route to Large Language Models Effectively, https://arxiv.org/abs/2401.13979  
[7] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, https://arxiv.org/abs/2403.04132

questions:
Q1. In Sec 3.1, authors introduced the cost threshold \lambda. It is unclear that how to choose the right cost threshold when user have specific cost budgets.

Q2. Some details in overhead analysis are unclear. Both SW ranking and matrix factorization rely on embeddings generated by text-embedding-3-small. Are the costs incurred by embedding extraction included in the overhead analysis? Also, given that the model size ratio between BERT-base (110M) and causal LLM (8B) is 110M / 8B ~= 1%, it is surprising to see the cost overhead of BERT-base is ~60% of the causal LLM, and the achieved throughput is only 60% higher, according to Table 7. More details on how the overheads are estimated could be very helpful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719727937

thread_type:
official_review

<thread_object>
creator: reviewer_v25n

<content>
summary:
This paper introduces RouteLLM, a framework for training router models that direct queries between stronger and weaker LLMs to optimise cost-performance tradeoffs. It employs preference data from Chatbot Arena. Empirical results show that data augmentation is important to letting the routers outperform a random baseline on MMLU and GSM8K. It also demonstrates that the routers generalise across domains, and do not need to be retrained.

strengths:
1. The paper presents a novel framework of directly using human preference data, as opposed to reward models, to route between a pair of LLMs.
2. The empirical evaluation is comprehensive, touching multiple architectures, LLMs and common benchmarks.
3. Both metrics of "average performance gap recovered" and "call-performance threshold" are clear reflections of real world considerations: the general ability for the router to close the performance gap between the better and worse model, as well as the cost to doing so for a minimum quality bar.
4. The increasing variation in LLM quality and cost makes it more important to be able to efficiently tradeoff between cost and performance. On MT Bench, RouteLLM is able to achieve comparable performance to GPT-4 with a cost saving of ~3.7x.

weaknesses:
1. The paper focuses on a binary routing of "strong" versus "weak" model, but doesn't consider other binary differences between models.
2. The paper does not discuss in depth why certain architectures perform better or worse, especially with respect to the improvement in the causal LLM's performance when faced with data augmentation.

questions:
1. How well do the routers do at separating models which are much closer in ability e.g. two different 7B models?
2. How well do the routers do if the two models are not strictly "stronger" or "weaker", but rather have been finetuned to do different tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732475416925

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Main Rebuttal

comment:
We thank all the reviewers for their insightful comments and questions! Since submission, we have conducted additional experiments in response to reviewers’ questions and sought to clarify concerns raised. 

**On deciding the cost threshold** Reviewer 2 and Reviewer 3 raised questions around how users should determine what cost threshold to use for each router. Our suggested approach for users is to calibrate cost thresholds based on using a sample of the types of queries they expect to receive. Using the cost of the 2 models used for routing and their specified cost budget, users can first determine the percentage of queries that they would like to route to the stronger model. Given the sample of queries, users can then execute the router over these queries to obtain the estimated win probability for each query. Based on this, users can calculate the appropriate cost threshold to maximize routing performance while respecting the desired cost budget. 

We provide a script `calibrate_threshold.py` in our open-source framework that automates this calibration (included as supplementary material in the submission). Notably, we used this process to deploy the causal LLM router for a live evaluation in collaboration with the Chatbot Arena team. Specifically, we calibrated the cost threshold based on a public dataset of 55k Chatbot Arena queries. By doing this, our router achieved a **12% ELO improvement** over the random router on new unseen queries, demonstrating its effectiveness. 

**On the differences between different architectures and how to select them** Reviewer 1 and Reviewer 3 requested details about the differences between routers and guidance around router selection. To this end, we note that the selection of the optimal router requires a comprehensive evaluation of latency, cost constraints, and the availability of training data for the router model. Our experiments show that even with Chatbot Arena data, consisting of 65K human-labeled samples, surpassing the random baseline remains challenging, highlighting the complexity of the routing problem. Non-parametric methods, such as SW Ranking and MF, perform consistently well on Chatbot Arena data, often outperforming LLM-based classifiers and exhibiting stronger generalization across different benchmarks. The availability of high-quality labeled data, generated cost-effectively via the LLM-as-a-judge approach, has proven to be more critical to the success of the routing model than the specific architecture employed. Furthermore, the Causal LLM router, with its large number of parameters, demonstrates a clear reliance on this additional data to achieve competitive performance, as it is particularly susceptible to overfitting in low-data regimes. We will include this discussion in the next revision of the paper.

We are happy to answer any further questions reviewers may have.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes RouteLLM to dynamically choose between a stronger and weaker LLM during inference, achieving 2x cost savings on 3 benchmarks with minimal impact on response quality. Empirical results show that data augmentation is important to letting the routers outperform a random baseline and generalise across domains without needing to be retrained.


Strengths:
* Demonstrates significant cost savings (over 2x) without compromising response quality [Reviewer V25N, 7nQc, GuJg]
* Introduces APGR to quantify performance relative to cost, effectively balancing quality and cost constraints [Reviewer 7nQc and V25N]
* Demonstrates robustness and adaptability, as the router generalizes effectively to unseen LLM pairs and domains without retraining [all reviewers]
* Authors also provide a process of calibration for estimating the right cost threshold to calibrate based on both the target data distribution and router used.

Weakness
* The model only considers  binary routing of strong and weak models, but does not consider other binary differences between models such as coding-specific versus generalist models or English-only versus multilingual models. To reviewer V25N comment on this, authors claim that it is applicable to these other binary differences between models, without any substantiations.