Routoo: Learning to Route to Large Language Models Effectively

## Abstract

Abstract LLMs with superior response quality—particularly larger or closed-source models—often come with higher inference costs, making their deployment inefficient and costly. Meanwhile, developing foundational LLMs from scratch is becoming increasingly resource-intensive and impractical for many applications. To address the challenge of balancing quality and cost, we introduce Routoo, an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement.
Routoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given prompt without executing them. The cost-aware selector module then selects the most suitable model based on these predictions and constraints such as cost and latency, significantly reducing inference costs for the same quality.
We evaluated Routoo using the MMLU benchmark across 57 domains employing open-source models. Our results show that Routoo matches the performance of the Mixtral 8x7b model while reducing inference costs by one-third. Additionally, by allowing increased costs, Routoo surpasses Mixtral’s accuracy by over 5% at equivalent costs, achieving an accuracy of 75.9%. When integrating GPT4 into our model pool, Routoo nearly matches GPT4’s performance at half the cost and exceeds it with a 25% cost reduction.
These outcomes highlight Routoo’s potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs.

## 1 Introduction

LLMs have achieved remarkable success across a wide range of natural language processing tasks. However, this success comes with a significant trade-off between performance and cost: larger models generally offer better response quality but incur higher inference costs than their smaller counterparts. This increased inference cost poses challenges for deploying LLMs in real-world applications where computational resources, latency, and cost-efficiency are critical considerations.

At the same time, developing foundational LLMs  (OpenAI et al., ; Dubey et al., ; Yang et al., ; DeepSeek-AI et al., ; Jiang et al., ; ; Touvron et al., ) from scratch is becoming increasingly capital-intensive, requiring vast computational resources and extensive, high-quality data (Minaee et al., ). As the field approaches the limits of network size and data capacity, the improvements gained from training ever-larger models are diminishing (Udandarao et al., ). This situation underscores the need for alternative approaches that can achieve high performance without incurring prohibitive development and inference costs.

Moreover, many practical tasks do not necessitate the intricate reasoning capabilities of the largest models like GPT-4; instead, they can be efficiently handled by smaller, more cost-effective models (Zaharia et al., ; Ding et al., ; Shnitzer et al., ; Chen et al., ). The capabilities of existing LLMs appear to be complementary to a significant degree. For example, on the MMLU benchmark (Hendrycks et al., ), selecting the optimal open-source model for each question could hypothetically yield an accuracy of 97.5% at a computational cost similar to that of a 13-billion-parameter model (see Appendix  A for details). In contrast, GPT-4 (OpenAI et al., ) achieves an accuracy of 86.4%, while Mixtral 8×7b (Jiang et al., ), one of the leading open-source models,(^1^11As of the time of writing this paper.) reaches 70%. These observations suggest that integrating the knowledge of multiple LLMs could lead to new state-of-the-art models without the need to train from scratch while significanly reducing the inference cost.
However, effectively leveraging the vast and rapidly growing ecosystem of LLMs poses significant challenges. With approximately 450,000 models available on Hugging Face,(^2^22[https://huggingface.co](https://huggingface.co)) keeping track of the latest advancements and integrating them efficiently is a non-trivial task. Traditional approaches like Mixture of Experts (MoE) models (Abdin et al., ; Lieber et al., ; Jiang et al., ; Fedus et al., ; Shazeer et al., ) are limited by the need to load all expert parameters onto a single high-end machine, hindering scalability and flexibility. There is a need for an architecture that can dynamically and efficiently leverage multiple existing LLMs to optimize performance while controlling inference costs.

To address these challenges, we propose Routoo , an architecture designed to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Routoo provides controllability over the trade-off between inference cost and quality, enabling significant reductions in inference costs for a given quality requirement. By intelligently leveraging a universe of trained LLMs, Routoo addresses both the inference cost problem and the development cost of building LLMs from scratch, creating a composite high-performance model without the need for extensive retraining.

Routoo comprises two key components: a performance predictor and cost-aware selector. The performance predictor is a lightweight LLM that estimates the expected performance of various underlying LLMs on a given query without executing them. Based on these predictions and constraints such as cost and latency, the cost-aware selector module selects the most suitable model. For instance, when a task is predicted to be performed nearly equally well by a smaller model or a larger, more expensive model, Routoo will opt for the smaller model when speed and cost-efficiency are prioritized. This approach ensures optimal resource utilization without compromising on quality.

Our architecture marks a significant departure from traditional MoE. While MoE relies on gating over various expert sub-networks within each layer to predict the next token, it requires all expert parameters to be loaded onto a single, high-end machine. This limitation hinders scalability in the number of experts. In contrast, each ’expert’ within our system operates independently and can be hosted on different machines, potentially utilizing a different neural network architecture. This flexibility enables Routoo to incorporate a vast array of experts, ranging from specialized domain models to general-purpose ones, and to scale to a large number of experts without the limitations imposed by MoE models.

We evaluate our Routoo on MMLU benchmark (Hendrycks et al., ). We show that it achieves competitive performance with Mixtral 8x7b (Jiang et al., ) while only consuming two-thirds of its inference cost. Increasing the cost budget allows Routoo to outperform Mixtral by 5% at the same cost level, reaching an accuracy of 75.9%. By adding GPT4 (OpenAI et al., ) as one of the underlying experts, our Routoo achieves competitive performance with GPT4 at half the cost and exceeds it with a 25% cost reduction.
These outcomes highlight Routoo’s potential to significantly reduce inference costs without compromising quality, and even to establish new state-of-the-art results by leveraging the collective capabilities of multiple LLMs. By providing controllability over cost and quality trade-offs, Routoo offers an efficient approach to high-performance language modeling without the need for expensive model training from scratch.

To summarize, our contributions are:

- •
We propose Routoo, an LLM-based system that intelligently identifies the best-performing LLM for a given query while considering constraints such as cost and latency, effectively integrating the knowledge of multiple LLMs to create a high-performance model without the need for training from scratch.
- •
We evaluate our architecture on MMLU benchmark, and significantly outperform Mixtral 8x7b by 5% with similar inference cost. Also, our Routoo achieves competitive performance with GPT4 at half the inference cost and surpasses it by reducing the inference cost by 25%.

## 2 Related Work

#### Mixture-of-Experts.

MoE architecture includes a gating mechanism to integrate various expert sub-networks within each layer, guiding the prediction of the next token (Shazeer et al., ). This approach is used in recent foundational models, including Mixtral 8x7b (Jiang et al., ). Sukhbaatar et al. () fine-tuned LLaMa 7b (Touvron et al., ) on four different domains to create domain-specific expert LLMs. They then proposed combining the FFNs of these expert LLMs to construct an MoE. Tang et al. () proposed merging most parameters while scaling up MLP layrers of Transformer into a weight-ensembling MoE module that dynamically integrates shared and task-specific knowledge based on the input. Wang et al. () introduces a fusion gate at each Transformer layer to generate weights for a weighted average of outputs from a set of pre-trained LoRAs.

Figure: Figure 1: Routoo architecture has three main components: a performance predictor, cost-aware selector, and universe constructor. The universe constructor identifies the optimal set of complementary models from all available models. The performance predictor predicts the correctness of experts for a specified query, and the cost-aware selector chooses the underlying model by considering the cost and efficiency of each model.
Refer to caption: x1.png

#### Model Selection.

Previous works in selecting the best LLMs mainly focus on identifying the one that generates the most optimal output for a given input. Liu & Liu (); Ravaut et al. () proposed specialized scoring or re-ranking models that can be used for the generation tasks (summarisation, here). Speculative decoding (Kim et al., ; Leviathan et al., ) accelerates the decoding of expensive models by using small, efficient decoders for the ’easy’ steps. This approach can complement routing methods. LLM-BLENDER (Jiang et al., ) is an ensembling framework to reach better performance by mixing the results of LLMs with a ranking module, followed by a generation module to generate from top candidates of the ranker. FrugalGPT (Chen et al., ) executes experts sequentially until an expert reaches the acceptable generation performance. ZOOTER (Lu et al., ) proposed a reward-guided routing approach that distills rewards from training queries to train a routing function. HybridLLM (Ding et al., ) and RouteLLM (Ong et al., ) propose a routing approach to direct queries to the suitable expert, utilizing one strong and one weak LLMs.

Different from previous work, our Routoo identifies the most suitable expert without executing the inference of underlying LLMs. Additionally, our approach can be efficiently generalized to scenarios involving many underlying LLMs (not necessarily a tuple of weak and strong LLMs).

## 3 Architecture

### 3.1 Problem Formulation

Given a set of $M$ models $m_{1},m_{2},\ldots,m_{M}$ and a set of $N$ queries $q_{1},q_{2},\ldots,q_{N}$,(^3^33Section 3.4 will provide a detailed explanation of how we build the set of models from the pool of all available models.) our goal is to assign the most cost-effective model $m_{i}$ to each query $q_{j}$ that can accurately answer the query. Correctness and cost scores of each query-model pair are calculated as:

$$ $\displaystyle s_{m_{i},q_{j}}=\operatorname{Eval}(m_{i}(q_{j}))$ $\displaystyle c_{m_{i},q_{j}}=\operatorname{Cost}(m_{i},q_{j})$ $$

where $\operatorname{Eval}(.)$ computes the correctness score given the generated response $m_{i}(q_{j})$ of model $m_{i}$ for query $q_{j}$. $s_{m_{i},q_{j}}$ is an integer that ranges from 0 to $K-1$, indicating the correctness of the model’s response, where 0 is unacceptable, and $K-1$ is optimal. The cost of executing model $m_{i}$ for query $q_{j}$ is calculated by $\operatorname{Cost}(.)$ function.

The objective is to maximize the total correctness scores across all queries while keeping the total cost within a budget $B$:

$$ $\displaystyle\max_{\pi}$ $\displaystyle\ \sum_{j=1}^{N}s_{\pi(q_{j}),q_{j}}$ s.t. $\displaystyle\ \sum_{j=1}^{N}c_{\pi(q_{j}),q_{j}}\leq B$ $$

Here, $\pi(.)$ is the function that assigns each query to a model from the set of $M$ models, while considering both cost and correctness. In this formulation, the cost can be multi-faceted, encompassing aspects like computational cost, speed, etc. However, for simplicity, we consider a singular budget constraint $B$ in this context. To solve this without exhaustively testing every model on every query—a prohibitive approach—we divide the problem into two manageable parts:

- 1.
Performance Predictor. This component approximates the correctness score $s_{m_{i},q_{j}}$ for each model on each query. Given a model $m_{i}$ and a query $q_{j}$, it estimates the model’s performance without generating the response for the query, thereby significantly reducing the inference cost.
- 2.
Cost-Aware Selector. This module selects the best model for each query, balancing accuracy and cost-effectiveness by using correctness estimations of performance predictor.

These two components work together to dynamically and efficiently allocate models to queries.
Additionally, given the vast array of LLMs available for text generation,(^4^44According to the Huggingface platform ([https://huggingface.co](https://huggingface.co)), there are nearly 47,000 models available for the text generation task.) and considering the practical constraints on the number of models that can be actively served, we introduce Universe Constructor, which builds an initial complementary universe of models ($m_{1},m_{2},\ldots,m_{M}$) from a set of all available models. This universe aims to maximize performance by ensuring that the selected models are complementary to each other.

All three components are illustrated in Figure 1. In the following sections, we will delve deeper into each component.

### 3.2 Performance Predictor

The performance predictor is a lightweight LLM designed to estimate the effectiveness of each underlying LLM for a given query. It estimates the output of evaluation function ($s_{m_{i},q_{j}}$, introduced in Section 3.1) without executing model $m_{i}$ on query $q_{j}$. This prediction is formulated as:

$$ $\displaystyle\hat{s}_{m_{i},q_{j}}=\operatorname{Pred}(m_{i},q_{j})$ $$

Figure: Algorithm 1 Greedy algorithm for the optimization of universe construction.

where $\operatorname{Pred}(m_{i},q_{j})$ is the predictive model. Similar to $s_{m_{i},q_{j}}$, $\hat{s}_{m_{i},q_{j}}$ is an integer that ranges from 0 to $K-1$, 0 indicates an unacceptable result and $K-1$ represents the optimal outcome.

The process of $\operatorname{Pred}(m_{i},q_{j})$ is as follows:

$$ $\begin{split}\begin{cases}h_{q_{j}}=\operatorname{Enc}(q_{j})\\ h_{m_{i}}=\operatorname{Emb}(m_{i})\\ \hat{s}_{m_{i},q_{j}}=\operatorname{Linear}(h_{q_{j}}-h_{m_{i}})\end{cases} \end{split}$ $$

where $\operatorname{Enc}(.)$ is the encoder of the input query. Inspired by Radford et al. (), we use a decoder-only LLM as the query encoder and extract the embedding of the last token as the representation of the input query ($h_{q_{j}}\in\mathbb{R}^{h}$). $\operatorname{Emb}(.)$ is an embedding layer, where each embedding is assigned to a specific model. As a result, $h_{e_{i}}\in\mathbb{R}^{h}$ is the embedding representation of model $e_{i}$. $h$ is the hidden representation of the decoder-only model used. $\operatorname{Linear}(.)$ function is a projection matrix ($\mathbb{R}^{h\times K}$), computing the estimation score $\hat{s}_{m_{i},q_{j}}$.

The model is trained by minimising the cross-entropy loss function (Good, ) of $s_{m_{i},q_{j}}$ and $\hat{s}_{m_{i},q_{j}}$ over all $M$ models, and $N$ queries as:

$$ $\operatorname{min}\frac{1}{N\times M}\sum_{i=1}^{M}\sum_{j=1}^{N}\operatorname {CE}(s_{m_{i},q_{j}},\hat{s}_{m_{i},q_{j}})$ (1) $$

where $\operatorname{CE}(.)$ is the cross-entropy loss.

During the inference time, $\operatorname{argmax}(.)$ function is applied for a given model and query.

### 3.3 Cost-Aware Selector

The second phase of our architecture involves the selection step, where the estimated scores from the performance predictor are used to determine the optimal assignment of models to queries. Our objective is to maximize the overall effectiveness of the responses within a given budget constraint. The optimization problem is formulated as follows:

$$ $\displaystyle\max_{\pi}$ $\displaystyle\ \sum_{j=1}^{N}\hat{s}_{\pi(q_{j}),q_{j}}$ s.t. $\displaystyle\ \sum_{j=1}^{N}c_{\pi(q_{j}),q_{j}}\leq B$ $$

where $\pi(.)$ determines the model assignments and $B$ represents the predefined budget constraint.

We propose a greedy algorithm to approximate this optimization. Noting the challenges of accurately estimating the exact cost of a model for a specific query, instead, we consider the average cost $c_{i}$ for a model $m_{i}$ responding to an average length query and response. The algorithm includes the following steps:

- 1.
Performance-to-Cost Ratio: For each query $q_{j}$ and model $m_{i}$, calculate the performance-to-cost ratio. Let $\hat{s}_{m_{i},q_{j}}$ be the estimated correctness score of model $m_{i}$ for query $q_{j}$, and $c_{i}$ be the cost for model $m_{i}$. The ratio is calculated as:
$r_{m_{i},q_{j}}=\frac{\hat{s}_{m_{i},q_{j}}}{c_{i}^{\alpha}}$
where $\alpha$ is a parameter that adjusts the emphasis on cost. A higher $\alpha$ value increases the weight on cost efficiency, thereby favoring cheaper models and preserving more of the budget for subsequent assignments.
- 2.
Sorting and Assignment: For each query, sort the models by their performance-to-cost ratio in descending order. Assign the query to the model with the highest ratio that remains within the available budget.
- 3.
Budget Management: Monitor the accumulated cost. If selecting a model would exceed the budget, move to the next best model in terms of the ratio.

This approach efficiently balances the trade-off between performance and cost, effectively identifying the most suitable model for each query while adhering to budget constraints.

Table: Table 1: Performance and cost of running LLMs on MMLU benchmark. Accuracy is calculated based on OpenLLM Leaderboard setting Beeching et al. ().

### 3.4 Universe Constructor

With the abundance of models and the practical limitations on hosting several models, we introduce an optimization approach to build a complementary subset of models ($m_{1},m_{2},\ldots,m_{M}$). The goal is to select models that are complementary to achieve the highest performance, given a predefined limitation on number of serving models.

The objective is to select a subset of models that collectively provide the best coverage and performance across a set of queries. Formally, given a big pool of models $\Omega:(m_{1},m_{2},\ldots,m_{\hat{M}})$, a set of queries $Q:(q_{1},q_{2},\ldots,q_{L})$, and correctness scores of the models for these queries $s_{m_{i},q_{j}}$, we seek to find the optimal subset of models, that maximizes the following optimization problem:

$$ $\max_{U\subseteq\Omega,|U|=M}S(U)=\frac{1}{L}\sum_{j=1}^{L}\max_{i\in U}~{}s_{ m_{i},q_{j}}$ (2) $$

where $M$ is the desired number of serving models, and $U$ is a subset of selected models from $\Omega$. The function $S(U)$ represents the highest score achievable by the set $U$, quantifying the combined performance of the selected models. The optimal subset $S(U^{*})$ is used as the set of models ($m_{1},m_{2},\ldots,m_{M})$ in the optimization of Section 3.1.

Given the submodular nature of the maximum operation in our objective function (Equation 2), a greedy algorithm (as illustrated in Algorithm 1) can be employed to find an approximate solution with acceptable error bounds (Krause & Golovin, ). This method is particularly effective for large-scale problems where exact optimization is computationally prohibitive.

## 4 Results and Discussion

### 4.1 Experiment Setting

#### Evaluation Setting.

The evaluation of baselines and Routoo models is conducted using MMLU benchmark (Hendrycks et al., ), which comprises multiple-choice questions across 57 diverse domains, such as mathematics, law, computer science, biology, and US history. To be compatible with OpenLLM Leaderboard (Beeching et al., ), we use Eleuther AI Harness (Gao et al., ) to evaluate our models. (^5^55Specifically, the following branch is used: [https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463](https://github.com/EleutherAI/lm-evaluation-harness/tree/b281b0921b636bc36ad05c0b0b0763bd6dd43463).) Precisely, it calculates the likelihood of each choice in the question, and selects the answer with maximum likelihood, then ’accuracy’ is used as the evaluation metric. The overall performance is calculated as the average accuracy of the model in 57 domains. Routoo models will be made publicly available for the AI community to test on various benchmarks.

Figure: Figure 2: The performance of different Routoo models and baselines on MMLU benchmark, given different budget limitations.
Refer to caption: x2.png

#### Baselines.

Our comparison includes a range of both open-source and closed-source LLMs. These comprise LLaMa2 (Touvron et al., ) models with 7b, 13b, and 70b parameters, Mistral 7b (Jiang et al., ), Mixtral 8x7b (Jiang et al., ) (employing token-level MoE), alongside with GPT3.5 and GPT4 (OpenAI et al., ) as closed-source models. We could not find any publicly available trained models for routing or model integration. To the best of our knowledge, there are currently no open-source models that perform routing or dynamic selection among multiple LLMs that we could directly compare against.

#### Architecture Setting.

For the pool of universe constructor ($\Omega$), we use top 1,000 available models of OpenLLM leaderboard (Beeching et al., ) (^6^66[https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard).), Then, we set the maximum number of serving models ($M$) to 56 in Equation 2, meaning that we use 56 models for the routing optimization, defined in Section 3.1. For the performance predictor, we use Mistral 7b (v0.1) (Jiang et al., ) (^7^77[https://huggingface.co/mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).) as the query encoder. The number of levels in the evaluation score ($K$) is set to 2 for MMLU benchmark, meaning that $s_{m_{i},q_{j}}$ and $\hat{s}_{m_{i},q_{j}}$ are either 0 or 1. (^8^88As accuracy metric is used for MMLU benchmark in OpenLLM leaderboard (Beeching et al., ).) For fine-tuning, we apply LoRA method (Hu et al., ) with $r=1024,~{}\alpha=16,~{}\text{dropout}=0.05$ on query, key, and value matrices.

### 4.2 Training Data Preparation

Since the MMLU dataset (Hendrycks et al., ) lacks a training set, we build synthetic data to train the models. The synthetic questions are originated by two approaches: Filtering available open-source datasets and generating synthetic queries using GPT4 model (OpenAI et al., ).

#### Filtering Available Datasets.

We begin by collecting various multiple-choice QA datasets, such as ARC (Clark et al., ), MC-TEST (Richardson et al., ), OBQA (Mihaylov et al., ), RACE (Lai et al., ), and TruthfulQA (Lin et al., ). To identify questions with a high training signal (i.e., difficulty), we employed SOLAR-10.7B-v1.0 (Kim et al., ) (^9^99Available in Huggingface platform: [https://huggingface.co/upstage/SOLAR-10.7B-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-v1.0). We chose this model, as it is performing relatively well on MMLU section of OpenLLM benchmark (Beeching et al., ).) on this aggregated dataset, and calculate the evaluation score.(^10^1010MMLU evaluation system (introduced in Section 4.1) is used.) Queries on which the model under-performed (accuracy of 0) were retained, resulting in a set of 45,645 challenging training samples from an initial pool of 101,434. Additionally, 10,000 simpler questions were randomly selected from the initial dataset, bringing the total to 55,645 queries.

Figure: Figure 3: Per sub-category performances of our Routoo models and baselines on MMLU benchmark.
Refer to caption: x3.png

#### Generation with GPT4.

To diversify our training dataset further, we generated 20,000 synthetic queries using GPT4 model. These queries are generated by using seed data, which randomly sampled from the dataset mentioned above. Detailed specifications of the input prompts used for generation are documented in Appendix B.

Final set of queries contains nearly 75,000 samples. Datatrove library(^11^1111[https://github.com/huggingface/datatrove](https://github.com/huggingface/datatrove).) is used for decontamination. The set of queries in Universe Constructor ($Q:(q_{1},q_{2},\ldots,q_{L})$) is created by randomly sampling 1,000 queries from the training dataset.

### 4.3 Main Results

#### Our Routoo Models.

We present two variations of Routoo model: Routoo (open-source) and Routoo (mix). The former refers to a model that leverages 7b, 13b ,and 34b open-source models available on Huggingface as the input to our universe constructor. By additionally integrating GPT4 (OpenAI et al., ) to underlying models of Routoo (open-source), we create Routoo (mix).

#### Overall Results.

Comparison of our models and baselines are illustrated in Table 1.(^12^1212Inference costs are calculated based on price documentation of the following providers at the time of writing the paper: [https://www.together.ai](https://www.together.ai), [https://openai.com](https://openai.com). For GPT3.5 and GPT4 costs, the average of input and output costs are considered.) The cost of Routoo (open-source) is adjusted to match Mixtral 8x7b (Jiang et al., ), the best generic open-source LLM (at the time of writing this paper). Our model significantly outperforms Mixtral 8x7b by 5.27% absolute points with the same inference cost. Also, our underlying models can be executed on 1 GPU (e.g. A100 with 40 GB memory), while Mixtral model requires access to high-end computing resources  (e.g. GPUs with 100 GB memory). Compared to LLaMa2 70b, our model achieves significantly better performance (+6%), while reducing the cost by 33%. Impressively, Routoo (open-source) achieves competitive performance with GPT3.5 while reducing the cost by 73.3%.
Compared to closed-source LLMs, our Routoo (mix) model significantly outperforms GPT3.5 by 14.9% absolute point. Routoo (mix) reaches competitive performance with GPT4, while reducing the cost by almost 50%.

Figure: Figure 4: Routing distributions of different variants of Routoo models. The inference cost is provided for 1 million tokens.
Refer to caption: x4.png

#### Cost-Aware Selection.

Given that the cost-aware selector module can compute different routing distributions within a specified budget, we illustrate the curve of performance-cost in Figure 2 for both open-source and closed-source baselines, alongside with variations of Routoo models.(^13^1313$\alpha$ parameter, defined in Section 3.3, is set to 1, 0.1, and 0.01 for data points from left to right of each sub-figure, respectively.) Notably, our Routoo (open-source) reaches the performance of Mixtral 8x7b (Jiang et al., ) while decreasing the cost by 33%. Interestingly, Routoo (mix) achieves better performance than GPT4 (86.64 vs. 86.40) while reducing the inference cost by nearly 25%.
In summary, our cost-aware selector effectively offers an optimal trade-off between cost and performance, enabling both open-source and mix variants of the Routoo model to outperform individual LLMs in terms of performance (or cost) during inference.

#### Per Domain Comparison.

To further investigate the source of improvement in our models, we illustrates the distributions of performances on 17 sub-categories (Hendrycks et al., ) in Figure 3. A standout area of success is in STEM domains, such as mathematics and computer science, where Routoo particularly excels. This impressive performance is largely attributed to the incorporation of specialized small models (around 7b parameters) that are fine-tuned for tasks in mathematics (Yu et al., ; Shao et al., ) and coding (Rozière et al., ; Guo et al., ) by the community. Furthermore, this approach facilitates the identification of domains where there is a scarcity of experts. Then, future research can focus on improving the performance of these areas by developing domain-specific effective experts.

#### Routing Distribution.

Figure 4 presents the aggregated percentage usage of expert models by size for each Routoo pricing tier ($ per 1M tokens). It reveals that higher-priced options tend to utilize larger models more frequently. A substantial inclusion of effective smaller, 7-billion parameter expert models significantly enhances the cost-to-performance efficiency. This suggests that strategically increasing the use of such smaller experts could offer a more economical solution while maintaining high-quality outputs.

Finally, the cost of training a router is significantly lower than building foundational models e.g. GPT4 (OpenAI et al., ) and Mixtral (Jiang et al., ). This could pave a new path for building new frontiers at a much lower cost by integrating knowledge of current LLMs.

## 5 Conclusion and Future Work

In this paper, we proposed our Routoo architecture, a lightweight LLM-based model that is designed to inteligently routes the input query to the most suitable expert model given other constraints e.g. cost. We evaluated our architecture on MMLU benchmark (Hendrycks et al., ), which is a MCQA dataset with 57 different domains coverage. Routoo (open-source) achieved competitive performance with Mixtral 8x7b model (Jiang et al., ) with two-thirds of the inference cost. Increasing the budget limitation allows Routoo (open-source) to outperform Mixtral model by 5% with the same level of cost. By integrating GPT4 (OpenAI et al., ) model to underlying LLMs, our Routoo (mix) achieved competitive accuracy with GPT4 while reducing the inference cost by 50%, and even surpassing it while saving 25% of the cost. In general, our Routoo models provide an efficient trade-off between cost and performance during the inference.

In future, the Routoo’s ability to assess and understand the performance of existing models allows researchers to identify gaps in the AI landscape. It pinpoints domains where no existing expert excels or where larger models are inefficiently juggling tasks. This insight is invaluable, enabling us to strategically develop domain-specific models where they are needed most. Furthermore, our architecture facilitates easy integration of additional optimization criteria, including cost, speed, and privacy considerations.

## Appendix A Optimal Routing of Open-source LLMs on MMLU

In this experiment, we utilize all publicly available LLMs from the OpenLLM benchmark (Beeching et al., ) that have fewer than 34 billion parameters. Given that each query is directed to the most effective and cost-efficient model by an ideal performance predictor module within Routoo, the upper bound for the Routoo model in this scenario is nearly 97.5%. The following table shows the distribution of usage among LLMs of varying sizes:

Table: Table 2: The Distribution of usage among underlying LLMs with different sizes on MMLU (Hendrycks et al., ) test set.

The average size of the distribution above could be considered as an abstract model with approximately 13 billion parameters.

## Appendix B Synthetic Data Generation by GPT4

The following input prompt is used for generating synthetic data with GPT4 model (OpenAI et al., ):

For generation, we used OpenAI API (^14^1414[https://platform.openai.com/docs/overview](https://platform.openai.com/docs/overview).) with $\text{temperature}=0.7$. At each iteration, $N=100$, and we run it for 200 times. SUBJECT is used when the seed samples contain subject field e.g. development set of MMLU benchmark (Hendrycks et al., ). Seed samples are chosen randomly from datasets introduced in Section 4.2, alongside with development set of MMLU benchmark.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: - What is the comparison of the cost for obtaining Routoo’s training data (including the cost of generating the synthetic training data from GPT-4) plus doing inference, versus the cost for just running on Mistral/GPT/etc?
- How accurate is the Performance Predictor with predicting whether a given LLM will, indeed, answer a query correctly? Does it struggle more with queries from specialized domains, or other specific kinds of queries?
- The Performance Predictor takes as input the difference between the LLM’s and query’s embeddings. Are there other ways of combining these two inputs, such as convolutions or concatenations, that could perform better?
- Experiment comparing performance for different budget sizes and model universe sizes
- I would be interested to see results for different hyperparameters, for instance especially K. Additionally, the choice of M=56 seems like a very specific number, how robust is Routoo to these sorts of changes?
- Can you provide more information on the training process? You mention in the “Architecture Setting” paragraph of Section 4.1 that the Mistral query encoder is being fine-tuned with LORA– are you training the model embedding function Emb() and the projection matrix Linear() without LORA, but during the same time you’re fine-tuning Mistral? Also, how are the Emb() and Linear() weights initialized?

Overall, I think that Routoo might be promising, but I do not have confidence in its capabilities without seeing additional experimental results. My score is 3, but I will consider raising it if the authors could include more evaluations.
soundness: 2
strengths: - The paper is approachable and well-written
- Framing the balancing of performance and cost as an optimization problem with a greedy approximation is an elegant approach
- The finding that much of the performance gap can be addressed by domain-specialized models is interesting and provides context on why Routoo performs well
- As the authors state, many prior routing approaches (such as the cascading technique) execute multiple LLMs in sequence. Routoo’s inference-time technique of predicting models’ performances before actually using the models is clever.
- Routoo is flexible in that the “cost” consideration can easily be expanded to include criteria such as latency, and that the “performance” consideration can be any integer-valued performance-related score instead of just accuracy.
summary: - The authors propose Routoo, an architecture that learns to select an LLM best-suited to answer a specific query when weighing the factors of performance (accuracy) and cost. The idea is, we don’t need to pay extra for a larger or closed-source LLM on queries where we expect that a smaller, cheaper LLM will perform similarly.
- Routoo chooses the LLM for a query from among a universe of complementary models, and the authors describe a “Universe Constructor” that creates this set automatically.
- The key components of Routoo are the “Performance Predictor” and “Cost-Aware Selector”.
  - At inference time, we want to be able to predict which LLMs are likely to perform well on a given query. The Performance Predictor is an LLM that takes as input the query and a model from the available set, then predicts whether that given LLM will answer the query correctly. The performance predictor consists of an LLM for embedding the query, an embedding model for embedding the candidate LLM, and a linear layer for predicting the performance, given the embedded LLM and the embedded query.
  - The Cost-Aware Selector uses each LLM’s cost and Performance Predictor score to select an LLM to answer the query. This component uses a greedy algorithm that calculates the Performance-to-Cost ratio for each candidate model, then chooses the candidate model with the best ratio that fits in the cost budget.
- Evaluation is performed using the MMLU dataset. An open-source version of Routoo is built from each size of Llama2, Mistral 7b and Mixtral 8x7b, while a mixed-source version adds GPT3.5 and GPT4-turbo. 
  - For the main results, the closed- and mixed-source Routoos are compared for cost and performance against each of these individual models.
  - The performance-cost curve is compared for Routoo with different budgets and the individual LLMs
  - The performance of Routoo is compared against Llama2 70b and Mixtral 8x7b for each domain within MMLU
weaknesses: - The experimental results are promising but a little bit sparse right now, I list some more experiments that could be useful in the Questions section below
- It seems challenging to extend this method beyond datasets like MMLU that have multiple-choice answers
- The Universe Constructor builds a universe of candidate models in accordance with the paper’s Equation 2. This equation uniquely takes into account the model performance, and does not consider model cost. This approach should be modified to also consider including cheaper models at times, for instance by maximizing for performance-to-cost ratio (or some weighted version of performance-to-cost ratio that allows users to specify how much they want to prioritize cost).
- “We could not find any publicly available trained models for routing or model integration”: looking online for a few minutes, it seems this research area does have a dearth of methods with publicly released code. However, there might be some things on GitHub (e.g., https://github.com/emingenc/llm_adaptive_router) and the cascading technique (using the cheapest model, then going to a more expensive one if the cheaper one fails) would be relatively easy to implement as a routing-based baseline. Right now, the evaluations only compare Routoo against each individual LLM, with no other routing methods included.
- I don’t understand the claim in the Introduction (and restated at the very end of evaluation) that “Routoo addresses … the development cost of building LLMs from scratch, creating a composite high-performance model without the need for extensive retraining”. This makes it sound like Routoo is making it less costly to develop novel LLMs, when really Routoo is about choosing which pre-existing LLMs to use. If this claim is saying that Routoo is cheaper than, for instance, combining all the LLMs using MoE into a single very large model, then the claim holds up. However, model routing is already an existing technique that addresses this claim from this non-MoE perspective: it’s not something that Routoo itself is contributing. Model routing (including Routoo) and MoE already have very different use-cases, so the author’s comparisons of Routoo to MoE don’t seem very natural or could perhaps be better-articulated.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Q1. In Sec 3.3, authors consider an average cost for each model to estimate the model inference cost for a given test query. Since the inference cost heavily relies on the output length which is unknown beforehand, it is intriguing to see how good the cost estimation is. Analysis on estimation error and variance would be very helpful.

Q2. In Sec 3.3, the formulated problem is really a knapsack problem - select a subset of LLMs (items) such that the overall performance (profit) is maximized subject to cost budgets (weight threshold) - which has been widely studied in prior literature [1]. Provided that, greedy algorithms to knapsack problem has been known to be non-optimal in general cases. Authors may want to discuss related work and justify the specific algorithm design.

References:  
[1] Knapsack Problems, https://link.springer.com/book/10.1007/978-3-540-24777-7
soundness: 2
strengths: S1. This paper proposed Routoo, a routing framework that intelligently identifies the best-performing LLM for a given query while considering constraints such as cost and latency.

S2. Routoo achieves competitive performance with GPT4 at half the inference cost and surpasses it by reducing the inference cost by 25%.
summary: Large language models (LLMs) with superior performance often come with high inference costs, making their deployment inefficient and costly. This paper developed a routing framework, Routoo, to optimize the selection of LLMs for specific prompts based on performance, cost, and efficiency. Evaluation results on MMLU benchmark and 57 models demonstrate the effectiveness of Routoo.
weaknesses: W1. Unclear novelty and limited technical contribution. Training a router to harness the respective strengths of different LLMs has been widely studied [1,2,3,4,5]. Specifically, [5] studied a very similar setup - select LLMs to maximize (resp., minimize) overall performance (resp., cost) subject to a pre-defined cost budget (resp., performance threshold) - and developed various strategies including the greedy algorithm proposed in this work, which authors did not discuss or compare to.

W2. No baseline. Provided the rich literature on LLM routing as discussed above, no baseline included in the evaluation seems insufficient. This work can be greatly improved if at least one baseline from the general routing [1-4] and one baseline from the constrained routing [5] were included and compared to Routoo.

W3. Authors leveraged the Mistral-7B model as the query encoder, which may cause non-negligible cost & latency overheads. Authors may want to perform overhead analysis to address this concern.


References:  
[1] Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models, https://arxiv.org/pdf/2311.08692  
[2] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing, https://arxiv.org/pdf/2404.14618  
[3] ROUTERBENCH: A Benchmark for Multi-LLM Routing System, https://arxiv.org/pdf/2403.12031  
[4] Large Language Model Routing with Benchmark Datasets, https://arxiv.org/pdf/2309.15789  
[5] Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling, https://arxiv.org/pdf/2308.06077

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The Mistral7b based prediction has an inference cost. Is that included in the inference cost?
2. Is a LLM needed for prediction or would a lightweight retrieval system that retrieves scores on similar queries from the training data work well?
3. In Section 3.3, step 3 of the algorithm (budget management) mentions that the next best model is chosen if the budget is exceeded. However, the budget is the total budget across all queries so how is the ordering of this step performed across queries? Wouldn't the earlier queries starve the budget of the later queries, leading them to choose worse models?
soundness: 3
strengths: 1. This work introduces heuristics for optimizing for multiple queries with a total budget which is novel.
2. This Routoo system has great results where it Pareto-dominates state-of-the-art single LLMs on inference cost and MMLU performance.
3. The gathering of non-MMLU dataset for training along with synthetic data is meticulous in avoiding data leakage during evaluation.
4. The approach is sound: the predictive model for score prediction and budget constrained selector heuristic makes sense. The per-LLM embedding allows for the model to be inferenced only once-per-query instead of once-per-query-per-LLM which saves compute cost.
summary: This paper introduces a LLM model routing system called Routoo that selects what LLM to route queries to from a pool of LLMs of various capabilities and inference costs. For a particular query, and a particular LLM, a predictor model (based on Mistral7b) predicts the accuracy score, and an average cost heuristic estimates cost. Given a list of queries and a total budget, an optimization heuristic assigns the appropriate LLM for each query to achieve the highest overall accuracy within the total budget. On the MMLU benchmark, the Routoo system achieves higher accuracy at lower costs than using any single state-of-the-art model. Higher accuracy is achieved since the pool of models have specialists for particular type of queries and lower cost is achieved since easier queries don't need more expensive models to solve them.
weaknesses: While the approach in this work is sound, the contribution of the work over previous works may not be significant enough. Furthermore, the work is more appropriate for machine learning systems without a significant learning aspect.
1. There are related works regarding predictive routing with multiple-LLMs that are not discussed [3,4]
2. The advantages of specialist models improving scores over strong single-LLM models are covered in [1,2] so that aspect of the system is not very novel.
3. The concepts of budget conscious LLM routing is covered in [3,4] so that aspect of the system is not very novel either.
4. The only learning aspect of this work is the predictive model (Mistral7b with learned classification head). Without accuracy measurements, and comparisons with other models, there is little insight to be gained.

[1] Large Language Model Routing with Benchmark Datasets

[2] Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routing

[3] Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing

[4] RouteLLM: Learning to Route LLMs with Preference Data

Meta-review: This paper proposes a routing framework, Routoo, to balance the inference ost and quality. It comprises a performance predictor and cost-aware selector. Experiments show Routoo matches the performance of Mixtral 8x7b while reducing inference cost, and also nearly matches GPT4 with less cost.

Pros:
- Well written, and a clear description of the performance-cost trade-off in the routing framework.
- Competitive performance with GPT4 at reduced cost on the MMLU benchmark

Cons:
- Unclear novelty and lack of comparison with prior works
- Limited empirical evaluation on the MMLU benchmark

Decision: Reject