Let the LLM Stick to Its Strengths: Learning to Route Economical LLM
Authors: Yi-Kai Zhang
Shiyin Lu
Qing-Guo Chen
Weihua Luo
De-Chuan Zhan
Han-Jia Ye
Venue: NeurIPS 2025 poster ()
Keywords: LLM Routing
Deep Learning
Primary area: deep_learning

Abstract: Recently, test-time scaling of Large Language Models (LLMs) has emerged as a practical alternative to parameter and data scaling. Reasoning tasks often require large-scale, RLVR-based LLMs, while more economical LLMs can handle simpler tasks. Routing an LLM tailored to *suitability* (*i.e.*, capability and cost) ensures usability and efficiency. We introduce LLMRec, which routes the most suitable LLM to the user query without pre-inference on the candidate LLM zoo. It pioneeringly reframes the LLM routing problem as a comprehensive recommendation system (RecSys) task. Our core insight is that an LLM's suitability for a query is a complex, latent signal equal to user-item preference. LLMRec systematically engineers features for candidate LLMs (intrinsic attributes and capability distributions), queries (general semantics and meta-dimensional info), and context (inference type, cost budgets). It also incorporates behavioral features to learn high-order interactions. LLMRec is designed to generalize to out-of-domain datasets and adapt to new LLMs as the model zoo evolves. We define the metric with the Pareto frontier under user-specified cost budgets. Across six datasets, LLMRec achieves an average cost reduction of over 38% while maintaining accuracy and consistently outperforming baselines in converging toward the Pareto frontier.

Let the LLM Stick to Its Strengths: Learning to Route Economical LLM
Yi-Kai Zhangl?
Shiyin Lu Qing-Guo Chen
Weihua Luo
De-Chuan Zhan Han-Jia Ye
School of Artificial Intelligence, Nanjing University
National Key Laboratory for Novel Software Technology, Nanjing University
3 AI Business, Alibaba Group
Abstract
Recently, test-time scaling of Large Language Models (LLMs) has emerged as practical alternative to parameter and data scaling. Reasoning tasks often require large-scale, RIVR-based LLMs, while more economical LLMs can handle simpler tasks. Routing an LLM tailored to suitability (i.e., capability and cost) ensures usability and efficiency. We introduce LLMRec, which routes the most suitable LLM to the user query without pre-inference on the candidate LLM zoo. It pio- neeringly reframes the LLM routing problem comprehensive recommendation system (RecSys) task. Our core insight that an LLM's s suitability for a query a complex, latent signal equal user-item preference. LLMRec systematically engi- neers features for candidate LLMs (intrinsic attributes and capability distributions), queries (general semantics and meta-dimensional info), and context (inference type, cost budgets). also incorporates behavioral features to learn high-order interactions. LLMRec is designed to generalize out-of-domain datasets and adapt to new LLMs as the model zoo evolves. We define the metric with the Pareto frontier under user-specified cost budgets. Across six datasets, LLMRec achieves an average cost reduction of over 38% while maintaining accuracy and consistently outperforming baselines in converging toward the Pareto frontier.
Introduction
The rapid growth of large language models (LLMs) [27 38 has produced diverse ecosystem
of models varying in scale, functionality, and performance. The parameter scaling gave rise to models ranging from 1B-parameter edge-device variants to 100B+ omni-models 488 57]. The subsequent test-time scaling introduced advanced reasoning LLMs like DeepSeek-RI RI (15), which leverage reinforcement learning and extended thinking processes for superior performance.
In practice, companies often provide L.MMs-a-APP-service the monthly expenses for these services can reach millions dollars D. [53] significant challenge arises from this diversity: the capabilities of large reasoning-heavy LLMs often far exceed the requirements of some downstream tasks, leading to unnecessary costs. Smaller-scale LLMs are typically sufficient for handling simple tasks, but more complex tasks, such code generation [10] or mathematical reasoning [12, often require techniques like chain-of-thought (CoT), or larger, reasoning LLMs with extended inference tokens 228 13 54 An intuitive idea emerges:
19 For medium-sized business clients,
intelligently route user queries to the most suitable LLM, balancing performance and cost?
A good model routing outcome hinges on cost considerations, primarily driven by two factors 3 the cost per token and the number of tokens generated. The former reflects the expense of producing
*Coreesponingg author, email: yehj @ lamda.nju.edu.cn.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

single token, typically scaling with the size of the candidate LLM. The latter is closely tied to the inference mode, or whether the LLM is RLVR-based. The total cost can be defined as the product of these two. A general observation is that larger L.Ms or long-chain inference increase costs but improve task completion and accuracy. From that, our model routing goal to minimize costs without sacrificing accuracy, approaching the Pareto optimum in the trade-off between cost and accuracy.
In this paper, we propose LLMRec, which pioneeringly reframes the model routing problem as a comprehensive recommendation system (RecSys) task. We find the underlying logic of these two problems is equivalent: (1) RecSys learns the complex match between a user and an item to predict a personalized Preference. (2) Model Routing learns the optimal match between a query and an LLM evaluate its comprehensive Suitability (i.e., capability and cost). Importantly, both tasks share an assumption: RecSys is effective because user's preference is a complex, latent signal. Similarly, an LLM's suitability for a query a profound, implicit match determined by its specific capabilities, domain expertise, and cost, which can be learned from massive historical performance logs.
Specifically, we introduce the feature engineering paradigm from RecSys into model routing. In LLMRec, queries and LLMs correspond to users and items. We systematically construct features for: (1) Model and Query: An LLM representation captures intrinsic attributes (like architecture, scale, origin, training details, etc.) and capability distributions (as performance on benchmarks at release and on a core set we constructed). For query representation, we consider general semantic embeddings and meta- dimenniimniio information (including high-level evaluations like answer difficulty, reasoning level, and domain category). (2) Context: As RecSys dynamically adjusts recommendations based on different times or locations, we use the user-specified infer type (e.g., self-consistency. Co',, and Tree-of- Thoughtt and cost threshold as contextual features, enabling adaptation to different service- level objectives.
(3) Behavioral sequences: Furthermore, we build dynamic interaction features based on the performance of candidate LLMS on i) a core set of tasks and ii) on top-k nearest training neighbors of the current query. Like modern RecSys, the core of the LLMRec framework learns the complex, high-order cross-information among model, query, behavioral, and contextual features. This drives it to accurately predict a suitability score for each candidate LLM, recommending the one that best approaches the Pareto optimum for the given cost constraint.
LLMRec is designed to be robust, generalizing to out-of-domail datasets, and adapting to new LLMs as the model zoo evolves. While maintaining accuracy, L.MReee reduces costs by an average of over 38% across six datasets. It consistently outperforms all baselines in converging toward the Pareto frontier under various cost budgets. Our main contributions are as follows:
We are the first to systematically apply recommendation systems to model routing, achieving economical, efficient, and iterative optimization.
We implemented model, query, behavioral, and contextual features, allowing users to specify the inference type and cost threshold to constrain the routing.
We develop a cost- ceat-budeet-bassee Pareto metric tailored for LLM routing applications.
We construct a large-scale training set and routing benchmark, including evaluations on unseen datasets and dynamic model zoos.
LLMRec is versatile, efficient, and adaptable to the evolving LLM landscape, offering scalable solution for real-world LLM API deployment.
2
Preliminary
We start by presenting the model, query, and pipeline of the LLM router, and discuss its connection to RecSys. We then introduce the deployment and related work of model routing.
2.1 Notations
Key Elements in LLM Routing Consider a scenario where, when using an LLM API, the system provides a candidate LLM zoo M (f), f .?..... M In this case, the user provides a task consisting of an instruction set Duett = {(x,, a; 1 where the LLM fr produces the output as o; = f (x;; I;) on input x; using inference mode I E I, and the correct answer a, The accuracy is given by Ace (os, a.). In this paper, we focus on the case that fr represents a decoder-only text generation L.M.. As established in the /section the core challenge of L.IM routing to predict the suitability of an LLM fr e M for given query (x; without first generating the actual output o,

---

Performance
Qwen2
Performance
Qwen2
GLM-4 96
Meta- Daata- 708
Phi-
Phi-3 small
Meta-Ulama-
Meta Llama-3 3 BB
GLM-4 9B
Easy Task
LLM Scale
Hard Task
LLM Scale
Small-scale LLMs can most simple tasks.
Comparison of the performance laws of LLMs with different scales, architectures, and tasks, For most LLM architectures, smaller-scale LLMs can accomplish most simple tasks efficiently; routing to smaller-scale LLM can reduce deployment costs.
Figure 1:
from each candidate model. This suitability conceptually consistent with the foundation of RecSys, which operates by predicting the complex user preference.
Brief Background on Recommendation System The core task of RecSys is to learn the complex matching relationships between and items. A typical process begins by collecting user and item features, their historical interaction logs, and contextual information. The model then learns a user's preference for different items under various scenarios. The ranking task in RecSys estimates the probability of user clicking on each item.
Cost-effective Routing Pipeline Our task is to select an LLM f for each instruction x, and the inference mode to minimize cost while maintaining accuracy. We define the per-token cost of an LLM as t", and the goal is:
n f" arg max E, [Ace (f" (x;;I)), a:).. s.t. Σ tT 10: c M'EMMLE
(1)
where o, = 5 (x;; The cost of the input sequence is omitted (as it is fixed for all methods). represents the user-specified cost threshold. In our formulation, exceeding the cost threshold incurs some penalty, but it is less critical than losing accuracy. The upper bound of LLM routing performance depends on the number of instructions for which no available LLM can produce correct answer. In practice, directly optimizing Equation aas hard-constraint problem is intractable. The budget is global constraint summed over the entire dataset, whereas the router must make local decision for each query x, Furthermore, router trained to satisfy a fixed is inflexible; it cannot adapt different user-specified budget without being retrained. To overcome this, we reframe the problem by treating the cost threshold as dynamic input feature. This converts the hard constraint into a learnable condition. The router learns a policy that associates different budget levels with corresponding routing strategies, enabling it to dynamically balance cost and accuracy based on the user's s needs. We formally define the router, frouter, as policy that selects a model fr M given an input x,, an inference mode I,. and the cost threshold
Definition of the Pareto Front based on Cost Intervals In practical applications, service providers tend prioritize accuracy as key performance indicator over cost. This preference stems from their desire for a predictable performance outcome rather than an uncertain level of accuracy after incurring expenses. Therefore, when LLM API service providers implement LLM routing, they often address the dual problem of the optimization presented in the above Equation. This dual problem aims to minimize costs subject to constraint on target accuracy. At this point, we introduce the concept of Pareto dominance. For any two routing solutions, say solution frouter and solution f frouter characterized by their respective metrics (e.g., accuracy and cost), we state that solution a Pareto dominates solution b (denoted as frouter > frouser) if and only if solution a is strictly better than solution in least one objective and not worse in the other objectives. If one solution has lower accuracy but also lower cost than another, neither solution dominates. They may belong to the same
3

---

Response
Resporse
Reqooue
...
MoE
BB
M
!
Ensemble
Router
Respanse
Re-ranking
A T D
A A
A M
Attention
Owen
O
O
Router
A ^
(a) Methods Inference Candidate LLMs
(b) Mixture-of--xperts
(c) L.IM Routing (Ours)
Figure 2: Comparison of Different Methods for Utilizing pre-trained LLM Libraries: Ensemble and Re-ranking methods require inference from all candidate LLMs. The Ensemble approach combines the outputs of multiple LLMs, while the Re-ranking selects the most suitable response from all generated outputs. Mixture-of-Experts (MoE) integrates routing within the Transformer layer structure. LLM Routing in our setup directs instruction to one of the candidate LLMs without requiring inference on the target instruction for all LLMs.
Pareto front. both solutions fall within the user's acceptable range for both cost and accuracy, they are both considered comparatively optimal.
2.2 Related Work
Leveraging the formalized setting, we revisit key features from thelsection
No pre-inference model interactions. Some model ensemble cascade strategies aim to se- lect and synthesize the optimal response from all candidate models, considering input-response relationships. Some scoring strategies have been applied in reinforcement learning-based LLM training. Others include classical machine learning techniques 7 40 20 21 and multi-perspective deep ensemble methods 35 22
However, these methods can introduce significant delays in LLM APIs, especially with large-scale candidates. In LLMRee, inferring each model for every target
instruction is not feasible.
Generalization in new scenarios. LLMRec is designed for LLM APIs with focus on router
scalability, While some transfer learning approaches [5] 29 59 39 use proxy torgee-target metries based on label distribution matching, they are limited by the target set seen during training [9]. On the other hand, LLM APIs have flexible natural language output space, meaning the router should generalize shot to unseen user instructions.
Extension on updated candidate models. Some application specific router frameworks
have addressed the issues mentioned above. However, most approaches fix the candidate model zoo order to stabilize routing training and deployment scenario [36 16.
17
46 37 Additionally,
some mixture-of-expe embedding a router to select among them to reduce inference costs. However, these methods tightly couple router and model parameters, when the expert zoo is updated, the router needs to adapt through complex incremental learning 4T which may introduce issues like hyperparameter sensitivity and catastrophic forgetting.
(MoE) 45 18 31
use MIPs within transformer blocks as experts,
In summary, while existing approaches have made significant strides in specific aspects of model routing, LLMRec distinguishes itself by offering a solution that holistically addresses the challenges of pre-inference overhead, zero-shot generalization new instructions, seamless extension to updated model libraries, and inherent cost-efficiency. Its novel use of a recommendation system framework, as depieted |Figure based on learnable representations of models and queries, allows for dynamic and efficient routing in large-scale LLM API environments. LLMRee introduces scalable approach

---

by creating a universal, learnable model representation, turning LLM routing into learning both model and query embeddings. The model representation encodes capabilities and behaviors and is optimized with a dynamic embedding vocabulary. This universality allows new LLMs to quickly index into the embeddings after lightweight inference, unlike the random initialization and re-training needed in Model Spider [63]. Finally, the routing process estimates the relationship between the model and query representations.
3 Learning to Route Test-Time Economical LLM
In this section, we outline LLMRec framework, discuss the construction of the recommendation representations, implementation details, and cost constraints in deployment.
3.1 Representation of Model & Query
The model is a black box to the router, and decisions must be made without inferring on all candidate models to minimize overhead. Directly extracting features from an LLM's high- dimensional parameters is infeasible. To address this, we construct a model representation that incorporates both intrinsic properties and capability distributions. This allows the router to learn how a model's potential impacts its generalization to new instructions. To achieve this, we also build comprehensive coreser of diverse evaluation data. The advantage that when a new model arrives. we can assess its core capabilities on this coreset with minimal overhead.
Motivation:
Model Representation For the candidate LLM fT we categorize its representation into intrinsic properties and capability distributions, with each dimension optional.
1. Intrinsic properties include model structure (e.g- publisher name, architecture, number of layers, layer types, total parameters, training details, precision, and feature descriptions). Additional information may include HuggingFace download count, open-source licenses, etc.
2. Capability distributions are divided into evaluated offline benchmark and online coreset.
(a) Given that most LLMs publish standard benchmark performances on release, we document model performance on benchmarks such as MMLU (24). MMLU-Pro [51]. BBH [47]. ARC- Challenge (6]. TruthfulQA (32], Winogrande [43], and HellaSwag [60]. For reasoning capa-
bilities, we consider domains like mathematics (MATH (25), MMLU-STEM, GSM8K (22)
and code generation (HumanEval [T], HumanEval+ 34 MBPP 4. MBPP+ (4). For offline capabilities, the router will not explicitly know which benchmark the current user query belongs to. (b) To expand the assessment of model capabilities, we also create online evaluation coreser: it acts as a bridge linking the model's historical behavior with its future expected performance. By sampling 20-shot examples from each of the 71 categories in MMLU and MMLU-Pro, we form a core set 1,415 instructions for online evaluation. In parallel, we select from specialized domains like mathematics, code generation, healthcare, law, and finance. We extract 5 keywords for each category as semantic descriptors and compute class-center embeddings. Detailed information is provided in the section 4
focus the average performance across these datasets, and defimitely,
Query Representation We divide the representation for user instruction x, into general semantic representation and meta-dimensional information.
General semantic embeddings: For a user instruction x,, we use 3 general encoders ap (e.g., GTEaage (2) 0.33B, 5.ren..5-0.5B--nnttrue (57). and RoBERTa-Large) to extract embeddings. 2. Meta-dimennionall information, e.g. answer difficulty, reasoning level, content diversity, tem- poral stability, conceptual ambiguity, and domain expertise, is extracted via specific prompts by encoder 1-7en2..57B-Inttruue [57] through few inference steps.
Context Representation We also construct contextual features to align routing decisions with service-level objectives. These include the user-specified inference type (e.g., self-consistency, CoT, Tree-of- Thouhht (ToT)) and the cost threshold, which we discretize into five levels.
The features described above are all attributes of either the query or the model side. In RecSys, these are known as first-order features. We note that any of these feature dimensions can be optional (in implementation, they are null null-padded). Modern RecSys, however, automatically constructs and learns
5

---

O
LLM API
User
EcoRouter w/ RecSys Kernel
Figure 3: The flowchart of LLMRec. The representation of model, query, context, and behavioral sequences is constructed. Then, the feature relationships are learned by the LLMRec with the RecSys kernel to route to the corresponding LLM.
from numerous high-order cross-fcatures to capture the complex, non-linear matching relationship between the two sides. We introduce some examples of explicit cross-features, such as those centered on the historical behavioral sequences of models relevant to the current query. Specifically, we consider the performance of candidate LLMs on i) our core set of tasks and ii) on the top-k nearest training neighbors of the current query. These are introduced as dynamic capability features that are more representative of and relevant to the query. While an inherent gap may exist between a model's static capabilities and query's semantics, the L.MReec framework, like modern RecSys, is designed learn from the complex, high-order relationships among all available model, query, contextual, and behavioral sequence features.
3.2 Routing as Recommendation
We formalize the routing problem ranking in RecSys. In this paradigm, the input query (x,) and candidate LLM (fr)) are mapped to the user and item. The user-specified context, such as the cost threshold and inference type, provides the situational context for the recommendation. The core objective is to learn a function that predicts the suitability for any given (query, model, context) tuple. This suitability is complex, non-linear match using high-order frattrfe,,us,s, corresponding to the core challenge in modern RecSys. LLMRec is designed to leverage principles from established architectures to learn these interactions.
For instance, methods like Wide & Deep excel by combining two components: "wide" part that memorizes explicit, low-order feature interactions (e.g.. manually crafted crosses) and "deep" part that uses MLPs to generalize and learn implicit, high-order relationships from dense embeddings. Manually designing cross- crosurfeatures is difficult. Models like DeepFM 24] automate this by integrating Factorization Machines (FM) with an MLP. The FM component efficiently models second-order feature interactions by learning low-dimensional latent vector for every feature and taking their dot products. These latent vectors are shared with the deep component, allowing end-to-end learning. Enhancements like AFM [55] introduce attention mechanism to weigh the importance of different feature interactions. Other models learn implicit interactions more directly: FGCNN convolutions the feature map capture local interaction patterns. FiGNN 30 employs graph neural networks (GNNs), treating features as nodes to capture complex, high-order relationships via aggregation.
33 applies
Furthermore, a model's suitability is not static; is highly dependent on the query. We draw inspiration from behavioral modeling in RecSys, such as the Deep Interest Network (DIN) [65). which uses an attention mechanism to dynamically weigh a user's historical behaviors based on their relevance target item. Analogously, LLMRee treats model's historical performance, such as our coreset or on the top-k nearest training neighbors of the current query, as a dynamic behavioral sequence. This allows the framework to learn a query-aware representation of model's capabilities.
By integrating first-order features (of query, model, context) and learning their complex, high-order interactions, the RecSys framework learns to accurately rank all candidate LLMs. This ranking

---

datasets (such as allows it to recommend the model that best approaches the Pareto optimum for the given query and service-level constraints.
3.3 Training Data Construction
training instance is (q, m, c, b) + 8, where the representations for the query (q), model (m), context (c), and behavioral features (b) map to target suitability score To learn this function, we construct a massive dataset of over billion model-query pairs. This dataset spans more than 50 LLM families (from 2024) evaluated on over 30 diverse benchmarks, We capture performance logs using various inference modes, including direct generation, self-consistency (50), Chain-of-Thougge (Col') 52 and Tree-of-Thoughe (ToT) [58
For each pair (x, o(r) (instruction x,, model m's output ","), record the key labels as:
1. Accuracy (Acc): Evaluated using dataset-specific metrics against the ground-truth answer a;-
2. Cost: Calculated as Cost(o.") = Cfm, x |o!" where cs,., is the estimated per-token inference cost (based on model scale) and oo is the length of response tokens.
To generate the ground-trudh truth suitability ranking s, we follow a Pareto- based principle that prioritizes cost-effectiveness. For a given query, we define the preference order as:
Ranking = Sort. (Co)) a.)20) @ Shuffle ([fm) a0)=0) Cost(o!")
where @ denotes ordered concatenation. All models that correctly solve the task (Acc > 0) are ranked first, sorted by their Cost(-) in ascending order (lower cost is better). All models that fail (Acc = 0) are shuffled and ranked last. Crucially, this ground-truth ranking is computed relative to the input contextual features c, especially the user-specified cost threshold. This teaches the L.MReec to adapt its routings, learning which LLM is "best" under different budget constraints.
4 Experiments
4.1 Implementation Details
Candidate LLMs Zoo of Training As mentioned, we consider 52 different LLMs. Among them, 32 models are under 10B parameters, 15 models between 10B and 20B parameters, and models are around 70B parameters. In practice, we test more than 80 models, but exclude early models that did have CoT capabilities.
Domains of Training Datasets We consider general evaluation datasets, commonsense reasoning, math reasoning, code generation, symbolic reasoning, and specific domain datasets such as medical, law, and financial datasets, totaling 35 datasets.
Interaction Scaling We perform inference with each candidate LLM on the target datasets to generate nearly 10m interaction pairs. We sample approximately 110k these for training, with eroonger-perfommigg pairs being assigned higher sampling weights. Out of the 35 datasets, 24 multiple-choice datasets, and 11 are fill-n-thee-lann or question-ame answer datasets, including token generation via the generate method. For these datasets, incorporate self- celf-consistency, COT, and Tree-of-Thouge (ToT) reasoning modes. We also include datasets where the performance did not improve even declined after applying complex reasoning modes.
Evaluation Metrics We evaluate the 24 multiple-choice datasets using perplexity (PPL). Most additional 11 fill-in-tee-blank datasets are evaluated using regular expressions to extract the final answers. We follow the corresponding evaluation libraries for some math extraction processes or code generation-yye tetrrttionttppeeee pass@k).
Table we have mixed 5 small-scale LLMs with fewer than 10B parameters, 2 LLLMs between 10B and 20B parameters, and 3 large-scale LLMSs with around 70B parameters ensure that the L.LM library contains varying capabilities, from small to large.
Candidate LLMs Zoo of Evaluation
shown

---

General
Comm. Reasoning Mean MMLU-stem
Method
#Params
MMLU
TruthfulQAAR---
Shohot
O-shol
25-thot
5.sho
Small-scale LLMS (<10B)
InternL.2.55
7.7B
69.88
54.36
60.75
65.31
62.63
Meta-Llama-3 Instruct
49
8.0B
65.59
51.63
62.12
58.32
59.42
Qwen2 Instrect [6
7.6B
69.13
55.49
61.43
63.45
62.38
GLM-4 [61
9.4B
69.28
59.32
66.13
64.45
64.80
Phi-3 Small 128K
2
7.4B
75,90
64.62
71.08
69.09
70.17
Best-Performing of Small-scale L.LMS
75.90
64.62
71.08
69.09
70.17
Large-scale LLMs (~70B)
Meta-Llama-2 Instruct
49
70B
79.89
61.83
71.67
73.92
71.83
Qwen2 Instruet [56)
72B
83.79
54.85
68.62
79.85
71.78
Mixtral-8x22B I0.truct-0.11
20
140B
77.63
68.19
72.78
71.64
72.56
Best-Performing of Large-scale LLMs
83.79
68.19
72.78
79.85
76.15
LLM Routing
Random Selection
32B
72.98
58.87
67.83
67.96
66.91
GTE Large
55B
74.72
61.08
69.62
69.12
68.64
LR
42
~ 33B
73.25
59.61
66.04
68.12
66.76
Deep & Wide 11
27B
82.32
67.44
71.67
77.37
74.70
Ours w/ DeepFM 23
26B
82.29
67.81
71.93
76.74
74.69
AFM 555
25B
79.84
63.53
70.39
77.50
72.82
DIN 65
~ 31B
83.92
66.83
72.95
78.36
75.52
Table Comprehensive Routing Evaluation on General, Commonsense, and Reasoning Tasks. We compare the response accuracy across various benchmarks (MMLU, TruthfulQA, ARC-C, and MMLU-stem). We categorize methods by model scale (Small-scale LLMs, Large-scale LLMs, and LLM Routing methods). We show the number of parameters ("#Params"), Bold represents the best performance, and underlined the second-best.
Evaluation Benchmarks Similarly, we have construct evaluation dataset that includes general evaluation benchmarks like MMLU (24). TruthfulQA [32], and commonsense reasoning tasks such as ARC-CChllennge 66) and MMLU-stem [24]. as well as mathematical reasoning benchmarks like GSM8K [12), and symbolic reasoning tasks like BBH [47]. Different models exhibit performance variations on these datasets. We reference the scale size and FLOPs to correspond with the cost, and by multiplying the response token count, calculate the total cost per unit each dataset.
Model Representation Construction. As outlined in section 3 using the descriptions and coreset. Each dimension is segmented into multiple values, with continuous values being bucketed. These segmented values are then mapped to randomly initialized embeddings. which are incorporated into the training process.
we construct model representations
Baseline methods. For the baseline, we consider randomly selecting from the LLM library and using all responses oi from the inference downstream datasets. We employ GTE to match o, with the instruction x,, and the LLM with the highest score selected.
Average scale calculation ("#Params" the Tables). We average the scales of all selected LLMs based the instruction dimensions of all evaluation data in both tables, which results in the LLM scale presented in "#Params".
the models underlined in our L.LM library have not appeared in the routing training set (one large-scale and one small-scale LLM).. All datasets, except for MMLU, considered unseen datasets.
Generalization to unseen models and datasets. As shown in Table

---

Math Reasoning Symbolic Reasoning
Mean
Method
#Params
GSM8K
BBH 3-shot, CaT
shol. CoT
Perf.
Leng.
Perf.
Leng.
Perf.
Leng.
Small-scale LLMs (<10B)
InternLM2.5
7.7B
74.37
371
68.13
452
65.50
412
Meta- Meta-Llama- Instruct
8.0B
56.18
273
60.93
399
59.13
336
Qwen2 Instruct
7.6B
78.92
368
62.92
526
65.22
447
GLM-4
9.4B
79.53
505
74.43
487
68.86
496
Phi-3 Small 128K
7.4B
82.34
449
73.94
521
72.83
485
Best-Performing
82.34
449
74.43
487
72.91
468
Large-scale LLMS (~70B)
Meta-L.amm-3 Instruct
70B
83.17
580
81.48
635
75.33
608
Qwen2 Instruct
72B
88.86
535
82.89
593
76.48
564
Mixtral-8x22h Instruct-vO.1
140B
84.31
553
79.54
610
75.68
582
Pest-Perfforing
88.86
535
82.89
593
79.39
564
LLM Routing
Random Selection
32B
77.71
489
73.58
534
69,82
512
GTE
55B
80.52
528
74.37
580
71.57
554
LR Deep & Wide
33B
83.40
457
73.84
532
70.71
495
27B
86.05
426
78.27
489
77.19
458
Ours w/
DeepFM
26B
87.87
411
78.02
495
77.44
453
AFM
25B
86.05
404
79.40
476
76.12
440
DIN
31B
87.19
446
79.94
500
78.20
473
Table 2: Comprehensive Routing Evaluation on Math and Symbolic Reasoning Tasks. We compare performance on math and symbolic reasoning tasks, showing response accuracy ("Perf.") and average token usage ("Leng."). Similar to Table we estimate the approximate model scale associated with each routing method. The total computational approximately proportional to the product of the model scale ("#Params") and the average token usage ("Leng."). Bold is the best, and underlined is the second-best.
of a method the data is
Results Analysis
illustrates the performance of LLMRee using different recommendation system kernels on General and Commonsense Reasoning tasks. We test routers at two scales, with small-scale LLMs showing some variation in performance. Among the small-scale LLMs, Phi-3 Small- 128K achieved the best results, scoring the highest across all tests with average of 70.17. Other small-scale LLMs, such as I2ternL.M2.5, Meta-Llama-3 Instruct, Qwen2 Instruct, and GLM-4, also deliver decent performance.
Table
In the large-scale LLM category, Qwen2 Instruct (72B) outperforms the others, notably earning high score of 79.85 on MMLU-stem. Meta-Llama-3 Instruct (70B) follows closely, showing strong performance across most tests. Mixtral-8x22B Instruct-vO.1 excels in specific metrics like TruthfulQA, but its overall average score was slightly lower than the other methods. I.LMRec manages the routing of instructions for all the above LLMs. Comparison methods were also used, such random selection and embedding matching using GTELage (with estimated LLM scales of 32B and 55B). Most of LLMRec's routes were the 20B+ range. When routing LLMs with the recommendation system's s DIN architecture, the best results are achieved when historical model-data interactions are included in the sequence, reaching an average accuracy of 75.52. Wide & Deep and DeepFM also achieve similarly high performance with relatively low overhead. Notably, on MMLU and ARC-Challenge, the LLM's performance surpasses the rop-performing models in the datasets.
In Table 2
we present the routing capability on deep reasoning tasks, including GSM8K and BBH datasets for math and symbolic reasoning. The candidate I.IM library and comparison methods are the same in Table I In this evaluation, the LLMs performed inference through generation-based

---

approach and assembled COT. The "Mean" in the Table represents the average across all data, including the results from Table I After inference, we calculate the average token length for each method. The overall cost for each method is roughly determined by multiplying the model scale by the token length. L.MMeec achieves superior performance while reducing the average output length by approximately 30% compared to others, resulting in the optimal average performance.
Limitations
LLMRec has not considered market factors such as fluctuations in LLM API pricing: instead, use the LLM parameter scale as approximation. The generalization capability of model representations under modified pricing schemes requires further investigation. The system's evaluation of new LLMss depends on a sufficient volume of historical interaction data. This creates a cold-start challenge, common issue for all routing systems the rapidly evolving LLM ecosystem. This limitation may be addressed by continuously updating the coreset to enhance the validity of the model representations.
5 Conclusion
In this paper, we introduced LLMRec, a novel framework that optimizes the cost-performance trade- off for LLM API services by pioneeringly reframing model routing as recommendation system (RecSys) task. Our core insight is that an LLM's suitability for a query, balancing capability and cost, is a complex, latent signal equal to user-item preference. LLMRec learns this signal by modeling rich, multidimensional features for models, queries, operational contexts (including user-specified cost thresholds), and behavioral sequences. Our extensive experiments demonstrate that LLMRec reduces inference costs by over 38% on average while preserving task accuracy. The framework shows robust zero-shot generalization to unseen tasks and scamless adaptation to dynamically evolving model zoo, validating intelligent routing as a viable solution for scalable and economical LLM deployment.
Acknowledgments and Disclosure of Funding
National Key R&D Program of China (2024YF00202800), NSFC(62376118), Collaborative Innova- tion Center of Novel Software Technology and Industrialization.
References
[1] Cost estimation
using GPT-3 real applications. https:// hww.blllr..eeeebogg////ooo
mon-muuc-does- 1esoos-it-ce cossssttttto--t-o moss--t-ure -s---e--eeeeeee spt-models- \gpt- --prrcing-explainee Accessed: 2024-
12-26.
[2] Marah Abdin, Ade Jacobs, Ammar Ahmad Awan, el Phi-3 technical report: A highly capable language model locally on your phone. CoRR, 011224444.12119 2024.
[3] Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay. Manaal Faruqui. and Mausam. Automix: Automatically mixing language models. In NeurlPS, 2024.
[4] Jacob Austin, Augustus Odena, Maxwell I. Nye. Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carric Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR,
07732, 2021.
[5] Yajie Bao. Yang Li, Shao- Huang. Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-theoreiicc approach to transferability in task transfer learning. In ICIP, 2019.
[6] Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, et al. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03315,. 2021.
[7] Leo Breiman. Stacked regressions. Machine learning. 24:49 64, 1996.
Zheng Cai. Maosong Cao. Haojiong Chen, al. Internlm2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
[9] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. CoRR, 0oo.//335.05166. 2023.
10

---

[10] Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. CoRR, 01/22007.03774, 2021.
[11] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. Wide & deep learning for recommender systems. Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS@ RecSys 2016, Boston, MA, USA, September 15, 2016, pages 7-10. ACM, 2016.
[12] Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert. Jemry Tworek, Jacob Hilton, Reiichiro Nakano. Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, 0bs/2110.14166 2021.
[13] Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Yan, Hao Li, Bin Ling, Yonghong Tian, and Li Yuan. Chatlaw: A multi-agent agent collaborative legal assistant with knowledge graph enhanced mixture-of- experts large language model. CoRR, abs/2306.1604 2024.
[14] Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John C. S. Lui. Cost-effective online multi-llm selection with versatile reward models. CoRR, abs/2405. 165877 2024.
[15] De,,Seck-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qi- hao Zhu, Shirong Ma, Peiyi Wang, et al. Deepscek-rl; Incentivizing reasoning capahility in llms via reinforcement learning. 2025.
[16] Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks V. S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: cost-efficient and quality- aware query routing. In ICLR. OpenReview.net, 2024.
[17] Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: graph-based router for LLM selections. CoRR, abs/2410.03834, 2024.
[18] Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: An efficient multitask tuning for large language models. CoRR, abs/2403.03443 2024.
[19] Roy Thomas Fielding and Richard N. Taylor. Architectural styles and the design of network-based software architectures. PhD thesis, 2000.
[20] Yoav Freund and Robert E Schapire. A decision-theoreticc generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1):119-13, 1997. ISSN 0022-0000. doi: https::/doi.org// 100/ 1r010...//.....
[21] Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189 118--1222,22 2001.
[22] Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N. Suganthan. Ensemble deep learning: review. Eng. Appl. Artif. Intell., 115:105151,, 2022.
[23] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: A factorization- machine based neural network for CTR prediction. In Carles Sierra, editor, IJCAI, pages 1725-1731. ijcai.org, 2017.
[24] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
[25] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In NeurlPS, 2021.
[26] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04000 04088, 2024.
[27] Alhert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. CoRR, 0002240004088. 2024.
[28] Chunyuan Li, Cliff Wong, Sheng Zhang. Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and and-vision assistant for biomedicine in one day, In NewrIPS, 2023.

---

[29] Yandong Li. Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, and Boging Gong. Ranking neural checkpoints. In CVPR, 2021.
[30] Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang. and Liang Wang. Fi-gnn: Modeling feature interactions via graph neural networks for CTR prediction. CoRR, abs/1910.05552, 2019. URL.nttp:/aaxxiv.rrg. orgpb////..rr///////////b/bbb/s 1910. 85552
[31] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 2024.
[32] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In ACL, pages 2252- Association for Computational Linguisties, 2022.
[33] Bin Liu, Ruiming Tang, Yingzhi Chen, Jinkai Yu, Huifeng Guo, and Yuzhou Zhang. Feature generation by convolutional neural network for click-through rate prediction. In Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza-Yates, and Leila Zia, editors, The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 1119-1129, ACM, 2019.
[34] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In NewrlPS, 2023.
[35] Yong Liu and Xin Yao. Ensemble learning via negative correlation. Neural networks, 12910441399-1444 1999.
[36] Aman Madaan and Yiming Yang. FLOWGEN: fast and slow graph generation. CoRR, abs/2207.07655 2022.
[37] Isaac Ong. Amjad Almahairi, Vincent Wu, Wei-Lin Chiang. Tianhao Wu, Joseph E. Gonzalez. M. Waleed Kadous, and Ion Stoica. Routellm: Learning to route ilms with preference data. CoRR, 2024.
[38] OpenAL. GPT-4 technical report. CoRR, abs/2303.08722 2023.
[39] Michal Pándy, Andrea Agostinelli, Jasper R. R. Uijlings, Vittorio Ferrari, and Thomas Mensink. Transfer- ability estimation using bhattacharyya class separability. In CVPR, 2022.
[40] Ross Quinlan. Bagging, boosting, and C4.5. In Proceedings of the Thirteenth National Conference on Artificial Intelligence and Eighth Innovative Applications of Artificial Intelligence Conference, AAAI 96. IAAl 96, Portland, Oregon, USA, August 4-8, 1996, Volume 1, pages 725-730. AAAI Press The MIT Press, 1996.
[41] Sylvestre-Alvis Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In CVPR, 2017.
[42] Matthew Richardson, Ewa Dominowska, and Robert Ragno. Predicting clicks: estimating the click- through rate for new ads. WWW, pages 521-530. ACM, 2007.
[43] Keisuke Sakaguchi. Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, pages 8732-8740, 2020.
[44] Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-modess meta-modeling. In ACM WSDM, pages 606 615. ACM, 2024.
[45] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Lee Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of-experr layer. 2017.
[46] KV Aditya Srivatsa, Kaushal Kumar Maurya, and Ekaterina Kochmar, Hamesssing the power of multiple minds: Lessons learned from LLM routing. CoRR, abs/2405.00040 2024.
[47] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. In ACL, pages 13003-13051. Association for Computational Linguistics, 2023.
[48] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288 2023.
12

---

[49] Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[50] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe L. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR. OpenReview.ner, 2023.
[51] Yubo Wang. Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi task language understanding benchmark. In NewrlPS, 2024.
[52] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. thaiigh prompting clicits reasoning in large language models. In NeurlPS, 2022.
[53] Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun. Newsha Ardalani, Kiwan Maeng. Gloria Chang, Fiona Aga Behram, Jinshi Huang, Charles Bai, et al. Sustainable AI: environmental implications, challenges and opportunities. In Diana Marculescu, Yuejie Chi, and Carole-Jcan Wu, editors, MLSys. mlsys.org, 2022.
[54] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski. Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomherggpt: A large language model for finance. CoRR, abs/2303. 17564, 2023.
[55] Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. Attentional factorization machines: Learning the weight of feature interactions via attention networks. In IJCI, pages 31 19-3122. ijcai.org, 2017.
[56] An Yang, Baosong Yang, Binyuan Hui, et al. Qwen2 technical report, 2024. URL httphttt//::/xi...oom abs/2407. 10671
[57] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. CoRR, Ahs/24122.5115 15, 2024.
[58] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NewrIPS, 2023.
[59] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of pre-trained models for transfer learning. In ICMI, 2021.
[60] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentenec? In ACL, pages
2019.
[61] Aohan Zeng, Bin Xu, Bowen Wang, et al. Chatglm: A family of large language models from GLM-130B to GLM-4 all tools. CoRR, hbss2406..1293,, 2024.
[62] Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. arXiv preprint arxiv:2407.11666 2024.
[63] Yi-Kai Zhang, Ting-Ji Huang, Mho-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model spider: Learning to rank pre-trained models efficiently. In NeurlPS, 2023.
[64] Wei Zhou, Qi-Wei Wang. Zhi-Hong Qi. Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Class- insseinncreettal lcarning: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 46( 12):9851-9877, 2024.
[65] Guorui Zhou, Xiaoqiang Zhu, Chengru Song, Ying Fan, Han Zhu, Xiao Ma. Yanghui Yan, Junqi Jin, Han Li. and Kun Gai. Deep interest network for thicu-th rate prediction. In ACM SIGKDD, pages 1059-1068, ACM, 2018.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly summarize the paper's core contributions and define the scope of the research presented, accurately reflecting the content discussed in the main body.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper is empirical and does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In the paper, we have provided detailed description of the model structure and training method, and in the experimental section, we have detailed the evaluation protocol and the datasets used for comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code for the methods is provided the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines hhttps://nips.co public/guico
wtt.:/ubbbsssioooooooyoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https! ssionPolicy for more details.
/ //nips. httpccccccbcccccccccccceceecc
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We specified the training pipeline, the evaluation settings and protocol, and the evaluation datasets in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the statistical significance of the experiments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the appendix, we describe the computational resources used to complete model training and experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:://eerrrrs.ccccoooolccettts
Answer: [Yes]
Justification:
The paper is compliant with the NeurlPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work has no potential negative societal impacts. The topic the paper is using large models for table understanding and question answering, which can be applied in areas such as finance, healthcare, and science to improve productivity.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssil swi thcode. swooodee com/ datasttsss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer[[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy what should or should not be described.
for
20


Reviews:
[{"review_id": "cHG3xT3OC2", "reviewer": "Reviewer_3Co2", "summary": "The paper introduces EcoRouter, a recommendation-style router that chooses, at inference time, the “cheapest” large-language model (LLM) and inference mode (direct, self-consistency, CoT, ToT) that will satisfy an accuracy target. Key ingredients are (i) multidimensional representations for both models and prompts (intrinsic metadata + capability “coreset” scores + cross-features), (ii) a Wide-&-Deep/DeepFM-family recommender that learns from >1 B synthetic interactions covering 52 LLMs and 30+ data sets , and (iii) a cost-interval Pareto-frontier metric for evaluation. On six benchmarks EcoRouter reportedly cuts average unit cost by > 38 % with no loss of accuracy and shrinks average output length by ~30 % compared to strong baselines", "questions": "please refer to the weaknesses", "limitations": "please refer to the weaknesses", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n(1) Timely, practically important problem – production users do face ballooning LLM-API bills; a routing layer that reasons about both model and reasoning depth is valuable.\n\n(2) Clear high-level formulation – turns routing into a constrained optimisation (Eq 1) with a Pareto-front interpretation; keeps accuracy as first-class objective.\n\n(3) Large-scale empirical effort – 50 + LLMs, 3 reasoning modes, 6 diverse data sets; zero-shot tests on unseen models/tasks.\n\n(4) Strong headline numbers – 38 % cost saving at equal accuracy; top accuracy among routing baselines on both general-knowledge (Table 1) and math/BBH (Table 2).\n\nWeakness:\n\n(1) Missing SOTA routers in the comparison. The experiments pit EcoRouter only against Random, GTELarge embedding match, and a few recommender baselines (§4.2 Tables 1–2). Recent cost-aware routers—FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM—are cited in Related Work but never evaluated, so it is unclear whether EcoRouter’s 38 % cost saving is actually new.\n\n(2) Heuristic cost model. “Token-level costs are estimated from model scale, architecture, etc.” (§3.3, Eq. 2) rather than using real API prices or wall-clock latency. Pricing tiers (e.g., input vs output tokens, 128 k context surcharges) and batching effects are ignored, so the 38 % figure may be optimistic.\n\n(3) No routing-time latency budget. EcoRouter adds three text-encoder passes plus a recommender forward for every query (§3.1/§3.2). The paper does not report milliseconds/query or amortised GPU cost. A router that is cheaper in tokens but slower than simply calling a large model may be unusable.\n\n(4) Over-reliance on scale as a cost proxy. Parameter count (“#Params”) is treated as proportional to cost in Tables 1–2, yet real inference cost is dominated by sequence length and hardware utilisation. For instance, a sparsely-gated MoE (Mixtral-8×22B) costs far less per token than dense 70 B models.\n\n(5) Potential train-test leakage via the 1 415-example ‘bridge’ coreset. The same MMLU categories are used to construct model capability features (§3.1) and to evaluate routing accuracy (Tables 1–2 include MMLU, ARC-C, etc.). If overlap is not carefully filtered, EcoRouter may implicitly see the test distribution during representation learning.", "was_revised": true, "final_justification": "Thanks for the detailed responses from the authors. I think most of my concerns are well addressed. Considering my score is already positive, I keep my score. Hope the authors could include the new content during rebuttal into the final version.", "rebuttal": "We sincerely thank Reviewer 3Co2 for their detailed and insightful review of our work.\n\nWe are pleased that the reviewer recognized several core strengths of our work, including the importance of the problem, the clarity of our formulation, and the effectiveness of our experiments.\n\nIn the following, we provide detailed explanations and clarifications for the points raised. We are committed to incorporating the corresponding revisions into the final version of our paper:\n\n### 1. Comparison with SOTA Routers\n\nWe have added several SOTA comparison methods, including FrugalGPT, GraphRouter, Hybrid-LLM, and RouteLLM. Furthermore, we have included additional experiments on RouToo, RouterDC, and EmbedLLM.\n\nIn addition to the existing six benchmarks, we have expanded our evaluation to include Hellaswag, Winogrande, MMLU-Pro, MATH, MBPP, HumanEval, and MT-Bench. This expansion covers a broader range of tasks, including general knowledge, mathematics, code generation, and multi-turn dialogue.\n\nOur method creates a detailed model representation for the routing process. By using a paradigm of `(model representation + instruction) -> predicted performance`, we consider various aspects of the models, such as their intrinsic information and capability distributions. In a sense, this approach encapsulates the principles of many of the compared methods.\n\nOur replication details are as follows:\n\n1.  **FrugalGPT (Cascade Strategy)**\n    * **Routing Mechanism**: This method employs a cascade calling and post-generation scoring mechanism. It does not predict performance but rather evaluates the generated answer after the fact.\n        * **Input**: `(generation result from the current model, original instruction)`. The output is similar to a reward model, assessing whether the result is satisfactory.\n    * **Replication Details**: We constructed an LLM cascade using `Qwen2-7B-Instruct`, `Qwen2-57B-A14B-Instruct`, and `Qwen2-72B-Instruct`. For the scoring function, we used the DistilBERT model from their official code, retrained on our own data. To ensure evaluation stability, we did not use the \"prompt adaptation\" or \"LLM approximation\" (caching) strategies from the original paper.\n\n2.  **GraphRouter**\n    * **Routing Mechanism**: Treats routing as an edge prediction task on a graph.\n        * **Input**: The instruction and a description of the models.\n    * **Replication Details**: Due to limitations in the official implementation regarding multi-GPU distribution for single-graph data, we completed the replication using a subset of the training data.\n\n3.  **Hybrid-LLM**\n    * **Routing Mechanism**: DeBERTa + a classifier.\n        * **Input**: The instruction. The model representation consists of the parameters of each DeBERTa classifier.\n\n4.  **RouteLLM**\n    * We primarily replicated its two core strategies (the other two are similar to Hybrid-LLM):\n        * **Similarity-Weighted Ranking**: **Input** is a `new instruction`. It is compared against historical data and modeled using the Bradley-Terry model. **Output** is a `full ranking of the models`.\n        * **Matrix Factorization**: **Input** is `(instruction embedding, model embedding)`, where the model embedding is derived from the factorization of a historical performance matrix. **Output** is a `predicted performance score`.\n\n5.  **Methods from RouterBench**\n    * We replicated its non-cascade routers:\n        * **k-Nearest Neighbors (KNN) Router**: **Input** is the `embedding vector of the request`. **Output** is a `performance estimate based on the average performance of the K-nearest neighbors` (where historical performance data serves as the model representation).\n\n6.  **Routoo**\n    * **Routing Mechanism**: Uses a decoder-only LLM to process the input, combined with a unique model embedding vector.\n        * **Input**: `(instruction, target model ID)`. The model ID maps to a learnable \"model embedding.\" We did not adopt this user-item style of learnable representation because it would require incremental retraining of the entire routing structure whenever a new model is added.\n\n7.  **RouterDC**\n    * **Routing Mechanism**: Trains the router based on contrastive learning.\n        * **Input**: `(instruction, learnable model embedding)`.\n\n8.  **EmbedLLM**\n    * **Routing Mechanism**: Learns compact representations of models via matrix factorization.\n        * **Input**: The instruction and relevant historical performance data.\n\n| Model | MMLU | TruthfulQA | ARC-C | MMLU-STEM | GSM8K | BBH |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Qwen2-7B-Instruct | 69.1 | 55.5 | 61.4 | 63.5 | 78.9 | 62.9 |\n| Qwen2-72B-Instruct | 83.8 | 54.9 | 68.6 | 79.9 | 88.9 | 82.9 |\n| Random Selection | 73.0 | 58.9 | 67.8 | 68.0 | 77.7 | 73.6 |\n| Wide & Deep | 82.3 | 67.4 | 71.7 | 77.4 | 86.1 | 78.3 |\n| DIN | 83.9 | 66.8 | 73.0 | 78.4 | 87.2 | 79.9 |\n| FrugalGPT | 81.8 | 63.6 | 72.6 | 73.1 | 83.2 | 77.5 |\n| RouterBench | 72.3 | 62.0 | 71.9 | 70.2 | 80.1 | 74.7 |\n| Hybrid-LLM | 74.1 | 59.6 | 69.3 | 70.8 | 77.0 | 73.3 |\n| RouteLLM (Bradley-Terry) | 79.0 | 62.8 | 69.6 | 71.0 | 81.7 | 74.1 |\n| RouteLLM (Matrix Dec.) | 77.5 | 60.8 | 66.5 | 70.7 | 81.3 | 74.5 |\n| GraphRouter | 80.2 | 59.8 | 69.7 | 75.3 | 80.4 | 75.8 |\n| ROUTOO | 80.9 | 63.6 | 71.7 | 75.5 | 83.2 | 75.2 |\n| RouterDC | 81.7 | 64.2 | 73.2 | 75.0 | 81.8 | 77.2 |\n| EmbedLLM | 82.0 | 62.4 | 72.3 | 70.6 | 81.6 | 76.4 |\n\n### 2. Cost in Relation to Real API Pricing\n\nOur cost model is based on estimations of model scale, architecture, and generation length. Our initial goal was to establish a **general and relatively stable** framework for measuring cost that does not directly depend on the frequently changing and complex pricing rules of specific API providers (like OpenAI or Google). This approach ensures our conclusions have broader applicability.\n\nDue to network instability, we have not yet completed supplementary experiments with real APIs. However, in the final version, we will purchase API access for the relevant models (with their real-world pricing) and include a comparison of the actual costs incurred.\n\n### 3. Lack of Discussion on Routing Latency\n\nThe online inference overhead of EcoRouter primarily comes from two components: 1) **Feature Extraction**: We use a small model like `GTE-large` (~0.33B) to extract text embeddings. Its parameter count and computational cost are significantly smaller than any LLM in the candidate pool. 2) **Recommendation Model Inference**: The recommendation framework we employ is essentially a shallow network, making its inference extremely fast with negligible overhead.\n\nWe have added the routing latency in the table below:\n\n| Average Latency on GSM8K | EcoRouter Routing Time | Qwen2-7B-Instruct Inference Time |\n| :--- | :--- | :--- |\n| Time | ~0.3s | 5~7s |\n\n### 4. Over-reliance on Scale as a Cost Proxy\n\nWe would like to clarify our cost calculation. In our framework, the unit token cost is estimated based on the model's scale. The total cost is then calculated as the product of this **\"model's unit token cost\"** and the **\"length of the generated output (in tokens)\"**.\n\nWe will update the paper to make this clearer in the final version, particularly in the tables. We appreciate this valuable suggestion.\n\n### 5. Potential Train-Test Data Leakage\n\nOur \"bridge coreset\" and the evaluation test sets are completely disjoint. We will state this more explicitly in **§3.1** and **§4.1** of the paper.\n\nFurthermore, we've observed that the learning difficulty increases significantly when there is a large distributional shift (measured by the Wasserstein distance) between the historical performance dataset and the target evaluation set.\n\n| Distributional Distance (Wasserstein) between Historical Data and GSM8K | EcoRouter Learning Difficulty (Convergence Steps) |\n| :--- | :--- |\n| 6.29 | ~2k |\n| 38.41 | ~5k |\n| 53.2 | ~8k |\n\nOnce again, we thank you for your valuable feedback. We will continue to work diligently to improve our paper."}, {"review_id": "OLOM8Vmyar", "reviewer": "Reviewer_N3dk", "summary": "The paper introduces EcoRouter, a system that frames LLM routing as a cost-aware recommendation problem. The approach represents candidate models and prompts using a combination of their complexity level, benchmark capabilities, and embedded features. These representations are then used to train a Wide-&-Deep recommender model on a 1B cost-labeled model-instruction pairs. In practice, the system is designed to select the most cost-effective model that satisfies a user-defined accuracy budget. It achieved over 38% in cost savings across six public benchmarks.", "questions": "1. The 1B training set is prohibitively expensive for most researchers. What happens if the corpus is 1–2 orders of magnitude smaller?\n2. How does EcoRouter perform on RouterBench, especially on coding tasks?", "limitations": "This work primarily focus on single-turn prompts, while multi-turn or tool-calling workflows not addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- By casting the LLM routing problem as a cost-aware recommendation task, the authors align it with well-understood paradigms and avoid the inefficiencies of expensive cascading approaches. \n- A key contribution is the introduction of a Pareto-front metric for evaluating cost-accuracy trade-offs which is missing in prior work. \n- EcoRouter embeds cost during training by ranking 1 B interaction pairs via a Pareto-front rule. Thus the model is explicitly cost-aware.\n- EcoRouter is capable of handling a large set of models and integrating cost directly into the training objective, rather than applying it as a post-hoc filter. \n- It also provides a clear strategy for handling new models and out of domain prompts, a \"cold-start\" problem not fully detailed in relevant works. \n\n**Weaknesses**\n- The most significant gap lies in its baseline comparisons. Random choice or embedding-similarity matching are far from the state-of-the-art routing methods. Comparing against competitive works, such as RouteLLM (ref 38 in the paper), Hybrid-LLM (ref 18 in the paper), could strengthen the effectiveness of the method further.\n- While the current evaluation covers six public benchmarks, it would be nicer to test on more practical and complexity tasks, like coding, retrieve, and conversational tasks. As a comprehensive benchmark for assessing efficacy of LLM routing system, RouterBench [1] could be a good option to test with.\n- The lack of an ablation study on feature groups makes it unclear whether the complex capability vectors offer a real advantage over simpler representations.\n\n[1] Hu, Qitian Jason, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer and Shriyash Kaustubh Upadhyay. “RouterBench: A Benchmark for Multi-LLM Routing System.” ArXiv abs/2403.12031 (2024): n. pag.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer N3dk for their recognition of our work's contributions, including: innovatively framing the LLM routing problem as a cost-aware recommendation task, introducing the Pareto-front metric to evaluate the cost-benefit trade-off, and explicitly embedding cost into the training process.\n\n### 1 & 2. Regarding Baseline Models and Evaluation Benchmarks:\n\nWe have added nine new comparative methods and seven new benchmarks (including tasks for mathematics, code, and multi-turn dialogue), taking into account all the evaluations mentioned in RouterBench. For details, please refer to our response to Reviewer 3Co2. We will incorporate the relevant experimental results and analysis in the final version of the paper.\n\n### 3. Regarding the Ablation Study on Feature Groups:\n\nWe conducted a study on combinations of different model features, as shown in the table below:\n\n| Intrinsic Properties | Capability Distributions (Offline Benchmark) | Capability Distributions (Online Evaluation) | EcoRouter Accuracy | Overhead |\n| :--- | :--- | :--- | :--- | :--- |\n| ✓ | | | 79.8 | 496 |\n| | ✓ | | 81.3 | 513 |\n| | | ✓ | 84.0 | 468 |\n| | ✓ | ✓ | 84.2 | 435 |\n| ✓ | ✓ | ✓ | 86.1 | 404 |\n\n*(Note: The checkmark '✓' indicates the feature group was included, and a blank space indicates it was excluded, corresponding to '1' and '0' in the original table respectively.)*\n\n### 4. Regarding the Scale of Training Data:\n\nWe are committed to open-sourcing all of our training data. This data was generated through LLM inference, with the majority of interactions with smaller LLMs being conducted on a 4090 GPU.\n\nFurthermore, training the recommendation model is not as time-consuming or expensive as training a large language model. The entire training process for our reported results took approximately 30 hours on a system with 8x NVIDIA A100 GPUs, indicating that the volume of training data is manageable.\n\nWe have supplemented our work with an analysis of routing accuracy under reduced training data sizes:\n\n| Number of Training Samples | 100k | 500k | 1100k (Reported in the paper) |\n| :--- | :--- | :--- | :--- |\n| **Average Performance** | 82.7 | 84.7 | 86.1 |\n| **Average Overhead** | 485 | 468 | 404 |\n\nWe will include this sensitivity analysis on the data scale in the final version of the paper. Thank you very much for this valuable suggestion."}, {"review_id": "6fwsV9CEOJ", "reviewer": "Reviewer_k8wC", "summary": "This paper works on the model routing task, which is to select from a pool of LLMs the one that can achieve certain tasks while having the lowest cost. The method proposes to represent each LLM with a set of features, including their intrinsic features and performance on evaluation benchmarks. Borrowing ideas from recommender system, a routing model is trained to recommend LLMs for data.", "questions": "- It would be better to compare with other model routing methods and conduct more comprehensive analysis on the model performance and cost.\n- There are several missing citations in the paper. It would beneficial from some proofreading.\n- The definition of Pareto frontier is interesting but it doesn't seem to be relevant to latter discussion.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- A comprehensive list of LLM features are used to train a recommender model, which looks to be a promising direction for LLM routing problem.\n- Experiments show the strong performance of the proposed method.\n\n### Weaknesses\n- The experiments could be improved to be more comprehensive. First, there should be inclusion of other routing methods, instead of purely comparing with basic LLMs.\n- Also, only reporting the overall performance is not sufficient. Including additional analysis on the model is necessary and often encouraged in the main content.", "was_revised": true, "final_justification": "My concerns are addressed during rebuttal and I have increased the rating to reflect my final judgement", "rebuttal": "We sincerely thank Reviewer k8wC for their meticulous review of our work. We have added relevant experiments and will present an improved study in the final version of the paper.\n\n### 1. Regarding the comparison with other model routing methods and a more comprehensive performance and cost analysis:\n\nWe have incorporated nine additional comparison methods and seven new benchmarks, including tasks in mathematics, coding, and multi-turn dialogue. This expansion covers all the evaluation metrics considered in RouterBench. For a detailed breakdown, please refer to our response to Reviewer 3Co2.\n\n### 2. Regarding the missing citations:\n\nThank you for your careful examination. We have conducted a rigorous proofreading of the entire manuscript and will ensure that all citations are complete and accurate in the final submission.\n\n### 3. Regarding the definition of the Pareto Frontier and its relevance to the subsequent discussion:\n\nWe introduced the concept of the Pareto Frontier to provide a **clear framework and ultimate goal** for the complex optimization problem of LLM routing. An ideal routing system should identify a **Pareto-optimal** solution within the two-dimensional space of \"cost-accuracy.\"\n\nIn our experimental tables, we establish a theoretical upper bound by creating a \"Best-Performing\" baseline—a composite model selecting the optimal model for each specific task. This represents a point on the theoretical Pareto Frontier that prioritizes maximum accuracy.\n\nFor instance, in Table 2, the average performance of our method (e.g., Ours w/ DIN at 78.20%) closely approaches the Pareto Frontier (79.39%). Crucially, the **average response length** required to achieve this performance (473) is significantly lower than that of large models like Meta-Llama-3 70B (608).\n\nThe table below presents further results demonstrating that by incorporating cost-accuracy control inputs into our routing process, the resulting router can form a superior Pareto Frontier.\n\n| Model Input Preference: | High Accuracy | Balanced | Low Cost |\n| :--- | :--- | :--- | :--- |\n| **Theoretical Pareto Frontier (Accuracy, Cost)** | (88.9, 535) | - | (56.2, 273) |\n| **EcoRouter (Accuracy, Cost)** | (86.1, 404) | (79.2, 304) | (78.5, 316) |\n\nIn the final version of the paper, we will include a \"cost-accuracy\" scatter plot to visually demonstrate how EcoRouter effectively pushes the overall system performance towards the theoretical Pareto Frontier."}, {"review_id": "x6rLu7YLmR", "reviewer": "Reviewer_MPGf", "summary": "This paper introduces EcoRouter, a novel framework for routing user instructions to the most cost-effective Large Language Model (LLM) from a given library, while maintaining high performance. The core problem addressed is the inefficient and expensive use of large, powerful LLMs for tasks that could be handled by smaller, cheaper models. EcoRouter frames this challenge as a recommendation system problem, where the goal is to \"recommend\" the best LLM (item) for a given instruction (user).", "questions": "1. Throughput analysis with other LLMs.\n2. Comparison with sota LLMs, e.g., Qwen2.5-14B-1M achieves 94.8 on GSM8K.\n3. Does the proposed method achieve steady improvement with more few-shot samples?", "limitations": "Please refer to the questions section.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The conceptualization of LLM routing as a recommendation system problem is highly innovative and practical. It elegantly transforms the task into a well-understood domain, allowing the use of powerful and mature architectures like Wide & Deep learning. This framing directly addresses the need to balance memorization (specific model strengths) and generalization (handling unseen instructions).\n2. The paper tackles a critical real-world challenge in the operationalization of LLMs: cost management. As LLM APIs become ubiquitous, the financial burden is a major concern for developers and businesses. EcoRouter offers a principled and data-driven solution to this problem.\n3. A major advantage of EcoRouter is its design for real-world API deployment. By avoiding pre-inference on all candidate models, it minimizes latency. Furthermore, its ability to incorporate new LLMs by generating representations and performing lightweight indexing, rather than full retraining, makes it adaptable to the rapidly evolving LLM landscape, a clear advantage over static Mixture-of-Experts (MoE) models.\n\nWeaknesses\n\n1. The success of EcoRouter is heavily dependent on the massive, pre-computed model-data interaction dataset. Creating and maintaining this dataset requires immense computational resources (running millions of inferences across dozens of models). This presents a significant upfront engineering and financial cost, which may limit the practical adoption of this approach for smaller organizations or research labs.\n2. While the paper emphasizes that it avoids the cost of inferring with all candidate LLMs, it is not fully transparent about the computational cost of the router itself. The data representation step involves extracting embeddings and \"meta-dimensions\" (which may require a separate LLM call), and the recommendation model itself has an inference cost. Quantifying this overhead (e.g., in milliseconds per query) is crucial for a complete cost-benefit analysis.\n3. The paper defines cost as per-token cost × total tokens. While a reasonable proxy, real-world LLM API pricing is often more complex, with different rates for input and output tokens, potential costs for specific tool usage, and tiered pricing models. The system's optimality is tied to this simplified cost function.", "was_revised": true, "final_justification": "The author's rebuttal partly resolved my doubts (I don't see any explanation about Q1 and Q3), and I decided to maintain the previous rating unchanged.", "rebuttal": "We sincerely thank Reviewer MPGf for the valuable feedback. We have carefully considered each point and provide our responses below:\n\n### 1. Regarding the high cost of constructing the model-data interaction dataset:\n\nFirst, we commit to open-sourcing the model-data interaction dataset. The cost of its construction is a one-time, offline expense. In application, this data resource can be continuously and seamlessly collected from interactions between the models and users.\n\nOnce the dataset is built and EcoRouter is trained on it, the online routing inference process is extremely efficient. (We have added an experiment **in our response to Reviewer 3Co2** to substantiate this claim).\n\nFurthermore, this construction cost can be amortized. The process is not GPU-intensive (as it primarily involves inference), and most of the data was generated using NVIDIA 4090 GPUs.\n\n* **Handling New Data:** Our routing system can generalize directly to new data as it emerges.\n* **Handling New Models:** When a new model is introduced, our method rapidly adapts by performing inference on a lightweight \"bridge\" coreset to update the capability distribution. This avoids the need for a full re-inference on all historical data and complete retraining, making the cost of small-scale updates manageable.\n\nTo enable the router to accurately learn the \"preferences\" and capability boundaries of different LLMs, a large-scale, high-quality history of interactions is indispensable. This is crucial for achieving precise and economical routing. We will elaborate on these upfront costs in the final version of the paper.\n\n### 2. Regarding the computational overhead of the router model itself:\n\nThe online inference overhead of EcoRouter primarily stems from two components:\n\n1.  **Feature Extraction:** We use a small model like `GTE-large` (~0.33B) [307] to extract text embeddings. Its parameter count and computational cost are significantly smaller than any LLM in our candidate pool.\n2.  **Router Model Inference:** The recommendation framework we employ [327] is essentially a shallow network, making its inference extremely fast with negligible overhead.\n\nWe have supplemented our paper with a table detailing the routing latency:\n\n| Metric | Routing Time at Deployment | Qwen2-7B-Instruct Inference Time |\n| :--- | :--- | :--- |\n| Average Latency on GSM8K | ~0.3s | 5-7s |\n\nWe will add a dedicated section in the final version to analyze the specific latency and other related overheads of the routing model in detail. Thank you for this valuable suggestion.\n\n### 3. Regarding the simplification of the cost function:\n\nThe formula `Cost = Price_per_token × Total_tokens` is indeed a simplification of real-world API pricing models. We adopted this approach for the following reasons:\n\n* This formula effectively captures the two most critical factors influencing cost: the model's scale (which determines the unit price) and the length of the generated content. Similar proxies are widely used in related work such as FrugalGPT, AutoMix, and Hybrid-LLM.\n* A core strength of EcoRouter is its flexibility concerning the cost function. The function is not hard-coded into the model architecture; it is primarily used to rank the training data (see Equation 2 in the paper). To adapt to more complex pricing strategies (e.g., different pricing for input/output tokens, tiered pricing), we only need to replace the cost function, re-rank the training data accordingly, and retrain the recommendation system. The fundamental methodology and framework remain unchanged.\n* We have added an experiment where cost requirements are provided as a descriptive input. The results are shown in the table below:\n\n| Model Cost Input | Prioritize High Accuracy | Balanced | Prioritize Low Cost |\n| :--- | :--- | :--- | :--- |\n| **EcoRouter (Accuracy, Cost)** | (86.1, 404) | (79.2, 304) | (78.5, 316) |\n\nWe will discuss this simplification more explicitly in the limitations section of our paper and emphasize that our framework can be easily adapted to more sophisticated cost models.\n\n### 4. Regarding the comparison with SOTA LLMs:\n\n* EcoRouter is **not a new LLM** but rather an **LLM routing framework or scheduling system**. Its performance ceiling is determined by the models within its candidate pool.\n* Our framework is designed to **leverage and orchestrate** these SOTA models, not to compete with them. To demonstrate this, we have conducted a new experiment by adding Qwen2.5-14B to our candidate model pool. This integration is lightweight: we only need to perform inference with the new model on the coreset to obtain its capability distribution, without any need for fine-tuning the router. The results are as follows:\n\n| Model | MMLU | TruthfulQA | ARC-C | MMLU-STEM | GSM8K | BBH |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| Qwen2-7B-Instruct | 69.1 | 55.5 | 61.4 | 63.5 | 78.9 | 62.9 |\n| Qwen2.5-14B-Instruct | 79.7 | 58.4 | 67.3 | 76.4 | 90.2 | 78.2 |\n| Qwen2-72B-Instruct | 83.8 | 54.9 | 68.6 | 79.9 | 88.9 | 82.9 |\n| **Wide & Deep (Before)** | **82.3** | **67.4** | **71.7** | **77.4** | **86.1** | **78.3** |\n| **Wide & Deep (After)** | **82.0** | **67.7** | **71.7** | **77.5** | **86.8** | **78.1** |\n\nAs shown, EcoRouter effectively routes relevant requests (e.g., for GSM8K) to the newly added, more capable Qwen2.5-14B to boost performance. Concurrently, for simpler tasks, it continues to select more economical models, thus maintaining cost-efficiency.\n\n### 5. Regarding the impact of few-shot sample quantity on performance:\n\n* **Inference Modes as Routing Options:** In EcoRouter, we treat the tuple `(model, inference_mode)` as a holistic candidate for routing. The input features for the router can be augmented with additional information, such as the number of few-shot samples or the reasoning strategy to be used (e.g., CoT, ToT [343]).\n* **Learning the Trade-off:** Increasing the number of few-shot samples generally improves accuracy but also increases the context length, leading to higher inference costs. EcoRouter is specifically designed and trained to learn this **trade-off**.\n\n***\n\nOnce again, we thank you for your valuable time and constructive feedback. We are confident that by incorporating these revisions based on your suggestions, the quality of our paper will be significantly enhanced."}]

Author rebuttal:

Meta-review: This paper introduces EcoRouter, a recommendation-system-based framework for routing user instructions to the most cost-effective LLM in a candidate pool. By constructing multidimensional model and data representations (intrinsic properties, benchmark capability scores, and cross features) and training a Wide&Deep/DeepFM-style recommender on >1B synthetic interactions, EcoRouter learns to predict which (model, inference mode) pairing best satisfies a user-defined accuracy target under cost constraints. A Pareto frontier–based evaluation metric is also proposed. Empirically, EcoRouter reduces costs by >38% on six benchmarks across 50+ LLMs while maintaining accuracy.

Strengths. Reviewers agreed the framing of routing as a recommendation problem is both novel and practically important (Reviewer k8wC, N3dk, MPGf). The introduction of a Pareto-frontier cost–accuracy metric was noted as a methodological contribution (Reviewer N3dk). The large-scale evaluation across diverse LLMs and reasoning modes demonstrated strong headline improvements (Reviewer 3Co2). Practical deployment advantages, such as avoiding inference on all candidate models and adaptability to new LLMs, were emphasized as real-world strengths (Reviewer MPGf).

Weaknesses. Several reviewers highlighted limitations in evaluation: missing competitive baselines such as RouteLLM, GraphRouter, Hybrid-LLM, and FrugalGPT (Reviewer 3Co2, N3dk, k8wC); absence of RouterBench coding/multi-turn benchmarks (Reviewer N3dk); and limited ablation analyses (Reviewer N3dk, k8wC). Cost modeling was flagged as simplified and heuristic, not tied to real API pricing or latency (Reviewer 3Co2, MPGf). Concerns about router overhead and whether cost savings persist in real deployments were raised (Reviewer 3Co2, MPGf). Reviewer 3Co2 also questioned possible train–test leakage from the MMLU-based coreset. Finally, the approach requires a massive precomputed interaction dataset (>1B samples), raising barriers for replication (Reviewer MPGf).

Discussion and Rebuttal. The authors’ rebuttal was substantive and constructive. They added nine missing baselines (FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM, etc.) and seven new benchmarks (including coding and dialogue), as well as ablations on feature groups and sensitivity to training data size. They clarified that router overhead is ~0.3s vs. 5–7s for a 7B LLM, and committed to adding real API pricing experiments. They also confirmed no leakage between the bridge coreset and test sets. Reviewers largely accepted these clarifications: Reviewer k8wC explicitly raised their score; Reviewer N3dk stated all concerns were fully addressed; Reviewer 3Co2 maintained a borderline-accept rating but acknowledged improvements; Reviewer MPGf appreciated the responses but held their borderline-reject score, citing only partial resolution of concerns.

Decision. While one reviewer remains unconvinced, the majority of reviewers shifted positively after rebuttal, recognizing the paper’s strong empirical contribution, novel framing, and practical significance for LLM deployment. The weaknesses (e.g., dataset construction cost, proxy cost model) are real but do not undermine the central technical contributions. Given its relevance to scalable and economical LLM usage—a timely and practically pressing problem—I recommend acceptance. However, due to residual concerns about evaluation completeness and cost realism, I do not recommend spotlight/oral; the work is valuable but not yet fully mature in evaluation rigor.

Follow-up for Camera-Ready. For the final version, please:
1. Incorporate the newly added baselines (FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM, etc.) and expanded benchmarks (e.g., coding, dialogue) into the main paper.
2. Provide empirical results under real API pricing and latency measurements, not only heuristic proxies, to strengthen cost-effectiveness claims.
3. Explicitly report routing overhead in ms/query and clarify the amortized cost relative to direct large-model inference.
4. Clearly document how the bridge coreset is separated from evaluation datasets to avoid any impression of train–test leakage.
5. Discuss the scalability and accessibility of the required 1B+ interaction dataset, and outline strategies for researchers with limited resources (e.g., smaller-scale training feasibility).
6. Add a short discussion of broader limitations, such as reliance on single-turn prompts and the simplified cost definition, as flagged by reviewers.

Recommendation: Accept.
Decision: Accept (poster)