Title: Let the LLM Stick to Its Strengths: Learning to Route Economical LLM
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Routing

Deep Learning
Abstract: Recently, test-time scaling of Large Language Models (LLMs) has emerged as a practical alternative to parameter and data scaling. Reasoning tasks often require large-scale, RLVR-based LLMs, while more economical LLMs can handle simpler tasks. Routing an LLM tailored to *suitability* (*i.e.*, capability and cost) ensures usability and efficiency. We introduce LLMRec, which routes the most suitable LLM to the user query without pre-inference on the candidate LLM zoo. It pioneeringly reframes the LLM routing problem as a comprehensive recommendation system (RecSys) task. Our core insight is that an LLM's suitability for a query is a complex, latent signal equal to user-item preference. LLMRec systematically engineers features for candidate LLMs (intrinsic attributes and capability distributions), queries (general semantics and meta-dimensional info), and context (inference type, cost budgets). It also incorporates behavioral features to learn high-order interactions. LLMRec is designed to generalize to out-of-domain datasets and adapt to new LLMs as the model zoo evolves. We define the metric with the Pareto frontier under user-specified cost budgets. Across six datasets, LLMRec achieves an average cost reduction of over 38% while maintaining accuracy and consistently outperforming baselines in converging toward the Pareto frontier.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces EcoRouter, a novel framework for routing user instructions to the most cost-effective Large Language Model (LLM) from a given library, while maintaining high performance. The core problem addressed is the inefficient and expensive use of large, powerful LLMs for tasks that could be handled by smaller, cheaper models. EcoRouter frames this challenge as a recommendation system problem, where the goal is to \"recommend\" the best LLM (item) for a given instruction (user).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The conceptualization of LLM routing as a recommendation system problem is highly innovative and practical. It elegantly transforms the task into a well-understood domain, allowing the use of powerful and mature architectures like Wide & Deep learning. This framing directly addresses the need to balance memorization (specific model strengths) and generalization (handling unseen instructions).\n2. The paper tackles a critical real-world challenge in the operationalization of LLMs: cost management. As LLM APIs become ubiquitous, the financial burden is a major concern for developers and businesses. EcoRouter offers a principled and data-driven solution to this problem.\n3. A major advantage of EcoRouter is its design for real-world API deployment. By avoiding pre-inference on all candidate models, it minimizes latency. Furthermore, its ability to incorporate new LLMs by generating representations and performing lightweight indexing, rather than full retraining, makes it adaptable to the rapidly evolving LLM landscape, a clear advantage over static Mixture-of-Experts (MoE) models.\n\nWeaknesses\n\n1. The success of EcoRouter is heavily dependent on the massive, pre-computed model-data interaction dataset. Creating and maintaining this dataset requires immense computational resources (running millions of inferences across dozens of models). This presents a significant upfront engineering and financial cost, which may limit the practical adoption of this approach for smaller organizations or research labs.\n2. While the paper emphasizes that it avoids the cost of inferring with all candidate LLMs, it is not fully transparent about the computational cost of the router itself. The data representation step involves extracting embeddings and \"meta-dimensions\" (which may require a separate LLM call), and the recommendation model itself has an inference cost. Quantifying this overhead (e.g., in milliseconds per query) is crucial for a complete cost-benefit analysis.\n3. The paper defines cost as per-token cost × total tokens. While a reasonable proxy, real-world LLM API pricing is often more complex, with different rates for input and output tokens, potential costs for specific tool usage, and tiered pricing models. The system's optimality is tied to this simplified cost function.\n\nQUESTIONS:\n1. Throughput analysis with other LLMs.\n2. Comparison with sota LLMs, e.g., Qwen2.5-14B-1M achieves 94.8 on GSM8K.\n3. Does the proposed method achieve steady improvement with more few-shot samples?\n\nLIMITATIONS:\nPlease refer to the questions section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-24"}

comments: SUMMARY:
This paper works on the model routing task, which is to select from a pool of LLMs the one that can achieve certain tasks while having the lowest cost. The method proposes to represent each LLM with a set of features, including their intrinsic features and performance on evaluation benchmarks. Borrowing ideas from recommender system, a routing model is trained to recommend LLMs for data.

STRENGTHS AND WEAKNESSES:
### Strengths
- A comprehensive list of LLM features are used to train a recommender model, which looks to be a promising direction for LLM routing problem.
- Experiments show the strong performance of the proposed method.

### Weaknesses
- The experiments could be improved to be more comprehensive. First, there should be inclusion of other routing methods, instead of purely comparing with basic LLMs.
- Also, only reporting the overall performance is not sufficient. Including additional analysis on the model is necessary and often encouraged in the main content.

QUESTIONS:
- It would be better to compare with other model routing methods and conduct more comprehensive analysis on the model performance and cost.
- There are several missing citations in the paper. It would beneficial from some proofreading.
- The definition of Pareto frontier is interesting but it doesn't seem to be relevant to latter discussion.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces EcoRouter, a system that frames LLM routing as a cost-aware recommendation problem. The approach represents candidate models and prompts using a combination of their complexity level, benchmark capabilities, and embedded features. These representations are then used to train a Wide-&-Deep recommender model on a 1B cost-labeled model-instruction pairs. In practice, the system is designed to select the most cost-effective model that satisfies a user-defined accuracy budget. It achieved over 38% in cost savings across six public benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- By casting the LLM routing problem as a cost-aware recommendation task, the authors align it with well-understood paradigms and avoid the inefficiencies of expensive cascading approaches. \n- A key contribution is the introduction of a Pareto-front metric for evaluating cost-accuracy trade-offs which is missing in prior work. \n- EcoRouter embeds cost during training by ranking 1 B interaction pairs via a Pareto-front rule. Thus the model is explicitly cost-aware.\n- EcoRouter is capable of handling a large set of models and integrating cost directly into the training objective, rather than applying it as a post-hoc filter. \n- It also provides a clear strategy for handling new models and out of domain prompts, a \"cold-start\" problem not fully detailed in relevant works. \n\n**Weaknesses**\n- The most significant gap lies in its baseline comparisons. Random choice or embedding-similarity matching are far from the state-of-the-art routing methods. Comparing against competitive works, such as RouteLLM (ref 38 in the paper), Hybrid-LLM (ref 18 in the paper), could strengthen the effectiveness of the method further.\n- While the current evaluation covers six public benchmarks, it would be nicer to test on more practical and complexity tasks, like coding, retrieve, and conversational tasks. As a comprehensive benchmark for assessing efficacy of LLM routing system, RouterBench [1] could be a good option to test with.\n- The lack of an ablation study on feature groups makes it unclear whether the complex capability vectors offer a real advantage over simpler representations.\n\n[1] Hu, Qitian Jason, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer and Shriyash Kaustubh Upadhyay. “RouterBench: A Benchmark for Multi-LLM Routing System.” ArXiv abs/2403.12031 (2024): n. pag.\n\nQUESTIONS:\n1. The 1B training set is prohibitively expensive for most researchers. What happens if the corpus is 1–2 orders of magnitude smaller?\n2. How does EcoRouter perform on RouterBench, especially on coding tasks?\n\nLIMITATIONS:\nThis work primarily focus on single-turn prompts, while multi-turn or tool-calling workflows not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces EcoRouter, a recommendation-style router that chooses, at inference time, the “cheapest” large-language model (LLM) and inference mode (direct, self-consistency, CoT, ToT) that will satisfy an accuracy target. Key ingredients are (i) multidimensional representations for both models and prompts (intrinsic metadata + capability “coreset” scores + cross-features), (ii) a Wide-&-Deep/DeepFM-family recommender that learns from >1 B synthetic interactions covering 52 LLMs and 30+ data sets , and (iii) a cost-interval Pareto-frontier metric for evaluation. On six benchmarks EcoRouter reportedly cuts average unit cost by > 38 % with no loss of accuracy and shrinks average output length by ~30 % compared to strong baselines\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n(1) Timely, practically important problem – production users do face ballooning LLM-API bills; a routing layer that reasons about both model and reasoning depth is valuable.\n\n(2) Clear high-level formulation – turns routing into a constrained optimisation (Eq 1) with a Pareto-front interpretation; keeps accuracy as first-class objective.\n\n(3) Large-scale empirical effort – 50 + LLMs, 3 reasoning modes, 6 diverse data sets; zero-shot tests on unseen models/tasks.\n\n(4) Strong headline numbers – 38 % cost saving at equal accuracy; top accuracy among routing baselines on both general-knowledge (Table 1) and math/BBH (Table 2).\n\nWeakness:\n\n(1) Missing SOTA routers in the comparison. The experiments pit EcoRouter only against Random, GTELarge embedding match, and a few recommender baselines (§4.2 Tables 1–2). Recent cost-aware routers—FrugalGPT, GraphRouter, Hybrid-LLM, RouteLLM—are cited in Related Work but never evaluated, so it is unclear whether EcoRouter’s 38 % cost saving is actually new.\n\n(2) Heuristic cost model. “Token-level costs are estimated from model scale, architecture, etc.” (§3.3, Eq. 2) rather than using real API prices or wall-clock latency. Pricing tiers (e.g., input vs output tokens, 128 k context surcharges) and batching effects are ignored, so the 38 % figure may be optimistic.\n\n(3) No routing-time latency budget. EcoRouter adds three text-encoder passes plus a recommender forward for every query (§3.1/§3.2). The paper does not report milliseconds/query or amortised GPU cost. A router that is cheaper in tokens but slower than simply calling a large model may be unusable.\n\n(4) Over-reliance on scale as a cost proxy. Parameter count (“#Params”) is treated as proportional to cost in Tables 1–2, yet real inference cost is dominated by sequence length and hardware utilisation. For instance, a sparsely-gated MoE (Mixtral-8×22B) costs far less per token than dense 70 B models.\n\n(5) Potential train-test leakage via the 1 415-example ‘bridge’ coreset. The same MMLU categories are used to construct model capability features (§3.1) and to evaluate routing accuracy (Tables 1–2 include MMLU, ARC-C, etc.). If overlap is not carefully filtered, EcoRouter may implicitly see the test distribution during representation learning.\n\nQUESTIONS:\nplease refer to the weaknesses\n\nLIMITATIONS:\nplease refer to the weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-06"}