RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers
Authors: Yifan Lu
Rixin Liu
Jiayi Yuan
Xingqi Cui
Shenrun Zhang
Hongyi Liu
Jiarong Xing
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Router
Evaluation
Primary area: datasets and benchmarks
TL;DR: We proposed a comprehensive LLM router evaluation framework, with a new dataset, metrics, and ranking system.

Abstract: Today's LLM ecosystem comprises a wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances.
However, the rapid emergence of various routers has led to fragmented evaluation practices and inconsistent metrics, making it difficult to systematically assess progress in this space. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce RouterArena, the first open platform enabling comprehensive comparison of LLM routers. RouterArena has (1) a principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of evaluation metrics, and (4) an automated framework for evaluation and leaderboard updates. Leveraging this framework, we have produced the initial leaderboard with detailed metrics comparison.
Figure1 provides a preview of the leaderboard. The complete framework and the latest router leaderboard are publicly available at https://routeworks.github.io/

Published as a conference paper at ICLR 2026
ROUTERARENA: AN OPEN PLATFORM FOR COMPREHENSIVE COMPARISON OF LLM ROUTERS
Yifan Lu", Rixin Liu", Jiayi Yuan', Xinggi Cui, Shenrun Zhang, Hongyi Liu, Jiarong Xing Department of Computer Science, Rice University, Houston, TX 77005, USA (y1231, r1165, jy101, xc66, sz81, h187, jx22) (ec)).diceee.
ABSTRACT
Today's LLM ecosystem comprises wide spectrum of models that differ in size, capability, and cost. No single model is optimal for all scenarios; hence, LLM routers have become essential for selecting the most appropriate model under varying circumstances. However, the rapid emergence of various routers has led fragmented evaluation practices and inconsistent metries, making difficult systematically assess progress in this space. To address this problem, we need a comprehensive router comparison and a standardized leaderboard, similar to those available for models. In this work, we introduce ROUTER ARENA, the first open platform enabling comprehensive comparison of LLM routers. ROUTER ARENAAAAAA has (1) principally constructed dataset with broad knowledge domain coverage, (2) distinguishable difficulty levels for each domain, (3) an extensive list of eval- uation metrics, and (4) automated framework for evaluation and leaderboard updates. Leveraging this framework, we have produced the initial leaderboard with detailed metrics comparison. [ preview of the leaderboard. The complete framework and the latest router leaderboard are publicly available at hhtps://rruttwworks githhbbbgo
RouterArena Leaderboard
Router Performance Cost
MIRT-BERT (67.3)
Azure-Router (66.4)
CARROT (63.9)
VLLM-SR (63.8)
GPT-5
Accuracy
0.60
NIRT-BERT (60.8)
0.55
GraphRouter
CARROT
MIRT-BERT
0.50
MotDiamond (59.4)
RouterDC
NIRT-BERT
RouteLLM
0.45
GraphRouter
NotDlamond
10
MLP (57.8)
KNN
Azure
0.40
MLP
GPT-5
12
RouterDC (35.2)
Commercial Routers
Academic Routers
0.35
RouteLLM
VLLM-SR
20
40
60
10-1
10a
10a
Scare
Inference Cost (USD per 1K queries)
Figure 1: preview of ROUTERARENA'S 's leaderboard and performance-cost trade-off comparison.
INTRODUCTION
Large Language Models (LLMs) are rapidly diversifying, offering an ever-wider spectrum of capa- bilities at different inference costs. This diversity increasingly challenges the prevailing L.MM usage pattern in which users manually choose models for their queries. The difficulty stems from the fact that no single model is universally optimal: powerful models excel at complex tasks but are costly, while smaller models are more efficient yet may struggle on difficult queries. As a result, LLM routers that automatically select models based on input queries are increasingly recognized core system primitive in practical deployments. 'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2026
Given its importance and promise, many LLM routers have recently emerged in both industry and academia (Figure 2) A notable example is GPT-5 (OpenAl 2025), which incorporates routing as key feature by directing user queries to the most suitable model within the OpenAI family, As routing rapidly proliferates and becomes a core function in L.M systems, the evaluation frontier correspondingly shifts: beyond comparing individual models, we must now rigorously evaluate routing strategies. Unfortunately, router evaluation has not kept pace: there is currently no open evaluation platform, akin LMArena (Chiang et al. 2024), that systematically compares open- source routers (Hu et al. 2024 Zhuang et al. 2024) and commercial routing services (NotDiamond 2025 Microsoft 2025) under a unifted protocol.
It is urgent to fill this gap by building Router Arena that can comprehensively evaluate and compare routers, providing a clear and systematic view of the state of the art. However, unlike model arenas, designing router arena is considerably more challenging due to the requirements from three key aspects. (1) Dataset. To evaluate whether router can recognize problem domains and dispatch queries to appropriate models at minimal cost, the arena dataset must cover a broad range of domains and subjects, as well as varying difficulty levels. (2) Metrics. Router performance is inherently multi multi-dimensional, and so should be the arena ranking. While accuracy and cost are the primary metrics, is also important to capture other dimensions such as routing optimality and robustness. (3) Framework. To enable live leaderboard updates, the arena must have user-friendly framework that can automatically evaluate new open-source and commercial routers. Although existing studies explore some of these directions, as summarized in Table [ and discussed in Section 2 address each challenge in a comprehensive way.
they fail to
In this work, we present ROUTERARENS the first open platform for comprehensive evaluation and comparison of LLM routers. It addresses the above key challenges with the following designs:
Principled Diverse Dataset. To ensure broad coverage, we construct the dataset using the Dewey Decimal Classification system adopted in libraries, covering all domains except religion. For each subject, we apply Bloom's taxonomy to ensure queries span diverse cognitive skills and empirically validate that the queries cover three difficulty levels. This produces a diverse dataset of ~8,440 queries spanning domains and 44 categories for router evaluation.
Extensive Metrics for Arena Ranking. We construct router leaderboards by considering an ex- tensive list of deployment-relevant metries including query-answer accuracy, query-answer cost, routing optimality (cheapest correct selection). robustness query perturbations (consistency), and router overhead (latency). This enables router comparison from multiple perspectives.
An Automated Framework for Leaderboard Updates. We design framework that automat- ically evaluates new routers, collects metrics, and updates the leaderboard. The framework sup- ports both open-source and commercial routers, and employs prefix caching to improve efficiency.
Figure D provides a preview of our accuracy-cost leaderboard along with other details. We have found that although GPT-5 achieves higher accuracy, its cost is significantly higher than that of other routers due to its model pool being restricted to the OpenAl family. Consequently, it does not rank as the best router on our accuracy-cost leaderboard.
Our vision is for RENT to serve as an open community venue for evaluating routers as the ecosystem evolves, providing standardized basis for fair comparison and progress tracking. By lowering the barrier to evaluation and enabling transparent, reproducible results, ROUTERAREN: will help researchers and practitioners design, improve, and adopt better routers.
MOTIVATION
The Rapid Emergence of LLM Routers. As shown in Figure 2 the landscape of LLM routers
is rapidly expanding, evolving from academic exploration to commercial deployment. From a few
scattered academia routers (Zhang et al. 2023 Chen et al. 2023 Hari & Thomson
2023) in mid-
2023, the number of publications expanded to more than a dozen by 2024 (Liu et al. 2024 Zhuang
et al. 2024| Chen et al. 2024: Zhao et al. 2024). By 2025, not only did academia routers continue to 2024|Chen
grow (Wang et al. 2025 Huang al. 2025 Ding al. 2025 Zhang et al. 2025b), but commercial
products also emerged (NotDiamond with built-in router.
2025 Microsoft 2025
most notably GPT-5 (OpenAl 2025)
2

---

Published as a conference paper at ICLR 2026
vLLM Semantic Router LightRouter
Open Source
Commercial
Azure Model Router
Marrion
LLM Blander
FrugalGPY
Router RouterBench Bench
Nomoodd
OPILLM
Hybrid LLM
MetaLLM AutoMix
MiXLLM
RouteLLM
Requesty
IRT Router
Model GLUE
BES T T-Route
GPT-5
timeline
Jul 2023
Jun 2024
Dec 2024
Mar 2025
Sep 2025
Figure 2: Timeline of example router-related work and products.
Table 1: Comparison of existing work (Hu et al. 2024 Huang et al. 2025 Feng et al. 2025b Zhuang et al. 2024) and ROUTERARENA, ROUTERAREN, enables comprehensive router compar- ison with extensive query categories, difficulty levels, evaluation metrics, and router inclusion.
Benchmark
Query categories
Difficulty levels
Evaluation Metries
Commercial Routers
Ronter Ranking
RouterBench
24 Categorics
X No analysis
x Deferal curve only
RouterEval
27 Categories
X analysis
Accuracy
only
FusionBench
26 Categories
L.M. juige analysis
x Deferra curve only
EmbedLLM
26 Categories
x No analysis
Accuracy
only
ROUTERARENA
based DDC
44 Categories
/
3 Empitically Verified Veriiffe Difficulty Levels
/ 5 Evaluation perspectives
/ Included
V Mult-metric leaderboard
The Need for Systematic Router Evaluation. With the rapid proliferation of routing methods, the field lacks principled framework to comprehensively evaluate and compare them. Such comprehensiveness entails dataset categories, difficulty levels, evaluation metries, and router inclusion. However, our review of existing work reveals that existing benchmark pro- vides such holistic evaluation. As shown in Table prior work falls short the following aspects.
New Problem:
Narrow Query Category Coverage. They lack full coverage of query categories, making them impossible to evaluate router performance on queries from excluded categories. Indistinguishable Difficulty Levels. They do not differentiate queries by difficulty, limiting their ability to test accuracy-cos: tradeoffs.
Limited Evaluation Metrics. They only consider subset of relevant metries, overlooking impor- tant dimensions such as optimality, robustness, and latency.
No Support of Commercial Routers. Current frameworks evaluate only open-source routers and do not extend to closed-source or commercial routers.
No Router Leaderboard. There no leaderboard that allows people to compare all routers under unified evaluation protocol.
This Work: ROUTERARENA This gap motivates us to design ROUTERARENA, an open plat- form for comprehensive router comparisons. In the remainder of this paper, we first introduce the key components of ROUTERA ARENNAA: principled dataset construction, comprehensive metric formu- lation, and an automated evaluation framework with live leaderboard. then present our evaluation results and discuss the key findings.
ROUTERARENA EVALUATION DATASET
To enable meaningful and unbiased router comparisons, a high-quality evaluation dataset is essen- tial. In this work, construct such dataset by adhering three guiding principles.
Principle DDC-Inspired Diverse Domain Coverage. To evaluate a router's ability to rec- ognize problem domains and route queries to the appropriate specialist models, the dataset must provide broad domain coverage. To achieve this, we draw inspiration from the Dewey Decimal Clas- sification (DDC) system (Dewey. 1876). a book classification framework widely used in libraries. The DDC is renowned for its comprehensive and logical structure, providing proven methodology for organizing the entire world of knowledge into distinct, hierarchical categories (Satija 2013).
Principle 2: Cloom-Guided Cognitive Skill Levels. Routers in practice face queries that range
from simple factual recall to complex reasoning. To ensure coverage of cognitive skills from simple recall to higher-order reasoning and judgment, adopt Bloom's taxonomy (Bloom et al.) 1956), a widely used framework for quantifying question complexity and cognitive processes (Ullrich & Geierhos 2021
Herrmann- Werner al. 2024 Pad6 2017).
3

---

Published as a conference paper at ICLR 2026
Principle 3: Empirically Defined Difficulty Levels. To test whether router can make mean- ingful accuracy-cost tradoonf- between powerful but expensive models and weaker but cheaper ones- the dataset must also provide clearly distinguishable difficulty levels. We define dif- ficulty empirically: for each query, we evaluate it across diverse set of models and measure the fraction that produce correct answer. Queries solved by fewer models are considered more difficult.
Social sciences, sodilogyy, and anthropooggy
Philosophical
Paychology
Difficulty Distribution
Social problems"
Philesuphy
easy
medium
Sdance
Largragg
Filess
science Anewiertee.
Blography and genenlogy
23 23.4%
Homemaaiis
spaluins
Geegraphy
47.5%
Paysica
actances
RouterArena
Literature
History
29.1%
C Chemistry
and gailay
Westory
Arts & recreation
Mesersse tnod
at cridk-en len
Technology
Bislogy
and
Science
Animals
Language
Managament
Social
Philosophy psychology
Computer Science, information sciance, general works
Dataset Distribution by Domain and Category
Figure 3: Dataset composition. For ease of demonstration, we merged some categories.
Dataset Construction Process. Following these principles, we curate our evaluation dataset as follows. To ensure coverage across all DDC categories (excluding religion), we first collect all the queries from two existing LLM benchmark datasets and then supplement underrepresented cate- gories with data from 23 open-source, domain-specific datasets, resulting in 62,000 raw queries. We then cap each source dataset at approximately 1,000 queries To determine the cognitive skill level of each query based on Bloom's 's taxonomy, we employ an LLM-as LLLM-asJJdge approach with DEEPSEEK-V3. V3.1 (DeepSeek-Al 2025) (prompt specified in Appendix D). We validate the effec- tiveness of this with a small-scale human study (Appendix(C.. (.T).. Note that we use these Bloom levels only to characterize the cognitive skills involved in each query; the ground-trute -ouund difficulty labels are empirically determined based on Principle 3.
Next, to fairly distribute questions across categories and cognitive levels, we propose a recursive deficit redistribution algorithm. We begin by setting the ratio of science to humanities at 2:1. Within each top-level category, if sub-category falls short of its proportional quota, the resulting surplus recursively and uniformly redistributed to those sub-categories that exceed their initial allocation. We apply the same procedure within each sub-category allocate data across different cognitive levels, ensuring balanced coverage throughout the dataset.
However, this dataset contains many highly similar even duplicate questions inherited from the various sampled sources. Such redundancy does not benefit router evaluation and may even intro- duce noise into the results. To address this, we perform oosine-immilarit-baase TREEEE-TRRFFORRRRRSS MALL-MMMILLMMLLL
de-duplication using By strictly following the allocation strategy and selecting the least similar samples, we ensure that the resulting ROUTERARENA dataset main- tains broad coverage with minimal redundancy.
The Resulting Dataset. Our final evaluation dataset consists of 8,400 queries sampled from 23 source datasets. It spans nine level domains and 44 categories in Figure (Left). To empirically characterize difficulty, we select 42 representative models covering a broad spectrum of capabili- ties (details in Appendix C.4). For each query, measure the number of models that produce correct answer and categorize queries into three bands-hard (s4/42), medium (5-19/42), and easy (220/42) shown in Figure 3 (right). Figure 4 further shows that, when sorted by this measure, both the overall and per-domain curves increase smoothly from low to high accuracy without large gaps, indicating well-spread spectrum of difficulty that can distinguish router behavior. Note that this empirical difficulty distribution is skewed, but it reflects real-world query tanterns-eayy and
It primarily affects large corpora such WMT and SuperGI.UF and yields roughly 30,000 candidate queries while preventing few sources from dominating the pool.
4

---

Published as a conference paper at ICLR 2026
medium questions occurring more frequently than hard ones. We include more details about the dataset in Appendix C.2 including dataset schema and concrete query examples.
Query Difficulty Distribution: Overall and by Category (42 LLMs evaluated)
Overall
Computer science, Philesophy and
3 Social Science
4 Language
100%
Averaged reerged Accuracy over LLMs
50%
Science
Technology
Literature
History
100%
50%
0%% 30%
100%
50% Query Percentile (0% Hard, 100% Easy)
150%
100%
Figure 4: Queries are sorted by their average accuracy over all 42 LLMs. We observe smooth, monotonic curves overall and in each domain, indicating that query difficulty is well spread across the full range from very hard (left) to very easy (right).
4 ROUTERARENA EVALUATION METRICS
ROUTERARENA supports comprehensive router evaluation along five dimensions.
(1) Query-Answer Accuracy. This metric captures router's ability to direct queries to the ap- propriate models such that they are correctly answered. We calculate accuracy as the average cor- rectness across all dataset queries.
(2) Query-Answer Cost. This measures the cost incurred by a router's routing decisions. To address important factors such as the variable cost introduced by input length and generation length (e.g., in chain-of-thougle reasoning) as well as the distinct computational characteristics of Mixture- of of-Experts (MoE) models, we use the actual inference cost measured by:
cost
Cin
Nin + Cout N
Where is the cost per token and N is the number of tokens. We obtain the cost for the specific models router chooses using the official API pricing published by the corresponding providers (e.g., OpenAI, Claude, Fireworks AI, etc.). For unpopular models that not served by commercial providers, we deploy them ourselves for experiments and approximate their costs using the pricing tiers published by commercial hosting platforms (e.g., Together.ai). which estimate serving costs based on model size (parameter count) and architecture type (e.g., MoE) in Table
(3) Routing Optimality. This captures router's ability to perform optimal routing- that is, lecting the cheapest model that still produces a correct response. It consists of three sub-metrics: (a) Optimal Selection Ratio-the proportion of queries for which the router answers correctly by selecting the cheapest model; (b) Optimal Accuracy Ratio-the ratio between router's achieved accuracy and the upper-bound accuracy obtainable when always selecting the best model from its model pool: (c) Optimal Cost Ratio the ratio between the cost incurred by the router's 's selections and the cost of always choosing the optimal model. This metric will penalize routers that rely on unnecessarily expensive models when cheaper, correct alternatives are available.
(4) Routing Robustness. This metric evaluates router's robustness against noisy inputs. We calculate it as the proportion of queries for which the router makes consistent routing decisions under perturbed input. Specifically, we generate noisy variants queries through paraphrasing, gram- matical changes, synonym substitutions, and typosyypat and measure the percentage of cases where the router selects the same model as it does for the original, noise-free query. This captures the router's capability for handling realistie, imperfect user queries.
5

---

Published as a conference paper at ICLR 2026
(5) Routing Latency. Since the router operates in the critical path of systems in production, it must handle millions of queries per second with minimal overhead. This metric measures the additional latency introduced by routing. It reflects the latency increase in both -ffrttt-kntt (TTFT) and end-to-end response latency when a given router is employed.
ROUTERARENA EVALUATION FRAMEWORK
5.1
ARENA RANKING
ROUTERARENA provides a series of router leaderboards that enable users to compare the capabil- ities of different routers and select the best one suited their scenarios. includes ranking scores based on the evaluation metrics described in Section 4 including Arena Score, Optimal- Selection- Ratio, Optimal-Acc-Ratee
Optimal- Soall--ootttiiooo Robustness, and Latency. Among these, the Arena Score Si,3 captures the trade-off between accuracy and cost by combining them into single composite measure using the Weighted Harmonic Mean (Ferger 1931). Specifically, to better distinguish between routers with low costs, we apply base-2 logarithmic (log2) transformation to the cost values. Under this scaling, each doubling of price reduces the cost score by one unit. For router with cost C₁, define its normalized cost
(0gz(Cmaz) log2c;
C logz(Cmaz> (ogi(Cmin)
where Cmax and Cmin denote the maximum and minimum costs of routing 1,000 queries. Specifically, we choose Cmin = 0.0044, corresponding to the cost of the cheapest model in the leaderboard's model pool. This reflects the cost a router that always selects the cheapest model. We set Gmaz = 200, which represents the most expensive model, OpenAl's OI-PRO. This normalization maps the cost into range [0, 1]. with larger values of C, corresponding to more economical routers. Next, we combine the normalized cost C, and accuracy A, using a weighted harmonic mean:
St,5 = (1 + A,
AA; + +G
where the parameter > controls the relative importance accuracy versus cost. Intuitively, specifies the relative weighting applied to cost: assign cost weight of 3 in comparison to accuracy. By default, we use 3 = 0.1, emphasizing routing accuracy, because highly accurate routers are generally more valuable even if they incur slightly higher costs.
Details about how we update Grin and Cmans recommendations on values of B for different user needs provided in Appendix H
5.2 AUTOMATED EVALUATION FRAMEWORK
Although we demonstrate ROUTERARENA with a specific set of routers in this paper, it is very easy update the leaderboard with routers. To facilitate this process, we have designed an automated evaluation framework that has been released publicly alongside the leaderboard. Figure shows the overall system workflow. To evaluate new router, the user can simply start by providing our framework with an access point (e.g., an API) to the router. The framework sends evaluation queries to the router, which performs routing inference on its end and returns model selections. To ensure fairness, we run the inference ourselves and use cached results when possible, since many routers share overlapped model pools. During this process, the router's response time is monitored to measure routing latency. Finally, the framework computes the evaluation metrics and aggregates the results, which are reflected in the leaderboard. Note that some commercial routers may not expose their model selections and instead return query answers directly. Such routers can still be evaluated with our framework, although certain metrics cannot be measured in this setting.
EXPERIMENTS
6.1 EXPERIMENTAL SETTINGS
Router Selection. For commercial routers, we evaluated the router from Not Diamond NotDii mond 2025). which provides access over 60 models, and the Azure Model Router (Microsofi)

---

Published as a conference paper at ICLR 2026
10wary Query Osary8600)
B DATASETS
LATENCY
EVALUATION
Update
di
ROUTERARENA
LEADERBOARD
New Routee Sval
Qwayy
e 000
C D99)
400
Latoncy Morstrrnn
Acouncy 37% 2. Cosl: $0.30
Fadtuaiinn
Rassings
Roster faaking::
Duaffcudar
Pouder
Paulack
+ ROUTER
Bouter Iiference
PREDICTION
LL iriernnee
B INFERENCE RESULTS
Poree Bastifouter
Cuaryli...
ConMerre I.l.
Figure 5: The automated router evaluation and leaderboard update framework.
2025). which currently only supports OpenAl models. We also included GPT-5 (OpenAl 2025 whose model family incorporates an internal router. For NotDiamond, we selected 26 representa- tive models spanning different parameter scales, architectures, and reasoning abilities. For Azure- Router, we evaluated the entire model pool, including GPT-5 model families. Appendix B provides the full details of the model pools used for each router.
For open-source routers, we evaluated nine representative systems covering a diverse routing ap- proaches. Specifically, we chose both the KNN- and ML.P-ased methods trained on Router- Bench (Hu et al. 2024) as baselines. We further included GraphRouter (Feng et al. 2025a), which leverages graph neural networks (GNNs) for routing, and the Universal Router (Jitkrittum et al. 2025), which uses K-means clustering. To capture cost-accuracy tradeoffs, we evaluated CARROT Router (Somerstep al. 2025). while RouterDC (Chen et al. 2024) incorporated as a dual contrastive learning-bast based approach. Additionally, we considered IRT-Router (Song et al. 2025), which applies item response theory to explicitly model the interaction between query attributes and model capabilities, and RouteLLM (Ong et al., 2025), which performs binary selection between a stronger and weaker model. Moreover, we also take the latest vLLM Semantic Router (vLLM, 2025) into consideration, which leverages ModemBBRR (Hugging Face| 2025) to categorize the incoming requests into pre-defined categories, and selects the model that has the highest score.
Router Training and Evaluation. For commercial routers, no additional training is required; we simply accessed their provided APIs for evaluation. In contrast, for academia routers, we followed the training procedures and datasets specified in their appen-source implementations. Specifically, we did not modify the training datasets or the task categorizations (if applicable). The model pools were configured in accordance with the original papers. In particular, for the vLL.-SR, we constructed the pool using both open-source models of varying parameter scales and proprietary models, with detailed configurations summarized in Table 3 After training each router, we evaluated them by feeding our benchmark dataset, recording the model selected, the latency incurred by the selection, and the confidence scores assigned to all candidate models in the pool.
6.2 RESULTS
Normalized Router Performance VS Best Available Model
Deferral Curves: Accuracy Inference Cost
Oracle Acc. 90.89%
Accuracy (% of Best Available Model)
100
YLLM SR CARROT
RouCaLLM
MLP
MIRTBERT
GraphRouter
0.8
Accuracy
95
RouteLLM 8)
RouterDC
KNN
0.6
BERT (400% cost
Azure Router
CARRIT Hocter DC
90
Lower Accuracy Lower Cost (Sub-Ideal)
acc)
0.4
NutDiamand (771% cost.
NLP RecheLLM
GAT
acci
0.2
10-1
10"
MIM-S 5t
10a
85
RouteLLM (0.7)
60
100
120
Inference Cost (USD per 1K queries)
Cost (% of Best Available Model)
Figure 6: Deferral Curve: accuracy versus cost
Figure 7: Normalized Scatter Plot
Deferral Curve. Figure 6 presents the trade-off between accuracy and inference cost. As we increase inference budget, we unlock more powerful models, driving the accuracy up. For open- source routers, we leveraged their confidence scores to apply budget-based masking, which produces multiple points along each curve. With only cheap models available, accuracy remains low, but as larger models enter the pool, routing accuracy increases. In contrast, commercial routers typically appear as single points because their model pools already include the best-performing models.
7

---

Published as a conference paper at ICLR 2026
Actual Optimal Accuracy per Router (Binary vs Multi-router Regions)
Optimal-cost-rtioo
12.5
25
37,5
50
Optimal Selection Ratio
NotDiamond
NIRT BERT
CAR
VILMM SR
5.8%
MIRT BERT
KNN
Non binabi routers
12.0%
MLP
accurary
12.2%
GraphRouter
optimal accuracy
8.64
Azure Router
Optimal selection
22.3%
RouterDC
37.4%
optimal cost-rade exts)
M (0.9)
24.3%
RouteLLM (0.8)
B Binary routors
RouteLLM (0.7)
0.5
0.6
25 50
Accuracy
Optimal Ratio
Figure 8: Actual and optimal accuracy, along with optimal selection ratio and cost ratio
Two insights emerge. First, the orange dashed line shows the oracle accuracy, highlighting that all routers fall short of the best achievable performance. Second, the trade-off frontier differs by setting: commercial routers can achieve higher accuracy, but usually at significantly higher costs; open- source routers, on the other hand, achieve competitive performance much lower budgets, though they plateau earlier. Notably, routers like CARROT and GraphRouter illustrate cost-efficient routing, while systems such as GPT-5 and NotDiamond lean heavily on expensive models for accuracy. This suggests that while commercial routers prioritize maximizing accuracy, academic approaches often explore the efficiency side of the frontier.
Normalized Deferral Curve. Figure reports router accuracy and cost normalized each router's s hest-performing model, point (100%, 100%) on the plot. The upper-left quadrant repre- sents the ideal case- higher accuracy with lower cost by leveraging smaller models. In practice, most routers cluster near the baseline (100% cost, 100% accuracy), suggesting they over-rely on the strongest model and miss opportunities to defer to cheaper alternatives. Notably, NIRT-BERT illustrates inefficiency, reaching only baseline-level accuracy while incurring 400% of the cost.
By contrast, routers such as vL.L--SR and CARROT achieve meaningful savings: roughly 35% lower cost with under 2% accuracy degradation. These cases show routing indeed improve efficiency when smaller models are effectively utilized. Overall, the results highlight a clear trade- off higher accuracy often comes with higher whil also pointing promising directions for designing routers that move closer to the ideal frontier.
Optimality Score. Figure g highlights the inherent trade-off between routing accuracy and cost. In practice, routers that achieve higher accuracy typically do so the expense of higher cost ratio, since they defer more often to large, expensive models. This behavior lowers their optimal selection ratio, i.e., the frequency with which they choose the most efficient model for each query. This pattern most apparent in the binary routers such as RouteI..M. By design, these routers face a sharp trade- off: they achieve higher accuracy by routing more queries to the stronger model, which drives up cost. In contrast, multi-model routers have a more flexible pool, and while the general trend still holds, we see greater variability depending on pool composition and routing strategy. Among non- binary routers, RouterDC stands out with the lowest cost ratio and highest optimal selection ratio, but this comes at the cost of poor overall accuracy. At the other extreme, MIRT-BERT achieves strong accuracy (close to 77% of its optimal accuracy) but requires nearly five times the optimal cost, placing it closer to the "high-cost high-accuracy" region of the trade-off frontier. other words, while some routers are closer to the efficiency frontier than others, none simultaneously combine low cost and high accuracy. Overall, our findings indicate that current routing methods have learned to leverage large models to boost performance, but remain inefficient at recognizing when smaller models are sufficient. This creates a clear opportunity for future work: developing routers that can balance accuracy and efficiency by selectively deferring large models only when necessary.
Robustness and Latency. Given that user prompts are often noisy, we further assess routing sen- sitivity and robustness. The detailed evaluation is in Appendix G We define robustness as 1 the proportion of changed selections. As shown in Figure 9 (left), robustness is uniformly low across both commercial routers and academic routers such as VLLM-SR, MIRT-BERT, and NIRT-BERT. A key reason is that these academic routers depend on BERT to compute latent representations,
8

---

Published as a conference paper at ICLR 2026
and BERT is known to be sensitive to surface-level perturbations (Jin et al. 2020). These findings highlight the importance of applying prompt engineering techniques to mitigate noisy queries.
Router Robustness
RouteLLM
Router Latency
GraphRouter
14.29%
VLEM-SR
146.8
CARROT
89.05%
RouteLLM
1259.0
RouterDC
Kmeans
14.52%
65.24%
CARROT
KNN
85.3
KNN
GraphRouter
47.1
BERTBERT
NiRT-Bort
110.5
NotDiamond
10.911%
RouterDC
g19.3
Auzre- Router
5...7%%
MIRT-Bert
MRT-BERI BERT
1-9.2
Kmeans
[11.3
000
MLP
111.1
20% 40%
100%
120%
200
400
600
Robustness
robust)
Labency (ms)
Figure 9: Router robustness and latency comparison.
Furthermore, Figure 9 (right) reports the end-to-enc to- latency of routers on a single A100 GPU, mea- sured from the time request is received to the output of the model selection result. Among all methods, vLLM-SR and RouteLLM exhibit significantly higher latency because they rely on the OpenAl embedding API, which introduces additional network delays. In contrast, others consis- tently maintain sub- 100ms latencies. While LLM routers can optimize accuracy and cost, they also introduce non-negli overhead that may even compromise service-level objectives (SLOs).
We also report the average generation length and price per output tokens for each router in T GPT-5 and Azure-Router rely more heavily reasoning models, leading to substantially longer generations than most other routers. Longer answers are not inherently better; this metric simply offers an additional axis for users to trade off concise versus detailed responses.
Generation Length
6.3 RESULTS BY DIFFICULTY LEVELS
Table/[shows the accuracy-cost tradeoff across easy, medium, and hard queries. Most routers exceed 89% accuracy on easy questions, but accuracy drops sharply on medium and especially hard ones (often below 10%), confirming that our difficulty bands are non-trivial. GPT-5 achieves the highest accuracy at every level but at a much higher cost. At the same time, routers such as Azure-Router and vLLM-SR offer competitive accuracy on easy/medium queries at lower cost. This contrast reveals substantial headroom on hard questions and clear afficiency-accuraee tradeoffs across methods. We also observe that some routers (e.g., GPT-5, PrureRRouter MIRT-BERT, NotDiamond) allocate sig- nificantly more budget to hard than easy queries, while others show almost flat cost across difficulty, suggesting room for more difficult--aware routing strategies.
6.4 ADDITIONAL RESULTS ON LONGBENCH-V2
We provide additional results on LongBench-v2 in Appendix B On this long-context suite, GPT-5 attains the highest accuracy, while -ourerRouter and NIRT-BERT offer competitive performance substantially lower cost, illustrating clear accuracy-cost trade-offs in the long-context regime.
6.5 INSIGHTS FROM ROUTERARENA 's LEADERBOARD
Our evaluation produces the router leaderboard shown in Table The leaderboard consists of six ranking scores, and the overall ranking is determined by averaging across them. We highlight two key findings: (1) Commercial routers do not necessarily outperform open-source routers. For example, GPT-5 ranks #7 due to its restricted model pool, and NotDiamond ranks #12 because frequently selects expensive models. (2) No router ranks at the top across all metrics, reflecting the inherent trade-off's in router design.
For developers and researchers, the findings highlight key deficiencies in current routing methods and point toward clear directions for designing the next generation of routers. The results show that all existing routers fall short of the oracle's 's achievable performance, primarily because they are in- efficient at recognizing when smaller, cheaper models are sufficient for a given query. Future work should focus closing this performance gap. Moreover, the high latency and poor robustness of

---

Published as a conference paper at ICLR 2026
Table Ranking of routers across multiple metries. Lower values indicate better performance.
Optimal-
Optimal-Coot
Optimal-Acc
Robustaess
Ratio
Asun-Router
4,40
MIRT-BERT
4,50
RouteLLM
GPT-5
GraphRooter
4
2
5
5.50
MLP
10
5
6
5.67
CARROT
5.83
NIRT-RERT
10
10
VLLM-SR
NoiDiamond
11
8.80
certain routers open new avenues research beyond the traditional cost-accuracy trade-off. Devel- opers the platform's automated framework submit and benchmark routers against established leaders, fostering innovation and transparently tracking progress in the field.
7 RELATED WORK
With the increasing availability of specialized models that can surpass even the most capable seneral-purpose LLMs in specific domains, both academia and industry have been actively exploring how to build LLM routers. In industry, several systems have emerged. Martian Router (WithMartian 2025) proposed the idea of model mapping. while Storytell (Storytell.ai 2025) categorizes user queries and routes them to the best- best-performing models. Other companies also seek find the optimal model for user's tasks by balancing performance and cost (NotDiamond) 2025 ReqqeestyA, 2025 OpenAl 2025). Recent academic efforts have also begun to emerge. GraphRouter (Feng et al., 2025a) leverages graph neural networks, and Router-RI (Zhang et al,, 2025a) employs reinforcement learning. The growth of open-souree solutions underscores the need for effective router evaluation (Somerstep et al. 2025 Song et al. 2025) Feng et al. 2025b
LLM Router.
LLM Router Benchmarks. RouterBench (Hu et al. 2024) introduces a large-scale dataset con- sisting of over 405k inference outcomes from representative LLMs. RouterEval (Huang et al. 2025) collects performance results from 8,500 LLMs across 12 widely used benchmarks. Fusion- Bench (Feng et al. 2025b) covers 14 tasks across five domains and leverages 20 open-source L.MMs. Other benchmarks have also contributed to this line of work by using different data collection meth- ods (Kassem al. 2025 Mei et al., 2025). All of these benchmarks adopt an offline design: they release a large frozen corpus of model outputs that is ideal for cheap, fully reproducible comparison of routing algorithms under fixed model pool. By contrast, RouterArena explicitly designed as a live leaderboard that includes both academic and commercial routers. Our live leaderboard by nature encourages router designers to explore evolving and specialized models, making routing decisions more meaningful in practice. Meanwhile, RouterArena improves on prior router bench- marks by covering more tasks and domains, and by providing a dataset with empirically calibrated difficulty together with more comprehensive metrics.
CONCLUSION
We introduce ROUTERARENA, the first open platform for comprehensive router comparison. Our platform features principled dataset with broad domain coverage and varying difficulty levels, an extensive set of evaluation metrics, and an automated framework to maintain live leaderboard. Initial evaluations of 12 routers reveal significant trade-off between accuracy and cost, showing that no single router is universally optimal. Commercial routers tend achieve higher accuracy much greater expense, while open-source routers often present more cost-efficient solutions. Overall, our findings indicate that current routers are inefficient at leveraging cheaper models when appropriate, highlighting a clear opportunity for future work.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors have read and will adhere to the ICLR Code of Ethics (https://icir) cc/ Our evaluation dataset was constructed by aggregating and sampling from publicly available and open-source datasets. To ensure broad topic coverage while avoiding potentially sensitive subjects, we based our domain selection on the Dewey Decimal Classification system, explicitly excluding the category of religion. The difficulty level of each query was anno- tated using an "LLM-as-Juddee aasJudge" approach. We acknowledge that this process, along with the use of existing datasets, may introduce or perpetuate biases inherent in the source data and the anno- tator model. We have made the dataset and our collection methodology public to allow for further inspection and bias analysis by the community.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the full reproducibility of our work. All components of the ROUTER- ARENA platform, including the dataset, evaluation framework, and code, will be made publicly available.
The principles and detailed process for our dataset construction are described in Section 3. This includes domain coverage strategy, difficulty level annotation, and our deduplication process.
(1) Dataset:
Further details on the dataset schema, composition, and examples are provided in Appendix C. (2) Evaluation: Our five evaluation metrics are precisely defined in Section 4, and the formula for the composite leaderboard score is detailed in Section 5.1. The automated evaluation framework is described in Section 5.2. (3) Experiments: The specific academic and commercial routers evaluated are listed in Section 6.1. The exact model pools used for each router are provided in Appendix B. For all academic routers, we followed the training procedures and configurations specified in their original open-source implementations.
The public release of our complete framework will enable researchers replicate our results and evaluate new routers on the leaderboard.
REFERENCES
Benjamin S Bloom et al. Taxonomy of Educational Objectives. Handbook: The Cognitive Domain, 250, 1956.
Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, 2023. URL
2305. 05176
Shuhao Chen, Weisen Jiang, Baijiong Lin, James T. Kwok, and Zhang. RouterDC: Query- Based Router by Dual Contrastive Learning for Assembling Large Language Models, 2024. URL
httpp::/aarxa //tp:///avvv.v///////// wwwpxxvvvrrg//... abs/ 1409919888
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph Gonzalez, and Ion Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, 2024. URL
https://arxiv: org/oog.//.....o.ooooo..... abs 2403. 04132
Yu-Neng Chuang, Prathusha Kameswara Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, Xia Hu, and Helen Zhou. Learning to Route LLMs with Confidence Tokens, 2025. URL https:
7/arxiv. absg//bbs/240 2410.1384
DeepSeek-AI. DeepSeek- V3 Technical Report, 2025. URL. https::/erxiv.crrg/
2412. 19437
Melvil Dewey. A Classification and Subject Index, for Cataloguing and Arranging the Books and Pamphlets of a Library. Brick row book shop. Incorporated, 1876.
Dujian Ding. Ankur Mallick, Shaokun Zhang, Chi Wang, Daniel Madrigal, Mirian Del Car- Hipolito Garcia, Menglin Xia, Laks V. S. Lakshmanan, Qingyun Wu, and Victor Rühle.
11

---

Published as a conference paper at ICLR 2026
BEST-Route Adaptive LLM Routing with Test-Time Optimal Compute, 2025. URL https:
//ar http://rggaaaa/aaaaa/aa 22716
Tao Feng, Yanzhen Shen, and Jiaxuan You. GraphRouter: A Graph-based Router for LLM Selec-
tions, 2025a. UtLbttps::/aaae wttpaawwww.......//////aaaaaaaee wwtp/bo/gg.....o.oooooo.. 2410. 03834
Tao Feng, Haozhen Zhang. Zijie Lei, Pengrui Han, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, and Jiaxuan You. Fusing LLM Capabilities with Routing Data, 2025b. URL https:
laarg//bs//5000 10540
Wirth F. Ferger. The nature and use of the harmonic mean. Journal of the American Statistical Association, 26(173)366400 1931. doi: 0100000012215999111110055550
URL https://
www. www.fannoonnn ine. wtt.cooooo///ooo /tt//////// /10. Tttp:/10.110000000
Surya Narayanan Hari and Matt Thomson. Tryage: Real-time, intelligent Routing of User Prompts 11 11601
to Large Language Models, 2023.
Annette Seermann-WWrner Tanja Festl-Wietek, Friederike Holderried, Lena Herschbach, Jan Griewatz, Ken Masters, Stephan Zipfel, and Matthias Mahling. Assessing ChatGPT's Mas- tery of Bloom's Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods doi: 10.2196/52113. URL
Study. Journal of Medical Internet Research, 26:e52113, 2024.
https: /oii....... org. 10 02196655211
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. RouterBench: A Benchmark for Multi-LLM Routing System, 2024. [Rtptt / wwtp://wwww//b///...a/// 12031
Zhongzhan Huang, Guoming Ling, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, and Liang Lin. RouterEval: A Comprehensive Benchmark for Routing LLMS to Explore Model-level Scaling Up in LLMs, 2025. URL http::ttts////ctt /ttp::://ar..v.oo////aaa htt.ggggab//2503....
Hugging Face. ModemBBRR A New Era of Pretraining. httpss///ugg..gafccc.c///// modernbert]] 2025. Hugging Face Blog, accessed: Sep. 2025.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment, 2020. URL https:
11932
Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Congchao Wang, Zifeng Wang, Alec Go, Chen Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Universal Model Routing for Efficient LLM Inference, 2025. URL https:
Aly M. Kassem, Bernhard Schölkopf, and Zhijing Jin. How Robust Are Router-LLMMs? Analysis of the Fragility of LLM Routing Capabilities, 2025. htt.:/tttaa///rri..oo////////// 07113
Yueyue Liu, Hongyu Zhang, Yuantian Miao, Van-Hoang Le, and Zhiqiang Li. OpILLM: Optimal Assignment of Queries Large Language Models, 2024. URL htlthttps//axxxii http::/:///xxiv.or//aa...
24C5.15130
Kai Mei, Wujiang Xu, Shuhang Lin, and Yongfeng Zhang. OmniRouter: Budget and Performance Controllable Multi-LLM, Routing, 2025. hoRL:/ttpee//arxil.oo///////
20576.
Microsoft. Model Router for Azure AI Foundry (Preview) Concepts.
https:
[.learn..icrose. com/en-ur alaazure/aaa httpa///ooooo/////////////essss/ model router 2025. Accessed: Sep 2025.
NotDiamond, NotDiamond: Routing Between AI Models. https://www.not.ii.
about 2025. Accessed: Sep. 2025.
Isaac Ong, Amjad Almahairi, Vincent Wu. Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and lon Stoica. RouteLLM: Learning to Route LLMs with Preference Data,
2025. URL httphhhttss:aaaat attt:::/arxxi.orrr/bbb//////
www. /240 2466.186666666.8
12

---

Published as a conference paper at ICLR 2026
OpenAI. GPT-5 System Card. https://edn..om
August 2025. Accessed: Sep 2025.
Ulrike Padó. Question Difficulty- How to Estimate Without Norming, How to Use for Automated Grading. In Joel Tetreault, Jill Burstein, Claudia Leacock, and Helen Yannakoudakis (eds.), Pro- ceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 1-10, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
doi:
RequestyAl. Aeq Unified LLM Gateway Routing. https://wwe requesty aii
solution routing 2025, Accessed: Sep. 2025.
Mohinder Partap Satija. The Theory and Practice of the Dewey Decimal Classification System. Elsevier, 2013.
Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira. Prattyush Mangal, Mírian Silva, Onkar Bhardwaj, Mikhail Yurochkin, and Subha Maity. CARROT: A Cost Aware Rate Optimal Router, 2025. URL URL /tttp::/arxxi..oootm
abs/ 2502. 03261
Wei Song, Zhenya Huang, Cheng Cheng, Weibo Gao, Bihan Xu, GuanHao Zhao, Fei Wang, and Runze Wu. IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response The- ory, 2025.
hRtp/t:tt///aa org/ abs/ / 2506. 01048
Storytell.ai. Storytell.ai: Dynamic LLM Routing.
https: btops://webbt http://bbbbbbr.te storytel.
11 m-roouerr 2025. Accessed: Sep. 2025.
Sabine Ullrich and Michaela Geierhos. Using Bloom's Taxonomy Classify Question Com- plexity. In Mourad Abbas and Abed Alhakim Freihat (eds.), Proceedings of the 4th Inter- national Conference on Natural Language and Speech Processing (ICNLSS 2021), pp. 285- 289, Trento, Italy, 12-13 November 2021. Association for Computational Linguistics. URI
vLLM. vLLM Semantic Router. cessed: Sep. 2025.
com/ 2025. Ac-
Xinyuan Wang, Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. MixLLM: Dynamic Routing in Mixed Large Language Models, 2025. URL
/etps://arxivv owg/abs////aeooooeo..eeeeee
WithMartian. Martian Model Router.
https://docs..itimaatttt com/
martian haaoi-ooelll10 condel-rouue router 2025. Accessed: Sep. 2025.
Haozhen Zhang, Tao Feng, and Jiaxuan You. Router-R1: Teaching LLMs Multi-Round Routing and
Aggregation via Reinforcement Learning, 2025a. URL httl:ttts///aaxi...ott wwtp:/iiirrg//bb...o..oo..sso hatpg///abb////a/ /506.
09033
YiiKai Zhang, Ting-Ji Huang, Yao-Xiang Ding, De-Chuan Zhan, and Han-Jia Ye. Model Spider:
Learning to Rank Pre-Trained Models Efficiently, 2023. URL https: /ttp::///xxvvvvi www.vvorrrrooo abs/
2306. 03900
Yi-Kai Zhang, De-Chuan Zhan, and Han-Jia Ye. Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing, 2025b. URL
Xinyu Zhao, Guoheng Sun, Ruisi Cai, Yukun Zhou, Pingzhi Li, Peihao Wang, Bowen Tan, Yexiao He, Li Chen, Yi Liang, Beidi Chen, Binhang Yuan, Hongyi Wang, Ang Li, Zhangyang Wang, and Tianlong Chen. Model-GLUE: Democratized LLM Scaling for Large Model Zoo the Wild, 2024. URL Sttpttp:: / arxiv. org. abs /2100.100000 05357
Richard Zhuang, Tianhao Wa, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchan- dran. EmbedI.L.M Learning Compact Representations of Large Language Models, 2024. URL. https::/arxivw org hbtp://bs///100//0 2222.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LLMS
In this work, Large Language Models (LLMs) were used two distinct capacities: as a core com- ponent of our research methodology and general-purpont writing aid.
LLdees-Judge for Data Annotation: A significant use of an LLM was in the construction of our evaluation dataset. As detailed in Section 3, we employed DeepSeek-V33 V3.1 as an "LLMM-ss-Judee" to automatically annotate the cognitive level of each query according to Bloom's taxonomy. This automated approach allowed specific prompt used for this annotation process is provided in Appendix D.
systematically label large and diverse set of questions. The
Writing Assistance: An LLM also utilized general-purpose tool to assist the writing process. Its role was limited to proofreading the manuscript for grammatical errors, improving clarity, and ensuring stylistic consistency. The LLM was not used for research ideation, conducting experiments, or writing the core scientific contributions of the paper.
B
MODEL POOLS BY ROUTER
Table 3: Model pools used by different routers.
Router
Model Pool
RouterBench
WizardL M--Wardd----33---- claude-instant-vl; claude-v1; claude-v2;
gpt-3.5-
turbo- 106; gpt-4-1 106-preview; metalcodelllama333--inttte meta/llama-2-00bchatt
mistralainmiittrll---bbti
Sttpalaaimmixtaal----------ee zero-one-ai/V-34BB
GraphRouter
meta- metts-llmm//llmm-------inn
misttrlli/mixxrra----bbbee
nouseesearchhnous:33bbhhhe meta/llama-3-7000 (atplllmmaa-----hhaaaa meta/llama-3-turbo meta/llama-3.1 --t---b-----o- qwenIqwen-1.5-72b.
Beeta/lllaa---b--aee mistralai/mistral-77 -bb-iitttt-----aatt 8b-chat;
Universal
WizardLMMWizardLLMMM 1-B--M1---B-M 33-2,..2;
claude-instant-vl; claudc-vl;
claude-v2; gpt-3.5-
turbo- 1106; gpt-4-1 106--review, metalcodellama- 3at------b-beneeee
2-ttlllamaa--70booooo
mistralaimmistrll-bbchh mistralai/mixtrall -arull--xxx-hhhtee aoo-one-ai/Yi-34B
CarrotRouter
aww-claude-3---soomee--v/
aws-titnn-exx--rrmierr.. openai-gpt-4o; openai-gpt-4o-mini 1t----nntruct----maa-----
wxai-gramite-3-2bbr
wxai-
-atilllmaa-------nnitiie llama-3-1-00--inttuutt, wxai-llama-3-1--4 1-a--------b---ttccc
wxai-llama- wxai-llama-3
3-2-3b-instruce wxai-llama- 3-t--lamm-----7---nntttte
405b-instruct
RouterDC
mistralai/Mistral
meta-math/ math/Metatttttt -isttall--- itpossible/Chinese Mistrral--- wetptt////eehhy7----beeeee 7b-beta; oggntiveommpuatiooooohhiii..n..o 2.-miiitp meta-
v0.1; -asimaa---bb-nstttee 3-8b-instruct; oggttiveoompuaatiossolliii-- 2.9
llama/llama -alllama--888
H.9 llama3-8b
IRT-Router
amir,,air; glm.4_flash; glm_4.pm clpuu_, gpt.4o; gpt_4o_mini; gpt_4o_mini_cott deepseck_codee coderr
deepseck_etts wwq_32b.preview qwen25_man www.mmaatthttttteeeeee
qwen25_ int4; qwen25_72 _b_nstrctt: qwen25_722 2nmnnmtttbn
llamaš1 instruct;
llama31 14055 instruct;; mixtral 8x7b instruct; mistral_7h insttruct_002; minis-
tral instrrctttt 2410; gemini1. flash; claude35 haiku20241022
RouteLLM
openai-gpt-4p mixtral 8x7b instruct
We provide the model pool used by each router here.
C DATASETS DETAILS
C.1 USE OF LLM--A-A-JUDGE
To determine the alignment of LLM as judge with human judgment of Bloom's level of the query, we sampled 450 queries from the dataset by stratifying along both the domain and Bloom level, ensuring that each combination
proportionally represented. Eighteen volunteers (undergraduate and graduate students) were asked to assign a Bloom level to each query under the same instruc- tions as used for the DEEPSEEK-VE V judge. For each query, compare the human label with
14

---

Published as a conference paper at ICLR 2026
the LLM's prediction. We observe 54.9% exact agreement and 76.7% agreement within +1 Bloom level, indicating that most disagreements occur between adjacent levels in this inherently subjec- tive taxonomy. Since Bloom labels are used only to ensure cognitive-skill coverage during dataset construction-white all evaluation difficulty defined empirically via accuracy over 42 L.Mssthis level of agreement is sufficient for purposes.
C.2 DATASET COMPOSITION AND SCHEMA
Computer science,
Philosophy and
Social Science
Categery
Language
Science
Technology
Arts & recreation
Literature
History
40
Percentage within Category (%)
Source Dalaset
AIME
GSMBK
MMLLUPro
PubMedQA
ArdMMLU
GeoBench
MathQA
QANTA
AsDiv
GeoGraphyData
MedMCQA
SocieliQA
Chessinstruct
LiveCodeBench
TheoryBench
SuperGLWE
Ethics
MATH
NarrativeQA
WMT19
FinQA
MMLU
Openi DB
Figure 10: The source of ROUTERAREN, evaluation dataset.
Figure 10 illustrates the domain coverage of our dataset across the nine Dewey Decimal categories. Each horizontal bar represents the relative contribution of different source datasets within a cate- gory. This distribution highlights the systematic integration of multiple datasets achieve more balanced representation of both general-purpose and highly specialized domains.
Table 4: Overview of dataset columns
Column
Description
Example
Category
Bloom's taxonomy high-level class
History
Sub Category
Bloom's is taxonomy sub-class
02 Library and information sciences
Dataset Name
Source dataset
AreMMLU
Global Index
Unique instance
AreMMLUU
Context
Supporting passage (if any)
Sasha decided to watch TV and get some food
Question
Input question
What is the capital of France?
Options
Multiple-choice options
['10", "20", "30", "40"]
Answer
Ground truth answer
Paris
Bloom Level
Bloom's laxonomy difficulty level
Understanding
Table 4 shows the detailed dataset columns.
C.3 DATASET EXAMPLES
Figure I1 shows some dataset examples.
15

---

Published as a conference paper at ICLR 2026
Table 5: The 42 models used for empirical difficulty labeling. Models span across range of sizes and performances, showcasing that ROUTERARENA could distinguish LLMss by providing diverse questions of difficulty.
Rank
Model
Score [%]
gpt-5
74.04
2
claude-3-5-sommet
67.73
deepssek/ddeppseekcchtt
66.86
4
gpt-4o
66.35
-lama---1-405--instuutt
65.43
6
glm-4-plus
62.02
gpt-4-1106-preveew
61.91
8
meta-llama
1err00--IIttttt-tuuuue Turbo
60.64
9
60.51
10
Qwen//Qwen2.55722--1nssre
59.98
11
59.89
12
Qwen/QwQ-32B
59.81
13
gpt-4o-mini
59.22
14
meta-llama Meta-MMeeMMaaaaaaa Llama-a--maa- Instructt
58.16
15
google/gemini - 1lash-1.5
57.21
glm-4-air
54.63
17
Qwen/Qwen 1-t-00eenll--7------ -5-72B-Char
49.86
Qwenn(wen2..5-7-Innne
47.70
19
glm-4-flash
47.62
mistrala/Miixtaal-k------nn
47.01
21
01-ai/Yi-34B-Chr
45.54
llama-3-1-Sb-intrruc
45.23
23
45.01
44.24
25 mctillaamalllama2277-bhhhhha
39.82
26 metw/llma--270b-cchal
39.49
mistrall/AVinsttaalBBBBnssrtttt
32.87
28
llama-3-233--isstruct
32.55
29
32.35
30
Qwen/QWeen2.5332B
32.26
misttalai/Miscrral/l---nnsuuue
31.83
32
meta-llamaalllaaa-----iinss
30.45
29.98
34
masrrlaiANissaal//Bnnnnntttt--.
28.05
35
23.69
23.03
37
Ilama-3---Ib-instruuc
22.10
38
Metta-llnnaLLLama2----bhhaaat
20.50
39
metal/ooeellmma-34--nnnntt
20.42
40
19.31
41
meta-llama Meta-Llama-2-77
18.92
42
meta-llama.Meta-llanaee
18.39
16

---

Published as a conference paper at ICLR 2026
Domain: Computer science, information, and general works Category: 02 Library and information sciences
Global Index: ArcMMLU. _1700
Context:
Question: "In Which five year plan the INFLIBNET' was established()"" Options: "Fourth five year plan", "Fifth five year plan", "Sixth five year, "Seventh five plan")
Answer: D
Bloom, level: remeber
Domain: Language
Category: Language
Dataset name: SuperGLUE-
Global Index: SwweereLLLE-aaaseeeeeeeeee
Context: Piolenn broke the nation."
Question: "what's the effect this?"
Options: [ ("Many citizens relocated to the capitol.", "Many citizens took refuge other territories."
Answer: 1.0
Bloom_ level: analyze
Figure 11: Dataset examples.
C.4 MODEL LIST FOR EMPIRICAL DIFFICULTY LABELING
We use the following 42 models to compute the empirical difficulty labels shown in Table These models are drawn from the model pools of the routers we benchmark, and were chosen to cover broad range of parameter scales, architectures, and providers. This diversity ensures that queries elicit wide spread of accuracies across models, which we indeed observe in their scores and use as the basis for our empirical difficulty measure.
17

---

Published as a conference paper at ICLR 2026
D LLM PROMPTS
D.1 BLOOM TAXONOMY PROMPT
We used the following prompt for classifying each question into different Bloom Levels:
Instruction: You evaluator tasked with classifying questions by cognitive difficulty using Bloom's Taxonomy (Revised 2001). Bloom's Taxonomy defines six levels cognitive processes:
1. Remember Recall or recognize facts, terms, or concepts. Understand Explain, summarize, interpret, or demonstrate comprehension. 3. Apply Use learned knowledge to solve problems in new or routine situations. 4. Analyze Break down information, examine parts, relationships, underlying causes. 5. Evaluate Make judgments decisions based on evidence, criteria, standards, Create Put elements together to form new structure, idea, or product.
Your Task: Given the details question, determine which Bloom's 's level represents the cognitive process required to answer it.
Output the result in structured JSON block with no additional text. If multiple levels might apply, choose the highest level required.
Output Format:
levell; "<Remenber
Understand
|
Apply
|
Analyze
Rvaluate | Create>"
Input Fields:
Dataset name: (dataset_nane nane)
Dataset classification (DDC category): (dataset_category)
Question: :question]
Question Index: (question_index)
Context (if any): : (context)
Options (if MCQ): (options)
Answer: (answer)
18

---

Published as a conference paper at ICLR 2026
Table 6: Router performance by difficulty level (accuracy and cost per 1k queries).
Router
Overall
Easy
Medium
Hard
GPT-5
74.0% ($14.02)
95.1% (S5.68)
68.6% ($14.80)
27.5% ($35.73)
Azure-Router
68.1% (S0.54)
93.3% (S0.30)
59.5% (S0.63)
17.9% (S1.05)
NotDiamond
68.0% (S9.34)
92.5% (S6.34)
61.9% ($10.29)
15.1% ($15.16)
LLLM-SR
67.3% (S1.67)
95.3% (S1.56)
57.9% ($1.70)
8.7% ($1.87)
CARROT
67.2% (S2.06)
95.0% (S1.79)
58.0% ($2.26)
9.0% ($2.36)
MIRT-BERT BERT
66.9% (S0.15)
95.9% (S0.10)
56.8% ($0.17)
7.1% ($0.26)
NIRT-BERT
62.0% (S0.69)
93.7% (S0.53)
47.0% ($0.78)
4.7% ($0,90)
MLP
61.6% (S4.83)
93.2% (S3.54)
46.6% (S5.74)
4.6% ($6.52)
KNN
58.7% (S4.27)
89.2% (83.08)
43.6% (85.13)
4.5% ($5.77)
GraphRouter
57.0% (S0.34)
89.7% (S0.25)
39.3% (S0.39)
2.4% ($0.46)
RouteLLM
47.0% (S0.27)
79.8% (S0.20)
24.4% (S0.33)
2.0% ($0.32)
RouterDC
32.0% (S0.07)
54.9% (S0.06)
15.3% ($0.08)
2.2% ($0.09)
Table 7: RouterArena routers nouters-long-comteeee long-context performance and cost. RouteLLM cannot be evaluated on this subset because its text encoder only supports inputs up to 8,192 tokens.
Router
Acc (%)
Cost 1K Queries (USD)
GPT-5
71
45.70
NotDiamond
70
59.30
Azure-Router
67
9.54
NIRT-BERT
60
6.60
RouteLLM
GraphRouter
42
23.60
vLLM-SR
42
64.80
CARROT
25
4.10
RouterDC
12
3.60
MIRT-BERT
60
7.50
MLP
65
286.70
KNN
69
300.30
D.2 EVALUATION PROMPT
The following example of an evaluation prompt
Please read the following multiple-choice choice questions and
consider which option best follows deontological ethics. An Context: (Context] \n
Question: (Question) \nin
Options: An(Options)/mn AnAn Provide the correct letter choice Wboxed (xx)), (X)), where is the correct letter choice.
Keep the explanation or feedback within 3 sentences.
E ADDITIONAL RESULTS AND TABLES:
Long-Context Routing, Table reports router performance the LongBench-v2 dataset. We randomly sampled 100 queries of short and medium length from the dataset. GPT-5 attains the highest accuracy (71 %) but also the highest cost per 1 queries. Azure-Router, MIRT-BERT, and NIRT-BERT achieve reasonably strong accuracy (60-67%) at one order of magnitude lower cost, whereas methods like MLP, KNN, NotDiamond, and vLLM-SR are substantially more expensive for comparable or worse accuracy. RouteLLM cannot be evaluated in this setting because its text encoder only supports inputs up 8,192 tokens, which excludes many of our long-context queries.
Generation Length and Cost. Table summarizes average generation length and cost per million output tokens. GPT-5 produces the longest responses and is also the most expensive per token,
19

---

Published as a conference paper at ICLR 2026
Table 8: RouterArena routers average generation length and cost.
-verrae
Router
Avg. Gen. Length
Cost M Tokens (USD)
GPT-5
970.65
10.00
NotDiamond
390.05
9.18
Azure-Router
270.90
1.79
NIRT-BERT
215,82
1.33
RouteLLM
140.24
0.68
GraphRouter
132.24
0.81
VLLM-SR
109.27
9.03
CARROT
105.56
12.58
RouterDC
102,83
0.20
MIRT-BERT
89.16
1.00
MLP
81.06
29.75
KNN
79.80
26.31
Router Radar (t Arena Score
CARROT
RouterDC
GraphRouter
MIRT-BERT
Robustness Score (1)
NIRT-BERT
Cost-ratio Score
b. 06 1.0
Latency Score (1)
Optimal-acc Score
Figure 12: The spider plot of ROUTER
while Router NIRT-BERT, and MIRT-BERT generate shorter outputs at much lower cost. In contrast, heuristic baselines such MIP, KNN, and CARROT have relatively short generations yet very high token-level costs, indicating that they are inefficient choices when generation cost is primary concern.
F LEADERBOARD
Figure presents the spider plot of RouterArena, which compares six routing methods (CARROT, RouterDC, GraphRouter, MIRT-BERT, NIRT-BERT, and RouteLLM) across five evaluation dimen- Arena Score. Cost-ratio Score, Optimal-acc Score, Latency Score, and Robustness Score. Each axis indicates higher performance in the outward direction, allowing direct visualization of trade-offs. For example, CARROT achieves strong performance in Arena and Latency Scores, while RouterDC excels in Cost-ratio Score.
sions:
G ROBUSTNESS EVALUATION
Specifically, we assign equal weight to the four noisy variants described in the definition. For each category and subcategories, we randomly sampled of the entire dataset, which 420 exam- ples and generated perturbed queries using GPT-4o. The exact prompt we used to generate perturbed queries as follows:
20

---

Published as a conference paper at ICLR 2026
You are an assistant that performs extremely strong and highly disruptive text transformations for robustness stress testing. You must follow the selected mode exactly. Transformations must be intense, obvious, and aggressive. Always preserve the requested structure, but you may heavily distort surface wording. Do NOT shorten or expand the text beyond the allowed limits.
There are four transformation modes:
PARAPHRASE (MAXIMAL TRANSFORMATION MODE) Rewrite the text so heavily that almost no original sentences or phrasing remain. You may reorder ideas, split or merge sen- tences, and aggressively substitute expressions. The meaning must stay intact, but the wording should look almost unrecognizable from the imput. Keep length approximately the same (#15
GRAMMATICAL RECONSTRUCTION (DEEP REWRITE MODE) Rebuild the text with fully restructured, polished, and elegant grammar. You may reorder clauses, shift emphasis, modernize tone, and transform the writing style completely. All information must remain, but the writing should feel dramatically improved and profoundly different from the original. Keep the length the same (=10
SYNONYM SATURATION (ULTRA-DENSE SUBSTITUTION MODE) Replace nearly ev- ery content word (nouns, verbs, adjectives, adverbs) with strong, less common, surprising synonyms. You may alter idioms, swap multiword phrases with equivalent expressions, and push semantic boundaries while preserving meaning. The output should visually look almost entirely changed. Maintain length (+10
INTENTIONAL CORRUPTION (HEAVY DEGRADATION MODE) Inject very high den- sity of diverse, realistic, and chaotic textual corruptions: misspellings, keyboard slips, letter swaps, missing characters, inserted random characters, spacing glitches, partial word breakage, and inconsistent capitalization. Meaning must remain barely decipherable but still recoverable. Maintain approximate length (+10
Input format:
MODE: <Paraphrase Grammar Synonyms Typos>
TEXT: cuser.provided_text
Output format (strict):
= TRANSFORMED TEXT == <extremely modified text>
NOTES <1-2 very short descriptions of the corruption intensity and strategy> Do not output anything else.
H DISCUSSION ABOUT HYPERPARAMETER (Cmin' Cmaas B)
In our framework, Cmar and Cmrn are not intended to static global constants; instead, they are tunable for flexibility. To prevent the most expensive router from being normalized zero, for now, we set Cmaz and Cmin to the prices of the most expensive and cheapest models on the market. However, whenever model's actual cost really falls below Cmin exceeds Craz. the only thing need to do is to adjust the Criin and Cmaas and re-calculate the Arena Score so that all normalized costs remain within (0, Note that this does require re-running the experiment.
The parameter 3 is also adjustable to modulate how much weight is assigned to cost compared to accuracy. Specifically, we made the tunable on our leaderboard for users. Here give three example settings:
Example 1: 3 = 0.01, Accuracy: Cost 100, accuracy-dominated ranking users who do not consider cost to be key constraint.
Example 2: 8 = 0.1, Accuracy Cost = 10, trade-off balanced ranking for users who primarily care about accuracy but still operate under budget constraints.
21

---

Published as a conference paper at ICLR 2026
Example 3:
Accuracy Cost equally weighted ranking for users who have
limited budget.
Router
Rank & Score @ B=0.01
Rank & Score @ B=001
Rank & Sore [=1
Accuracy
Norm.Cost
GPT-5
(0.7282)
(0.6272)
(0.3688)
0.7428
0.2453
Azure-Router
(0.6780)
(0.6640)
(0. (0.6010)
0.6798
0.5386
MIRT-BERT
(0.6731)
1 (0.6729)
(0.6718)
0.6731
0.6705
CARROT
(0.6682)
(0.6386)
(0.5229)
0.6720
0.4266
VLLM-SR
(0.6633)
(0.6385)
(0. (0.5368)
0.6665
0.4494
NotDiamond
(0.6565)
8 (0.5935)
11 (0.3998)
0.6651
0.2858
RouteLLM
(0.6175)
(0.5800)
(0.4440)
0.6224
0.3451
NIRT-BERT
(0.6151)
(0.6083)
(0.5764)
0.6159
0.5416
MLP
9 (0.6143)
10 (0.5780)
9 (0.4449)
0.6191
0.3472
outer
10 (0.6070)
(0.6054)
(0.5976)
0.6072
0.5884
KNN
11 (0.5867)
11 (0.5577)
(0 (0.4464)
0.5905
0.3588
RouterDC
12 (0.3362)
12 (0.3522)
7 (0.4627)
0.3344
0.7507
Figure 13: Recommended B value for different user needs.
I PRICE OF MODELS
Table shows the price we used for the self-served model. It was used by Together.ai at the time the experiment was conducted, and the price reflects hardware requirement for the increasing model size.
Table Dense and MoE large language model size and price per million tokens.
Dense Models
Mfxture-of Models
Model Size
Price
Model Size
Price
Up to 4B
$0.10
Up to 56B
$0.60
4.1B 8B
S0.20
56.1B 176B
$1.20
8.1B 21B
S0.30
176. 480B
$2.40
21.1B 41B
$0.80
41.1B - 80B
$0.90
80.1B 110B
$1.80
22


Reviews:
[{"review_id": "OLfcvgNCZH", "reviewer": "Reviewer_T3An", "summary": "The paper tackles a timely problem—standardized router evaluation—with a clear systemization (dataset + metrics + framework) and substantive empirical coverage (open-source and commercial).", "questions": "I think this is a very interesting topic. Specifically, I have the following two questions:\n\nQ1. Your evaluation reports per-query accuracy as a binary outcome and aggregates it—without explicit weighting by Bloom difficulty levels—into the composite Arena score $S_{i,\\beta}$. \n-  Is “completion” strictly 0 or 1 correctness per query? If so, why not support rubric-based partial credit for partially solved answers (e.g., correct plan but incomplete final step)?\n- Some longer (higher-token) first-round answers can enable success in later turns. Do you plan a multi-turn setting that measures cross-round utility versus added first-round cost (e.g., a “round-2 success gain” vs. round-1 token spend)?\n\nQ2. Among answers that are equally correct, a longer response may offer clearer reasoning, citations, or actionable steps, potentially increasing user satisfaction even at a higher token cost. Do you collect any user-satisfaction / explanation-quality signal (human Likert ratings or a calibrated LLM-as-judge) in addition to accuracy/cost?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. Well-scoped problem & gap analysis.\n\nS2. Good dataset, metric, and evaluation design.\n\nS3. Comprehensive experiments and visualizations.", "weaknesses": "W1. The $\\log 2$ cost normalization with fixed ( $c_{min}⁡ = 0.0044$, $c_{max} ⁡ = 200$ ) and $\\beta=0.1$ may bias rankings toward certain price bands; no sensitivity analysis shown in Sec. 5.\n\nW2. LLM-as-judge labeling (DeepSeek-V3.1) lacks human validation studies or inter-rater checks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zPPg5WrziO", "reviewer": "Reviewer_LQ8R", "summary": "This paper introduces \"RouterArena,\" a new open-source platform for benchmarking Large Language Model (LLM) routers. The authors argue that as the LLM ecosystem has produced numerous models with varying costs and capabilities, routers have become essential for selecting the right model for a given query. This, in turn, has created a new need for a standardized way to evaluate the routers themselves.\n\n\nAn automated framework and leaderboard designed to be \"live,\" allowing researchers to submit new routers (both open-source and commercial) for evaluation and comparison.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a practical problem. As model routing becomes a standard component in AI stacks, the need for a robust, standardized benchmark to compare routers is high. This work is well-motivated and highly relevant to the community.\n\n2. Principled Dataset Construction: A major strength of this paper is its novel and well-justified dataset construction.\n\n3. Novelty: An automated, \"live\" platform and leaderboard, distinct from a static dataset. It is designed for continuous benchmarking and community engagement, allowing researchers to submit and compare new open-source and commercial routers.\n\n4. Comprehensive, Multi-Dimensional Evaluation: The paper correctly identifies that router performance is multi-faceted. The inclusion of Routing Optimality is a key metric, as it reframes the goal from just \"being correct\" to \"being correct efficiently.\" Measuring Robustness and Latency further strengthens the benchmark's utility for real-world applications.\n\n5. Interesting Initial Analysis and Insights: The initial evaluation of 12 routers provides valuable insights. The findings—that commercial routers do not necessarily lead, that all routers are inefficient, and that performance is a complex trade-off—are important takeaways for the field.", "weaknesses": "Discussion of Benchmarking Philosophy: The paper positions itself as superior to prior work like RouterBench (Table 1), but it misses an opportunity for a deeper discussion. RouterArena evaluates live routers (a \"hot\" evaluation), whereas RouterBench uses a large, static dataset of pre-computed outcomes for \"offline\" evaluation. This offline approach is significantly cheaper and faster for iterating on router designs, presenting a different benchmarking philosophy. A more nuanced discussion of the pros and cons of these different approaches would improve the paper's contribution.\n\nDataset Scale: While the dataset is principled in its design, its size (~8,400 queries) is a potential limitation. When spread across 44 categories and 3 difficulty levels, some cross-sections may be too small to draw statistically significant conclusions. The paper would be stronger if it addressed this limitation or included an analysis of the sample size per category.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LHAfoFB9LF", "reviewer": "Reviewer_ZQTb", "summary": "This paper introduces RouterArena, which the authors claim to be the first comprehensive open platform for evaluating and comparing large language model (LLM) routers. The platform aims to meet the growing need for systematic router evaluation and provides: (1) a principled dataset consisting of 8,400 queries spanning 9 domains and 44 categories, organized using the Dewey Decimal Classification system and incorporating Bloom’s taxonomy to define difficulty levels; (2) a comprehensive set of evaluation metrics covering accuracy, cost, routing optimality, robustness, and latency; and (3) an automated evaluation framework that supports both open-source and commercial routers. The authors evaluate 12 representative routers and present a multidimensional leaderboard summarizing their overall performance.", "questions": "Q1. Regarding the validation of Bloom’s taxonomy classification—was any human sampling verification conducted, and how consistent were the results with the LLM’s judgments?\nQ2. The robustness definition in Section 4 differs from the implementation described in Section 6.2 (“adding irrelevant keywords”). If only keyword-insertion was used, does this cause inconsistency between the definition and the actual test?\nQ3. Figure 6 looks more like a scatter plot than a curve plot. Would the authors consider renaming it for accuracy?\nQ4. The paper builds a dataset with three difficulty levels, but all evaluations are aggregated. Why not provide router performance broken down by difficulty level?\nQ5. For commercial routers that cannot expose routing decisions, is there a plan to design alternative indicators to enhance ranking reliability?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "S1. The paper addresses a critical gap in the LLM ecosystem, the research motivation is clear and timely.\nS2. The evaluation metrics are comprehensive, with five well-defined dimensions that reflect real-world deployment considerations.\nS3. The automated framework enables dynamic leaderboard updates and supports the evaluation of both open-source and commercial routers.", "weaknesses": "W1. The overall contribution of the paper is limited.\nW2. For commercial routers, internal routing decisions are inaccessible, making many metrics uncomputable and thus limiting a full evaluation.\nW3. The use of DeepSeek-V3.1 for automated difficulty annotation may introduce systematic bias; no quantitative bias analysis or annotation-consistency study is provided.\nW4. The robustness test is limited, and the actual evaluation method appears inconsistent with the definition of robustness given in Section 4.\nW5. The paper spends substantial space describing the construction of a dataset with distinct difficulty levels but does not subsequently analyze results based on these levels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8iXIX7wkBw", "reviewer": "Reviewer_vKx3", "summary": "This paper proposes a benchmark for LLM Routers that evaluates different routing systems along multiple axes—including *accuracy*, *cost*, *latency*, and *routing robustness*. Concretely, the authors curate *8,400* queries from 21 open-source datasets and, via an LLM-as-Judge procedure using DeepSeek-V3.1, assign each query to one of three difficulty levels: easy, medium, or hard. On the collected dataset, the paper conducts objective evaluations of both open-source routers (e.g., KNN- and MLP-based methods) and commercial routers (e.g., GPT-5, Azure Router).", "questions": "See above weaknesses", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "1. It is necessary and practically valuable to objectively evaluate LLM routers, as this helps users who are unfamiliar with model details select configurations that best fit their needs.\n2. The benchmark combines 5 evaluation perspectives (including cost, accuracy, robustness, latency, and routing optimality). In addition, it supports commercial routers, making the evaluation more complete than prior work.", "weaknesses": "### Weaknesses\n\n1. Query selection bias. The query set consists primarily of objective questions and excludes creative/open-ended tasks. For objective questions, users often prioritize accuracy over cost; however, for open-ended or batch-processing tasks (e.g., large-scale data filtering), cost can be the primary concern, leading to different routing preferences. This preference shift can further affect the validity of conclusions drawn in the Experiments section. It is advisable to include open-ended queries and update the results and analyses accordingly.\n\n2. Difficulty grading misalignment with real-world scenarios. Difficulty is determined solely via an LLM-as-Judge setup, which may introduce model-induced bias; human annotation should be incorporated. Moreover, the benchmark should expand to more realistic routing demands, such as *long-context generation*, *tool use*, *code agents*, and *deep research*. In real deployments, achieving good performance often involves *orchestrating multiple models* collaboratively rather than invoking a single model.\n\n3. Limited comparability due to heterogeneous model pools. Different routers operate over different model pools. A router that can access a cheaper yet strong model may rank higher, but this does not necessarily demonstrate a better routing algorithm. Introducing a unified model pool and a corresponding leaderboard would improve comparability.\n\n4. Evaluation dimensions remain limited. Additional aspects—such as *response time* or *response length*—should be considered. For example, if the pool contains a cheaper *reasoning* model and a more expensive *instruction* model, then under the cost formulation\n$ \\text{cost} = C_{\\text{in}} \\cdot N_{\\text{in}} + C_{\\text{out}} \\cdot N_{\\text{out}} $,\ntwo routing strategies might achieve the *same cost* and *same accuracy*, yet differ meaningfully in behavior; how should their relative quality be judged?\n\n### Notation\n\n1. *All displayed formulas lack numbering*, which reduces ease of reference and clarity in discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)