Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning
Authors: Haozhen Zhang
Tao Feng
Jiaxuan You
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
LLM Routers
LLM Selection
Reinforcement Learning
Primary area: deep_learning
TL;DR: We propose Router-R1, an RL-based framework that interleaves multi-round reasoning with dynamic LLM selection, supports zero-shot integration of new models, and optimizes performance-cost trade-offs

Abstract: The rapid emergence of diverse large language models (LLMs) has spurred the development of LLM routers that assign user queries to the most suitable model. However, existing LLM routers typically perform a single-round, one-to-one mapping (\textit{i.e.}, assigning each query to a single model in isolation), which limits their capability to tackle complex tasks that demand the complementary strengths of multiple LLMs. In this paper, we present \textbf{Router-R1}, a reinforcement learning (RL)-based framework that formulates multi-LLM routing and aggregation as a sequential decision process. Router-R1 instantiates the router itself as a capable LLM, leveraging its reasoning ability to interleave "think" actions (internal deliberation) with "route" actions (dynamic model invocation), and integrates each response into its evolving context. To facilitate learning, we employ a lightweight rule-based reward comprising format rewards, final outcome rewards, and a novel cost reward for optimizing the balance between performance and cost, opening a pathway toward enhancing performance-cost trade-offs via RL. Router-R1 also conditions only on simple model descriptors such as pricing, latency, and example performance, enabling strong generalization to unseen model selection. Experiments on seven general and multi-hop QA benchmarks show that Router-R1 outperforms several strong baselines, achieving superior performance while maintaining robust generalization and cost management.

Router-R1: Teaching LLMs Multi-Round Routing and A Aggregation via Reinforcement Learning
Haozhen Zhang Tao Feng Jiaxuan You
University of Illinois at Kibana-Champar
(haozhenz, taofenn22 jiaxuan] 0illinois. edu, 'wazhz146gmail. com
Abstract
The rapid emergence of diverse large language models (LLMs) has spurred the development of LLM routers that assign user queries to the most suitable model. However, existing LLM routers typically perform a single-round, one-to-one map- ping (i.e., assigning each query to single model in isolation), which limits their capability to tackle complex tasks that demand the complementary strengths multiple LLMs. In this paper, present Router-R1, a reinforcement learning (RL)-based framework that formulates Multi-LLM routing and aggregation as a se- quential decision process. Router-RI R1 instantiates the router itself as a capable LLM, leveraging its reasoning ability to interleave "think" actions (internal deliberation) with "route" actions (dynamic model invocation), and integrates each response into its evolving context. To facilitate learning, we employ a lightweight rule-based reward comprising format rewards, final outcome rewards, and novel cost reward for optimizing the balance between performance and cost, opening a pathway toward enhancing performance-cost trade-off's via RL. Router-R1 also conditions only on simple model descriptors such as pricing, latency, and example perfor- mance, enabling strong generalization unseen model selection. Experiments seven general and multi-hop QA benchmarks show that Router-RI outperforms several strong baselines, achieving superior performance while maintaining robust generalization and cost management.
ulab-uiuc/Routeer-BB
Hugging Face Collection
Introduction
Large language models (LLMs) have proliferated at unprecedented pace, with new architectures and fine-tuned variants released on a monthly or even weekly basis [21, 1J. To harness the comple- mentary strengths of multiple LLMs (e.g., one model's fluency versus another's factual accuracy), LLM routers have emerged as a critical infrastructure component, dynamically dispatching user queries to single selected model to maximize answer quality and efficiency [3, 6. 2, 5, 26. 30, 4]. While this one-shot routing strategy can improve average performance, it overlooks the fact that truly complex reasoning tasks often require coordinated interactions among multiple models, orchestrating not just a single choice but a sequence of model calls to leverage their complementary strengths. This observation raises key challenge: how routing and aggregation process to jointly solve complex tasks?
coordinate multiple LLMs in multi-round
Answering this challenge is non-trivial. First, the discrete decision process of selecting which LLM to call at each round is inherently non-differentiable, precluding straightforward end-to-end training via backpropagation. Although prior work has applied gradient-basse based methods single-shot routing [3, 6, 2, 5]. extending them to multi-round selection and aggregation rapidly becomes intractable.
WWrk done intern University of Illinois at Sabaana-Champaige
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Second, existing routers operate in single-step regime: given query, pick one model, collect its output, and stop [3. 6]. However, complex tasks (e.g. multi-hop question answering) demand sequence of interleaved reasoning and model-selection decisions, and a one-off choice rarely suffices.
Therefore, we must design an interplay mechanism that alternates between "long thought" reasoning (internal deliberation) and targeted LLM selection to refine an answer iteratively.
To address these challenges, we introduce Router-R1, a reinforcement learning- based framework for multi-round LLM routing and aggregation. Rather than making a single dispatch decision, we formulate LLM coordination as a sequential decision-making problem. At each step, Router-R1 chooses whether to perform internal reasoning ("think"") or to invoke a specific model from a pool of available LLMs ("route"), gradually constructing an answer through iterative interaction. To support this interplay between reasoning and routing, we instantiate the router itself as capable LLMM, leveraging its inherent reasoning capability to perform long-form deliberation and targeted model selection. This flexible reasoning and model selection interleaving enables Router-RI 1 to adaptively compose the strengths of multiple L.MMs in a task-aware manner. optimize this decision policy, we adopt reinforcement learning and design simple yet effective rule-based reward function composed of three parts: a format reward for producing well-structured outputs, a final outcome reward based on task correctness, and a cost reward that penalizes excessive use of expensive routed models, providing Router-R. R1 the capability to navigate performance-cove cost trade-offs during training. Additionally, Router-RI exhibits strong generalization capability to newly added LLM candidates without the need for retraining by conditioning its routing decisions on simple descriptors such as pricing, latency, and example performance. Together, these components make Router-RI robust and flexible solution for coordinating multiple L.MS to solve complex reasoning tasks. Through comprehensive experiments seven diverse QA benchmarks, covering both general and multi-hop question answering, we demonstrate that Router-R1 consistently outperforms several strong baselines and achieves state-of-the-at performance. Further analyses on cost-aware routing and generalization to unseen LLMs also highlight the flexibility and robustness of our approach.
Our contributions are summarized as follows:
We propose Router-R1, a reinforcement learning-based framework for multi-round LLM routing and aggregation. By instantiating the router itself capable LLM, Router-R1 naturally interlcaves internal reasoning and external model selection, enabling adaptive coordination across multiple LLMs for complex task solving.
We design a simple and effective rule-based reward function consisting of format rewards, final outcome rewards, and cost rewards, enabling Router-R1 1 to navigate performance-coe trade-offs. Additionally, by conditioning on simple model descriptors, Router-R1 can generalize to unseen LLMs without retraining.
Through extensive experiments on seven question-answering benchmarks, we show that Router-RI outperforms several competitive baselines, achieving superior performance and robust generalization.
2 Related Work
2.1 Query-based Routers for LLM Selection
The rapid rise of various LLMs has spurred the development of query-based LLM routers, which aim to direct queries the most appropriate model to improve response quality and efficiency. HybridI.LM, LL [5] proposes a dynamic router that selects between small and large LLMs based on predicted query difficulty and a user-defined quality budget. GraphRouter [6] frames LLM selection as inductive edge prediction over beck-query-modeee graph, enabling cost- cost-performance estimation and effortless integration of new LLMs. To explicitly balance performance and cost, FrugaIGPT [2] adopts an LLM cascade approach, while FORC [26] routes queries to appropriately sized models for cost-effective inference. TO-Router [30] unifies multiple comminn-specifi expert LLMS under one interface and dispatches queries based on task needs. C2MAB-V V [4] leverages cost-aware combinatorial multi-armed bandit to dynamically select optimal LLM subsets. RouterDC [3] enhances routing via dual contrastive learning between queries and LLM embeddings. Finally, Routel.L [22] utilizes human preference data to dynamically choose between strong and weak L.M,s, effectively reducing costs while maintaining quality.
2

---

In contrast to previous methods, Router-RI treats routing as sequential decision process, interleaving internal "think" steps with multi-round model routing to refine its answer by instantiating the router itself as capable I.M. Moreover, its RL-baased training harnesses a cost reward to balance performance and cost, enabling flexible performance-cov trade-offs and resource-aware routing.
2.2 Optimizing LLM Behaviors via Reinforcement Learning
In recent years, reinforcement learning (RL) has emerged as a powerful paradigm for fine-tuning large language models (LLMs) better align with human preferences. Early work like RLHF (Reinforcement Learning from Human Feedback) [23], trains a reward model on human judgments and then applies poliey-optimizztion algorithms such as PPO [27] steer the base language model toward more desirable outputs. Building on this foundation, RLAIF [17] shows comparable or better performance on tasks like summarization and dialogue, and its direct variant (d-RLAIF) removes the need for an explicit reward model, improving efficiency. RRHF [37] ranks model-generatee external responses train preference without reward predictor, achieving results on par with RLHF More recently, Direct Preference Optimization (DPO) series methods [25, 20] take further step by training directly on human preference data, avoiding RL sampling and complex tuning while matching or outperforming RLHF. Alongside these advances in reward-driven fine-tuning, other RL-based techniques such as Search-RI [13] have empowered LLMs to adaptively interact with external tools like search engines, allowing them to dynamically retrieve and incorporate external information during inference. These approaches highlight the potential of RI. in optimizing L.LM behaviors beyond static prompt-based generation, particularly in tasks requiring real-time information access and decision-making.
and
Router-R1
In this section, we detail our proposed Router-R1 into three parts. Section 3.1 introduces the reinforcement learning formulation with an L.M routing pool. Section 3.2 presents the reward curation strategy, which includes format rewards, final outcome rewards, and cost rewards. Section 3.3 describes the multi-round interaction training process, including training prompt template and multi- round interaction with an LLM routing pool. We show the architecture of Router-RI in Figure 1.
3.1 Reinforcement Learning via Coordination with a LLM Routing Pool
In Router-R1, adopt a general policy optimization objective with an LLM routing pool P formulated as:
TV x; P)
max Be y (a:P) ro(2,y) y) 3 log
(I)
Trerl x; P)
where π denotes the policy LLM to be optimized, and nee is reference LLM that may be fixed or updated iteratively for stable training. x represents input samples from the dataset D, and y denotes the generated outputs sampled from the policy Tee(y P;;P), which interleaved with results obtained from accessing the LLM routing pool P. (x, is the reward function, and P is the LLM routing pool, which provides a set of candidate LLMs available for selection. The KL regularization term ensures that the updated policy remains close to the reference, with the regularization coefficient controlling this trade-off. This formulation is general and encompasses various regularized reinforcement learning algorithms such as PPO [27]. GRPO [28], and KL-constrained methods, allowing for flexible policy updates over a pool of LLM candidates.
Moreover, under this optimization objective, the policy LLM can dynamically select a candidate LLM from the LLM routing pool P multiple rounds, and obtain auxiliary information about the input sample by providing the candidate [.LM with relevant context, thereby enhancing the reasoning process of the policy LLM. In this case, the policy LLM can be regarded as coordinator, selecting and coordinating multiple candidate LLMs to jointly solve complex tasks.
3.2 Reward Curation
To provide Router-R1 with reasonable and effective supervision signals, we carefully design the reward function, including format rewards, final outcome rewards, and cost rewards, which we will describe in detail below.
3

---

(a)
Single -rugee Routing
Query
Query
Single-round
Router
Selected
Answer
LLM Routing Pool
(b) Multi-round Routing
Sub-query
Route
Selected LLM #1
Information
Query
Router-R1 (Policy LLM)
Interleave
O
Sub-query IN
Selected
Reason
#N
Figure I: Router-RI architecture. (a) Single-round Routing: A conventional router assigns each query single LLM in isolation via a one-shot decision. without internal reasoning or multi-model coordination. (b) Multi-round Routing (ours): Router-RI casts multi-LLM routing as a sequential decision process, which leverages an LLM-based router to interleave internal reasoning with external LLM routing and integrates retrieved information into its evolving context. This enables adaptive multi-model coordination for complex tasks, surpassing single-round routing with better performance.
3.2.1 Format Rewards
Inspired by [8], to stabilize the training and ensure that the LLM response rollouts conform the predefined format (detailed helow in Section 3.3), we impose strict format validation on Router-R1 Specifically, format reward assigned according to the following rule: if the response rollouts do not satisfy the required format, the format reward is set to -1; otherwise, it is set to 0:
(-1,
if the format is incorrect
Rermat
(2)
0,
if the format is correct
3.2.2 Final Outcome Rewards
In Router-RI, we adopt Exact Match (EM) to measure the correctness between the answer predicted by LLM and the ground truth, and utilize the only final outcome reward to guide the optimization of Router-R1:
Routome
EM yas 9t),,
where ya is the predicted answer extracted from the generated output y and gr denotes the ground truth. EM emphasizes the complete and strict matching of the predicted answer with the golden answer, which has been proven by many works be a concise and effective rule-based reward [8, 13].
3.2.3 Cost Rewards
To balance the additional cost of invoking candidate LLM from the routing pool, introduce the computational cost incurred by querying a candidate LLM as a cost reward. This design enables Router-RI to potentially optimize only for model performance but also for the trade-off between performance and computational efficiency.
Formally, the cost reward is inversely proportional to both the number of output tokens produced by the candidate LLM and the model-dependent cost-per-token function, which reflects the computational price of using different models:
Resst x - -m( PIIM) Toat
where PLLM denotes the number of parameters of the selected candidate LLM and Tout is the number of output tokens generated by it. mí-) is predefined cost function that maps model size to its per-token computational cost (e.g., based on pricing tiers of the LLM API service). Note that the cost reward will be normalized between and during training.
In this way, the larger the model size, the more output tokens, and the smaller the cost reward, which provides Router-RI R1 with the capability to achieve performance-cost balance.

---

3.2.4 Overall Rewards
To sum up, the overall reward function of Router-RI can be defined as:
( y) Rormat + (1 a) Rautome + a
(5)
where a serves as a hyperparameter controlling the balance between model performance and cost.
In particular, to mitigate reward hacking [29] and improve optimization stability, we introduce a hierarchical reward refinement to the overall reward function (not shown in Formula 5 for brevity). In Router-RI, , the three reward components are assigned different priorities, decreasing from left to right in Formula 5. Concretely, if the format reward -1, the remaining two rewards nullified (set to zero), regardless of their original values. This hierarchical design enforces critical constraints before optimizing for performance computational efficiency, contributing the stable and reliable training of Router-RI.
In our experiments, such clear rule-based reward combination is sufficient to optimize Router-R1 well, which also demonstrates the rationality and effectiveness of our reward function design.
3.3 Multi-Round Interaction Training Paradigm
In this section, we describe the training prompt template of Router-RI and the multi-round interaction with the LLM routing pool.
3.3.1 Training Prompt Template
Inspired by [8, 13], we construct the training prompt template for Router-RI as shown in Figure 2. To ensure accurate and well jell-justified responses, adopt structured prompt combining internal rea- soning and selective external querying. Upon receiving question, Router-R1 first performs internal analysis within <think> and </think> block assess whether additional information is required. If so, queries a suitable specialized LLM via <search> Candidate LLM: Query </search> based on predefined LLM routing pool and model descriptions that include details such parameter size and task specialization (we detail it in Appendix C. ). Retrieved information is returned within <info> and </info> tags, and the process may iterate to gather complementary insights. The final answer is output within an <answer> and </answer> block.
It worth noting that the model descriptions here serve only LLM. During the policy optimization process, the policy LLM adaptively learns the strengths and weaknesses of each candidate L.M through interaction and feedback. To further enhance adaptability, the prompt design also supports the seamless integration of new candidate L.MMs without requiring retraining. Specifically, Router-RI can achieve this generalization capability by incorporating the descriptions of newly added LLMs directly into the prompt. This flexibility allows Router-RI to dynamically expand its routing pool and effectively accommodate the rapid and ongoing emergence of new LLMs.
initial prior for each candidate
Answer the given question. Every time you receive new information, you must first conduct reasoning inside <think> and </think?.
After reasoning, if you find you lack some knowledge, you can call specialized LLM by writing a query inside <search> Candidate LLM: Query </search> Before each LLM call, you must explicitly reason inside <think> and </think> about "why external information needed" and "which LLM from the list is most suitable for answering your query," based on the brief model descriptions provided below.
When you call an LLM,, the response will be returned between <info> and </info>, You are encouraged to explore and utilize different LLMS multiple times to better understand their respective strengths and weaknesses, as well as gather more comprehensive information.
Description of LLM Candidates: (candidates intro) If you find that no further external knowledge is needed, you can directly provide your final answer inside <answer> and </answer>, without additional explanation illustration. Question: (question)
Figure 2: Training prompt template for Router-RI (some texts are omitted for page space).
5

---

3.3.2 Multi-Round Interaction with a LLM Routing Pool
According to the prompt in Figure 2, Router-R1 first analyzes the input question to identify the necessary information and selects the most appropriate candidate LLM from a routing pool to query via sub-question. During training, it learns to decompose complex queries and route them adaptively based on the strengths of different LLMs.
The routing process is triggered whenever a special <search> tag appears in the generated sequence, specifying the target LLM and intended sub-query. Upon detection, Router-R1 queries the designated LLM and inserts its response back into the sequence to continue reasoning (to ensure stable training. external responses marked by <info> tags are excluded from the loss computation). For complex tasks, Router-R1 I can perform multi-round routing, iteratively integrating information from multiple sources arrive at a final answer. For simple questions, Router-R1 can rely solely on the policy LLM's internal knowledge to produce answers, demonstrating its ability to judge whether external information needed.
Experimental Setup
4.1 Datasets and Metrics
We evaluate Router-R1 on seven question-ann question-answering answering (QA) datasets, i.e., (1) General QA: Natural
Question (NQ) [16]. TriviaQA [14], POpQA [19]; (2) Multi-Hop QA: HotpotQA (HpQA) [36], 2WikiMuliHHoPAA (2wiki) [9], Musique [32]. and Bamboogle (Bamb) [24]. These datasets cover both single-hop and multi-hop QA benchmarks, providing comprehensive testbed for evaluating the performance of Router-R1. For metries, the correctness of the predictions generated by Router-R1 evaluated using the Exact Match (EM) and FI-Score (F1) against the ground truth.
4.2 Baselines
To have a comprehensive evaluation, we compare our proposed Router-RI with dozens of baselines:
Basic Baselines. (1) Direct Inference (Direct), (2) Chain-of- Thouugh (CoT) Prompting [34], (3) supervised fine-tuning (SFT), (4) Retrieval-Augmented Generation, which utilizes Wikipedia-18 [15] as the external knowledge and E5 [33] as the retriever (RAG), (5) Search- RI [13].
Query-based LLM Routers. These baselines follow a similar setup: one or more candidate LLMS are selected from the routing pool to answer the input question (or its sub-questions, unless specified), and their responses are integrated into the base model's input to generate the final answer, simulating Router-RI I's response generation process for fair comparison. (6) Prompt the base LLM model to select candidate LLM (Prompt LLM), (7) always select the largest LLM (Largest LLM). (8) KNN Router [10], (9) MLP Router [10], (10) BERT Router [22], (11) RouterDC [3], (12) GraphRouter [6]. (13) prompt the base LLM to decompose the original question into sub-queries and assign each to a candidate
LLM (Prompt LLM"), (14) prompt the base LLM to decompose the original question into
sub-queries and utilize KNN Router to assign each to a candidate LLM (KNN Router*).
To train query-based LLM routers, we additionally construct dedicated router training dataset. Specifically, each training question is independently fed to every model in the LLM routing pool multiple times with temperature sampling. The responses are then evaluated using the EM metric to assess answer quality. This process yielded, for each question, a set of EM scores corresponding to all candidate LLMs in the routing pool, effectively labeling each LLM with its performance on that question. These question-LLM score pairs form the supervision signal for training the router.
4.3 Implementation Details
We conduct our experiments using Swwen.5--B-Inntteee [35] and LLaMA-3.2-3B-Innst [7] as base models for training. To empower LLMs with multi-round routing capabilities, we design Router-RI with maximum of 4 routing steps per input query. The model is trained using veRL for
http://gghhub.com//vdeeggggeeeet

---

reinforcement learning in LLMs, employing the Proximal Policy Optimization (PPO) as the default algorithm. The batch size is set to 64, with a maximum of 225 training steps. The cost coefficient is set to 0.0 in our main experiment unless otherwise specified.
To incentivize both single-round and multi-round routing capabilities during training, we construct joint dataset consisting of 7K samples each from the NQ and HotpoiQA datasets, respectively. This results in a 14K sample training set, which we find sufficient to induce effective routing strategies without requiring extensive data filtering or complex sampling procedures. As demonstrated in our experimental analysis in Section 5, this modestly sized dataset enables robust routing and aggregation behavior learning.
After training, we evaluate in-domain performance NQ and HotpotQA datasets, where Router-RI has seen similar data during training, and assess out-of-domain generalization performance across five other QA datasets mentioned above. For each evaluation dataset, randomly sample 500 test instances (except for Bamboogle, which contains only around 120 test examples in total). All baseline models are trained (if' needed) and evaluated under the consistent dataset and settings to ensure fair comparison. To support routing strategies, we adopt a diverse LLM routing pool comprising six representative models of varying sizes and families: Qwen2.5-7BBIIstrruc [35], 1-M-A-----8B-Inttruee [7], -..MMA-3.1-70B-mmm
[12), Mixtral- [31]. The base model training is conducted on
[7], Mistral-7B-Instreee
8x22R-It [11], and
NVIDIA A6000 GPUs, while routing LLMs are accessed via NVIDIA NIM APIs'. More details
about experimental details and prompts are described in Appendix A and C. We also provide case
study of Router-R1 in Appendix B.
5 Experimental Analysis
In this section, we present comprehensive empirical studies to evaluate the effectiveness of Router-Re R1, We analyze its performance across diverse QA benchmarks (Section 5.1), explore its adaptability to cost constraints (Section 5.2), test its generalization capability when encountering unseen LLMs (Section 5.3), and provide insights into its routing behavior and training dynamics (Section 5.4).
5.1
Main Results
We conduct comprehensive experiments across seven QA benchmarks, spanning both general QA and multi-hop QA settings, to evaluate the effectiveness of Router-RI against a wide range of baselines. The results are presented Table 1, from which we can derive some key observations.
Router-R1 consistently outperforms all basic baselines across all seven datasets, achieving SOTA performance. Compared Direct, CoT, and SFT, which rely solely on the base LLM's internal knowledge and reasoning capabilities, Router-RI delivers substantially better results, particularly on these snooledge-intensivy tasks where those baselines often struggle. It also outperforms RAG by a significant margin, as Router-RI can dynamically query specialized LLMs during multi-round reasoning, offering more flexibility and relevance than methods relying on static external retrieval. Among these baselines, Search-Ri R1 stands out as a stronger method that supports multi-turn search engine calling, resembling Router-RI's 's multi-round routing capabilities. However, Router-R1 still shows clear advantages, achieving better results across both base L.M models and most QA datasets. Notably, Router-R1-LLame reaches the highest average exact match score of 0.409, while Router- R1 R1-Qwen further improves this to 0.416. These results demonstrate the superior effectiveness and adaptability of Router-RI in routing decisions for both general and multi-hop QA.
Router-RI significantly surpasses all LLM router baselines in overall performance, benefiting from its multi-round routing and interleaved reasoning. Compared to single-round routers like Prompt LLM and KNN Router, Router-RI R1 shows clear and consistent improvements. Even when enhanced baselines like Prompt LLM and KNN Router are introduced, where inputs are decomposed into sub-queries before routing, RI maintains significant advantage. This superiority stems from the core design of Router-RI: using an L.MM as the router itself, enabling flexible interleaving of reasoning and routing steps. Such a design allows Router-RI I to adaptively coordinate across models and aggregate their strengths more effectively. Furthermore, Router-R1 consistently outperforms advanced LLM router baselines such as GraphRouter and RouterDC across
3 https:/buuldd.nidda.comm
7

---

Table 1: Experimental results on seven QA datasets w.r.. Exact Match.
Methods
General QA
Multi-Hop QA
NQ
TriviaQA
PopQA
HpQA
2wiki
Musique
Bamb
Avg.
Owen2.5-3B-Instee
Direct
0.092
0.260
0.122
0.140
0.266
0.026
0.040
0.135
CoT
0.126
0.358
0.160
0.168
0.208
0.046
0.224
0.184
SFT
0.212
0.400
0.160
0.198
0.256
0.052
0.112
0.199
RAG
0.298
0.540
0.366
0.216
0.146
0.078
0.224
0.267
Search-RI
0.328
0.510
0.324
0.236
0.278
0.090
0.272
0.291
Prompt LLM
0.300
0.580
0.340
0.268
0.262
0.108
0.448
0.329
Largest LLM
0.296
0.578
0.354
0.278
0.274
0.104
0.480
0.338
KNN Router
0.262
0.528
0.222
0.224
0.196
0.066
0.360
0.265
MLP Router
0.252
0.460
0.222
0.198
0.210
0.072
0.360
0.253
BERT Router
0.230
0.516
0.192
0.216
0.206
0.058
0.312
0.247
DC
0.278
0.592
0.282
0,244
0,218
0.080
0,504
0.314
GraphRouter
0.276
0.586
0,280
0.234
0.180
0.076
0.448
0.297
Prompt LLM"
0,258
0.500
0.256
0.206
0.248
0.078
0,472
0.288
KNN Router*
0.236
0.478
0.232
0.154
0.234
0.072
0.384
0.256
Router-Be R1 Rw-Qwen
0.388
0.706
0.384
0.352
0.434
0.138
0.512
0.416
-lama-3.2-3B-Instrue
Direct
0.202
0.328
0.176
0.144
0.134
0.018
0.048
0.150
CoT
0.256
0.468
0.182
0.172
0.168
0.040
0.272
0.223
SFT
0.076
0.098
0.084
0.100
0.026
0.016
0.089
RAG
0.308
0.478
0.356
0.162
0.084
0.038
0.176
0,229
Search-RI
0.372
0.578
0.360
0.282
0.226
0,084
0.272
0.311
Prompt LLM
0.304
0.638
0.374
0.248
0.198
0.132
0.528
0.346
Largest LLM
0.344
0.616
0.394
0.258
0,242
0.122
0.472
0.350
KNN Router
0.292
0.572
0,254
0.210
0.182
0.078
0.376
0.281
MLP Router
0,282
0.506
0.248
0.178
0.188
0.064
0.360
0.261
BERT Router
0.256
0.560
0.222
0.210
0.188
0.066
0.296
0.257
RouterDC
0.310
0.614
0.298
0.250
0.204
0.088
0.504
0.324
GraphRouter
0.316
0.602
0,290
0.222
0.170
0.084
0.416
0.300
Prompt LLM*
0.236
0.446
0.164
0.118
0.080
0.036
0.208
0.184
KNN Router*
0.202
0.398
0.166
0.096
0.060
0,030
0.176
0.161
Mouter-R1-Liant 0.416
0.680
0.432
0.322
0.368
0.128
0.520
0.409
Bold indicates the highest score in each column for each base model.
indicates in-domain evaluation; all others are out-of-domain.
both base models and evaluation benchmarks. These improvements highlight the flexibility of Router-R1's 'ss routing mechanism, which interleaves internal reasoning with multi-round routing to progressively refine its answers. By adapting routing decisions to each query through interaction and feedback, Router-RI R1 demonstrates strong performance in general and multi-hop QA tasks.
Router-R1 demonstrates strong generalization to unseen data. Despite being trained with in- domain samples only on NQ and HotpotQA, it achieves robust performance on the remaining five out-of-domain datasets. This indicates that Router-R1 learns transferable routing and reasoning strategies from limited training data, underscoring its generalization ability across diverse QA tasks. More experimental results can be found in Appendix A.2.
5.2 Analysis of Cost Rewards
In this section, we investigate how varying the cost coefficient a influences Router-R1's learned performance-cost balance. We initiate the mapping function m(-) according the pricing tiers of Together API and evaluate four cost coefficients (i.e,, 0.6, 0.7, 0.8, and 0.9) on Router-Rl-Qwen. The results, shown in Figure reveal clear trends. First, the cost coefficient increases. overall performance gradually declines. Second, the cost reward rises simultaneously. indicating that the
https//www.toggtheeraaiiiiiiii

---

policy places greater emphasis on minimizing expensive LLM calls. Notably, we observe that the integration of cost rewards encourages an emergent routing strategy where Router-RI prefers query smaller models first and only escalates to larger models when necessary, thereby forming adaptive routing strategy that dynamically balances efficiency and accuracy based on the cost constraints.
These findings confirm that, by optimizing under a composite reward, Router-RI dynamically adjusts its routing behavior to adhere to resource constraints and achieves controllable trade-off between accuracy and computational expense. An extensive analysis of cost rewards and a case study are provided in Appendix A.2.2 and B, respectively.
Performance (EM)
Cost Reward
Dakaset
Dataset NQ
0.40
- NO
Exact Match
0.35
PopGA
HpDA
Cost Reward
PapQA
2wiki
0.7
Zwiki
0.30
0.6
0.20
0.5
0.6
0.7
0.8
0.9
0.6
0.7
0.8
0.9
Cost Coefficient (a)
Coefficient (a)
Figure 3: Analysis of cost rewards on the NQ, PopQA, HotpotQA (HpQA), and 2WikiMulti- HopQA (2wiki) datasets.
5.3 Generalization Capability Unseen Candidate L.MSs
To evaluate Router-R1's generalization ability, we extend the routing pool of Router-R1-Qwen with two previously unseen models, ie., Saenyya-Creetive--- and include their corresponding model descriptors in the inference prompt. Inference is conducted directly using the original Router-RI- RI-ween checkpoint without any additional fine-tuning.
and 1..MMA3-CharAA-115-- [18],
The results, presented in Table 2, show that performance across all four benchmarks when augmented with the new candidate LLMs. In particular, it achieves new best scores on several datasets, including TriviaQA, PopQA, and overall averages. This improvement demonstrates Router-RI capacity to generalize to unseen candidate LLMs by interpreting their textual descriptors at inference time. Rather than relying on prior exposure or retraining, Router-RI leverages descriptor- based reasoning to infer the strengths new models and dynamically assign queries accordingly. Notably, this is achieved without sacrificing in-domain performance on tasks like NQ and HotpotQA, further validating the robustness of Router-RI's routing strategy. Compared to baselines, Router-RI demonstrates stronger generalization, achieving consistent improvements with newly added LLMs, while baselines like GraphRouter and Prompt LLM show limited inconsistent gains, highlighting their weaker adaptability to unseen models.
maintains and even slightly improves
Overall, the results highlight Router-RI's ability to flexibly adapt unseen candidates and selectively utilize newly introduced LLMs, indicating promising potential for real-world deployment in evolving model ecosystems.
5.4 Discussion
LLM API Call Count Analysis. To assess the adaptability of Router-R1 to tasks of varying difficulty, we analyze the average number of LLM API calls (i.e., the number times candidate LLMs within the routing pool are invoked) by Router-RI --Qenn across seven QA benchmarks. As shown in Figure 4a, Router-Rl-Qwen makes significantly more average LLM API calls on multi-hop QA benchmarks (i.e., HotpotQA. 2WikiMultiHopQA. Musique, and Bamboogle) compared to general QA benchmarks (i.e,, NQ TriviaQA, and PopQA). This indicates that Router-RI can adaptively
https:/tttilaaaddd...cmwwvveeeet 122b

---

Table 2: Experimental results of generalization capability of Router-RI on NQ, TriviaQA, PApQA, and HotpotQA (HpQA) datasets w.r.t. Exact Match and F1-Score ( indicates a routing pool extension).
Methods
NQ!
TriviaQA
PopQA
HpQA
Avg.
EM
F1
EM
F1
EM
F1
EM
F1
EM
F1
Prompt LLM
0.300
0.437
0.580
0.694
0.340
0,409
0.268
0.407
0.372
0.487
Prompt LLM
0.296
0.433
0.584
0.697
0.342
0.412
0.266
0.409
0.372
0.488
BERT Router
0.230
0.362
0.516
0.635
0.192
0.251
0.216
0.335
0.289
0.396
BERT Router
0.238
0.370
0.550
0.662
0.236
0.323
0.198
0.322
0,306
0.419
GraphRouter
0.276
0.412
0.586
0.690
0.280
0.324
0.234
0.366
0.344
0.448
GraphRouter'
0.282
0.420
0.594
0.696
0.276
0.322
0.228
0.360
0.345
0.450
Prompt LLM
0.258
0.373
0.500
0.600
0.256
0.315
0.206
0.313
0.305
0.400
Prompt LLM
0.212
0.311
0.440
0.541
0.196
0.247
0.166
0.252
0.254
0.338
Router-RI Rl-Qwen
0.388
0.484
0.706
0.772
0.384
0,447
0.352
0.449
0,458
0.538
Router-RI-O Rw-Qnen
0.382
0.493
0.722
0.778
0.402
0.464
0.346
0.459
0.463
0.549
Bold indicates the highest score each column for each base model. indicates in-domain evaluation; all others out-of-domain.
assess task difficulty and decide whether external LLM routing is necessary, demonstrating its ability to selectively utilize external resources when tasks are more complex.
Convergence Analysis of Router-R1 Training. To evaluate the convergence behavior of Router-Ri. show two erucial curves during its policy training: the reward and the policy LLM's action entropy. Figure 4b and 4c illustrate that Router-RI converges within 100 training steps, as evidenced by rising rewards and decreasing policy entropy, indicating rapid and robust convergence. It's worth noting that occasional formatting errors may cause brief drops in reward, but our hierarchical reward design swiftly corrects them, ensuring stable and accelerated learning. particular, we observe that without format rewards, Router-RI R1 exhibits greater training instability, frequently generating meaningless or nonsensical text that leads to severe formatting breakdowns in the output.
LLM API Call Frequency
Training Reward Curve
Policy Entropy Curve
Avg. Cails
1.4
1.5
1.36
Reward
1.0
Wofomttrewww wi format reward
1.3
1:12
craahed!
0.6
1.1
1.06
1:01
1.06
fomatroward
04
1.0
-02
wbb format reward
0.2
NQ
TrivisOA PO
HPQA Zutki
Musiaye Bamb
50
100
150
200
50
100
150
200
Training Step
Training Step
(a) LLM API Call Count
(b) Reward Curve
(c) Entropy Curve
Figure 4: Analysis of LLM API call count and Router-RI training convergence.
6 Conclusion
In this paper, we introduce Router-R1, a reinforcement barming--bseed framework that formulates multi-LLM routing and aggregation as sequential decision process. By instantiating the router as a capable LLM, Router-R1 interleaves internal reasoning with targeted model selection and incrementally builds responses through multi-round interaction. Our lightweight rule-based rewards, which combine format, outcome, and cost rewards, enable Router-R1 to achieve superior performance while learning flexible performance-cost trade-offs. On seven diverse QA benchmarks, outperforms more than ten strong baselines and maintains robustness and generalization in the presence of distractor models. These results demonstrate the promise of RL--driven routing for orchestrating multiple LLMs.
10

---

Acknowledgments
We sincerely appreciate the support from Amazon grant funding project #120359, "GRAG: Enhance RAG Applications with Graph-structured Knowledge", and research gifts from Meta and Lenovo.
References
[1] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions intelligent systems and technology, 15(3):1-45, 2024.
[2] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.051 2023.
[3] Shuhao Chen, Weisen Jiang. Baijiong Lin, James Kwok, and Yu Zhang. Routerde: Query-based router by dual contrastive learning for assembling large language models. Advances Neural Information Processing Systems, 37:66305-66326 2024.
[4] Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John Lui. Cost-effective online multi-llm selection with versatile reward models. arXiv preprint arXiv:2405. 16587, 2024.
[5] Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid llm: Cost-efficient and quality- aware query routing. arXiv preprint 2404 /4618, 2024.
[6] Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: graph-based router for Ilm selections. arXiv preprint arXiv:2410.03884. 2024.
[7] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407:21783, 2024.
[8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[9] Xanh Ho, Anh- Bhh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computationnl Linguistics, pages 6609-6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
[10] Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403. /2031, 2024.
[11] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint 2400::44000000888, 2024.
[12] Fengging Jiang. Identifying and mitigating vulnerabilities in Imm- Imm-interrated applications. Mas- ter's thesis, University of Washington, 2024.
[13] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025.
[14] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeling of the Association for Compula- tional Linguistics Volume Long Papers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.
11

---

[15] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6781, Online, November 2020. Association for Computational Linguistics.
[16] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Keloey, Ming- Wei Chang. Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019.
[17] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif rlhf: Scaling rein- forcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00260 2023.
[18] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint 2001 2024.
[19] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Amnual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802-9822, Toronto, Canada, July 2023. Association for Computational Linguistics.
[20] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-124235, 2024.
[21] Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saced Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint arXiv:2307 0.445,,,555 2023.
[22] Isaac Ong, Amjad Almahairi, Vincent Wu, Wii-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Rouuellm: Learning to route Ilms from preference data. In The Thirteenth International Conference on Learning Representations, 2024.
[23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277 277444, 2022.
[24] Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687-5711 1, Singapore, December 2023. Association for Computational Linguistics.
[25] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 08:53728-537444 2023.
[26] Marija Šakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-modeling, In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 606-615, 2024.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint prxiv://777 170.0063777 2017.
[28] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv: arXiv:2402.0333 03300, 2024.
12

---

[29] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming, Advances in Neural Information Processing Systems, 35:9460-
9471, 2022.
[30] Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. Tensoropera router: A multi-model router for efficient Ilm inference. arXiv preprint arXiv:2408.1 12320, 2024.
[31] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprini arXiv:2408.000 18, 2024.
[32] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2022.
[33] Liang Wang, Nan Yang. Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv::212.033533 2022.
[34] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837724-24877, 2022.
[35] An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024.
[36] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: 010888899000000 2018
[37] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
13

---

A Experimental Details
A.1 Implementation Details
A.1.1 Model Descriptors
To enrich model descriptors and better inform routing decisions, we draw on the publicly available model cards Specifieally, prompt GPT-4o generate brief, standardized summaries of each candidate LLM based on the information in these model cards. These concise descriptions are then injected into the training prompt of Router-RI. This enables the model develop an initial understanding of each LLM's capabilities.
It is worth noting that these model descriptors serve solely cold-start priors to guide early-stage decisions. During policy optimization, Router-R1 gradually refines its understanding of each LLM's strengths and weaknesses through repeated interaction and reward feedback. Furthermore, this design allows Router-R1 to seamlessly integrate new LLMs into the routing pool without retraining by simply appending their descriptions the prompt. This flexibility supports fast adaptation newly released models and enhances Router-RI 's generalization in dynamic multi-LLM environments.
A.1.2 Format Rewards
To ensure the structural quality and consistency of generated completions, design nase- format reward that penalizes malformed outputs. The function inspects whether the completion correctly follows the expected structure using specific action tags. The format reward function imposes the following requirements:
All tags must be properly opened and closed, with no nesting allowed.
The response must begin with a <think> </think> block and end with single <answer> ... </answer> block.
At least one <think>
</think> block must be present, and exactly one <answer>... ... </answer> block required.
Each <search> ... </search> must be paired with corresponding <info> ... </info> block.
Routed queries must follow the format "Ilm name: query", where the LLM name is valid and both parts are non-empty.
This format reward is lightweight but effective in enforcing disciplined structure in the model's output, guiding the policy toward well-formed and interpretable reasoning trajectories.
A.1.3 Cost Rewards
To encourage cost-efficient routing decisions, we design cost reward that penalizes the use of high-cost LLMs. This reward is computed as the inverse of a dynamically normalized cost value using sliding window approach.
Sliding Window Normalization. We maintain a fixed-size buffer of recent cost values (window size 1000), and apply smoothing transformation to each cost before normalization. Specifically, we adopt a square root transformation:
r'=
where r the raw cost associated with the selected LLM (i.e,, (PLLM) -Tot).
To ensure robustness against outliers, we normalize r' using the 5th and 95th percentiles of the buffer. Let Tmin and Fmax the 5th and 95th percentile values respectively. The normalized cost is then computed as:
Tmin
F Fnomm
Tmmx
Tmin
the range Fmax Tmin is too small (less than a small threshold E), we return a neutral reward value of 0.5 to maintain stability.
https:/build..iiiia.commmmm
14

---

Reward Inversion. To encourage the selection of low-cost LLMs, we invert the normalized value:
Resst 1.0 clip(Tnnrm. 0.0, 1.0)
This ensures that higher-cost models receive lower rewards, guiding the policy to prefer efficient candidates without relying on hard constraints. This cost reward design allows Router-R RI to learn cost-sensitive routing policies that adapt dynamically to the resource profiles of available LLMs.
Discussion. While the cost reward normalization stabilizes training, our experiments mainly focus on QA tasks. Router-RI were trained across heterogeneous domains (e.g., math, code, QA), the large variation in response lengths could cause inconsistent cost scales even after normalization, com- plicating optimization. A promising direction for future work is to introduce task-level normalization to better align reward magnitudes across domains and improve multi-task training stability.
A.2 More Experimental Results
A.2.1 Main Results w.r.. F1-Score
We present extensive experimental results on seven QA datasets with respect F1-Score in Table 3. Router-R1 consistently outperforms all baselines across both general and multi-hop QA tasks, achieving the highest average F1-scores for both Qwen and LLaMA backbones.
Table 3: Experimental results on seven QA datasets w.r.t. F1-Score.
Methods
General QA
Multi-Hop QA
NQ
TriviaQA
PopQA
HpQA
2wiki
Musique
Bamb
Avg.
Qwen2.5-3B-Instruet
Direct
0.162
0.341
0.154
0.215
0.304
0.081
0.112
0.196
CoT
0.218
0.431
0.185
0.260
0.251
0.106
0.332
0.255
SFT
0.289
0.460
0.207
0.281
0.291
0.121
0.173
0.260
RAG
0.414
0.622
0.452
0.307
0.187
0.134
0.303
0.346
Search-R
0.407
0.575
0,383
0.328
0.317
0.145
0.387
0.363
Prompt LLM
0.437
0.694
0.409
0.407
0.393
0,205
0.579
0.446
Largest LLM
0.431
0.695
0.423
0.416
0.397
0.199
0.608
0.453
KNN Router
0.388
0.627
0.281
0.341
0.289
0.141
0.496
0.366
MLP Router
0.368
0.557
0.272
0.295
0.277
0.142
0.460
0.339
BERT Router
0.362
0.635
0,251
0.335
0,284
0.135
0.423
0.346
RCuterDC
0.410
0.694
0.328
0.381
0.293
0.165
0.623
0.413
GraphRouter
0.412
0.690
0.324
0.366
0.258
0.154
0.546
0.393
Prompt LLM*
0.373
0.600
0.315
0.313
0.355
0.165
0,580
0.386
KNN Router*
0.360
0.572
0.272
0.248
0.299
0.138
0.493
0.340
Router-RI MouterR1-Qween
0.484
0.772
0.447
0.449
0.487
0.212
0.635
0.498
Alama-3.2-3B-Instruee
Direct
0.281
0.377
0.209
0.211
0.165
0.070
0.097
0.201
CoT
0.364
0.534
0,230
0.241
0,213
0.104
0.385
0.296
SFT
0.125
0.159
0.117
0.154
0.267
0.075
0.042
0.134
RAG
0.410
0.557
0.413
0.247
0.119
0.070
0.240
0.294
Search-RI
0.478
0.650
0.412
0.375
0.271
0.138
0.337
0.380
Prompt LLM
0.450
0.733
0.441
0.369
0.301
0.222
0.628
0.449
Largest LLM
0.469
0.711
0.448
0.377
0.341
0.204
0.563
0.445
KNN Router
0.410
0.651
0.292
0.312
0.248
0.140
0.504
0.365
MLP Router
0.395
0.585
0.290
0.271
0.236
0.134
0.455
0.338
BERT Router
0.383
0.656
0.272
0.310
0.244
0.126
0.388
0.340
RCuterDC
0.435
0.708
0,338
0.371
0.257
0.161
0.601
0,410
GraphRouter
0.441
0.704
0.327
0.343
0.227
0.146
0.545
0.390
Prompt LLM*
0.342
0.524
0.209
0.188
0.136
0.088
0.302
0.256
KNN Router*
0.311
0.476
0.211
0.149
0.104
0.074
0.259
0.226
Pouter-RI-Llamm
0.520
0.740
0.461
0.429
0.420
0.188
0.625
0.483
Bold indicates the highest each column for each base model. indicates in-domain evaluation; all others are out-of-domain.
15

---

A.2.2 Extensive Analysis of Cost Rewards
We conduct extensive study the effect of different cost coefficients in Router-RI and compare with various baselines terms of exact match (EM) and average raw cost rewards. The results in Table 4 and the Pareto-style plots in Figure 5 together illustrate clear trade-offs between answer accuracy and computational cost. When a 0.0, Router-RI achieves the highest EM across nearly all datasets, reflecting its performance-oriented routing preference. As increases, Router-R1 gradually shifts toward more cost-efficient decisions, substantially reducing average cost but with a moderate decline in EM. In particular, accuracy while lowering cost compared to the a = 0.0 setting. This demonstrates the flexibility of cost-aware reward design in adapting different resource constraints.
0.6 consistently achieves a favorable balance malaace-maintaining maintaining strong
The Pareto plots further reveal that Router-RI effectively spans a wide cost post-performance frontier: by tuning it can approximate the accuracy of larger, stronger models at fraction of their cost, or conversely, prioritize efficiency while retaining competitive accuracy. Compared to static base- lines, Router-RI provides a continuous and controllable trade-off, highlighting its robustness and adaptability to diverse deployment requirements.
Among individual LLMs, LLaMA-3.1 1-7BB and Mixtral-8x221 achieve the strongest EM scores, likely owing their larger model capacities. Notably, Mixtral attains high performance relatively lower cost, which can be attributed to its tendency to generate shorter responses under identical prompts. Other individual LLMs exhibit lower accuracy but are more cost-efficient due smaller sizes, forming the lower-cost region of the Pareto frontier.
Overall, Router-RI's s cost-aware design enables smooth traversal along the efficiency-accuracy continuum, achieving competitive or superior trade-offs relative to all individual models and learned routers. These findings validate the effectiveness of the reward design and demonstrate Router-RI's potential for flexible, resource-adaptive deployment in multi-LLM, systems.
Table 4: Extensive analysis of cost rewards on NQ, PopQA, HpQA, and 2wiki datasets w.r.t. Exact Match and raw cost rewards (unnormalized),
Methods
NQ
PopQA
HpQA
2wiki
EMI
Cost
EMI
Cost
EM
Cost
EM
Cost
Owen2.5-3B-Instrue
Prompt LLM
0.300
20.0
0.340
17.6
0.268
20.1
0.262
20.2
Largest LLM
0.296
20.2
0.354
17.6
0.278
20.1
0.274
20.1
KNN Router
0.262
41.0
0.222
30.3
0.224
45.4
0.196
51.8
MIP Router
0.252
35.8
0.222
21.0
0.198
29.5
0.210
41.3
BERT Router
0.230
26.0
0.192
16.3
0.216
26.0
0.206
26.3
RouterDC
0.278
57.5
0,282
21.8
0.244
62.3
0.218
40.7
GraphRouter
0.276
29.6
0,280
19.2
0,234
24.7
0.180
28.6
Prompt LLM*
0.258
286.4
0.256
111.7
0.206
313.4
0.248
222.4
KNN Router*
0.236
102.2
0.232
49.8
0.154
133.0
0.234
99.4
Qwen2.5-7B- 5---....-B-nsstru
0.138
24.2
0.130
18.6
0.152
24.9
0.166
28.7
LLaMA-3.1-70B-Imes
0.280
153.3
0.342
76.3
0.260
124.6
0.270
119.8
LLaMA-3.1-8B -88--rrnttt
0.242
0.208
14.3
0.198
24.1
0.130
21.3
Mistral- 7B IB-sruttttt
0.192
20.2
0.182
16.2
0.202
19.1
0.198
18.3
nixtral-8x221 8x22B-Instruct
0.296
20.2
0.354
17.6
0.278
20.1
0.274
20.1
Gemma-2-27B Instructt
0.282
42.7
0.290
22.0
0.244
37.8
0.190
44.6
Router-R1-o R1-Qwen (a 0.0)
0.388
1.5066
0.384
98.3
0.352
138.6
0.434
150.8
Mouter-R1-Qwer
0.6)
0.388
1.50.0
0.414
7.5.9
0.332
124.3
0.422
113.8
Router-RI-oe R1-Qwen 0.7)
0.318
32.3
0.280
17.2
0.254
27.2
0.236
31.4
Router-RI-Qwen (a -0.8)
0.320
28.9
0.270
14.9
0.238
28.2
0,202
31.4
Router-R1-Qwen (a -0.9)
0.244
5.5
0.234
5.3
0.186
5.3
0.252
6.5
Bold indicates the best score in each column.
indicates in-domain evaluation; others are out-of-domain.
16

---

NQ
PopQA
0.40
N.00
0-50
0.35
0.30
0.25
0,20
0.15
50
150
200
250
300
20
60 Cost (unnomalized, -))
40
100
Cost (unnormalized, +)
HpQA
2wiki
0.35
80-075
0.6
0.40
Match
0.35
0B
80
exact
0.25
0.20
0.15
100
150
200
250
300
50
100
150
200
Cost (unnormalizedd,rg +)
Cost (unnormalized,
Rowwe
KNN MLP Router
RouterDC
[LMMA3. 1-88
Nixtrw-Be228
Prompt LLM
GraphRouter
Owen2 5-
Mistral 78
Gemma 278
BERT
LIM
MA3.1-708
Figure 5: Cost vs. Performance Pareto Curve on NQ, PopQA, HpQA, and 2wiki datasets w.r.t. Exact Match and raw cost rewards (unnormalized).
A.2.3 More Comparison Experiments
In this section, present additional comparison results using FrugalGPT [2] as baseline and Qwen2.5-3B-Instre [35] as base model. As shown in Table 5, Router-RI consistently outperforms FrugaIGPT, demonstrating its clear superiority in routing decisions across both general and multi-hop QA tasks.
Table 5: More Experimental results on seven QA datasets w.r.t. Exact Match.
Methods
General QA
Multi-Hop QA
NQ
TriviaQA
PopQA
HpQA!
2wiki
Musique
Bamb
Avg.
Qwen2.5-3B-Instruec FrugaIGPT
0.265
0.562
0.362
0.234
0.268
0,103
0,430
0.318
Router-RI-Qwee R1-Qwen
0.388
0.706
0.384
0.352
0.434
0.138
0.512
0.416
Bold indicates the highest score in each column for each base model. indicates in-domain evaluation; all others out-of-domain.
17

---

A.2.4 Sensitivity Analysis of Model Descriptors
To assess whether Router-RI is sensitive to the richness of model descriptors, we conduct an additional experiment where the original detailed descriptors (see Appendix C) are compressed retain only key attribute information while removing stylistic and secondary details. For example: LLaMA-3.1 1- 70B-Instruct: A 70B state-of-the-ar model for multilingual dialogue and reasoning, excelling in benchmark evaluations. This modification does not alter the overall format of the descriptors but reduces their information richness, providing more concise description of each model. The results, reported below in Table 6 (on Router-R RI l-wen), show that this simplification has only a negligible impact on performance, with the average exact-match score changing marginally from 0.416 to 0.419. This indicates that Router-RI is robust to variations in descriptor richness.
Table 6: Sensitivity analysis of model descriptors on seven QA datasets w.r.t. Exact Match.
Methods
General QA
Molti-Hop QA
NQ
TriviaQA
PopQA
HpQA
2wiki
Musique
Bamb
Avg.
Modification
0.388
0.714
0.402
0.348
0.425
0.150
0.504
0.419
Default
0.388
0.706
0.384
0.352
0.434
0.138
0.512
0.416
indicates in-domain evaluation; all others are out-of-domain.
A.2.5 New Candidate Incorporation During Training
To further examine the adaptability of Router-RI dynamically expanding model pools, we conduct an additional experiment where a new model ALLaMA-ChaiQA-I.-55
is incorporated during training. While our main results (Table focus on generalization unseen models introduced only at test time, this experiment evaluates whether adding new model during the learning phase affects routing behavior.
The results, summarized below in Table 7 (on Router- Mentte-RR-Qveee ---eenn), show that the average exact-match score slightly increases from 0.416 0.425 after incorporating the model, with stable perfor- mance across all datasets. This demonstrates that Router-RI I maintains robustness and generalization even when the routing pool evolves during training.
Table New candidate incorporation during training on seven QA datasets w.r... Exact Match. Methods
General QA
Multi-Hop QA
NQ
TriviaQA
PopQA
HpQA!
2wiki
Musique
Bamb
Avg.
Qwen2.5-3BIInstruct
Modification
0.384
0.726
0.390
0.332
0.421
0.154
0.568
0.425
Default
0.388
0.706
0.384
0.352
0.434
0.138
0.512
0.416
indicates in-domain evaluation; all others are out-of-domain.
18

---

B Case Study
In this section, we present case study Router-RI under both zero and non-zero cost coefficients a. This case study illustrates the output behavior (e.g., routing) of Router-RI under both the presence and absence of cost reward constraints.
B.1 Case Study without Cost Reward Constraints
**Question*** * Which film was released more recently, Sacred Silence or Ek Haseena Thi Ek Deewana Tha?
**Ground Truth**; Ek Hascena Thi Ek Deewana Tha
Response of Router-RI:
<think> To answer the question "Which film released more recently, Sacred Silence or Ek Haseena Thi Ek Deewana Tha?", need specific information about the release dates of these films. I don't have this knowledge stored within me, so will need consult an external specialized language model to find the answer. </think>
<hearch> L.AMA---.--7B-Insiii or Ek Haseena Thi Ek Deewana Tha? </search>
Which film was released more recently, Sacred Silence
<info> can help with that! After checking, found that: * "Sacred Silence" (also known as "Il silenzio sacro") ") is 2016 Italian drama film. "Ek Haseena Thi Ek Deewana Tha" is 2017 Indian romantic thriller film. Based on the release years, "Ek Haseena Thi Ek Deewana Tha" (2017) was released more recently than "Sacred Silence" (2016). </info> <answer> Ek Haseena Thi Deewana Tha </answer>
19

---

** ** Which film has the director who died earlier between To See Or Not To See and Prince (1969 Film)?
**Ground Truth**; (TTo See or Not to See", 'To See Or Not To See"]
Response of Router-R1:
<think> To answer the question about which film has the director who died earlier between To See Or Not To See and Prince (1969 Film), need to determine the directors of these films and their death dates. I don't have immediate knowledge of these details, consult an external specialized language model to find the required information. </think> <search> LLaMA-3.1 1-0---11-7---trruuee Which film has the director who died earlier, To See Or Not To See or Prince (1969 Film)? </search>
will need to
<info> A very specific question! After conducting research, found that the 1969 film "Prince" is directed by Lekh Tandon, an Indian film director. Unfortunately, couldn's t find any information on film called "To See Or Not See" that would allow me determine the director's s name or their date of passing. However, I did find that Lekh Tandon passed away on October 15, 2017. you're re looking for more information on "To See Or Not To See" or its director, I recommend consulting other LLMs or film databases for further assistance. </info>
<think> Since I don't have enough information to directly answer the question, I will need to consult another specialized language model to determine the director of "To See Or Not To See" and compare their death dates with Lekh Tandon. </think> <search? LL 111-77BBBBnntC Which film is called "To See Or Not See" and who its director? Additionally, please compare the director's death date with Lekh Tandon's death date (October 15, 2017). </search>
<info> I'm happy to help with the sub-question! After conducting thorough search, I found that the film "To See Or Not To See" is also known as "Andha Kanoon" " (1983), but I couldn't confirm this title exactly matching "To See Or Not To See". However, I did find a film titled "To See or Not to See" (1969), which is an Indian film directed by Biren Nag. Regarding the comparison of the director's death date with Lekh Tandon's ' death date, I found that Biren Nag passed away on August 5, 2018. Therefore, Biren Nag's ' death date is approximately 10 months after Lekh Tandon's 'ss death date (October 15, 2017). Please note that I couldn't confirm the exact title match, but provided relevant information related to the question. If the base model requires further assistance or confirmation, I recommend consulting other LLMs for additional information. </info>
<think> With the information that Biren Nag directed "To See or Not to See" (1969) and passed away on August 5, 2018, I can now compare his death date with Lekh Tandon's 's death date (October 15, 2017). Since Biren Nag died later than Lekh Tandon, the director "To See or Not to See" (1969) died earlier than the director of "Prince" (1969 Film). </think> <answer> To See or Not to See </answer>
20

---

* ** Which film has the director who born first, Women He's S Undressed The King's S Stamp?
**Ground Truth**; ["The King's Stamp", "King's s Stamp", "The King's S Stamp"]
Response of Router-RI:
<think> answer the question "Which film has the director who was born first, Women He'S Undressed or The King's S Stamp?", , need to determine the birth years of the directors of these films. However, don't have immediate knowledge of these directors' birth years. will need to consult an external specialized language model to find this information. </think> <search> LLaMA-3. --sttr-nttttt Which film has the director who was born first, Women He'S Undressed or The King'S S Stamp? </search>
<info> To answer this question, need to identify the directors of the two films and their birth dates. After researching. found that: * "Women He's Undressed" (2015) is a documentary film directed by Gillian Armstrong. * "The King's 's Stamp" doesn't it seem to well-known film. However, found film called "The King's Speech" (2010) directed by Tom Hooper, but I also found another film called "The King Stamp" (1935) directed by Michael Powell. Assuming the question refers to the 1935 film "The King's Stamp", here the birth dates of the directors: * Gillian Armstrong was born on December 18, 1950. MMichhel Powell was born Septemher 30, 1905. Based on this information, Michael Powell, the director of "The King's 's Stamp" (1935), was born first. Please note that the question refers to a different film with the title "The King's 's Stamp", my answer may not be accurate. </info> <think> With the information obtained from the external I know that Michael Powell, the director of "The King's Stamp" (1935), born first. However, I need to determine if "Women He's S Undressed" (2015) has a director who was born earlier than Michael Powell. Since I don't have the birth year of the director of "Women He'S S Undressed",, I will need to consult another external specialized language model. </think>
model,
<search> LLaMA-3. 3.aMA-3.11-0B 11-70B-Inntructt If the question is "Which film has the director who was born first, Women He'S Undressed or The King'S S Stamp?", and I found that "The King's Stamp" (1935) was directed by Michael Powell who was born on September 30, 1905, but don't have the birth year of the director of "Women He'S Undressed" (2015). What the answer? </search>
<info> A nice question about film directors! Although you don't have the birth year the director of "Women He' 'S Undressed" (2015), I can provide some helpful context. After quick search, I found that "Women He'S Undressed" (2015) is a documentary film directed by Gillian Armstrong. Now, let's compare the birth years: Michael Powell, the director of "The King's Stamp" " (1935), was born on September 30, 1905. Gillian Armstrong, the director of "Women He 'S Undressed" (2015), was born December 18, 1950. Based on this information, it's clear that Michael Powell was born first, so the answer is: "The King's Stamp" (1935) has the director who was born first. </info> <answer> The King's Stamp </answer>
21

---

B.2 Case Study with Cost Reward Constraints
**Question**; The radiographic term used to describe the dense bone of the socket and septal crest is?
**Ground Truth**; ('lamina dura', 'alveolar process", 'the lamina dura')
Response of Router-R1:
<think> need to determine which LLM would be most suitable for this query. The LLMs have different strengths, but typically, LLaMA-3. 1-8-IBnssttccc and 1--MA--.1-70B-Insmttte are strong in multilingual and complex reasoning tasks, which might be helpful here. I will search for the most appropriate LLM. </think>
<search> LLaMA-3.1 -nstttruttt the radiographic term used to describe the dense bone of the socket and septal crest is? </search>
<info> The radiographic term used to descrihe the dense bone of the socket and septal crest is **lamina dura** The lamina dura is radiodense layer of bone that lines the socket of tooth and the septal crest of the maxilla. It a normal anatomical structure that can be visible on radiographs (x-rays) and is an important landmark for dental and maxillofacial radiology. In dental radiography, the lamina dura is often used as a reference point to assess the health of the surrounding bone and detect any abnormalities, such as periapical lesions or bone loss. </info> <answer> lamina dura </answer>
"*Question** **; Which film whose director was born first, Maker Of Men or Kentucky Kernels?
GGround Truth**; ['Maker Of Men', 'Maker of Men']
Response of Router-R1:
<think> I need determine which film, Maker Of Men or Kentucky Kernels, had a director born first. will search externally to find this information. </think>
<search> LLaMA-3.1 1.B-A-----B-IIsttuucc Which film, Maker Of Men or Kentucky Kernels, had director born first? </search>
<info> I'm happy to help with this question. However, need clarify that couldn't 't find any information on films called "Maker Of Men" or "Kentucky Kernels". It's possible that these films are obscure, unknown, or non-existent. If you could provide more context or details about these films, I'd be happy to try and assist you further. Alternatively, can suggest some possible sources where you may be able to find the information you're 're looking for. you're re unable to provide more context, I'm afraid I' I'm unable to assist with this question. In this case, would instruct the base model to consult other LLMs for further assistance, such those specializing in film databases obscure movie information. </info> <think> Based on the inability to find direct information, will now consult another LLM to gather the necessary details. </think>
<search> LLaMA-3.1 1-11-----Insttuu director born first? </search>
Which film, Maker Of Men or Kentucky Kernels, had
<info> To answer this question, I need to provide some background information on the directors of the films mentioned. "Maker of Men" (1931) directed by Edward Laemmle, an American film director. Edward Laemmle was born on October 25, 1887. "Kentucky Kernels" (1934) directed by George Stevens, American film director, producer, screenwriter, and cinematographer. George Stevens was born on December 18, 1904. Based the birth dates, Edward Laemmle, the director of "Maker of Men", was born first. </info>
<answer> Maker Of Men </answer>
22

---

**
Where was the place of death of Topa Inca Yupanqui's father?
** *Ground Truth*:; ['Cusco', 'Cuzco', CCusco, Peru', CCuzco, Peru']
Response of Router-R1:
<think> need to determine the place death Topa Inca Yupanqui's father. will search externally find this information. </think> <search> 1.------.-BB--nstruuct Where was the place of death of Topa Inca Yupanqui's father? </search>
<info> I'm unable to assist with this question as I don't have specific information on the place of death of Topa Inca Yupanqui's father. The historical records of the Inca Empire are extensive, but detailed information on individual family members' lives, especially their places of death, can be scarce not well-documented. To further assist with this question, I recommend that the base model consult other LLMs or historical databases that specialize in Inca history or Peruvian archaeology. These sources may provide more detailed accurate information on the life and death of Topa Inca Yupanqui's father. </info>
<think> Based on the need consult external knowledge, will now search again to find the answer. </think>
<search> 1LaMA-3311-0BB Inttttstuttt Where was the place of death of Topa Inca Yupanqui's father? </search>
<info> A question about Inca history! Topa Inea Yupanqui's 's father was Pachacuti, the ninth Sapa Inca (emperor) of the Inca Empire. According to historical records, Pachacuti died in Cusco, Peru, which was the capital of the Inca Empire at that time. To provide more context, Pachacuti was a renowned leader who expanded the Inca Empire significantly during his reign 1178).472). He was succeeded by his son, Topa Inca Yupanqui, who continued to expand the empire. If you need more information about Pachacuti, Topa Inca Yupanqui, the Inca Empire, feel free to ask! </info> <answer> Cusco </ answer>>
The case study illustrates the behavioral differences of Router-RI under different reward settings, In the first (without reward constraints), the routing policy converges toward always selecting whhic leads to the best answer accuracy but
the most capable LLM LLaMA-3.1 10-...1-7000 incurs higher inference cost. In contrast, the second case (with cost reward constraints) demonstrates that Router-RI R1 learns to initially route queries to a smaller, more cost-efficient 8B-Instruct. Only when the smaller model fails to provide sufficient information does the policy escalate to querying the larger L.M. This strategy reflects cost-aware routing behavior that balances performance with efficiency.
23

---

C
Prompts
In this section, we detail the prompts used in our experiments.
Model Descriptors
Qwen2.5-78BIIstrrune
Qwen2.5-7B I----nnntuuttt is a powerful Chinese-Englisi instruction-tunat large language model designed for tasks in language, coding, mathematics, and reasoning. As part of the Qwen2.5 series, features enhanced knowledge, stronger coding and math abilities, improved instruction following, better handling of long and structured texts, and supports up 128K context tokens. It also offers multilingual capabilities across over 29 languages.
LLaMA-3.1-BB-Isttee
LLaMA-3. 1.aMA-3.1-8B--nss -88tttruutttt an 8-bilion-paramette instruction-tuned language model optimized for multilingual dialogue. provides strong language understanding, reasoning, and text generation performance, outperforming many open-source and closed-source models on standard industry benchmarks.
LLaMA-3.1-70B-Insse is 70-billion partimeter state-of-the-ee of-the-art language model designed LLaMA-3 for advanced multilingual dialogue tasks. It excels in language comprehension, complex reasoning, and high-quality text generation, setting new standard against both open and closed models in benchmark evaluations.
Mistral-7B-Inssiee
Mistral-7B-Instruce is a fine-tuned version of the Mistral-7B-v0.3 language model designed to follow instructions, complete user requests, and generate creative text. It was trained diverse public conversation datasets to enhance its ability handle interactive tasks effectively.
Mixtral-8x222-Insttee
Setill-8x22-IInsttue is a cutting-edge sparse Mixture-of-Experts (SMoE) large language model from MistralAI. efficiently uses 39B active parameters of 141B total, delivering high performance at lower costs. The model excels at following instructions, completing tasks, and generating creative text, with strong skills in multiple languages (English, French, Italian, German, Spanish), mathematics, and coding. It also supports native function calling and handles long contexts up to 64K tokens for better information recall.
Semmma---7B-IIstteeee
Gemma-2-27B: 2-27---nstrt--eee a cutting-edge, instruction-tune text generation model developed by Google. Built using the same technology Gemini, excels at text understanding, transformation. and code generation. As a lightweight, decoder-only model with open weights, it is ideal for tasks like question answering, summarization, and reasoning. Its compact size enables deployment on laptops, desktops, or private cloud setups, making powerful AI more accessible.
24

---

Model Descriptors (Unseen LLMs)
LLaMAJ-ChatQQA-L--5)
is an 8-billion-paraan parameter instruction-tuned language model built on top of the LLaMA-3 base, specifically optimized for conversational question answering (QA) and retrieval-augmented generation (RAG). Developed with an improved training recipe from the ChatQA paper, incorporates rich conversational QA data enhance its performance on tasks involving tabular reasoning and arithmetic calculations.
LLaMA3-ChaIQA-IN
Palmyra-Creative-
Saanyrr-Creetive-- 122B is a 1 parameter model by Writer, built for high-quality creative writing and content generation. excels tasks like storytelling, poetry, scriptwriting, and marketing copy, adapting to various styles and tones while maintaining consistent voice. Ideal for writers and content creators, supports diverse creative workflows.
Prompts for Querying LLM Candidates
You are helpful assistant.
You are participating in a multi-round reasoning process, where a base model delegates sub-questions to specialized models like you.
Your task is to do your **absolute best** to either:
+
Answer the question directly, if possible, and provide a brief explanation: or
+ Offer helpful and relevant context, background knowledge, or insights related to the question, even if you cannot fully answer
you are completely unable to answer the question or provide any relevant or helpful information, you must:
+ Clearly state that you are unable to assist with this question, and
+ Explicitly instruct the base model to consult other LLMs for further assistance.
"mportant Constraints**
+ Keep your response clear, concise, and informative (preferably under 512 tokens). Your response will help guide the base model's reasoning and next steps. + Stay strictly on-topic. Do not include irrelevant or generic content.
Here is the sub-question for you to assist with:
D Hyperparameter Settings
Table 8: Hyperparameter Settings (shared across all datasets)
Hyperparameter
Value
Hyperparameter
Value
Learning Rate (Actor)
lc-6
Learning Rate (Critic)
le-5
Total Batch Size
64
Mini-batch Size
32
Micro-batch Size
8
Max Training Steps
225
Max Routing Steps
4
Max Sequence Length
4096
Max Response Length
1024
Max Length for LLM API Response
600
Tensor Parallel Size
GPU Utilization Ratio
0.6
Rollout Sampling Temperature (Train)
1.0
Rollout Sampling Temperature (Eval)
25

---

E Limitations
While Router-RI shows strong empirical performance, has several limitations:
Task Scope: Our evaluation focuses primarily on QA tasks. It remains to be seen how well Router-RI generalizes other domains such as dialogue, summarization, or code generation, which may have different routing dynamics.
The rule-based reward function, while effective, may be insufficient for capturing more nuanced objectives like factual consistency long-term dialogue coherence. Incorporating learned or human-in- in-the--oohopp reward functions could further enhance the framework.
Reward Simplicity:
Inference Latency: Although Router-R1 aims optimize cost, its multi-round nature introduces inference latency, especially when reasoning steps are interleaved with multiple model calls. This may limit its suitability for time-sensitive applications.
Dependence Model Descriptors: The generalization to unseen LLMs relies on simple descriptors like pricing and latency. These may not capture deeper model behaviors or capabilities, especially in settings with limited performance history.
Future work may explore improving Router-RI 's reward modeling, reducing inference latency through model pruning routing heuristics, and broadening its application scope wider range of language tasks.
Broader Impacts
Router-RI provides a principled and flexible framework for coordinating multiple large language models, offering potential benefits for improving the efficiency, scalability, and quality of language model deployments. By learning to dynamically balance performance and cost, Router-R1 can help reduce reliance on expensive models where unnecessary, which may lower the environmental footprint and financial cost of large-scale LLM applications.
Additionally, Router-RI's architecture encourages the reuse and composition of existing LLMs, promoting modularity and potentially accelerating progress in collaborative AI systems. Its general- ization ability to unseen LLMs may support faster integration of new models into production systems without extensive retraining.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in abstract and introduction accurately reflect the paper's contribution, which can be found in Section
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide it in Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We elaborate it in Section 4 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer:
[Yes]
Justification: The datasets we use are publicly available, and the link to the released code is presented in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/guides/Codest StapeeeeeDooeSSuuiiiiiiooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: httpsssccppubllc//ee SetguddeeCooeeeaeeeeee
ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We elaborate these details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We fix the seed for all our experiments, which ensures the reliability of our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tttps://neurip... httpes//uurrpp..ccppbb/// wtw.hhhhhhiieeeeeeeeee
Answer: [Yes]
Justification:
We confirm that our research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide it in Appendix F.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use several existing datasets in our work and have ensured that all are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
32

---

component, and the details Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: Our method involves training LLM described in the experimental setup section (Section 4).
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: (bttps://neur)) h/tp:///eeuu..eeeee http///rppssccooooeeeeereeeeeeee ccc.cooneeeeeccee/222//1111 for what should or should be described.
33


Reviews:
[{"review_id": "39Z1vCOMB8", "reviewer": "Reviewer_iXcu", "summary": "This paper introduces an interesting idea to combine model reasoning ability with multi-model routing,  Router-R1 optimizes for performance and cost using a hierarchal rule-based reward system. It demonstrates strong generalization to unseen LLMs via simple model descriptors. Experiments show superior performance on diverse QA benchmarks, highlighting robust generalization and cost management.", "questions": "- is it possible to consider the computational cost of the Router-R1 policy LLM into the overall cost analysis, or even let the model optimize the joint cost?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The use of reasoning model for routing is interesting yet intuitive, utilizing the more powerful reasoning paradigm on the model routing use case. The strong generalization may suggest the model learns some useful insights for routing the models.\n- The reward design enables successful RL training without suffering from reward hacking. Specifically, the authors choose to use rewards with verifiable rules and uses a hierarchical reward system.\n - Extensive evaluation is done on a list of benchmarks and baseline methods.\n\nWeakness:\n- The multi-round, iterative nature of Router-R1 introduces increased inference latency due to the reasoning steps. Depending on the task, this may be negligible or significant. An analysis on this would help improve the paper.\n - The application shown here are generally QA tasks. A complex routing method may work well on complex scenarios (such as SWE where the router works like a PM). It would be interesting to see application in similar domains.", "was_revised": true, "final_justification": "My original recommendation of this paper is positive. While I and some reviewers are concerned about the cost of the router, I consider that is a factor that might change as technology develops so I wouldn't use that as the main reason to reject this work. \n\nMy recommendation is in between borderline accept/accept, but in general I think the idea is novel and I try not to use \"borderline\" as the \"rating\" intruction suggested.", "rebuttal": "Thanks for your valuable feedback and appreciation of the quality, clarity, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.\n\n## **Inference Latency**\n\nWe thank the reviewer for the insightful and constructive comment.\nWe would like to clarify a key point: **the multi-round, iterative reasoning of Router-R1 does not necessarily lead to increased inference latency.** Instead, it adapts the number of reasoning steps to the difficulty of each task, which can actually reduce latency for simpler questions and avoid unnecessary computation.\n\n**(1) Adaptive reasoning reduces latency for simple tasks**\n\nAs correctly noted by the reviewer, the latency impact depends on the task complexity. We conduct a detailed analysis in **Figure 2a, Section 5.4 (Page 9)**, where we measure the average number of LLM calls across seven QA benchmarks. In comparison with single-hop benchmarks, multi-hop tasks (e.g., Musique, 2WikiMultiHopQA) require more reasoning steps. This task-dependent adaptivity helps minimize latency where possible. (**Lines 315–322**)\n\n**(2) Router-R1 offloads reasoning to lightweight models**\n\nRouter-R1 can reduce latency by delegating parts of the reasoning process to smaller, faster models in the routing pool, instead of relying solely on the policy model. This is particularly beneficial when the policy LLM is larger than some candidate models (a setting that is increasingly relevant in practical deployments).\nAlthough in our current setup the policy LLM is relatively lightweight, the design of Router-R1 naturally supports scalable offloading, where a larger central controller coordinates a pool of efficient LLMs, potentially reducing the end-to-end latency compared to monolithic processing by a single large model.\n\n**(3) Case studies demonstrate latency-efficient routing behavior**\n\nIn Appendix B (Page 22), we present case studies showing that with cost-aware RL training, Router-R1 prefers small models when possible, escalating to larger ones only when necessary. While the main optimization objective is cost, this behavior often correlates with lower latency, since smaller models respond faster in practice.\n\n**(4) Configurable cost-performance tradeoff**\n\nWe further provide a systematic study of different cost coefficients $\\alpha$, showing that Router-R1 allows flexible tuning between performance and efficiency. While this primarily reflects monetary cost, lower cost settings often result in fewer or faster model calls, which can also benefit latency.\n(The “cost” in the table denotes unnormalized raw cost rewards)\n| **Methods**                | **NQ EM↑** | **NQ Cost↓** | **PopQA EM↑** | **PopQA Cost↓** | **HpQA EM↑** | **HpQA Cost↓** | **2wiki EM↑** | **2wiki Cost↓** |\n| -------------------------- | ----------- | ------------- | ------------- | --------------- | ------------- | --------------- | ------------- | --------------- |\n| **Qwen2.5-3B-Instruct**    |             |               |               |                 |               |                 |               |                 |\n| Prompt LLM                 | 0.300       | 20.0          | 0.340         | 17.6            | 0.268         | 20.1            | 0.262         | 20.2            |\n| Largest LLM                | 0.296       | 20.2          | 0.354         | 17.6            | 0.278         | 20.1            | 0.274         | 20.1            |\n| KNN Router                 | 0.262       | 41.0          | 0.222         | 30.3            | 0.224         | 45.4            | 0.196         | 51.8            |\n| MLP Router                 | 0.252       | 35.8          | 0.222         | 21.0            | 0.198         | 29.5            | 0.210         | 41.3            |\n| BERT Router                | 0.230       | 26.0          | 0.192         | 16.3            | 0.216         | 26.0            | 0.206         | 26.3            |\n| RouterDC                   | 0.278       | 57.5          | 0.282         | 21.8            | 0.244         | 62.3            | 0.218         | 40.7            |\n| GraphRouter                | 0.276       | 29.6          | 0.280         | 19.2            | 0.234         | 24.7            | 0.180         | 28.6            |\n| Prompt LLM\\*               | 0.258       | 286.4         | 0.256         | 111.7           | 0.206         | 313.4           | 0.248         | 222.4           |\n| KNN Router\\*               | 0.236       | 102.2         | 0.232         | 49.8            | 0.154         | 133.0           | 0.234         | 99.4            |\n| **Router-R1-Qwen (α=0.0)** | **0.384**   | 149.4         | 0.391         | 97.5            | **0.313**     | 130.4           | **0.393**     | 145.5           |\n| **Router-R1-Qwen (α=0.6)** | **0.384**   | 150.3         | **0.421**     | 75.1            | 0.305         | 123.7           | 0.382         | 113.4           |\n| **Router-R1-Qwen (α=0.7)** | 0.314       | 31.4          | 0.285         | 16.9            | 0.236         | 25.9            | 0.214         | 31.2            |\n| **Router-R1-Qwen (α=0.8)** | 0.317       | 28.7          | 0.275         | 14.7            | 0.221         | 28.5            | 0.183         | 30.1            |\n| **Router-R1-Qwen (α=0.9)** | 0.241       | **5.4**       | 0.238         | **5.3**         | 0.176         | **5.4**         | 0.228         | **6.3**         |\n\n**(5) System-level latency can be minimized in practice**\n\nWhile API-based access to external models (as used in our experiments) may introduce network overhead, this is not inherent to Router-R1. In practical deployment, models can be hosted locally or optimized via standard acceleration techniques. System-level latency engineering is orthogonal to the algorithmic contributions of this work and is left to future work.\n\n**In summary, Router-R1 does not introduce unnecessary latency, but rather provides a task-adaptive mechanism that adjusts inference effort based on question complexity. This design enables a flexible and efficient reasoning strategy that balances latency and accuracy.**\n\n## **Application in Similar Domains**\n\nWe thank the reviewer for the thoughtful and encouraging suggestion. While our current experiments focus on QA tasks, the modular and hierarchical design of Router-R1 is general and well-suited to complex domains such as SWE, where the router can coordinate submodules like a project manager. We are excited to explore such applications in future work, and we believe Router-R1 holds strong potential for such applications.\n\n\n## **The cost of policy LLM**\n\nWe appreciate the reviewer’s thoughtful suggestion. In our current setup, we use a relatively small 3B policy LLM, whereas the candidate models are significantly larger.\nMoreover, as shown in Appendix B (Case Study), the vast majority of output tokens come from the candidate LLMs. The policy LLM primarily acts as a controller, assigning subtasks and summarizing information, with a minor computational footprint. For this reason, we did not include it in the cost calculation.\nThat said, we agree this is a valid and practical extension, especially in industrial settings where larger policy models may be used. In such scenarios, it is indeed feasible to integrate the policy LLM cost into the overall cost analysis or even let Router-R1 jointly optimize for the total cost.\nWe view this as a valuable direction for scaling Router-R1 beyond the academic setting and plan to explore it in future work.\n\n\n\nThank you again for your constructive feedback, and we look forward to further engaging in discussions to improve the quality and impact of our work."}, {"review_id": "RZ4CrzZkGN", "reviewer": "Reviewer_Gukk", "summary": "The paper tackles the limitation of current LLM routers that make a single one-shot model choice. It formulates routing as a sequential decision process in which a router-LLM alternates between internal reasoning,  external model calls and final answering. The router is trained end-to-end with PPO on a composite reward (format, exact-match accuracy, and cost). Experiments on seven QA benchmarks show sizable gains over strong one-shot routers and retrieval baselines, while a cost-coefficient study demonstrates controllable performance/price trade-offs.", "questions": "1. How sensitive is performance to the number or quality of model descriptors supplied in the prompt?\n\n2. Will the approach scales to larger policies with parameter-efficient tuning like LORA?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- Clear formulation of multi-round routing as RL.\n- Lightweight yet effective reward design. \n- Strong empirical results demonstrate generalization beyond in-domain datasets.\n\nWeaknesses\n- Not consider multi-round Router cost in comparison. The cost of policy router in Router-R1 can dominate latency when the called model is a small model.  Other baselines perform a single forward pass, omitting the router’s share makes Router-R1’s cost look lower than it really is. \n- Missing baselines. The comparison set lacks other multi-step methods. For example: Lingjiao Chen, Matei Zaharia, James Zou. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.\n- Router scalability. The router is a 3 B-parameter model trained with full-parameter PPO,  it is unclear whether the approach scales to larger policies with parameter-efficient tuning.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and appreciation of the quality, clarity, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.\n\n## Inference Costs\n\nWe thank the reviewer for raising this important point and would like to offer some clarifications and insights regarding the cost evaluation of Router-R1.\n\n> (1) Goal of the main experiment: performance-oriented comparison\n\nIn the main results table (Table 2), we set the cost reward coefficient $\\alpha = 0.0$ to evaluate the performance ceiling of Router-R1, focusing on its multi-round reasoning capabilities. The primary goal here was not cost-efficiency but answer quality under best-effort routing. Due to space limitations, cost-performance tradeoffs were deferred to a separate analysis.\n\n> (2) Cost-performance tradeoff analysis\n\nWe conduct a comprehensive study on different cost coefficients $\\alpha$ in Router-R1, comparing with baselines in terms of exact match (EM) and raw cost rewards (unnormalized).\nAs shown in the following table, with $\\alpha = 0.0$, Router-R1 achieves the highest EM, demonstrating its effectiveness in performance-first scenarios. As $\\alpha$ increases, the total routing cost drops significantly, while EM degrades moderately—highlighting the controllable trade-off between accuracy and efficiency.\n| **Methods**                | **NQ EM↑** | **NQ Cost↓** | **PopQA EM↑** | **PopQA Cost↓** | **HpQA EM↑** | **HpQA Cost↓** | **2wiki EM↑** | **2wiki Cost↓** |\n| -------------------------- | ----------- | ------------- | ------------- | --------------- | ------------- | --------------- | ------------- | --------------- |\n| **Qwen2.5-3B-Instruct**    |             |               |               |                 |               |                 |               |                 |\n| Prompt LLM                 | 0.300       | 20.0          | 0.340         | 17.6            | 0.268         | 20.1            | 0.262         | 20.2            |\n| Largest LLM                | 0.296       | 20.2          | 0.354         | 17.6            | 0.278         | 20.1            | 0.274         | 20.1            |\n| KNN Router                 | 0.262       | 41.0          | 0.222         | 30.3            | 0.224         | 45.4            | 0.196         | 51.8            |\n| MLP Router                 | 0.252       | 35.8          | 0.222         | 21.0            | 0.198         | 29.5            | 0.210         | 41.3            |\n| BERT Router                | 0.230       | 26.0          | 0.192         | 16.3            | 0.216         | 26.0            | 0.206         | 26.3            |\n| RouterDC                   | 0.278       | 57.5          | 0.282         | 21.8            | 0.244         | 62.3            | 0.218         | 40.7            |\n| GraphRouter                | 0.276       | 29.6          | 0.280         | 19.2            | 0.234         | 24.7            | 0.180         | 28.6            |\n| Prompt LLM\\*               | 0.258       | 286.4         | 0.256         | 111.7           | 0.206         | 313.4           | 0.248         | 222.4           |\n| KNN Router\\*               | 0.236       | 102.2         | 0.232         | 49.8            | 0.154         | 133.0           | 0.234         | 99.4            |\n| **Router-R1-Qwen (α=0.0)** | **0.384**   | 149.4         | 0.391         | 97.5            | **0.313**     | 130.4           | **0.393**     | 145.5           |\n| **Router-R1-Qwen (α=0.6)** | **0.384**   | 150.3         | **0.421**     | 75.1            | 0.305         | 123.7           | 0.382         | 113.4           |\n| **Router-R1-Qwen (α=0.7)** | 0.314       | 31.4          | 0.285         | 16.9            | 0.236         | 25.9            | 0.214         | 31.2            |\n| **Router-R1-Qwen (α=0.8)** | 0.317       | 28.7          | 0.275         | 14.7            | 0.221         | 28.5            | 0.183         | 30.1            |\n| **Router-R1-Qwen (α=0.9)** | 0.241       | **5.4**       | 0.238         | **5.3**         | 0.176         | **5.4**         | 0.228         | **6.3**         |\n\nWe will include this expanded analysis in the camera-ready version for completeness.\n\n> (3) Ensuring fairness in comparison\n\nTo ensure fairness when comparing with single-step methods, we introduced two multi-round routing baselines (Prompt LLM*, KNN Router*) that simulate step-by-step decomposition.\nFurthermore, we note that most prior LLM router works only include the cost of routed LLM calls and similarly omit the router's cost in their analysis[1][2].\nTo ensure consistency and fair comparison, we follow this practice and focus our cost evaluation on the routed candidate LLMs, which dominate the total computational overhead.\n\n> (4) Policy LLM cost is minimal, and Router-R1 enables scalable offloading\n\nWe acknowledge the concern about the policy LLM’s potential latency or cost, especially when routing to smaller models. However, in our current setup, the policy LLM is only 3B, significantly smaller than the candidate LLMs.\nMoreover, as shown in Appendix B (Page 23), the policy LLM performs only lightweight coordination tasks, and the vast majority of output tokens are generated by candidate LLMs.\nIn fact, Router-R1 can reduce latency by delegating parts of the reasoning process to smaller, faster models in the routing pool, instead of relying solely on the policy model.\nThis architecture is particularly beneficial when the policy LLM is larger than some candidate models (a setting increasingly common in practical deployments). Even though our current policy model is lightweight, the Router-R1 framework naturally supports scalable offloading, where a central controller coordinates efficient model usage. This can even lead to lower latency and cost than using a single monolithic model, especially for simple tasks. Interestingly, this resembles ideas in speculative decoding[3], where small models generate drafts and large models verify them, reducing end-to-end inference time and cost.\n\nIn summary, Router-R1 adopts a flexible and extensible architecture that balances reasoning quality and computational efficiency. We take care to ensure fair cost comparison, and we appreciate the reviewer’s thoughtful feedback, which we will further address in the final version.\n\n[1] GraphRouter: A Graph-based Router for LLM Selections. ICLR 2025.\n\n[2] RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models. NeurIPS 2024.\n\n[3] Fast Inference from Transformers via Speculative Decoding. ICML 2023.\n\n## Missing Baselines\n\nWe thank the reviewer for pointing out this baseline.\nWe did evaluate FrugalGPT in our experiments. However, we chose not to include it in the main results table, as its design prioritizes cost reduction, whereas Router-R1 is primarily aimed at maximizing performance under multi-round decomposition, especially in complex reasoning settings.\nWe agree that including FrugalGPT helps contextualize our method across different optimization objectives. Below we provide its results for completeness:\n| Model           | NQ    | TriviaQA | PopQA | HotpotQA | 2wikimultihop | Musique | BAMB  | Avg   |\n|-------------------|-------|----------|-------|----------|----------------|---------|--------|--------|\n| FrugalGPT         | 0.265 | 0.562    | 0.362 | 0.234    | 0.268          | 0.103   | 0.430  | 0.318 |\n| Router-R1-Qwen    | 0.384 | 0.699    | 0.391 | 0.313    | 0.393          | 0.150   | 0.531  | 0.409 |\n\nWe will include this comparison in the camera-ready version to better position our method within the broader landscape of routing strategies.\n\n## Router scalability\n\nThank you for raising this question.\nOur method is built upon the VERL framework, which already supports parameter-efficient tuning methods such as LoRA. This means that Router-R1 can be directly scaled to larger policy models using LoRA or similar approaches, without requiring full-parameter training.\nHowever, our focus in this work is not on optimizing parameter efficiency, but rather on proposing a multi-round, cost-aware LLM routing strategy. Exploring how Router-R1 behaves under different tuning paradigms (e.g., LoRA or QLoRA) is certainly interesting, and we consider it promising future work.\n\n## Sensitivity of Model descriptors\n\nThank you for the insightful question.\nTo evaluate the sensitivity of model descriptors, we conducted an additional experiment where we compressed the original prompts (shown in Appendix B) to retain only basic attribute information. For example:\n> LLaMA-3.1-70B-Instruct: A 70B state-of-the-art model for multilingual dialogue and reasoning, excelling in benchmark evaluations.\n\nCompared to the more elaborate descriptors used in our default setup, this version is significantly more concise.\nThe results are shown in the table below (on Router-R1-Qwen). We observe that such modification has minimal impact on overall performance, indicating that Router-R1 is not highly sensitive to the model descriptors.\n| Setting         | NQ    | TriviaQA | PopQA | HotpotQA | 2wikimultihop | Musique | BAMB  | Avg   |\n|-----------------|-------|----------|-------|----------|----------------|---------|--------|--------|\n| Modification  | 0.384 | 0.707    | 0.409 | 0.309    | 0.385          | 0.163   | 0.523  | 0.411 |\n| Default | 0.384 | 0.699    | 0.391 | 0.313    | 0.393          | 0.150   | 0.531  | 0.409 |\n\nAs discussed in the paper (**Line 188–195**), model descriptors primarily serve as a cold-start prior, helping the router bootstrap the learning process. As training progresses, Router-R1 learns to distinguish the strengths and weaknesses of candidate LLMs through interaction, rather than relying solely on initial descriptors.\nWe will include this result in the camera-ready version.\n\n\nThank you again for your constructive feedback, and we look forward to further engaging in discussions to improve the quality and impact of our work."}, {"review_id": "7QXGEPo4rO", "reviewer": "Reviewer_7Web", "summary": "Router-R1 is a multi-round collaborative routing and aggregation framework for LLMs, designed to overcome the limitations of existing routing approaches that assign each query to a single model in a one-shot manner. It formulates multi-model collaboration as a sequential decision process and introduces a *think–route* alternating mechanism: the router itself is an LLM with reasoning capabilities, which iteratively alternates between internal reasoning (*Think*) and model invocation (*Route*) across multiple rounds to progressively construct the final answer. To enhance training efficiency, Router-R1 employs a lightweight regularized reward function composed of format rewards, answer correctness rewards, and model invocation cost penalties, effectively guiding the model to balance accuracy and computational cost.", "questions": "Please see Weakness.", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The introduction of the cost reward is interesting and aligns well with the motivation for routing methods.\n2. Experiments are comprehensive, but multi-run repetition experiments to validate stability appear lacking.\n3. The paper is well written and clearly presented.\n\n**Weaknesses**\n\n1. The LLM routing pool lacks some state-of-the-art models, such as GPT-4o and DeepSeek-v3.\n2. Although NeurIPS does not require reviewers to check arXiv versions of papers, I found a version of this paper on arXiv. I would like the authors to explain the discrepancy between the experimental results in the two versions, where the results are very different. If the paper is accepted, which version’s experimental results will be used in the camera-ready submission? (This will not affect my evaluation of the paper.)\n3. In Section 5.3, the authors demonstrate the effect of introducing new models at test time for Router-R1. I am curious how Router-R1 would perform if new models, such as LLaMA3-ChatQA-1.5-8B, were incorporated during training.\n4. Router-R1 requires multiple calls to LLMs, but Figure 2 shows that the average LLM API call count is only slightly above 1.0. This is puzzling because it implies most queries require only one call. Is multi-round calling really necessary? Does the current dataset adequately demonstrate the effectiveness of the proposed method? Additionally, how does Router-R1’s runtime efficiency compare with single-round invocation models?\n5. The novelty of the paper is limited (though the introduction of the cost reward is commendable). While Router-R1 is presented as a routing method, from a modeling perspective, it resembles tool learning. For example, the authors mention search r1. The main difference here is that the authors treat the models in the LLM routing pool as callable tools. Although the distinction between these two approaches is discussed in Section 2.2, I find it not fundamentally different. The authors may consider modifying search r1 to better fit the routing scenario as a new baseline.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and appreciation of the quality, clarity, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.\n\n## The LLM routing pool lacks some state-of-the-art models\n\nWe sincerely thank the reviewer for raising this point.\nDue to limited GPU resources and budget constraints, our experiments rely on public LLM APIs to access candidate models (Lines 257–263). We carefully selected a representative and diverse set of models. However, some cutting-edge models like GPT-4o are either unavailable via the APIs we use or incur prohibitive costs and latency, making them infeasible for repeated runs in academic settings.\n\nMoreover, Router-R1 is inherently model-agnostic—its routing policy can generalize to any future LLMs, including stronger ones, without requiring algorithmic changes. We hope our work will inspire industrial labs with greater compute resources to further evaluate Router-R1 at scale with larger routing pools.\n\nAs for reproducibility, we ran multiple repetitions and observed consistently low variance (typically within ±0.01 EM), which does not affect the conclusions. Reported results are averaged across runs, and we omit variance values for clarity. We will also open-source our code upon publication to support reproducibility.\n\n## The results in arXiv version\n\nThanks for your feedback.\nTo ensure compliance with NeurIPS's double-blind review policy and maintain fairness in the reviewing process, we have communicated the relevant details to the Area Chair.\nWe appreciate your attention and understanding regarding this matter.\n\n## New model incorporation\n\nThank you for the suggestion. While we primarily evaluated Router-R1’s generalization to new models introduced at test time (Table 3), we also experimented with incorporating a new model (LLaMA3-ChatQA-1.5-8B) during training.\nBelow we report the average EM scores across datasets when this model is added to the routing pool at training time (Router-R1-Qwen):\n| Setting         | NQ    | TriviaQA | PopQA | HotpotQA | 2wikimultihop | Musique | BAMB  | Avg   |\n|-----------------|-------|----------|-------|----------|----------------|---------|--------|--------|\n| Modification | 0.380 | 0.719    | 0.397 | 0.295    | 0.381          | 0.167   | 0.589  | 0.418 |\n| Default | 0.384 | 0.699    | 0.391 | 0.313    | 0.393          | 0.150   | 0.531  | 0.409 |\n\nThis result aligns closely with the test-time-only setting (Table 3), suggesting that Router-R1 generalizes well to new models and remains robust even when the routing pool is dynamically extended.\n\n## Multi-round Calling Frequency\n\nThe number of LLM calls required per query depends on multiple factors, notably **(1) the complexity of the dataset**, and **(2) the composition of the routing pool**.\nTo understand how task complexity affects routing depth, we conduct a quantitative analysis using the average EM performance across all candidate LLMs on each dataset. This gives a proxy of dataset difficulty (harder datasets typically have lower EM scores). \n| Dataset | NQ    | TriviaQA | PopQA | HotpotQA | 2Wiki | Musique | BAMB  |\n| ------- | ----- | -------- | ----- | -------- | ----- | ------- | ----- |\n| Avg EM  | 0.257 | 0.537    | 0.220 | 0.296    | 0.298 | 0.133   | 0.412 |\n\nThis aligns with intuition (e.g., multi-hop datasets like Musique are harder), and when correlated with Figure 2a, it shows that more difficult datasets tend to invoke more LLM calls. This supports the motivation for our multi-round routing framework, which flexibly allocates computational resources based on query difficulty.\n\nMoreover, the routing pool itself affects the observed call frequency. Due to API constraints, we use a pool of API-accessible LLMs; stronger routing pools would naturally lead to fewer calls per query. **However, this does not undermine the necessity or effectiveness of multi-round routing, especially when combined with cost-aware rewards.** Importantly, multi-round routing enables behaviors that one-shot methods cannot learn. For example, with cost reward enabled, Router-R1 often learns to first query smaller models, escalating to larger ones only when needed (Appendix B.2). This adaptive, efficient strategy is not achievable with one-shot routing. Finally, while current experiments focus on QA tasks, we believe the general-purpose design of Router-R1 allows easy extension to more complex reasoning domains in future work.\n\n## Runtime efficiency\n\nWe acknowledge the importance of runtime efficiency. However, a direct comparison between Router-R1 and single-round baselines in terms of wall-clock runtime would be unfair, as our setup involves heterogeneous inference environments. Candidate LLMs are accessed via APIs (e.g., Together AI), which often apply advanced backend optimizations for fast inference. In contrast, Router-R1's policy model runs locally on a single GPU without such system-level optimizations.\nInstead, we report inference cost w.r.t. raw cost rewards, which fairly reflects efficiency and is commonly used in prior work. As shown in the following table, Router-R1 achieves a strong cost-performance trade-off under this metric.\n| **Methods**                | **NQ EM↑** | **NQ Cost↓** | **PopQA EM↑** | **PopQA Cost↓** | **HpQA EM↑** | **HpQA Cost↓** | **2wiki EM↑** | **2wiki Cost↓** |\n| -------------------------- | ----------- | ------------- | ------------- | --------------- | ------------- | --------------- | ------------- | --------------- |\n| **Qwen2.5-3B-Instruct**    |             |               |               |                 |               |                 |               |                 |\n| Prompt LLM                 | 0.300       | 20.0          | 0.340         | 17.6            | 0.268         | 20.1            | 0.262         | 20.2            |\n| Largest LLM                | 0.296       | 20.2          | 0.354         | 17.6            | 0.278         | 20.1            | 0.274         | 20.1            |\n| KNN Router                 | 0.262       | 41.0          | 0.222         | 30.3            | 0.224         | 45.4            | 0.196         | 51.8            |\n| MLP Router                 | 0.252       | 35.8          | 0.222         | 21.0            | 0.198         | 29.5            | 0.210         | 41.3            |\n| BERT Router                | 0.230       | 26.0          | 0.192         | 16.3            | 0.216         | 26.0            | 0.206         | 26.3            |\n| RouterDC                   | 0.278       | 57.5          | 0.282         | 21.8            | 0.244         | 62.3            | 0.218         | 40.7            |\n| GraphRouter                | 0.276       | 29.6          | 0.280         | 19.2            | 0.234         | 24.7            | 0.180         | 28.6            |\n| Prompt LLM\\*               | 0.258       | 286.4         | 0.256         | 111.7           | 0.206         | 313.4           | 0.248         | 222.4           |\n| KNN Router\\*               | 0.236       | 102.2         | 0.232         | 49.8            | 0.154         | 133.0           | 0.234         | 99.4            |\n| **Router-R1-Qwen (α=0.0)** | **0.384**   | 149.4         | 0.391         | 97.5            | **0.313**     | 130.4           | **0.393**     | 145.5           |\n| **Router-R1-Qwen (α=0.6)** | **0.384**   | 150.3         | **0.421**     | 75.1            | 0.305         | 123.7           | 0.382         | 113.4           |\n| **Router-R1-Qwen (α=0.7)** | 0.314       | 31.4          | 0.285         | 16.9            | 0.236         | 25.9            | 0.214         | 31.2            |\n| **Router-R1-Qwen (α=0.8)** | 0.317       | 28.7          | 0.275         | 14.7            | 0.221         | 28.5            | 0.183         | 30.1            |\n| **Router-R1-Qwen (α=0.9)** | 0.241       | **5.4**       | 0.238         | **5.3**         | 0.176         | **5.4**         | 0.228         | **6.3**         |\n\n\n## Novelty\n\nOur work is motivated by the need for multi-round, adaptive routing in complex reasoning tasks—something one-shot routing fails to handle effectively. This requires a centralized policy LLM to coordinate step-wise model calls, where each candidate LLM functions as a specialized reasoning module rather than a generic tool. Inspired by DeepSeek-R1, we introduce reinforcement learning to train the policy LLM, enabling fine-grained and adaptive routing. To our knowledge, Router-R1 is the first to apply R1-style RL to LLM routing, unifying multi-round reasoning with cost-aware inference (**Lines 31–39**). This sets it apart from both tool learning paradigms and prior router-style works.\n\nA core innovation is our cost reward design, which allows Router-R1 to learn efficient, hierarchical routing behaviors—favoring small models and escalating only when needed (Appendix B.2). This dynamic trade-off between cost and performance is crucial in real-world, cost-sensitive deployments and cannot be achieved by fixed, one-shot methods.\n\nWhile Search-R1 also uses RL, it addresses tool use rather than model routing. Unlike tool selection among heterogeneous modules (e.g., calculators)[1,2,3], Router-R1 routes across semantically similar LLMs differing in capability and cost, which is a fundamentally different challenge. We also include Search-R1 as a baseline (Appendix A.2), but emphasize that Router-R1 is a standalone contribution tailored for multi-round LLM routing, not an extension of tool learning.\n\n[1] Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023.\n\n[2] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR 2024.\n\n[3] ToolRL: Reward is All Tool Learning Needs. arXiv 2025.\n\nThank you again for your constructive feedback, and we commit to enriching the content of the paper based on the above in the revised version. We look forward to further engaging in discussions to improve the quality and impact of our work."}, {"review_id": "bYQ22qxXJ8", "reviewer": "Reviewer_f8xo", "summary": "This paper proposes Router-R1, a new way of routing LLMs in multi-turns. Their solution is based on RL, where the router is trained to maximize a combination of different rewards.", "questions": "How does Eq 5 mitigate reward hacking?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The approach seems novel in two different ways:\n    1. It uses RL for routing;\n    2. It enables multi-turn routing which has not been explored before.\n\nWeaknesses:\n- The main issue with the paper is the following. One main experimental results is missing (but could probably be easily built from other results in the paper: the cost vs performance Pareto curves (shown, for example, in the RouterBench paper you cited). Ideally, you would have a plot for each benchmark and have a curve for each routing strategy. From that, it will be much clearer how your method compare to others; from the current results, I can't tell which method is performing better. This is the main reason I am suggesting borderline reject.\n- In Eq 4 T_out is assumed to be known which is a limitation. Maybe you can predict it? See for example this paper: \n\nSomerstep, Seamus, et al. \"CARROT: A Cost Aware Rate Optimal Router.\" arXiv preprint arXiv:2502.03261 (2025).", "was_revised": true, "final_justification": "They authors provided a new set of results I asked. Even though I have mixed feelings about the full set of results, this work offers a new approach to routing, which can handle multi-turn interactions (which is novel). I will increase my rating to 4.", "rebuttal": "Thanks for your valuable feedback and appreciation of the clarity, significance, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.\n\n## cost vs performance Pareto curves\n\nThank you for pointing this out, and we fully agree that visualizing the cost–performance trade-off is valuable for understanding routing strategy behavior.\n\n**Router-R1 was originally designed to optimize for answer performance**, and thus our main results (Table 2) focus on the α = 0 setting, where cost is not explicitly penalized. **The cost-aware reward term was included to give Router-R1 the flexibility to balance cost and performance when needed**, which enhances its practical value, though it was not the main focus of our comparison experiments due to space constraints.\n\nNevertheless, we have already conducted an extensive study varying the cost coefficient α, evaluating how Router-R1 performs across different efficiency constraints. While we are unfortunately unable to include figures during the rebuttal phase (per NeurIPS policy), we include below a representative table of EM vs raw cost reward (unnormalized) across α values and baselines:\n\n| **Methods**                | **NQ EM↑** | **NQ Cost↓** | **PopQA EM↑** | **PopQA Cost↓** | **HpQA EM↑** | **HpQA Cost↓** | **2wiki EM↑** | **2wiki Cost↓** |\n| -------------------------- | ----------- | ------------- | ------------- | --------------- | ------------- | --------------- | ------------- | --------------- |\n| **Qwen2.5-3B-Instruct**    |             |               |               |                 |               |                 |               |                 |\n| Prompt LLM                 | 0.300       | 20.0          | 0.340         | 17.6            | 0.268         | 20.1            | 0.262         | 20.2            |\n| Largest LLM                | 0.296       | 20.2          | 0.354         | 17.6            | 0.278         | 20.1            | 0.274         | 20.1            |\n| KNN Router                 | 0.262       | 41.0          | 0.222         | 30.3            | 0.224         | 45.4            | 0.196         | 51.8            |\n| MLP Router                 | 0.252       | 35.8          | 0.222         | 21.0            | 0.198         | 29.5            | 0.210         | 41.3            |\n| BERT Router                | 0.230       | 26.0          | 0.192         | 16.3            | 0.216         | 26.0            | 0.206         | 26.3            |\n| RouterDC                   | 0.278       | 57.5          | 0.282         | 21.8            | 0.244         | 62.3            | 0.218         | 40.7            |\n| GraphRouter                | 0.276       | 29.6          | 0.280         | 19.2            | 0.234         | 24.7            | 0.180         | 28.6            |\n| Prompt LLM\\*               | 0.258       | 286.4         | 0.256         | 111.7           | 0.206         | 313.4           | 0.248         | 222.4           |\n| KNN Router\\*               | 0.236       | 102.2         | 0.232         | 49.8            | 0.154         | 133.0           | 0.234         | 99.4            |\n| **Router-R1-Qwen (α=0.0)** | **0.384**   | 149.4         | 0.391         | 97.5            | **0.313**     | 130.4           | **0.393**     | 145.5           |\n| **Router-R1-Qwen (α=0.6)** | **0.384**   | 150.3         | **0.421**     | 75.1            | 0.305         | 123.7           | 0.382         | 113.4           |\n| **Router-R1-Qwen (α=0.7)** | 0.314       | 31.4          | 0.285         | 16.9            | 0.236         | 25.9            | 0.214         | 31.2            |\n| **Router-R1-Qwen (α=0.8)** | 0.317       | 28.7          | 0.275         | 14.7            | 0.221         | 28.5            | 0.183         | 30.1            |\n| **Router-R1-Qwen (α=0.9)** | 0.241       | **5.4**       | 0.238         | **5.3**         | 0.176         | **5.4**         | 0.228         | **6.3**         |\n\nFrom this table, we observe:\n- At α = 0.0, Router-R1 consistently achieves the highest EM across benchmarks, confirming its strong performance orientation.\n- As α increases, cost drops significantly while EM gradually decreases, showing a clear trade-off.\n- Compared to other baselines, Router-R1 achieves better performance when cost is lightly constrained (small α), and more efficient routing under tighter budgets (larger α).\nThese trends echo the Pareto-like patterns seen in works like RouterBench, and validate the strength and flexibility of our cost-aware reward design. We will include full cost–performance plots and analysis in the camera-ready version.\n\n\n## T_out is assumed to be known which is a limitation\n\nThank you for the suggestion. We believe there may be a slight misunderstanding here.\n\nIn contrast to approaches like CARROT that rely on explicit cost and performance prediction to guide routing, Router-R1 does not require predicting T_out in advance. Instead, we adopt a reinforcement learning framework where the policy LLM learns to select the most suitable candidate LLMs by observing the actual outcomes of its decisions.\n\nSpecifically, during training, the policy LLM selects a candidate LLM, receives its response (whose token count T_out is observed at runtime, not predicted), and incorporates that response into its own context. The reward is computed based on the final answer accuracy and the actual token cost (T_out). This reward then guides policy learning via RL, without requiring any explicit modeling of future token usage.\n\nIn other words, Router-R1 delegates cost-performance estimation to the policy LLM itself, which is already a powerful model capable of learning complex trade-offs through experience, rather than relying on heuristic or learned cost predictors.\n\nWe greatly appreciate the reviewer’s suggestion. Integrating ideas from CARROT, such as lightweight cost prediction, to further improve training efficiency or cold-start performance could be a promising future direction, and we plan to explore this in follow-up work.\n\n## How does Eq 5 mitigate reward hacking?\n\nThank you for the helpful question.\n\nWhile Eq. 5 defines the overall reward structure, we would like to clarify that Router-R1 adopts a hierarchical reward mechanism that is not explicitly shown in Eq. 5 for brevity (**as noted in Line 167**). This design plays a key role in mitigating reward hacking and improving training stability (**described in Line 166–172**).\n\nConcretely, Router-R1 assigns strict priorities among the three reward components: format > outcome > cost. If the format reward is -1 (e.g., due to invalid or unparseable output), the outcome and cost rewards are ignored (set to 0); If the outcome is wrong (i.e., outcome reward = 0), then cost reward is only considered if the outcome is correct, otherwise the total reward only depends on the format reward.\n\nConsider a situation where the output has the correct format but produces an incorrect answer. Without hierarchical reward, the model could still receive a positive total reward by minimizing the number of output tokens (i.e., reducing total T_out) and maximizing the cost reward. Over time, this may encourage the model to learn a shortcut: always produce very short, low-cost but incorrect answers, as this strategy still yields non-zero reward. This is a classic case of reward hacking. With hierarchical reward, however, once the outcome is incorrect, the cost reward is completely disregarded, and the total reward remains zero regardless of how low the cost is. This effectively eliminates such degenerate behaviors and ensures that correctness is prioritized over cheapness.\n\nIn our experiments, we found that this hierarchical reward structure significantly stabilizes training and effectively mitigates reward hacking, leading to better final performance and robustness. We appreciate the reviewer for pointing out this important detail.\n\nThank you again for your constructive feedback, and we commit to enriching the content of the paper based on the above in the revised version. We look forward to further engaging in discussions to improve the quality and impact of our work."}]

Author rebuttal:

Meta-review: This paper proposes Router-R1, a RL framework for LLM routing and aggregation. Unlike single-round routers that assign a query to a single model, Router-R1 formulates this as a sequential decision-making process. The router itself is a reasoning LLM that alternates between internal deliberation and external model calls. It learns a policy to select the most suitable model for a given sub-task, integrating the response into its context to progressively build a final answer. The training is guided by a novel, hierarchical reward function that balances performance (answer quality), format, and cost. Experiments on seven QA benchmarks demonstrate that Router-R1 outperforms strong baselines and provides a flexible, cost-aware trade-off.

However, a number of significant weaknesses were raised, including:
- Reviewers noted missing baselines, a lack of runtime efficiency analysis, and a need for clearer Pareto curves to illustrate the cost-performance trade-off.
- One reviewer argued the method is not fundamentally different from existing tool-learning frameworks.
- The paper's experiments are limited to a small 3B router model, and it is unclear how the approach would perform with larger policies or with different parameter-efficient tuning methods.
- The multi-round, iterative nature of the method may introduce significant latency and cost, which the authors' analysis does not fully address.

The authors provided rebuttals to address these concerns.
- The authors added results for an expanded analysis of the cost-performance trade-off across different alpha values. They also added results for individual models to contextualize the trade-offs
- The authors clarified that Router-R1 is distinct from tool-learning methods like Search-R1 because it learns to route among semantically similar LLMs. They emphasized that the core novelty lies in applying RL to unify multi-round reasoning with cost-aware routing.
- The authors argued that Router-R1's adaptive nature can reduce latency for simpler tasks, though they acknowledge that direct wall-clock runtime comparisons were not possible due to their experimental setup. They also noted that the cost of the small 3B router model is minimal compared to the larger models in the pool.

The paper presents a original idea with a clear design and reasonable empirical results. The authors' rebuttals were thorough and responsive, addressing most of the reviewers' concerns with additional experiments and insightful clarifications. The introduction of cost-aware RL for LLM routing is a practical contribution. While some concerns about efficiency and a lack of certain comparisons remain, they are not severe enough to warrant rejection. The potential for this work to inspire future research in multi-round, cost-aware LLM routing makes it a valuable contribution to the NeurIPS community. The work's strengths outweigh its weaknesses.
Decision: Accept (poster)