DiSRouter: Distributed Self-Routing for LLM Selections
Authors: Hang Zheng
Hongshen Xu
Yongkai.lin
Shuai Fan
Lu Chen
Kai Yu
Venue: ICLR 2026 Poster (2026)
Keywords: query routing
model selection
distributed system
self-awareness of LLM
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce DiSRouter, a distributed system where LLMs leverage "self-awareness" to route queries among themselves, outperforming conventional centralized routers.

Abstract: The proliferation of Large Language Models (LLMs) has created a diverse ecosystem of models with highly varying performance and costs, necessitating effective query routing to balance performance and expense. Current routing systems often rely on a centralized external router trained on a fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully understand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, a query traverses a network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness—its ability to judge its competence. This distributed design offers superior flexibility, scalability, and generalizability. To enable this, we propose a two-stage Self-Awareness Training pipeline that enhances each LLM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effectively distinguishes between easy and hard queries, and shows strong generalization to out-of-domain tasks. Our work validates that leveraging an LLM's intrinsic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems.

Published as a conference paper at ICLR 2026
DISROUTER: DISTRIBUTED SELF-ROUTING FOR LLM SELECTIONS
Hang Zheng's ; Hongshen Xu Yongkai Lin Shuai Fan Lu Chen 12.3.5f Chen!.2.3.5 Kai Yu 13.5t X-LAACE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Shanghaii Innovation Institution, Shanghai, China
Jiangsu Key Lab of Language Computing. Suzhou, China
4 AAlSpeech Co., Ltd., Suzhou, China Suzhou Laboratory, Suzhou, China (azure123, chenlusz, kai, mai.yu)es yu)es wtw.:du.uccooo edu.
ABSTRACT
The proliferation of Large Language Models (LLMs) has created a diverse ecosys- tem of models with highly varying performance and costs, necessitating effective query routing balance performance and expense. Current routing systems often rely centralized external router trained on fixed set of LLMs, making them inflexible and prone to poor performance since the small router can not fully un- derstand the knowledge boundaries of different LLMs. We introduce DiSRouter (Distributed Self-Router), a novel paradigm that shifts from centralized control to distributed routing. In DiSRouter, query traverses network of LLM agents, each independently deciding whether to answer or route to other agents based on its own self-awareness-its ability to judge its competence. This distributed de- sign offers superior flexibility, scalability, and generalizability. To enable this, we propose two-stage Self-Awareness Training pipeline that enhances each L.LM's self-awareness. Extensive experiments demonstrate that DiSRouter significantly outperforms existing routing methods in utility across various scenarios, effee- tively distinguishes between easy and hard queries, and shows strong generaliza- tion to oft-oan tasks. Our work validates that leveraging an L.LM's 's intrin- sic self-awareness is more effective than external assessment, paving the way for more modular and efficient multi-agent systems.
INTRODUCTION
The proliferation of Large Language Models (LLMs) has marked a new era, demonstrating remark- able capabilities across tasks. However, the superior performance of of-th-aa-t- LLMs is often accompanied by prohibitive computational and financial costs, limiting their widespread adoption. Concurrently, a rich and diverse ecosystem of LLMS has emerged, ranging from small, efficient models for edge deployment to powerful, cloud-based L.MSs (Chen et al. 2025). This diversity presents both critical challenge and significant opportunity: how navigate the LLMs pool to select the most cost-effective agent for given query without compromising performance?
This challenge, often termed "query routing" or "model selection," " has drawn significant research attention -arraggt-RReil et al., 2025) and found industrial application, with systems like the latest GPT-5 employing a unified architecture with multiple models and real- -ime router (OpenAl|2000 2025). Conventional approaches typically utilize centralized router, such as scoring model, to assess query difficulty or predict the potential performance of specific I.I,, subsequently dispatching queries to the smallest capable model (Ong et al., 2024 Chen et al., [023 [eng et al. 2024). Despite their effectiveness, these centralized systems suffer from two fundamental limitations: (I) Limited Flexibility of External Routers. External routers are usually trained on fixed set of candidate LLMs. Once trained, any modification, such as adding or updating an agent, necessitates a costly retraining of the entire routing system, which makes the system rigid and undermines scalability.
"Han Zheng and Hongshen Xu contribute equally to this work.
Lu Chen and Kai Yu are the corresponding authors.

---

Published as a conference paper at ICLR 2026
(a)
Centralized
(b) Distributed
ILMI
Cascade
Queries
Queries
Router
LLMZ
LLM3
LLMZ
Tree
Queries
LLME
Inflexibilitt
Router
x
Original LLM
Unawareness
Self-route LLM
(More)
Student
Mathematician Physicist
Comparison between (a) centralized and (b) distributed routing. Centralized routing relies on an external router, which has limitations in adapting to new LLMs (Inflexibility) and ac- curately assessing the knowledge boundaries of each LLM (Unawareness). contrast, distributed routing is not dependent on an external router and can be organized into various structures.
Figure 1:
(2) Inaccurate Capability Assessment. External routers are often implemented as relatively small models. Lacking the capacity to fully comprehend the intrinsic knowledge boundaries of large- scale LLMs, they struggle to accurately assess the capabilities of each model for a given query and determine the most suitable model. Consequently, their limited capability of assessment often becomes major bottleneck for the routing system.
To overcome these limitations, we propose a paradigm shift from centralized control to distributed routing, as shown in Figure [ We introduce DiSRouter (Distributed Self-Router), a novel frame- work that replaces a single central router with a network of agents capable of self-assessment and autonomous routing decisions. The core idea is to empower each agent to evaluate its own compe- tence on a given query and independently decide whether to answer it or delegate it onward, rather than relying on external router make routing decisions. In experiments, we instantiate DiSRouter as a cascade of homogeneous L.LM agents of increasing sizes. Each smaller agent first attempts assess and solve the query; if determines that it cannot provide reliable answer, autonomously routes the query to the next larger agent in the cascade. This design fosters fully distributed, flexible, and modular system that naturally supports the seamless addition or removal of agents. While our evaluation focuses on the cascade architecture, the DiSRouter framework is not limited to this setting. principle, DiSRouter can instantiated with alternative network topolo- gies, such as tree mesh structures, a prospect we explore further in the discussion section $6
The effectiveness of DiSRouter highly depends on the self-awareness of each agent, namely its ability to accurately assess the boundaries of its own capabilities. To this end, we propose Self- Awareness Training pipeline comprising Supervised Fine- Tuning (SFT) followed by Reinforcement Learning (RL). A key innovation is the design of a localized reward function that enables each agent to be trained completely independently and in parallel. To further align the model with diverse scenario requirements, the reward function incorporates global preference factor (a), allowing the entire system to adaptively shift its collective behavior from performance-firg to a cost-first stance based on user preference, without requiring any direct inter-agent communication.
Our contributions are summarized as follows:
We propose DiSRouter, novel distributed, self-routing system that achieves superior mod- ularity, scalability, and robustness compared to traditional centralized routers.
We design Self-Awareness Training pipeline with localized reward that enables fully independent and parallel training of reliable agents in a distributed routing system.
Our extensive experiments demonstrate that DiSRouter effectively balances performance and cost, while dynamically adapting its routing strategies to user-defined preferences.
2 PROBLEM FORMULATION
The primary objective of a routing system is to develop an optimal strategy that assigns each incom- ing query to the most suitable agent, effectively balancing system performance and cost.
2

---

Published as a conference paper at ICLR 2026
2.1 ROUTING SYSTEM
A routing system comprises three main components: a set of queries X, pool of models M, and a routing policy и. The policy maps each query Ij to specific agent T(z E M, denoted as
X M. Let's consider set of N queries X {ær) and pool of K LLMs M {m,) where X is drawn from data distribution D. We define the score of model m; query Ij as A(m;, I;)) E {0, 1). where I denotes a correct solution and O otherwise. To facilitate comparison of costs from different model sources, such API call prices GPU seconds, assign predefined and fixed cost Q E [0, 1] to each model m,. Without loss of generality, we assume the models are ordered by increasing cost, such that my has the lowest cost and mg has the highest cost. In addition to the inference cost of the models, the routing process itself incurs an extra overhead, which typically negligible (as discussed in Appendix A.I). and therefore not considered in this work. Thus, the total cost function for an agent m, is simply C(m,) Ci.
The system's overall performance and cost can then calculated as follows, where ni; = T Txx)::
N
N ((m.)
1
Performance
A(m.,x.)
Cost = N 1=0
(1)
N
We adopt a widely used metric, Utility. to quantify the effectiveness of a routing policy by combin-
ing both Performance and Cost (Tsiourvas et al. 2025 Xu et al. 2025), which defined as:
Utility = Per formanccce - a Cost
where [0, 1] is hyperparameter named "preference factor" representing the user's preference between performance and cost. A smaller a prioritizes accuracy, while larger a emphasizes cost efficiency. Specifically, we define three scenarios referencing that in GraphRouter (Feng et al., 2024): "Performance First", "Balance", and "Cost First", with set to 0.2, 0.5, and 0.8, respectively.
2.2
ROUTING SYSTEM ARCHITECTURE
In this section, we delineate the architectures of centralized and our proposed distributed routing.
2.2.1
CENTRALIZED ROUTING
In centralized routing architecture, a single entity, typically trained model set of predefined rules, acts as the central router. This router embodies the routing policy π and independently assigns each query a specific model in the pool. this setup, the overarching goal of the router is to learn an optimal routing mapping x" that maximizes the expected utility over the data distribution D:
π = arg mar (m)D(A(T()). - a C(m(x)))
2.2.2 DISTRIBUTED ROUTING
In distributed architecture, the system operates without central router. Instead, the pool of agents forms a routing network where each agent independently decides whether to execute the query itself delegate it to another agent. In this setup, the routing policy is function. Instead, it is comprised of local policies distributed among each agent, denoted as 11 (2..,..... k where each π, the local policy for agent my. For a given query x, the policy (z) selects a target agent my from a set of possible actions. If the agent selects itself (j = i). it commits to executing the query. If it selects another agent (j * i), it routes the query to m,. To simplify the evaluation, we enforce that an agent can only route to agents of higher index (and thus higher cost). The action space for agent m, is therefore:
single, monolithic
(σ) E {m, E M Mi
(4)
To guarantee that every query is eventually processed, the policy of the final agent, mx. is fixed to always execute, i.e., л (3)) mm for all æ., serving fallback "expert of last resort."
3

---

Published as a conference paper at ICLR 2026
(1) DiSRouter Routing Procedure
Agent pool: ordered LLM size/cost
Self -ssesssesnmmnn
Rejected
Resporss
Ansein
Execute
smcll
Forword
mediu
Forword
longe
Queries
Execute
Execute
Execute
Rejactt
Erny
Think step by step
Madium
low
med
Asswor high cost
don't
Reliable and cost-efficient solutions
(2) Awarenesss Training
(3) Scenario Adaptability
Stage
resporcees
acourocy
constructed
Agent level
Scenarie: Performance Pirst
Prompt 2
rejections ettempps
Qaary
Sompling
2/
Qsery
Uncortain
c-0.3
Query
Scenario: Cost First
System-level
Parformance First
Cost First
Stage 2: RL
Reutting
Cuery
Somplnn
Rmwand
Traning
Movill Fartt
Mode Cost
Figure 2: The DiSRouter Framework. (1) The routing procedure of DiSRouter: Queries of varying difficulty enter routing cascade ordered by LLM sizelcost, where each LLM agent uses self-assessment to decide whether to answer the query or reject it and route it to the next agent. (2) The Self-Awareness Training pipeline. Green and red indicate correct and incorrect answers, respectively, while blue represents the rejection behavior ("I don't 't know"). (3) DiSRouter possesses scenario adaptability: At the agent level, LLMs adjust their behavior toward uncertain queries based on scenario requirements. At the system level, the overall routing distribution shifts with the scenario: greater emphasis on cost leads more queries being routed smaller models.
For any query the routing process begins at entry-point agent, typically mı. This generates routing path, or trajectory, P(x) (m,, mig...... mi.), where
1. Each subsequent agent in the path is determined by the policy of the previous one, such that Miş+ ni (x) for j < k. The path terminates until an agent mig decides to execute, i.e., Tı (x) = miy The final agent to process the query is thus denoted as [I(x)= my, Since routing costs are negligible, the total cost is simply the inference cost of the final model: C(I[(z)) = C((m..) = Ciz.
The objective is to find the optimal set of local policies II that maximizes the system utility:
II arg max Eg>D[A((T()) - a C(I[())))
(5)
Distributed Optimization A key advantage of the distributed architecture is that the complex systemic optimization problem can be decomposed, allowing each agent's routing policy m, to be optimized independently. The policies of different agents can vary, leveraging either the inherent self-knowledge of LLM (Yin et al., 2023) or a further aligned and improved self-awareness (Xu al. 2024p. Under this paradigm, each agent m, learns its optimal local policy गद by maximizing its own expected local utility Ug based solely on locally available information. which can be formulated as follows, where the design of U, can be tailored each individual agent.
arg max (T(T))
This distributed optimization framework allows individual agents be trained or updated without requiring a full-system retraining, enhancing the scalability and flexibility of the entire system.
3 DISROUTER
To overcome the limitations of conventional centralized routers, we propose DiSRouter (Distributed Self- RouteR) framework built distributed architecture. It relies on the self-awareness of each
4

---

Published as a conference paper at ICLR 2026
agent its local routing policy, which define an agent's intrinsic ability to accurately assess a query falls within its knowledge boundaries and can be solved correctly. Beyond the inherent advantages of distributed structure, DiSRouter also possesses Scenario Adaptability, the ability to dynamically adjust its collective behavior based on scenario requirements, from Performance-Fing to Cost-First. This system-level capability is contingent each agent being scenario-adaptive.
3.1 ROUTING PROCEDURE OF DISROUTER
To facilitate a clear and focused evaluation, we adopt a simple cascade structure for DiSRouter in this work, with the routing procedure illustrated in Figure (1). Agents ordered by increasing cost, and each agent m4 action space is limited to either executing the query or routing it to the next agent, m4+1- We realize this through "reject" behavior: an agent answers query if confident, otherwise it rejects and routes it to the next agent, which transforms the abstract notion of self- awareness into a concrete action. Consequently, DiSRouter shifts the paradigm from training a single, potentially limited router to training set of distributed, reliable self-assessment policies. To this end, we propose the following Self-ewsreness Training pipeline for LLM alignment.
3.2 SELF-AWARENESS TRAINING
The DiSRouter routing system hinges on the self-awareness of each agent. Although original LLMs possess some inherent self- eeff-nnowledge, and certain uncertainty-basee methods have been proposed to evaluate LLM output confidence (Huang et al., 2024), they often struggle to accurately assess an LLM's capacity for specific queries (Zheng et al. 2025). closed-source LLMs (e.g., GPT-4) may already possess strong intrinsic self-awareness and can be integrated without additional training, as discussed in Appendix [A.9 To address the limitations of open-source LLMs, introduce two-stage training pipeline augment L.M self-awareness, comprising an SFT (Supervised Fine- Fine-Tuning) stage for foundational self-assessment abilities, fol- lowed by an RL (Reinforcement Learning) stage for further improvement. The data construction process is illustrated in Figure (2). Beyond self-awareness, we also train agents to be scenario- adaptive: they should be more conservative in a "Performance First" scenario to minimize errors, and more aggressive in "Cost First" scenario for greater efficiency. The scenario adaptability of DiSRouter is shown in F (3).
worth noting that high-capability
3.2.1 SCENARIO ADAPTABILITY INSTRUCTIONS
For given query, the current scenario's requirements are conveyed to the LLMs via instructions. We add instructions corresponding to different scenarios into the prompts, explicitly conveying user preferences to enable nuanced behavioral adjustments. For detailed prompts, see Appendix A.12.
3.2.2 SELF-AWARENESS SFT
We adopt a data construction method similar to prior work on improving LLM reliability (Xu et al, 2024 by introducing explicit rejection behavior: models are trained respond with "I don't know" when uncertain. We use the target LLM to perform N CoT (Chain-of-Thought) inference trials per training query, with the frequency of correct answers measuring its capability for each query. Queries with a correctness frequency below a certain threshold 8 are deemed unanswerable, and their responses are replaced with the rejection statement: "I don't know." For the remaining high- capability queries, which the model is expected to answer, their thought processes and answers are extracted and standardized into an "Answer" template (see Appendix A.12.2
To accommodate different scenarios with varying preference factors a. we set the rejection threshold 8 as 1 a. For instance, "Performance First" scenario (a 0.2). the model should only answer when highly confident (6 0.8). We construct an equal amount of training data for three distinct scenarios and mix them during training, while maintaining an equal proportion of accepted and rejected samples to prevent bias.
3.2.3 SELF-AWARENESS RI.
To further enhancing self-awareness and scenario adaptability, we propose a scenario-conditioned reward for RI. For given query x, the reward is:
5

---

Published as a conference paper at ICLR 2026
1,
if answer correctly if answer incorreetly
reward (x) =
0, (1 a), if reject
Here, is the preference factor that represents the scenario's emphasis cost, with larger indi- cating greater cost importance. is the reliability factor, introduced to ensure the model maintains sufficient reliability and does not excessively sacrifice precision for cost. Both parameters range from O to 1. For RL training, we maintain an equal proportion of data from all three scenarios, which are then combined to form the final training dataset.
Reward Rationality Analysis A model's decision to answer query x or not is essentially based on whether the expected reward for answering is greater than the expected reward for rejecting. Let these be E[answer] and E'reject], and p(z) denote the model's capability in answering æ. Then:
Elanswer] [p(x) 1+ (1 -0(x))) 0. = p(z)
Elreject] (1 a)²²
(9)
E Flnnswrr]
F(jeeett] = p(x) (1 xt)
(10)
This implies that LLM's capability query z must reach tain threshold before it chooses to answer. This threshold decreases increases, meaning that the model prioritizes cost more, its ca- pability threshold for answering decreases, making the LLM more aggressive. This behavior aligns with our requirements. If the reli- ability factor 7 = 1, this trend is linear. However, the human pref- erence for accuracy and cost is often nonlinear; we typically desire the system to maintain high reliability most of the time, only allow- ing a significant decrease accuracy when extreme cost-saving is prioritized. The graph of the function (ax = (1 a is shown in Figure 3 Thus, using a y value between 0 and can effectively
0.4
V=0.5
y-1.0
02
04
066
1.0
Figure 3: f(a) = (1 a
represent this demand. In this work, use 7 0.5 to reflect pursuit of high reliability.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Datasets We utilize widely used benchmarks covering various domains as our test tasks, including mathematical reasoning, common sense question answering, and reading comprehension. These comprise seven datasets: GSMSK (Cobbe et al. (021), A12 Reasoning Challenge (ARC) (Clark ct al., 2018). MMLU (Hendrycks et al.) (020). RACE.HIGH (Lai et al. 2017). Openbook QA (Mihaylov et al. 2018). DROP (Dua et al. 2019). and Cosmos QA (Huang et al. 2019). These incorporate several datasets as out-of-domain tasks to assess the generalization capability of the various methods, including: SQuAD (Rajpurkar al. 2018). HellaSwag (Zellers al., 2019), and HeadQA (Vilares) & Rodezzeod 2019). 19). For datasets with a hidden test set, we utilize the validation set as an alternative. We evaluate LLMs in zero-shot Thain-of-Thougee (CoT, Wei al. 2022) manner and use accuracy as the evaluation metric for all these datasets. Further details on dataset descriptions,
datasets serve in-domain data for model training and evaluation. Additionally,
statistics, and prompts for testing can be found Appendix A.2
Our model pool comprises five distinct sizes of the Twen2.5-Innsrru. series, spanning from 0.5B to 14B (Qwen et al. 2023). These L.LMs vary significantly in terms of performance and inference cost. Follow- ing GraphRouter (Feng et al.) 2024), we as- sign each LLM a normalized, fixed cost rang- ing from to 1. Table I summarizes their av- erage performance across datasets and corre- sponding costs. Details LLM performance and actual inference time every dataset provided in Appendix A.3
LLM Pool
Table Performance and cost for LLMs of dif- ferent sizes. ID: in-domain, OOD: out-of-domain.
Model Size
ID Accuracy
0OD Accuracy
Model Cost
0.5B
38.01
35.82
0.1
1.5B
58.95
54.69
0.2
73.08
61.57
0.4
7B
80.90
69.89
0.7
85.45
74.43
0.9

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of different methods in-domain datasets across three scenarios. We calculated and report the average values across multiple datasets.
Method
Performance First: o 0.2
Balance: a = 0.5
Cost First: a = 0.8
Accuracy
Cost
Utility
Accuracy
Cost
Utility"
Accuracy
Cost
Utility
Topline
Orucle
0.93
0.29
0.87
0.93
0.29
0.79
0.93
0.29
0.70
Naive
Smallest LLM
0.38
0.30
0.36
0.38
0.30
0.33
0.38
0.10
0.30
Baselines
Largest LLM
Random
0.85
0.90
0.67
0.85
0.90
0.40
0.85
0.90
0.13
Routel LM
0.44
0.16
0.41
0.44
0.16
0.36
0.44
0.16
0.31
FrugulGPT
Router Baselines
Automix
0.85
1.10
0.63
0.58
0:58
0.29
0.44
0.33
0.18
FORC
0.78
0.65
0.65
0.69
0.45
0.47
0.60
0.29
0.37
GruphRouter
0.82
0.68
0.68
0.77
0.46
0.53
0.61
0.20
0.45
DiSRoater
SFT
0.84
0.59
0.72
0.83
0.52
0.57
0.77
0.42
0.43
(Ours)
+ RL
0.81
0.40
0.73
0.77
0.32
0.61
0.75
0.29
0.52
Baseline Methods We compare our proposed method against the following baselines. We first introduce three intuitive Naive Baselines: (I) Largest / (2) Smallest LLM: Always routes queries to the largest / smallest available LLM. (3) Random: Randomly routes each query to an LLM.
We also incorporate several representative routing methods: (4) RouteLLM (Ong et al. 2024), (5)
FrugalGPT (Chen al. 2023). (6) Automix (Aggarwal et al., 2024). (7) FORC (Sakota et al, 2024), and (8) Graph (Feng et 20.4), Details on the description and implementation of these methods can be found in Appendix [A.4] Notably, Automix relies on the self-verification the LLMs, and we consider the time overhead for this process equivalent to the LLM's response cost.
Additionally, we introduce a theoretically optimal routing strategy (9) Oracle, as a topline for com- parative analysis, which always routes queries to the smallest capable LLM.
Metrics We compare the average accuracy and average cost of each routing system across multiple datasets. To evaluate the overall routing performance, we utilize the utility metric introduced in $2.1. Equation 2 which can be adjusted for different scenarios by setting a corresponding value for a.
Implementation Details For Self-Awareness SFT, we randomly selected 10,000 training samples from all in-domain datasets and trained for one epoch. The learning rate was set to x 10-5, using four A800 GPUs with batch size of four per GPU. For Self-Awareness RI., we employed the Verl (Sheng et al., 2024) framework with the Reinforce++ algorithm. We used 10,000 training samples with a total batch size of 128. The learning rate was 10 6 and each query was sampled eight times with temperature T of 1. Training was conducted for one epoch on eight A800 GPUs.
4.2 RESULTS AND ANALYSIS
Table presents the performance of DiSRouter in-domain datasets compared to various baselines. Across all three scenarios, our method consistently achieves the highest utility, outperforming all baselines. Notably, DiSRouter reaches at least 74.29% of the Oracle topline, demonstrating the effectiveness of our distributed routing approach. Compared to Naive Baselines, our system is better balancing performance and cost according to the demands of each scenario. Furthermore, our method generally achieves higher accuracy with lower costs than existing Router Baselines. A more detailed analysis of why DiSRouter routes better is provided in Section 4.3
Generalization Analysis We evaluate the generalization capabilities of DiSRouter from three crit- ical dimensions: task domains, model architectures, and task types. First, regarding task domains, we compare the performance of different baselines and DiSRouter on several out-of-domain datasets (Table (3). Our findings indicate that DiSRouter effectively distinguishes whether query is within its capability even on unseen datasets (see Appendix [A.5
(A.3) for full results). Second, regarding model architecture, we validate our framework on a heterogeneous agent pool comprising diverse LLM families (e.g.. Gemma-3 (Team et al., 2025), Phi-4 (Abdin et al. 2024)). As detailed in Appendix A.10 results demonstrate that training pipeline and distributed architecture remain highly effec- tive across different LLM architectures. Third, regarding task types, we extend our evaluation to open-ended generative tasks (e.g., text summarization) where correctness is not binary. As shown in
7

---

Published as a conference paper at ICLR 2026
Performance First
3,4%
32.9%
27.0%
25.8%
11.0%
Query Distribatioon
Answer Rate
100
Performane Pist
Balunce
25.8%
Cost Fiast
Balance
6.9%
4%.1%
23.7%
20.5%
3.8%
26
50
11.15
Cost First
8.1%
51.3%
22.0%
17.7%
0.9%
0.51
1,5B
3B
7B
14B
0.5B
1,5B
Model Size
Model Size
Figure 4: System-level routing distribution.
Figure 5: Agent-level answer rate.
Appendix A.II, by adapting the reward formulation to continuous metrics, DiSRouter successfully generalizes to these scenarios, distinguishing high-quality generations from low-quality ones.
Modularity and Flexibility Analysis The distributed structure of DiSRouter inherently offers "plug-and and-play" modularity. To validate this, we reduced the agent pool from the original five mod- els to a three-agent system (1.5B, 3B, and 14B), without the need for any retraining or modification to the agents. Results for the "Balance" scenario are presented in Table results). Our three-agent DiSRouter system still achieved superior utility across all three scenarios, confirming the excellent modularity of DiSRouter. Furthermore, the modest utility drop compared to the five-agent system was expected, indicating that our full system effectively leverages the two additional models for performance gains. This scaling of system performance with the diversity of available agents further validates the soundness of our distributed, self self-routing paradigm.
Appendix[A.6) for full
Table 3: Comparison of different methods on out-of-domain datasets for Balance scenario.
Table 4: Performance comparison on re- duced 3-agent system for Balance scenario. RouteLLM and Automix are excluded they require retraining for agent pool modifications.
Balance: 0.5
Method
Accuracy 1
Cost J
Utility 1
Topline
Oracle
0.72
Method
Balance: a 0.5
Naive
Smallest LLM
0.36
0.10
0.31
Accuracy
Cost
Utility
Buselines
Largest LLM
0.90
Topline
Oracle
0.91
0.38
0.72
Random
0.59
0.46
0.36
Smallest I.I.M
0.59
0.20
0.49
RowieLLM
0.42
0.22
0.31
Raselines
Largest LLM
0.85
0.90
0.40
FrugaGIPT
0.55
0.25
0.42
Random
0.72
0.50
0.47
Baselines
Router
Automix
FORC
0.63
0,55
0.53
0.60
0.36
0.25
Roster Raselines
FrigaGPP
FORC
0.63
0.72
0.48
0.48
GraphRouser
0.49
0.39
GraphRouter
0.75
0.46
0.52
DiSRouter (Ours)
SPT
0.74
0.62
0.43
DISRouter (Ours)
SFT
0.83
0.54
0.56
RT.
0.69
0.43
0.48
RL
0.78
0.36
0.60
Scenario Adaptability Analysis DiSRouter dynamically adjusts its collective behavior via the preference factor a across different scenarios, capability we term Scenario Adaptability. Crucially, unlike baselines like GraphRouter that are scenario-dependent and require re-training for each sce- nario, DiSRouter is single, unified system that adapts diverse scenarios using scenario-specific prompt. We analyze this capability at both the system and the individual agent levels.
At the system level, figure shows clear strategic shift in the distribution of queries routed to each agent. As the scenario transitions from "Performance First" to "Cost First", more queries are routed to smaller models, indicating that DiSRouter shifts strategy from maximizing accuracy to prioritizing cost. This systemic adjustment stems from the adaptability of each individual agent. At the agent level, we plot the answer rate-the percentage of queries an agent chooses to answer rather than reject-for each agent across three scenarios in Figure As increases towards cost- first priority, every agent's 's decision threshold is effectively lowered, making them more "willing" to answer. This synchronized adaptation is the fundamental mechanism enabling the system to achieve global objectives without explicit coordination, confirming that our localized, scenario-conditioned reward function successfully instills the desired Scenario Adaptability in each agent.
8

---

Published as a conference paper at ICLR 2026
4.3 WHY DISROUTER ROUTES BETTER?
In this section, we explore the reasons for DiSRouter's superior routing utility. We first show that DiSRouter is better distinguishing between easy and hard queries than router baselines, and after that we analyze the self-awareness of our trained LLMs, revealing their strong intrinsic capacity determine whether a problem falls within their capabilities and can be answered correctly. It is important to note that DiSRouter's performance gains do not stem from a direct improvement in task performance due to training. point we validate with experiments in Appendix A.8
System-level Analysis We define queries solvable by 3B and smaller models "Easy," and all others as "Hard." An effective routing system should direct easier queries to smaller LLMs for lower average costs. Under the "Balance" scenario, analyzed the average cost for easy and hard queries for three Aaselines-Grrphheee evident that the router baselines lack or exhibit only limited discriminative capability. Our system, in contrast, effectively differentiates queries of varying difficulty.
FORC, and FrugalGPT-comppnte
to our method (Figure 6).
External Router vs. Intrinsic Assessment A core advantage of DiSRouter is its reliance on intrinsic seff-assessment over an external router. To demonstrate this, we formalize the question "whether a LLM can solve query x" binary classification problem. We tar- geted the 7B model, using the correctness of its greedy-search response each query ground truth. To ensure a balanced test set, we randomly sampled 5,000 instances from each class, We trained two external classifiers (one BERT-based, one LLM-based using Llama3-8B. Instruct (Grattafiori al. 2024)), and compared their performance against our model. For our DiSRouter-aligned LLM, its decision whether to reject query serves as the classification output. Implementation details for classifiers are in Appendix A.7 with results summarized in Table[5]
Table 5: Comparison between external routers and self-assessment DiSRouter.
Type
Classifier
Size
Accuracy
F1
Naive
Random
0.50
0.50
Bert-based
0.13B
0.56
0.63
External
LLM-based
8B
0.71
0.77
Intrinsic
DiSRouter
7B
0.80
0.81
Our self-assessment method demonstrates superior classification accuracy and F1 score, indicating its ability to accurately discern solvable queries and prevent itself from queries beyond its capa- bilities. External routers exhibit two primary limitations: (1) I Limited Performance due to Con- strained Parameters. While router performance improves with larger models, using a separate, larger router introduces significant overhead, problem DiSRouter avoids by integrating the jection action. (2) Inferiority of External Evaluation. Even a similar-sized LLM--based external classifier lags behind intrinsic approach. We therefore argue that integrating the assessment function directly into the LLM, as DiSRouter does, is a more rational and efficient solution.
Easy queries
1.0
0.36
0.84
0.91
0.6
Hard queries cost
Accuracy
Cost
0.4
0.45
0.47
0.34
0.5
0.39
0.43
3
Answened
Rej ected part
0.63
0.24
0.28
0.0
0.3B
L.5
5B
BB
0.2
Model Size
0.0
Figure 7: Accuracy of answered and rejected queries for DiSRouter-aligned LIMSs in the "Performance First" scenario. The accuracy of our models on answered queries significantly surpasses that of their rejected counterparts, pro- viding evidence of effective assessssessmnt..
GraphRouter
FORC
FrugalGPT
DISRouter
Figure 6: Comparison of average Cost of easy and hard queries for "Balance" scenario. A larger discrepancy between the two indicates stronger ability to differentiate query difficulty.
We further analyzed the self-awareness of each aligned LLM based on their query acceptance and rejection behaviors. Figure 7 compares their accuracy answered versus rejected queries under the "Balance" scenario. For the rejected queries, we employed prefix injection approach (Li et al. 2025) by appending affirmative prefix to the end of the prompt.
Self-Awareness Ability
9

---

Published as a conference paper at ICLR 2026
This forced the models to generate responses, enabling calculate their underlying accuracy. is evident that our aligned models possess a strong self-awareness capability, demonstrating a significant capacity to distinguish whether problem falls within their capabilities.
These findings validate our core motivation: methods that rely on a centralized external router to assess query difficulty for routing decisions are inherently limited. Such approaches primarily learn to route queries to LLMs with higher expected utility rather than making judgments based on the ac- tual difficulty of the query. Our DiSRouter framework, which leverages the models' self-awareness ability, is not only logically more sound but has also been proven to be more effective.
RELATED WORK
Research in LLM query routing is largely dominated by centralized architectures. Early systems like FrugalGPT (Chen et al. 2023) used scoring model to directly evaluate the quality of an LLM response to decide whether to proceed to more powerful agent. More recent methods focus on predictive routing, where central router assesses queries before execution. These routers typically predict either the difficulty of a query (e.g., Hybrid LLM (Ding et al.) (024), RouteLLMM (Ong) al., 2024)) or the expected performance of each available LLM, using techniques like bandit- based models (e.g.. C2MAB-V (Dai et al.] (2024) or graph neural networks (e.g., GraphRouter (Feng et al. 2024)). ). To address the inherent rigidity of centralized networks, recent works have explored multi dilti-dimensional profiling (Shi et al. (025) and decoupled regression predictors (Wang) jtt al. 2025). While effective, these systems share fundamental limitations: their overall performance is bottlenecked by the capability of the external router, and their centralized architectures are often rigid, lacking the modularity to easily adapt to dynamically changing pool of agents.
To bypass the bottlenecks of external classifiers, alternative approaches explore the intrinsic knowl- edge of LLMs for routing. For instance, AutoMix (Aggarwal et al.) 2024) utilizes an LLM's self- verification to guide routing decisions; however, its multi-step verification incurs significant time overhead and relies on static, non non-adaptive thresholds. To mitigate such latency. methods like Early Abstention cascades (Michael et al. (2025) trigger routing decisions at early decoding stages based on error probability distributions. Moving towards fully distributed topologies, AgentNet (Yang ett al. 2025) shares a similar motivation with our work by equipping each agent with independent router for decentralized decision-making
6 DISCUSSION AND CONCLUSION
Discussion Distributed Routing Networks While our work mainly discusses simple cas- cading structure, the distributed nature of DiSRouter theoretically supports more complex network topologies, such cross-level cascading or tree-like structures with multiple routing directions. To implement these, we would need to introduce system-level information during the inference phase, practice common in distributed communication (e.g., Content Distribution Network, CDN). This would allow each model to be aware of subsequent models' information when making routing deci- sions, enabling more efficient routing, such as directing the most difficult queries the final model.
Future Work We acknowledge several avenues for future research. (1) Our current Self- Awareness Training pipeline is relatively basic. Future work could explore incorporating "reasoned refusals" or more sophisticated distributed reinforcement learning (RI.) rewards to further enhance model self- alf-enwresssss especially for smaller models. (2) We will explore more complex distributed network structures and implement the system information exchange during inference to generalize the framework to a true intelligent agent network.
This paper introduces the Distributed Self-Router (DiSRouter), novel framework designed to address the challenge of cost-effective query routing in multi-agent systems. By identi- fying key limitations in prevailing centralized routing architectures, DiSRouter eliminates the cen- tral router and instead empowers each LLM agent with intrinsic self-awareness, creating fully distributed, scalable, and plug-and-play system. Our extensive experiments validate the superiority of DiSRouter, which consistently achieves higher utility than competitive baselines across various cost-sensitive scenarios. key advantage of DiSRouter its reliance on intrinsic self-awareness, which our experiments prove outperforms external evaluation.
Conclusion
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is funded by the China NSFC Projects (62120106006, 62576212, 92370206, and U23B2057) and Shanghai Municipal Science and Technology Project (25X010202846).
ETHICS STATEMENT
The authors declare that this work was conducted in full accordance with the released code of ethics. We confirm that there are ethical issues to be disclosed.
REPRODUCIBILITY STATEMENT
We are committed to the principles of reproducible research. Our work is highly reproducible as all components are based on publicly available resources and are described in detail throughout this paper. The models and datasets utilized in this study are open-source, with further details provided Section 4. A comprehensive description of our data construction process is presented in Sec- tion 3.2 Furthermore, all implementation details, including training parameters and experimental settings, are thoroughly documented in Section 4.1 We believe that these provisions will enable the research community to fully replicate our findings.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. Mendes, Anh Nguyen, Eric Price, Gustavo Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report, 2024. URL/tttps:/ http:::://wxwxwo/.///.//ooo www. wttp.rr//abb/11.... 08905
Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang. et al. Automix: Automatically mixing language models. Advances in Neural Information Processing Systems, 37:131000-131034, 2024.
AI-MO. Aimo validation ame dataset on hugging face, 2024a. URL httph:ttttppggeeee http:///hhggiggcc..... UtLttps://uuggiiaae
so/GataasessAAI-Mo mo datioon-
AI-MO. Aimo validation dataset on hugging face, 2024b. URL hww.hhttss//hngggigfaae.
datasets/AI wwwaammovalidaatte.....oooaaaaa.
aime
Daman Arora, Himanshu Gaurav Singh, and Mausam. Have Ilms advanced enough? challenging problem solving benchmark for large language models, 2023. URL
abs/ 2ttp/2355.20 15074
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint 1:Xxi:2305.01766 2023.
Zhijun Chen, Jingzheng Li, Pengpeng Chen, Zhuoran Li, Kai Sun, Yuankai Luo, Qianren Mao, Dingqi Yang. Hailong Sun, and Philip S. Yu. Harnessing multiple large language models: A survey on Ilm ensemble, 2025.
ptattps://ariiie org/ wtt../////......1.3333
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 18033 05457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.14168 14168, 2021.
11

---

Published as a conference paper at ICLR 2026
Xiangxiang Dai, Jin Li, Xutong Liu, Anqi Yu, and John Lui. Cost-effective online multi-llm selec- tion with versatile reward models. arXiv preprint arXiv:240S. 2024.
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. Hybrid Ilm: Cost-efficient and quality-aware query routing. arXiv preprini arXiv:2404. 14618, 2024.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume / (Long and Short Papers), pp. 2368 2378, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18655//11111191266 URL https:
7/ Taclanthologyy. wwg.logy..ooooooooooo.oo NN19 12119-12666 1246/
Tao Feng, Yanzhen Shen, and Jiaxuan You. Graphrouter: A graph-based router for Ilm selections. arXiv preprint arXiv:2410.03883 2024.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, Association for Computational Linguistics, 2019. doi: 10. 18653/v1/119-5499. URL htth:t:::/ / doi org/ 10. 18653/ 9-0-999-4999
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark. Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBudawy. Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh. Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoy- chev, Niladri Chatterji. Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic. Roberta Raileanu. Rohan Maheswari. Ro- hit Girdhar, Rohit Patel, Romain Sauvestre. Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang
12

---

Published as a conference paper at ICLR 2026
Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajofeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern. Grant Herman, Grigory Sizov, Guangyi. Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri. Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog. Igor Tufanov, Ilias Leontiadis, Irina-Elens Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings. Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liinnpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parckh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager. Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay. Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe. Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho. Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang. Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary Veto,, 7ef Rosnbrick, Zhao- duo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The Ilama 3 herd of models, 2024. URL https: ttapss://rriiie www./as...//oooooooo///.o//.aaaa 2407 21783
13

---

Published as a conference paper at ICLR 2026
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual mul- timodal scientific problems, 2024. http:ttts:/// /aattps:/arxiv...o org wttp///rggabb///////////// 14008
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhann bert with disentangled attention. In International Conference Leurning Representations, 2021.
URL htttts:://pperreiieeeeee/ <ttp://eeeeeeeeeeeoooooo///////- forrmmidd=x2IIIattuts
net
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint TDt::2009.03000, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
URL https: http:///axivvvvo wewzxiv.orggaaai wttp//ggg//////oo
03874
Hsiu-Yuss Huang, Yutong Yang, Zhaoxi Zhang, Sanwoo Lee, and Yunfang Wu. A survey of uncer- tainty estimation in llms: Theory meets practice. arXiv preprint arXiv:2410.1520 15326, 2024.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Process- ing (EMNLP-IJCNL)), pp. 2391-2401, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 2.118653/1/DD19 1243. URL https: /ttpss://aclas http:///aaaaanhooyyy..o
bxg/D19-1243/
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Cher, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang, Dahua Lin, Yu Qiao, and Pengfei Liu. Olympi- carena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai, 2025. URL
https: pttps://arxii> org. wtg.abss////oooo....o.....s....a / 2406 12753
Minwu Kim, Anubhav Shrestha, Safal Shrestha, Aadim Nepal, and Keith Ross. Reinforcement
learning vs. distillation: Understanding accuracy and capability in Ilm reasoning, 2025. URL
https::/arxiis http://..rrrab//////.// http:s//aarax.orrrraa////555.. 14216
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large- scale ReAding comprehension dataset from examinations. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference Empirical Methods in Natural Language Process- ing, 785-794, Copenhagen, Denmark, September 2017. Association for Computational Lin- guisties. doi: 18.18863/v 1/D77-11/111-0822
acl http:///aaaaaatooooooo////111/1
Yakai Li, Jickang Hu, Weiduan Sang, Luping Ma, Jing Xie, Weijuan Zhang, Aimin Yu, Shijie Zhao, Qingjia Huang, and Qihang Zhou. brasell-based jailbreak: A novel approach of bypassing 11m safety boundary. arXiv e-prints, pp. arXiv-2504, 2025.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. URL https: arxiv. wwt./br///////////// 11692
J Zellinger Michael, Rex Liu, and Matt Thomson. Cost-saving 1lm cascades with early abstention. arXiv preprint XXX:2500.00954 2025.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv: 1809.02789, 2018.
14

---

Published as a conference paper at ICLR 2026
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization, 2018. URL https:
Isaac Ong. Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route Ilms with preference data. arXiv preprint arXiv:2406. 18665, 2024.
OpenAl. Introducing gpt-5. gust 2025.
gpt Au-
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: /tops::/aaee hwt.aaxxivvv..... abs// 2412.44222222 15115
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad, 2018. URL
org/ abs/ /1066.80.... 03822
Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 606-615, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Haochen Shi, Tianshi Zheng. Weiqi Wang, Baixuan Xu, Chunyang Li, Chunkit Chan, Tao Fan, Yangqiu Song, and Qiang Yang. Inferencedynamies: Efficient routing across Ilms through struc- tured capability and knowledge profiling. arXiv preprint arXiv:2505. 16303, 2025.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Cas- bon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xi- aohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Cole- man, Yi Gao, Basil Mustafa, lain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan- Tan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju. Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang. Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng. Dimitris Paparas, Divyashree Shivaku- mar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eu- gene Kharitonov, Frederick Liu, Gagik Amirkhanyan. Glenn Cameron, Hadi Hashemi, Hanna Klimezak-Plucinsk Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini. Jean Pouget-Abadic, Jetha Chan, Joe Stanton, John Wi- eting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma. Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Ni- lay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Ruben- stein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu. Piotr Stanezyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Poder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi
15

---

Published as a conference paper at ICLR 2026
Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow. Yuvein Zhu, Zichuan Wei, Zoltan Egyed. Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Bar- ral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukeuoglu, Clement Farabet, Elena Buchatskaya, Jean-Barne Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. Gemma 3 technical report, 2025. URL https: wawaa.iiivv... wttp:iv.rrgaabb//b//a/ wtw.:////..//...aa////// 19786
Asterios Tsiuruass, Wei Sun, and Georgia Perakis. Causal Ilm routing: End-to-end regret minimiza- tion from observational data, 2025. URL
Clovis Varangot-Reille Christophe Bouvard, Antoine Gourru, Mathieu Ciancone, Marion Schaeffer, and François Jacquenet. Doing more with less: A survey routing strategies for resource op- timisation in large language model-based systems, 2025. URL
2502. 00409
David Vilares and Carlos GódrzzRod HEAD-QA: A healthcare dataset for complex rea- soning, In Anna Korhonen, David Traum, and Lluís Märquez (cds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 960-966, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/w1/P19-10922 URL http:p/://cclaatttooooo..ooooooe
Xinyuan Wang. Yanchi Liu, Wei Cheng, Xujiang Zhao, Zhengzhang Chen, Wenchao Yu, Yanjie Fu, and Haifeng Chen. Mixllm: Dynamic routing in mixed large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Lingnistics: Human Language Technologies ( (Volume 1: Long Papers), pp. 10912-10922, 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1117. 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Cfain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 14:28244-44877 2022.
Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, and Kai Yu. Rejection im- proves reliability: Training ilms to refuse unknown questions using rl from knowledge feedback, 2024. btRphttps: // arxiv. wwwpexiw...ggaaaaa abs/ / /tabbb//24430 2403..44 18349
Hongshen Xu, Zihan Wang, Zichen Zhu, Lei Pan, Xingyu Chen, Lu Chen, and Kai Yu. Alignment for efficient tool calling of large language models, 2025. URL https:///rxxiii org/ abs/ 2503.00 06708]
Yingxuan Yang, Huacan Chai, Shuai Shao, Yuanyi Song, Siyuan Qi, Renting Rui, and Weinan Zhang. Agentnet: Decentralized evolutionary coordination for llm-based multi-agent systems, 2025.
Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. Do large language models know what they don't know?, 2023. URL htttp:://axxiv... /
2305. 18153
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?, 2025. URL /ttetps://arxiv... wttp:/vvwrr//b///5...1.... 13837
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Märquez (eds.). Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-48000 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 18653/v1/P19-14 1472. URL https: / anthol htw.hholooo.orrr/11111141
16

---

Published as a conference paper at ICLR 2026
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger. and Yoav Artzi Bertscore: Evalu- ating text generation with bert, 2020. URL https::/axxxi..ooo////////////// http:/vv..rrabb///1go0// 09675
Hang Zheng, Hongshen Xu, Yuncong Liu, Lu Chen, Pascale Fung, and Kai Yu. Enhancing 1lm reliability via explicit knowledge boundary modeling, 2025, URL https:/ http:///axiv.orr.
abs/ abs/2503 2503. 02233
17

---

Published as a conference paper at ICLR 2026
Appendix
TABLE OF CONTENTS
A.1
Time Overhead Analysis
19
A.2
Dataset Descriptions
20
A.3
LLM Pool Performance and Cost
20
A.4
Router Baselines: Descriptions and Implementation Details.
A.5
Out-of-domain Datasets Detailed Results
A.6
Reduced 3-Agents System Results.
A.7
External Routing Classifiers.
A.8
Whether Task Capability Improved After Training?
A.9
Experiments Intrinsic Sell-Awareness
24
A.10
Experiments on Different Model Families
A.11
Experiments Open-Ended Generative Tasks
26
A.12
Prompts and templates
18

---

Published as a conference paper at ICLR 2026
THE USE OF LARGE LANGUAGE MODELS (LLMS)
In the preparation of this paper, we utilized Large Language Models (LLMs) auxiliary tool to support our research and writing process. Specifically, LLMs were employed for two primary purposes: 1) to aid in the discovery of relevant literature and assist in the paper reading by gener- ating summaries, and 2) to aid in manuscript refinement through the translation of specific phrases and the polishing of sentence structures for improved clarity and readability. We affirm that the authors critically reviewed, revised, and edited all content suggested by the LLMs, and we take full responsibility for the accuracy and integrity of the entire work.
A.1 TIME OVERHEAD ANALYSIS
the total cost of routing system comprises the routing cost (the time overhead the decision process) and the inference cost (the execution time the selected LLM). In this section, we conduct a fine- ane-grained statistical analysis of the actual time consumption. To accurately measure the inference time for each sample, we excluded parallel acceleration techniques typically provided by inference engines like vLLM (Kwon al. 2023 Instead, we utilized sequential, sample-by-sample inference approach for precise timing.
As discussed in Section 2.1
Table 6: Comparison of average time consumed by rejection and answering responses. Statis- ties were gathered using predictions from the base model (for the 14B L.M) and from SFT-trained models (for the others). Proportion refers to the proportion of time consumption of rejection to swering.
Type
0.5B
1.5B
3B
7B
14B
Answer (s)
0.825
1.399
2.446
3,895
5,398
Reject (s)
0.039
0.053
0.066
0.083
Proportion (%)
4.7
3.8
2.7
2.1
Table 7: Analysis of routing overhead in DiSRouter. We report the time consumption for both average and worst-case (routing to the final 14B model) situations under "Balance" scenario. Pro- portion refers to the proportion of Routing Cost to Inference Cost.
Inference Cost (s) Routing Cost (s)
Proportion (%)
Average
2.270
0.081
3.56
Worst Case
5.398
0.241
4.46
DiSRouter Overhead Analysis In the DiSRouter framework, the routing decision is intrinsic, manifested as the time taken by an agent to generate a "rejection" response. Therefore, the routing cost corresponds to the cumulative time consumed by all agents in the routing path to generate their rejections. Our empirical analysis shows that the average rejection time for Self- Awarenesss trained model is less than 5% of its inference time (Table Besides, further analyzed both the average routing overhead across all samples and the worst-case scenario (where the query traverses the entire cascade the final 14B model). As shown in Table 7 the routing overhead constitutes less than 5% of the total inference time in both average and worst-case scenarios. Consequently, we consider this routing cost negligible and do not factor it into the utility calculations in the main text.
Table 8: Comparison of System Latency (Time to First Token, s) across different methods.
RouteLLM
FrugalGPT
Automix
FORC
GraphRouter
DiSRouter
Average
0.048
2.620 9.565
3.475 11.34
0.070 0.076
0.024 0.028
0.081
Worst Case
0.063
0.241
19

---

Published as a conference paper at ICLR 2026
System Latency Comparison To evaluate the real-time responsiveness of different systems, benchmarked the System Latency across different methods, which defined as the time required to generate the first token of the final answer. measured the time consumption of router base- lines under a similar sample-hy-samp The comparison is presented in Table g The results reveal distinct characteristics among routing strategies.
configuration, using 100 randomly sampled test queries.
1. Predictive routers utilizing lightweight external classifiers (RouteLLM, FORC and GraphRouter) exhibit the lowest latency, their routing decisions are nearly instantaneous.
2. Generative/Cascading routers that rely on intermediate model outputs (FrugaIGPT) or extensive self-verification steps (AutoMix) incur significant latency, often exceeding the inference time of the LLMs themselves.
3. DiSRouter demonstrates balanced profile. While its latency is marginally higher than that of predictive routers due to the token generation process for rejection, it is significantly faster than other cascading verification-basae -based approaches. Given that the rejection sponse typically consists of only a few tokens, the introduced overhead remains minimal and does not negatively impact the user experience.
A.2 DATASET DESCRIPTIONS
This section provides brief overview of the datasets utilized in this work.
GSM8K: A dataset focused on mathematical problem-sol requiring models to comprehend natural language problems, apply correct operations, and provide solutions.
ARC: Designed to evaluate models' ability to answer questions based on scientific texts, involving comprehension, critical thinking. and scientific reasoning.
MMLU: A comprehensive dataset testing broad knowledge and reasoning across various domains, featuring multiple-choice questions from diverse subjects.
RACE HIGH: A reading comprehension benchmark for high school-level texts, testing deep com- prehension, logical reasoning. and inference skills.
OpenbookQA: Crafted to test model's ability to apply elementary science knowledge to novel situations, requiring integration of facts with commonsense reasoning.
A challenging dataset evaluating system's 's ability to perform discrete operations over text, such as addition or counting, on adversarially created questions.
DROP:
CosmosQA: Focuses on commonsense-basse reading comprehension, requiring models to infer implicit causes, effects, intentions, or plausible outcomes from narratives.
SQuAD: An extractive reading comprehension benchmark where the answer is text span directly found within corresponding Wikipedia passage.
HellaSwag: A dataset for commonsense reasoning tasks, where models predict the most plausible continuation for diverse incomplete situations.
HeadQA: Evaluates multilingual and multi-domain question answering in healthcare, featuring multiple-choice questions from real medical exams in Spanish and English.
Statistical information for these datasets and the prompts for testing are summarized and presented in Table
A.3 LLM POOL PERFORMANCE AND COST
We present the performance and actual computational costs of the five different-sized models from the Dwen2.5- series our selected datasets in tables 9 and respectively.
20

---

Published as a conference paper at ICLR 2026
Table 9: Performance of LLMs (aceuracy %.)
Table 10: Cost of LLMs (inference time, s).
Dataset
0.5B
1.5B
3B
7B
14B
Dataset
0.5B
1.5B
3B
7B
14B
GSM8K
42.76
68.31
84.00
86.81
91.58
GSM8K
0.79
1.48
2.35
3.62
4.70
ARC
37.71
62.71
75.51
83.02
85.58
ARC
0.79
1.31
2.26
3.65
4.90
MMLU
34.97
55.19
63.77
74.04
79.21
MMLU
0.72
1.28
1.72
3.11
4.83
RACE_HIGH
43.57
63.69
77.67
85,79
89.14
RACE
RACE_HIGH HIGH
0.73
1.64
2.76
4.15
5.90
OpenbookQA
42.40
66.40
81.20
87.80
94.2
OpenbookQA
0.43
1.09
1.40
3.62
4.98
DROP
28.82
43.37
62.72
70.37
73.25
DROP
0.72
2.45
3.40
6.11
7.98
QA
35.81
53,00
66.67
78,46
85.19
CosmosQA
0.78
1.85
2.85
5.07
5.71
SQUAD
44.48
62.87
61.17
73.63
71.76
SQUAD
0.62
1.73
3.37
4.86
5.95
HellaSwag
29.91
48.71
62.24
63.91
72.78
HellaSwag
0.55
1.15
2.78
4.28
5.31
HeadQA
33.08
52.48
61,31
72.14
78.74
HcadQA
0.63
0.73
2.54
3.92
3.69
Average
37.35
57.67
69.62
77.6
82.14
Average
0.68
1.47
2.54
4.24
5.40
Table 11: Statistical information and CoT prompts for datasets.
Train samples
Test samples
Dataset
CoT prompt
Given the following math problem, reason and give final answer to problem.
GSM8K
7,473
1,319
Choose the most reasonable answer based scientific reasoning and give linal answer. The final answer must be one of: [A. B. C. DJ without any prefix suflix.
ARC
1,119
1.172
Choose the most reasonable answer of the multiple choice question hased scientific reasoning and give a final answer. The final an- must be one of: [A. B, C. DJ without any prefix suffix.
MMLU
4,993
14,042
Read the following passage carefully, choose the most suitable an- swer for the multiplechoicce question based on commonsense rea- soning and give final answer. The final answer must be one of: [A, B. C, DJ without any prefix suffix.
RACE HIGH
6,245
3,498
Use both common sense and basic science knowledge to choose the best answer to the multiple-choice question. The final answer must he the one of: |A, B, C. D)] without any prefix or suffix.
OpenbookQA
4,957
500
Read the following passage carefully and answer the question. You should give the final answer without any prefix or suffix.
DROP
15,480
9,535
Read the given passage and choose the most plausible answer to question based on context and common sense reasoning. The final answer must he one of: [A, B, C, DJ without any prefix or suftix.
CosmosQA
25,262
2,985
Answer the questions based on provided context. Your response must follow the format requirement. The final answer must the direct answer to the problem without any prefix or suffix.
SQuAD
86,821
5,928
Choose the most reasonable continuation the given text, reason and give final answer. The final answer must be one [A. C. DJ without any prefix or suffix.
HellaSwag
39,905
10,042
Choose the most reasonable answer of the multiple choice question hased scientific reasoning and give a final answer. The final an- swer must be one of: [A, B, C, D] without any prefix or suffix.
HeadQA
2,657
2,742
A.4 ROUTER BASELINES: DESCRIPTIONS AND IMPLEMENTATION DETAILS
This section details the router baselines incorporated into this study, along with their respective implementation specifics.
RouteLLM Ong et al. (2024): This method frames the routing problem as a classification task, employing Bert-style classifier to assign each request to the LLM that yields the highest reward. In the context of this paper, this translates to routing the query to the smallest model capable of resolving the problem. For our implementation, utilized the base version of the RoBERTa model
21

---

Published as a conference paper at ICLR 2026
the backbone for the classifier. Training conducted with batch size of 128 under various learning rate configurations, selecting the model that performed best on the validation set (with a learning rate of 5 10 50-5).
Chen et al. (2023) This approach adopts a cascading strategy, where queries are pro- gressively routed from smaller to larger models. score model is trained to assess the output relia- bility each model until a pre-determined, empirically searched threshold is exceeded. While the original paper employed trained DistilBERT the score model, observed poor performance in our initial trials. Consequently, we opted for a higher-perforning trained Swen2.5-005B-Insttee as a substitute.
FrugalGPT
Automix Aggarwal et al. 20224; Also employing a cascading mechanism, Automix relies on the self-verification results generated after each model's response. models the routing problem as a decision process and solves it using a Partially Observable Markov Decision Process (POMDP). The model undergoes ten reflection steps for each query and response. The number of bins for the self-verification probabilities within the POMDP instance was also set to 10. We configured the self-verification cost of the model to equal to its response inference cost.
FORC Sakota et al. (024)) This method utilizes trained meta-model predict the score of each LLM on a given query, subsequently routing the query to the LLM possessing the maximum predicted utility. Similar FrugalGPT, the original paper specified a trained DistiBBERT as the score model. We replaced this with more capable trained Qwen2.5-0.55 1.e.nn.5-...BIInsiiiii Instruct Training parameters for this substitute model were as follows: a maximum sequence length 1024, batch size of 16, a learning rate of 2 x 10-3, 3, and training for 10 epochs.
Feng et al. (2024): This approach incorporates textual descriptions related to tasks and LLMs as context, employing graph neural network to predict the effect and cost of LLMs on queries. Queries are then routed to the LLM with the maximum predicted utility. We trained this model using the officially released code, modifying the reward calculation formula to align with our defined utility. We extensively searched for the optimal learning rate and training mask rate parameters, which are as follows for each scenario:
GraphRouter
Performance First: learning rate (Ir) = 1 x 10-4, training mask rate (Imr) = 0.
Balance: (r 2 10 tmr 0.5
Cost First: Ir 10 tmr 0.7
A.5 OUT-OF-DOMAIN DATASETS DETAILED RESULTS
We have presented the results for out-of-domain datasets under the "Balance" scenario in the main text. Here, we present the results for the "Performance First" and "Cost First" scenarios, as shown in Table 12 and Table 13 respectively.
Table 12:
Comparison of methods on 0OD
Table 13:
Comparison of methods on 0OD datasets for Cost First scenario.
datasets for Performance First scenario.
First:
Cost First:
Method
Method
Accuracy
Cast
Utility
Accuracy f
Cost
Utility
Topline
Orucle
0.89
0.33
0.82
Topline
Orucle
0.89
0.33
0.63
Smallest LLM
0.36
0.10
0.34
Smallest
0.36
0.10
Naive Baselines
Largstt I.I.M
0.74
0.90
0.56
Naive Baselines
Largest I.I.M
0.74
0.90
0.02
Random
0.59
0.46
0.50
Random
0.59
0.46
0.22
RouteLLM
0.42
0.22
0.38
LouteLLM
0.42
0.22
0.25
FrigalGPT
0.63
0.41
0.54
FriglGPP
0.55
0.21
0.38
Baselines
Router
Automix
0.74
1.10
0.52
Router Baselines
Automix
0.40
0.29
0.17
FORC
0.70
0.75
0.56
FORC
0.54
0.32
0.29
GraphRouter
0.69
0.67
0.56
GraphRouser
0.55
DiSRonter (Ours)
SFT
0.73
0.62
0.61
DiSRonter (Ours)
SFT
0.69
0.56
0.25
RT.
0.75
0.67
0.62
RT.
0.67
0.36
0.39
22

---

Published as a conference paper at ICLR 2026
A.6 REDUCED 3-AGENTS SYSTEM RESULTS
In Section 4.2 we validate the modularity of DiSRouter using a reduced 3-agent system. The results for the "Performance First" and "Cost First" scenarios are presented in Tables [4 and respec- tively.
Table 14: Comparison on reduced 3-agent system for Performance First scenario.
Comparison on a reduced 3-agent system for Cost First scenario.
Table 15:
Method
Performance First: a
0.2
Method
Cost First: a
0.8
Accuracy
Cost
Utility
Accuracy f
Cost
Utility
Topline
Oracie
0.91
0.38
0.83
Topline
Oracle
0.91
0.38
0.61
Naive Buselines
Smallest
0.59
0.55
Naive Baselines
Smallest LLM
0.59
I Largest L.M
0.85
0.9
0.67
I Lagestt I.I
0.85
0.9
0.13
Random
Router Buselines
FrugalGPT GraphRouter
0,71
0.34
0.64
Router Baselines
FrugalGPT
0.61
0.21
0.44
0.75
0.63
0.62
GraphRouter
0.59
0.2
0.43
DiSRonter
0.84
0.72
DiSRonter (Ours)
0.83
(Ours)
RT.
0.84
0.53
0.73
RT.
0.75
0.29
0.52
A.7 EXTERNAL ROUTING CLASSIFIERS
we introduced two external binary classifiers for our comparative study: BERT- based model and an LLM--ased model. Here, we detail their training specifics.
In Section 4.3
Data We used the Qwen2.5-7B I-Inttrcct model to generate ground truth labels for classification. The correctness of its greedy-search response each query across all in-domain datasets used to create binary labels (1 for "capable, " 0 for "not capable")) We randomly sampled 10,000 instances for the training set, the same those used for the Self-awareness training of Dwwen255-7B-Innsrrue development set of 500 samples was created. For the final evaluation, we constructed a 10,000- sample test set by randomly sampling 5,000 instances from each class prevent class bias.
Training Details
BERT-based Classifier: We fine-tuned a classifier based on RoBERTa--bae (Liu et al.,
2019). The training parameters were follows: learning rate of 5e-6, batch size of 64, and a weight decay of 0.01. The model was trained for two epochs, saving the best checkpoint based on validation performance.
LLM-based Classifier: We fine-tuned classifier based LLAMA338B--nstmue
(Grattafiori et al.) 2024). The training parameters were: a learning rate of le-5, batch size of 64, and an evaluation on the development set every 20 steps. The model was also trained for two epochs, saving the best checkpoint.
A.8 WHETHER TASK CAPABILITY IMPROVED AFTER TRAINING?
Compared with most current routers, our method involves training the LLMs themselves. It would be an unfair comparison this training procedure were to enhance not only their self-awareness ability but also their inherent task-solving capabilities. To quantitatively assess this, we utilize the Pass K metric (Wang et al.) 2022) with
10. Specifically, for each query, we sampled 10 responses with temperature T 1. Consistent with the standard definition, query is considered "answerable by the model" at least one of the 10 samples is correct. We then measure the performance fluctuation, denoted as APass @ 10, to quantify whether the model's fundamental problem-solving capability has shifted after training.
Figure plots the APass@ @ 10 for RL-trained LLMs in the "Balance" scenario. As observed, the performance improvement is sufficiently small to be considered negligible. This aligns with our expectations because corrective information introduced during the SFT stage, and the RL stage
23

---

Published as a conference paper at ICLR 2026
APass
0.010
SFT
RL
0,85%
0.79%
0.8%
0.74%
0.005
0.50%
0.000
0.08%
0.58
1.5B
3B
7B
Model Size
Figure 8: Pass 10 for aligned LLMs with various sizes. The performance improvement (mea- sured by APass 10) for all models, trained with both SFT and RI., did not exceed 1% and can be considered negligible.
does not bring additional knowledge or expand the models' capability boundary, but only adjusts the models' behavioral preferences indicated by numerous existing works Yue et al. 2025 Kim ct al. 2025). Therefore, the overall utility increase of our routing system should be attributed to the enhanced self-awareness ability of each model, which is consistent with our motivation.
A.9 EXPERIMENTS ON INTRINSIC SELF-AWARENER
The DiSRouter paradigm relies fundamentally on the self-awareness of the constituent agents. In the main text, we utilized our proposed Self-Awareness Training Pipeline to enhance this capability in the Qwen2.5 series models. However, posit that truly reliable model should inherently possess high degree of intrinsic self-awareness, enabling it to autonomously reject queries it cannot correctly solve. In this section, we evaluate and analyze the native self-awareness of LLMs by using prompt engineering to guide models to either answer or reject queries. Our evaluation two-fold: (1) experiments open-source Qwen2.5 models, and (2) experiments SOTA closed-source GPT series LLMs. For both setups, we employ the identical prompt template, as shown in Table 25
A.9.1 EXPERIMENTS ON OPEN-SOURCE LLMS
We conducted experiments on four models from the Qwen2.5 series using the seven in-domain datasets described in the main text. To quantify self-awareness, we utilize the following metrics:
Answer rate (Ratiouns): The average proportion of queries the model chooses to answer.
Answer F1: We treat the decision answer or reject as a binary classification problem. Using the correctness of the model's ' response under standard evaluation as the ground truth, we calculate the score. Ideally, model should choose to answer only those queries can solve correctly and reject those it cannot.
Answered part accuracy (Accans): The accuracy on the subset of queries the model chose answer.
Rejected part accuracy (Accre)): The accuracy on the subset of queries the model rejected (had been forced to answer).
An ideal model should exhibit Answer Rate close its actual task accuracy, high Answer F1
score, and an Accans significantly higher than Accrej- The results are presented Table 16
Table 16: Self-Awareness evaluation of Open-Source LLMs (Qwen2.5-Instruct series).
Model Size
Ratio
F1
Accans
Accrej
0.5B
84.77
32.7
32.07
38.07
1.5B
6.45
23.46
60.59
58.9
3B
39.21
48.33
70.98
69.1
33.95
41.29
75.89
73,89
As observed, the open-source models exhibit poor intrinsic self-awareness. They demonstrate unstable Answer Rates and low Answer F1 scores, indicating an inability to accurately discern
24

---

Published as a conference paper at ICLR 2026
whether sample lies within their competence. Furthermore, Accans is not significantly superior to Accres-and in the case of the 0.5B model, it is even lower. This suggests that the rejection behavior of these models is largely random rather than reflection of true uncertainty. Based on these results, we conclude that current open-source LLMs lack sufficient inherent self-awareness for reliable routing without further alignment.
A.9.2 EXPERIMENTE ON CLOSED-SOURCE LLMS
Closed-source LLMs, such as the GPT series, are widely regarded as SOTA models. Given their strong task capabilities, we hypothesize they may also possess superior intrinsic self-awareness. We selected three models representing different capability levels and costs: GPT4.1-NANO. GPT4.1- MINI, and 04-MINI. Since these models achieve near-perfect performance on the in-domain tasks used in the main text (rendering them non-discriminative) we conducted this evaluation
suite
of more challenging mathematical reasoning tasks. These include GSM8K (Cobbe et al. 202 MATH-500 (Hendrycks et al. 202T). AIME 2024/2025 (AI-MO 2024b), AMC (AI-MO 2024a)
JEEBench (Arora et 2023). OlympiadBench (He et a 2024), and OlympicArena (Huang et al, 2025). For the routing experiments, GSM8K and MATH-500 are treated in-domain tasks (using
their training sets to train router baselines), while the others serve as out-of-domain (OOD) tasks.
We assigned normalized costs to these models proportional to their official pricing
The average
performance and cost of these models are summarized in Table I7
Table 17: Performance and normalized cost configuration for the GPT series models. ID: in- domain (GSM8K, MATH-500), OOD: out-of-domain (AIME, AMC, etc.).
Model Name
ID Accuracy
OOD Accuracy
Model Cost
GPT4.1-nam
86.07
42.02
0.1
GPT4.1-ming 1-mini
89.86
61.11
0.4
04-mini
92.34
79.52
1.0
We employed the same prompting to induce rejection behavior. Figure 9 iillustrates the relationship between Answer Rate and Performance for the GPT4.1-NANO model, and Figure I0 compares Accans against AcCrej across different tasks (GPT4.1-MIN exhibited similar trends). Two key observations emerge: (1) There a strong positive correlation between the LLM's Answer Rate and the task difficulty (performance), implying the model flexibly adjusts its rejection frequency based on difficulty. (2) The accuracy of the answered subset (Accans) is consistently and significantly higher than that of the rejected subset (Accrej). These findings confirm that these LLMs possess the intrinsic capacity to define their knowledge boundaries i.c., strong self-awareness.
DiSRouter with Intrinsic Self-Awareness To further validate this, we constructed a DiSRouter system using these three models, relying solely on their intrinsic rejection behavior for routing decisions without any additional training. We compared this against three router baselines under the "Balance" scenario (a = 0.5). The results are shown in Table [8] Leveraging the inherent self-awareness of the GPT series, DiSRouter achieved performance comparable to baselines in- domain tasks but outperformed them on out-of-domain tasks. This not only validates that an ef- ficient DiSRouter system can be constructed without training-proviine self-aware but also validates the poor generalization capability of existing router baselines.
the underlying LLMs are
We posit that self-awareness should be viewed as a critical dimension of LLM capability, essential for system reliability. It is plausible that the training of the GPT series already incorporates phases designed to mitigate hallucinations or incorrect responses, a capability that current open-source models appear to lack.
A.10 EXPERIMENTS ON DIFFERENT MODEL FAMILIES
In the main text, our experiments were conducted exclusively on the Qwen2.5-Inss -nnstrtct series. In this section, investigate the generalization capabilities of the DiSRouter framework and the Self-
hhtps///opeeai.com/////nnnnggg
25

---

Published as a conference paper at ICLR 2026
Answer Rate
1.0
Answered part
Rejected pari
Accuracy
0.8
0.67
0.57
0.6
0.5
0.62
0.4
0.56
A² =
0,22
0.38
p-value 9.66e-05
0.2
40 Performance
0.0
0.18
0.14
0.15
COMA
MATI1--000
ADME
AlMES
AMCn
Olym Bank
Op Arena
In-domain
Trend Line 95% C1
Out-of-domain
Task
Figure 9: Correlation between Answer Rate and Model Performance (Task Difficulty) for GPT4.1-mano. The strong positive correlation indicates that the model autonomously rejects more frequently on harder tasks.
Comparison of accuracy on answered queries (AcCana) versus rejected queries (Accrez) The model achieves higher ac- curacy questions it chooses to answer, demon- strating effective self-assessment.
Figure 10:
Table 18: Comparison of DiSRouter against router baselines on GPT series models under the Balance scenario. DiSRouter demonstrates superior generalization on out-of-domain (OOD) tasks.
In Domain
Out of Domain
Method
Accuracy f
Cost
Utility 1
Accuracy
Cost 1
Utility
Topline
Oracle
0.97
0.18
0.88
0.82
0.58
Smallest LLM
0.86
0.10
0.81
0.4
0.35
Naive
Largest L.M
0.9
1.00
0.40
0.8
1.00
0.30
Baselines
Random
0.89
0.49
0.65
0.62
0.53
0.36
Rontel
Router Baselines
FrugalGPT
0.87
0,16
0.79
0.43
0.13
0.37
GraphRouter
0.92
0.4
0.72
0.57
0.39
0.38
DiSRouter
GPT-series
0.84
0.17
0.76
0.61
0.41
0.41
Awareness Training pipeline across diverse model families. Benefiting from the inherent "plug-and- play" modularity of DiSRouter, we modified the original five-agent configuration by removing the 0.5B, 1.5B, and 3B models and substituting them with aligned LLMs from other model families. Specifically, we incorporated Gemma3- B---IBBruccttt (Team et al., 2025) and Phi4-mini Inst-utt (Ab- din et al. 2024) to construct a heterogeneous four-agent routing system (comprising models of sizes 1B, 4B, 7B, and 14B).
We applied the Seff-Awareness Training pipeline to these two newly introduced models, maintain- ing the exact training configuration described in the main text equip them with rejection capabili- ties. We evaluated the self-awareness of these aligned LLMs using the metrics defined in Appendix A.9.I with results presented in Table 19 The results demonstrate that our Self-Awareness Training pipeline generalizes effectively across different model families, significantly enhancing model reli- ability. Meanwhile, the characteristic of scenario adaptability is also generalized, and the behavior of the model is adjusted in line with expectations as the requirements of the scenario change.
Furthermore, we trained router baselines on this four-agent configuration and compared them with DiSRouter. The results, summarized in Table 20 indicate that DiSKouter consistently achieves superior utility. These findings confirm that the advantages of DiSRouter over traditional centralized routers are generalizable, provided the constituent models possess high degree of self-awareness.
A. EXPERIMENTS ON OPEN-ENDED GENERATIVE TASKS
In the main text, experiments focused verifiable tasks where clear, objective ground truth exists. In this section, we extend the Awlrenesss Training pipeline to open-ended generative tasks, such text summarization and creative writing. A fundamental premise of any routing system
26

---

Published as a conference paper at ICLR 2026
Table 19: Self-Awareness evaluation of aligned LLMs from different model families across three scenarios.
Backbone Model
Performance First:
0.2
Balance: 0.5
Cost First: 0.8
Ratioass
Accana
Accres +
Ratiouss
Accana
Accres 4
Rationss
Accana 1
Accres
Gemma--3 3-1B-instruct Phi Phi-4-mini-ti instruct
6.58
85.34
38.27
8.82
76.66
38.31
11.98
68.03
38.75
64.51
86.89
58.39
72.07
85.64
56.62
76.58
84.61
54.84
Table 20: Performance comparison of DiSRouter and baselines the heterogeneous four-
agent system. DiSRouter maintains superior utility, demonstrating strong generalization across model families.
Method
Performance First:
0.2
0.5
0.8
Accuracy
Cost
Utility
Accuracy
Cost
Utility
Accuracy
Cost
Utility
Topline
Onicle
0.94
0.41
0.86
0.94
0.41
0.74
0.94
0.41
0.61
Smalicst LLM
Baselines
Largest LLM
0.85
0.90
0.67
0.85
0.90
0.40
0.85
0.90
0.13
Random
0.71
0.56
0.60
0.71
0.56
0.43
0.70
0.56
0.25
Routel.LM
0.78
0.64
0.78
0.64
0.46
0.78
0.64
0.27
Router Baselines
FrugalGPT
0.81
0.68
0.67
0.66
0.35
0.49
0.49
0.19
0.34
GraphRouter
0.72
0.70
0.26
DiSRoater
SFT
0.82
0.64
0.69
0.80
0.59
0.51
0.80
0.58
0.34
(Ours)
+ RL
0.82
0.56
0.71
0.80
0.53
0.54
0.80
0.52
0.38
that the task utility must quantifiable. Unlike verifiable tasks (e.g., QA or Math) where the evaluation score A(m, x7) is binary {0, 1). open-ended tasks typically yield continuous quality score S(m, .)) E [0, 1]. An unevaluable task renders routing meaningless, as the optimal strategy would trivially default the cheapest model. Therefore, we generalize the binary scoring function used in the main text to continuous scoring model, which can be instantiated as learned reward model an LLLM-as-a-Judge
A.11.1 METHODOLOGY ADAPTATION
To accommodate continuous scores, we adapted both the SFT and RL stages of our pipeline.
SFT Data Construction In the verifiable setting, we determined rejection based on a fixed prob- ability threshold da factor a. To address this, we employ a percentile-based thresholding strategy. We define the deci- sion threshold da as the a--th percentile of the base model's score distribution on the training set. Which can be formulated as:
However, continuous scores lack direct mapping to the preference
8a (S- Dtrain))
(11)
For example, in the "Performance First" scenario (a- = 0.2). the threshold is set to the 80th percentile score (Pso). Consequently, the model is trained to answer only if its expected output quality belongs to the top 20% of its capability, and to reject otherwise. This data-driven approach effectively preserves the system's Scenario Adaptability within continuous scoring landscape.
RI. Reward Formulation We extend the reward function to align with this percentile logic. The reward for given sample is defined as:
x(, if answer
r(z, a)
(12)
Pi-a
if reject
Here, Pi-a serves the baseline reward for rejection. The model learns answer only when its generated response quality 5(m, ær) is expected to exceed the threshold Pi-a: mirroring the logic of the verifiable task setting.
27

---

Published as a conference paper at ICLR 2026
A.11.2 EXPERIMENTAL SETUP AND RESULTS
We validated the effectiveness of this extended pipeline on the text summarization task, utilizing the SAMSum (Gliwa et al. 2019) and XSum (Narayan et al. 2018) datasets. We employed BERTScore
Zhang et al. 2020) as the evaluation metric, using the recommended DEBERTA-XLARGE. -LAAGE--AANNNN (He) et al. 2021)> as the backbone. The baseline BERTScore performance of the Qwen2.5 InIstrucct series is presented in T2b We observe that performance relatively saturated across model sizes a common challenge in evaluating open-ended generation. While more sophisticated reward models could offer finer granularity, our primary objective is to assess whether Self- Self-Awareness Training can effectively teach the model to reject samples prone lower scores based on the given metric signal.
Table 21: Baseline performance (BERTScoree of Owen2.55Innsrru. series models SAMSum and XSum datasets. The relatively narrow performance gap across model sizes reflects the charac- teristies of the metric and the task.
Task
0.5B
1.5B
3B
7B
SAMSsum
67.5
68.97
69.64
70.71
XSum
64.4
66.21
65.04
67.34
Average
65.95
67.59
67.34
69.03
We conducted Self-Awareness Training on four model sizes, maintaining the same hyperparameters as in the main text. The training dataset consisted of 4,000 samples per scenario (Total 12,000 samples). The Answer Rate and the BERTScore comparison between answered and rejected subsets illustrated in Figure I and Figure 12 respectively.
Answer Rate
Balance
Perfomancce
77.4%
78 284%
Bert Score
0.8
0.71
0.75
0.73
Cost Fire
0.7
54.2%
50
36.1
33.0%
M.35%
0.6
0.66
0.64
0.65
23.29
Answered part
Rejected part
0.50
1.50
7B
Model Size
0.5B
1.5B
3B
7B
Model Size
Figure 11: Answer Rate of models summa- rization tasks across different scenarios. The models exhibit consistent scenario adaptability, answering fewer queries as requirement for performance (high scores) increases.
Comparison of BERTScore be- tween answered and rejected queries. The models achieve consistently higher scores on queries they choose to answer.
Figure 12:
The results indicate that: (1) The trained models retain robust Scenario Adaptability, flexibly adjust- ing their answer rates according to the preference factor o; and (2) The average BERTScore of the answered subset is significantly higher than that of the rejected subset. This demonstrates that the model successfully learns anticipate its performance and reject lower-quality outputs. These find- ings suggest that our Self-Awareness Training pipeline generalizes effectively to open- open-ended tasks, provided reliable reward model is available.
1.
PROMPTS AND TEMPLATES
A. 1.12 SCENARIO PROMPTS
To imbue each DiSRouter-alignet agent with Scenario Adaptability, prepend scenario-specifee instructions to the prompt each query, as detailed in Table [2 This ensures that the agent's behavior aligns with the requirements of the given scenario.
28

---

Published as a conference paper at ICLR 2026
Table 22: Prompts for each scenario.
Scenario
Prompt
Scenario: Performance First, You must ensure your answer is correct. Only choose to answer if you are completely certain of your ability to answer the question; otherwise, choose not to answer.
Performance Fisst
Scenario: Balance. You need strike balance between answering and not answering. can choose to answer questions you confident and choose answer those you are uncertain about.
Balance
Scenario: Cost First. You need to answer questions as much as possible. Choose to answer as long as you believe you have the ability to answer correctly: only choose not to answer if you feel there is absolutely no possibility of giving correct answer.
Cost First
A. 12.2
SELF-AWARENESS TRAINING RESPONSE TEMPLATES
When conducting Self-Awareness Training, a model can choose to either answer or reject a query. In the SFT stage, we use distinct response templates for these two behaviors, as shown Table 23 For rejection, the model directly responds with "I don't 't know!" For answering. we use an extraction LLM (Qwen2.5-14p) 1-n-2.5--4B-Innsrrccl to extract the thinking steps and the final answer from the original Qwen model's response on the training samples, and then use the "Answer" template in Table 23 to construct the training data. The prompt used for the extraction of the large model is shown in Table
24
Table 23: Response template for training data.
Action
Template
Answer
Let's think step by step: < <thinking steps>>.
The final unswer is: <final answer>>
Reject
I don't know!
Table 24: Instruction for the Extraction LLM. The model extracts thinking steps and final answer from the responses of the original LLM.
Extraction Prompt
There a question and the corresponding output from model. would now like you to help me extract the reasoning part and the final answer from the model's oulput. Please note that only the reasoming part and the final answer needed, and any other content should be excluded, and do not generate any other prefix or suffix. Use the following template reply: The Extracted Reasoning Parts: <>>, The Extracted Final Answer: <>, Now, start the task: Input Question: Question:'/n<quuee <<<question>>.. \n Model Output:colputt> The content you extracted:
A. A..3 REJECTION PROMPT FOR INTRINSIC SELF- AWARENESS EVALUATION
29

---

Published as a conference paper at ICLR 2026
Table 25: The prompt template used to evaluate intrinsic self-awareness. The same prompt is applied to both Open-source and Closed-source experiments.
Rejection Prompt
You are "Hyper-Cautious Expert" AI. Your primary directive is answer only when you 100% certain of the entire, complete, and accurate
You must follow these rules:
If you have ** ANY DOUBT at all because the question is ambiguous, complex, unverifiable, unsolved problem, or you are 100% sure of the correct steps you **MUST** refuse. Your **ONLY** response must the three-word phrase: don't know!"
2. If you are 100% CERTAIN** you can provide a flawless, correct, and complete solution, you **UST** provide the answer. Your answer == follow this exact format:
It must begin with "Letts think step by step."
Your reasoning must be #Fconcise and #t focusing *tonly on the key logical steps*#3
Avoid verbose explanations. Your entire response must be as brief as possible, ideally under 2000 tokens. must conclude with the final answer enclosed in "\ (boxed{))" All mathematical formulas and symbols you output should represented with LaTeX!
*CCRITICAL INSTRUCTIONS]*
+*DO NOT** apologize you refuse.
**DO NOT** explain *why* you don't know.
+*DO N provide partial answers.
Your choice is binary; either concise, reasoned, boxed answer (Rule 2) or simple refusal (Rule 1).
*(BEHAVIORAL EXAMPLESJ**
[FIVE SHOTS]
**[TASK SEGINS]**
Now, apply these rules strictly to the following question.
30


Reviews:
[{"review_id": "XKc4HTHIu8", "reviewer": "Reviewer_xhvr", "summary": "This paper introduces **DiSRouter (Distributed Self-Router)**, a framework that replaces centralized query routing in LLM systems with a distributed paradigm where each agent autonomously decides to answer or route queries based on self-awareness. Key contributions include:\n\n- A **distributed routing architecture** that enhances modularity and scalability compared to centralized routers.\n- A **Self-Awareness Training pipeline** with supervised fine-tuning (SFT) and reinforcement learning (RL) using a localized reward function, enabling independent agent training.\n- **Scenario adaptability** via a preference factor $\\alpha$ to balance performance and cost dynamically. Experiments on datasets like GSM8K and MMLU show DiSRouter outperforms baselines (e.g., RouteLLM, FrugalGPT) in utility. The core idea is that intrinsic self-assessment is more effective than external evaluation.", "questions": "1. I am still confused that how the the local routing policy $\\pi_i$ of agent $m_i$ is trained, it seems that $m_i$ only output \"I don't know!\" if choosing reject, how does $m_i$ select the next agent?\n2. Can DiSRouter's self-awareness training be applied to LLMs with different architectures (e.g., encoder-decoder models)? Please provide results with at least one non-Qwen model family to validate broad applicability.\n3. How does DiSRouter perform with large-scale agent pools (e.g., >10 agents)? \n4. What are the practical limitations (e.g., latency, fault tolerance) in deploying DiSRouter? Case studies or simulations with real-world constraints would be valuable.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Originality**: The shift from centralized to distributed routing is novel, creatively integrating self-assessment concepts (e.g., LLM uncertainty) into a routing framework. The scenario-adaptive reward function is an innovative touch.\n- **Quality**: Experimental design is rigorous within its scope, with utility metrics and modularity tests. The pipeline is methodically described.\n- **Clarity**: The problem formulation in Section 2 is precise. Writing is concise.\n- **Significance**: The \"plug-and-play\" modularity addresses a key pain point in LLM systems, though real-world impact depends on scalability beyond the tested settings.", "weaknesses": "- **Limited Generalizability**: Experiments use only the Qwen2.5-Instruct series, neglecting other LLM families (e.g. Llama). This casts doubt on DiSRouter's applicability to diverse models. The authors should include cross-architecture validation.\n- **Training Efficiency Concerns**: The Self-Awareness Training (SFT + RL) is computationally heavy, but the paper dismisses routing cost as \"negligible\" without quantifying training overhead. A cost-benefit analysis is missing.\n- **Limited Robustness Testing**: There is no evaluation under adversarial conditions (e.g., ambiguous queries) to test the robustness of self-assessment. This is critical for real-world reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IR7pPRVdkL", "reviewer": "Reviewer_qdad", "summary": "This paper proposes DiSRouter—a distributed self-routing framework that can propagate queries step-by-step through different LLM agents without a central router.\nA two-stage Self-Awareness Training (SFT+RL) is designed to teach individual LLMs to \"answer if they know the answer and reject if they don't,\" and the rejection threshold can be adaptively adjusted according to user preferences (performance priority/balance/cost priority).", "questions": "1.In Section 3.2.2 and Appendix C, what is the purpose of extracting and reformulating the original model's answer using a stronger model? The intervention of the model may introduce additional errors and information, resulting in unpredictable performance after training.\n\n2.Is the true \"routing overhead\" underestimated? There may be multiple levels of consecutive rejections on the cascading path (the experiment used a 5-model approach, and the worst case was only 4 rejections to reach 14 bytes). Could you please provide the end-to-end latency of the first token for comparison with other methods?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1.Experiments show that DiSRouter's utility score (performance-α·cost) is significantly superior to both baselines and existing router systems across various performance-cost scenarios, with both in-domain and out-of-domain data. It is more flexible and has \"plug-and-play\" modular capabilities.\n\n2.The hypothesis that training does not inject new knowledge but only improves self-boundary awareness can be supported by experimental results: ∆Performance <1%, which verifies that the performance improvement mainly comes from better routing rather than the model itself becoming stronger.", "weaknesses": "Although the experimental results have shown that the authors' method is far superior to the baseline and other routing methods, I believe that some additional perspectives are still beneficial for future work:\n\n1.The cost introduced by the two-stage training approach may require further evaluation or comparison with other methods, as each agent needs to perform independent multiple-time inference to prepare its SFT data and conduct RL training.\n\n2.For more open agent systems, data preparation is more challenging, as some questions do not have a single correct answer (e.g., summarizing, creation/generation questions) while still have varying difficulty, making the self-awareness training phase tricky. If only verifiable questions data are synthesized and trained, unprepared tasks may suffer from catastrophic forgetting, and the model's self-awareness on such tasks cannot be guaranteed. The author may provide the performance of the model and the systems on these more complex out-of-domain tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6yw99h60N0", "reviewer": "Reviewer_dfTT", "summary": "The paper introduces a distributed routing paradigm, enabled by a concrete self-awareness training pipeline, to address LLM cost-performance trade-offs. However, the core claims are undermined by significant methodological flaws, including the failure to account for cumulative latency and a lack of generalization evidence beyond a single, homogeneous model family for narrow, verifiable tasks.", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a \"Distributed Self-Routing\" framework, replacing the external centralized router with intrinsic \"self-awareness.\" \n\n2. It provides a concrete technical contribution with the two-stage \"Self-Awareness Training\" pipeline (SFT + RL), enabling models to internally assess their knowledge boundaries and execute a \"reject\" action.\n\n3. The framework is inherently modular (\"plug-and-play\").", "weaknesses": "1. The core utility metric (Utility = Performance - α * Cost) is flawed. It completely ignores the significant cumulative latency incurred by the cascade structure. A query rejected multiple times will have a real-world response time far exceeding that of a single-shot centralized router, a critical cost factor this paper overlooks.\n\n2. The experiments are confined to a homogeneous model pool (all Qwen-family models). This is a major threat to validity. The paper provides no evidence that the \"self-awareness\" training paradigm generalizes to a realistic, heterogeneous ecosystem of diverse model architectures (e.g., Llama, Mistral, Gemma), which is the primary motivation for such a system.\n\n3. The entire methodology (self-awareness training, binary accuracy evaluation) is fundamentally dependent on tasks with clear, verifiable answers (e.g., math, multiple-choice QA). It is unclear how this approach would apply to open-ended generative tasks (e.g., summarization, creative writing) where \"correctness\" is ill-defined, severely limiting its practical scope.\n\n4. The evidence supporting the superiority of \"intrinsic assessment\" is marginal. In Table 5, the purpose-trained 7B DiSRouter (0.80 F1) barely outperforms a general-purpose 8B external classifier (0.77 F1). This negligible gap fails to provide a strong justification for the added complexity of the distributed training approach over a simpler, well-trained external router.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PfjBSj7pTI", "reviewer": "Reviewer_JRcd", "summary": "A recent strategy to minimize compute involves routing user prompts to an LLM which has just the right size to handle the request, instead of always prompting the most capable model.\n\nCurrently, the most popular routing strategy involves using a routing model, which determines which LLM to prompt with the user's request. This paper proposes to instead integrate the routing process directly into the available LLMs. Specifically, the LLMs are trained to defer to the next biggest model when they deem they are not capable of providing a correct answer. The paper shows that this results a better balance between cost and accuracy.", "questions": "None", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem formulation is clear and the motivation is sound.\n- This idea will be of interest to the community.\n- The results show that the idea has merit and should be further investigated.\n\nOverall, I believe that this paper should be accepted, although authors should address the weaknesses mentioned below.", "weaknesses": "My main issue is that the authors barely discuss/explore the (potential) limitations of the method. For instance, the method first prompts the smallest model. Because of its size, it might sometimes struggle to properly reason whether it is capable of accurately answering a given question. Although the results seem positive, have the authors observed such phenomena, our found that for certain type of tasks this happened at a greater rate?\n\nAnother issue is the computational cost of the method. The most difficult prompts must go through all models before the final largest model answers. This is the worse case scenario, and it is a downside of the method. Authors do not talk about this much. The method does seem more costly than certain router baselines, but performs slightly better. Have the authors considered potentially skipping an LLM when say the first model's confidence is very low, or outputting more nuanced reject answers, such as \"I don't now.\", \"I really don't know.\", and depending on such output, skip an LLM or two in the chain?\n\nAnother important question is how does internalizing the routing decision degrade a model's capabilities compared to standard instruction-tuning. It is pretty important to know if this creates a gap and if so, how big is it.\n\nAside from discussing the (potential) limitations of the method, the paper's methodology section would also gain from being clearer. The current description is somewhat high level and leaves a lot of the details unspecified. For instance, during RL fine-tuning, how are the model's answers judged as correct or incorrect?\n\nFinally, it would be interesting to know how the authors designed the prompts for the \"Performance First\", \"Balance\", and \"Cost First\" scenarios in Table 7 of the Appendix. Were multiple prompts tested or did the authors simply use the first prompt that came to mind?\n\nAddressing these aspects would help strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)