Universal Model Routing for Efficient LLM Inference
Authors: Wittawat Jitkrittum
Harikrishna Narasimhan
Ankit Singh Rawat
Jeevesh Juneja
Congchao Wang
Zifeng Wang
Alec Go
Chen-Yu Lee
Pradeep Shenoy
Rina Panigrahy
Aditya Krishna Menon
Sanjiv Kumar
Venue: ICLR 2026 Poster (2026)
Keywords: model routing
adaptive computation
learning to defer
efficient inference
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a principled model routing framework which allows new models to be added to or removed from the serving pool without having to retrain the routing model.

Abstract: Model routing is a simple technique for reducing the inference cost of large language models (LLMs), wherein one maintains a pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning a router for a fixed pool of LLMs. In this paper, we consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, a new approach to this problem that relies on representing each LLM as afeature vector, derived based on predictions on a set of representative prompts. Based on this, we detail two effective instantiations of UniRoute, relying on cluster-based routing and a learned cluster map respectively. We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound. Experiments on a range of public benchmarks show the effectiveness of UniRoute in routing amongst more than 30 unseen LLMs.

Published as a conference paper at ICLR 2026
UNIVERSAL MODEL ROUTING FOR EFFICIENT LLM INFERENCE
Wittawat Jitkrittum
Harikrishna Narasimhan hmarasimhan@googleee
Ankit Singh Rawat
sattawatjagmail... com
wewitssrawat@google.ccoom
Jeevesh Juneja
Congchao Wang
Zifeng Wang
beeveshjuneja@google.cmm
ccwangāvt.edu
zi fengwagoogle..om
Alec Go agoāgoogle.com
Chen-Y Yu Lee chenyulee@@ooglee.co
Pradeep Shenoy shenoyoradeep@google.cco
Rina Panigrahy
Aditya Krishna Menon adi tedttykmmonn@@gggle.ccco com
Sanjiv Kumar sanj weiiiikkggge..om
rinapāgoogle.com
Google
ABSTRACT
Model routing is simple technique for reducing the inference cost of large lan- guage models (LLMs), wherein maintains pool of candidate LLMs, and learns to route each prompt to the smallest feasible LLM. Existing works focus on learning router for fixed pool of LLMs. In this paper, consider the problem of dynamic routing, where new, previously unobserved LLMs are available at test time. We propose UniRoute, new approach to this problem that relies on representing each I.I.M as a feature vector, derived based on predictions on set of represen- tative prompts. Based on this, we detail two effective instantiations of UniRou relying cluster-hased routing and learned cluster map respectively. We show that these are estimates errors via an excess risk bound. Experiments range of public benchmarks show the effectiveness of UniRoute routing amongst 30+ unseen LLMs.
theoretically optimal routing rule, and quantify their
INTRODUCTION
Large language models (LLMs) have seen a flurry of development (Radford et al., 2018; 2019; Brown al., 2020; Touvron et al., 2023; Anil al.. 2023; Grattafiori et al., 2024: DeepSeek-Al et al., 2024). While LLMs have demonstrated impressive abilities, their inference cost can be daunting (Li al., 2024a; Wan al., 2024; Zhou et al., 2024b). Several techniques aim to ease this issue, such as speculative decoding (Stern et al., 2018; Chen et al., 2023a; Leviathan et al., 2023), early-exiting (Schuster et al., 2022), quantisation (Chee et al., 2023), compression (Frantar & Alistarh, 2023; Agarwal et al., 2024; Rawat al., 2024), and others (Pope et al., 2023; S et al., 2024).
Our interest is in model routing efficient inference. Here, one maintains pool of candidate LLMs of various sizes and capabilities. Given prompt, one learns to predict the lowest-cost LLM which can reasonably address the prompt. In doing so, one can learn to use high-cost LLMs sparingly, only the (relatively) few "hard" inputs. This is conceptually simple but effective technique, which has seen a surge of recent interest (Hendy et al., 2023; Hari Thomson, 2023; Ding et al.,
2024; Šakota et al., 2024; Chen et al., 2024b; Hu et al., 2024b; Shnitzer et al., 2023; Wang et al.,
2023; Stripelis et al., 2024; Ong al., 2025; Zhuang al., 2024; Srivatsa et al., 2024; Feng et al., 2024; Lu et al., 2024: Zhao et al., 2024b: Dann et al., 2024; Aggarwal et al., 2024; Lee et al., 2024a; Mohammadshahi et al., 2024; Chuang al., 2025; Huang et al., 2025).

---

Published as a conference paper at ICLR 2026
Clustered training sample (unlabeled)
Training example
Valiation example
Cluster centroid
Cluster membership
*
*
*
Cost adjustment Model
Error scores
Prompt
Promet Embedder
1.0
0.0
0.0
0.3
LLM, 0.5
0.6
LLM2
0.1
LLM,
1688
LLM Embedder
0.3
0.6
0.0
Dot
product
LLM,
0.6
0.2
0.4
Unseen during training
*
*
*
argmin
Per-oluster errors
Labelled validation semple assigned training clusters
Route to LLM,
Figure Illustration of our proposed UniRoute with a cluster-based router (see $5.1). We first perform K-means on training set to find K centroids, and then partition the validation set into K representative clusters. Each test-time L.M can then represented as K-dimensional feature vector of per-cluster errors. This yields an intuitive routing rule: for each test prompt, we route to the LLM with the smallest cost-adjusted average error on the cluster the prompt belongs to. The prompt embedder may either be completely unsupervised (as shown in the figure), or fitted via supervised learning using labels from a set of training LLMs different from those seen during test time ($5.2).
Existing works largely focus on routing over a fixed pool of LLMs. In practice, however, the pool of LLMs can constantly change; e.g., older LLMs may be deprecated in favor of newer, more performant LLMs. To leverage such new LLMs, perhaps the simplest approach is retrain the router. However, with frequent changes to the LLM pool, this may be impractical owing to non-trivial overhead from repeated retraining and redeployment (see $3 for further discussion and considerations)
In this paper, we propose UniRoute, a new approach to this problem based representing each LLM as a feature vector, derived from their prediction errors on a set of representative prompts. By learning router over these LLM features, we enable generalisation to previously unseen LLMs without any retraining. Building on the observation that K-NN routing (Hu et al., 2024b) a special case of UniRoute, detail two concrete instantiations of UniRoute relying supervised prompt clustering respectively. While conceptually simple, empirically, these enable effective routing with a dynamic LLM pool across several benchmarks. In sum, our contributions are:
wasupervised and
we formalise the problem setting of model routing with dynamic pool of LLMs ($3);
iii) we propose UniRoute ($4), a novel routing approach relying on an LLM feature representation based on its prediction error vector
small set of representative prompts ($4.
(iii) propose two simple yet effective instantiations of UniRoute based on unsupervised or super- vised clustering 1$5.1, $5.2, Figure 1), with an accompanying excess risk bound ($5.3);
(iv) empirically illustrate ($6) UniRoute effectively handles 30+ new LLMs across multiple benchmarks (Zhuang et al., 2024; Hu et al., 2024b; Ong et al., 2025; Somerstep et al., 2025).
BACKGROUND: MODEL ROUTING WITH A STATIC LLM POOL
Language models (LMs). Given a finite, non-empty vocabulary of tokens V, a language model (LM) a probability distribution p over sequences of tokens V* = U V" Our interest is in using large language models (LLMs) for prediction problems mapping input prompts x E X C V to targets y y. Our goal is construct a predictor h: X + minimising R(h) = Ei [f(x y, h(x))] Here, f(x: y. /())) denote loss function measuring the loss of predicted response h(x) on given (prompt, target) pair (x, y). For example, f(x, y, h(x)) = 1 # h(x)) penalises responses that are not exact string matches of the target. We may construct predictor using LLM p via
2

---

Published as a conference paper at ICLR 2026
h(x)) = predict(p(- z)). Here, predict is some (task-specific) prediction function mapping LLM output strings to targets; e.g., we may employ a standard decoding algorithm (Ficler Goldberg, 2017: Fan et al., 2018; Holtzman et al., 2020), followed by stripping non-numeric tokens.
Model routing. Model routing is a means for achieving inference efficiency by sparingly calling larger models only on "hard" prompts. More precisely, suppose we have M 2 2 LLMs (y())) Jm
with corresponding inference costs {c!")) [me[M) denoting, c.g., their average monetary costs for [
processing one prompt. We assume e!)) < x( S... .... cMM) Let h(m) = predict(p(m)) x))
denote the predictor for the m LLM. re :: X - [M] be a router that, given a prompt, predicts the most suitable L.MM. The standard ("stattic") routing problem seeks to solve (ef. Chen al. (2023b);
Dekoninck et al. (2025); Woisetschläger et al. (2025); Somerstep et al. (2025))
min 1: 1::2-|M JM]
Σ F( ((((()) = m) f( y, hm)] Mc[[M]
Σ Mc[M]]
F(a
[(((a)) = m) c.(m)
S B. (1)
Here, B 2 0 is some fixed budget on the cost of the routed solution.
Model routing strategies. The most naive routing strategy randomly assigns prompts to the various models, potentially pruning inadmissible solutions (see Appendix D). A more refined strategy is to learn a router (see Appendix 1). Hu al. (2024b) (see also Narasimhan al. (2022)), proposed to construct predictor (mm)
X + I of the expected loss incurred by each LLM (mm),, and route via
r(x) argmin mE(M]
(m)) (2)) + x cl
(x))
Here, À 2 is hyper-parameter trading between cost and quality. In $5, we formalise that (2) is a plug-in estimator of a theoretically optimal routing rule (cf. also Somerstep et al. (2025)). Given training sample Sy = {xx(1, ()) j there are several approaches construct y. For example, given a text embedder :p: X - IR Dr (e.g.. BERT (Devlin et al., 2019), Sentence-T5 (Ni et al., 2022)):
(m)
(x) w ( + bm,
Vim
where Wm E RDr, bm E IR and (optionally) yp may be fit with suitable empirical loss such as the mean squared error. A related matrix factorisation approach operating over frozen embeddings was proposed in Ong et al. (2025), fitted on either pairwise (Ong et al.. 2025) or pointwise (Zhao et al.. 2024a) supervision. Alternativll,, we may use a K-NN estimator (Hu et al., 2024b, Section 5.1):
(m²) TENN (x=)
Σ
((x(1) yy( (1((m)))
(a,,)
where NN(x, denotes the k-nearest neighbours of (the embeddings of) x in S..
3 MODEL ROUTING WITH DYNAMIC LLM POOL
We now formalise the model routing problem when the set of LLMs may vary dynamically.
Routing with dynamic pool: motivation. The routing setup in (1) assumed a static pool of LLMS: indeed, (3) only estimates parameters ((wm; mE[M] for fixed pool of L.MMs, namely, (pim)) Jee[M]- We are interested in routing with dynamic pool of LLMs. The importance of this setting is twofold. First, new models are frequently released, and old models can be deprecated. It is highly desirable for router reflect these changes: e.g., smaller LMs are especially released very frequently, and leveraging them is critical for effective routing in low-cost regime. Second, even with fixed set of candidate I.LMs, the pool of L.MSs available for routing at test-time can change dynamically. For example, routing API services may changes owing the availability of GPUs for serving, licensing requirements, and suitability to the user's particular task.
There are two natural strategies to handle such dynamic LLM pool: (i) reuse an existing router trained against an older pool, or (ii) retrain a router with each change to the pool. While simple, neither approach is wholly satisfactory. The first approach leverage newer LLMs can result in a poor user experience for end-users a routing system. The second approach can incur non-trivial overhead (cf. Appendix H): for every newly released L.M,, one
be practically undesirable: failing to
3

---

Published as a conference paper at ICLR 2026
needs to perform router retraining and re-deployment. Further, such retraining requires annotating labelled samples with each new LLM, which runs the risk of overfitting when using too few samples, or incurs high annotation cost otherwise. These motivate the following alternate setup.
Routing with a dynamic pool: setup. Let Han denote the set of all possible LLM predictors, where for simplicity assume [Haul +oo. Let HH = 2 denote the set of all subsets of Hanl Let Har = (her ( ,.,, ( evaluation, seek to route amongst HLe { (1)
E Hi denote the set of L.M predictors observed during training. During
..... ) H. The original routing problem in (1) corresponds to Hir = Han However, we aim to allow Har o Hta including H4r n 2.
To accommodate such dynamic LLM pool, first modify our router accept both a prompt and a set of candidate LLMs, with the goal to pick the best option from this set; i.e., we consider dynamic routers R = (r(-;))): X - [[]]]] H. H). Next, assume that the set of training LLMs H. is itself drawn from some mneta-distribttion 5) over H. Rather than perform well on the specific set H.r. would like generalise any set of LLMs drawn from h. Concretely, wish to solve:
1
min E rEc
1(r(x, m)) (xx y, h(m)) m
Σ
E
Σ 1(r(x,)() = -m) -e(hlmi) mE||33]
S B,
(5)
where before 2 denotes a cost budget, H = (h(") A(M) 5 denotes a sample of M LLMs. c: Fanl + R denotes the cost of given LLM, and E is a shorthand for (m...
Bayes-optimal routing with a dynamic pool. To guide the design the Bayes-optimal rule for (5). Interestingly, the Bayes-optimal rule decomposes across each of the constituent LLLMs. The result is known for fixed pool H E H (Jitkrittum et al., 2023, Lemma F.I), (Dekoninck et al., 2025; Somerstep et al., 2025). The distinction arises for varying pool, where the result closely mirrors (Tailor et al., 2024, Eq. 6), derived in related setting (see Appendix I).
dynamic router, we study
Proposition 1 (Optimal dynamic routing). Under mild regularity condition on IP, for any input X, LLM pool H. E H. and budget B > there exists a Lagrange multiplier Xs 2 0 such that the optimal dynamic router r* for the constrained optimization in (5) is
r" (x) = argmin Egll [e(z.y, y, h(mo) Xo c(h(m)). me[]a[]]
The above uses standard L Lagrangian analysis to convert the original constrained objective into an unconstrained one. Intuitively, it is optimal to route the model with the lowest expected loss, after applying cost adjustment of Äs c(h(m)) to the loss. The parameter \s the expected quality and the average cost. one wishes to sweep B to trace a cost-quality deferral curve (Appendix F.2), one may equally treat Ào tunable constant. See also Appendix E.
allows trade off
Consider a setting where an LLM response may be compared to a ground-truth target based on an exact match criteria: ie., we pick the 0-1 loss f(x, y, ((m)) = 1[h(m) (x) + y). with values either (incorrect) or I (correct). Here, the optimal rule in (6) becomes:
r" (x,,H) argmin (1x,))))) As c(h(m))) where (" TP y 7 M(x))) mE]]K[]]
mr]
For simplicity, we will focus on the the 0-1 loss henceforth, and consider (7) as the optimal router; our results can readily adapted (as we shall see empirically in $6) other loss functions via (6).
Plug-in routing with a dynamic pool. The above suggests a simple router for dynamic LLM pool Hap = =(h(r))
(1) INe[N]: construct a plug-in estimator 7(x., h) of 7" (æx, h), and estimate (7) via
r(x, Kie) argmin RE(N]
[>(2,, ((xe)((()) ((²)) x c((h(a))
(8)
A key question arises: how should parameterise h)² We study this question next.
4

---

Published as a conference paper at ICLR 2026
UniRoute: Routing with a Dynamic LLM Pool
Train base router. Fit parameters of Yuni from (9) on training set S = {(0(0) )
Embed test LLMs. Compute & (hee) for each test LLM hiv Hee. e.g,, via (10) on validation set Saa = {(*l) ())
(3) Route on new prompts. Given a new input prompt æ. pick the best test LLM via (8).
UniRoute: UNIVERSAL ROUTING VIA AN LLM FEATURE REPRESENTATION
To enable routing with a dynamic LLM pool, we propose to parameterise by representing both prompts and LLMs as feature vectors. Specifically, for fixed K > 1 let $ - IR. k and w Hall + R denote feature maps for prompts and LLMs respectively. Then, we propose:
Tuni (x. h) ((x)) (h).
(9)
We may now fit any parameters associated with + T on the training set Su., and then route as before via (8), Crucially, provided we define an easily computable V, this seamlessly handles any h E Hall. including one unobserved during training; this is analogous to semantic output codes for zero-shot classification (Palatucci et al., 2009). Thus, (9) provides an approach for universal routing with dynamic LLM pools. The bilinear form is a natural generalisation of the standard model identity based routing, wherein one could interpret the model embedding as a one-hot vector. To realise the potential of this approach, it remains to specify + (x)) and (h).
Prompt representation T. Good choices for ((x) E IR.K have been well-studied in prior work (Hu al., 2024b). standard choice is to leverage (a linear projection of) a general-purposs. text embedding such OpenAl (2025); Lee et al. (2025); Wang al. (2024b); Lee et al. (2024b).
LLM representation V. A good choice for (h) less apparent than that for ((x). Observe that linear router (3) for a static pool Hy = [h (1) "...,,A, ( corresponds to one-hot LLM representation oo (h) = 1(h ((()))
mE(M] which is inherently tied to the LLM pool Hg. We
now examine alternative representation based LLM's performance subset of prompts.
4.1 REPRESENTING AN LLM VIA THE PREDICTION ERROR VECTOR
A key desideratum for is that (h)) & (h') ought to be large for pair (h, h²) of "similar" LLMs, and small for a pair of "dissimilar" LLMs. We posit that two LLMs are "similar" if they have comparable performance on a validation prompt set (i.e., set of prompts following the same distribution as those encountered in deployment). This is akin to proposals in Thrush et (2024); Zhuang et (2024).
Concretely, suppose that we have access to small (labelled) validation set Syat (xx ) N.o Further, suppose that any LLM Hall including new LLMs unobserved during training can be evaluated efficiently on these prompts. Then, one may represent the LLM based on its prediction error vector on prompts from Swat: for suitable F: R. - R. K (e.g., a linear projection), we choose:
(h) = F ((1())) & ((200))) je(Nml)
R
(10)
A special case of this is the K-NN method (4) from Hu et al. (2024b) applied to Seat: for V km (h) (1(ي(5) * h(z(")))] fE[N..] E N and (2:) E {0, 1 indicating which validation samples are the k-nearest neighbours of æ, (9) exactly reduces to (4). general, however, it can prove useful to parameterise F and learn some compressed LLM representation in K < Nval dimensions.
4.2 DISCUSSION: UniRoute VERSUS STANDARD ROUTING
central assumption in UniRoute is that for any new LLM, one may efficiently compute (:):; for (-) given by (10), this in turn assumes that any L.MM can be efficiently evaluated on S.a.- This
5

---

Published as a conference paper at ICLR 2026
only requires black-box access to the LLMs in the pool (e.g., offered by standard APIs); the LLM weights need not be exposed. We stress some further important points regarding this.
Choice of validation prompt set. The success of UniRoute critically requires that the validation prompts Saa reasonably reflect those encountered during deployment. In our experiments, we largely take Saa be random subset of the training set Sy since the core assumption of training router on Sy is that these prompts reasonably reflect test prompts, a random subset can also be assumed to mimic test prompts. More generally, the validation prompts could be hand curated based domain knowledge, or drawn from a standard benchmark suite (which is often available for any new LLM).
Comparison to router retraining. The assumption that each new L.M can be evaluated on Stal raises a natural question: why not simply retrain a conventional (static) router on S This approach has two drawbacks. First, such retraining susceptible to overfitting: this is because Syol assumed to be of modest size (e.g.. (10"),, such that performing inference for a new LLM on Sya is not pro- hibitive. Consequently, UniRoute can thus yield better quality (as we shall demonstrate empirically). Second, router retraining and redeployment imposes additional overhead (e.g.. engineering effort) compared to Uni Route. Further to these, UniRoute offers a more flexible approach that can handle bespoke routing use-cases, such as routing amongst dynamic model subsets; see Appendix H for more discussion.
Comparison to K-NN. K-NN routing (Hu et al., 2024b) which, noted in $4. is special case of UniRoute does support new LLMs without retraining. Indeed, one may readily compute JKNN in (4) based solely the prediction eтror vector. However, with naïve retraining, this approach may not generalise favourably as Ska is assumed to be of modest size; indeed, even in moderate data regimes, Zhuang al. (2024) observed that K-NN may underperform compared to their proposed Matrix Factorization approach ("MatFac" in our experiments), Further, it does not exploit any information from the (potentially large) training set Sq. Nonetheless. K-NN offers simple means of making non non-linear routing decisions. We now study alternate means of leveraging non-linearity in L
5 UniRoute WITH CLUSTER-BASED LLM FEATURE REPRESENTATIONS
Building on the above, we now consider certain cluster-based LLM representations.
5.1 REPRESENTING AN LLM VIA PER-CLUSTER PREDICTION ERRORS
We propose an instantiation of (10) that represents any LLM h Hall through its average errors I "clust (h) E [0, 1 on K >> 1 pre-defined clusters. Similarly, we represent prompts via their cluster membership elaa (a=) E (0, K This yields the following instantiation of (9):
Velust (xx h) = 4 Pelust (x) L cllus(hh). (h).
(11)
Intuitively, ( (æ. h)) estimates the performance of given LLM prompt by examining its performance on similar prompts, i.e., those belonging to the same cluster.
Naively, one may directly cluster Syal: however, this is prone overfitting, since (by assumption) the set is of modest size. Thus, we instead cluster the prompts in the training set Sg. We then use this to group the validation prompts into K disjoint clusters, and compute per-cluster errors for a new LM using Sea- Concretely, given a text embedder yp: X + R Dr, we compute Tb &elus (x), L eluss (h) via:
(i) Cluster the training set embeddings (t)() N,2, to construct K non-overlapping clusters. This yields a cluster assignment map & X + {0, 1)xK, where the k-th index is when æ belongs to cluster i (i.e.. it on average closest samples cluster k). This step does not require labels.
iii Assign each validation set prompt to a cluster. Let C2 = {(x, (xx y) Sval, & eelust, (xx)
1} be the subset of the validation set that belongs to cluster k.
(iii For any LLM h Hal. compute & clust( (hlus((h) [0, 1] using its per-cluster validation errors:
V clust, (h) = |Crl
Σ
1 + h(z)].
(12)

---

Published as a conference paper at ICLR 2026
Plugging these into (11), we may now approximate the expected loss for lhe input prompt using the average error of the LLM on the cluster the prompt is assigned to, and route via (8).
This cluster-based instantiation of UniRoute given any new test LLM, we simply need to estimate its average per-cluster clusters for all validation prompts. This does not require any expensive gradient updates, and one-off cost: further routing can operate entirely on this vector, and is oblivious to any further changes in the LLM pool.
route with new LLMss in highly efficient manner:
natural choice of clustering algorithm in step (ii) is K- --mans (MacQueen, 1967). For router devolves to the ZeroRouter from (Hu et al.. 2024b) (see Appendix D). Empirically, we find UniRoute to reasonably robust the choice of K (Appendix G.2).
the
5.2 FROM FIXED TO LEARNED CLUSTER ASSIGNMENT MAPS
To improve the above cluster-based router, may further seek to exploit the labels in Sw given the same clustering, we propose to learn a cluster assignment map Polust(: 0) E [0 that can better map prompt to a distribution over clusters. Specifically, we set & Pelust, (æ;; 0) o exp (0) <x(x) for parameters E IR K De and text embedding sp Analogous to (11), we have
Telust (x, h; 0) 4 clust (x; 0) V clust (h),
where, as in (12), V cluust (h) denotes the per-cluster errors estimated from the validation set S To pick 0, we minimize the log loss on Spr against the correctness labels for the training LMs Hg;
Σ Σ
1 1 h(z)]
log Yelust (x, h) + yy = h(x)] log (1 Yolust (x. h: 0))
(s,))sS,, NE7G,
5.3 EXCESS RISK BOUND
We now present excess risk bound for our cusserrbbssed routing strategy. Suppose represent the underlying data distribution over (æ, y) by mixture of latent components: P(x, y)
E T P(x, y component and Teg P(2 k). For a fixed k. we may denote the probability of incorrect predictions seeks mimic the following rule:
k).. where z denotes latent random variable that identifies the mixture
for h H conditioned on = & by (h) = = (h(x) & y) Then, cluster-based routing
F* (x. Hee) argmin nE(N)
Σ kE[K]
P(2 k|/x) (z(h(()) + X c(b(e))
(13)
Proposition 2. Let r* be per (7). For any Hee (m(0)) (
HH, h( Ha Let Roi (r,He)) =
and x E X, let: Aula, (m))) = Pp u/a,==k (n) (x) + y
Σn P Mi (x)) + yyyrr H(o) = n denote the 0-1 risk. Then under a regularity condition on TP,
Exg. [Ro
- Egg. Rom (r" [Heo) S Excaa
max 1 kE]
Ae( ((m))
This suggests that the quality gap between cluster-based routing in (13) and the optimal rule in (7) is bounded by the discrepancy between the per-cluster and per-prompt errors: i.e., the gap between the LLM's error on a prompt versus the average constituent cluster error (see Appendix C.3 for proof).
5.4 DISCUSSION: UniRoute VERSUS EXISTING WORK
UniRoute allows a trained router to make of new test-time LLMs without retraining. This is unlike most, though not all existing approaches; indeed, recent works Feng et al. (2024); Zhao et al. (2024b); Li (2025) have also proposed dynamic routing approaches. In comparison, approach that avoids dependence on exogenous task label information for prompts, and relies established statistical learning primitives.
provide
7

---

Published as a conference paper at ICLR 2026
More precisely, Li (2025) also proposes the use of certain LLM embedding vector, and is thus the most related work. However, there are important distinctions to UniRoute. First, Li (2025) rely on RL-based policy gradient optimisation; the significant recent advances in RL notwithstanding, this raises the risk of training instability, and computational burden. By contrast, we consider standard statistical learning (akin most recent routing works, e.g., Somerstep al. (2025)), and rely on standard gradient descent. Second, the LLM embeddings in Li (2025) depend on other LLMs in the pool, as well as the order in which the LLMs are added. By contrast, our embeddings are pool agnostic, and highly intuitive (per-cluster errors). Third, to construct the LLM embeddings. Li (2025) perform an iterative optimisation of variational objective, which further relies on estimation of prompt difficulty. By contrast, our correctness- vector based representation requires a simple calculation of per-cluster errors. We defer a full discussion to Appendix
EXPERIMENTAL RESULTS
We demonstrate the effectiveness of UniRoute when observing new LLMs at test time.
Experimental setup. We present results on EmbedLLM (Zhuang et al., 2024), RouterBench (Hu et al., 2024b), a Math+Code dataset from Dekoninek al. (2025) (containing subset of Minerva Math and LiveCodeBench (Lewkowyez et al., 2022; Jain et al., 2024)), SPROUT o3-mini (Somerstep et al., 2025), Headlines (Sinha & Khandait, 2021; Chen et al., 2023b), OpenLLMv2 (Fourrier et al., 2024), and Chatbot Arena (Zheng et al., 2023). All datasets use binary accuracy as the evaluation metric. Thus, all methods rely on the deferral rule described in (7).
LLM pool construction. We partition the set of LLMs into two disjoint sets: training models (HGr in $5) and testing models (. For EmbedLLM (112 LLMs) and SPROUT o3-mini (15 LLMs), use random subset of 2/3 for training and 1/3 for testing. For Headlines, we hold out six LLMs for training and use the other six LLMs for testing (details in Appendix G.3). For RouterBench (11 LLMs in total), details on hyper-parameter selection.
random 50% for training and the rest for testing. See Appendix G.2 for
Example construction. Over 400 independent trials, for EmbedLLM, we randomly split examples into 60%/10%/30% for training, validation, and testing (for SPROUT o3-mini, Headlines, and RouterBench, we use 400 examples for validation). The training portion for training router, and only has correctness labels of training models. The validation split is the small dataset used to represent each test-time LLM as a feature vector (see $5.1). 1). All baselines are evaluated on the test examples and only on the test LLMss We use Gecko 1B (Lee et al., 2024b) (frozen) to produce a 768-dimensional prompt embedding for all methods.
Evaluation. We evaluate each method with a deferral curve (Appendix F.2; (Jitkrittum et al., 2023; Wang et al., 2024a; Hu et al., 2024b)), which plots the trade-off of average quality against cost. The trade-off is realized by varying As in (6). We report two evaluation metrics: (i) the quality-neutral cost (QNC), or the minimum relative cost nceded to match the quality of the most accurate test LLM; and (ii) the area under the deferral curve (Area). The QNC is analogous to the call- call-performance threshold in Ong et al. (2025). For EmbedL.LM, we use the number of parameters of the LLM as a proxy for the cost of processing one prompt. For SPROUT 03-mini, HeadLines, and RouterBench, we use each LLMs' API calling costs (in USD) as provided in the datasets.
We evaluate Uni e cluster-based routing using both the unsupervised and supervised cluster assignment maps 165.1, 1, $5.2) - against certain dynamic & static routing baselines, described next.
Baselines:
dynamic routing. There are simple, but surprisingly strong dynamic routing base-
lines:
ZeroRouter (Hu et al., 2024b). A router that randomizes between two LLMs, chosen from those attaining a certain Pareto-optimal performance on the validation sample (details in Appendix D).
K-NN (Hu et al., 2024b; Shnitzer et al., 2023). A special case of UniRoute e (see $4.1) 1) that for each prompt, looks up the K nearest prompts in the validation set in the space of prompt embeddings, computes y for each test LLM using (4) (with the 0-1 loss), and routes via (2).
LLMBandit (Li, 2025) is also a natural baseline per $5.4. Unfortunately, an exact replication of Li (2025) is challenging, owing to the lack of public implementation, coupled with their
8

---

Published as a conference paper at ICLR 2026
Dataset
EmbodLLM
SPROUT o3-mini
Headlines
RouterBench
Method
QNC
Area
QNC | Aren
QNC |
Area
QNC
Area
ZeroRouter (Hu et al., 2024b)
87. 87.5%
.607
100.0%
820
88.0%
819
99.9%
.89
K-NN (Hu et al., 2024b; Shnitzer et al., 2023)
45.9%
.636
29.69
.84"
43.7%
.80
99.7%
.077
Retrained MLP (Hu
2024b)
80.9%
829
74.2%
99.2%%
.710"
Retrained MatFac (Ong et al., 2025; Zhuung et al., 2024)
36.6%
.640
84.2%
.825
80.9%
.821
99.3%
.788
UniRoute (K-Means)
33.7%
649
19.6%
850
56.9%
828
99.4%
712
UniRoute (LearodMapp)
33.1%
.652
23.4%
.846
34.9%
832
99.6%
.711
EmbeiiLU
SPROUT c8
Meadlincs
BoukerBunch
0800
0315
3ocl-
040-
M855
03r
0625
bettuired MLP
Number
290 250 200 2i0 cie sin
Number valisation samples
299
Number samples
250
19 ahe di Number divviiaation samples
Figure 2: Top: We report the Quality-Neutral Cost (QNC), i.e., the minimum relative cost to achieve the same performance as the most accurate LLM, and the area under the deferral curve. indicates the method is statistically significantly worse than UniRoute (LearnedMap) at significance level = 0.01. Bottom: Areas under the deferral curve (1) with 96% CI on unseen test LLMs for varying validation sample sizes. UniRoute (K- (K-mean)) is often better than the baselines across sample sizes. The re- reairri MLP underperforms on two datasets due to overfitting to the smaller validation samples.
approach involving many non-trivial components (e.g., policy gradient training with a replay buffer). Nonetheless, Appendix I.1 presents (favourable) comparison against the paper's quoted results.
static routing. We reiterate that most existing routers in the literature are inherently tied to the LLMs they are trained with: thus, such static routers are impractical the dynamic rout- ing setting. This is true of the multi-layer perceptron (MLP) (Hu et al., 2024b), matrix factorization (Ong et al., 2025: Zhuang et al., 2024) and BERT-based routers (Ong et al., 2025; Ding et al., 2024; Chen et al., 2024b; Mei et al., 2025), all of which have fixed number of outputs (one per LLMM). Despite this, we include two representative methods for completeness:
Baselines:
Retrained MLP & MatrixFac: An MLP router (Hu et al., 2024b) and factorised router (Ong et al., 2025; Zhuang et al., 2024), with one output per test LLM, trained on prompt embeddings to estimate y. For each new LLM, these routers need to be retrained with additional output using the small labeled validation sample available for the new LLM. These basclines are impractical as they prone to overfitting to the small validation sample, and incur frequent retraining costs.
We excluded static routers with more complex architectures (c.g., Ding et al. (2024)) because they even more susceptible to overfitting and less practical to retrain frequently.
Experimental results. We present deferral curves on EmbedLLM in Figure 3, and on other datasets in Appendix G.3. Each isolated x represents the cost and average test accuracy of one test LLM. The table in Figure 2 summarizes Area and QNC for four datasets; additional results in Appendix G.3.
UniRoute is effective under dynamic LLM pools. UniRoute yields competitive quality-cost trade-offs. with the gains over baselines being statistically sig- nificant. In particular, on EmbedLLM - featuring 30 unseen LLMs we provide compelling gains over K-NN. Further, on all datasets, we consistently outperform ZeroRouter, which was noted as a strong baseline in Hu et al. (2024b). We are also signifi- cantly better than the re-trained static routers on most datascts; as expected, these basclines underperform as a result of overfitting to the small validation set. Appendix G.4 further shows UniRoute is effective when the LLM representations are constructed from Chatbot Arena, and evaluated nut-of-distributioon EmbedLLM prompts.
0.70
Accuracy
0.60
K-NN
Retreined MLP Retrained MatFac
0.55
UnviRoute (K-Maans]
UniRoute (LeamedMap)
ZeroRouter
20
60
Cost budget (model size)
Figure 3: Deferral curves EmbedLLM.
In Appendix I.1, show that UniRoute performs favourably compared to results from Li (2025).

---

Published as a conference paper at ICLR 2026
UniRoute is effective even with small validation sample. We show in Figure (bottom), that proposal is often significantly better than K-NN across a range of validation sample sizes. One potential reason for this is the requirement in K-NN that only the retrieved neighbors from the validation set can be used to estimate test models' performance. It hence unable to exploit the training in any way. In contrast, our methods are able exploit the training data either in unsupervised (K-ni (K-mman)) or supervised (LearneMMap) manner.
UniRoute robust to choice of clusters K. Appendix G shows that in general, UniRoute is effective for several different K values, and our hyperparameter selector picks an effective K.
UniRoute maintains effectiveness under static LLM pools. While the dynamic pool setting the focal point of our work, we show in Appendix G.5 that even in the static LLM pool setting, UniRoute typically performs comparable to most basclines.
UniRoute's 's LLM representations capture similar LLMs. Appendix G.6 shows that similar LLMs are close in the space of bussed LLM embeddings of UniRoute (Section 5).
LIMITATIONS AND FUTURE WORK
While UniRoute makes progress the problem of dynamic routing, there some limitations which present opportunities for future research. First, in the fully static LLM pool setting, UniRoute is not guaranteed to recover the performance of existing methods; designing hybrid static-dynamic routers that leverage the best of both paradigms would be of interest. Second, while we proposed certain natural cluster-based routers, there could be other instances of UniRoute; a more systematic study of this design space would be worthwhile. This may be particularly fruitful in settings without a clean cluster structure, wherein UniRoute (K-Means) is seen to underperform UniRoute (LearnedMap), Finally, while the choice of a random subset of training prompts for Seat is intuitive, further optimizing for diverse subset (i.e., finding coreset) can potentially give an even better representation; this complementary our current proposal, and is an interesting topic for future study.
REPRODUCIBILITY STATEMENT
We have made efforts to ensure reproducibility of our work. Our experiments on dynamic routing rely training, validation, and test data splits. Each data split may contain training LLM, test LLMs, or both. These details are explained in Appendix F.1. The model architecture of our dynamic router is detailed Appendix F.3. Tuning of spper-parameterss and parameter choices are clearly described in Appendix G.I, and Appendix G.2, respectively. All datasets used are publicly available and properly attributed. Data preprocessing steps are in Section and Appendix G.3. For our theoretical results, the complete proofs are given in Appendix C, along with several auxiliary lemmas.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6429-6438, 2019, doi: 10. N0.1109//CCV22210006653.
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthiu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In The Twelfth International Conference on Learning Representations, 2024.
URL https:://peereviieeeee wwweeeewwwweeeeeeeeeeeeeeeeeee wtwp://wwwww..eee/eooooooaaaaaaa
Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei
Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui, and Mausam Automix: Automatically mixing language models. In The Tiihty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:
//openreview. www. mret/forummice www.mii:ewwwwwwww...
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark,
10

---

Published as a conference paper at ICLR 2026
Laurent El Shafey, Yanping Huang, Kathy Helee--ellsteem,, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hermandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette- Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng. Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittyceeiiah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Mísra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni. Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang. Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM 2 technical report, 2023.
Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 959); 2008. URL http://jmtr.org popp//pprs///////// v9/bartlett html.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for fast test-time prediction. In International Conference on Machine Learning, 2017.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4) 1952.
Matthew Brand. Incremental singular value decomposition uncertain data with missing values. In Anders Heyden, Gunnar Sparr, Mads Nielsen, and Peter Johansen (eds.), Computer Vision ECCV 2002, pp. 707-720, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg. ISBN 978-3-540- 47969-7.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Ruisi Cai,, Sauray Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Flextron: Many-in-one flexible large language model. In International Conference on Machine Learning (ICML), July 2024a.
Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. MEDUSA: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024b.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QulP: 2-bit quantization of large language models with guarantees. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Boyuan Chen, Mingzhi Zhu, Brendan
Muhammad Shafique, and Siddharth Garg. Model cascading for code: Reducing inference costs with model cascading for LLM based code generation, 2024a. URL https: httpp/sss///wwww///sssss/em 15842.
11

---

Published as a conference paper at ICLR 2026
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, fean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318. 2023a.
Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305, 055677 2023b.
Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang, RouterDC: Query-based router by dual contrastive learning for assembling large language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024b.
François Chollet et al. Keras. https://keras: io, 2015.
C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1):41-46, 1970.
Yu-Neng Chuang, Prathusha Kameswara Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, Xia Hu, and Helen Zhou. Learning to route LLMs with confidence tokens. In Forty- International Conference on Machine Learning. 2025. URL
Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. ALT, 2016. URL pdd.
https://cs.. http://cccuuu... nyu. www.eeduuooooreeeeeeeee
Jordan Cotler, Kai Sheng Tai, Felipe Hernández, Blake Elias, and David Sussillo. Analyzing populations of neural networks via dynamical model embedding, 2023. URL https://arxiv..
org/abs/2302.1400 14078
Christoph Dann, Yishay Mansour, Teodor Vanislavov Marinov, and Mehryar Mohri. Domain adaptation for robust model routing. In Adaptive Foundation Models: Evolving for Personalized and Efficient Learning, 2024. URL https://openrevieww ne t htt.//oooddooeeeooeeeeeee
DeepSeek-Ar AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin. Fuli Luo. Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang. Haowei Zhang, Honghui Ding. Huajian Xin, Huazuo Gao, Hui Li. Hui Qu, L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang. Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Rumxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nic, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xic, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng. Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseck-v2: A strong, economical, and efficient mixture-of-expert language model, 2024. URL wtw.tttsss///.......tt wwwp://////aa.a....o.ooo.. /2005 64434.
Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for LLMs, 2025. URL https://arxiw.or aat/////////// 10347.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June
12

---

Published as a conference paper at ICLR 2026
2019. Association for Computational Linguistics. doi: 10.18653/v1/N19 1423. URL https:
//aclanthology. www.oooooyooogg//ggg wogg/19-- 1423..
Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati. Tim Dettmers, Kaifeng Chen, Inderjit S Dhillon, Yulia Tsvetkov, Hanmaneh Hajishirzi, Sham M. Kakade, Ali Farhadi, and Prateek Jain. MatFormer: Nested transformer for elastic inference. In The Tiirth- Annual Conference Neural Information Processing Systems, 2024. URL https::/openrevieeeee http://pprreiiieeeeeefoorrrr//// wwtt:oorummiid--faaeelll..
Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks S. Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevien www.neettooooeeee formmmdd=02ff
Arthur Douillard, Qixuan Feng, Andrei A. Rusu, Adhiguna Kuncoro, Yani Donchev, Rachita Chha- paria, lonel Gog, Mare' Aurelio Ranzato, Jiajun Shen, and Arthur Szlam. DiPaCo: Distributed path composition, 2024. URL htttps://arxiv.org///
10616.
Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed Aly, Beidi Chen, and Carole-Jean Wu. Layerskip: Enabling early exit inference and self-speculative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 12622-12642. Association for Computational Linguistics, 2024. doi:
10. 3:653//1/2022.acl-loog
URL //tt:://dx..e www.ddddoi.orro.... orgoi.rgg///a 10. (13863/v1/222/0 Lon-oog............. 681.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/188/ 1082. URL httpp:://actaatthlooeeee wwwphhhwwwooooooo//oooeeoee 1082.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2022. URL https://arxiv. orgg //////2111..9111/ 03961.
Tao Feng, Yanzhen Shen, and Jiaxuan You. GraphRouter: A graph-based router for Ilm selections, 2024. URL httpss://aaaiii. wwwp:wwwwwww.rrarooooe.ee org/ ha////bb////00/ 2410. 03834.
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects neural language generation. In Proceedings of the Workshop Stylistic Variation, pp. 94-104, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 1/177499/1/W1---912 URL https:
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open IIm leaderboard v2. https://huggingface..o/// 11m leadebboard,, 2024.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10323-10337. PMLR, 23-29 Jul 2023. URL
mlr ppwss//ee2// http:///2222///aea//e/// http///aaaarraaaaaaaa/
Yonatan Geifman and Ran Elniv. SelectiveNet: A deep neural network with integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2151-2159, PMLR, 09-15 Jun 2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle
13

---

Published as a conference paper at ICLR 2026
Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olamo, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee. Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafilld Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Contala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang. Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Pumit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Railcanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou. Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery. Elconora Presani, Emily Hahn. Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers. Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpemn Grant Herman, Grigory Sizov, Guangyi. Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Aina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li. Kiran Jagadeesh, Kun Huang. Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish
14

---

Published as a conference paper at ICLR 2026
Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vattkkvv, Mikayel Samvelyan, Mike Clark. Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng. Sungmin Cho, Sunny Virk, Suraj Subramanian. Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar. Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama herd of models, 2024. URL https: / /awpiv.vvvvvaaa org/ 2bS/2407.2178.
Neel Guha, Mayee F Chen, Trevor Chow, Ishan S. Khare, and Christopher Re. Smoothie: Label free language model routing. In The Tiihtt-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL
net/ fo
Neel Guha, Mayee F Chen, Trevor Chow, Ishan S. Khare, and Christopher Re. Smoothie: Label free language model routing. In The eiirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https:///openevveeeeeee//eee/eee
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024. URL httpps://opeeeeeiieee
net/forum? Kgag
Surya Narayanan Hari and Matt Thomson. Tryage: Real-time, intelligent routing of user prompts to large language models, 2023. URL https://arxiv www./or////////ee/eoeeeeeeee//e/ abb////b./.......//// 11601.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Mat- sushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are GPT models at machine translation? comprehensive evaluation, 2023. URL 09210.
Ari Holtzman, Jan Buys Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https: //tpp//wwwww.nee///eeeeeeeeeeeee totee//orrrmmii-ddrrrrrrrerr
Jinwu Hu, Yufeng Wang, Shuhai Zhang, Kai Zhou, Guohao Chen, Yu Hu, Bin Xiao, and Mingkui Tan. Dynamic ensemble reasoning for 11m experts, 2024a. URL https://arxiv http//axxxv...////b/// wrgg/bs/////aooeeo..soeeeers //t.///4...//// .7448..
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. RouterBench: benchmark for multi-LLM routing system. In Agentic Markets Workshop at ICML 2024, 2024b. URL htt.ntss///oeeeeiiew.eee forum?id=IVXvvBUxx IVXVVUUxwh
Zhongzhan Huang, Guoming Ling, Vincent S. Liang, Yupei Lin, Yandong Chen, Shanshan Zhong, Hefeng Wu, and Liang Lin. Routereval: A comprehensive benchmark for routing Ilms to explore model-level scaling up in LLMs, 2025. URL htttpp://arxxvvorr//tttt 19657.
15

---

Published as a conference paper at ICLR 2026
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79 87, 1991. doi: 10. htt.1///ecco.199.......
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint urXiv:2403.07977 2024.
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. Jorg. 2023.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-BBlnder: Ensembling large language models with pairwise ranking and generative fusion. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 14165-14178, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: org/2023 acn- 792/.
URL https:haaclaattthll.s.
Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiy Kumar. When does confidence- basen cascade deferral suffice? In Thirth- Conference on Neural Information Processing Systems, 2023. URL https://openreeiiew. nett/ mimmmidddk2ZZzSPUU
M.I. Jordan and R.A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. In Proceedings of 1993 International Conference on Neural Networks AJCNN-93-Nagoyy. Japan), volume 2, pp. 1339-1344 vol.2, 1993. doi: 10.1109/JCCN. 1993.7767999
Anil Kag, Igor Fedorov, Aditya Gangrade, Paul Whatmough, and Venkatesh Saligrama. Efficient edge inference by selective query. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https::/openreette hwtpw/wwwwwweeeeeeeeeeeeeeeeeeee httet//ooommiddddeeeeeeeee
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models, 2025. URL https://arxiv.org/ www.or//g/b///oeeeeeeeeee 17428.
Lee, Hao Cheng, and Mari Ostendorf. OrehestraLLMM Efficient orchestration of lan- guage models for dialogue state tracking. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1434 1434-1445, 1445, Mexico City, Mexico, June 2024a. Association for Computational Linguistics. doi: http://///22...aaaaa/a.mm
URL httts:::/aclattthhoot
org/ 2etg/2224.oo0 naacl- long. 79/.
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi Luan, Sai Meher Karthik Duddu, Gustavo Hernandez Abrego, Weigiang Shi, Nithi Gupta, Aditya Kusupati, Prateck Jain, Siddhartha Reddy Jonnalagadda, Ming-Wei Wei Chang, and Iftekhar Naim. Gecko: Versatile text embeddings distilled from large language models, 2024b. URL htt.a//...ttttp///aaaa.o.oo//// httpaa/www...rr////////......../
20327
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023.
Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer. Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. Llm inference serving: Survey of recent advances and opportunities, 2024a. URL httpt:////aate wwwpeiiiiioorraeee.ee www.oog/////..e absp////.//../.../// 12391.
Tianle Li, Wei-Lin Chiang, and Lisa Dunlap. Introducing hard prompts category in Chatbot Arena. https://lmsys. wwgg/blo//aae 2224/2224-022 17-category hard, 2024b.
16

---

Published as a conference paper at ICLR 2026
Tianle Li. Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-han hard and benchbuilder pipeline, 2024c. URL https://arxiv,
Yang Li. LLM Bandit: Cost-efficient Ilm generation via preference-coonitionee dynamic routing, 2025. URL http://aaxiv..orra///////////
Yuhui Li. Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: speculative sampling requires rethinking feature uncertainty. Proceedings of the 41st International Conference on Machine Learning, ICML'22. JoLR.org. 2024d.
Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2:
Faster inference of language models with dynamic draft trees. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. --432, Miami, Florida, USA, November 2024e. Association for Computational Linguistics. doi: 2tt55///1112224..mmmmmaaaa 422/.
URL
emn mamin..
Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing the expert: Efficient Soward-guide ensemble of large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1964-1974, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: https: http://ooocaatoooooooyoooo org/ 2024./2224..22 www.l-lacolongggo long. 169/.
URL
J. MacQueen. Some methods for classification and analysis of multivariate observations. Proc. 5th Berkeley Symp. Math. Stat. Probab., Univ. Calif. 1965/66, 1, 281-297 (1967)., 1967.
David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NeurIPS' 18, pp. 6150-6166 Red Hook, NY, USA, 2018. Curran Associates Inc.
Anqi Mao, Christopher Mohri, Mehryar Mohri, and Yutao Zhong. Two-stage learning to defer with multiple experts. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024a. Curran Associates Inc.
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Principled approaches for learning to defer with multiple experts. In Reneta P. Barneva, Valentin F. Brimkov, Claudio Gentile, and Aldo Pacchiano (eds.). Artificial Intelligence and Image Analysis, pp. 107-135, Cham, 2024b. Springer Nature Switzerland. ISBN 978 778-3-031-63333-33
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Realizable hi-consistent and Bayes-consistent loss functions for learning to defer. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024c. URL https::t////ttt
Anqi Mao, Mehryar Mohri, and Yutao Zhong. Regression with multi-expert deferral. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 34738-34759, PMLR, 21-27 Jul 2024d, URL htattp::/pproceeinngg .ng..g.................m....mm.. .mlr. wtt.pp/pee///aaasssse htmll.
Kai Mei, Wujiang Xu. Shuhang Lin, and Yongfeng Zhang. Omnirouter: Budget and performance controllable multi-llm routing. arXiv preprini arXiv:2502. 2025.
Alireza Mohammadshahi, Arshad Rafiq Shaikh, and Majid Yazdani. Routoo: Learning to route to large language models effectively, 2024.
Yannis Montreuil, Axel Carlier, Lai Xing Ng. and Wei Tsang Ooi. Adversarial robustness in two-stage Gaaming-to-defée Algorithms and guarantees, 2025. URL httts:://arkee httppss//aaxivv.or//ab///2// 01027.
17

---

Published as a conference paper at ICLR 2026
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to expert. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 7076-7087. PMLR, 13-18 Jul 2020.
Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Post-hoc estimators for learning to defer to an expert. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
net/ www.nntt/ofoooe htfffoormmi// id= _jg65f6tuF7.
Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, Neha Gupta, and Sanjiv Kumar. Learning to reject for balanced error and beyond. In The Twelfth International Conference on Learning Representations, 2024a. URL https::/openreeiew.. net/ wtt.wwwwwwneeooummineeeeeeeeeeee
Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, and Sanjiv Kumar. Plugin estimators for selective classification with out-of-distributtin detection. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview/ wttp://wwwweeeeetnneeiieee DASh78r37g.
Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Seungyeon Kim, Neha Gupta, Aditya Krishna Menon, and Sanjiv Kumar. Faster cascades via speculative decoding. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview. hwtpwwiwwww..eeeeeeeeeeeeeeeeeee met//orumiiddoog 220wsmd.
Jerzy Neyman and Egon Sharpe Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231(694-706):88 -37, 1933.
Jianmo Ni, Gustavo Hernandez. Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yin- fei Yang. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Associa- tion for Computational Linguistics: ACL 2022, pp. 1864-1874, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 00.18653/112222.fnndgg 146. URL https://ac http:///aclaathooo..... www.hoooo..orggooee... 2022. htw.://..rrrroo....nn.a------aae findings-acl....
Lunyiu Nie, Zhimin Ding, Erdong Hu, Christopher Jermaine, and Swarat Chaudhuri. Online cascade learning for efficient inference over streams. In Proceedings of the 41st International Conference on Machine Learning, ICML'22 24.. JMLR.org. 2024.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and lon Stoica. RouteLL.M Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025.
OpenAl. New embedding models and API updates.
new- www---ebeddnngmmmeeee models -ndeees-aaad----ppdttee-s.s 2025.
Mark Palatucci, Dean Pomerleau, Geoffrey Hinton, and Tom M. Mitchell. Zero-shot learning with semantic output codes. In Proceedings of the 23rd International Conference on Neural Information Processing Systems, NIPS'09, pp. 1410-1418, Red Hook, NY, USA, 2009. Curran Associates Inc. ISBN 9781615679119.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer infer- ence. In MLSys, 2023. URL httpsttss///ooreeddng mlsys. org/ http:/per..pppppppp/// wtt.s///appe///////eeeep html.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving lan- guage understanding by generative pre-training https://cdn. ittps://cdnoooeai..... com/researcch-ooerss... wttp-uuuuuuussseeeaauuuuaaee wwwpuunerstadnnn__paapppp pdf, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners, 2019.
18

---

Published as a conference paper at ICLR 2026
Mathieu Ravaut, Shafiq Joty, and Nancy F Chen. Summareranker: A multi-task mixture-of--xxpeee re-ranking framework for abstractive summarization. arXiv preprint arXiv:2203.06566 2022.
Ankit Singh Rawat, Manzil Zaheer, Aditya Krishna Menon, Amr Ahmed, and Sanjiv Kumar. When in doubt, summon the titans: Efficient inference with large models. arXiv preprint arXiv:21 10.10 2021.
Ankit Singh Rawat, Veeranjaneyulu Sadhanala, Afshin Rostamizadeh, Ayan Chakrabarti, Wittawat Jitkrittum, Vladimir Feinberg, Seungyeon Kim, Hrayr Harutyunyan, Nikunj Saunshi, Zachary Nado, Rakesh Shivanna, Sashank Reddi, Aditya Krishna Menon, Rohan Anil, and Sanjiv Kumar. A little help goes long way: Efficient LLM training by leveraging small LMs, 2024. URL
https://aran /ttp:///axvvvv/. org/ abs/2410.18779
Aishwarya P S. Pranav Ajit Nair, Yashas Samaga, Toby Boyd, Sanjiv Kumar, Prateek Jain, and Pranceth Netrapalli. Tandem Transformers for inference efficient LLMs, 2024. URL https:
//arxiv.org/ www.orrgg.bb...aea/eete
Sara Sangalli, Ertune Erdil, and Ender Konukoglu. Expert load matters: operating networks at high accuracy and low manual effort. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Simone Scardapane, Michele Scarpiniti, Enzo Baccarelli, and Aurelio Uncini. Why should we add early exits to neural networks? Cognitive Computation, 12:954-966 2020.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:://pennrevieww.eeeee wwwp://wwwweeeeeeeeeeeeeeaeeeeee wttp::/eeeeooormniduuuaaaaaaaoo idcLLLC4L3CBAA
Avital Shafran, Roei Schuster, Thomas Ristenpart, and Vitaly Shmatikov. Rerouting LLM routers, 2025. URL htttttss//aaxii.orrgaaaaaaa/e 01818
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets, 2023. URL 15789. Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 22045-22055. Curran Asso- ciates, Inc., 2020. URL htttps:/ httppp/reeeennnnnunrrsre..ro.mme meurips.cccpaper (92697869f$64844400 wtw.a
petps://arxiin
file//
Paper.pdf.
Ankur Sinha and Tanmay Khandait. Impact of news on the commodity market: Dataset and results. In Kohei Arai (ed.), Advances in Information and Communication, pp. 589-601, Cham, 2021. Springer International Publishing. ISBN 978-3-030-73103-8.
Seamus Somerstep, Felipe Maia Polo, Allysson Flavio Melo de Oliveira, Prattyush Mangal, Mírian Silva, Onkar Bhardwaj, Mikhail Yurochkin, and Subha Maity. Carrot: A cost aware rate optimal router, 2025. URL https: http:::/wwaxx..///////// org. haigg//bs/25500 03261.
Kv Aditya Srivatsa, Kaushal Maurya, and Ekaterina Kochmar. Harnessing the power of multiple minds: Lessons learned from LLM routing. In Shabnam Tafreshi, Arjun Akula, João Sedoc, Aleksandr Drozd, Anna Rogers, and Anna Rumshisky (eds.), Proceedings of the Fifth Workshop on Insights from Negative Results in NLP,, pp. 124-134, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. CoRR, abs/18 181 1.01155, 2018. URL http://arxiv, wwwpxiivoorraaaeeeee aag//bs// 111111111111,...1111111111111111
Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. TensorOpera router: A multi-moded router for efficient LLM inference, 2024. URL https:t/aaxxii......// www.:/or/////o///eoeeeee 240...////....00 12320.
19

---

Published as a conference paper at ICLR 2026
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
Swabha Swayamdipta, Ankur P. Parikh, and Tom Kwiatkowski. Multi-mention learning for reading comprehension with neural cascades. International Conference on Learning Representations, 2018. URL
ie--iddne-yyeeeeeeee RW.
Dharmesh Tailor, Aditya Patra, Rajeev Verma, Putra Manggala, and Eric Nalisnick. Learning to Defer to Population: A Meta-Leaming Approach. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, 2024.
Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. BranchyNet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016, Cancún, Mexico, December 4-8, 2016, pp. 2464-2469. IEEE, 2016.
Tristan Thrush, Christopher Potts, and Tatsunori Hashimoto. Improving pretraining data using perplexity correlations, 2024. URL https://arxii org/ abtp///b//////.////////// 05816.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023.
Vivien An optimal lossy variant of speculative decoding, 2023. URL https://github. com/viiveenooo/mentooo http://eeeeeeeedddedne......... Unsupervised Thoughts (Blog). URL: https://github. hwap///iiieeneee/eetttrrddddddoo decoding.
Kirill Trapeznikov and Venkatesh Saligrama. Supervised sequential classification under budget constraints. In Carlos M. Carvalho and Pradeep Ravikumar (eds.), Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31 of Proceedings of Machine Learning Research, pp. 581-589, Scottsdale, Arizona, USA, 29 Apr-01 May 2013.
PMLR.
Neeraj Varshney and Chitta Baral. Model cascading: Towards jointly improving efficiency and accuracy of NLP systems. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11007-11021, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
P. Viola and M. Jones. Rapid object detection using boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pp. I-1, 2001. doi: 10.1109/VVP..2201.99955
Marija Šakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? Cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24, pp. 606-615, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400703713.
doi: 10.1145/3616855.66355
URL
https://doi.org
3145/3616855... 3635825
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Survey Certification.
https://openreviie.. wtwp:/iwwwweeeeooondiddoooooc
Congchao Wang, Sean Augenstein, Keith Rush, Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Aditya Krishna Menon, and Alec Go. Cascade-aware training of language models, 2024a. URL https::/arxiv.or wttp://..rrraabss.....oooo. 00060.
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models, 2024b. URL https: httpps:://///iii///eee http://ariiv...r///////////// e8368.
20

---

Published as a conference paper at ICLR 2026
Xiaofang Wang. Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Movshovitz-Atiaa, and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models. In International Conference on Learning Representations, 2022. URL httpss//openteiiew.ne/// forum? ? iid -v02ttoobbb.-
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. IDK cascades: Fast deep learning by learning not to overthink. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty- Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 580-590. AUAI Press, 2018.
Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. Tabi: An efficient multi-level inference system for large language models. In Proceedings of the Eighteenth European Conference
on Computer Systems, EuroSys '23. pp. 233-248. New York. NY. USA, 2023, Association for Computing Machinery. ISBN 9781450394871. doi: 10.1 10.114553552326.35744388 URL
https://doi.org//ee
3tt.5//555322..3355 3587438.
Herbert Woisetschläger, Ryan Zhang, Shiqiang Wang, and Hans-Aro Jacobsen. Dynamically learned test-time model routing in language model zoos with service level guarantees, 2025. URL
https://arxiv, org. org/abs////a/ao//oe/..//eee aataaabs////// 2250//555..99
Xin, Rodrigo Nogueira, Yaoliang Yu, and Jimmy Lin. Early exiting BERT for efficient document ranking. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pp. 83-88, Online, November 2020. Association for Computational Linguistics.
Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representa- tion learning help neural architecture search? In NeurlPS, 2020.
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfih International Conference Learning Representations, 2024. URL https://openrevien.net// forumm?id 6okaSfANzh.
Zesen Zhao, Shuowei Jin, and Z Morley Mao. Eagle: Efficient training- free router for multi-llm inference. arXiv preprint urXhv:2409.15511 2024a.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. LoraRetriever: Input-aware LoRA retrieval and composition for mixed tasks in the wild. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Associa- tion for Computational Linguistics: ACI. 2024, pp. 4447-4462, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10.18653/1120024.fddng org/2024. Findings-acl.266
URL
Lianmin Zheng, Wei Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging juddae with MT-bench and Chatbot Arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: wwwp:/eeeeeetoooodueeeeemmmm
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. BERT loses patience: Fast and robust inference with early exit. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18330-18341. Curran Associates, Inc., 2020. URL httttpss//proceedinng meigss.neurssp... cttppp.cccppppr...ppppppppp files/paper/2e200
Fot.993399922ccccccceeeeeeeeeee Paper. pdf.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du Yanping Huang, Vincent Y Zhao, Andrew M. Dai, Zhifeng Chen, Quoe V Le, and James Laudon. Mixture-of-experts with expert choice routing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho hett:/oorrrmiddjjdooottiiii Information Processing Systems, 2022. URL https::/ net
in Neural
Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Frangen Kagy, and Rishabh Agarwal. DistillSpec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning Repre- sentations, 2024a. URL httpss:/oppnreevie.. www.:neee/oouumiiirrsreaeeee
21

---

Published as a conference paper at ICLR 2026
Zixuan Zhou, Xuefei Ning. Ke Hong. Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan. Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. A survey efficient inference for large language models, 2024b. URL https: //arxiv.org/abs//uee 14294.
Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. EmbedLLM: Learning compact representations of large language models, 2024. URL https:
22

---

Published as a conference paper at ICLR 2026
Universal Model Routing for Efficient LLM Inference
Supplementary Material
Table of Contents
Societal Impact
24
B Static Dynamic Routing
25
C Proofs of Results in Main Body
26
C.I Intermediate Results
26
C.2 Proof of Proposition
28
Proof Proposition
29
D Zero Routing
32
D.1 Special Case: Optimal Cluster Routing Rule for K
32
E Budget Control and Deferral Curves
33
F Experimental Setup
34
F.1
Splitting and LLMs
F2
Evaluation: Deferral Curve
................................
34
F33
Implementation Details of UniRoute (LeameedMa))
................................
34
F4
Router Cost
................................
35
G Additional Experimental Results
36
G.1 Hyper-parameter Tuning
36
G.2 Hyper-parameter Choices & Statistical Significance ................................
36
Deferral Curves Additional Comparisons
39
G.4 Train Chatbot Arena and Test EmbedLLM
41
G.5 Static LLM Pool Setting. ................................
41
G.6 Visualisation of LLM Embeddings
42
H Comparison to Router Retraining
44
H.1 Challenges with Router Retraining
44
Theoretical Analysis of Router Retraining Costs ................................
44
Empirical Analysis Router Costs
45
Additional Related Work
47
I.1 Comparison to Li (2025)
49
23

---

Published as a conference paper at ICLR 2026
A SOCIETAL IMPACT
Our primary contribution a mathematical formalism for routing amongst multiple dynamic LLMs, with concrete instantiations based on prompt clustering. These are coupled with empirical results on public benchmarks, typically involving LLM accuracy as the primary metric.
We do not foresee immediate negative societal impact from our mathematical framework. While well- studied in the literature, the underlying routing problem itself could lend itself to some undesirable outcomes, e-g., a router may overly favour models that produce outputs that are systematically biased on certain data sub-groups. We believe that our framework is amenable to such constraints, and believe that accounting for these is worthy subject of future research.
24

---

Published as a conference paper at ICLR 2026
B
STATIC VS DYNAMIC ROUTING
STATIC ROUTING
New LLM C Carrives
LLM A
LLM
LLM A
LLM B
LLM
Static Router
Stario Router
Prompt
Prompt
Train new router from scratch
DYNAMIC ROUTING
A
New LLM arrives
LLM
LLM
LLM C
LLM features
LLM A features
LLM B features
Inexpensive feature
computation
LLM C festures
LLM features
Prompo
Prompt ombedder
Prompt
Frompt
Figure 4: An illustration of static routing and dynamic routing.
Static routing Static routing refers to the setting where the pool of LLMs that a router is trained on the same one that the router will see at deployment time. That is, the pool is fixed, and there is no need for the router to account for new LLMs that may be available after it is trained. If new LLMs are be part of the LLM pool, then in general the router has to be retrained from scratch. Existing works largely focus on the static routing setting.
Dynamic routing In dynamic routing, one seeks a router that can adapt to a changing LLM pool at deployment time. In particular, dynamic router is able to add new LLMs to the LLM pool without retraining. Dynamic routing is the focus of our work. As formally described in Section 3, our proposed Uni R framework relies representing each LLM as feature vector, derived based predictions on a set of representative prompts.
An illustration to contrast these two settings is provided in Figure
25

---

Published as a conference paper at ICLR 2026
C PROOFS OF RESULTS IN MAIN BODY
In what follows, we use rex (x: and r(x, interchangeably. As a shorthand, we define
S() = 13300
Σ
1(((x, m) ((x, y, hlm))
mE||3(]
((r) = (m,
Σ
1(r(z,)) m.) c((((m))
The constrained optimization problem in (5) equivalent
min S()) subject to C(r) S B.
(14)
rER
It will be useful to consider the following related unconstrained optimization objective:
M(r S(r) A C(r).
(15)
For any À 2 O, let r be the minimiser of L(r, A).
C.I INTERMEDIATE RESULTS
We first provide results that will be useful for providing the main claims in SC.2 and $C.3. Lemma 3. Given À 2 0, the optimal solution r E arg min, L(r, X) the unconstrained problem is
x (1, E argminmel 20] Ell [((x y. hmm))) + X c((((m)).
Proof. Starting with the definition of L,
L(r, A) = 5()) A C(r)
Σ 1(r(2, = m) A(x, y, klm)) + B(o,,00)
Σ
1(r(x,])) -m) -c(b(m))
metpii
(a) EgrBer Byyle
Σ I(((x, m) { ((2, h(m)) + x c(him)))
x
mE|13c]]
Egc Exn
Σ
(() =m)
(yyl Elx v, (mm) x + x c(h(m)))
me([?3]]
Coc
where (a) uses the fact that the draw H is independent of the draw (æ, y). The last line makes it clear that for any fixed H fj, and any fixed æ, minimize the overall loss, the router ought to route to the model that has the lowest cost-adjun adjtsadeeseed loss Cyc.. Thus,
(ax H) E argmin Eyja [(a;, y, 1 {m))] + X c(him))).
Lemma 4. For any X 2 0, let r be the minimiser of T(r, A). Then, for any r, ifC()) 2 C(r), then S(r)) < S(r).
Proof. Since r minimises L(r, A), for any r. we have L(rū, < L(r, That is,
S(r)) A C(r)) < S())
(16)
- 5(r)) < +()) (C(r) ((()))))))
Since À 2 0, it follows that S(r3) S()) by the assumption that C(r) C(r)) 5.
n
26

---

Published as a conference paper at ICLR 2026
Lemma 5. Let r3, be the minimiser of L(r, For X, 2 A X if and only if C(r3) - C(r)))
In other words. X + C(r)) is a non- itt.eancceaanngg..e increasinngfunctioon Hence, supx2o C(r)) = C(r)).
Proof. Since r E arg min, L(r, A), by definition, we have
S(r + À ((x)) - S(r) + A.A.C())
for any r, including rx. This means
S(r)) + (()5) S S(x)) (..())))
In symmetric manner,
S(xx.) (-XXX(C(())))))))) & 5(1)) + + -((2))
Adding the above two inequalities, we have
+ S(+x) X ((1)) S +(12) = A. C(((()))+ +x))))))))) A +(
+ + -(3)))))++++ S(5)) + -((3))
= A. -(())) +
= (A-x) C( S..
The last inequality above implies that À 2 X if and only if C())) S C(r)).
Lemma 6. Let X, X 2 0 such that < X. Given x E X and H E H. (x, + (2,
then there exists a model pair (m), (0) E 3C with (m)) + (b) such that
E ylo (xx y. ] - Eyjn [E( y. h(m))
A
S X
c(h(m)) - c(h(1))
Proof. Given H. define Am (A, x) = Eyja [((x y, h(m))]] + A c(h(m)). Recall that (x 9() argmii,melist Am (),
Observe that given (z, for each e [[C[]] Am x) is one- dieddeenniooal affine function of X. So, (mx, H( is the index of the affine function that gives the lowest value when evaluated at A. Fix æ.. Since À r Am (A, is continuous, for any m, if (ax (H) # r (ax, H),, it must mean that the index of the lowest affine function changes as we move from À to X'. This implies that there is an index pair m and (with m jf k) for which + Am æ) and A + As(A, x) cross, as we move from À to X' The cross point is precisely at À such that
Am (π, x) = Ax x))
Eyi l
((x, y, 1 (m)))
)
+ A (-c(h(m)) = Eyize (((x, + A 00)
Eyja
[l( y, h(0)]
- Egylar [((2,, y, h(m))]] c(h(m)) ) - -c(h(())
Lemma 7, Assume is a continuous random vector. Then, X + C(+)) is continuous.
Proof. We will show that for any AA E R, limAA-u (C(r)+aN) -C())] = 0. Observe that
[e (1(( E(m,or) [c (117)6e0200) ]|
- -
E E(m,or)
(0)
S P00000 c (nllitaattee00)) -C(10X(1900)))
/ le (ultā (1l3) (A03(000)) p(x) p(a)de= do
-Eor
Eor
/EE
27

---

Published as a conference paper at ICLR 2026
where we applied Jensen's inequality at (a). Define SAA be the set of input points for which r and *AA make different decisions. Precisely,
(3)( = { rx It) * (x),
Let Crnax ma. c(h) i.e., the maximum per-query cost of any LLM in our finite universe. The quantity
can be bounded further with
lo (al12taa))))))) -1(0)((000)))) p(xx) dx
Ex
2cmax
p(x) dæ
(30)
2mmx Ear (S))())()))
The proof now amounts to showing that limar-oo Byc Exp((Sx(H() = O.
Define a shorthand Am,d(a;; (x, H() Pil [e(or v. n(m) - 3/12 (((..n//mm)))
where (mm,, h(0)) E H. We
a(him)) -c(h(())
start by expanding (H) to have
(3)) = { (1) H0) # (0)
C {x 36 +m: (m, (x, H) E + ANJ
U for (m, H) E [2, A AA]}
m
FAx (JC),
where at (a) the subset relationship is due Lemma 6. Since SAA(H) S FH),H), we have
2mm Ex ((FA
20m
Σ m
IP (fæ)
(2, E AA,A+ (АJ})
2cmax Ex
Σ
P ( E [A AX]),
mifk
where at (a) we use the union bound. Note that there are a finite number of summands because Han is finite. Since is continuous random variable, for any H. (x, is also continuous random variable. As AA + O, we have IP (Am.ad (x.H) E [A, A + AA)) + 0 for any m, k. This 0 and thus limax-oo (C(r)+a>) C(r))=
means
Proposition Suppose P(zyaaz) and (xx) are continuous in æ. Let rā E arg min, L(r, 0) where [(r, S(r) + X. C(r) (see also (15)). For any budget B E (0, C(ri)) in the constrained problem in (5) (or equivalently (14)), there exists Asy 2 such that the minimiser of L(r, Ass) also minimises (5).
Proof. Since P(y|æ) and P(w) are continuous in æ, and + C(r)) is continuous (by Lemma 7), there exists Äg 2 0 such that C(r).) = B. Applying Lemma 4 with this choice of X implies that
S(1 ) < S(r),
for any r such that C(r) S C(r = B. This proves that rg is the minimiser of (5). Setting Ag Äg completes the proof.
C.2 PROOF OF PROPOSITION
Proposition (Restated). Assume that x ~ P and - P(y x) are continuous random variables. Then, for any LLM meta- iistaddistibbttion 5). and budget B (0, C(rj)] (see SC for the definitions of C and ro), there exists Xs) 2 0 such that the optimal dynamic router r* for the constrained optimization (5)
-
r" (x = argmin Eyjo km))] +Ao c(h(m)) me[[*]]]
28

---

Published as a conference paper at ICLR 2026
Note that the continuity assumption on the data distribution applies to the setting where can be represented via a fixed-length sentence embedding (i.e., a Euclidean vector). This is the primary setting of this work.
Proof. Under the continuity assumption on P. by Proposition 8, there exists Ag) 2 0 such that the minimiser of the unconstrained objective (r, As) (see (15)) also minimises (5). By Lemma 3, is established.
C.3 PROOF OF PROPOSITION
Proposition (Restated). For set of LLMs H, let r* denote the Bayes-optimal routing rule in Proposition For any x E X and ((m) E H, let:
(x (x(x,/lm)) h(m)) Py y (h(x) # y) - (h((m))
(10 Let Roi (r, (re = En Me (m) # A r(,, H(e) denote the 0-1 risk. Then under the regularity condition on IP Proposition 1, the difference 0-1 risk between 7 and Y can be bounded as:
ExG Roi (f* HGo)]
< Ex,, Roo [Ha] E(
max [el[[aaal].k]j]]
(xx, ((2)))
Remark C.I. The bound in Proposition provides us with intuition for when UniRoute (K (K-meens) is effective in practice. Specifically, it suggests that the quality gap between cluster- routing and the optimal rule in (7) is bounded by the gap between the LLM'' ss error on a prompt versus the average error on its constituent cluster. So when this gap is small, we expect cluster-based routing to provide a good quality-cost trade-off. This happens when the input queries form clear clusters, and is the case, for instance, in EmbedLLM which aggregates multiple benchmark datasets. As shown in Figure 2, UniRoute (K -MMMan)) is highly performant on this dataset. By contrast, on Headlines, effective because all prompts about news related to finance, and the cluster structure is less clear.
not as
We now start with preliminary for proving Proposition 2. For simplicity, we will refer to Hee simply by H,, and (m) by (m)) We define a proxy risk objective:
Roi (r,3)) Ex,e
Σ
(h(hm))
[r(x H( = m]
(17)
mE
where the expectation is over the joint distribution over (x, 2) (and not (æx y)).
Consider solving variant of the constrained optimization problem in (5) where the original risk objective is replaced with the proxy risk in (17):
min Eor
[Roi (r, H)]
s.t.
E (0.)
Σ m
(m) 1[r(x, = ml B.
(18)
We can then show that the optimal solution above proxy constrained optimization problem admits the same form as the cluster- based routing rule j in (13):
Lemma C.2. Under the assumption on in Proposition 2, for any set of models H, the minimizer of the proxy constrained optimization problem in (18) is given by:
F* (x, = argmin
P(2 = k|z) (h(()) + X (m))
mE[[30]] kE[K]
for some
29

---

Published as a conference paper at ICLR 2026
We will also find it useful to bound the difference between the original risk Roi (C, and the proxy risk in (17): Lemma C.3. For any routing rule and fixed H.
Roi (r,H) - Roi (r, (H) & Eur
An(ac, ll(m))
max mee ke[]
We are now ready prove Proposition 2:
Proof. The excess risk we wish to bound is given by:
Eor [Ro (F", ,T) (orr"(")
Eoo { [oo1 (F ,30) Roi (F* ,303 + Roi (2" ,30)
Roi Bon(r",30 ,3) + Roo (r*,30)
Roi (rr,
()) Eee RF*, [H)]
Ex [Ro (rr
]0]] (
max [e([
(6)
(x h(m)))
x 0 0+2 2 (m)30))
max mE|[][]].k[AK]
desired. To derive (a), apply Lemma C.3 bound the first and third terms. To derive (b), use the fact that F is the minimizer of the proxy-risk Eac Roi (.2() subject to the budget constraint in (18); since r" also satisfies the same budget constraint, it has an equal or higher expected risk than f*.
We now prove Lemmas C.2 and C.3.
Proof of Lemma C.2. Under the assumption on P. the constrained problem in (18) equivalent to minimizing the following Lagrangian objective for some Lagrange multiplier (Neyman & Pearson, 1933):
C = (0,2000
()(m())) 1 [r(x, HC) = ml]
+ A (,,
1(r(,, 30) = m) (m)
mE[1KK]
(0) Ege ExelExnEEEaa
Σ (T) = m) (/ + X cmii
me(|o<]]
=
=Egr Eng
1 (((x,H) = =m)
Σ KE[K]
P(= = kl(x) (2(h)ml) + cl)) }
mE [[]]]
Cor
where (a) uses the fact that the draw of K is independent of the draw of (æc y). The last line makes clear that for any fixed H. and any fixed æ, to minimize the overall loss, the router ought to route to the model that has the lowest cost-adjusted loss Ly. Thus,
F" (xx H( = argmin Σ
P(= = k(x) (1) + X (m)
mE|190] KE[R]
Proof of Lemma C.3. Expanding the original risk, we have:
Roi (r, E(x.v)
(m) (xx) w) 1 [r(x, =m]
1 (m)))aa) + y] [r(ar, = m]]
Byyla
30

---

Published as a conference paper at ICLR 2026
Σ
l1!ll (20) + v) 11([,, (t) ml
m m [|
Ex
Σ
Σ Thk Pul (x)) + yy (r(,, ml
(19)
me|lKK KE[K]
Recall that:
Ax him))
IP =,,== (m (x)) + (1((m))))
y 20,0=
h (m) (x:) +
Pa 2// l=
(x)) + e/)
We may next bound (19) in terms of Am (m h:m)):
Roi (r)
< Ex
Σ
Σ
Th
( (Par (1(m0 (x') * 3/ + Ax(xe, 1 (r (C) = ml]
mE |9C] kE[K]
Ex
Σ Tk Σ
w''==> (mmm))
1 [r(,, - mi]
[E(
me((9r]
Ex Σ TK
Σ
Ax(x))((m))
11 3() ml
KE(K)
mEi!
(0)
E,
Σ mc[
Per Wel (m(m) (se') # wr] (r(,, ml
(xm))
E
TTk
(ar lilml)
kE[K]
KE(K]
(b)
Ex
Σ T
Σ
Par /// [1(m) # vs] . [r(x, ml
+ Ex
max mE|]
(x Hlmm))
KE|K]
mE||KK]]
a(
Σ
(h(mm)) [r) ml
+ Ex
max [mE[[o[].<<[[K]
Ax (mm)))
mE|
Roi (C) + En
max mE((3T])AE(KK]
Ac(a, (i(es)
where (a) uses the fact that Σm 1 (r( = ml = 1 and (b) follows from the fact that Ee 1.
One can similarly show that:
Roi J() 2 Roi (r, En
As(x, ((m)))
max
[me([99]).EE(K]
which completes the proof.
31

---

Published as a conference paper at ICLR 2026
D ZERO ROUTING
An elementary approach to multi-model routing is to identify the points on the non-decreasing convex hull the set of cost-risk pairs {(c(m) R(h(m))= : m E [M]] and to route amongst the corresponding LLMs (Hu et al., 2024b). Specifically, given a budget
(c()), c(M)), (())) we may pick the two nearest costs e < B < (mp)) from the non-decreasing convex hull, and route a query to LLM / (mm))
c/ma)_B and to h (m2)) with probability c(mz))_(m1))
B B-c!nn)
with probability c(ma)_e(m)
This approach can be seen as random router that randomizes between two LLMs, where the LLMs and mixing coefficients are chosen to maximize the expected quality on the validation sample, while satisfying the budget constraint. Despite its simplicity (i.e., the routing decision being agnostic to the input), this approach was noted as a strong baseline in Hu et al. (2024b).
D.
SPECIAL CASE: OPTIMAL CLUSTER ROUTING RULE FOR K =
When the number of clusters K 1, the routing rule in (8) returns the same LLM for all queries æ:
((xHHee argmin
| (M( (r)) + À ))
(20)
nE[N]
where (A(aa)) 1
Neal Eimy)E 1h (m) (a) * y). This rule closely aligned with the Pareto-
random router.
Proposition 9. For any E R20, the routing rule in (20) returns an LLM on the non non-decreaiing convex hull of the of cost-risk pairs {(((h()) rox((())))
[N], where roi (h
Nyai ( 1/e
(n) (x)) + yy)
Proof. Suppose there exists λ E Rzo and E argmin,, (((("))) + A clm) such that (clm)) Foz (h(m)))) is not the non-decreasing convex hull. Then there exists (1m2)) E H such that either c(ma) < (m1) and roi (mm) < roi (h(m))) )), cm < clm)) and roi (mma) ruz (mm())
In either case, (h((a))) + d mma))) = roi (he (mma) + A c(ma) < roi (oi (m11)
(m(()))) + d c(m))
(m)) y (h(mee)) )) + A1 c(m1)) which contradicts the fact that my E argmin, [V + d c(m)). O
32

---

Published as a conference paper at ICLR 2026
E BUDGET CONTROL AND DEFERRAL CURVES
Ideally, one wishes to find a router that exactly satisfies given cost budget B in (5). Per Proposition 1, following standard Lagrangian theory, this may be reduced to tuning certain soft penalty coefficient Ag. This raises the question: how exactly does one pick Asy corresponding to a given IB?
We note the outset that this question is relevant for any routing method (not only ours). Indeed, even in the static routing setting, the question of how to tune (i.e., the quality-cost trade-off parameter) independent of the core technical question of how to design a good router r. To separate these concerns design of a good routing score estimate, versus design of good trade-off parameter selection it is standard the literature to sweep over (i.e., report a deferral curve).
This said, for completeness, we include a discussion on how to tune Äs. First, note that A is not determined during the training phase. Instead, we would need to tune it to meet the budget constraint whenever new set of LLMs are included in the LLM pool. In practice, this can be done using an inexpensive line search: find the smallest for which the routing rule satisfies the budget constraint. This line search is one-off procedure needed to be run when new LLMs arrive; it requires re-training, gradient updates, or labeled samples.
Second, we highlight that are able to decouple the training phase from the estimation of thanks to the optimal rule in (8):
r (x) = arg min (Loss of LLM (m)) on x)) + As (cost of calling h (m)).
With this strategy, we may guarantee that the average cost of inference on the test prompts is within the specified budget. More specifically, suppose we have access to set validation prompts that are drawn from the same distribution as the test prompts. by standard generalization theory, the average cost of inference on the test set also guaranteed (up to sampling errors) to be within budget. This guarantee is similar to a classifier trained the training set generalizing well to new test samples (the analogue of the classifier here the routing rule with a single tuning parameter).
33

---

Published as a conference paper at ICLR 2026
F EXPERIMENTAL SETUP
We provide more details on the experiments discussed in Section 6.
F.1 SPLITTING DATA AND LLMS
In the experiments, we split the data into three disjoint portions: train, validation, and test. The set of all LLMs available each dataset is also split into two disjoint sets: training models and testing models. The relationship of data splits and model splits is summarized in the following table.
Data split
Train
Validation
Test
Model split
Training models
Testing models
Training set. The training examples are meant for router training. Only information the training models (not testing models) is available in this data portion. The only exceptions are the clairvoyant oracle baselines which are allowed access to correctness labels of testing models training examples. In other words, unlike other baselines, these oracle methods observe all models during training, and are trained on both training and validation portions. These baselines are meant to establish performance achievable if router has access to all models.
Validation set. The validation examples are meant to be used to represent new L.LMs. For instance, for our proposed UniRoute (K-Means) approach, the validation set is used to compute per-cluster performance metrics of each testing L.LM observed at test time, to represent it feature vector. We assume that the validation set is small as to keep the cost adding a new LLM low.
Test set. The test examples are only used for evaluating routing methods.
Testing models represent new models that arrive at deployment time, and are not available for training (except to the clairvoyant fixed-pool router baseline). Training models meant for router training. For instance, our UniRoute (K (K-Me)ns) approach learns to route among the training models, and is tested the test route among the testing models.
F.2 EVALUATION: DEFERRAI. CURVE
Routing performance may be assessed via a deferral curve (Jitkrittum et al., 2023; Wang et al., 2024a; Hu et al., 2024b) e = {BB R(hRM(-)T))) B E [c()) c(M)], tracing the tradeoff between the cost budget B and loss of the resulting routed model. Specifically, one varies the cost budget B (c())e(M)) cM]] computes router rz(-) (-) for this budget; and plots the resulting expected loss R(hRM(s rB)). We may also use quality metric (e.g., accuracy) in place of the loss to capture quality-cost trade-offs.
F.3 IMPLEMENTATION DETAILS OF UniRoute (LEARNEDMAP)
In this section, we give details on the architecture and training of our proposed UniRoute (LearnedMap).
Architecture Recall from Section 5.2 that the LearnedMap attempts to learn æ + T cllust (z; 0) parameterised by 0. This function that maps an input prompt to probability vector Polust (2;; 0) E A where A denotes the K-dimensional probability simplex. In experiments, clust (:: O) is defined by an MLP with two hidden layers:
Pelusr (æ; 0) = (Softmax FC(K) o H' o H BN up) (æ),
(21)
[ReLU BN o FC(128)),
(22)
where:
34

---

Published as a conference paper at ICLR 2026
denotes function composition,
H and H' denote the two, separate (i.e., no parameter sharing) hidden layers of the same architecture as described in (22),
[=(:) denotes fully connected layer with outputs,
ReLU denotes the rectified linear unit i.e., ReLU(a) = max(0, a))
Softmax denotes softmax layer,
BN denotes the batch normalization layer, and
ep denotes frozen text embedding.
Recall from the experiments in Section 6 that we use Gecko Lee al. (2024b) for the text embedding 5.
We use Keras (Chollet et al., 2015) for implementing (21). experiments, Route (LearneMMap) is trained for only 5 epochs using Adam the optimization algorithm. We observe that training for too long can lead to overfitting. Training batch size is set to 64, and the learning rate is 0.005. Since yp is frozen, training celust 0) amounts training the MLP with two hidden layers. is sufficient to use CPUs for training. For one trial, training takes only a few minutes to complete.
Training
F.4 ROUTER COST
All routing methods rely on frozen prompt embedding model s. described in Appendix G.3, yp is set be the token probability quantiles of the Gemma 2B model in the Headlines dataset. For other datasets, we use Gecko 1B model (Lee et al., 2024b) for yp Importantly, all methods rely on the same embedding model, and thus share the same overhead for prompt embedding.
At inference time, for UniRoute (K-Means), deciding the LLM to route to for a given test query involves computing the text embedding, and finding the nearest centroid out of K centroids (recall that -mmenns is run only once on the training set). For UniRoute (LearnedMap), after computing the prompt embedding, pass the embedding through small MLP described in (21). Compared to
the sizes candidate LLMs (up to 60+B in EmbedLLMM for instance), invoking the text embedding model and performing few operations after (centroid lookup, invoking small MLP) incur negligible cost.
When plotting the deferral curves, do not include the cost of the routing model. Including the router's overhead will simply shift all the deferral curves to the right by a small amount, without changing the relative ordering of the methods.
35

---

Published as a conference paper at ICLR 2026
G ADDITIONAL EXPERIMENTAL RESULTS
We present additional experimental results we omitted in the main text.
G.1 HYPER-PARAMETER TUNING
To tune K in K-NN, K-means, and the proposed LearnedMap, for each candidate K, we represent the training LLM using correctness labels in the training set, evaluate the routing rule for the training LLMs on the validation set, and measure the area under the deferral curve. This evaluation metric can be seen as the average improvement in quality per unit cost. The parameter with the maximum area is then chosen.
G.2 PPPER-PRRMMEER CHOICES & STATISTICAL SIGNIFICANCE
There are three methods that we consider in the experiments Section 6 that depend on a hyperpa- rameter K. Specifically, K in K-NN refers to the number of nearest neighbors, and K in UniRoute (K-means) and UniRoute (LearnedMap) refers to the number of clusters. In Figure 2, we report the performance of these methods with the best K found on each dataset separately. We now describe the validation procedure we used to select the best K.
K-NN For each candidate K, and each prompt in the validation set, find the K nearest queries in the training set (in the Gecko embedding space). Route each prompt in the validation set to the most appropriate training LLM according to the routing rule (7) where 7 (m)) estimated with (4). Produce deferral curve on the validation set, and compute the normalized area under such curve. Select K that maximizes the area. The list of candidate K's is set to be from to one third of the validation sample size.
UniRoute (K-Means) For each candidate K, perform K-means on the training set using Gecko embeddings @Lee et al., 2024b). Compute the feature vector representation of each training LLM on the training set using (12). For each prompt in the validation set, find the nearest cluster, and route the prompt to the most appropriate training LLM according to the routing rule (8). Produce deferral curve on the validation set, and compute the normalized area under such curve. Select K that maximizes the area. The list of candidate K's s is from 3 to the number of validation sample size, divided by 50.
UniRoute (K-Means Attributes) An alternate approach that we consider in Appendix G.4 is to construct a binary vector of prompt attributes, denoting whether a prompt possesses certain characteristics (Li et al., 2024b;c), e.g., whether it requires multi-step reasoning, seeks a single correct answer, and so on. These can be seen as generalised "task". Compared to a general purpose text embedding, such a representation model routing, this can help mitigate overfitting.
coarser representation; the other hand, for the purposes of
Concretely, parameterize the prompt embedding model be ((x)) = (VT (a))cio where V e 766877 and a denotes the sigmoid function. Train each head v, E 768 (with Gecko frozen) by minimizing the sigmoid cross entropy to predict whether the j-th semantic attribute is active on each input prompt. We use the seven prompt difficulty attributes as described in Li et al. (2024b), and prompt Gemini 1.5 Pro 002 to annotate each binary attribute on each training example. Once the prompt embedding model & is trained, we freeze it, and perform the same hyperparameter selection procedure as used for K-means (Gecko) by replacing the Gecko embedding function with Φ.
UniRoute (LearnedMap) For each candidate K, perform K-means on the training set using Gecko embeddings. Compute the feature vector representation of each training LLM on the training set using (12). Parameterize the cluster assignment map with a softmax-dense layer as described in Section 5.2, resulting in a parameter to learn. Learn the parameter by minimizing the binary sigmoid cross entropy the training with the labels given by the correctness labels of the training LLMs. With the cluster map trained, for each prompt in the validation set, route the prompt to the most appropriate training LLM according to the routing rule (8). Produce deferral curve the validation set, and compute the normalized area under such curve. Select K that maximizes the area. The list of candidate K's is from 3 to the number of validation sample size, divided by 50.
36

---

Published as a conference paper at ICLR 2026
Effect of K Figure 5 shows the area under the deferral curve (on the validation set) vs candidate pa- rameter K. Importantly, the testing models and the test set are never used in the above hyperparameter selection process.
EmbedLLM
RouterBench
0.30
0.673
0.50
0.572
0.665
0.50
Area the deferal curve
0.602
4rio under the defernal curve
Number neighbors
Number of nearesd neightern
(a) K--N
(b) K-NN
0.672
0.674
0.672
0.51
0.570
0.50
0.39
Area under the deferaal rurve
0.665
Aria under the delunnal curve
Number of Clustors
Number of clstters
(c) UniRoute (K (KeMM)
(d) UniRoute (MMMns))
0810
0.672
0.828
0.662
0.808
0.667
0.824
Area unter the deferaa curve
150
0.665
clsters
102
153
kumber clustes
(e)
UniRaute
(K-Means
At-
(f) UniRoute (K (K-Means Attributes)
tributes)
0.822
0.600
0.679
0.688
0.672
0.888
0.663
0.84
Area under Uhe defundl uriee
Aroa unde Uhe delennll sarve
150
40
Number of dlusters
kumber clusters
(g) UniRoute (LearnedMap)
(h) UniRoute (LearnedMap)
Figure 5: Validation performance of methods considered in Figure 2 and Appendix G.4: K-NN, UniRoute (K K-Means), UniRoute (K-Means Attributes), and UniRoute (LearnedMap). See Ap- pendix G.2 for more details.
MLP (Clairvoyant) For this oracle baseline, as per (Hu et al., 2024b), fix the number of hidden layers to two and the number of hidden nodes in each hidden layer to 100. We fit the MLP on the combined training and validation set to predict quality score for each test LLM, and route via (2). For training, we ran 10 epochs of Adam optimization with learning rate of 10-3 and batch size 64, and picked the checkpoint that yielded the best quality metric on the validation set.
Matrix Factorization (Clairvoyant) For this oracle baseline, we fit factorized model A B. with E R Dpxd E d where Dp is the dimension of the pretrained query embedding, is an intermediate dimension, and N is the number of test LLMs to route to. We choose d using the same procedure used to pick the hyper-parameter K above. For query x and pre-trained (x) R2d Pr, this router predicts N LLM scores via A B <p(x), and routes via (2), For training,
37

---

Published as a conference paper at ICLR 2026
ran 10 epochs of Adam with learning rate of 10- 3 and batch size 64, and picked the checkpoint that yielded the best quality metric on the validation set.
Statistical Significance For the table in Figure 2 (Top), we repeat the experiments over 400 trials, and report statistical significance (via the sign test) at signficance level a 0.01. For the plots of area vs. validation sample size in Figure 2 (Bottom), we repeat the experiments over 100 trials, and report 96% confidence intervals (i.e. two-sigma standard errors).
38

---

Published as a conference paper at ICLR 2026
G.3 DEFERRAL CURVES AND ADDITIONAL COMPARISONS
Dateet
SPROUTT
63-mini
Hedlines
RosterRench
Math+
Meshaad
QNC |
Area
QNC
Arres
QNC
Area
QNC
Aren 1
QNC
Area
ZeroRouter (Tl et al., 2024b)
87.5%
.607"
100.05
58.0%
319
99.9%
669
82.8%
395
X-NN (Ht al., 2024b; Shaitzer et al,, 2023)
45.9%
.6336
29,666
3544
43.79%
X30
99.75%
.707
25.7%
.477
Retrained MI.P (Ht et al., (0224)
$5.95%
-641
50.95
320
74.2%
323
19.22
710
Retrained MadFac (Ong et al., 2025; Zhuung al., 2024)
36.61/11
.6400
54.2%
.825
80.9%
.21
99.3%
.708
UniRucte (X-Menns)
33.75
649
19,65
350
56.9%
528
99.4%
712
25.7%
490
UntRoute (Leumedldap)
33.15
23.6%
8-16
34.9%
832
99.6%
211
Clairooant MLP
26.9%
4.5%
.399
18.1%
835
95.2%
.23
25.1%
500
Cloiroyant MatFue
0,70
0.86
Accuracy
Quality
0.84
0.60
K-NN
GAN
nntraied MLP
0.82
Retrained MLP Retrained Matfac (GMeaas)
Retrained Matherr (CMlears)
UniRoute (LearnesMapp)
UniRoute eeamm
ZeroRouter
ZersRouter
0.50
20
40
60
Cost budget. et (model size)
Cost budget (USD)
(a) EmbedLLM
(b) SPROUT o3- 03-mini
0.75
0.830
Accuracy
Accuracy
0.025
0.70
KNN
CAN
0.820
Retrolied HLP
Retralned Notfac
0.65
Retrained MLP
Retrained Matfas
0.815
(LeamedMag)
UniRoute &earmedMap! Zemtouttrr
0.010
ZoroRoutor
0.60
0.2
1.0
2 Cost budget (USD) per 1K queries
Relative cost (USD)
(c) Headlines
(d) RouterBench
Figure 6: Deferral curves and router evaluation metrics (QNC, and Area) for different methods in the dynamic pool setting. Clairvoyant MLP (Hu et al., 2024a) and Clairvoyant MatFac (Ong et al., 2025; Zhuang et al., 2024) are oracle methods that observe correctness labels of testing LLMs on the large set of training prompts. ZeroRouter (Hu et al., 2024b), K K-NN Hu et al. (2024b); Shnitzer et al. (2023), and retraining variants of MLP and matrix factorization are baselines applicable to the dynamic LLM pool setting.
As described in Section 6, we randomly split examples of each dataset into training, validation and testing over 400 independent trials. Figure 6 presents the mean deferral curves of all the methods we consider. The Area and QNC statistics reported in Figure 2 are derived from these curves. All results here are for the dynamic LLM pool setting as considered in Figure 2. In Figure 6, we present two additional oracle methods:
Clairvoyant MLP and Clairvoyant MatFac: Idealized versions of the MLP and factorised routers, with one output per train and test LLM. These are trained on the combined train- ing and validation set (with correctness labels from both the train and test LLMs), and serve as oracles to estimate an upper bound on router performance on unseen LLMs (as- sume the of LLMs is fully known). These oracle baselines provide an estimate of the performance achievable when all LLMS are observed. Recall that other non-clairvoyant methods (including UniRoute) do not observe test LLMs on the (large) training data split (see Appendix F.1).
Headlines We consider the Headlines dataset (Sinha & Khandait, 2021; Chen et al., 2023b) consist- ing of 10K prompts in total. Each prompt asks an LLM determine the price direction (up, down, neutral, or none) of an item in a list of news headlines. There are 12 LLMS of various sizes in this dataset. Each LLM has distinct cost (USD) of processing one prompt, which is function of the
39

---

Published as a conference paper at ICLR 2026
imput length, output length, and fixed cost per request (see Table in Chen et al. (2023b)). We hold out six LLMs for training and the other six LLMs for testing:
Training LLMs: Textsynth FAIRSEQ. OpenAl GPT-4, OpenAl GPT-Curie, Textsynth GPT- Neox, A121 JI-Large, Cohere Xlarge;
Test LLMs: OpenAl ChatGPT, OpenAl GPT-3, Textsynth GPT-J, A121 J1-Grande, A121 J1-Jumbo, Cohere Medium.
Over 400 independent trials, randomly partition the data into 4000, 400, 5600 examples for training, validation, and testing, respectively. Unlike other datasets, we do not use Gecko for embedding prompts for Headlines. Rather, the frozen text embedding y is constructed based on the output of the Gemma 2B model': for each prompt, its embedding is the seven equally spaced quantiles of the per-token probabilities of the output tokens from Gemma 2B.
The mean deferral curves over 400 trials are shown in Figure 6c. We observe that our proposed UniRoute (LearnedMap) has higher accuracy than K-NN throughout the whole cost range.
Math+Code The Math+Code dataset is from Dekoninck et al. (2025) (containing subset of Minerva Math and LiveCodeBench (Lewkowyez et al., 2022; Jain et al., 2024)). We use 4 LLMs for testing; consequently, the training data is unlabeled, meaning that we cannot train UniRoute (LearnedMap) this case. The parameter tuning procedure (Appendix G. for K cannot be applied to the Mote+Code dataset, where we have no training L.LMs. In this case, we set K to v /Nval for K-NN and to Nva1/50 /5 for Mcaas.
'Gemma 2B model: http::///inggghgaaac.cooogggg egggmaa-gggabaa 2b.
40

---

Published as a conference paper at ICLR 2026
G.4
TRAIN ON CHATBOT ARENA AND TEST ON EMBEDLLM
Method
Areat
QNCl
Peak Acc.
0.55
Pareto-random router
.507
51.9%
Accuracy
0.50
K-NN
.472
00
52.1%
K-means (Geckol
K-means (Gecko)
.529
54.5%
0.45
K-means (Attributes)
K-means (Attributes)
.545
.97
55.8%
Individual
0.40
Pareto-random router
20
Number of parameters (Billions)
Figure 7: Deferral curves of routers trained on Chatbot Arena with pairwise comparison labels, and tested EM with per-prompt correctness labels.
We observe that representing each prompt with small number of binary attributes that capture its inherent hardness shines when there is a prompt distribution shift at test time. To illustrate this, we use the seven binary difficulty attributes proposed in Li al. (2024b), and prompt Gemini 1.5 Pro to annotate each attribute for each training prompt. We then construct a query embedder
(xx) (V Gecko(æ))) E [0, 1)";
(23)
where V E 768x7 is distilled using the training set to predict the 7-category attributes for any new prompt æ.
We compare Gecko-based prompt representation and attribute-based representation by training on Chatbot Arena conversation data (Zheng et al., 2023), and testing on EmbedI.LM, which contains mostly Q&A prompts. To reduce confounding factors, datasets (26 LLMs), and test on the same set of LLMs (i.e., no unseen LLMs at test time). After appropriate filtering, the Chatbot Arena dataset has 8447 records left. The filtering step ensures that we only deal with LLMs that are present both datasets. These examples are split further into 90% training and 10% validation splits.
train all LLMs that present in both
The Chatbot Arena dataset contains pairwise comparison labels: each user prompt is responded to by two random L.Ms,, to which the user selects the better response. To evaluate per-cluster performance for representing each LLM,, we fit the -eelley-Teery-aaae pairwise comparison labels within a cluster and estimate the pointwise quality scores for each LLM for that cluster. We use the full EmbedLLM dataset for testing.
model (Bradley & Terry, 1952) to the
The results are shown in Figure 7 where we compare two variants of proposed UniRoute (K- means):
1. K-means (Gecko). This UniRoute (K-means) (see Section 5.1) where the text embedding is set to the Gecko model (Lee et al., 2024b).
2. K-means (Attributes). This is UniRoute (K-means) where the text embedding is set to (23).
We observe that K-means (Attributes) in this case performs better than K-means (Gecko), suggesting that using prompt hardness attributes helps improve robustness to prompt distribution shifts. In fact, this routing approach is the only method that can reach the performance of the most accurate model in the pool, thus attaining a finite quality- cost (QNC). The reason the Pareto-random router has decreasing trend is because the Pareto-optimal LLMs are chosen using the validation set, and turn out to be not optimal for the test set.
G.S STATIC LLM POOL SETTING
While the dynamic pool setting is the focal point of our work, we show in Table I that even in the static LLM pool setting, UniRoute is typically comparable to most baselines. The MLP baseline alone often has slight edge. This is because, unlike our methods, which tied to particular input LLM representation, the MLP approach has the flcxibility of learning its own representation for each fixed L.M. Note that this is possible only in the static L.LM pool setting.
41

---

Published as a conference paper at ICLR 2026
In this case, all LLMs seen during training, and hence the training and validation samples have correctness label annotations from all LLMs. We tune the ppper--arameters. such as K K-NN and K-Means, the number of hidden nodes in MLP, and the intermediate dimension in Matrix Factorization, to maximize the area under the deferral curve on the validation sample. We pick K from the range (5, 10, 25, 50, 75, 100, 150, 200, 250, 300} for K-NN and K K-Means, pruning values that are too large for the given validation sample size.
For Uni Route MLea) nodes as MLP. We set the number of clusters to be the same as the chosen number of clusters for UniRoute (K-Means). We employed Adam with learning rate 0.001 and batch size 64 for training. picking the checkpoint with the best quality metric on the validation set.
used two hidden layers, with the same chosen number of hidden
For these experiments, we additionally consider the Headlines dataset (see Appendix G.3) where all LLMs are fully observed during training. We additionally consider the LiveCodeBench coding bench- mark (Jain et al., 2024), and include from 15 LLMS for which the model size was publicly available (the Math+Code dataset contains two LLMs from this benchmark). We split the LiveCodeBench dataset into 60% for training. 10% for validation, and 30% for testing. For Headlines, we use 40%, 10%, 50% for training, validation, and testing, respectively.
Table Comparing Route with existing methods for the static LLM pool setting, where Hyr Hee- We report the area under the deferral curve (t). The best baseline results are highlighted, and the best Uni U results are boldfaced. Even the static setting, our approach competitive compared to most baselines. The MLP baseline often has slight edge. This is because, unlike our methods, which are tied to particular input LLM representation, the MLP approach has the flexibility of learning its own representation for each fixed LLM. Note that this is possible only in the static LLM pool setting, and not the dynamic setting, which is the focus of this paper.
CldeCeefee
EmbbilLLM
Mislostnuct
RouterBeuch
Mali+Code
Headlins
Dataset
Method
ZeroRouter MLP Hu et al. (2024b) Matrix Factorization (Ong et al., 2025: Zhuang al., 2024)
2024b)
.483
.457
.834
.689
.0500
.747
.483
480
.522
.682
.0503
.744
.482
.82
.849
K-NN Hu al. (2024b); Shnitzer al. (2023)
.636
.5100
.744
.475
.474
.844
UniRoute (K- (K-Meuns)
.62
.0544
J44
.480
.881
.445
Uni Unuto (LeamedMap)
.0502
.463
.479
.854
G.6 VISUALISATION LLM EMBEDDINGS
We visualise the cluster-based LLM embeddings V learned from our clustering procedure on the EmbedLLM and RouterBench datasets in Figures 8 and 9. These heatmaps illustrate the Gaussian kernel similarity between all pairs of LLM embeddings on these datasets. The results are largely intuitive: e.g., on RouterBench, we find that the claude family of models are highly similar each other. Similarly, EmbedLLM, code-focussed models generally tend to demonstrate a high degree of similarity.
We emphasise again that EmbedILLM previously also considered representing LLMs as feature vectors. Importamtly however, their representation depends on a fixed pool of LLMs, and does not readily generalise to new LLMs (without further retraining).
42

---

Published as a conference paper at ICLR 2026
340 bcto
Meta Larna Lama-3-7700 3-738-nsttucct
mpp-7b-chat
CockFuse-DDepSaek333
Codel Jama-1 Ib-iretruct-fft
snala-1 38-H1
tulo-2-dpo-70e
CodeLibma
Temma 34b
Meta-Liama. Guard 2.88
Hermes 13b
CodeLiama- 34b Instructfff
Meta Uama 3 88 Instruct
Mixtral_1BAk 11Bx2_MoE__1
Starting-
Jamm-2-788-phynt
AiphaMonarch-22
BioMistral--B
tigerboe1 1b- base
mp--30b-rnsrrutt
vicuna-13b-v1.5
Starlings M-7B-bela
Llama-2. 1amma-2-1330 2f-cht-htt-
WizardLM-
openbusdy codellama2
Y--6 200K
Moe 4ith--mttthhhhheemmmmm reason-code
neursi-ctat-Tbx-
Sc/Phi Nistral 7B 32k
costin some the semuerteee
sicuma will dies 7o0 Gostuue 28 mrzzennn awd co
tesmen 340
Redaa Uame Guand es
Nou comman 180 cepere tae di cen conn anvenn
coveeeee intrra th12 Mos you 221 sostsss oper seeercree tett cocorn Air
lee toooeeee O.O.
stee T'IE /
STA 80 artouctin
spoc wateoo upeu
suseu on2 Z ewen
sotiee t-ewer
1eta gu
sou in muama Ald aoseraaaa
Coodet coneeaan
Spen
zewr 0000 fopnquddo
Figure 8: Gaussian kernel similarity between pairs of LLM embeddings EmbedLLM dataset.
claude-v1
gpt-3.5-turbo-1106
claude-v2
claude-instant-vl
miktrai-8x7b-chat
Yi-34B-Chat
WizardLM-13B--11.
gpt-4-v106-preview
mistral-7b-chat
llama-2-70b-ee
code-llamma-instuuct-aa-hhhhha
claude-v1
3.5-turrb--1111 1106 1106
claude-v2
claude flade--nssan in--insttttvv
mixtral-BxX7 -x7-cha bhchat
Yi- 343---C Chat
13B-V1.2 2
distral--b-chat
llama-2 2--0a------hha
code-llama instrunttttttto.i 34b- chot
Wizard 1a-rdLM--3BBB11
gpt-4-3 1..---10--ppeeveee
gpt-3.5 gpt-3.5--aa
Figure 9: Gaussian kernel similarity between pairs of LLM embeddings RouterBench dataset.
43

---

Published as a conference paper at ICLR 2026
H COMPARISON TO ROUTER RETRAINING
H.1 CHALLENGES WITH ROUTER RETRAINING
Per the discussion in the body, naive static router retraining conceptually appealing approach to the dynamic routing problem. While there may be certain specific scenarios where this approach is indeed advisable, we now argue why for many settings, may fall short.
First, the central requirement in router retraining is the existence of labelled sample of (prompt, target) pairs, along with the outputs of any new LLM. If this set of prompts is small which the setting considered in UniRoute, with the validation set Seal then retraining runs the risk of overfitting. On the other hand, if this set of prompts is large, then the annotation cost can be cumulatively prohibitive: every new LLM will need to perform inference the large prompt set. Further to this, the cost of retraining can be non-trivial, as typically involves many steps of gradient-based optimisation (see the next section). thus potentially delaying availability of the new router.
Second, regardless of the size of the labelled prompt set, retraining and redeployment involve a non-trivial engineering cost, as model redeployment is rarely fully automated. Third, retraining may fail adapt certain bespoke settings. For example, suppose that we wish to allow users to select a subset of models to route amongst (c.g., to enable consistency with their existing workflows). Supporting arbitrary subset of dynamic models would naïvely require exponential number of retraining rounds. By contrast, UniRoute would seamlessly handle such settings without any retraining.
H.2 THEORETICAL ANALYSIS OF ROUTER RETRAINING COSTS
We analyse the runtime complexity of UniRoute (K (K-Means) and a static routing method (such as "MLP (Clairvoyant)": see Appendix G.2) that retrains from scratch on the full training set, whenever the LLM pool changes. We assume that the static routing method is feedforward network given by dense layer on top of a frozen query embedding network. This assumption agrees with our experimental setup.
For this analysis, we use the following notation.
Symbol
Deseription
N
Training sample size
Nval
Validation sample size used represent LLMs in UniRoute. Note that Nyal N.
Number of training epochs
I
Maximum number of iterations in the K-means algorithm
M Number of LLMs in the current pool
K Number of clusters in UniRoute. Note that K Nval-
D Embedding dimension of each query
Cost for computing the embedding of one query.
Cost of router training
UniRoute (K-Means): O(BN + DNIK), which is the cost of running K-Means once on the training sample. In our experiments, we use / = 300 iterations. Note that the complexity is independent of the LLM pool size M.
Static router: O(BN + DNM),, where we assume that the cost of gradient update on example is O(DM). Note that in the more general setup where the query embedding network is also trained jointly with the routing module, the gradient update cost will depend the size of the embedding network, and the cost will be much higher.
Cost of adding a new LLM Suppose that the router has been trained. Consider the scenario where a new test L.MM arrives at test time. Let be the cost of calling the new LLM for generating response for one prompt.
44

---

Published as a conference paper at ICLR 2026
UniRoute (K-Means): O(Cival + BN + DN The first term (CCi,a)) is the cost of generating responses for all prompts in the validation set. The second term for generating responses for prompts in the validation set. The third term is for computing cluster assignments. Note that this is independent of M.
Static nuter:: O(CN + BN + DNMT). The first term for generating responses for all prompts in the training set.
Observe that there is a large difference in the costs of the two methods for adding a new LLM. In particular, the cost of adding new LLM with UniRoute (K-MMans) is independent of the training sample size N and the size of the current LLM pool M. By contrast, retraining a static router would clearly depend on these quantities.
Cost of performing LLM inference At test time, the runtime complexity for routing input query is as follows:
UniRoute (K-Means): O(B + DK + M). The second term is the cost of determining the nearest centroid. The third term is the cost of applying the routing rule in (8).
Static router: O + DM).
H.3 EMPIRICAL ANALYSIS OF ROUTER COSTS
We empirically analyse the actual wall-clock time of various key operations of UniRoute and retrained router.
We consider the same experimental setting as in Figure 2 for the EmbedLLM dataset. We compare the wall time of our proposed UniRoute (K (K-Means) to the strongest baseline, which is Retrained MLP in this case. Note that all approaches studied in this work (including UniRoute) start from embedding queries with the Gecko embedding network. So, to minimize the variance in time measurement, pre-compute Gecko embeddings and exclude this step from the following results. The following table reports time in seconds.
Operation
UniRoute (K-Means)
Retrained MLP
Training
1.41 s
8.18 s
Route one test example
0.37
10-3
3.19 10-3
Add a new LLM the pool
57.90 X 10
8.18 s
For adding new LLM to the pool, we exclude the time to run batch inference to generate responses for the prompts in the validation set because this common step for both approaches.
We observe that UniRoute is faster than Retrained MLP for both training and inference. In fact, it is also better terms quality (see Figure 2). Since Retrained MLP does not support dynamic routing, adding new LLM to the pool requires retraining the router (hence the same wall time for training and for adding a new LLM). By contrast, UniRoute only requires estimating per-cluster errors of the new LLM. Unlike the MLP approach, this does not require any expensive gradient updates. The above wall time is consistent with the runtime complexity analysis in Appendix H.2.
We make the following additional remarks.
1. We emphasize that the above K K-means clustering uses much larger dataset compared to MLP. This because K-Means clustering (training) requires only unlabeled input queries, whereas the MLP router requires labeled queries. Requiring no labels allows one to easily afford a large set of training queries. In the above experiment, the UniRoute uses 18k unlabeled examples, whereas Retrained MLP uses 3k labeled (validation) examples. Recall from Section 4.1 that we consider the setting where the validation set (i.c., containing labels from test L.Ms)) is limited.
2. The above latency analysis is in an illustrative setting where the query embedding network (i.e., Gecko in this case) is frozen. In reality, we expect router retraining to take much longer if the query embedder is also trained. In such case, Retrained MLP would incur
45

---

Published as a conference paper at ICLR 2026
significant cost every time the LLM pool changes. Our UniRoute approach would require backbone training only once.
3. Inference latency and training time are only one aspeet of the overall cost when deploying router. In practice, there are engineering and maintenance costs that one needs to take into consideration.
4. There are many applications where full router retraining is infeasible either because smaller (open-soureed) models are released frequently, or because the number of possible subsets of LLMMs one can use is exponentially large. Even if the underlying L.LMs are fixed, the test-time LLM pool can change frequently. In particular, the exact pool of LLMs available to a user may vary, depending on e.g., the availability of GPUs for serving, LLM licensing requirements, suitability to the user's particular task, etc. One can imagine use cases where custom- made LLMs of different sizes have to updated frequently (e.g.. LLM assistants for stock market news), and possibly proprietary data (e.g., LLMs for internal documents a large company).
46

---

Published as a conference paper at ICLR 2026
I ADDITIONAL RELATED WORK
Works without lask labels
Unseen models at test time
Candidate LLMs
Adaptive computation
Routing approach
Training signals
Reference
Query embeddings. No label required. Pointwise evaluation.
Smoothic
Any
x
Guha (2024a)
Cascading with loken-level features
2
x
Gupta al. (2024)
Annotations from teacher
Summon the tilams
2
X
Rawal et al. (2021)
model.
Pairwise comparison metrics. Pointwise evaluation, query embeddings. Task information
RouteLLM
2
X
Ong ct al. (2025)
Hu et al. (2024h))
K-NN router
Any
4
Shnitzer (2023)
GraphRouter
Any
Feng et al. (2024)
Pointwise evaluation, query clusters
Our proposal
Any
This work
Table A qualitative comparison of recently proposed model routing approaches. Adaptive com- putation refers to the ability to trade quality for reduced inference cost. A: The K-NN approach considered in Hu et al. (2024b); Shnitzer al. (2023) for a fixed pool of L.MSs. However, the approach can be straightforwardly extended to support unseen models at test time (i.c., dynamic pool).
Model routing. Model routing has emerged as simple yet effective strategy to lower LLMs' infer- ence cost (Hendy et al., 2023; Hari & Thomson, 2023). Recent works have studied various strategies to learn a router, including training an explicit "meta-model" based on a neuronal network (Ding et al., 2024; Šakota et al., 2024; Chen et al., 2024b; Aggarwal et al., 2024), k-nearest neighbours (Hu et al., 2024b; Shnitzer al., 2023; Stripelis al., 2024; Lee et al., 2024a), matrix factorisation (Ong et al., 2025; Zhuang et al., 2024; Li, 2025), and graph neural networks (Feng al., 2024). Works have also explored the role of supervision in training a router (Lu et al., 2024; Zhao et al., 2024b), and enhancing router robustness (Dann al., 2024; Montreuil al., 2025; Shafran al., 2025). Typically, the router orchestrates amongst multiple independent LLMs, although it is also possible to route amongst implicit sub-models, such those defined by MatFormer (Devvrit et al., 2024; Cai et al., 2024a).
Model cascading. Cascading is a closely related technique for orchestrating amongst multiple models, wherein one sequentially invokes each model in order of cost. One then uses statistics from the resulting model output (e.g., the confidence) to decide whether or not proceed to the next costlier model. Cascading has a long history in computer vision applications (Viola & Jones, 2001; Wang et al., 2018; Swayamdipta et al., 2018; Rawat et al., 2021; Wang et al., 2022; Kag et al., 2023; Jitkrittum et al., 2023). Recently, cascades have also been successfully proven in the case of LLMs (Varshney & Baral, 2022: Chen et al., 2023b; Gupta et al., 2024; Yue al., 2024; Chen et al., 2024a; Nie et al., 2024; Chuang et al., 2025).
Selective classification and learning to defer. The formal underpinnings of routing and cascading can be traced to three closely related literatures: learning to reject (Chow, 1970; Bartlett & Wegkamp, 2008; Cortes et al., 2016), selective classification (Geifman & El-Yaniv, 2019; Narasimhan et al., 2024a;b), and learning to defer to expert (Madras et al., 2018; Sangalli al., 2023). Follow- ing pioneering studies of Trapeznikov & Saligrama (2013); Bolukbasi et al. (2017); Mozannar & Sontag (2020), a series of works have studied the routing and cascading problem through these lenses (Narasimhan et al., 2022; Mao et al., 2024a;b:c:d).
Model fusion Model routing may be contrast to model fusion, where the primary goal is to leverage multiple models to improve quality, potentially at the expense of efficiency. This can involve invoking multiple models prior to generating an output (Ravaut et al., 2022; Jiang et al., 2023; Guha al., 2024b; Wang et al., 2024b; Hu et al., 2024a), or producing a single fused model (Singh & Jaggi, 2020).
47

---

Published as a conference paper at ICLR 2026
Mixture of experts (MoE). Classically, MoE models focused on learning parameters for independent models, along with a suitable routing rule (Jacobs et al., 1991; Jordan & Jacobs, 1993). These have proven an plausible alternative model specialisation (Jang et al., 2023; Douillard et al., 2024). Such models are typically of the same size; thus, cost considerations do not factor into the router design. More recently, MoEs have focussed on sub-models within a single larger model, e.g., Transformer (Fedus et al., 2022: Zhou et al., 2022).
Early-exiting. Early-exiting enables adaptive computation within single neural model, by allowing computation to terminate at some intermediate layer (Teerapittayanon et al., 2016; Scardapane et al., 2020; Zhou et al.. 2020). Often, the termination decision is based on a simple model confidence (akin to simple model cascading), but learning approaches have also been considered (Xin et al., 2020; Schuster al., 2022).
Speculative decoding. Speculative decoding is another technique that leverages two models to speed up inference, by using the smaller model draft tokens and having the larger model verify them (Stern et al., 2018; Chen et al., 2023a; Leviathan et al., 2023; Tran-Thien, 2023; Sun et al., 2024; Zhou et al., 2024a; Cai et al., 2024b; Li et al., 2024d;e). Recent works have studied approaches to combine speculative decoding with early-exits (Elhoushi et al., 2024) and cascades (Narasimhan et al., 2025).
Model representation. Broadly, the idea of representing models via compact representations has been studied in various contexts (Achille et al., 2019; Yan et al., 2020; Cotler et al., 2023). In the context of LLMs, our proposal relates to several recent strands of work (Tailor et al., 2024; Thrush et al., 2024; Zhuang et al., 2024; Feng et al., 2024; Li, 2025; Zhao et al., 2024b), which merit individual discussion.
The idea of representing an LLM via a prediction error vector has close relation to some recent works. In Tailor et al. (2024), the authors proposed to represent "experts" via predictions on a small context set, so as to enable deferral to a new, randomly selected expert at evaluation time. While not developed in the context of LLMs (and for a slightly different problem), this bears similarity to our proposal of representing LLMs via a prediction error vector on validation set; however, note that the mechanics of routing based on this vector (via suitable clustering) are novel. Thrush et al. (2024) considered representing LLMs via their perplexity on set of public benchmarks, for subsequent usage pre-training data selection. This shares the core idea of using LLM performance on set of examples to enable subsequent modelling, albeit for wholly different task. Zhuang et al. (2024) proposed to construct a generic embedding for LLMs based on performance on public benchmarks. This embedding is constructed via form of matrix factorisation, akin to Ong et al. (2025). While Zhuang et al. (2024) discuss model routing as a possible use-case for such embeddings, there is explicit evaluation of embedding generation in the case of dynamic LLMs; note that this setting would naïvely require full re- re-comuutation of the embeddings, or some version of incremental matrix factorisation (Brand, 2002).
Some recent works have considered the problem of routing with a dynamic pool of LLMs. Feng et al. (2024) proposed graph neural network approach, wherein LLMs are related to prompts and lasks (e.g., individual benchmarks that a prompt is drawn from). Such pre-defined task labels for input prompts may be unavailable in some practical settings.
Li (2025) proposed to use LLM performance on benchmark data to construct a model identity vector, which trained using a form of variational inference. This has conceptual similarity with our prediction error vector proposal (and the works noted above); however, our mechanics of learning based on this vector (i.e., the cluster-based representation) are markedly different. We reserve a separate discussion of this work in Appendix 1.1.
Finally, Zhao et al. (2024b) consider the problem of routing to a dynamic LoRA pool, where LoRA modules are natively represented by aggregating (learne)) embeddings a small subset of training examples. These embeddings are learned via contrastive loss, constructed based on certain pre- defined task labels. As such labels may not be provided in many settings, Chen et al. (2024a) implemented a variant wherein these are replaced by unsupervised cluster assignments. While similar in spirit to our proposals, the details of the mechanies different; e.g., we use the clustering to compute a set of average accuracy scores for each LLM, rather than training an additional embedding.
48

---

Published as a conference paper at ICLR 2026
Finally, note that Chen et al. (2024a) also consider the use of clustering part of router training. However, the usage is fundamentally different: Chen et al. (2024a) regularise the learned prompt embedding & based on cluster information, while we clustering to construct the LLM embedding V
Our proposed approach is similar in spirit methods such K-NN, where the prediction for prompt is based on the labels associated with queries in its neighborhood; in our case, we consider pre-defined clusters instead of example-specifie neighborhoods.
1.1 COMPARISON TO LI (2025)
In this section, we contrast our proposed UniRoute to the the LLM Bandit approach of Li (2025), which is one of the few routing techniques relevant to the dynamic routing setting. The two techniques are fundamentally different.
Li (2025) consider an online routing setting wherein bandit approaches are advisable, whereas we consider and analyse conventional supervised learning setting. Also of note is that the mechanics of producing an LLM embedding are different in their setting. Briefly, a model identity vector their work is parameter For each LLM, such a vector is optimized so that the network has low cross-entropy loss, regularized by KL term encourage the vector to be close to standard Gaussian prior. Because this is an expensive procedure to add new LLM to the pool, they propose an efficient version wherein each prompt is rated for its difficulty. The difficulty of a prompt is defined the mean of cross entropy losses on the prompt across all LLMs in the current pool. They then apply stratified sampling on the full prompt set, using this difficulty score as the strata. They run the same above procedure on the stratified sample to find model identity vectors.
feedforward network that predicts the correctness of an LLM from prompt.
In more detail: in Li (2025), the representation of each LLM in the pool is the parameter vector of a feedforward network. For each LLM h, the parameter is optimized so that the network can predict the correctness score 1yy j h(x)] of the LLM given an input prompt æ (sec Section 5.2 for a related discussion). Adding new LLM thus involves expensive gradient-based training procedure. While Li (2025) also proposed an efficient procedure that requires only 50 prompts for training. the procedure involves counting the number of LLMs in the current pool that can correctly respond to each prompt in training set. This means that the representation of the (K + 1)-th LLM will necessarily depend on the K models in the current pool. In other words, the order in which LLMs arrive in the pool influence the learned identity vectors of future LLMs.
Our proposed LLM representation in (12) is computationally efficient: It does not require expen- sive gradient updates, does not depend on the LLM pool (see the runtime complexity analysis in Appendix H). In UniRoute, LLM representations computed independently of each other (i.e., a function of the order of LLMs).
Empirical comparison Since there no source code released by Li (2025), we present a compari- son against the numbers reported in Figure 3b of Li (2025) for the Ope dataset (Fourrier et al., 2024),² 2 In this experiment, we consider the setting of using four Qwen 2.0 model variants as training LLMs, and six Qwen 2.5 models as test LLMs. See Table E.4 and Table E.5 in Li (2025) for details. The LMv dataset has 20046 examples in total, from which we use only 800 randomly selected examples the validation set for UniRoute, 6014 examples for testing (30%). ), and the rest of 13232 examples for training. We report Quality-Neutral Cost (QNC) as in Figure 2 of our work.
dataset:
http:///bugingffaccco/ssssssse Umm ttmm-Leddebbor//oeen 11m_
leaderboard
49

---

Published as a conference paper at ICLR 2026
Method
QNC
ZeroRouter
86.4%
K-NN
81.0%
Retrained MLP
84.1%
Retrained MatFac
81.0%
UniRaute (K-Means)
82.2%
UniRoute (LeamedMap)
80.2%
LLM Bandit (Li, 2025)
90%
In the above results, the QNC LLM Bandit is directly estimated from Figure 3b in (2025) by considering the operating point of "PCDR (10)" with the highest evaluation score. More precisely. that operating point has cost of approximately $1.8. With the most performant model in the pool costing $2.0, the QNC is at least $1 31.8/S2.0 = 0.9 90%. The results indicate that UniRoute is able to achieve the same performance the largest model in the pool (Qwen2.55722B I2B-Inst) with 80.2% of its cost.
50


Reviews:
[{"review_id": "GkujNWj1UZ", "reviewer": "Reviewer_vMrd", "summary": "The paper studies routing among multiple LLMs when the model pool can change after training. The idea is to fingerprint each model by taking its error to a validation set, giving us a way to then route promots to a cluster and let us estimate which model will do best. They cover two approaches, one which is K-means clusters and one which is a learned assignment. This produces budget-and-quality frontier improvements over baselines on a range of benchmarks and models.", "questions": "How stable is the cost-tuning knob across datasets and model pools?\nHow does this look on non-binary metrics?\nAre there subgroups or adversarial examples which can reduce performance?\nIs it possible to combine this with a learned router for seen models to get improvements when comparing to a static pool?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The set of evaluations is very broad and shows consistent gains against reasonable baselines\n2. The test-time approach is amenable to many real use cases\n3. The method is cognisant of cost, as opposed to naively just optimising for performance", "weaknesses": "1. The set of metrics is relatively narrow; it would be good to cover generation metrics as some other methods do\n2. It is unclear how robust this is to shifts in distribution compared to the validation set", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kDvcZ83MUm", "reviewer": "Reviewer_n7Fp", "summary": "This paper studies the problem of dynamic model routing for LLM inference: given a prompt and a pool of candidate LLMS of different cost, select the cheapest model that will still answer well, even when the candidate pool dynamically changes. Existing work mostly assumes a static pool, and retrains the router whenever a new model appears, which is impractical and expensive. The authors formalize routing with a dynamic pool and propose UniRoute, which represents each LLM as a feature vector using its prediction error on the validation dataset, and represents each input prompt via a learned embedding. Then, they estimate for each model and prompt pair the probability of error plus a cost term and pick the minimum. Two examples of UniRoute are presented; they are clustering-based routers where prompts are grouped into clusters learned using the training dataset. The authors argue that these are approximations to an optimal routing rule and provide an excess risk bound. Experimentally, the authors evaluate UniRoute against multiple baselines that cover static and dynamic routing methods on several public routing benchmarks. Empirically, UniRoute attains better cost accuracy tradeoff than baseline methods.", "questions": "- UniRoute represents the performance of the new LLM via the bilinear form $\\Phi(x)^|\\top\\Psi(x)$. Can you explain why the bilinear form is expressive enough to model the interaction between prompts and models? Can there be more complex models that uses $\\Phi(x)$ and $\\Psi(x)$ as input features and estimates the performance?\n\nTypos\n- Line 330, \"... incorrect predic let...\"\n- Line 386, missing \".\" at the end of the sentence.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a realistic and under-explored setting. Most prior work focuses on a fixed pool of LLMs and require retraining when a new model is added, UniRoute explicitly designed to handle a dynamic pool of LLMs.\n- To my knowledge, the solution proposed by the authors is novel and quite simplistic. I think that the cluster-based example setups are practical and easy to implement. It is also cost-effective to adapt to new models.\n- The paper gives an explicit optimal routing rule that shows the tradeoff between the predicted error and the model cost via a Lagrangian multiplier. It also shows how UniRoute approximates that.", "weaknesses": "- The given clustering-based examples rely on the representativeness of the validation dataset. While this works, the paper gives limited insight into sensitivity. What happens if clusters are not well aligned with the task structure? \n- The proposed method assumes that you have access to a validation dataset with labels. How could this be generalized to noisy labels for a validation dataset or a validation dataset without labels? (I think that an intuitive answer would be enough instead of running a whole set of experiments.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XJJXd7rIpF", "reviewer": "Reviewer_fvKV", "summary": "The paper studies LLM routing under dynamic model pools, where test-time available LLMs may differ from training-time ones. It proposes UniRoute, which learns a bilinear scoring function γ(x, h) = Φ(x)ᵀΨ(h). Prompts are embedded (and clustered) once; LLMs are represented by their per-cluster error rates on a small validation set. Routing is then simple cost-adjusted argmin over any test-time pool. The paper gives a Bayes-optimal rule for the dynamic setting and an excess-risk bound for the cluster-based instantiation. Experiments on four public routing-style benchmarks with 30+ unseen LLMs show UniRoute outperforms strong dynamic baselines (ZeroRouter, K-NN) and avoids retraining costs.", "questions": "1. Can you add results or at least a justified comparison against the most recent approach(es) that use a similar intermediate representation? Right now the reader has to guess how your method would fare against them.\n2. Ablation depth: If you ablate the proposed module but keep the training data / hyperparams the same, how much of the gain remains? This would confirm it’s not just better tuning.\n3. Compute: What is the actual training cost (GPU-hours, model size) and inference latency delta vs. the strongest baseline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Relevant problem: tackles a practical, current task where LLM/agent methods are still brittle, the problem choice is well motivated.\n- The components fit together: the modeling choices are consistent with the objective, and the pipeline is implementable without exotic assumptions.\n- Across multiple datasets/settings, the method shows consistent improvements over the stated baselines, not just one cherry-picked case.\n- There is at least some attempt at digging into why it works (error breakdown / qualitative examples), which is better than many incremental papers.\n- Reproducible direction: The paper uses mostly standard toolchains; another group should be able to re-implement this without reverse-engineering half the setup.", "weaknesses": "- Ablations are thin: key modules are turned on/off only in one setting; we don’t see if the effect is stable across datasets/scales. A 2–3 row ablation table per main component would already help.\n- Limited robustness reporting: no real stress test (distribution shift, noisier inputs, or lower-resource regime). Right now, the method looks tuned to a friendly setup.\n- Clarity on compute/cost: method adds some overhead but the paper doesn’t quantify it clearly; for adoption, people will want to know the inference/training cost relative to the simplest baseline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qf5ZvykYjp", "reviewer": "Reviewer_44vL", "summary": "Model routing is an important problem in LLM serving. The focus has been on static model routing, but there is a need for dynamic model routing. The paper formulates the dynamic model routing problem and characterizes the optimal solution. From this, the authors derive a practical algorithms using prompt and model embeddings.", "questions": "- Prop 1: If I understand correctly, the Lagrangian multiplier $\\lambda_{\\mathcal{h}}$ is specific to an instance of $x$ and $\\mathcal{H}$, while $\\mathcal{h}$ is the distribution of $\\mathcal{H}$. Therefore, denoting this multiplier as $\\lambda_{\\mathcal{h}}$ is confusing. Please correct me if I am wrong.\n- Implication of Prop 1: the optimal $r^\\star$ for Eq 5 is the minimizer to Eq 6 for some $\\lambda_{\\mathcal{h}}$. However, it does not mean that optimizing over $\\lambda_{\\mathcal{h}}$ and Eq 6 is going to give us the optimal $r^\\star$ for Eq 5. Therefore, the suggestion of tuning $\\lambda_{\\mathcal{h}}$, while being something sensible to do, is simply ignoring Prop 1. I want to hear the reasoning on why this is still a good idea. Maybe we are implicitly optimizing another optimization problem when doing so. The main problem is that it is not clear whether the values of $\\lambda_{\\mathcal{h}}$ satisfying Prop 1 can be found constructively.\n- Unclear formulation in Sec 4: for simplicity, let $\\Phi \\in R^{|X| \\times K}$, $\\Psi \\in R^{|H| \\times K}$. Hence, $\\Phi \\Psi \\in R^{|X| \\times |H|}$ (I'm assuming the transpose in (9) is a typo). Suppose that $|X|=1$, then $\\gamma$ is a vector function? From (8), I got the impression that the domain of $\\gamma$ is $R_+$. Therefore, something about the current formulation does not add up.\n- Line 271: underperform vs what?\n- Unclear motivation for clustering in Sec 5: What is the reason to clustering here? Is it only for computational efficiency?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- Good formulation of the dynamic routing problem.\n- Good results on standard benchmarks.", "weaknesses": "- Lack of rigor in Sec 5 (See my questions).\n- Unclear motivation for introducing clustering from first principles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)