Cost-Aware Contrastive Routing for LLMs
Authors: Reza Shirkavand
Shangqian Gao
Peiran Yu
Heng Huang
Venue: NeurIPS 2025 spotlight ()
Keywords: LLM Routing
Inference Efficiency
Contrastive Learning
Model fingerprints
Large Language Models
Primary area: deep_learning
TL;DR: CSCR embeds both prompts and LLMs into a shared space using fast logit or perplexity fingerprints. A cost‑banded InfoNCE loss trains the space to balance quality against cost. It generalizes to unseen models and out‑of‑distribution prompts.

Abstract: We study cost-aware routing for large language models across diverse and dynamic pools of models. Existing approaches often overlook prompt-specific context, rely on expensive model profiling, assume a fixed set of experts, or use inefficient trial-and-error strategies. We introduce Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework that maps both prompts and models into a shared embedding space to enable fast, cost-sensitive selection. CSCR uses compact, fast-to-compute logit footprints for open-source models and perplexity fingerprints for black-box APIs. A contrastive encoder is trained to favor the cheapest accurate expert within adaptive cost bands. At inference time, routing reduces to a single $k$‑NN lookup via a FAISS index, requiring no retraining when the expert pool changes and enabling microsecond latency. Across multiple benchmarks, CSCR consistently outperforms baselines, improving the accuracy–cost tradeoff by up to 25\%, while generalizing robustly to unseen LLMs and out-of-distribution prompts.

Cost-Aware Contrastive Routing for LLMs
Reza Shirkavand Department of Computer Science University of Maryland - College Park rezashkvccs. umd. edu
Shangqian Gao Department Computer Science
Florida State University sgaoßes isu.edit
Peiran Yu
Heng Huang"
Department of Computer Science and Engineering University of Texas at Arlington peiran-yuuuta. yu@uta. edu
Department of Computer Science University of Maryland College Park heng@cs umd. edu
Abstract
We study cost-aware routing for large language models across diverse and dynamic pools of models. Existing approaches often overlook prompt-specific context, rely on expensive model profiling, assume a fixed set of experts, or use inefficient trial- and-error strategies. We introduce lightweight framework that maps both prompts and models into shared embedding space to enable fast, cost-sensitive selection. CSCR uses compact, fast-to-cone commpte logit footprints for open-source models and perplexity fingerprints for black-box APIs. contrastive encoder is trained favor the cheapest accurate expert within adaptive cost bands. At inference time, routing reduces to single k-NN lookup via FAISS index, requiring no retraining when the expert pool changes and enabling microsecond latency. Across multiple benchmarks, CSCR consistently outperforms baselines, improving the accuracy-cost tradeoff by up to 25%, while generalizing robustly to unseen LLMs and out-of outtof-distribution prompts.
Spectrum Contrastive Routing (CSCR). a
1
Introduction
After burst of sinforreemen-learmiiie and specialized finetuning, the Large Language Model (LLM) [65] (66] 6, [6 [7 ecosystem has fractured: code models excel at generating code but hallucinate outside programming contexts, math-tuned variants solve AIME [46] yet mishandle open-ended dialogue, and instruction chatbots trade being precise for fluency. Production systems therefore host a pool of models with different sizes, licenses, and domain strengths, and decide at run time which one to call for every user prompt, worse: burden the users with picking the model they need.
router (a. model selector, mixture-gate) adjudicates that choice online. It dynamically selects the most appropriate LLM from pool for each input. Without it, users either over-pay by defaulting to the largest model or risk quality regressions by selecting cheaper ones. Current predictive routers [33 for a pool of LLMs fall into two broad camps: parametric routers and con-parametric
32 51 76 ones.
Parametric routing methods, such as softmax-based classifiers 24 optimize exclusively for top-1 accuracy without explicit consideration of inference costs. Consequently, they tend default to selecting expensive models and require full retraining whenever new models are introduced.
"This work partially supported by NSF IIS 2347592. 2348169, DBI 2405416. CCF 2348306. CNS 2347617, RISE 2536663.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

0.90
8.35
0:0
PRATTO PVEDOM
MARUO ANEOO
POGTEC MANOO
0.2
0.0
Cos (lggurts
(a) Embeed
(b) MixInstruct
(c) RouterBench
Figure 1: Accuracy-costtsize deferral curves on three expert pools. Across all benchmarks, our Sost-Spectrum Contrastive Router (blue) consistently dominates the Pareto frontier, achieving higher accuracy at lower model size and latency (left, middle) and reduced cost (right).
Recent non-parametric approaches, such UMR 40 enhance generalization by routing across joint space of prompt clusters and model footprints. On the other hand, bandit-based methods [61] 34 and UCB (3. track simplified quality-cos metrics that
[88. including Thompson sampling [55 2 disregard detailed prompt characteristics, resulting in slower convergence and reduced effectiveness, especially when facing heterogeneous and diverse input distributions. All these methods remain cost-agnostic during training, solely relying on post-hoc hyperparameter tuning to achieve an effective balance between cost and accuracy.
In our view, routing boils down to similarity search. we can embed both prompts and experts in metric space where cosine distance trades off (quality, cost routing reduces microsecond Nearest Neighbor query - no brittle softmax gate, no retraining when the pool changes.
In this paper we introduce:
Universal ultra-compact descriptors: Two lightweight and fast to compute fingerprints that work across the full spectrum of LLMs. (1) Logit footprints that require only <10 forward passes through an open-weights model. (2) Perplexity fingerprints that score any black-box API with small public LM, enabling vendor-agnostic routing.
Cost-Spectrum InfoNCE: A novel contrastive objective that (1) selects correct positives within adaptive cost bands, (2) temperature-scales each band separately, and (3) down- weights negatives in proportion to their cost. This aligns the learned metric with the accuracy-cost Pareto frontier.
Routing Efficiency: A shared space turns routing into single k-NN lookup, eliminating brittle softmax gates and retraining whenever the pool changes. Then a lightweight FAISS index makes routing a microsecond lookup.
We evaluate method three routing benchmarks span- ning both open-source checkpoints and proprietary APIs. It achieves up to 25% higher accuracy-cor cost efficiency a fixed pool of L.MMs and demonstrates strong robustness unseen models and out-of-distribution prompts at inference time.
Comprehensive Evaluation:
Related Work
2.1 LLM Routing
Routing, Non-predictive methods generate outputs from one more models before making a selection. FrugaIGPT 110 uses sequential strategy and a response quality threshold to minimize cost. Other works adopt layered inference architectures to escalate hard queries to more
powerful models 191]. or leverage cascades with verffferiiination
[54] 43 51].
contrast, predictive routing aims select the best model before any inference is performed. Strategies include supervised learning [76), reward-model-baae routing [32]
Predictive Routing.
and meta-models trained to predict LLM performance given an input 7 Router models vary widely
in implementation, including neural networks [9 88
k-nearest neighbors 33
76
80 43
matrix factorization 105 48), and graph neural networks
25
Others incorporate model-specific
tokens or train across multiple domains [18 7

---

Academic routers usually assumed the expert set is static. Recently routing with dynamic pool of experts has been explored 40 48 UMR [40] clusters probes and stores coarse capability footprints but still rebuilds them offline whenever the model pool changes. L.MM-andit (48 optimizes cost but ignores prompt semantics and offers no cold-start prior for unseen experts.
2.2 Routing within MoE and Hybrid Architectures
Routing LLMs can be viewed as coarse-grained MoE, where each expert a full LLM. Routing is central mechanism in MoE models [36 75 where expert modules are dynamically activated based on imput. While classical MoEs involved equally-sized sub-models, modern approaches like Switch Transformer [24] and Mixtral (38) employ sparse activation to minimize cost. Approaches like UltraFuser [20) highlight recent advances in combining model specialization and flexibility.
2.3
Model Fusion, Merging and Cascading
Fusion strategies synthesize outputs from multiple LLMs to improve output quality [68] 39
89
Fusion approaches often rely on unsupervised metrics [99] 73 97 or ensemble voting to determine the final output [44]. A related but distinct technique is model merging [50], where weights from multiple pre-trained or fine-tuned models combined, either directly via methods like weight averaging (92), Task Arithmetic (35). or Fisher merging [56]. In contrast, cascading invokes models sequentially (often ordered by computational cost) and halts
[2
satisfactory output is
generated [10 98 30).
To the best of our knowledge, prior work simultaneously (i) embeds both prompts and arbitrary experts into unified metric space, (ii) incorporates inference cost explicitly into its learning objective, and (iit) generalizes effectively to new LLMs and out-of-distributte prompts using simple, efficiently computable descriptors.
Method
This section formalizes our Cost-Spectrum Contrastive Router (CSCR) and its two drop-in, model- agnostic descriptors: logit fingerprints and perplexity fingerprntss.. CSCR is trained once on fixed pool of LLMs and deployed without modification any subset of that pool. At inference time it performs a k-NN lookup in a FAISS 21 index to return the k most cost-effective experts for a prompt. Throughout this section, let H (1) ....,........................... (aM) } denote the available LLMs, c(h) their normalized cost, and ( R
our frozen query encoder with trainable MLP head ge
3.1 Model Fingerprints
We map every LLM to a compact, task-independent vector d, E', Both descriptors are gradient- free. They can be computed off-line, cached, and shipped without IP-sensitive weights.
3.1.1 Logit-Footprint Descriptors for Transparent LLMs
Let Sprobe = fN autoregressive LLM h, denote by
[i=1 be a fixed set of short, diverse prompts shared across all experts. For an
pu x1,) softmax(logits,,
& V 121
(1)
the probability that emits vocabulary token generation step t conditioned on the prompt prefix t. We compress these probabilities into a fixed-length logit footprint:
diogin (h)
[pn(ux x0, ,0) R
NT
where T' is small horizon, and {ux /k- the K most frequent tokens across all probes. We Eg-normalize disga( (h) to live on the unit hypersphere, after which cosine similarity a proxy for KL divergence between the first token distributions.
We use FAISS IndexFlatIP

---

Why logits? Equation 2 directly samples the model's 's internal predictive distribution Ps ( (-)-the the objective log Po 6 therefore encodes both
very quantity trained by maximum-likeliiee topical preference and generation style while remaining inexpensive (only N forward passes with greedy decoding). Thus, we use (2) as the primary descriptor whenever logits are available. See Appendix C.1 for more detailed discussion.
3.1.2 Perplexity Fingerprints for Black-Box or API-Only LLMs
Closed-source APIS (e.g., GPT-o3 37], Gemini-2.5 expose responses but hide almost all logits. For such models we adopt per-prompt cross-entropy fingerprint:
1
(x(x)
log pil Wi
(3)
Lu
(h) = normalizo (Zx()))) E ERV
where WI,L, are the gold target tokens (ground-truth answers if available, or probe continuations): normalize(.) denotes mean-centering and
scaling. Practically, we approximate B with
a lightweight open LM that scores the API output ÿn. (x) instead of inaccessible pk:
(x(x))
Σ log Parae ())=)))
(5)
(x)]
See Appendix C.2 for a more detailed discussion.
Why perplexity? Cross-entropy is proportional to (L(pure Ph) plus entropy of the data distribution. It there- fore quantifies text fit and has long been proxy for LM qual-
ity [58 66]. Prior work has shown that perplexity can serve
as an effective metric for distinguishing between human- generated and LLL--genratedd output [31]. Moreover, although 4 is an approximation, the descriptor vector in still captures how hard each prompt is for given expert. So routing on these vectors recovers much of the benefit of logit footprints while remaining viable for black-box L.MMs. Fig- ure vre[]shon shows that, despite using the same model to compute the perplexity of text generated by different LLMs, we still observe a clear separation between the expert descriptors.
Unified metric space. We can make both descriptors re- K (i.e., the
Figure 2: Cosine similarity of per- plexity descriptors for RouterBench LLMs. Despite using shared scorer, the descriptors distinctly separate experts.
side on the unit sphere S1 by setting number of top tokens in logit fingerprints equals the number of prompts in perplexity fingerprints) with cosine similar- ity (di, d2) = d d2. A single k-NN router can thus mix open-weight experts (logit fingerprints) and API experts (per- plexity fingerprints) without altering downstream training loss and only the descriptor extraction pipeline has to change (See Table 4).
3.2 Cost-Spectrum Contrastive Router
A contrastive router learns a shared embedding space where each query vector is pulled toward the descriptor of the right-sized expert and pushed away from less suitable ones. This yields three key pay-offs. First, because routing reduces to a nearest-neight neighst lookup in that space, inference is microsecond operation that adds virtually no latency to serving large expert pools. Second, contrastive objectives let the router exploit implicit supervision ("expert X solved this prompt while expert Y failed' or "X cheaper than an equally accurate Y") so can trained with only correctness or cost signals with dense human annotations required. Third, the geometry learned by contrastive

---

learning naturally generalizes: queries that look semantically or structurally similar land near the same expert regions, which improves robustness to distribution shift and unseen prompts, a behavior long noted in contrastive representation learning. Together, these properties make a contrastive router an efficient, supervision-light and highly adaptable choice for directing traffic in modern multi-LLLM systems.
3.3 Background: The Classic INFONCE Loss.
1 and memory bank of M keys {em JM A query encoder Xo:X' - Rd produces representations q, ==((x)/ (l.) and the keys are fo-norallieed in advance, 6m Em/||Emlll (here, Fm is the expert descriptor from Equation 2 or 4 i.e. Fm = d(hm))) For each query let P(i) C [M] denote the positives (e-g. correct experts) and
Let minibatch contain queries {x:}_=
(ii) [M] Pii) the negatives. The vanilla InfoNCE objective [87 maximizes log-softmax over
cosine similarities
Σ exp 9 mEP(()
1
LInONCE
log
(6)
B
M
exp
C.p
m'=1
where 7 > 0 is a temperature. It encourages queries to be close to any positive but far from all negatives, thereby learning metric embedding.
3.4 Cost-Spectrum InfoNCE.
Routing must balance two competing axes: quality (the expert is correct) and inference cost Gn (e.g. dollars or latency). The classical InfoNCE loss ignores Cms the encoder can satisfy the objective by clustering any correct experts-most often the cheapest ones, since there usually more of them in the pool-around the query embedding. Compounding this, easier prompts tend to occur more frequently, so training examples are skewed toward cases where cheap experts suffice. Once those low-cost positives are nearby, there is training signal to learn where the slightly more expensive but markedly more accurate models live. Empirically, this drives the router over-use the bargain-bin checkpoints, hurting accuracy and leaving significant potential untapped, even though paying little more would buy a large quality jump (see Table 5).
Why Incorporate Cost:
We therefore introduce cost-aware spectrum version that:
1. Selects all positive per cost band, preventing domination extremely cheap extremely costly experts
2. Assigns band-specific temperatures so that harder (costlier) positives yield smoother gradi- ents
3. Penalizes negatives proportionally to their cost, pushing the encoder to prefer cheaper mistakes if it must err.
Formally we first normalize costs Cm [0, 1] and partition them into K disjoint percentile bands B, = {m Cm (++)) with quantiles 30 = <... < 8x = 1. For each query and band let Piu Pii) n BR denote the set of correct experts in that band. All experts in Pik are treated as positives, and weighted by a softmax over similarities scaled by a band-specific temperature
T Tmin + a Ck:
where is the mean cost of experts in Bg.
With (x) E PR being our frozen query encoder with a lightweight trainable MLP head go).), the loss for query qi ge(((x)))/ (lgg (f(x)))lla is then an average over all non-empty bands:
5

---

Emepie exp
q. Cm
1
log
Kii
Em'=1
9.
exp
where K, is the set of cost bands that contain at least one positive and 7 0 controls the negative cost penalty, Averaging over the minibatch yields Lcs
-1=1 CCS.
B
Banded positives. By retaining all positives within each cost band, ensure that high-cost, correct experts still receive gradient signal, even when low-cost models also answer correctly. This prevents cost-collapse, the failure mode discussed in 3.4. where training signal concentrates on cheap experts due to prompt and model imbalances.
Cost-dependent temperature. Higher bands (larger 0&) get larger Tke flattening their softmax and avoiding vanishing gradients when few difficult positives exist. In contrast, cheap bands keep a low temperature, sharpening the push towards inexpensive correct experts.
Negative cost penalty. Subtracting 7cmm in the denominator (not the numerator) means that expen- sive wrong experts contribute more to the partition function, hence increase the loss; the encoder is thus encouraged to separate from them first.
Thus align three signals in the same metric space: (i) semantic proximity via the query encoder, (ii) expert capability via fingerprints, and (iii) user preference via cost scaling. Previous cost-aware objectives for retrieval weight the final scoring function inference time (e.g. 48 40)). Our formulation also integrates cost during representation learning, inducing a feature geometry that naturally interpolates accuracy and cost. Equation (8) collapses to standard InfoNCE when K = 1 and 7 0- See Appendices
C.3 and C.4 for a more detailed discussion.
3.5
Inference Router
Given test prompt we retrieve
f(x) arg max (cos(go(((x)) dr) - Ac(hh]]
(x)
is the cost weight and Top, (x) retrieves the k most similar experts to the prompt from the FAISS index. We = by default. During training, similar composite score appears in the cost- spectrum InfoNCE objective (Equation (8), allowing the encoder to rank candidate models by similarity to expert descriptors plus the cost term ye(h), just as the inference rule of Equation We show in the next section that this alignment between training and inference is highly effective.
Experiments
4.1 Experimental Settings
Baselines We compare our proposed method against a comprehensive set of baselines. Specifi- cally, we include UMR 40 a recent technique that clusters prompt embeddings to route queries to LLM pools efficiently; Thompson Sampling [48] 22, which frames routing as bandit explo- cxtlootxploitttion problem balance cost and accuracy dynamically; Pareto- oottmml routing [34), strategy that selects models by explicitly considering the cost-accuracy Pareto frontier; and two extreme baselines: Random, which selects models uniformly at random to represent naive routing without intelligent selection, and Oracle 40), which always selects the most accurate model at the lowest possible cost and thus represents a theoretical performance ceiling. Additionally, we evaluate against parametric-softmaa gating methods inspired by mixture-of-experts architectures(c.g (62]) and SoftMoE [64]. which models router decisions via differentiable soft gating functions.
Datasets & Benchmarks We train our router and evaluate it on three datasets: EmbedL. L.M [155]
InxInstruct (39], and RouterBench (34 For EmbedI.LMM and MixInstruct, we sample 192 probes from their respective validation sets. Each probe is processed to extract logit- based descriptors by

---

EMbedLLM
Mix-Instruct
RouterBench
Router
AUDC
QNC 4
Peak
AUDC
QNC 4
Peak
AUDC
QNC 4
Peak
Oracle (upper bound)
0.960
2.87
0.979
0.079
10.17
0.081
0.891
0.290
0.910
UMR
0.515
58.61
0.562
0.049
10.35
0.050
0.568
0.487
0.615
Thompson
0.472
0.514
0.044
10.09
0.045
0.622
1.634
0.787
Soft-MoE
0.404
70.00
0.577
0.030
10.09
0.045
0.599
1.659
0.794
Parametric
0.506
70.00
0.555
0.039
12.00
0.052
0.691
1.658
0.794
Pareto-Random
34
0.369
16.03
0.393
0.036
6.16
0.043
0.5172
0.589
0.545
Random
0.379
18.09
0.392
0.037
6.16
0.043
0.5147
0,585
0.542
CSCR (Ours)
0.541
43.28
0.578
0.051
9.32
0.052
0.7110
1.660
0.794
Table 1: Deferral curve metrics across three benchmarks. Our Cost-Spectrum Contrastive Router achieves the highest area under the deferral curve (AUDC), competitive superior peak accuracy and lower quality-neutral cost (QNC) compared to key baselines. The Oracle router serves as an upper bound, retrospectively selecting the lowest-cost LLM that yields the correct answer.
capturing the top K 256 tokens
horizon of T 10 tokens (Equation ( For RouterBench, we sample 192 probes from its training set. We compute perplexity- -based descriptors on RouterBench binary accuracy as the per-sample
and use GPT-2 [66]. On both EM and RouterBench, evaluation metric. For MixInstruct, we employ exponentiated BARTScore 997 as the evaluation
metric, following the approach in prior work 40 39
Training We frozen bentence-transformert Mat.ormarrrrall-iiiiii-------S 70 model the bedding backhone across all experiments. Our trainable router component is a two-layer MI.P which projects prompt embeddings into the expert descriptor space. We train our contrastive router on the training splits of each dataset. For the cost spectrum loss (Equation (8). we set the number of cost bands K and the negative cost penalty temperatures (Equation are set as = 0.25 and Tmin = 0.05. See D.I for full details.
0.2. The hyperparameters for band-specific
Evaluation We evaluate each routing strategy using deferral curve [40] which plots the average response quality against the total inference cost. Sweeping the routing penalty parameter over the interval A [0, Amax] (Equation (9) traces the deferral curve. For the EmbedLLM and MixInstruct datasets, we define the cost of processing prompt as the number of parameters in the LLM, proxy for computational resources and latency. In the case of RouterBench, we utilize the actual API call costs in USD, as provided in the dataset. Following 400 we employ evaluation metrics including Area Under the Deferral Curve (AUDC), peak accuracy and Query-Normalize Cost (QNC), the minimum relative cost required to match the performance of the most accurate tested L.M..
4.2 Results
Table D presents Deferral curve metries across the benchmarks. Our CSCR consistently outperforms all relevant baselines, achieving the highest AUDC and demonstrating competitive or superior peak accuracy. Notably, it attains lower QNC, indicating more cost-effective routing decisions. These results show the effectiveness of our cost-aware router learning approach in balancing performance and inference cost. The Oracle router, which selects the optimal LLM for each query, establishes an upper bound for performance. The benchmarks are dominated by lower cost experts, hence the lower QNC for random baselines. See Appendix D.6 for results on statistical significance.
4.2.1 Generalization New LLMss
We also evaluate the robustness of our method and baselines in scenarios where new LLMs are introduced during testing. Specifically, we select two-thirds of the EmbedLLM training models to train our router and the baselines following 40), using only responses from these selected models. Table 2 summarizes the performance metrics, while Figure iillustratte the corresponding deferral curves. Testing is conducted exclusively on the unseen L.M. pool. Our results indicate that our approach exhibits superior robustness under these conditions.
7

---

0.55
0.55
0.50
Accunscy
0.45
0.45
0.40
0.40
UNI
0.35
UNR
RANDOM
RANDOY
0.30
FARETO RANDOM
0.30
SOFTMOE
0.25
THOMPSON
CSCR
0
10
20
30
40
50
80
70
10
20
40
50
BO
(R)
Params
Figure 3: Deferral curves of test on new LLMs.
Figure 4: Deferral curve of OOD prompts.
EmbedLLM
EmbedLLM
Router
Router
AUDC
QNC
Peak f
AUDC f
QNC 4
Peak f
Oracle (upper bound)
0.9111
4.118
0.951
Oracle (upper bound)
0.9511
2.872
0.979
UMR
0.4766
64.300
0.518
UMR
0.4249
47.910
0.459
Thompson
0.4478
68.904
0.514
Thompson
0.4863
69.983
0.574
Soft-MoE
0.4109
18,282
0.485
Soft-MoE
0.4207
70,017
0.555
Pareto-Random
0,3812
15.662
0.407
Pareto-Random
0.3676
17.417
0.396
Random
0,3829
15.372
0.403
Random
0,3717
17,499
0.397
CSCR (Ours) 0.4848
70.000
0.565
CSCR (Ours)
0.5146
42.338
0.568
Table 2: Deferral curve metries on new LLMs.
Table 3: Deferral curve metrics on OOD prompts.
CSCR shows better robustness to new LLMs.
CSCR shows superior robustness.
4.2.2 Generalization to
Prompts
We evaluate approach on out-of-distrrbbtiin (OOD) prompt test time scenario. We split the EmbedLLLM dataset into two subsets: one focusing on STEM-related prompts (e.g.. science and technology) and the other comprising all remaining categories (see Appendix D.2.1 for detailed experimental settings). As illustrated in Figure 4 and quantified in Table 3 CSCR significantly outperforms all baselines across all key metrics. Specifically, our router achieves an AUDC of 0.5146 compared to the next- best baseline, Thompson, at 0. 0.4863, highlighting its superior robustness and accuracy when handling diverse OOD prompts. This performance advantage demonstrates that our method generalizes exceptionally well, maintaining reliable eecision-maaing capability across varied and harsh distributional shifts.
4.3 Ablative Studies
4.3.1 Descriptor Choice
compares two descriptors on Mix-Instruct, the only benchmark where we can compute both. Perplexity descriptors (obtained by running every candidate answer through an auxiliary language model) lift the AUDC from 0.461 to 0.467. The absolute peak accuracy, however, changes less than 0.1%. Because the perplexity pipeline requires (i) generating text with the model in the pool and (ii) a second forward pass through public LM,, often at least 2x slower. Logit descriptors, in contrast, need only single pass on open-weights models and still deliver competitive AUDC. We therefore adopt logit-based descriptors for all open LLMs and fall back to perplexity descriptors only when logits are inaccessible. The "mixed" row in represents the results obtained by using logit descriptors for randomly selected LLMs and perplexity descriptors for the remaining This shows that both descriptors can be combined within the results. In fact, performance slightly improves compared to using only a single descriptor type. This observation further supports discussion of the "unified metric" in Section 3.1.2
pool without negatively affecting the
4.3.2 Cost-Aware Training

---

MixInstruct
EmbedLLM
Router
Router
1
QNC 4
Peak f
AUDC
QNC +
Peak 1
Logit Desc. 0.0461
6.426
0.046
Vanilla
0.3421
6.382
0.362
Perp. Desc. 0.0467
8.967
0.047
Cost-Aware 0.4951
11.065
0.540
Mixed 0.0473
6.233
0.047
Table 5: Effect of cost-aware training: injecting cost awareness into the contrastive loss prevents the router from concentrating and boosts the AUDC and peak accuracy of the rotter. The trade-off is higher QNC.
Table 4: Effect of Descriptor Type. Perplexity descriptors slightly improve AUDC but require extra pass compared to the faster logit descriptors. Mixing descriptors has no impact on results.
cheap experts,
Ω
Replacing the vanilla InfoNCE loss with our cost-spectrum variant significantly increases the AUDC (0.342 + 0.495) and raises the peak at- tainable accuracy from 36% to 54% shown in Table 5 The trade-off
F13) 8 and P13) - then F13.02) is approximately
6 Eeacitrs
A,-8088
higher Quality- Neutral Cost QNC, meaning the router leans more on expensive but accurate models; however, the large AUDC gain shows that, for any realistic cost budget, users receive better overall. This confirms discussion in Section that explicitly teach- ing the encoder respect the cost hierarchy of experts crucial.
7.90
8.02
Dissuuk much 7ulnkisc
0.382
Figure 5: Real sample of routing decisions made by UMR, vanilla contrastive router, and CSCR. CSCR chooses a more expensive but accurate expert than the vanilla router, while also selecting cheaper option than UMR, achieving better accuracy-cost trade-offs on both ends.
Furthermore, Figure and Table show that while cost-aware contrastive training incurs more cost than the vanilla variant, it remains far more efficient than the baselines, achieving lower QNC by learning to distinguish good cheap experts from both bad cheap and bad expensive ones. See Figure for an example.
4.3.3 Cost-Spectrum Granularity
shows that performance varies with the number of cost bands. Using bands yields the best results, with the highest AUDC and Peak accuracy, suggesting a good balance between flexibility and generalization. Fewer bands (e.g., 2) limit routing precision, while too many bands (e.g., 15 or 20) degrade performance, likely due to wverffragmentttion and increased decision noise. This highlights the importance of tuning the number of bands per dataset avoid both under- and overfitting. See Ap- pendix D.2 for more ablations and detailed discussion.
70
Figure 6: Deferral curves across differ- ent numbers of cost bands.
Theoretical Analysis
Set-up- Let X' be the space of prompts and H. (hi...., ) fixed pool of experts with per-call cost
X' and ground-truth write ((,,8, 4) [x) * y] for the 0-1 loss and ((z, h) = 8, 1] for the Bayes error. The cost-adjusted risk of a router
c(h) RR For query
(x(r) E( (x hr(s)) d(tr()))
where λ Az trades accuracy for cost 23
Our router embeds queries via 4 X Rdd and experts via E = Bui....edt MRd using either
(i) logits fingerprints (EmbedLL.M. Mix-Instruct) (ii) perplexity fingerprints (RouterBench), Given query z, the k-NN rule selects

---

fx(x; arg min mc(M]
Σ
( hm) +dc(Atm)]
(eN
local
where (1) are the l nearest training prompts to x in (0,())2
5.1 Excess-risk of cost-spectrum k-NN
Assumption 5,1 (Lipschitz Bayes error). There exists L > 0 s.t. for all x & and h E H.
(r h) -x(2', h)] S (1 Dg(2')ll..
Theorem 5.2 (Excess risk), Let fy be trained on i.i.d. prompr embeddings. Under Assumption 5.1 for any A 2 and any k n,
E [[A((F))] - Rx(r") & cVV
where C depends only on L and the diameter of (X),, and r* is the Bayes-optimal rule r" (2)
arg minm [7(2, hm)) + )c(hm)).
The proof follows the classical k-NN bound of 53 5 with extra Xc(h) term that is constant w.r.t. x and therefore preserves the rate.
5.2 Consistency of Cost-Spectrum InfoNCE
Fingerprints live on gd-1, so dot products are scaled similarities Sim n. cm with band-dependent To
temperature Tk (Equation (7). Let Bu 0 < B1 < ... < 3x 1 partition costs into bands B₃ = {m c(hm) [8.. {x+1)}. and denote Pa: = P(t) n By the correct experts for query i that fall in band k. The Cost Cost-Spectrum InfoNCE loss for single query is
Σ
expl (Sm))
log
mEP
KC
AEK.
Σ
exp Sim 7 Crn)
m/
where K., [k: Psk #2 and i is the negative cost penalty. Lemma 5.3 (Directional alignment with cost bands). At any stationary point of Los = E, pCs.
for every query and any m+ EPu. m Ei( with Cn Smm- 4 cm+ > q. cm-
Lemma 5.3 shows the optimum ranks cheaper correct experts ahead of expensive or wrong ones, explaining the empirical benefit the cost term.
5.3 Discussion
Theorem 5.2 guarantees that if query-expert descriptors are Lipschitz, CSCR converges to the Bayes-optimal router at the usual k-NN rate. Lemma 5.3 justifies the specific form of our InfoNCE objective. Sec. 4 verifies these claims on three benchmarks. See proof' in Appendix
Conclusion
We presented CSCR, simple and efficient framework for cost-aware routing across a pool of LLMs. Our method uses two lightweight expert descriptors and trains contrastive encoder select the cheapest accurate expert within adaptive cost bands. Despite its simplicity, CSCR outperforms more complex routing baselines and demonstrates strong generalization to unseen models and out-of- distribution prompts. Our findings highlight the importance of embedding cost-awareness directly into the training objective of routers, rather than deferring it to test time. As model pools grow size and diversity, activating the right-sized expert per query is critical for minimizing latency and cost. We see CSCR as a step toward more sustainable and adaptive L.M deployments, and believe this line of research is essential to avoid defaulting to unnecessarily large models for simple tasks.
10

---

7 Acknowledgments
The authors would like to thank Zahra Miri for her assistance in preparing the figures. This work was made possible by NSF IIS 2347592, 2348169, DBI 2405416, CCF 2348306, CNS 2347617.
References
[1] Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang. Shyam Upadhyay, Manaal Faruqui, and Mausam Automix: Automatically mixing language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systemx, 2024, /oget seapeerrriieenee/foorue id=e6MrwIvgzX
[2] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi ammed bandit problem. In Conference on learning theory, pages 39 1.91.10 JMLR Workshop and Conference Proceedings, 2012.
[3] Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-2566 2002.
[4] Peter L. Bartlett and Marten H. Wegkamp. Classification with reject option using hinge loss. Journal of Machine Learning Research, 9659):1823-1840, 2008. htttp://mmmr.rgg//peeeeeee bartlett08a html
[5] Gérard Biau and Lue Devroye. Lectures on the nearest neighbor method, volume 246. Springer, 2015.
[6]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin. Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner,
McCandlish. Alec Radford. llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[7] Ruisi Cai, Saurav Muralidharan, Greg Heinrich, Hongxu Yin, Zhangyang Wang, Jan Kautz, and Pavlo Molchanov. Flextron: Many-in-one flexible large language model. In International Conference on Machine Learning (ICML), July 2024.
[8] Tianle Cai, Yuhong Li, Zhengyang Geng. Hongwu Peng. Jason D. Lee, Deming Chen, and Dao. MEDUSA: Simple LLM inference acceleration framework with multiple decoding heads. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org. 2024.
[9] Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean- Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023.
[10] Lingjaao Chen, Matei Zaharia, and James Zou. Frugalgpt: How large language models while reducing cost and improving performance. arXiv preprint arXiv: 2xxiv-2305.05// 2023.
[11] Shuhao Chen, Weisen Jiang, Baijiong Lin, James Kwok, and Yu Zhang. RouterDC: Query- based router by dual contrastive learning for assembling large language models. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. httptttt:t//ppreeeeeeee www./eetnuurneeeee id=7RQvjayHr
[12] C Chow. On optimum recognition error and reject tradeoff. IEEE Transactions information theory, 16 (1):41-46, 1970.
[13] Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et Unified scaling laws routed language models. In International conference on machine learning, pages 4057-4086. PMLR, 2022.
[14] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In ALT, 2016. URL https://cs nyu. edu/ edw/ "wth///wuh//////eeeee caauud/rejppdf
[15] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):2122, 27, 1967.

---

[16] Christoph Dann, Yishay Mansour. Teodor Vanislavov Marinov, and Mehryar Mohri. Domain adaptation for robust model routing. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning, 2024. URLlhttps://ppenrrviies http://w.eee/oooommmddddeeeeerr
[17]
DeepSeck-AL Deepseek-v2: A strong, economical. and efficient mixture-of-experts language model, 2024. URL http:/ttt//aaiii.or///////////// wbapg//gaaa2440555 .44344
[18] Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit S Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham M. Kakade, Ali Farhadi, and Prateek Jain. MatFormer: Nested transformer for elastic inference. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL wtw.ttts:://////ttt htttp:://operreiicccccc
[19] Dujian Ding, Ankur Mallick, Chi Wang. Robert Sim, Subhabrata Mukherjee, Victor Rühle, Laks Lakshmanan, and Ahmed Hassan Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations, 2024, URL https; //openreview
[20] Ning Ding, Yulin Chen, Ganqu Cui, Xingtai Lv, Weilin Zhao, Ruobing Xie, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Mastering text, code and math simultancously via fusing highly specialized language models, 2024.
[21] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Femmn-em Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss lihrary. arXiv, 2024.
[22] Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. Chapman and Hall/CRC, 1994.
[23] Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17. pages 973-978. Lawrence Erlbaum Associates Lid, 2001.
[24] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers:
Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1 -3, 2022.
[25] Tao Feng, Yanzhen Shen, and Jiaxuan You. GraphRouter: A graph- router for Ilm selections, 2024.
URL htt/tttts///axiii org/abs/////oooo.ooo......... /4100. .08334
[26] Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evei. Scaling laws for sparsely- connected foundation models. arXiv preprint wwx.vv2300.08522 2023.
[27] Yonatan Geifman and Ran EI- Elniv..v. SelectiveNet: A deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 2151-2159. PMLR, 09 15 Jun 2019.
[28] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2112.00752, 2023.
[29] Neel Guha, Mayce F Chen, Trevor Chow, Ishan S. Khare, and Christopher Rc. Smoothie: Label free language model routing. In The Thirty-iggtth Annual Conference on Neural Information Processing Systems, 2024. URL https://openrrevier htt..s/aoorrrriiipppsasaaaagpgpp
[30] Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrituum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfih In- ternational Conference on Learning Representations. 2024. /Rt//httss//ppeeeeraii wwwpeeewiiiieeeeeoourreeeeee id=KgaBScZ4VI
[31] Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Spotting Ilms with binoculars: Zero-shot detection of machine-generated text, 2024. URL htt.thttt://aaatt wwwp://////aaaaeeeeeeee.e hbee//bb//2//// 2401. 12070
[32] Surya Narayanan Hari and Matt Thomson. Tryage: Real-time, intelligent routing of user prompts to large language models. CoRR, 2023.
[33] Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are GPT models at machine translation? comprehensive evaluation, 2023. URL battps://arxee org/ abs/ 2w1.22222..02222
[34] Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. RouterBench: A benchmark for mulli-LLMM routing system. In Agentic Markets Workshop at ICML 2024, 2024. URL httL:hhtts:////tt httpps://openrrevie net mett/formm?dde IVXmV8Uxwh
12

---

[35] Gabriel Ilbarco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi. and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023, URL ftiap:://opnneviee.
jrjj
[36] Robert A. Jacobs, Michael I. Jordan, Steven Nowlan, and Geoffrey Hinton, Adaptive mixtures of local experts. Newral Computation, 3(1):79-87, 87, 1991. doi: (1116//ncco.1991....77
[37] Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed EI- Hi-kkik,, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024.
[38] Albert Q Jiang, Alexandre Sablayrolles. Antoine Roux. Arthur Mensch, Blanche Savary, Chris Bamford. Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 04008, 2024.
[39] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pair- wise ranking and generative fusion. Anna Rogers, Jordan BradeGr, and Naoaki Okazaki, editors, Proceedings of the 61xt Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 14165-144 14178, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 1 (1118865/1/12/ 1/2013 lal.-lln...777 792. ftt.ttp:: pttttps://acaaanthoolo
org/ 22wg./222.... acl- long. 792
[40] Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Zifeng Wang, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Universal model routing for efficient Iim inference, 2025. URL http:ttt/txxx...oott org/ htw..//22..07777
[41] M.I. Jordan and R.A. Jacobs. Hicrarchical mixtures of experts and the EM algorithm. In Proceedings of 1993 International Conference on Neural Networks -UNNN-93-Nagoya, Japan), volume 2. pages 1339-1344 vol.2, 1993. doi: 10.1 109///CN, 1999.916799
[42] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, 2020.
[43] Chia-Hsuan Lee, Hao Cheng, and Mari Ostendorf. Orchestrallm: Efficient orchestration of language models for dialogue state tracking, CoRR, 2023.
[44] Young-Suk Lee, Md Arafat Sultan, Yousef KulKur Tahira Naseem Asim Munawar, Radu Florian, Salim Roukos, and Ramón Fernandez Astudillo. Ensemble-instruct: Generating instruction-tuiin data with heterogeneous mixture of Ims. arXiv preprint arXiv:2310.1396 2023.
[45] Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274-19286. PMLR, 2023.
[46] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi. Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang. Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024.
[47] Margaret Li, Sneha Kudugunta, and Luke Zettlemoyer. (mis) fitting: A survey of scaling laws. arXiv preprint arXiv:2502. 18969, 2025.
[48] Yang Li. LLM Bandit: Cost-efficient llm generation via preference-oondttioned dynamic routing, 2025.
URL https: //arivv///x....//a wwwxii.org//eee. org/ ab2//222.252..... 02743
[49] Yueyue Liu, Hongyu Zhang, Yuantian Miao. Van- Van-Hoang Le. and Zhiqiaan Li. Opillm Optimal assignment of queries to large language models. In 2024 IEEE International Conference on Web Services (ICWS). pages 788-798 IEEE, 2024.
[50] Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. Merge, ensemble, and cooperate! a survey on collaborative strategies in the era large language models. arXiv preprint arXiv:2407.00089 2024.
[51] Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward- reward-guided guided ensemble of large language models. CORR, 2023.
[52] Xiaoding Lu, Zongyi Liu, Adian Liusie, Vyas Raina, Vineet Mudupalli, Yuwen Zhang, and William Beauchamp. Blending is all you need: Cheaper, better alternative to trillion parameerss 11m. arXiv preprint arxiv: 2401 02994, 2024.
13

---

[53] Ulrike von Luxburg and Olivier Bousquet. Distance-based classification with lipschitz functions. Journal Machine Learning Research, 5(Ju)):6696655, 2004.
[54] Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang. Shyam Upadhyay, Mausam, and Manaal Faruqui. Automix: Automatically mixing language models. CoRR, 2023.
[55] David Madras, Toniann Pitassi, and Richard Zemel. Predict responsibly: Improving fairesss and accuracy by learning to defer. Proceedings of the 32nd International Conference on Neural Information Processing Systems, NeurlPS' 18, page 6150-6160, Red Hook, NY, USA, 2018. Curran Associates Inc.
[56] Michacl Matena and Colin Raffel. Merging models with fisher-weighted averaging, 2022.
[57] Quinn McNemar. Note on the sampling emorr of the difference between correlated proportions or percentages. Psychometrika, 12(2):1531177 1947.
[58] Tomas Mikolov and Geoffrey Zweig. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT). pages 234-239. IEEE, 2012.
[59] Yannis Montreuil, Axel Carlier, Lai Xing Ng, and Wei Tsang Ovi. Adversarial robustness in two-stage learningtt--defe: Algorithms and guarantees, 2025. URL (tttsss::://artt org/ wwt./oo//////eeeeeeeoo awtpab////// 01027
[60] Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, Neha Gupta, and Sanjiv Kumar. Learning reject for balanced error and beyond. In The Twelfih International Conference Learning Representations, 2024. URL /ttettps://ppeeevvviie
[61] Quang H Nguyen, Duy C Hoang, Juliette Decugis, Saurav Manchanda, Nitesh V Chawla, and Khoa D Doan, Metallm: A high-performant and cost-efficient dynamic framework for wrapping llms. arXiv preprint 2rXi 10834, 2024.
[62] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph Gonzalez, M Walced Kadous, and Ion Stoica. LouteLLM: Learning to route LLMs with preference data, 2024, URL httpps://axii /arxivv org/ abaa 2406. 18665
[63] Michael Poli, Armin W Thomas, Eric Nguyen, Pragaash Ponnusamy, Björ Deiseroth, Kristian Kersting, Taiji Suzuki, Brian Hie, Stefano Ermon, Christopher Ré, et al. Mechanistic design and scaling of hyhrid architectures. arXiv preprint arXiv:2403. 178444 2024.
[64] Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts, 2024. URL httphttts///xx..ooot org/ www.rr///bbaee........oee ab//2
2308 00951
[65] Alec Radford. Karthik Narasimhan, Tim Salimans. Ilya Sutskever, et al. Improving lan- guage understanding by generative pre-training. htttps//ccdi openai. www.ecoo//raeeeeeee www./reeearchcooeeeeeee language- wwtpssuusuuuueesese/ssaesee htwpr//wwweeeeeggggagunueeeeennr paper pdf, 2018.
[66] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)9 2019.
[67] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. ArXiv, 2023.
[68] Mathicu Ravaut, Shafiq Joty, and Nancy F Chen. Summarcranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. arXiv preprint arXiv:2203.06633 2022.
[69] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv2200.005330 2024.
[70] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 11 2019. BtRL/htps::/// //axxx. /rrvvv org/abs/ 1etp//998..111 10084
[71] Marija Šakota. Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-mod arXiv preprini vrXivv2338.06077 2023.
[72] Tal Schuster, Adam Fisch. Jai Gupta, Mostafa Dehghani. Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave. and Kyunghyun Cho, editors, Advances in Newral Information Processing Systems, 2022. URL https //openreview att//forumr..eru.ood.3.........
14

---

[73] Thibault Sellam, Dipanjan Das. and Ankur Parikh, BLEURT: Learning robust metrics for text generation. In Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, 2020.
[74] Avital Shafran, Roci Schuster, Thomas Ristenpart, and Vitaly Shmatikov. Rerouting LLM routers, 2025.
URL ltttps://arxiii org hwwg/abs/2501... 2b0.//....0111 01818
[75] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-cxperts layer. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017 Conference Track Proceedings. OpenReview.net. 2017. URL http:: : http://ppprreeiieeeem net wwwpffortmmiiacc
BickMDqig
[76] Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets. CoRR, 2023,
[77] Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. CoRR, abs/1811.03115 2018. http://:///axti org/abs ftabs//811... 003115
[78] George Stoica, Daniel Bolya, Jakob Bjorerr, Pratik Ramesh, Taylor Heam, and Judy Hoffman. Zipit! merging models from different tasks without training, 2024.
[79] Charles Stone. Consistent nonparametric regression. The annals of statistics, pages 595-6255-0099 620, 1977.
[80] Dimitris Stripelis, Zijian Hu, Jipeng Zhang. Zhanzhuo Xu, Alay Diliphhai Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. TensorOpera router: A multi-madel router for efficient L.M inference, 2024. URL /tttpp:////a org/ aba/2408 12320
[81] Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, et al. Branch-train-mix: Mixing expert Ilms into a mixture-of-experts 1lm. arXiv preprint arXiv:2403.07816 2024.
[82] Mingjie Sun, Yida Yin, Zhiqiu Xu, Zico Kolter, and Zhuang Liu. Idiosyncrasies in large language models. Forty- International Conference on Machine Learning, 2025. URL /ttpss///opeerreeeee net fott/forumidd=FC23 JVzm1
[83] Ziteng Sun, Ananda Theertha Suresh, Jac Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024.
[84] Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. BranchyNet: Fast inference via early exiting from deep neural networks. In 23rd International Conference on Pattern Recognition, ICPR 2016, Cancún, Mexico, December 4-8, 2016, pages 2464-2469, IEEE, 2016.
[85] William R Thompson. On the likelihood that unknown probability exceeds another in view of the evidence two samples. Biometrika, (5(344) 1933.
[86] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anm Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023.
[87] Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. URL http://t/////axi....... org/ www./br/////aaeeeeeeeeeeeeeeeaee
http://aa///..//.//// 1807 03748
[88] Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat cannon? Cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM '24, page 606-615, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400703713,
doi: 10.1 10.1145/3616855 6655.33385.. URI. hhttps:///dl.. org/ 100/10
1145/361 066655666855... 3635825
[89] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang. Rangan Majumder, and Furu Wei. Improving text embeddings with large language models, 2024. holl:attps:/arxvvoorr//////
00368
[90] Xuezhi Wang, Jason Wei, D. Schuurmans, Quoe Le, E. Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. International Conference on Learning Representations. 2022. doi: 10. 00.48550arxii.2223.1111
15

---

[91] Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. Tabi: An efficient multi-level inference system for large language models. In Proceedings of the Eighteenth European Conference on Computer Systems, EuroSys '23, page 233-248, New York, NY. USA, 2023, Association for Computing Machinery. ISBN 978145039487.. doi: 10.1145/3552326.355722 URL httpps/::/// 3587438
1145/35522/32
[92] Mitchell Wortsman, Gabriel Iharco, Samir Ya Gadre, Rehecca Roelofs, Raphael GontjooLLppes Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbliihh, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965-23998. PMLR, 2022.
[93] Jiahao Wu, Wenqi Fan, Jingfan Chen, Shengeai Liu, Qing Li, and Ke Tang. Disentangled contrastive learning for social recommendation. In Proceedings of the 31st ACM International Conference on 2022.
Information & Knowledge Management, pages
[94] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023, URL http:: nips. cttpp..cc/ppaee cc/paper__p wwwes//ppereeeeeeeeeeeeaaaaeeeee httpppaaarrr/////aam 22/3//22222//////// (G444394228877911774444682222222 Abwpaaaoooeetaatttt---ooeeoeeeee
html
[95] Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. CoRR, abs/2310.02575. 2023. doi:
(0.48550/AXXIV23310000277 URL lttps://::///oi org/10 10. 48550/8553513 artpv.riiii.e 2310. 02575
[96] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, T. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Neural Information Processing Systems, 2023. doi: 00.48550/arXiw.2355... 10601.
[97] Weizhe Yuan. Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 11-27263-27777 2021.
[98] Murong Yue, Jie Zhao, Min Zhang. Liang Du, and Ziyu Yao. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. CoRR, 2023.
[99] Tianyi Zhang*, Varsha Kishore Felix Wu?, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020, URL net/forumm httt/forumidd-Skettte Sdw.uuuCCV
[100] Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yigi Wang. Zhen Wen, Fei Wang. Xiangyu Zhao, Jiliang Tang, et al. Recommender systems in the era of large language models (Ilms). IEEF Transactions Knowledge and Data Engineering, 2024.
[101] Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Lo- raRetriever: Input-aware LoRA retrieval and composition for mixed tasks in the wild. In Lun-Wei Ku, An- dre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 4447-4462, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ 1/2024.inndngg acl.263. Uttptt : //ac/ wwwacaannnooooooooooooooooo org/2024 ftl.fiiddi findings ac1.263/ 263/
[102] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Lc, and Ed Chi. Least-to-mo tosstt prompting enables complex reasoning in large language models. arXiv, 2023.
[103] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAulcy, Ke Xu, and Furu Wei. BERT loses patience: Fast and robust inference with carly exit. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 18330-18344 Curran Associates, Inc., 2020. URL https::/ https://prroceedinge wwtpp/awww..p.ppepeeeeeeeeeeeee. wwtppappeea.laeeepeeeeeeeeeeeeee files/ filea/papr////// 2020/file/ 200o0//44A44dd1aa1333332222aeee
pdf
[104] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M. Dai, Zhifeng Chen, Quoe V Le. and James Laudon. Mixture-of-experts with expert choice routing. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advunces in Neural Information Processing Systems, 2022. URL httts::/ mett:/oruum.dddddooHHH.....
[105] Richard Zhuang, Tianhoo Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. Embedllm: Learning compact representations of large language models. arXiv preprint 2410022100.0222 2024.
16

---

Limitations and Broader Impact
This paper proposes a routing framework that improves inference efficiency in large language models (LLMs).. By directing simple queries to smaller models, it cuts computation and memory overhead, lowering costs and environmental impact. Although the framework built for large-scale deployments, could not test very large LLM pools due resource limits. Still, our experiments validate the concept. Future work will examine routing across specialized subnetworks and conditional computation within single L.M.M Moreover, future work answer: How big must model be to recognize a problem's 's difficulty even if it can't solve the problem itself? Future work can study whether size and fine-tuning helps, would RL LLMs makes them able to purely as routers, and whether there is model size when difficulty awareness kicks in.
A Related Work
A.1 Enhancing and Optimizing LLMs
Large language models (LLMs) have demonstrated remarkable capabilities across diverse NLP tasks 666 6]. To further improve their performance and efficiency, numerous strategies have been proposed.
Single-LLM Techniques.
Enhancement approaches targeting individual LLMs include fine- tuning 677, prompting strategies like Chain-of-Thought (CoT) 102
90,, Tree-of-Thoughts [96],
and inference-eee accencc-acellrttionn techniques such as early exiting 88 103
72 and speculative decod-
ing [77 83 8). Additionally, Mixture-of-Expers (MoE) architectures [36 [5] 24 104 38
route inputs through sparse sub-models or "experts", reducing cost while retaining performance. However, these methods typically operate within a single L.LM's 's structure and may not generalize to multi-model scenarios.
Model Fusion and Merging, Fusion strategies synthesize outputs from multiple LLMs to improve output quality [68
39
29
89 52]. Fusion approaches often rely on unsupervised metrics (99 73
or ensemble voting to determine the final output 44] A related but distinct technique is model
merging [50), where weights from multiple pre-trained or fine-tuned models are combined either
directly via methods like weight averaging [92], Task Arithmetic [35], or Fisher merging [56] or
using more sophisticated techniques like TIES 194,, AdaMerging [95], and Ziplt (78).
Cascading. In contrast, cascading invokes models sequentially often ordered by computational cost and halts once satisfactory output is generated [10] balance between quality and efficiency, making them particularly attractive in production settings.
98 10 30) Such approaches strike
A.2 LLM Routing
Routing methods dynamically select the most appropriate LLM from pool for each input, aiming to optimize performance and cost without querying all models. Two primary strategies dominate: non-predictive and predictive routing.
Non-Predictive Routing. Non- Non-prediceive methods generate outputs from one or more models before making selection. FrugalGPT [10] exemplifies this category, using sequential strategy and a response quality threshold minimize cost. Other works adopt layered inference architectures to escalate hard queries more powerful models 191], or leverage cascades with self-verification [54
98
Predictive Routing. contrast, predictive routing aims select the best model before any
inference is performed. Strategies include supervised learning [76), -owuurd--ool--baee routing [32
51, and meta-models trained to predict LLM performance given an input (71. Router models
vary widely in implementation, including neural networks [9
88
k-nearest neighbors R4
76 80 43), matrix factorization [62] 105 48), and graph neural networks 25 Others incorporate
model-specific tokens train across multiple domains [18] 7
17

---

Theoretical Foundations and Robustness. ture, including selective classification [27 60 learning to defer [53], and learning reject [2] 14)
Routing and cascading are grounded in broader litera-
Several works explore supervision levels 51 101 robustness [16 59 74 and evaluation frame-
works for routers 33 32
A.3 Routing Recommendation
Routing can also be framed as a recommendation problem, wherein the input query plays the role of "user", the pool of LLMs corresponds to "items", and past performance metries form the implicit interaction history [100 93]. However, unlike conventional recommender systems, routing has limited "user" features (i.e., input metadata), making label collection and generalization especially
challenging 61 49
Matrix factorization, attention-based models, and graph neural networks used in both recom- menders and routers [62] 105 25], reinforcing the close link between the two domains.
A.4 Scaling Laws and Architecture Trends
Scaling laws 42 47 describe predictable trends between model size, data, and performance, guiding the development of efficient LLM architectures. These insights have been extended MoEs 24 13, sparse models [26]. and hybrid systems [28
63
offering context for when routing or merging
approaches might most beneficial.
A.5 Routing within MoE and Hybrid Architectures
75 where expert modules are dynami- cally activated based on input. While classical MoΣs involved equally-sized sub-models, modern approaches like Switch Transformer 24 and Mixtral 38
Routing is a central mechanism in MoE models 36
employ sparse activation to minimize cost.
Routing LLMs can be viewed as a coarse-graines MoE, where each expert is full LLM. Approaches like UltraFuser (20). Branch-Train Train-MiX [8]) and token-level fusion highlight recent advances in combining model specialization and flexibility.
B Proofs of Theoretical Results
B.1 Notation and Preliminaries
[T<x [1 hm) + Acm] and define the excess Let the oayes-optimal router be r (z) = arg minme[[]] cost-error gap
(=(x) = (0x, hm) + Acm (7(2,, (x))
) + (c)r
Hence (z) 0 and Am (x) 2 0. For query let (2x be the radius of the ball B(x, r) C gd-1 (in the cosine metric) that contains the k-th nearest training neighbor. If the marginal on (())
has density, E (x) < Cjk/n [1 79.
B.2 Proof of Theorem 5.2
Writing g, = q(x;) to lighten notation, decompose
E.
E
(x) (=se)))))NN + (AANC)(())ENN)))) Arteo
(A)
(B)
Term (A): Lipschitz bias. Assumption 5.1 gives 1r(x,) -71x < [(((a) - q(x')122, so
((A) < Lr()). Taking expectations and using [xrx(x)] - (C,k/n) yields E[(A)) S Cak-1/d, 1/d
Term (B): Estimation variance.
Let the empirical cost-adjusted risk be
Am (x)
1
Σ
[T hm) + Acm]
T (EN((2)
18

---

Hoeffding's 's inequality bounds
(1) - -E|åm (x) 21 - 2e 2kx²
and union bound over m S M plus integration gives
E mmxx (x) (1) xi]] S C3v log M m
Because (x) minimizes (x), (B) & 2 maxx (x) E E|Âm (2) x]]- Combining with (A) implies
(fR) RA(r*) ((Vk//+
B.3
Proof of Lemma
For convenience write Sim V Cm/ The when E Bg. The single-query loss (Equation X
is
Emepa exp(Sim)
log
KC
2m²=1 exp(Sim' 7Cmm
kE
Let
Pim = exp(Sim)) Σ exp(Sii)
JEPi
when E Pik and
M Jim exp(Sim 7cm)/ Σ
exp(S. 770;)
Then
y R. Σ log Σ
Pim/ Tim
mEPsk
Taking the gradient w.r.t. 9s and setting it to zero gives Em (pim - qim em = 0. Project onto q:
Em (pim Sim) = 0. Fix m Pik. m E Ni) with Cma Cm- Because Pim = while Pimtt 0, the equality forces qin > qim-, hence Sint -ymm > S.m- 7mm- Rearranging
yields q, em > q, em establishing directional alignment.
C Method
C.1
Logit-Footprint Descriptors
Why take the most frequent tokens? We the most frequent tokens the basis is shared and stable: they appear in all models, give low-noise estimates with few probes, and make calibration comparable across experts. They give less noisy estimates because they get non-negligible probability tokens. In all experiments, we set K 256 and T 10 (D. which is large enough that the basis isn dominated by few function words.
across many contexts, so their averaged log-probs vary less than rare or
Are frequent tokens trivial? These tokens aren't used for their meaning. They 're probes of each model's 's output behavior. Even common words get scored differently across models (temperature, punctuation/number handling, style). By averaging over many prompts and steps, the descriptor captures overall model behavior, not any single word's 's semantics. Also, a recent work [82) shows that LLMs exhibit stable, word-level idiosyncrasies (as the authors call them) that enable near- perfect model attribution using only the first few generated tokens (even after paraphrasing or translation), implying that common tokens still provide discriminative signals about a model's predictive calibration.
19

---

Shared token set vs. per-model selection. A shared basis gives all descriptors a common coordi- nate system. If each model used a different token set, cosine distances would mix basis changes with true behavior, hurting comparability. It would also require computing many more probes to align per-model bases that are different across models.
Beyond raw frequency. We deliberately kept the descriptors simple to isolate and quantify the contrastive router's contribution. Nonetheless, frequency is pragmatic, not necessarily optimal, choice. Two variants that we considered and could be explored are:
-Te-Dhe-weighted selection over the probe corpus.
Picking tokens with the largest across-model log-prob variance.
These can be dropped into Equation 2 without changing downstream training or inference.
C.2 The Step from Equation to Equation
Equation (3) defines a per-prompt token NLL that requires access to expert's next-token distribution Pa(-1 I). For API-only (black-box) experts, logits/rrobabilities are not exposed, so Equation (3) is computable. Our remedy is to (a) let the API expert h produce deterministic continuation ýn (2)) for prompt x (greedy decoding), and (b) evaluate that sequence under a single shared, public scorer ps (kept fixed across all experts). This yields Equation (5),
(x))
(x( = Uh (x)
Σ
logps((n.. (0,<st))
which we then normalize (similar Equation (4)) and use as the fingerprint coordinate(s) for black- box experts.
If ün. is typical (high-probability) output of h (i..., Dn ph)) then averaging the pseudo perppee-eyy (h,<s) over many prompts/tokens is equivalent taking
(x) [z]] Σ, log (ūhh log expectation over y p>h
Eyy-p. logps(u)] = Hph, = H(p)) + KT(pA || ps).
Here (ph. ps) is the cross-entropy of ps with respect to ps. which decomposes into the entropy of h's own distribution H(p)) and its divergence from the scorer (L(pxllps).
Because ps is fixed for all experts, Hyn, ps) ) stable, model-specific quantity that makes descrip- tors comparable across experts ("same yardstick"), " It is not the true NLL under pa. but it preserves differences between experts via H (pa) and their mismatch to ps via KL(p), plps). In practice we use deterministic (greedy) decoding reduce variance, averaging over many prompts/tokens makes the empirical &, (x) closely track the expectation above. Figure empirically validates this argument.
C.3 Band-Specific Temperatures and Smoother Gradients
Most prompts in everyday interactions (and in our datasets) can be handled by cheaper models; plus there are usually fewer very expensive experts overall. These expensive experts are only needed for small fraction of hard prompts, so within those high-cost bands there are fewer suitable positives per query. With few positives, the similarity distribution becomes very peaky.
larger Tæ in Equation flattens the per-band softmax, reducing gradient variance and preventing the update from collapsing onto single rare positive. Formally, for band k the per-query gradient w.r.t. the query embedding
M
V JgLk = Σ
Cm
+ m
Pmn
Cm'
(10)
Tk
mEPa
ml
where p(+) and p(-) are the band-restricted softmaxes over positives and all experts (with the negative cost penalty in the denominator). As increases, both softmaxes become less concentrated, so (i)
20

---

the gradient magnitude scales like 1/TT and (ii) its direction is averaged over more positives, lowering variance across minibatches.
This is the sense in which band-specific temperatures yield smoother gradients. It is especially helpful in high-cost bands that otherwise have few positives and highly variable similarities. Without band-specific temperatures, the router can exhibit oscillatory updates on hard prompts (rare positives dominate, then vanish), slowing convergence and encouraging over-use of cheap experts. Empirically, we observed that bands and band-specific temperatures are important (Table
C.4 Dense Human Annotations
We considered using human annotations but intentionally avoided them: model pools change quickly, so adding/replacing experts would require fresh labels that are costly and often unavailable. Instead, we train with sparse correctness and cost signals, which remain portable across experts. If dense feedback is available, it could help in several ways:
Positive sets P(i) with preference structure. Replace binary "correct expert" labels with pairwise preferences (cheap-and-good > expensive-and-simiin > clearly wrong), yielding band-aware positives and margin constraints. This can be implemented by expanding P(i) and adding a lightweight pairwise ranking (DPO-style) regularizer within each cost band.
Difficulty-awan reweighting. Use human "hardness" scores upweight rare/hard prompts when computing the contrastive loss, especially in higher cost bands. this could balance the effective sample sizes across easy vs. hard (and cheap vs. expensive band) cases so the gradient isn't 't dominated by the abundant, easy examples.
Band calibration. We can ask users how much quality they're re willing to trade for lower cost, then use that to set the cost bands and the penalty for picking more expensive models, so the router's choices match what users actually prefer.
D Experiments
D.I Experimental Settings
Baselines We compare our proposed method against comprehensive set of baselines designed to capture key routing strategies and their trade-offs. Specifically, we include UMR [40], a recent state-of-the-arr technique that clusters prompt embeddings to route queries L.MM pools efficiently; Thompson Sampling [48] 21, which frames routing as a bandit exploration-exploiiee
problem to balance cost and accuracy dynamically; Pareto-optimal routing [34], a strategy that selects models by explicitly considering the cost-accuracy Pareto frontier; and two extreme Maselines-Randone which selects models uniformly at random to represent naive routing without intelligent selection, and Oracle (Clairvoyant Upper-Boune Bound 440), which always selects the most accurate model at the lowest possible cost and thus represents theoretical performance ceiling. Additionally, we evaluate against parametric gating methods (Parametric Softmax Router) inspired by classical mixture- of-experts architectures [62] and SoftMoE, which models router decisions via differentiable soft gating functions (64). Collectively, these baselines enable us to rigorously assess whether our contrastive routing approach delivers meaningful improvements in performance, cost-efficiency, and generalization capabilities relative existing strategies.
Datasets, Benchmarks, and Evaluation We train our router and evaluate it on three datasets: EmbedLLM [105), MixInstruct (39), and RouterBench [34]. For EmbedLLM and MixInstruct, we sample 192 probes from their respective validation sets. Each probe is processed to extract logit-based descriptors by capturing the top K 256 tokens over a horizon of T = 10 tokens (Equation (2). resulting in 2.6-dimensional vector per model. For RouterBench, we sample 192 probes from its training set, ensuring these probes are excluded from the training data used for the contrastive router. We compute perplexity-basee descriptors on RouterBench and use GPT-2 (66). On both EmbedL.M, and RouterBench, we use binary accuracy as the per-sample evaluation metric, meaning an LLM response is classified strictly as correct or incorrect. For MixInstruct, we employ exponentiated BARTScore [97] the evaluation metric, following the approach in prior work 44 (3).
21

---

Table 6: Full breakdown of training and testing categories used in OOD experiments.
Set
Categories
mat cot_ shht, apqummiia zeroshot gpga_miin_ shoo geea main, zerosbor goqr, diamond shot,, apgp_ digppddamoddddoo cot scroshot, gpqa, dppaoiimoeddd_.mm puo. gesu diamond zerochot, spea extended_ cot n sushet, gpe eegeeeeeededd.cm cut pecoshot,
asdiv, gamsk, medineqo, marhga, piga, legign, gpgumain
extended n shot, gpea exiended zeroshot, munly vollege medicine. mill, astrenomy, mmlu, comoppuual physics, mmlu, college computer science mmlaa college biology, mmlu, electrcall engineering. mmlu, medical genetics, mmlu, moalu_oollggee college pavaee physics, mmlu migh_hight school chemistry, mmia computer securty, minla clinical knowledge, c
Train (STEM)
mmlla virology, mmlu machine Jearning, mmia college mathemmiicss mmkk, elementary. dicneetary_maaaaaeee matbematics, munka, professional, medicine, mmlu college chemistry mulu, high school, biology, mmiu, anatomy, midu high school statisics, mmlu high schhoochollll physics. mmbu high school computer scienc, mmlu high school muthematicss
social, iqa, truthfulga mel, mmlt high school cumpeun, history, minla uX foreign, polayy mmlu, high school micreceeonics mmlu basiness othics mmltu public relations, mmlu, jurispradence, mmit, nutrition. mmbo high school world history, mmmla miscellaneves, mmlu farmal logic, mmlu management, mmlo high school psychology. politis,, mmlu high, school geogaphy, mmla world religions, mmsha intemutional muila, human, agingg usala, saciolagy, mmlu, professional accounting, mmlu prehistory, miin, logical, fallicies, mmlu, psoral disputes, mmlu, human sexuality, mmlu, professional Laspoaal_pyeee psychology, mmlu, _ school us history, mmlu, high. school maccorconmmiss, mmln cherne algohra, mmln glohal ficts mmll security sndies, mmll philoooply, mmlu professional law, mralu, moral semarios, mmlu marketing
Test (Non-STEM)
mmlu high, school government
We evaluate each routing strategy using a deferral curve 440 which plots the average response quality against the total inference cost. Sweeping the routing penalty parameter À over the interval A E [0, mmax] (refer to Equation (9) traces the deferral curve. For the EmbedI.LM and MixInstruct datasets, we define the cost of processing prompt as the number of parameters in the LLM, serving as a proxy for computational resources and latency. In the case of RouterBench, we utilize the actual API call costs in USD, provided in the dataset. Following [40] we employ evaluation metrics including Area Under the Deferral Curve (AUDC), Query-Normalized Cost (QNC), and peak accuracy. QNC is the minimum relative cost required match the performance of the most accurate tested LLM.
Training We use a frozen bedding backbone (bxz) in Section3) across all experiments. Our trainable router component is a two-layer MLP, denoted as ge(.), which projects prompt embeddings into the expert descriptor space. We train our contrastive router on the training splits of each dataset, excluding the probe examples from RouterBench. Training is performed for 10 epochs using the Adam W optimizer with a batch size of 512 and a learning rate of 5 x 10-4, For the cost spectrum loss (Equation (8). we set the number of cost bands to K = 5 and the negative cost penalty to À = 0.1. The hyperparameters for the linear schedule of band-specific temperatures (Equation 77) are set as a = 0.25 and Tmin = 0.05. All training and descriptor extraction are done on dTX6000Ada GPUs with 48GB GPU memory.
[70] model as the em-
D.2 Results
D.2.1 Out-of-Distribution Prompt Experiments
In the out-of-distribution (OOD) experiments, we divided the prompts in the EmbedLLM dataset into two challenging sets based their categories: STEM-related (Science, Technology. Engineering, and Mathematics) and Non-STEMM-reltted (covering Social sciences, Humanities, Arts, etc.). A detailed summary of the train and test categories for the OOD experiments is provided in Table 6 Our splitting yielded 18,193 193 out of 36,054 total training questions and 1, 1,060 distinct test prompts. Training and testing is done all available LLMs across both splits.
D.3 Ablation of Cost Bands
We measured AUCD and peak accuracy with and without bands. Table 7 presents these results, showing having cost bands is indeed effective.
Setting
AUCD
Peak Accuracy
Without bands With bands
0.4574 0.4951
0.482
0.540
Table 7: Effect of banded cost temperatures on model performance
22

---

Router (Bands)
AUDC f
QNC
Peak
0.55
bands:2
0.5357
47.693
0.563
bands:5
0.5480
66.057
0.590
0.50
bands:
0,5173
30.412
0.547
ALCUszYy
0:45
bands:15
0.4460
13,474
0,496
bands:20
0.4649
11.541
0.477
0.40
Table 8: Deferral-curve metrics across different
0.35
numbers of cost bands.
0.80
10
20
30
40
50
80
Params IB)
Figure 7: Deferral-curve metries across different numbers of cost bands.
K
AUDC
cost max acc
0.5186
7.665
2
0.5294
7.759
0.5344
7.831
8
0.5378
7.940
0.5402
8.034
Table 9: Effect of Number of Nearest Neighbors (K) on AUDC and Average Cost
D.3.1 Ablation of Number of Cost bands
Table 8 and Figure 7 show that performance varies with the number of cost bands. Using 5 bands yields the best results, with the highest AUDC (0.5480) and Peak accuracy (0.590), suggesting good balance between flcxibilityy and generalization. Fewer bands (c.g.. 2) limit routing precision, while many bands (c.g., 15 or 20) degrade performance, likely due to wre-frragmentaioon and increased decision noise. This highlights the importance of tuning the number of bands to avoid both under- and overfitting
D.3.2 Ablation of the Number of Neighbors
Increasing h (the number of ANN neighbors selected before cost-aware scoring) generally provides modest improvements before reaching a plateau. (The baseline UMR [40] also included an ablation on k in a K-NN router.) Beyond small K, the gains in accuracy AUDC become minimal, while both latency and the likelihood of choosing unnecessarily expensive experts increase. We selected and worked reasonably well did not do further hyperparameter tuning. We
default of K performed an ablation on a subset embedl 1m prompts. The results are shown in Table 9 which are consistent with the trend observed in UMR.
D.3.3 Ablation of Negative Cost Penalty
is soft deterrent against assigning probability mass to costly, wrong experts during training (it appears in the denominator of the band-softmax in the loss in Equation 8 We ablate this hyperparameter in Table TO too small, the router learns to over-consider expensive hub experts (cost creep). too large, it over- ver-penalizes cost and undertrains on valid high-cost positives, hurting hard prompts. We picked 0.2 as it's light regularizer: enough to push apart costly negatives first, but not strong that drowns the similarity signal for truly necessary expensive experts.
D.3.4 Ablation of Band-Specific Temperature Slope
In the band-specific temperature schedule = Tmin + č. (Fquation 7 the slope sets how much flatter the softmax higher-cost bands. Increasing a raises тę for expensive bands, which flattens their per-band softmax over experts. This reduces gradient variance in those bands (where each query
23

---

Router
AUDC
cost max
y=0
0.5518
0.5980
55.543
7=0.1
0.5566
0.5850
44,499
7=0.2
0.5526
0.5720
43.083
7-3.3
0.5272
0.5307
14,901
0.5
0.5129
0.5147
8.836
Table 10: Gamma ablation
AUDC
max_acc
cost max_ace
0
0.5567
0.5930
64.292
0.1
0.5704
0.6033
55.680
0.25
0.5701
0.6003
48.360
0.4
0.5557
0.5657
32.854
0.5
0.5448
0.5517
29.175
Table 11: Ablation over the band slope in т Tmin + a Cg. Moderate a (0.1-0.25) maximizes
AUDC and lowers the cost required to reach peak accuracy. Large oversmooths high-cost bands and reduces peak accuracy.
has fewer suitable positive) mitigating collapse onto single rare positive and preventing cost-creep during training.
See Table 11 for an ablation of this hyperparameter. As a increases from 0 to a moderate value (0.1-0.25), AUDC improves and the max drops, indicating we reach peak quality lower cost, while max_acc remains comparable. For larger a (2 0.4), the high-cost bands become oversmoothed, weakening discrimination among expensive experts max_ _ac and AUDC decline despite further cost reductions. This validates our choice of adopting moderate setting (default a-0.25, with 0.1 performing slightly better).
D.3.5 Ablation of Cheapest Cost Band
Tmin is the softmax temperature for the cheapest cost band in Equation 7
All other bands inherit Tk Tmin Cg. A very small Tmin makes the cheap-band softmax sharp (highly discriminative but prone to noisy, peaky gradients) while a larger Tiin smooths the distribution, lowering variance but also blurring differences among cheap experts.
Table 12 ablates Tmin- Raising Tiin from to 0.02 increases AUDC and peak accuracy, showing that touch of smoothing stabilizes learning without hurting discrimination. At Tmin 0.05 keep nearly the same AUDC while cutting the cost needed to achieve peak accuracy by ss 14% (from 51.7 44.6). Pushing to Fmin 0.08 oversmoothes the cheap band: accuracy at low cost rises slightly, but max_aat acc and AUDC both fall. Thus a moderate setting (Tmin RJ 0.02-0.0) offers the best efficiency-stability trade-off.
D.4 Larger Encoder Size
We kept the router is deliberately small: a frozen centence-transformer plus a 2-layer MLP. We performed ablation where increased the dimension of the middle layer in Table 13 As the router size increases, AUDC improves, but latency also increases.
Tmin
AUDC
max acc
cost @ acc
0
0.5484
0.5877
51.232
0.02
0.5602
0.6037
51.704
0.05
0.5581
0.5883
44.633
0.08
0.5515
0.5707
38.395
Table 12: Ablation the base temperature Tiii (with 0.25). Moderate values improve AUDC and lower the cost required for peak or near-peak accuracy. Very small or large values underperform.

---

Router
AUDC
MLP-x1
0.5189 0.5384
MLP-x4
Table 13: Effect of Router Size on AUDC
Band Index
Count
0
533
1
837
2
805
3
694
4
131
Table 14: Prompt Counts by Band Index
Broadly, future work can answer: How big must model be to recognize a problem's difficulty even it can't solve the problem itself? Future work can also study whether size and fine-tuning helps and whether RL on LLMs makes them able to purely as routers? Future work can also study different model sizes to pinpoint when difficulty awareness kicks in.
D.5 Qualitative Insights and Interpretability of Routing
We performed analysis on embedllm, 1m, which features large pool of models with diverse costs. Note that these observed trends are specific the dataset and may not generalize other datasets or prompt pools with more challenging examples.
D.5.1 Selection Profiles
Qwen/Qwen1 .............00 5B-Chat (selected 91 times), progle/gemm--2uer (233), and microsoft/phi-2 (292) are smaller experts that are selected frequently. Some expensive experts (e.g., Qwen/Qwen-725,
are rarely chosen, since
less costly correct expert typically exists in the dataset.
We also report per-expert selection rates by band in Table 14
D.5.2 Routing Error Breakdown
We present a confusion-style breakdown of routing errors in Table 15
Too cheap-the router selects a cheap but incorrect expert
Too expensive-the router selects an unnecessarily costly expert, though correct
Optimal-the the router selects a correct and minimally costly expert
No correct-no available expert produces a correct answer
Outcome
Count
Too cheap
235
Too expensive
586
Optimal
773
No correct
63
Table 15: Routing Outcome Breakdown
25

---

Dataset
A AUDC
95% CT
p (bootstrap)
n10/no1
p (McNemar)
EMBEDLLM
+0.053
[0.037, 0.069]
2.0 10T4
6.08 560/366
9.76 x 10-11
Table 16: Paired significance versus the strongest baseline. A AUDC = AUDCeScR AUDCUM (area under the deferral curve. Higher is better). "95% CT and the one-sided come from paired bootstrap over prompts (NV 3000, = 5000, H1 : A > 0). c* is the matched budget used for McNemar. Rio/nno noo are discordant counts (CSCR correct baseline correct), and "p (McNemar)" is the one-sided exact binomial p for CSCR > baseline at ct. CIs that exclude 0 and small p- indicate a statistically significant improvement of CSCR.
D.6 Statistical Significance
We perform full evaluation on Embedlim [055]
using paired, prompt-level significance tests to concretely assess statistical significance. Specifically, computed a paired bootstrap [22] (sampling 3,000 prompts with replacement, 5,000 times) to obtain a 95% confidence interval for AAUDC AUDCescR AUDCUMR (UMR is the best baseline overall). We also report one-sided p-value for the hypothesis > O. Additionally, we ran McNemar's 's [57] test at a matched budget (using the median of the combined cost grids) to compare per-pon per-prompt wins and losses at equivalent operating cost. These tests quantify uncertainty over the test prompts.
AAUDC is positive with CIs that exclude zero, and McNemar shows win rates above 0.5 with very small p-values the matched budget. In other words: CSCR's deferral curve encloses more area (higher accuracy at the same lower cost on average), and at fixed budget it wins more prompts than it loses. This complements the Pareto-fronticr plots: the gains are not an artifact of a single operating point or random variation, but hold paired, prompt by prompt. The paired test establishes statistical significance.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide theoretical analysis in Section and empirical results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Provided in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
We provide a theoretical analysis with proof' and assumptions in Section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experimental details are provided in Section and D.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification: All datasets used are open source. We will provide the code for our experiments after paper decision is available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssicnlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental details are provided in Section and D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We could not provide eтror bars deviation due to computational costs. No other prior work and baselines in our work do this either.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experimental details are provided in Section and information.
D.1
They include this
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
There are not ethical concerns that we know of.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: discussion of broader impact provided in Section 7 Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All original owners of assets have been properly cited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
32

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
33


Reviews:
[{"review_id": "jdwzgLMjWM", "reviewer": "Reviewer_7VYj", "summary": "This paper introduces Cost-Spectrum Contrastive Routing (CSCR), a framework for cost-aware routing across pools of large language models (LLMs). ​ CSCR addresses limitations in existing routing methods, such as reliance on expensive profiling, inefficiency, and lack of adaptability to dynamic model pools.", "questions": "The paper is quite strong and very well-written. The related work section is extensive, crisp and very detailed. I have a few questions that I have provided below:\n\n1. In equation 2, what is the logic of taking the K most frequent tokens? If K is not too large, then they will most likely correspond to trivial tokens such as a, an, the etc - isn't that correct? Should we use something more like tf-idf to select the tokens? Or do you think that not having a common set of tokens across the models will lead to an embedding that will not work. Can you please clear up this confusion?\n\n2. How can dense human annotations further help the algorithm? In some cases, for a small number of datapoints, detailed human feedback might be available - how can it be further helpful? \n\n3. Please expand on L167 further - why do smoother gradients help and how do they results from assigning band-specific temperatures? It is not clear to me why gradients are smoother because of the clustering. What happens if the gradients are not smooth for the hard examples?", "limitations": "No limitations to the best of my understanding", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths: The paper is exceptionally well-written with abundant explanations and intuitions guiding the reader. The problem is clearly stated and intuitions behind the solutions and its advantages are clearly highlighted\n\nNo such weaknesses", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the careful read, the positive assessment, and the great questions. We address the questions below and will incorporate the suggested clarifications into the paper.\n***\n# 1 why take the $K$ most frequent tokens? Will they be trivial? Should we use TF-IDF instead?\n\nIn short, we use the most frequent tokens so the basis is shared and stable: they appear in all models, give low-noise estimates with few probes, and make calibration comparable across experts.  They give less noisy estimates because they get non-negligible probability across many contexts, so their averaged log-probs vary less than rare or OOV tokens. In all experiments, we set $K=256$ and $T=10$ (Appendix C1), which is large enough that the basis isn’t dominated by a few function words. We will add a short paragraph to the paper stating this rationale explicitly.\n\n## 1.1 Are frequent tokens \"trivial\"?\nThese tokens aren’t used for their meaning. They’re probes of each model’s output behavior. Even common words get scored differently across models (temperature, punctuation/number handling, style). By averaging over many prompts and steps, the descriptor captures overall model behavior, not any single word’s semantics. Also, a recent work shows that LLMs exhibit stable, word-level *idiosyncrasies* (as the authors call them) that enable near-perfect model attribution using only the first few generated tokens (even after paraphrasing or translation), implying that common tokens still provide discriminative signals about a model’s predictive calibration [1]. \n\n## 1.2 Shared token set vs. per-model selection.\nA shared basis gives all descriptors a common coordinate system. If each model used a different token set, cosine distances would mix basis changes with true behavior, hurting comparability. It would also require computing many more probes to align per-model bases that are different across models. \n\n## 1.3. Beyond raw frequency.\nWe deliberately kept the descriptors simple to isolate and quantify the contrastive router’s contribution. We agree that frequency is a pragmatic, not necessarily optimal, choice. Two variants that we considered and could be explored are: (i) TF–IDF-weighted selection over the probe corpus, just like the reviewer suggested. (ii) picking tokens with the largest across-model log-prob variance. These can be dropped into Eq. (2) without changing downstream training or inference. \n\nIf the reviewer believes that ablating this choice would strengthen the paper, we are happy to include it.\n\n# 2. How could dense human annotations help?\nGreat question. We considered using human annotations but intentionally avoided them: model pools change quickly, so adding/replacing experts would require fresh labels that are costly and often unavailable. Instead, we train with sparse correctness and cost signals, which remain portable across experts. If dense feedback is available, it could help in several ways:\n* **Positive sets $P(i)$ with preference structure.** Replace binary “correct expert” labels with pairwise preferences (cheap‐and‐good $\\\\succ$ expensive‐and‐similar $\\\\succ$ clearly wrong), yielding band-aware positives and margin constraints. This can be implemented by expanding $P(i)$ and adding a lightweight pairwise ranking (DPO-style) regularizer within each cost band.\n* **Difficulty-aware reweighting.** Use human “difficulty” scores to upweight rare/hard prompts when computing the contrastive loss, especially in higher cost bands. this could balance the effective sample sizes across easy vs. hard (and cheap vs. expensive band) cases so the gradient isn’t dominated by the abundant, easy examples.\n* **Band calibration.** We can ask users how much quality they’re willing to trade for a lower cost, then use that to set the cost bands and the penalty for picking more expensive models, so the router’s choices match what users actually prefer.\n\nWe will add a brief paragraph in Appendix describing these straightforward extensions.\n\n\n# 3 Expanding L167: why do band-specific temperatures yield smoother gradients? What if gradients are not smooth on hard examples?\nThank you for catching this. We originally included a full explanation in the paper (lines 183-186), but it was accidentally removed during our edits for space and wasn’t moved to the appendix.\n\nMost prompts in everyday interactions (and in our datasets) can be handled by cheaper models; plus there are usually fewer very expensive experts overall. These expensive experts are only needed for a small fraction of hard prompts, so within those high-cost bands there are fewer suitable positives per query.  With few positives, the similarity distribution becomes very peaky.\n\n*Effect on gradients:* Increasing $\\tau_k$ makes the softmax less concentrated, so (i) gradient magnitudes shrink roughly like  $1/\\tau_k$ and (ii) directions average over more positives, reducing variance and preventing updates from collapsing onto a single rare positive, especially in the high-cost bands.\n\n*If gradients are not smooth for hard examples.* Without band-specific temperatures, we observed the router can have unstable training (possibly due to oscillatory updates on hard prompts) and slower convergence. It also performs worse (please see the additional ablations in our response to Reviewer PkxW).\n\n# 4. References\n[1] Idiosyncrasies in Large Language Models, Sun et al., ICML 2025\n\n***\nThis concludes our response to the reviewer’s question, but we invite them to read section 3 of our response to Reviewer  PkxW for additional ablation experiments including ablations on cost spectrum hyperparameters.  We hope these clarifications address the reviewer’s questions. We truly appreciate the insightful questions as they help increase the clarity of our work."}, {"review_id": "suaSzWX9AM", "reviewer": "Reviewer_PkxW", "summary": "This paper proposes cost-spectrum contrastive routing (CSCR), which improves inference efficiency by routing simpler queries to smaller models, while maintaining comparable accuracy. CSCR first involves mapping every LLM to a model fingerprint vector (either logit-based for open LMs, or perplexity-based for API-only LLMs). Second, a contrastive router is trained to learn a shared embedding space where query vectors are pushed closer to right-sized experts, using a modification of the InfoNCE loss. At inference time, a given prompt is routed to the most cost-effective expert. CSCR is a lightweight, 2-layer MLP and is tested on the baselines EmbedLLM, MixInstruct, and RouterBench, achieving up to 25% higher accuracy-cost efficiency.", "questions": "Questions:\n- I am not sure if I’ve missed this anywhere, but is there any ablation of the k used in Equation 9? In general, I'm curious about the selection of hyper-parameters and further ablation studies would be helpful. \n- What happens if you ablate the encoder used to learn the shared embedding space? Would a larger encoder improve performance? \n\nTypos:\n- L34: Routing → routing\n- L148: classical -> classic", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- This is a very relevant and timely problem; as LM inference with many different options and pricing schemas is very common these days, so figuring out the optimal tradeoff between cost and accuracy is important. \n- Method relies on a small, efficient encoder and seems efficient at run time.\n- The authors introduce a new InfoNCE loss and ablate it in 4.3.2, finding it to be more cost-effective.  \n\nWeaknesses:\n- Are the main results (Tables 1, 2, 3) statistically significant? I know that the authors cited computational cost as why they do not include statistical significance (which is not a good reason IMO given that even statistical estimation is typically not prohibitive), but some reported metrics (e.g., AUDC) look very close together, which makes me hesitant to draw any definitive conclusions. Further, this is a little confusing to me since the low cost and efficiency seems to be the main advantage of the method. \n- I wonder if there are any qualitative insights to be made about the router decisions (i.e., if interpretability could be a potential advantage). For instance, are there any LMs that punch above their weight class and tend to be chosen more often despite being cheaper to serve? Similarly, are there any more expensive LMs that are almost never selected? \n- Writing is unclear and confusing in many places, and the manuscript may benefit from a more thorough proofreading. In particular, the introduction has lots of gratuitous bolding and italics, which may be distracting to the reader.", "was_revised": true, "final_justification": "As stated below, I've elected to keep my score. The authors have responded to all my points sufficiently and I think this paper would be a good addition to the venue.", "rebuttal": "We sincerely thank the reviewer for the positive assessment, the thoughtful feedback and actionable suggestions.\n***\n# 1. On statistical significance \nOur router consistently outperforms all baselines: AUDC is higher on every dataset, and the deferral curves align with or extend the empirical accuracy–cost Pareto frontier, rather than simply trading wins and losses across different regimes. Furthermore, AUDC is an average metric across the full deferral curve, not just a single accuracy point on a split dataset. If the reviewer is referencing a specific result, could they please clarify which values?\n\nWe had considered running robustness experiments with different probe sets to report mean and standard deviation, but this would require repeating each experiment at least three times, which is computationally prohibitive. That is what we referred to in the checklist.\n\nAs requested by the reviewer, we re-ran the full evaluation on EmbedLLM using paired, prompt-level significance tests to concretely assess statistical significance. Specifically, we computed a paired bootstrap [1] to obtain a $95\\\\%$ confidence interval for \n$\\Delta_{AUDC}$ (Difference between AUDC of CSCR and AUDC of UMR as the best baseline overall). \nWe also report a one-sided p-value for the hypothesis $\\Delta > 0$. Additionally, we ran McNemar’s test [2] at a matched budget (using the median of the combined cost grids) to compare per-prompt wins and losses at equivalent operating cost. These tests quantify uncertainty over the test prompts, which we believe matches the reviewer’s intended notion of significance.\n\n|Dataset|$\\Delta_{AUDC}$|$95\\\\%$ CI|$p (\\text{bootstrap})$|$c^\\star$ | $n_{10} \\/ n_{01}$|$p (\\text{McNemar})$|\n|-|-|-|-|-|-|-|\n| EmbedLLM|+0.053| [0.037,0.069] |2.0×10⁻⁴|6.08|560/366|9.76×10⁻¹¹|\n\n $95\\\\%$ Confidence Interval and the one-sided $p$ come from a paired bootstrap over prompts ($N=3000$, $B=5000$; $H_1 : \\Delta > 0$).  $c^\\star$ is the matched budget used for McNemar. \n$n_{10}/n_{01}$ are discordant counts (CSCR correct / UMR correct), and “$p$ (McNemar)” is the one-sided exact binomial $p$ for CSCR $>$ baseline at $c^\\star$. \n\n$\\Delta_{AUDC}$ is positive with CIs that exclude zero, and McNemar shows win rates above 0.5 with very small p-values at the matched budget. In other words: CSCR’s deferral curve encloses more area (higher accuracy at the same or lower cost on average), and at a fixed budget it wins on more prompts than it loses. This complements the Pareto-frontier plots: the gains are not an artifact of a single operating point or random variation, but hold paired, prompt by prompt. The establishes statistical significance.\n\n# 2. Writing clarity\nWe will remove the bold/italics in the introduction and other sections, clarify writing, and do another full proofread. We appreciate the pointers and have fixed the two typos the reviewer noted.\n\n# 3. More Ablations\nMost of our choices for the contrastive loss were based on our past experience training contrastive models. We conducted additional ablation studies during the rebuttal period as per the reviewer's request.\n\n## 3.1 $K$ in Eq. (9)\nWe selected a default of $K=4$  and it worked reasonably well so we did not do further hyperparameter tuning. However, we agree with the reviewer that an ablation would be useful, so we performed one on a subset of EmbedLLM prompts.\n\n|K|AUDC|cost@max_acc|\n|-|-|-|\n|1|0.5186|7.665|\n|2|0.5294|7.759|\n|4|0.5344|7.831|\n|8|0.5378|7.940|\n|16|0.5402|8.034|\n\nIncreasing $K$ generally provides modest improvements before reaching a plateau. Beyond a small $K$, the gains in accuracy or AUDC become minimal, while both latency and the likelihood of choosing unnecessarily expensive experts increase. The baseline UMR also included an ablation on $K$ in a K-NN router. The trends are similar.\n\n## 3.2 Banded Costs\nWe measured AUCD and peak accuracy with and without bands to validate having banded cost.\n\n|Setting|AUCD|Peak Accuracy|\n|-|-|-|\n|W/ bands|0.4574|0.482|\n|W/O bands|0.4951|0.540|\n\n## 3.3  $\\gamma$ in Eq. 8\n$\\gamma$ is a soft deterrent against assigning probability mass to costly, wrong experts during training (the denominator of the band-softmax in Eq. 8). If too small, the router learns to over-consider expensive hub experts. If too large, it over-penalizes cost and undertrains on valid high-cost positives, hurting hard prompts. We picked 0.2 as it’s a light regularizer: enough to push apart costly negatives first, but not so strong that it drowns the similarity signal for truly necessary expensive experts. \n\n| $\\gamma$| AUDC   | max_acc | cost@max_acc |\n|--|-|-|-|\n|0| 0.5518 | 0.5980  | 55.543  |\n| 0.1| 0.5566 | 0.5850  | 44.499 |\n| 0.2 | 0.5526 | 0.5720  | 43.083 |\n| 0.3 | 0.5272 | 0.5307  | 14.901 |\n|0.5| 0.5129 | 0.5147  | 8.836  |\n\n## 3.4 $\\alpha$\n\nIn the band-specific temperature schedule $\\tau_k=\\tau_{\\min}+\\alpha\\,\\bar c_k$, the slope $\\alpha$ sets how much flatter the softmax is in higher-cost bands. Increasing $\\alpha$ raises $\\tau_k$ for expensive bands, which flattens their per-band softmax over experts. This reduces gradient variance in those bands (where each query has fewer suitable positive) mitigating collapse onto a single rare positive and preventing cost-creep during training.\n\n| $\\alpha$ | AUDC   | max_acc | cost@max_acc |\n|--|-|--|--|\n| 0     | 0.5567 | 0.5930  | 64.292       |\n| 0.1   | 0.5704 | 0.6033  | 55.680       |\n| 0.25  | 0.5701 | 0.6003  | 48.360       |\n| 0.4   | 0.5557 | 0.5657  | 32.854       |\n| 0.5   | 0.5448 | 0.5517  | 29.175       |\n\nAs $\\alpha$ increases from $0$ to a moderate value ($0.1$–$0.25$), *AUDC* improves and the *cost@max\\_acc* drops, indicating we reach peak quality at lower cost, while *max\\_acc* remains comparable. For larger $(\\alpha \\ge 0.4$), the high-cost bands become oversmoothed, weakening discrimination among expensive experts; *max\\_acc* and *AUDC* decline despite further cost reductions. This validates our choice of adopting a moderate setting (default $\\alpha=0.25$, with $0.1$ performing slightly better).\n\n## 3.5 $\\tau_{\\text{min}}$\n$\\tau_{\\min}$ is the softmax temperature for the cheapest cost band; all other bands inherit $\\tau_k=\\tau_{\\min}+\\alpha\\,\\bar c_k$.  A very small $\\tau_{\\min}$ makes the cheap-band softmax sharp (highly discriminative but prone to noisy, peaky gradients) while a larger $\\tau_{\\min}$ smooths the distribution, lowering variance but also blurring differences among cheap experts.\n\n| $\\tau_{\\text{min}}$ | AUDC   | max_acc | cost@max_acc |\n|-|-|-|-|\n| 0 | 0.5484 | 0.5877 | 51.232  |\n| 0.02  | 0.5602 | 0.6037 | 51.704 |\n| 0.05  | 0.5581 | 0.5883 | 44.633 |\n| 0.08  | 0.5515 | 0.5707 | 38.395 |\n\nRaising $\\tau_{\\min}$ from $0$ to $0.02$ increases AUDC and peak accuracy, showing that a touch of smoothing stabilizes learning without hurting discrimination. At $\\tau_{\\min}=0.05$ we keep nearly the same AUDC while cutting the cost needed to achieve peak accuracy by $\\approx14\\\\%$ (from 51.7 to 44.6).  Pushing to $\\tau_{\\min}=0.08$ oversmoothes the cheap band: accuracy at low cost rises slightly, but max\\_ac and AUDC both fall.  Thus a moderate setting ($\\tau_{\\min} \\approx 0.02$–$0.05$) offers the best efficiency–stability trade-off.\n\n# 4. Q2: What if we ablate or enlarge the encoder?\nWe’re glad you asked as this is exactly the direction we’re exploring next.\nIn this work we deliberately kept the router small: a frozen sentence-transformer plus a 2-layer MLP. We have added the following ablation where we increased the dimension of the middle layer of the MLP. As the router size increases, AUDC improves, but latency also increases.\n\n| Router | AUDC |\n|-|-|\n| MLP-x1 | 0.5189 |\n| MLP-x4 | 0.5384 |\n\nWe’re now testing larger routers  by replacing the frozen query encoder with a larger LLM to see whether richer semantics improve difficulty judgment. More broadly, we’re excited to answer: How big must a model be to recognize a problem’s difficulty even if it can’t solve the problem itself? We want to study whether size and fine-tuning helps; we’re exploring whether RL on LLMs makes them able to purely as routers and sweep model size to pinpoint when difficulty awareness kicks in.\n\n# 5. Qualitative Insights and Interpretability\nWe agree that interpretability is extremely valuable. We performed an analysis on embedllm, which features a large pool of models with diverse costs. Note that these observed trends are specific to the dataset and may not generalize to other datasets or prompt pools with more challenging examples.\n\n## 5.1 Selection Profiles\n$\\texttt{Qwen/Qwen1.5-0.5B-Chat}$ (selected 91 times), $\\texttt{google/gemma-2b-it}$ (233), and $\\texttt{microsoft/phi-2}$ (292)—are smaller experts that are selected frequently. Some expensive experts (e.g., $\\texttt{Qwen/Qwen-72B}$, $\\texttt{ibivibiv/alpaca-dragon-72b-v1}$) are rarely chosen, since a less costly correct expert typically exists in the dataset. \n\nWe also report per-expert selection rates by cost band. \n\n| Band Index | Count |\n|-|-|\n| 0 | 533 |\n| 1 | 837 |\n| 2 | 805 |\n| 3 | 694 |\n| 4 | 131 |\n\n## 5.2 Taxonomy of Errors\nWe present a confusion-style breakdown of routing errors:\n1. Too cheap—the router selects a cheap but incorrect expert \n2. Too expensive—the router selects an unnecessarily costly expert, though correct\n3. Optimal—the router selects a correct and minimally costly expert \n4. No correct—no available expert produces a correct answer\n\n| Outcome | Count |\n|-|-|\n| Too cheap | 235 |\n| Too expensive | 586   |\n| Optimal | 773 |\n| No correct | 63|\n\nWe will add more comprehensive analysis and charts to the Appendix.\n\n# 6. References\n[1] An Introduction to the Bootstrap, Efron and Tibshirani, 1994\n[2] Note on the sampling error of the difference between correlated proportions or percentages, McNemar, 1947\n\n***\nWe truly appreciate the reviewer’s suggestions; the above additions directly address significance, interpretability, and clarity or our work."}, {"review_id": "lLPygNnwx4", "reviewer": "Reviewer_9rPU", "summary": "This paper introduces Cost-Spectrum Contrastive Routing (CSCR), a lightweight framework for dynamically selecting the most cost-effective LLM from a diverse model pool to handle a given user prompt. It works by embedding both prompts and LLM experts into a shared embedding space, and selecting the most suitable model for a given user prompt based on the embedding similarity. The authors give two methods to obtain model embeddings (Fingerprints): Logit–Footprint Descriptors for Transparent LLMs, and Perplexity Fingerprints for Black-Box LLMs. A query encoder is trained to obtain the prompt embedding via a contrastive training objective. At inference time, routing is reduced to a fast k-NN lookup in this learned embedding space, and the final expert is selected based on both the embedding similarity and cost. Experiments show that CSCR consistently outperforms other routers, while also demonstrating strong generalization to unseen LLMs and out-of-distribution prompts.", "questions": "None", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed method is novel and intuitive.\n2. CSCR outperforms baselines mehods, while being lightweight, cost-aware, and generalizable.\n3. The evaluation is comprehensive.\n\nWeaknesses\n1. ​​The presentation could be improved. For instance, in Section 3.1.2, the transition from Equation 3 to Equation 5 isn't sufficiently clear. The rationale behind this approximation needs better justification or explanation.", "was_revised": true, "final_justification": "This response addresses my concern, so I'll keep my positive rating.", "rebuttal": "We sincerely thank the reviewer for the positive assessment and for pointing out the presentation gap between Eq. 3 and Eq 5.\nBelow we state the missing bridge explicitly and justify the approximation.\n***\n# The step from Eq. (3) to Eq. (5).\nEq. (3) defines a per-prompt token NLL that requires access to an expert’s next-token distribution $p_h(\\cdot \\mid \\cdot)$.\nFor *API-only (black-box) experts*, logits/probabilities are not exposed, so Eq. (3) is not computable.\nOur remedy is to \n1. let the API expert $h$ produce a deterministic continuation $\\hat{y}_h(x)$ for prompt $x$ (greedy decoding), and \n2. evaluate that sequence under a single shared, public scorer $p_S$.\n\nThis yields Eq. (5), a *pseudo-perplexity*:\n\n$$\n\\tilde{\\ell}_h(x) = -\\frac{1}{|\\hat{y}_h(x)|} \\sum _{t = 1} ^{|\\hat{y}_h(x)|}  \\log p_S ( \\hat{y} _{h,t} | \\hat{y} _{h,<t} )\n$$\n\nwhich we then normalize (similar to Eq. 4) and use as the fingerprint coordinate(s) for black-box experts. If $\\hat y_h$ is a typical (high-probability) output of $h$ (i.e., $\\hat y_h \\sim p_h$) then averaging the pseudo-perplexity $\\tilde{\\ell} _h(x)  = -\\tfrac{1}{|\\hat{y}_h(x)|} \\sum _{t} \\log p_S ( \\hat{y} _{h,t} | \\hat{y} _{h,<t})$\nover many prompts/tokens is equivalent to taking an expectation over $y \\sim p_h$:\n\n$$\n\\mathbb{E}_{y\\sim p_h}\\big[-\\log p_S(y)\\big]\n= H(p_h,p_S)\n= H(p_h) + \\mathrm{KL} \\big(p_h || p_S \\big).\n$$\n\nHere $H(p_h,p_S)$ is the *cross-entropy* of $p_h$ with respect to $p_S$, which decomposes into\nthe entropy of $h$’s own distribution $H(p_h)$ and its divergence from the scorer $\\mathrm{KL}(p_h\\|p_S)$.\n\n\nBecause $p_S$ is fixed for all experts, $H(p_h,p_S)$ is a stable, model-specific quantity that makes descriptors\ncomparable across experts (same yardstick). It is not the true NLL under $p_h$, but it preserves differences between experts\nvia $H(p_h)$ and their mismatch to $p_S$ via $\\mathrm{KL}(p_h\\|p_S)$. In practice we use deterministic (greedy) decoding to\nreduce variance; averaging over many prompts/tokens makes the empirical $\\tilde{\\ell}_h(x)$ closely track the expectation above.\nFigure 2 in the paper empirically validates this argument.\n\nWe will make sure we add an explanation so the transition from Eq. 3 to Eq. 5 is smooth.\n\n***\nWe thank you for your feedback again. This improves the clarity of our work. We have also conducted additional ablation experiments which we invite you to read in our response to reviewer PkxW."}]

Author rebuttal:

Meta-review: This paper introduces Cost-Spectrum Contrastive Routing (CSCR), a framework for cost-aware routing across LLM pools that uses compact model descriptors and a contrastive objective incorporating inference cost. The approach is technically sound, lightweight, and achieves consistent improvements in accuracy–cost efficiency across multiple benchmarks, with demonstrated robustness to unseen models and out-of-distribution prompts. Reviewers noted strengths in novelty, clarity, and empirical validation, while raising concerns about statistical significance, interpretability, and presentation. The authors addressed these issues through additional experiments, analyses, and revisions during the rebuttal, which resolved the main concerns. Overall, the contribution is solid, timely, and practically relevant, and the AC recommends spotlight acceptance.
Decision: Accept (spotlight)