Title: Confident or Seek Stronger: Exploring Uncertainty-Based Small LM Routing From Benchmarking to Generalization
Venue: NeurIPS 2025
Accepted: no
Keywords: Large Language Model

LLM routing

Uncertainty Quantification
Abstract: Small language models (SLMs) are increasingly deployed on edge devices for personalized applications, offering efficient decoding latency and reduced energy consumption. However, these SLMs often generate inaccurate responses when handling complex queries. One promising solution is uncertainty-based SLM routing, offloading high-stakes queries to stronger large language models (LLMs) when resulting in low-confidence responses on SLM. This follows the principle of If you lack confidence, seek stronger support to enhance reliability. Relying on more powerful LLMs is yet effective but increases invocation costs. Therefore, striking a routing balance between efficiency and efficacy remains a critical challenge. Additionally, efficiently generalizing the routing strategy to new datasets remains under-explored. In this paper, we conduct a comprehensive investigation into benchmarking and generalization of uncertainty-driven routing strategies from SLMs to LLMs over 5000+ settings. Our findings highlight: First, uncertainty-correctness alignment in different uncertainty quantification (UQ) methods significantly impacts routing performance. Second, uncertainty distributions depend more on both the specific SLM and the chosen UQ method, rather than on downstream data. Building on the insight, we propose a proxy routing data construction pipeline and open-source a hold-out set to enhance the generalization on predicting the routing curve for new downstream data. Experimental results indicate that proxy routing data effectively bootstraps routing performance without any new data.


Peer reviews:
comments: SUMMARY:
This paper investigates the use of uncertainty-based routing from small language models (SLMs) to large language models (LLMs) as a practical approach for improving reliability in edge-device deployment scenarios. It presents a comprehensive benchmark over 5000+ settings involving 15 datasets, 12 SLMs, 4 LLMs, and 8 uncertainty quantification (UQ) methods. The authors identify critical alignment patterns between uncertainty and correctness that affect routing efficacy. To tackle the challenge of generalizing routing strategies to unseen tasks, the authors propose a proxy routing data construction pipeline, enabling routing threshold estimation without accessing new downstream data. The proposed framework addresses an important efficiency–reliability trade-off in real-world LLM deployment.

STRENGTHS AND WEAKNESSES:
***Strengths:***
1. The paper is well-organized and clearly written, especially the motivation and experimental sections.

2. The experimental evaluation is thorough. This paper conducts a comprehensive empirical evaluation across a wide range of models, datasets, and UQ methods.

***Weaknesses:***
1. It is unclear how existing baseline methods—such as RouterBench—perform in unseen domain settings. A direct comparison in Figures 5 and 6 would greatly help contextualize the benefit of the proposed proxy routing strategy.

2. It remains unclear how well the approach generalizes to code-related tasks, which are a common and important application domain for LLMs. More experiments on HumanEval or MBPP datasets are expected to be conducted.

QUESTIONS:
Have the authors evaluated existing multi-LLM routing systems under real or simulated on-device conditions, specifically in terms of latency and memory usage?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-30

comments: SUMMARY:
The paper evaluates 8 uncertainty quantification (UQ) methods across 14 datasets to assess how well uncertainty aligns with prediction correctness in SLMs, which is crucial for effective routing. Based on the finding from the benchmarking, the authors introduced a proxy routing data construction pipeline and a hold-out dataset to generalize routing strategies without prior knowledge of new downstream data.

STRENGTHS AND WEAKNESSES:
Strengths

The paper provides a rigorous benchmark involving 8 UQ methods, 8 SLMs, 2 LLMs, and 14 datasets, reflecting real-world deployment diversity and offering strong empirical support for their claims. The work addresses a real and growing challenge — balancing accuracy and cost in edge AI deployments—and provides actionable insights that help minimize LLM calls while preserving performance. Finally, the introduction of a proxy routing dataset to predict routing behavior in unseen tasks is both novel and practical, especially for scenarios where data collection is costly or infeasible.

Weaknesses

Authors could provide in-depth analysis of why certain UQ methods fail on specific datasets or tasks, which would be valuable for future improvements.
Despite the generalization claims, the proxy routing approach is still indirectly tuned using benchmarked datasets, which may bias the proxy data toward known distributions.
The pipeline for generating proxy routing data is not deeply explained in terms of how it adapts to drastically different downstream domains (e.g., multimodal tasks, non-English text), which limits its universality. 
Can the authors propose some guidelines from the observations?

QUESTIONS:
Already stated in weakness.

LIMITATIONS:
No.
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a setting in which small language models (SLMs) attempt to answer a question first, and then defer to a large language model (LLM) if the estimated confidence in their answer is too low.\n\nA routing curve describes the performance of such a system for a given SLM-LLM pair as one uses a specific uncertainty quantification (UQ) metric to select which examples to escalate to the LLM. The curve shows how tweaking the proportion of examples in a dataset passed to the LLM (from none to some to all) changes the performance of the system on the entire dataset.\n\nThe authors first evaluate several UQ metrics to see which performs better at selective prediction on different datasets (measurable by AUC ROC). Then, they produce routing curves to see which UQ metrics perform best at routing for each SLM & dataset.\n\nFinally, the authors argue and present evidence that the confidence distribution of UQ over a dataset typically depends on the UQ method and the SLM, but **not the dataset** -- arguing that the routing curve for a new dataset can be predicted by the routing curve on data from other datasets on the same SLM&UQ. They formalize this notion with Algorithm 1, a method of estimating routing curves with \"proxy routing data\" from other datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n**S1**\n\nExperiments are sufficient in scale and execution for this type of paper. They are also presented in a sufficient way to argue the main points. The breadth of UQ methods examined is sufficient, and several different SLMs are explored. Contribution as a benchmark is strong.\n\n**S2**\n\nWriting is generally clear. I do have an issue with the section 4; I will elaborate later.\n\nWeaknesses:\n\n**W1**\n\nThis field is not new; there are many works (acknowledged in this paper draft) that have used uncertainty for SLM-->LLM routing--and this paper's main methodology is mostly non-novel. \n\nOne of the main ways in which this work is justified as distinct compared to these prior works is the presence of improved guidelines for better dataset/domain generalization. However, it seems that the main conclusion of this work with respect to generalization is that changing the dataset actually does NOT present much of an obstacle for generalization--and I find the proxy routing data contribution slightly overcomplicated (see my W3).\n\n**W2**\n\nThe paper argues that other works \"face significant challenges when encountering new downstream tasks, as such data falls outside the distribution of the existing training data\" (133-135?). I find it not fully explained or explored why this paper's experiments, which again use a very standard setup, did not result in the same cross-domain issues that this paper argues these other papers displayed. It is certainly possible that specific tweaks in the setup led to these differences--but I think understanding and explaining any such differences is a key part of the remit of this work.\n\n**W3**\n\nThe proxy routing contribution is presented in 4.1 and Algorithm 1. I do not find the notation in Algorithm 1 very well-explained (what is $\\mathbf{X}_i$? What is $j$?) and furthermore, I think the presentation of a proxy routing dataset to be a slightly overcomplicated stratified random sample with no real justification. Why not just sample 10% of the proxy samples instead of binning and sampling? Is there a meaningful difference in the results?\n\nIf there is such a difference, would we not just counteract it by sampling more than 10%? If the justification on subsampling 10% is to save on the cost of inference when producing routing curves, don't we still have to do forward pass inference and UQ inference on every example to put it in the correct bin before sampling -- meaning that we still have done inference on every example?\n\n**W4**\n\nSections 2 and 5 occupy a great deal of the work. Section 2 mostly deals with in-depth literature review (explaining each type of UQ), and Section 5 deals mostly with speculation about future research directions. Either or both could be sacrificed, condensed, or relegated to the appendices in favor of more in-depth explanation of the novelty of this work and differences between this work and other work's findings.\n\nQUESTIONS:\nMost of my questions are addressed in the Weaknesses section above. Mostly, I am concerned about the necessity and utility of the proxy routing data concept--as well as with the novelty of this work and the resolution of this work's differing conclusion with respect to OOD routing as compared to other cited works.\n\nLIMITATIONS:\nThe authors quickly address potential negative social impact of their work. I am inclined to agree that there is not much such impact.\n\nHowever, I think the authors might address in more detail address a few more limitations of their analysis, such as:\n\n* The cost of producing the routing dataset, including assigning buckets, is at least one inference forward pass + UQ forward pass in the full routing set before sampling 10%\n* The routing analysis does not consider tweaking the LLM as much as it considers tweaking the SLM. This is another possible axis of analysis.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the problem of limited capabilities in small models and the high cost of deploying large models, this paper analyzes various existing methods for switching to large models based on the uncertainty of small models.\n\nSpecifically, the paper provides extensive analysis and experiments involving 16 models (12 SLMs, 4 strong LMs), 8 uncertainty quantification methods, and 15 benchmarks. Furthermore, the authors introduce a pipeline to construct the proxy routing data. This data can generalize its routing standards across various new downstream datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n\n\n- The main strength of this paper lies in its substantial empirical contributions. The large scale benchmark (with 16 models including 12 SLMs and 4 strong LMs, 8 uncertainty methods, and 15 benchmarks) is an important and valuable resource for the NLP community, providing a good foundation for the paper's claims and future research in this area.\n\n\n\n- This paper is not just a simple analysis paper. In order to solve the problem of how to transfer routing standards to new and unseen tasks, the paper further proposes a pipeline to construct the proxy routing data, which can generalize its routing standards across various new downstream datasets.\n\n\n\n- The paper is written with clarity and the open source code makes the work easier to reproduce\n\n\n\n\n\n**Weaknesses:**\n\n\n\n- The work suffers from a lack of deeper theoretical explanation for the empirical findings. For example, uncertainty distributions depend more on both the specific SLM and the chosen UQ method, rather than downstream data.  A more in-depth analysis will further enhance the contribution of this paper\n\n\n\n- The training data sizes for the \"trained probe\" (a subsample of one dataset) and the \"OOD probe\" (the entirety of other datasets) are vastly different. It is unclear if the strong performance of the \"OOD probe\" could be maintained under a more controlled comparison with an equal amount of training data, making it difficult to assess the true cost and benefit of this approach.\n\nQUESTIONS:\n1. I am curious about constrained tasks like translation (e.g., WMT'14), which should arguably have lower uncertainty than open-ended tasks. How well does your main experiment and pipeline for constructing proxy routing data work on this task?\n\n\n\n2. Your experiments find that perplexity (ppl) is a good metric. Recently, entropy has also become a very popular measure of uncertainty, even being used in reinforcement learning. Could you please supplement your results with an evaluation using entropy as an uncertainty measure?\n\n\n\n3. The OOD Probe method is crucial for generalizing to new datasets. However, in your experiments, it was trained on significantly more data (the entirety of other datasets) than the Trained Probe (a subsample of one dataset). To conduct a fairer comparison and better quantify the trade offs, could you run an experiment where the OOD Probe is trained on a data volume comparable to that of the Trained Probe? This would help clarify the extent to which its effectiveness stems from data diversity versus data volume itself.\n\n\n\n4. How sensitive is the final routing curve prediction to the hyperparameters of your pipeline, such as the number of bins for the distribution (M=30) and the sampling rate (10%)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}