Query-Level Uncertainty in Large Language Models
Authors: Lihu Chen
Gerard de Melo
Fabian M. Suchanek
Gaël Varoquaux
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Frugal AI
Uncertainty Quantification
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: In this work, we propose a method to detect knowledge boundaries via Query-Level Uncertainty, which estimates if a model is capable of to answering a given query without generating any tokens.

Abstract: It is important for Large Language Models (LLMs) to be aware of the boundary of their knowledge, i.e., the mechanism of identifying known and unknown queries. This type of awareness enables models to perform adaptive inference, such as invoking retrieval-augmented generation (RAG), engaging in slow and deep thinking, or abstaining from answering when appropriate. These mechanisms are beneficial to the development of efficient and trustworthy AI. 
In this work, we propose a method to detect knowledge boundaries via \textbf{\emph{Query-Level Uncertainty }}, which estimates if a model is capable of to answering a given query before generating any tokens. 
To this end, we propose a novel, training-free method called \textbf{\emph{Internal Confidence}}, which leverages self-evaluations across layers and tokens to provide a reliable signal of uncertainty.
Empirical studies on both factual question answering and mathematical reasoning tasks demonstrate that our internal confidence can outperform several baselines. Furthermore, we showcase that our proposed method can be used for adaptive inference,  such as efficient RAG and model cascading, thereby reducing inference costs while preserving overall performance.

Published as a conference paper at ICLR 2026
QUERY-LEVEL UNCERTAINTY IN LARGE LANGUAGE MODELS
Lihu Chen', Gerard de Melo?, Fabian M. Suchanek Gaäl Varoquaux* Imperial College London, United Kingdom Hasso Plattner Institute / University of Potsdam, Potsdam, Germany
Telecom Paris, Institut Polytechnique de Paris, France
*Soda, Inria Saclay, France
lihu. wihu.chen@iiie wwweeeimeeereee ac. uk, gerard. demelo@hpi.co
fabian.
ecoo fr, gael. waroquauxeinte inria fr
ABSTRACT
It is important for Large Language Models (LLMs) to be aware of the bound- ary of their knowledge, distinguishing queries they confidently answer from those that lie beyond their capabilities. Such awareness enables models to per- form adaptive inference, such as invoking serieval-augmented generation (RAG), engaging in slow and deep thinking, or abstaining from answering when appro- priate. These mechanisms are key to developing efficient and trustworthy AL. In this work, we propose a method to detect knowledge boundaries via Query- Level Uncertainty. which estimates if model capable of answering a given query before generating any tokens, thus avoiding the generation cost. To this end, we propose novel, training-free method called Internal Confidence, which leverages self-evaluations across layers and tokens to provide reliable signal uncertainty. Empirical studies on both factual question answering and math- ematical reasoning tasks demonstrate that our Internal Confidence outperforms several baselines in quality of confidence while being computationally cheaper. Furthermore, demonstrate its benefits in adaptive inference settings, showing that for RAG and model cascading it reduces inference costs while preserving overall performance. The code is available O https:/ htttps://iithbb.ooo tigerchen52/ /ecbrerrnn////ry..orraa query level uncertainty
MSP
64.0
64.1
Intornal Confidenice
62
66-
Perplexity
60.55
64
CCP
(%6)
Lexical Similarity
58
$77.8
AUROC
Semantic Entropy
60
A55.1
54-
50-
RAG Interna Confidence RAG (seeelne)
Laplacian
Running
per sample)
Running (seconds)
(a) Comparison of performance of distinguishing answerable and non- answerable queries run-
(b) Trade-off between running time and perfor- mance under different Internal Confidence thresh- olds for deciding on RAG invocation (Phi-3.8B on TriviaQA) compared against always using RAG.
ning time between our query-level Internal Con- fidence method and existing answer-level uncer- tainty measures (Qwen- 14B GSM8K).
Our Internal Confidence method improves performance running time tradcoff's in factu- ality assessment and RAG settings.
Figure I:

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Large language Models (LLMs) have their knowledge boundaries (Li et al. 2024 Yin et al. 2024
Ren al., 2025), which means that there are certain problems for which they cannot provide accurate answers. It is crucial for LLMs to be self-aware of their limitations, i.e., to know what they know and know what they do not know (Kadavath al. 2022 Amayuelas et al. 2024).
Clear awareness of knowledge boundaries is central to improving AI, both for efficiency and trust- worthiness. The rising usage of L.MMs and agents has introduced significant computational and monetary costs (Varoquaux et al. 2025). For example, agentic workflows may cost 5x-25x more per query compared to a simpler LLM prompt (Anthropic, 2025). Regarding efficiency, LLMs can distinguish answerable from -nswerbeeeeele simple from hard queries, they smartly perform adaptive inference to navigate the trade-offs between computational cost and output quality (Chen & Varoquaux 2024). For queries beyond their parametric knowledge, they can actively trigger RAG to obtain external knowledge (Lewis et al. 2020) or tool calls (Schick et al. (023). When faced with hard problems, L.MS can engage in slow (or deep) thinking to improve their outputs, which is also known as test-time scaling (Snell et al. 2024 Zhang et al. 2025). Alternativly, they can defer complex problem larger model via model cascading (Dohan et al. 2022] Gupta et al.) 2024). This adaptive inference ensures efficient allocation of computational resources, reducing costs while maintaining performance, especially for agentic scenarios. Beyond efficiency, estimating whether a query is answerable also enhances honesty and trustworthiness of LLMs. When faced with highly uncertain queries, models can adopt an abstention strategy (Wen et al. 2024) to withhold potentially misleading responses, important in high-stakes domains like healthcare (Tomani et al. 2024).
In this work, we introduce the concept of Query-Level Uncertainty to estimate a model's s knowledge with regard to given query. The central research question here is: Given query, can we deter- mine whether the model can address it before generating any tokens? Most existing work focuses on answer-level uncertainty, which measures the uncertainty associated with specific answer and is commonly used to assess the reliability of model outputs (Shorinwa et al.) 2024) Vashurin et al.] 2025). In contrast, our approach shifts from post-generation pre-generation, measuring how con- fidently an LLM can solve a given query, prior to answer generation, as illustrated in Figure 2 This approach avoids the computational cost of generating potentially long answers.
Prior research has explored different strategies for uncertainty estimation. One line of work learns a probe of internal states to predict uncertainties of queries (Gottesman & Geva 2024 Kossen al.)
2024). Another branch of work attempts to teach LLMss to explicitly express "I don It know" in their responses via fine-tuning methods (Amayuelas et al. 2024, Kapoor et al.) 2024, Cohen et al. 2024] Zhang et al.) 2024a). One common issue of these studies is that they require fine-tuning and training
Query: What is the capital of France?
Paris
post-generation
Query
Lyon
(Paris, 0.2)
Toulouse
Answer-Level Uncertainty
RAG
Low
An Awwerable
pre-generation
Deep Reasoning
Query
Invoke
High
Non-
Answerable
Cascading
Query-Level Uncerainty
Abstention
Figure 2: Illustrating the difference between answer-level and query-level uncertainty. Query-level uncertainty estimation distinguishes answerable from non-answerable queries (knowledge bound- ary) before generating answers, which is useful for adaptive inference, c.g., efficient RAG, flst reasoning, or cascading models with different abilities.
2

---

Published as a conference paper at ICLR 2026
samples, which introduces additional overhead and may restrict their generalizability across models and domains. To address this gap, we introduce a training-free approach to estimate query-level uncertainty that is both simple and effective.
Our training-free freee and generation-free approach, termed Internal Confidence, leverages self- evaluation across internal layers and tokens. It is grounded in simple assumption: LLMs can internally self- assf-sssess the boundaries of their knowledge through a single forward pass over the given query, without generating an explicit answer. Inspired by the uncertainty measure P(TRUE) (Ka-) davath et al. 2022). we prompt LLMs with a yes-no question to self-assess if they are capable of answering a given query. and define the probability assigned the token YES as the confidence level, denoted as P(YES) To fully exploit the latent knowledge within LLMs, our improved Internal Confidence approach computes this sort of P P(YES) at each layer and token position. Subsequently, we aggregate these signals to obtain the overall confidence score. This aggregation is motivated by prior work showing that leveraging logical consistency across layers can improve outputs (Burns) et al. 2022: Chuang et al. 2023 Xie al. 2024). Concretely, compute a weighted sum across layers and tokens, and the weights are derived from attenuated encoding (Chen et al. 2023), which enables fineggrineed control of the influence of adjacent units.
To validate the effectiveness of our proposed Internal Confidence, we conduct experiments on three datasets that cover factual QA and mathematical reasoning tasks. For fair comparison, adapt existing answer-level methods to the query level. Experimental results demonstrate that our pro- posed Internal Confidence can distinguish between answerable and non-answerable queries more accurately than a range of baselines, while being substantially faster than answer-level approaches (Figure Lap. In terms of applications, we showcase that our proposed method can support efficient RAG and model cascading. On the one hand, Internal Confidence can guide users to assess the trade-offs between cost and quality when invoking additional services. On the other hand, it reveals an "opttiaa point", where inference overhead can be reduced without compromising performance (Figure TT). In conclusion, we introduce the notion of query-level uncertainty and propose a simple yet effective training-free method to estimate it, which enables models to determine whether query can be addressed without generating any tokens.
2 RELATED WORK
2.1 UNCERTAINTY ESTIMATION AND LLMS
Existing approaches to LLM uncertainty primarily focus on estimating the uncertainty of LLM- generated responses, by providing score intended to reflect the reliability of query- answer pair (Geng et al., 2024; Shorinwa et al., 2024; Mahaut et al., 2024; Vashurin et al., (025). These approaches often rely on internal states (Chen et al. 2024a) or textual responses (Kuhn al.. 2023). and commonly use calibration techniques to mitigate issues such as overconfidence (Zhang et al.) 2024b) and biases (Chen et al., 2024b). Notably, these methods assess post-generation reliability, i.e., uncertainty regarding specific answer after it has been produced. In contrast, relatively little work has explored how to quantify a model's ability to address a query prior to token generation. For example. Gottesman & Geva (2024) propose training a lightweight probe on internal repre- sentations to estimate the model's 's knowledge about specific entities. Similarly, Semantic Entropy Probes (Kossen et al., (024)) suggest that internal model states can implicitly encode semantic un- certainty, even before any output is generated. To the best of our knowledge, this work is the first to formally define query-level uncertainty and investigate it systematically.
2.2 KNOWLEDDG BOUNDARY DETECTION
LLMSs should be able to faithfully assess their level of confidence in answering a query. This aware- ness of knowledge boundaries (Li et al., 2024 Yin et al.) 2024 Wang et al.) 2024) is essential for building reliable AI systems, particularly in high-stakes domains such as healthcare and law. A pio- neering study by Kadavath et al. 2002 explores whether language models can be trained to predict when they "know" the answer to a given query, introducing the concept of "I Know" (IK) prediction. Based on this idea, subsequent work has proposed methods to help LLMs become explicitly aware of their knowledge limitations through fine-tuning strategies (Amayuelas et al.) 20244] Kapoor et al.
2024). Cohen et al. (2024) further advances this line of research by introducing special [TDK] (")
3

---

Published as a conference paper at ICLR 2026
don't know") token into the model's vocabulary, allowing the direct expression of uncertainty in its output. Similarly, R- R-Tuning (Zhang et al., 2024a) tunes LLMs to refrain from responding to ques- tions beyond their parametric knowledge. While these abstention- -based approaches show benefits in mitigating hallucinations (Wen et al. 2024), they often require additional fine-tuning, which in- troduces overhead and may limit generalizabilit across models and tasks. In this work, propose a training-free method to identify the knowledge boundary of an LLM, which offers a more efficient alternative that can be applied across models and tasks.
3 PROBLEM STATEMENT AND METHOD
In this section, we define the problem and introduce our Internal Confidence, a training-free and generation- uncertainty that reflects whether an L.MM can address query in its own knowledge, prior to generating tokens.
3.1 PROBLEM STATEMENT
Given query (including prompt tokens) x (Z1,..., IN), we aim to quantify the query-level un- certainty, Ux), without generating an answer y. This differs from existing uncertainty approaches that estimate the uncertainty associated with a specific generated answer, an answer-level uncertainty that can be denoted as U(x.y² y). We define a query as being within the model's 's knowledge boundary if the LLM can produce correct answer under greedy decoding, i.c., by selecting the highest- probability token at each step without sampling. Conversely, failure to produce the correct answer suggests the query falls beyond the model's boundary, and it does not possess sufficient knowledge to answer it. While greedy decoding ensures deterministic measurement, it may not always reflect the optimal performance of a model (Song et al. 2024), as alternative decoding strategies like beam search may elicit better answer. We stick with greedy decoding for the following reasons: (1) Efficiency - Our method treats a successful greedy decode as a signal that the model knows how to answer, which is fast proxy. In contrast, non-greed non-greedy decoding requires the configuration of beam numbers, probability thresholds, and sampling numbers, which complicates both the definition of knowledge boundary and the assessment cost. (2) Reproducibility Greedy decoding outputs single deterministic output for a given input, which offers a stable and reproducible bascline and benchmark.
Therefore, this pragmatic framework serves as a heuristic indicator of internal knowledge, rather than absolute measure. We use this standard evaluate the estimated query-level uncertainty, ie., a lower uncertainty indicates a model is more likely to output the correct answer.
Our problem formulation mostly targets epistemic uncertainty of the model, though specific queries and datasets may contain aleatoric effects (see details in Section and the definition of the knowl- edge boundary is aligned with the parametric knowledge boundary (Li al. 2024). This boundary of a model is the set of all knowledge encoded in its parameters that can be verified by at least one input-output pair. Our study focuses on queries with definite and clear-cut answers, as in factual QA and mathematical reasoning, which have broad applications and allow for clear evaluations. While contentious queries with open and subjective answers are also important in areas such politics and philosophy, they remain beyond the scope of this work.
3.2
METHOD: FROM P(YES) TO INTERNAL CONFIDENCE
Studies have revealed that LLMs can express verbalized uncertainty in their responses (Tian et al., 2023 Xiong et al., 2024), and they can self-evaluate whether they know the answer to a question without reference to any specific proposed answer (Kadavath et al., (022), which indicates that LLMs possess an internal mechanism for assessing the correctness of their outputs. At the same time, recent work indicates that answerable and non-answeeable questions are also linearly separa- ble in hidden states (Slobodkin et al. 2023). Building on this observation, one can explicitly prompt an LLM to self-assess its confidence in answering given query by constraining the response to a no binary format: "Respond only with 'Yes' or 'No' to indicate whether you are capable of answering the (Query) accurately. Answer Yes or No:". Following that, can compute the
4

---

Published as a conference paper at ICLR 2026
probability assigned the token P(YES) the last token (2:)): ):
P(YES) = softmax WWrrssN (L) [YES.No] Diy YES
(1)
Here, N is the index of the last token in the query and L is the index of the last layer of the model. h Rd is the hidden state, where d the dimensionality of the hidden representations. Wunemb RDv xd is the unembedding matrix that maps the hidden state h to logits over the vocabulary V. The unembedding layer provides meaningful and comparable probabilities, whereas the raw logits not directly interpretable in this way. The probability P P(YES) can serve query-level confidence score here, which is similar to the process of linear probing (Alain & Bengio (016), but without any training steps. While this measure correlates with verbalized uncertainty, a key distinction is that it requires only single forward pass of the query, without generating any answer tokens.
However, P YES) considers only the final hidden state of the LLM, although the intermediate inter- nal states of LLMs preserve rich knowledge and latent information (Chen et al. (025), especially for uncertainty estimation (Azaria & Mitchell 2023 Chen et al. 2024a). Furthermore, prior work demonstrates that incorporating logical consistency across layers can improve outputs (Burns al.)
2022 Chuang al. 2023 Xie et al., 2024).
Motivated by these insights, we propose the Internal Confidence, a method that leverages latent knowledge distributed across multiple layers and tokens. Formally, let fo denote the transformation function for computing hidden states, parametrized by 0. The hidden state for the token Ty of the input query at layer l is computed
h = So (h) (1-1) y... b(-1))
(2)
In total, the model contains N x L such latent representations, and we can use Equation to compute the P(YES) for each (!)
Figure 3al plots the average P PEE)) of Llama-8B on mathematical queries-th the validation set of GSM8K (Cobbe et al. 2021 across layers and query tokens We observe that the P(YEs) generally increases from lower to higher layers and from left to right positions. If we treat each P(YES h as confidence score and compute the Area Under the Curve (AUROC), we can ob- tain an AUROC heatmap that illustrates how effectively each internal representation can distinguish answerable and non-answerable queries. As shown in Figure 33b] the highest score does not neces-
sarily appear at the top right position. Instead, the representation h (27) yields the best AUROC, and the performance gradually declines in regions surrounding this point. We refer to this optimal point as the decision center, where the model most effectively separates answerable from -nswerbleee queries.
To improve the vanilla P P(YES), we can apply weighted average centering around the decision center, which serves as ensemble strategy to enhance calibration and expressivity (Zhang et al.) 2020 Stickland & Murray 2020). We refer to this process as Internal Confidence (IC), formally defined
IC(h) Σ
W P(YES h.)).
where (i denotes the weight assigned to the hidden representation h. The equation describes a hierarchical two-step aggregation process. In the first step, for each individual token, we compute a weighted sum of confidence scores across layers. In the second step, we aggregate these token- level scores using another weighted average. Conceptually, this process be parameterized by layer weight vector w layer & R-_ for the first step and a token weight vector w woken & R for the second step. The obtained IC(h) value provides single, refined confidence score that integrates rich information across both layers and tokens.
In our implementation, we adopt the top-right cell (corresponding to the last token and last layer) as the decision center, since we observe that the decision center tends to be located near the later layers
Herr, consider the last & tokens of query, assuming that model has seen the entire query and is able infer its knowledge gap.
5

---

Published as a conference paper at ICLR 2026
31
31
035
Decay weights different
locaiity 0.1
26
004
locality 0.2
localit 0.6
Model Layers
0.8
locailty - 1.0
21
-035
Weight Value
0.6
16
Model
16
-038
-04
0.4
-045
0.2
044
0.0
Query Tokens
Query Tokens
Position
(a) P(YES)
(b) AUROC
(c) Decay Weights
Figure 3: Left: the internal P(YES) across tokens and layers. Middle: the AUROC of P(YES)
tokens and layers. Right: decay weights with different localities. Model: Llama-8B; Dataset: GSM8K validation set.
and final tokens across various architectures and tasks. While, in principle, the optimal decision center may also lie elsewhere, identifying such optimal center would require hold-out set of training data, which conflicts with our goal of developing a training-free approach. To address this, rather than relying model- or task-specific tuning of the decision center, incorporate information from the neighborhood of the fixed top-right cell. This strategy allows us to have the potential benefits of the optimal decision center while maintaining generalizability and avoiding dependence on additional training samples.
To reflect the observation that the AUROC performance gradually decays away from the decision center, we adopt Attenuated Encoding, as proposed by CChn et al. (023) to compute the above weight vectors Equation
exp( - a i - }
8(1)
(4)
Σj=1 exp( - jP)
where is the index of the decision center, / - jl is the relative distance, and a > 0 is a scalar param- eter that controls the locality value. Locality a metric that measures the extent which weights are concentrated in adjacent positions of center. Given a weight vector 5(6) {8(1) g(() ......... 85
and assuming that the center index is we define its locality as
( E [0,
Σ 2
(5)
Here, value of implies that the vector perfectly satisfies the locality property, which means weights are extremely concentrated the decision center. low locality means weights are more uniformly assigned to neighborhoods. Figure Be plots the weights obtained from Equation 4 for varying degrees of locality. This shows that we can account for the influence of neighboring layers and tokens during the averaging process.
Our proposed Internal Confidence training-free and computationally efficient, requires only single forward pass for given query. Since model responses are frequently longer than input prompts and invoking external services such as RAG and deep thinking adds significant overhead, propose this pre-gemeration uncertainty support adaptive reasoning.
4 EXPERIMENTS
4.1 SETTINGS
Models. Our experiments consider three different LLM sizes: 33-mmmi-4-iinstrucae (Abdin et al,
2024), Llama-3. Instructt (Grattafiori et al. 2024). and 1wwen.55-14B-Instruue (Team 2024

---

Published as a conference paper at ICLR 2026
TriviagA
SciQ
GSMBK
Avg
Method
AUROC PRR
ECE AUROC PRR
FCC
AUROC PRR
ECF
AUROC
PRR
ECE
Phi-3.88
Max log p)
55.5
10.0
51.4
55.0
11.3
54.0
8.1
Prodictive Entropy
58.9
17.9
51.2
63.6
25.7
57.9
Min-K Entropy
59.9
20.0
52.7
4.9
60.4
17.9
57.7
14.3
Attentioaa Entropy
52.4
Perplexity
61.8
24.3
57.7
16.5
53.6
6.9
57.7
15.9
Internal Semantic Similarity
-2.4
-5.9
47.9
47.8
15.9
P(YES) (oop right)
64.9
27.7
5,4
61.3
24.4
5,9
53.3
9.4
11.3
59.8
20.5
7,5
P(YES) (naive avg)
64.1
28.3
17.0
57.5
18.8
6.4
50.5
9.3
25.4
57.4
18.8
16.3
Internal Confidence
30.1
25.8
53.9
59.8
Loma--B
Max (- log p)
54.9
11.1
51.4
53.3
10.4
53.2
7.8
Predictive Entropy
58.5
17.7
51.4
3.2
66.1
28.0
58.7
16.3
Min-K Entropy
58.1
17.4
53.5
7.9
57.5
13.2
56.4
12.8
Attentional Entropy
Perplexity
58.6
17.1
58.3
15.1
53.2
4.3
56.7
12.2
Internal Semantic Similarity
52.7
P(YES) Goop right)
55.4
10.2
31.7
58.4
17.2
23.7
52.6
5.2
11.9
55.5
10.9
22.4
P(YES) (naive oug)
65.9
33.0
12.6
57.9
14.9
20.4
61.1
18.5
33.5
61.7
22.1
22.2
Internal Confidence
68.7
38.5
25.4
58.1
15.7
16.7
65.7
34,9
3.1
64.2
28.7
15.1
Owen
Max log
56.5
12.4
54.1
54.3
13.5
55,0
10.9
Predictive Entropy
59.3
18.9
53.2
66.4
32,6
59.6
19.5
Min-K Eatropy
39.9
20.0
55.7
11.3
63.0
30.9
59.5
20.7
Attentional Entropy
17.2
59.4
19.2
54.9
Perplexity
59.1
17.8
60.1
20.7
54.0
7.3
57.7
15.3
Internal Semantic Similarity
2.5
-7.7
14.9
47.5
48.0
-3.3
16.7
P(YES) (top right)
67.8
36.0
30.3
60.0
21.7
24.1
55.0
11.7
6.4
60.9
23.1
20.3
P(YES) (naise org)
67.0
33.9
3.5
59.5
17.9
14.6
64.0
32.3
32.4
63.5
28.0
16.8
Internal Confidence
71.9
43.3
26.5
62.6
23.6
18.2
66.8
28.2
5.7
67.1
31.7
16.8
Table Overall results of different query-level uncertainty estimation methods. performing methods are highlighted using boldface and second-best results are underlined.
The best-
This allows us to assess whether Internal Confidence generalizes across different model sizes. is worth noting that Internal Confidence can also be applied to models without instruction tuning.
Implementations. For Llama and Qwen, Internal Confidence is computed in the zero-shot setting, whereas for Phi, we use two shots in the prompt, since smaller models benefit from demonstration- based guidance (See details in Section D.2). All LLMs employ greedy decoding ensure deter- ministic outputs. The decision center is fixed to the last layer and last token, and we set a = 1.0 (Equation 4 across all models and datasets.
Evaluation Datasets. We evaluate on two factual QA datasets and one mathematical reasoning dataset: TriviaQA (Joshi al. 2017). SciQ (Welbl al.) 2017). and GSM8K (Cobbe al.) 2021) The first two tasks aim to assess factual knowledge stored parameters, while GSMSK requires models self-evaluate their reasoning capabilities. The ground truth for factual QA tasks takes the form of a short answer with entity-related facts. GSMSk as well calls for a short ans wer, but the intermediate reasoning steps are evaluated well, following prior work (Kadavath al. 2022). The three datasets consist of 10,000, 10,000, and 5,000 samples, respectively, with 1,000 samples from each reserved for validation.
We elicit responses from the model using greedy decoding strategy. If the answer aligns with the ground truth, we consider the model as possessing sufficient knowledge and the query falling within its knowledge boundary. For the first two datasets with short answers, answers are deemed correct the ROUGE-L (Lin & Och 2004) the ground truth is greater than 0.3, which con- sistent with prior work (Kuhn et al. 2023). For the GSM8K dataset,
LLM evaluator, Mistral-Large (MistralAl 2024), to assess both reasoning steps and the final answer. We evaluate the reasoning steps GSM8K because verifying the reasoning chain is essential to ensure the model truly understands the problem rather than outputting the correct results by chance. Subsequently, each query is paired with binary label reflecting whether the model is capable of addressing

---

Published as a conference paper at ICLR 2026
Baselines. For comparison, adapt ofthe-o answer-level methods to quantify the pre-
generation uncertainty (see details in Section C (1) Max(-log) log (Manakul et al. 2023). (2) Pre-
dictive Entropy (Malinin & Gales, 202I), (3) Min-K Entropy (Shi et al., 2024). (4) Attentional En-
tropy (Duan et al. 2024), (5) Perplexity, (6) Internal Semantic Similarity (Fomicheva et al. 2020),
(7) P P(YFS) (top right), corresponding to Equation |
(8) P(YEs) (naive avg) is variant of In- ternal Confidence that adopts naive averaging to aggregate scores across different tokens and layers.
We evaluate uncertainty by assessing whether method can distinguish an- swerable and non antewwbwwerbl queries, which can be treated ranking problems, i.e., lower uncer- tainty means model is more likely to know the answer to the query. Following prior work (Man- akul et al. 2023 Kuhn et al., 2023), we adopt the Area Under the Receiver Operating Characteristic Curve (AUROC) and Prediction Rejection Ratio (PRR) (Malinin al. 2017 as metries measure this. Additionally, we compute the Expected Calibration Error (ECE) to assess the calibration of different methods.
Evaluation Metrics.
4.2 INTERNAL CONFIDENCE CAN IDENTIFY ANSWERABLE AND NON-ANSWERABLE QUERIES
Table D summarizes the overall results comparing different query-level uncertainty methods. First, we can observe that our proposed Internal Confidence consistently outperforms other baselines in knowledge boundary detection, reflected in both average AUROC and PRR. The advantage be- comes more pronounced for larger models such as Llama-8B and Qwen-14B. For instance, on Owen-14B, it obtains average AUROC of 67.1 and PRR of 31.7, clearly surpassing all other methods. Regarding the calibration (ECE), Internal Confidence is found to consistently achieve a lower error across models and tasks. These findings indicate the effectiveness of Internal Confi- dence. Finally, we note that the variants, P(YES) (top right) and P(YES) (naive avg), generally underperform the full method, which highlights the importance of the attenuated encoding and its decay weights in effectively aggregating signals from different layers and tokens.
4.3 INTERNAL CONFIDENCE MUCH FASTER THAN ANSWER-LEVEL APPROACHES
We compare our query-level Internal Confidence with several popular answer-level uncertainty methods on GSMSK using Qwen-14B, including Perplexity (Fomicheva et al. 2020). Semantic Entropy (Kuhn et al. 2.23), P(TRUE) (Kadavath et al., 2022). Lexical Similarity (Fomicheva et al.
2020), SAR (Duan et al. 2024). Maximum Sequence Probability (MSP), CCP (Fadeeva et al. 2024)
and EV Laplacian (Lin et al. (023).
Table compares the effectiveness and run- time across different approaches. While answer- level approaches such as Perplexity, P(TRUE), and SAR require significantly higher computa- tion time (ranging from nearly 10 seconds up more than 180 seconds per sample), our In- ternal Confidence method achieves the best AU- ROC (66.8) with an average running time of only 0.3 seconds. This corresponds to speedups of over 30x to 600x compared to existing baselines. These results demonstrate that Internal Confi- dence achieves competitive performances compared to answer-level uncertainty approaches while being extremely faster, which can be practical choice for tasks requiring longer and more complex answers.
Method
1 AUROC
Time
Speedup
Perplexity Semantic Entropy
65.5
9.8
60.0
151.8
506x x
P(TRUE)
65.2
22.3
74x
Lexical Samilarity
62.4
170.3
567x
SAR
65.7
180.6
x
MSP
68.5
25.1
84x
CCP
64.2
61.7
206
Laplacian
153.9
Intemaal Confidence
66.8
0.3
Table 2: Comparison with answer-level uncer- tainty methods (Qwen-14B on GSM8K).
Notably, the running time for Internal Confidence remains constant, independent of the length of answers. Figure 4 shows that the runtime of the best answer-level approach. SAR, grows with the answer length, reaching nearly 500s for answers over 600 characters. In contrast, Internal Confi- dence achieves large acceleration ratios, with speedups increasing as answers become longer, which demonstrates its scalability and efficiency. See results of other datasets in Table A1
8

---

Published as a conference paper at ICLR 2026
0.65
Ph--300
Avg running time (6)
Internal Confidence
0.60
Liama-88
AUROC
Qwen-14B
0.63
0.62
200
0.60
100
736.2x
1025.0x
1347.6x
1672.1x
x200
200-400
400-600
>600
0.1
0.2
0.3
0.4
0.5 Locality
0.6
0.7
0.8
0.9
Answer length (chars)
Figure 4: Acceleration ratio comparison be- tween answer-level SAR and our Internal Con- fidence.
Figure 5: Impact of locality on validation set performance. We report the average AUROC across the three considered datasets. See details Section D.3
4.4 INTERNAL CONFIDENCE MAKES LLM REASONING MORE EFFICIENT
Recent studies advance LLM reasoning by introducing additional resources, such as using RAG to obtain external knowledge (Lewis et al. 2020) and inference-tine scaling to improve outputs (Snell et al. 2024). However, it is not always necessary to use additional resources, especially for simple queries. Here, we use our proposed Internal Confidence for adaptive inference, determining when to invoke RAG, slow thinking, or model cascading.
We conduct experiments for two scenarios: (1) Efficient (or adaptive) RAG. This assumes that the Internal Confidence can serve as signal of the knowledge gaps of a model. If the score is greater than threshold, the model sufficiently confident to address the query directly using its paramet- ric knowledge. Otherwise, it requires invoking RAG. Existing studies have explore adaptive RAG
through learned classifiers (Jeong et al. 2024 Marina et al. 2025) and inswer-leve uncertainty
approaches (Jiang et al. [2023 Su al. 2024 Yao et al.] Moskvoretski al., 2025), which actively decide whether and when to retrieve documents. However, these approaches require train- ing samples or generating answers measure the uncertainty. In contrast, Internal Confidence method is training-free ffree and significantly faster than answer-level approaches (as shown in Table 2). which can serve
potentially efficient way guide adaptive RAG. We use the TriviaQA dataset for evaluation. This dataset provides web search results for query, which can be used as retrieved contexts for RAG. (2) Model Cascading. This task aims achieve cost- cost-perfoie performance trade-offs by coordinating small and large models (Dohan et al.) 2022; Gupta et al.) 2024). The smaller models are responsible for easy assignments. Whenever they determine that the task exceeds their capabil- ities, it can be delegated to larger model. We use two-model cascade setting with Phi-3.8B and Llama-8B on the TriviaQA dataset. If the Internal Confidence the smaller model is high, we do not invoke the larger model. Otherwise, the hard query is deferred the larger model.
Figure 6 presents the results of applying Internal Confidence scores to efficient RAG (left) and model cascading (right). In both cases, the trade-off region illustrates how adjusting the confidence threshold allows us to balance efficiency and performance by controlling the frequency of external service calls larger model invocations. The optimal point highlights thresholds where additional resource usage can be reduced without sacrificing accuracy. Results across the two tasks further confirm the effectiveness of Internal Confidence in identifying knowledge gaps. Our method offers practical benefits by reducing inference overhead, which can be applied to token-heavy agentic frameworks.
4.5 LOCALITY AFFECTS UNCERTAINTY PERFORMANCE
Our method incorporates attenuated encodings to aggregate probabilities centering around decision point. The locality of the encoding may affect the accuracy of estimated uncertainties. To study the influence of the locality, vary the in Equation obtain encodings with different localities and observe how they affect the estimations. Figure reppots the average AUROC across three datasets and models. The results indicate that the effect of locality depends both the task type and the

---

Published as a conference paper at ICLR 2026
Accuracy Efficient, MAG
1000
70.0
Accuray of Cascading
100
Langer Model Calls (163
Cost
Cest
64
Optimal Point
80
Accuracy Fraction of RAG Calls
67.5
Optimal Point
80
62
03.0
Accuracy
60
82.5
58
40
40
57.5
Trade-off Region
54
55.0
Wado-off Rogion
52
52.5
Scores
Threshold Confidence
(a) Efficient RAG (see the running time in
(b) Model Cascading
Figure lb
We use estimated Internal Confidence to decide whether to invoke RAG. If the Internal Confidence exceeds a threshold, the model answers the query using its parametric knowl- edge. Otherwise, it relies on external knowledge. The plot shows the accuracy of Phi-3.8B on the TriviaQA dataset under this setting. Right: We implement model cascading setting with Phi-3.8B (small) and Llama-8B (large) on the TriviaQA dataset. The Internal Confidence of the smaller model determines whether answers the query or defers to the larger model when confidence is low. The green lines indicate the baseline accuracy achieved by the simple model or complex model.
Figure 6: Left:
model architecture. Although the optimal locality may vary with model and dataset (see details in Section D.3), we find that default setting a (corresponding to Locality 0.7) yields consistently competitive performance that generalize well.
CONCLUSION
In this work, we propose the new notion of query-level uncertainty, which seeks to assess whether model can successfully address query without generating any tokens. this end, we propose the novel Internal Confidence technique, which leverages latent self-evaluation to identify the boundary of a model's knowledge. Extensive experimental results confirm the effectiveness of our approach on both factual QA and mathematical reasoning. Our method is capable of identifying knowledge gaps with a substantially faster speed compared to answer-level approaches. Furthermore, we ap- ply Internal Confidence two practical scenarios of adaptive inference, efficient RAG and model cascading. Our findings reveal that our method can identify two regions: a trade-off region and an optimal point. The former means that one can strike balance between cost and quality by carefully selecting threshold of confidence scores. The latter means that one can reduce inference overhead without compromising performance.
In conclusion, these results highlight Internal Confidence as a strong and general-purpose baseline for estimating query-level uncertainty. While there remains room for refinement, our study can serve as strong baseline for this task, and we hope this study can stimulate future studies in this area.
LIMITATIONS
There are several main limitations of this work. (1) Our proposed query-level uncertainty measure relies on access a model's internal states, which not feasible for fully black-box APIs. (2) We adopt straightforward fixed hyperparameters across all experiments for efficiency and general- izability, but this choice does not yield optimal performance in all settings. As discussed Section D.5 our additional experiments show that the optimal decision center location varies across models and tasks. (3) Although internal confidence can serve as a strong baseline for detecting knowledge boundary, its performance still lags behind answer-level approaches. We hope this work inspires future research on more refined and robust ways to detect the knowledge boundary of foundation models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404.14211 2024.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint drX1v:1610.01644, 2016.
Alfonso Amayuelas, Kyle Wong. Liangming Pan, Wenhu Chen, and William Yang Wang. Knowl- edge of knowledge: Exploring known-unknown. uncertainty with large language models. In Findings of the Association for Computational Linguistics ACI. 2024, pp. 6416-6432, 2024.
Anthropic.
The cost of thinking: Agentic ai, inference economics, and the future of hybrid intel- ligence. https: /med /medii um com @ahi oftp://eaaaaaaaeeeeoooooooooo-o-
agentiic a ai-18 Ui-infeerence-ecconmee cs org.::/ooo.oo........e.eeeooeeen ligencee
1393182abd13 2025.
Amos Azaria and Tom Mitchell. The internal state of LLM knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967-976, 2023.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference on Learning Rep- resentations, 2022.
Chao Chen, Kai Liu, Ze Chen, Yi Gu, Yue Wu, Mingyuan Tao. Zhihang Fu, and Jieping Ye. INSIDE: LLMs' internal states retain the power of hallucination detection. In ICLR, 2024a.
Lihu Chen and Gaöl Varoquaux. What is the role of small models in the LLM era: A survey, arXiv preprint ar arXiv:2409.0668 2024.
Lihu Chen, Gael Varoquaux, and Fabian Suchanek. The locality and symmetry of positional encod- ings. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14313- 14331. 2023.
Lihu Chen, Alexandre Perez-Lebel, Fabian Suchanek, and Gaël Varoquaux. Reconfidencing LLMs from the grouping loss perspective. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1567-1581, 2024b.
Lihu Chen, Adam Dejl, and Francesca Toni. Identifying query-relevant neurons in large language models for long- form texts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. -26604, 2025.
Yung-Sune Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 2021.
Roi Cohen, Konstantin Dobler, Eden Biran, and Gerard de Melo. I don't know: Explicit modeling of uncertainty with an [IDK] token. Advances in Neural Information Processing Systems, 37: html
10935-10958, 2024. URL wwww www.eedicceeeeeeeooo com wttpa/ggsoocooooooooooa1a00
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? Does it matter? Structural safety, 31(2):105-112, 2009.
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A Saurous, Jascha Sohl-Dickstein, et al. Language model cascades. arXiv preprint arXiv:2207.103: 2022.

---

Published as a conference paper at ICLR 2026
Jinhao Duan, Hao Cheng, Shiqi Wang. Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu, Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 5050-5063, 2024.
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, et al. Fact-checking the output of large language models via token-level uncertainty quantification. arXiv preprint arXiv:2403.0400 2024.
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8: 539-555, 2020.
Jiahui Geng, Fengyu Cai, Yuxia Wang. Heinz Koeppl. Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 6577-6595 2024.
Brendan S Gillon. Ambiguity, generality, and indeterminacy: Tests and definitions. Synthese, 85(3): 391-416, 1990.
Daniela Gottesman and Mor Geva. Estimating knowledge in large language models without gen- erating single token. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3994-4019, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint urXiv:2407.21783, 2024.
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024.
Stephen C Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 54(2-3):217-223, 1996.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 110())457-556, 2021.
Soyeong Jeong, Jinheon Back, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-RAG:
Learning to adapt retrieval- augmente large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7029-7043, 2024.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969-7992, 2023.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings the the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Doddss Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.0522 2022.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine M Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large language models must be taught to know what they don't 't know. In The Tiirty- Annual Conference Neural Information Processing Systems, 2024.
12

---

Published as a conference paper at ICLR 2026
Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. Semantic entropy probes: Robust and cheap hallucination detection in LLMs. arXiv preprint arXiv:2406.15927, 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In Proceedings of the Eleventh In- ternational Conference on Learming Representations (ICLR), Kigali, Rwanda, May 1-5 2023.
OpenReview.net. URL
net/ formm? 200 ottpa//aeeaeaa..eeeeooooooooonan
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. tion for -nwllede-intensive NLP tasks. Advances in neural information processing systems, 33: 2020.
genera-
Moxin Li, Yong Zhao, Yang Deng, Wenxuan Zhang, Shuaiyi Li, Wenya Xie, See-Kiong Ng, and Tat- Tat-Seng Chua. Knowledge boundary of large language models: A survey, arXiv preprint arXiv:2412.12472. 2024.
Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04), pp. 605-612, 2004.
Stephanie Lin, Jacob Hilton, and Owain Eyans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings the 60th annual meeting of the association for compulational linguistics (volume 1: long papers), pp. 3214-3252, 2022.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models. Transactions Machine Learning Research, 2023.
Matéo Mahaut, Laura Aina, Paula Czarnowska, Momchil Hardalov, Thomas Mueller, and Lluis Mårquez. Factual confidence of LLMs: reliability and robustness of current estimators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Pupers), pp. 4554-4570, 2024.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2021.
Andrey Malinin, Anton Ragni, Kate Knill, and Mark Gales. Incorporating uncertainty into deep learning for spoken language assessment. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 45-50, 2017.
Potsawee Manakul, Adian Liusie, and Mark Gales. Selfcheckgpt: Zero-resource black-box halluci- nation detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9004-9017, 2023.
Maria Marina, Nikolay Ivanov, Sergey Pletenev, Mikhail Salnikov, Daria Galimzianova, Nikita Krayko, Vasily Konovalov, Alexander Panchenko, and Viktor Moskvoretskii. L.M-independent adaptive RAG: Let the question speak for itself. arXiv preprint arXiv:2505.04253, 2025.
MistralAI. Mistral large: A purposs language model. https::/mistrall.at/
mnssralllaarge 2407/ 2024.
Viktor Moskvoretskii, Maria Marina, Mikhail Salnikov, Nikolay Ivanov, Sergey Pletenev, Daria Gal- imzianova, Nikita Krayko, Vasily Konovalov, Irina Nikishina, and Alexander Panchenko. Adap- tive retrieval without self knowledge? Bringing uncertainty back home. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6355-6384, 2025.
Bálint Mucsányi. Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disentangle- ment: Specialized uncertainties for specialized tasks. Advances in neural information processing systems, 37:50922-51038, 2024.
13

---

Published as a conference paper at ICLR 2026
Ruiyang Ren, Yuhao Wang, Yingqi Qu Wayne Xin Zhao, Jing Liu, Hua Wu, Ji-Rong Wen, and Haifeng Wang. Investigating the factual knowledge boundary of large language models with retrieval augmentation. In Proceedings of the 31st International Conference Computational Linguistics, pp. 3697-3715, 2025.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to tools. Advances in Neural Information Processing Systems, 36:685399- 68551, 2023.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference Learning Representations, 2024.
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. arXiv preprint arXiv:24/2.05565 2024.
Aviv Slobodkin, Omer Goldman, Avi Caciularu, Ido Dagan, and Shauli Ravfogel. The curious case of hallucinatory (un) answerability: Finding truths in the hidden states of over-confident large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3607-3625, 2023.
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.0333 033144, 2024.
Yifan Song, Guoyin Wang. Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. arXiv preprint arXiv:2407.10457 2024.
Asa Cooper Stickland and Iain Murray. Diverse ensembles improve calibration. arXiv preprint arXiv:2007.042066 2020.
Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 11991-13013, 2024.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. WRRLittts://qwen.l.
github. hooblogg///ooooooooo http:///oooggggee....5.. qwen2.5/
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5433-5442, 2023.
Christian Tomani, Kamalika Chaudhuri, Ivan Eytimov, Daniel Cremers, and Mark Ibrahim. Uncertainty-basee abstention in LLMs improves safety and reduces hallucinations. arXiv preprint arXiv:2404. 10960, 2024.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Gaël Varoquaux, Sasha Luccioni, and Meredith Whittaker. Hype, sustainability, and the price of the bigger-is-better paradigm in ai. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, pp. 61-75, 2025.
Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah. Kirill Grishchenkov, al. Bench- marking uncertainty quantification methods for large language models with LM-Polygraph. Transactions of the Association for Computational Linguistics, 13:220-248. 248, 2025.
14

---

Published as a conference paper at ICLR 2026
Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Huimin Wang, Guan- hua Chen, and Kam-fai Wong. Self-de: When to reason and when to act? Self divide-and-conquee for compositional unknown questions. arXiv preprint arXiv:2402.13514 2024.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04388 2024.
Johannes Welbi, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User- seeer-eneerted Text, pp. 94- 106, 2017.
Binghing Wen, Jihan Yao, Shangbin Feng. Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. Know your limits: A survey abstention in large language models. arXiv preprint
1rXi:22407.18418, 2024.
Zhihui Xie, Jizhou Guo, Tong Yu, and Shuai Li. Calibrating reasoning in language models with internal consistency. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024.
Zijun Yao, Weijian Qi. Liangming Pan, Shulin Cao, Linmei Hu, Liu Weichuan, Lei Hou, and Juanzi Li. Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. In Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 27022-27042 2025.
Xunjian Yin, Xu Zhang, Jie Ruan, and Xiaojun Wan. Benchmarking knowledge boundary for large language models: A different perspective on model evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2270- 2286, 2024.
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models say 'I don't 't know'. Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 7106-7132, 2024a.
Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference machine learning, pp. 11117-11128 128. PMLR, 2020.
Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xipeng Qiu. Calibrating the confidence of large language models by eliciting fidelity. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 2979.2979, 2024b.
Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. A survey on test-time scaling in large language models: What, how, where, and how well? arXiv preprint arXiv:2503.24235, 2025.
A FUNDAMENTAL CONCEPTS
A.1 ALEATORIC AND EPISTEMIC UNCERTAINTY
Uncertainty in machine learning is commonly categorized into two main types: aleatoric and epis-
temic uncertainty (Horal 1996; Der Kiureghian & Ditlevsen, 2009 Hüllermeier & Waegeman, 2021]. These distinctions are often overlooked in the context of LLM uncertainty estimation. Aleatoric uncertainty arises from inherent randomness in the data, such ambiguous inputs or
15

---

Published as a conference paper at ICLR 2026
Known Quries
Ouries Quries
Fraction of Samples
Quiles
Fracion of Samples
5
Fraction Samples
2.5
1.5
0.5
Justeall mill. 0.2
0.2
0.4
0.6
0.
10
6.0
3.4 56
1.0
0.3
0.4
0.5
U.5.
0.7
0.8
0.9
Internal Confidence
Internal Confidence
Internal Confidence
TriviaQA
(b) SciQ
(c) GSM8K
Figure Al: Distinguishing between answerable and non-answerable queries using Internal Confi- dence for Phi-3.8B.
conflicting annotations. This type of uncertainty is irreducible, as it reflects intrinsic noise in the in- put data. In contrast, epistemic uncertainty stems from lack of knowledge, often due to insufficient training data and limited model capacity. Unlike aleatoric uncertainty, epistemic uncertainty is ducible with additional data or advanced modeling. In this work, we focus specifically on epistemic uncertainty, with the goal of evaluating whether an LLM possesses sufficient knowledge to answer given query. For evaluation, we adopt factual QA and mathematical reasoning benchmarks, which are designed have clear-cut answers. We assume these datasets are well-curated to minimize alcatoric uncertainty, such ambiguous questions and inconsistent labels. However, acknowl- edge that residual ambiguity may persist, given the inherent nature of linguistic ambiguity (Gillon 1990) and the difficulty of fully disentangling aleatoric from epistemic uncertainty (Mucsányi al. 2024 We treat such aleatoric effects as negligible for the purposes of focusing on epistemic uncer- tainty.
A.2 UNCERTAINTY AND CONFIDENCE
In the context of LLMs, the terms uncertainty and confidence are often used interchangeably (as antonyms). However, the two concepts have subtle differences. As noted by Lyn al. (2023), un- certainty is holistic property of the entire predictive distribution, while confidence refers to the model's estimated confidence level associated with a specific answer. For example, given a query = =Whatt is the capital of France". estimating uncertainty conceptually requires the distribution over all plausible answers, e.g., Paris, Toulouse, Lyon, etc., as operationalized by the semantic entropy framework (Kuhn et al.) (023), which clusters semantically equivalent outputs before com- puting entropy. In contrast, the conditional probability P(Y Paris x) can serve as an indication of confidence here, reflecting how strongly the model supports that particular response. Given that it is unfeasible to enumerate all possible responses in our context of query-level uncertainty, we pragmatically treat uncertainty and confidence as antonyms.
B PROMPT
We use the following prompt template for experiments. The query x consists of both prompt and question tokens.
You are helpful assistant that assesses whether you can provide an accurate response a question. Respond only with 'Yes" or 'No' to indicate whether you are capable of answering the following question. { (Examples)
(Imput Input Question).
C BASELINE DETAILS
We adapt existing answer-level methods to quantify the pre- tre-generation uncertainty, e.g., logit-based uncertainty. Given query (including the prompt) for each token ((x, (zn)) by performing a forward pass. (1) The baseline Max( log p) measures
(#1..... ZN), obtain probability
the query's uncertainty by assessing the least likely token in the query (Manakul et al. 2023) (2)
16

---

Published as a conference paper at ICLR 2026
TriviaQA
SciQ
GSMBK
Phi-3.8B
Liama-88
Qwen-14B
0.60
0.55
0.50
2
3
k-shot
k-shot
k-shot
Figure A2: Impact of the number in-context-learming examle pairs on validation set performance.
Predictive Entropy is defined as the entropy over the entire query token sequence (Malinin & Gales
2021
N PE(x) = Σ
log P(rn (ten)
(A.1)
(3) Min-K Entropy combines the ideas of Max(- log p) and predictive entropy, by selecting the top- K tokens from the query with the minimum token probability (Shi et al.) (024).. (4) Attentional Entropy modified version of the predictive entropy that considers weighted sum
AE(x) Σ On log P( Ieen),
(A.2)
where Os, are the attentional weights for tokens Ine The intuition here is that tokens contribute to the semantic meanings in different ways, such that we should not treat all tokens equally (Duan et al. 2024). (5) Perplexity reflects how uncertain model is when predicting the next token:
PPL = exp (- 1 Σ log P(æ,, x<e))
exp
(A.3)
(6) Internal Semantic Similarity measures the average similarity among hidden states of different
layers (h ..... hy which is inspired by lexical similarity (oomicheva al. 2020). (7) P(YEs) the probability of self-evaluation, as defined Equation (8) Internal Confidence (w/ naive avg)
simplified variant of our proposed Internal Confidence. The difference is that compute
naive average to aggregate all scores.
Mehool
AUROC
Time
Speodup
D ADDITIONAL EXPERIMENTS
TriviaQA
Perplesity Enmmpy
75.1
5.6
28%
D.1 CALIBRATION PERFORMANCE
Lexical Similarity
P(TRUK)
77.2
142.3
712x
113x
SAR
76.5
160.8
804
Figure A1 illustrates the distributions of Internal Confidence for answerable versus non-answerable queries across three drivistssTriviiQAA, SciQ, and GSM8K-using Phi-3.8B. all cases, answerable queries (green) exhibit noticeably higher Internal Confidence, with distributions concentrated toward the upper end of the confidence range. In con- trast, non-answerable queries (blue) show substan- tially lower Internal Confidence, typically forming broader left-shifted distributions. This clear sep- aration demonstrates that Internal Confidence effec- tively distinguishes between seen and unseen inputs, supporting its usefulness as an internal signal for as- sessing familiarity and reliability within the model.
73.3
37.6
188
EV Laplacion
78.1
12.4
62x
Internal Confidence
71.9
SciQ
Perplesity Semsatic Entropy
71.5
12.9
65x
66.3
132.8
664
P(TRUE)
Sintlarity
826
SAR
70.5
165.7
829
MSP
70.3
3.85
CCP
Laplacian
65.7
23.6
118x
Internal Confidence
62.6
0.2
Table A1: Comparison of query-level Internal Confidence with answer-level uncertainty methods (Qwen-14B on TriviaQA and SciQ).
17

---

Published as a conference paper at ICLR 2026
TriviaQA
SciQ
GSMBK
Phi-3.8B
Llama-88 86
Qwen-148
0.7
AUC
0.6
0.6
0.4
0.6
Locality
Locality
Locality
Figure A3: Impact of locality on validation set performance.
D.2 INTERNAL CONFIDENCE DOES NOT RELY ON IN-CONTEXT LEARNING
Figure A2 shows the effect of the number of in-context learning example pairs (k-shot) on model performance across three datasets and models. Here, we randomly select pairs of positive and negative samples. We plot the AUROC as function of k-shot values from to 5. Overall, Llama- 8B and Qwen-14B maintain relatively stable performance with slight improvements as k increases, while Phi-3.8B exhibits more fluctuation, especially on TriviaQA. These results suggest that benefit of additional in-context examples varies across both models and datasets. Therefore, our Internal Confidence can obtain strong performance even without in-context learning from examples, which can reduce the computational cost.
D.3 IMPACT OF LOCALITY
presents the impact of locality on AUROC performance across three datasets (TriviaQA, SciQ, GSM8K) and three models (Phi-3.8B, Llama-8B, Qwen-14D5 14B). For Phi-3.8B, AUROC im- proves gradually with increasing locality across all datasets, with TriviaQA exhibiting consistently higher discriminability than SciQ and GSM8K. For Llama-8B, the performance remains fairly stable across different locality values, showing only minor fluctuations, particularly for SciQ and GSM8K. For Qwen- 14B, the AUROC increases with the locality for all datasets up to a certain point, after which it either plateaus slightly declines; this trend is most evident for GSMSK.
Figure A3
Locality has a non-trivial effect on the performance of Internal Confidence, and its optimal value varies slightly by model and dataset. Phi-3.8B and Qwen-14B benefit more clearly from tuning locality, while Llama-8B appears more robust to changes. Overall, high locality values often yield competitive optimal performance.
D.4 INTERNAL. CONFIDENCE CAN BE GENERALIZED TO MORE CHALLANGING DATASETS
To validate whether our proposed internal confidence can be generalized to more challenging tasks, we conduct experiments on three additional datasets: (1) SimpleQA (Wei et al.) 2024). This is a benchmark that evaluates the ability of language models to answer short, fact-relevant questions, which is less likely to be contaminated by the pre-training stage. (2) MuSiQue (Trivedi et al.]
2022). This dataset that requires proper multihop reasoning, which is more difficult and harder to cheat via disconnected reasoning. (3) TruthfulQA (Lin et al. 2022). This is benchmark to measure whether a language model is truthful generating answers to questions. The authors crafted questions that some humans would answer falsely due to false belief or misconception. To perform well, a model has avoid generating false answers learned from imitating human texts. For each dataset, we use the validation or test partition for comparison, which contains a reasonable number samples (2-4K). For the first two datasets, we apply the default configuration internal confidence. Regarding the TruthfulQA dataset, we observe that the task exhibits distinct decision center, which tends to appear in the middle layers rather than the upper layers across all three model architectures. For example, at layers 9, 7, and 23 for Phi-3.8B, Llama-8B, and Qwen-148 respectively. To consider this, we learn the decision center specifically for TruthfulQA using 100-sample validation set. The overall
100-sample held-out validation set, the decision centers appear
18

---

Published as a conference paper at ICLR 2026
31
31
46
1.10
10
41
105
36
Model Layers
21.
21
31
40
26
Model
11-
16
11
1.
3
2
Query Tokens
Query Tokens
Query Tokens
(a) Phi-3.8B
(b) Llama-8B
(c) Qwen- 14B
Figure A4: Learned decision centers of Phi-3.8B.
31
46
26
15
26
36
Model Layers
21
452
355
31
3.55
16
3M
16
352
21
359
3.05
34
3
2 3
4
2
Query Tokens
Query Tokens
Query Tokens
(a) Phi-3.8B
(b) Llama-8B
(c) Qwen- owen-14B
Figure A5: Learned decision centers of Llama-8B.
results shown in Table A2
We observe that our internal confidence outperform other query-level uncertainty consistently across datasets and architectures.
D.5 THE OPTIMAL DECISION CENTER VARIES ACROSS MODELS AND TASKS
We use the top right position as default decision center, which offers a training free and pragmatic solution, but it is the optimal center. We conduct experiments study the leamned decision center across different models and tasks. Figure A4 Figure A5 and Figure A6 show the locations of decision centers. We can observe that the center tends to appear at the top right place for TriviaQA and SciQ while the math reasoning task of GSM8K has distinct behavior. The center is located in the lower layers. Although the current default center (top right) is sub-optimal, it offers training- free, strong baseline, which can be generalized across different applications.
E USE OF LARGE LANGUAGE MODELS
In this work, we employed LLMSs in two complementary ways. First, LLMs were used to aid and polish the writing of the manuscript. This includes grammar checks and sentence polishing, mainly for readability and clarity. Second, LLMs were leveraged for retrieval, particularly in the section of related work. By querying LLMs to retrieve relevant references, sought identify additional references and obtain comprehensive coverage of prior research.
19

---

Published as a conference paper at ICLR 2026
31
46
10
10
26
Model Layers
21
31
16-
21
16
14
3.45
3
Query Tokens
Query Tokens
Query Tokens
(a) Phi-3.8B
(b) Llamm-8B
(c) Qwen-14B
Figure A6: Learned decision centers of Qwen-14B.
SimpleßA
MusiQue
Truthfu1gA
Avg
Method
AUROC PRR
ECE
AUROC
PRR
ECE
AUROC
PRR ECE
AUROC
1 PRR
ECE
Phi-3.88
Max(- log p)
50.5
53.2
4.7
-1.3
3.0
Predictive Entropy
54.0
5.9
65.6
29.6
55.7
15.6
58.4
17.0
Min-K Entropy
13.2
59.9
21.9
55.1
13.9
16.3
Attentional Entropy
50.1
5.1
54.7
11.6
52.4
8.3
52.4
8.3
Perplexity
52.4
5.8
55.2
7.1
54.1
53.9
5.6
P(YES)
61.3
17.8
18.8
65.4
29.8
9.5
39.6
-16.2
27.1
58.4
10.5
18.5
P(YES) (naive ovg)
59.8
17.8
69.9
65.5
29.5
63.2
49.3
-2.0
25.5
58.2
15.1
$2.9
Internal Confidence
22.7
Llama-8B
Max( - log p)
50.1
-2.9
53.2
6.3
52.4
4.1
51.9
2.5
Predictive Entropy
Min. K Entropy
49.8
0.1
56.0
15.2
57.8
21.0
54.5
12.1
Attentional Entropy
57.4
7.7
Perplexity
50.1
-3,8
54.2
5.9
54.3
4.8
52.9
2.3
P(YES) (top right)
53.6
5.2
78.9
64.1
27.3
74.5
43.3
-11.9
55.7
53.7
6.9
69.7
P(YES) (naive
54.9
5.8
28.5
63.2
31.9
18.5
47.0
3,4
55.0
13.2
16.8
Internal Confidence
55.6
11.6
67.4
64.3
29.8
74.4
63.2
26.8
15.4
61.0
22.7
52.4
Qwen-148
Max(- log
50.8
-1.2
52.5
6.8
51.0
51.4
3.4
Prodictive Entropy
50.5
1.3
53.8
9.4
59.9
21.1
54.7
10.6
Entropy
Atlentional Entropy
48.7
-2.9
54.9
13.2
50.7
3.2
51.4
4.5
Perplexity
P(YEs) (top right)
55.6
11.4
35,5
57.7
14.4
22.4
42.6
-19.0
53.4
52.0
2.3
37.1
P(YES) (naive ovg)
57.6
16.6
58.4
16.9
49.7
-3.5
8.6
55.2
10.0
Internal Confidence
10.3
57.6
14.6
5.4
164
0.5
57.2
14.7
Table A2: Additional results of different query-level uncertainty estimation methods.
20


Reviews:
[{"review_id": "YunW1UpHD5", "reviewer": "Reviewer_ZpJR", "summary": "The paper proposes a training-free, pre-generation measure of query-level uncertainty called Internal Confidence (IC). Instead of generating an answer and then judging it, IC probes the model’s hidden states at many token positions nnn and layers lll using the model’s unembedding rows for {YES, NO}. These probe scores are then aggregated with decay weights around a “decision center” (empirically near the last token & last layer) to produce a single confidence score. The paper motivates the center by showing that discrimination between known/unknown queries often peaks at an interior token×layer location, then ensembles nearby positions via an “attenuated encoding”. Experiments on factual QA and math reasoning claim better AUC/PRR and calibration (ECE) than query-level baselines, and show major speedups versus answer-level methods (e.g., SAR), enabling applications like cost-aware RAG routing and model cascading.", "questions": "I appreciate the training-free setup and the promise of OOD-robust, pre-generation uncertainty. However, the evidence feels incomplete primarily because key, state-of-the-art baselines are missing. If the authors add stronger baselines (e.g., CCP, MSP, TokenSAR, DegMat/EigVal, RAUQ, robust verbalized-uncertainty) and evaluate on harder datasets, I could  revise my view. I also recommend reframing the main claim around efficiency (with explicit token/latency budgets) and presenting accuracy as on-par with SOTA where that is borne out.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Pre-generation, single-pass signal. IC avoids generating answers and extra prompts; it’s computed from internal states with one forward pass, which is appealing for latency/cost.\n\nCenter-weighted ensembling across tokens×layers. The “decision center” idea is well-motivated by heatmaps showing the best separator isn’t always exactly the last position; the attenuated average reduces variance while keeping locality.\n\nPractical routing use-cases. Clear demonstrations for RAG triggering and small→large model cascading with thresholding on IC (trade-off/optimal regions).", "weaknesses": "1) Baselines. It is puzzling that the baselines differ between Tables 1 and 2; they should be consistent to allow a fair comparison of self-knowledge and efficiency across models and tasks. Moreover, the state-of-the-art claim cannot be substantiated without including a broader set of strong baselines. Following the recent TACL benchmark on uncertainty quantification [1] , at minimum the evaluation should cover half of these baselines (that outperform SAR as a top baseline from the paper): CCP, Maximum Sequence Probability (MSP), EigValLaplacian, Lexical Similarity (ROUGE-L), DegMat, TokenSAR, representative verbalized-uncertainty methods, and the recently proposed RAUQ. For references and straightforward replication, please see LM-Polygraph.\n\n\n2) Datasets. The current dataset selection provides limited coverage of multi-hop and genuinely challenging questions. Are the proposed methods generalizable to such settings? TriviaQA and SciQ contain many single-fact lookups and shallow cues, which can inflate confidence estimates. Please consider adding SimpleQA [2]  (challenging for LLMs and less likely to be contaminated by pretraining), TruthfulQA[3]  (tests suppression of “popular but false” answers), and multi-hop benchmarks such as MuSiQue[4] and 2WikiMultihopQA [5].\n\n        For mathematics, GSM8K largely features short, templated chains, where uncertainty may     correlate with sequence length or final-step carry errors rather than true multi-step epistemic gaps. We recommend evaluating on GSM8K-Hard[6], which require longer chains and introduce more realistic failure modes.\n\n3) Adaptive RAG. What you refer to as “efficient RAG” is a fast-moving area with a substantial body of adaptive RAG/adaptive retrieval baselines. To substantiate the contribution, it’s important to compare against representative prior work—for example DRAGIN [7], FLARE [8], SEAKR[9],  AdaptiveRAG[10], and recent approaches that leverage uncertainty [11]  and LLM-independent features [12]. Including these would better situate your method within the existing literature.\n\n4) Access assumptions. IC needs logits / unembedding and hidden states. That’s fine for open-weights models, but not available for black-box APIs, where answer-level methods (e.g., P(True)) might be more deployable. Can the method work if black-box models provide access to the last layer, some per-output-token log-probabilities?\n\n\n5) Definition of “knowledge boundary.” The authors evaluate answerability under greedy decoding as a proxy; this can misclassify queries where non-greedy decoding would succeed, so “known vs unknown” labels are not absolute.\n\n\nTypos:: Notation mismatch: “w = 1.0” vs Eq. (4). In §4.5 the text says “vary the w in Equation 4 … default w = 1.0 (Locality ≈ 0.7).” But Eq. (4) defines the attenuation with parameter α controlling locality. This reads like a symbol mix-up; likely they meant α = 1.0. Please unify notation.\n\n\n[1] Vashurin, R., Fadeeva, E., Vazhentsev, A., Rvanova, L., Vasilev, D., Tsvigun, A., ... & Shelmanov, A. (2025). Benchmarking uncertainty quantification methods for large language models with lm-polygraph. Transactions of the Association for Computational Linguistics, 13, 220-248.\n\n[2] Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., ... & Fedus, W. (2024). Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368.\n\n[3] Lin, S., Hilton, J., & Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.\n\n[4] Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). ♫ MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10, 539-554.\n\n[5] Ho, X., Nguyen, A. K. D., Sugawara, S., & Aizawa, A. (2020). Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060.\n\n[6] Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., ... & Neubig, G. (2023, July). Pal: Program-aided language models. In International Conference on Machine Learning (pp. 10764-10799). PMLR.\n\n[7] Su, W., Tang, Y., Ai, Q., Wu, Z., & Liu, Y. (2024). DRAGIN: dynamic retrieval augmented generation based on the information needs of large language models. arXiv preprint arXiv:2403.10081.\n\n[8] Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., ... & Neubig, G. (2023, December). Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 7969-7992).\n\n[9]  Yao, Z., Qi, W., Pan, L., Cao, S., Hu, L., Liu, W., ... & Li, J. (2024). Seakr: Self-aware knowledge retrieval for adaptive retrieval augmented generation. arXiv preprint arXiv:2406.19215.\n\n[10] Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. C. (2024). Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv preprint arXiv:2403.14403.\n\n[11] Moskvoretskii, V., Lysyuk, M., Salnikov, M., Ivanov, N., Pletenev, S., Galimzianova, D., ... & Panchenko, A. (2025). Adaptive retrieval without self-knowledge? bringing uncertainty back home. arXiv preprint arXiv:2501.12835.\n\n[12] Marina, M., Ivanov, N., Pletenev, S., Salnikov, M., Galimzianova, D., Krayko, N., ... & Moskvoretskii, V. (2025). LLM-Independent Adaptive RAG: Let the Question Speak for Itself. arXiv preprint arXiv:2505.04253.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dAl5faRWOj", "reviewer": "Reviewer_wAg8", "summary": "This paper proposes Query-Level Uncertainty (QLU) for LLMs: a training-free method that estimates whether the model is likely to answer a query correctly before any generation. The core signal Internal Confidence (IC) aggregates self-evaluation logits (YES/NO) across layers $\\times$ tokens using a locality-aware attenuation around a fixed decision center (last token, last layer). Experiments on factual QA (TriviaQA, SciQ) and math reasoning (GSM8K) across multiple open models show that IC outperforms post-hoc, answer-dependent baselines in ranking “known vs unknown” queries, while being faster.", "questions": "Please refer to the previous weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Framing “knowledge boundary” as pre-answer uncertainty is useful for agentic pipelines (RAG, slow-thinking, model cascades).\n* Training-free and fast. Single forward pass speed largely independent of answer length, high leverage for long-form tasks and tool-heavy agents.\n* IC uses only standard hidden states + unembedding, attenuation over layers and tokens captures the information where the “can I answer?” signal concentrates.\n* Fixed decision center and a single locality hyper-parameter yield out-of-the-box generalization without per-dataset tuning.\n* The paper demonstrates adaptive inference use cases: gating RAG and model cascading to reduce cost at similar accuracy.", "weaknesses": "1. Operational definition of “knowledge boundary.” The binary label is tied to greedy decoding success. This could confuse capability with decoding heuristics and underestimate the success of non-greedy decoding. Sensitivity analysis of decoding strategies would help strengthen these claims.\n2. IC requires hidden states and full model access. Many production black-box APIs don’t expose them.\n3. The method fixes the decision center (last-layer, last-token) and selects a single decay schedule. A more complete ablation including alternative centers, learned decay parameters, or token-type weighting, would reveal the robustness–complexity frontier.\n4. Baselines could include some selective prediction and abstention methods (e.g., calibrated confidence, dedicated refuse-to-answer training). This will allow IC to be compared with the strongest alternatives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "71M3QCk4SG", "reviewer": "Reviewer_hjve", "summary": "This paper introduces Query-Level Uncertainty, which estimates whether an LLM can answer a query before generating tokens, addressing efficiency and trustworthiness concerns. The proposed Internal Confidence method aggregates P(YES) signals across layers and tokens using attenuated encoding weights, obtaining calibrated query-level uncertainty in a single forward pass. Experiments on factual QA and mathematical reasoning tasks show improvements over adapted answer-level methods. Applications to efficient RAG and model cascading demonstrate practical benefits with 30-600× speedups over answer-level approaches.", "questions": "1. Why is P(YES) specifically the right formulation? Have you tried other self-evaluations (e.g., confidence level on a 1-10 scale)?\n\n2. How does the method perform on adversarial inputs where models give confident wrong answers?\n\n3. Can you provide correlation analysis between Internal Confidence scores and actual greedy decoding accuracy?\n\n4. How sensitive are results to the specific threshold used for deciding \"know/don't know\"?\n\n5. For the decision center analysis (Figure 3b), why does performance peak at h_5^(27) rather than final layer?\n\n6. Have you tested on models with different architectures (MoE, retrieval-augmented)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Query-level uncertainty is a genuinely different and practical perspective from existing answer-level methods. Rather than generating long answers to assess uncertainty, the method predicts answerability before token generation, directly addressing efficiency bottlenecks in real-world systems. The Internal Confidence method is elegantly simple—it aggregates P(YES) signals across layers and tokens with attenuated encoding weights, producing calibrated uncertainty in a single forward pass without requiring training.\n\nThe empirical results are strong and consistent. The method shows improvements over adapted baselines across three diverse datasets and works reliably across different model sizes (Phi-3.8B, Llama-8B, Qwen-14B). More impressively, it achieves 30-600× speedups compared to answer-level approaches like Semantic Entropy and SAR, making it genuinely deployable. Ablation studies systematically examine the impact of locality and hyperparameters, providing confidence in design choices.\n\nThe experimental design is rigorous with proper out-of-domain evaluation demonstrating robustness to domain shift. The applications to efficient RAG and model cascading are immediately actionable, with identified \"optimal points\" that practitioners can use directly. The paper is well-written with clear motivation and effective visualizations that make the approach accessible.", "weaknesses": "The theoretical justification is weak. While P(YES) seems intuitive as a proxy for query answerability, the connection to actual answering capability is assumed rather than proven. Similarly, attenuated encoding is presented as effective, but other aggregation schemes aren't systematically compared to justify this choice. The \"decision center\" concept, which is central to the method, lacks theoretical grounding—the paper doesn't explain why the top-right position is optimal or how it should adapt across different models and tasks.\n\nThe evaluation scope is narrow and raises generalization concerns. Experiments are limited to factual QA and mathematical reasoning across just three datasets with 10K samples each. There's no evaluation on open-ended tasks, creative writing, code generation, or other generation types. Additionally, the method assumes greedy decoding for determining ground truth, which may not reflect real model behavior under beam search or sampling-based decoding strategies.\n\nGround truth definition is somewhat arbitrary and validation is incomplete. Correctness is defined solely by greedy decoding accuracy, and the paper lacks comparison against human-annotated \"answerability\" judgments. It remains unclear whether the model's internal uncertainty truly reflects knowledge or is confounded by input formatting, instruction phrasing, or other factors. The paper also doesn't analyze systematic biases—does the method correlate with question difficulty, answer length, or other confounds?\n\nThe method's claims of being \"training-free\" are overstated. While no training is required, hyperparameter selection (α value and decision center location) depends on validation data, and cross-dataset transfer shows these vary by task and model. The paper uses fixed hyperparameters across all settings despite acknowledging they're suboptimal, and the inconsistency between task-specific optimal centers and the fixed top-right choice isn't well justified.\n\nBaseline comparisons are limited to adaptations of answer-level methods, with no comparison to other query-level uncertainty approaches or simpler heuristics like maximum token probability of the query. Code availability isn't mentioned, and specific prompts and model configurations lack complete documentation, hampering reproducibility and follow-up work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "THxfQMvGKz", "reviewer": "Reviewer_pkEN", "summary": "This paper proposes a training-free, pre-generation method Internal Confidence to investigate the knowledge boundaries of LLMs and reduce inference costs.\nThe authors prompt LLMs with a yes–no question to self-assess whether they can answer a given query before generating an answer, and they adopt layer-level consistency to measure the reliability of the answers.\nExperiments show some efficiency improvements over simple baselines.", "questions": "- \"We define a query as being within the model’s knowledge boundary if the LLM can produce a correct answer under greedy decoding, i.e., by selecting the highest-probability token at each step without sampling.\" --- I’m not sure how or if the answers’ correctness correlates with the model's knowledge boundary. Are there any previous works that support this statement?\n\n- Why introduce the unmapping weights instead of directly using the last-layer pre-activation logits for a clearer description? Did you modify the unmapping weights to keep only the rows corresponding to Yes and No tokens for computing the logits?\n\n- Why introduce Lines 211--235? Are the parameters theta retrained or from external layers? Why not simply state \"pre-/post-activation output of each Transformer block\"?\n\n- It is unclear what the word \"query\" refers to in different contexts. In general, it seems to indicate the model's prompt; but in the example in Lines 194–195, it seems to refer to the question; and in the discussion in Lines 237–245, both meanings do not make sense, and it seems \"query\" here refers to both the prompt and the model's answer. This is extremely confusing.\n\n- In Figure 3 and Lines 237--245, it is unclear which tokens these probabilities/scores are associated with. Suppose these are answer tokens; then we may expect these results, since the last token is highly likely to be EOS, so the next-to-last token is the actual Yes/No token. If not, I’m not sure how to interpret these results. In either case, the discussion does not carry much useful information.\n\n- Lines 253--258: I don't get the value of this paragraph. Equation (3) looks like a simple weighted average, so why form it into a \"hierarchical aggregation\" process?\n\n- Equations (4) and (5): How is delta related to epsilon? How is locality related to the weights w?\n\n- When calculating the runtime, is the time for generating the actual answers included? Why not include predictive entropy in Table 2? Why not include length-normalized predictive entropy in Tables 1 and 2? Semantic aggregation methods such as Semantic Entropy and SAR are not designed for a task with a deterministic answer, such as GSM8k; why include them in Table 2? Why not use TriviaQA?\n\n- Line 376: Area under what curve? I know the answer, but it may confuse others who are not so familiar with UQ.\n\n- Line 315: What is the necessity of evaluating the reasoning steps for GSM8k? What would happen if we do not do so?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The idea of identifying an LLM’s knowledge boundaries before allowing it to generate answers or take agentic actions is both interesting and impactful.\nAs the paper demonstrates, when used properly, such methods can significantly reduce the costs of applying LLMs to downstream tasks.", "weaknesses": "- The method description is confusing, with some simple concepts made complex and other critical details missing.\n- The experimental results are not sufficiently robust.\n- The article is ambiguous throughout.\n- Although not required, the authors are encouraged to discuss the limitations of their work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)