Reasoning Models Better Express Their Confidence
Authors: Dongkeun Yoon
Seungone Kim
Sohee Yang
Sunkyoung Kim
Soyeon Kim
Yongil Kim
Eunbi Choi
Yireun Kim
Minjoon Seo
Venue: NeurIPS 2025 poster ()
Keywords: Reasoning Models
Confidence Estimation
Calibration
Large Language Models
Primary area: deep_learning
TL;DR: We show that slow thinking behaviors elicited in the chain-of-thought (CoT) process of reasoning models help large language models (LLMs) express their confidence more accurately.

Abstract: Despite their strengths, large language models (LLMs) often fail to communicate their confidence accurately, making it difficult to assess when they might be wrong and limiting their reliability. In this work, we demonstrate that reasoning models that engage in extended chain-of-thought (CoT) reasoning exhibit superior performance not only in problem-solving but also in accurately expressing their confidence.
Specifically, we benchmark six reasoning models across six datasets and find that they achieve strictly better confidence calibration than their non-reasoning counterparts in 33 out of the 36 settings. Our detailed analysis reveals that these gains in calibration stem from the slow thinking behaviors of reasoning models (e.g., exploring alternative approaches and backtracking) which enable them to adjust their confidence dynamically throughout their CoT, making it progressively more accurate. In particular, we find that reasoning models become increasingly better calibrated as their CoT unfolds, a trend not observed in non-reasoning models. Moreover, removing slow thinking behaviors from the CoT leads to a significant drop in calibration. Lastly, we show that non-reasoning models also demonstrate enhanced calibration when simply guided to slow think via in-context learning, fully isolating slow thinking as the source of the calibration gains.

Reasoning Models Better Express Their Confidence
Dongkeun Yoon
Seungone Kim
Sohee Yang*
Sunkyoung Kim²
Soyeon Kim²
Yongil Kim
Eunbi Choi²
Yireun Kim?
Minjoon Seo
KAIST
LG Al Research
CMU
*UCL
dkyoon@kaist.ac.kk
Abstract
Despite their strengths, large language models (LLMMs) often fail communi- cate their confidence accurately, making it difficult to assess when they might be wrong and limiting their reliability. this work, we demonstrate that reasoning models that engage in extended chain-of-thought (CoT) reasoning exhibit supe- rior performance not only problem-solving but also in accurately expressing their confidence. Specifically, we benchmark six reasoning models across six datasets and find that they achieve strictly better confidence calibration than their non-reasoning counterparts in 33 out of the 36 settings. Our detailed analysis reveals that these gains in calibration stem from the slow thinking behaviors reasoning models (c.g., exploring alternative approaches and backtracking) which enable them to adjust their confidence dynamically throughout their CoT,, mak- ing it progressively more accurate. In particular, we find that reasoning models become increasingly better calibrated as their CoT unfolds, a trend not observed in non-reasoning models. Moreover, removing slow thinking behaviors from the CoT leads to a significant drop in calibration. Lastly, we show that non-reasoning models also demonstrate enhanced calibration when simply guided slow think via in-context learning, fully isolating slow thinking as the source of the calibration gains
Introduction
A persistent weakness of large language models (LLMs) is their tendency to sound confident even when they are wrong [3 45). This overconfidence threatens their reliability, especially in high-stakes
scenarios 332 49
Meanwhile, recent reasoning models like OpenAl's ol (30) and Deepseek-R1 4
have demonstrated strong problem-solving capabilities through chain-of-thoughh (CoT) 43 eliciting
slow thinking behaviors such exploring alternative approaches and verifying their answers 24 6].
Yet, it remains underexplored whether such slow thinking behaviors also help LLMs to "know what they know" [16] and accurately communicate the limits of their knowledge.
To this end, we present an extensive study of reasoning models' ability accurately estimate and express confidence in their output; a process known as verbalized confidence estimation [40] 7]. We demonstrate that reasoning models exhibit superior confidence calibration compared non-reasoning models, and that this improvement stems from slow thinking behaviors, which allow LLMs to dynamically adjust their confidence throughout the reasoning process. For instance, as shown in Figure I the model's confidence increases when it first verifies its answer ("Oh, and he also created
Wook done during internship at LG Al Research. Our code is available at at/nttss//ggitee htt/://tt....ooaeat/t//////aaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

100
Question
Who's the Athenian reformer that set up the council of 500?
Confidence (%)
90
Model CoT
80
(2)
(4)
Hmm, remember that Athens had several important reform- ers, like Solon, Cleisthenes, and Pericles. maybe wasn't t Solon. Then there's Cleisthenes. (1) Oh, and he also created the council 500, which was (2) Pericles came later,
70
(3) but don 't think he was the one who established the council of 500. (4) So putting all together, Cleisthenes is the reformer who don't think I'm mixing up any details here.
(1)
(3)
60
100
200
300
400
CoT Progress (in tokens)
Figure 1: RI-Distill- Sestill--wen-322 32B dynamically refines its confidence throughout CoT (left) as it engages in various slow thinking behaviors (right). We collect the model's answer and confidence at each token position by appending "</think> AnAnswer:" "</think>\nAnswer:: to terminate the reasoning process early. Note that the model's answer is consistent and correct ("Cleisthenes") at all points, while the confidence fluctuates. For visual clarity, the data was smoothed using Butterworth low-pass filter. See Appendix A.4 for the full untruncated CoT..
the council of 500"), and decreases when it considers alternative ("Pericles came later"). The confidence then rises again as it rejects the alternative ("but don't 't think he was") and reaches its highest level with further verification ("I don't think I' I'm mixing up any details").
We begin by benchmarking six reasoning models derived from four backbone LLMs against their non-reasoning, instruction-tuned counterparts across two knowledge-focuse focused datasets (TriviaQA and NonambigQA) [13] 23 [7] and four reasoning- inteensie datasets (subsets of SuperGPQA and MMLU-Pro) 42 21). In 33 out of 36 settings, reasoning models strictly outperform their non- reasoning counterparts across all measured calibration metrics. Notably, reasoning models achieve knowledge-focune
better calibration non-reasoning models. This highlights that the calibration gains are not simply derived from superior task performance.
datasets, despite having comparable task accuracy to
We then conduct detailed analysis to support our claim that the calibration gains reasoning models stem from slow thinking, which allows them to dynamically adjust and correct their confidence throughout the reasoning process. (1) We measure changes in calibration metrics throughout the CoT process and observe steady, gradual improvement for reasoning models, with statistically significant trends (p < 0.05). Non-rcasoning models show no significant trend and in some cases, surprisingly, even exhibit a worsening pattern. (2) In our ablation study, we find that the non-lineas structure of slow thinking (6. such as the model's ability to explore alternatives and revise its reasoning, plays a critical role in improving calibration. Conversely, prompting the model to explicitly reason about its confidence yields only limited gains in calibration.
(3) Lastly, we observe that non- non-oresnning models also benefit from slow thinking when simply guided via in-context learning [5], supporting claim that these improvements arise from the slow thinking process itself, rather than from some other factors inherent to reasoning models.
2 Related Work
Confidence in LLMS Confidence refers to the model's estimated probability that its answer is
correct, while calibration measures how well this confidence aligns with actual correctness [8] In
other words, well-calibrated model's confidence is reliable indicator of prediction correctness [27
Therefore, assessing an LLM's contidence and enhancing calibration enables more reliable and
trustworthy models (7], supporting wide range of applications such as hallucination detection 22
41), ambiguity detection 12]. uncertainty-guiee data retrieval [4, and uncertainty- LLM agents [0].
Various approaches have been proposed for confidence estimation, which the process of inferring
an LLM's confidence in its predictions [7]. This includes measuring token-level probabilities of answers [ training proxy probe models on internal hidden states [16] and applying supervised
fine-tuning with labeled confidence data (46.. Nonetheless, these methods assume access to a model's

---

internal states or weights, which are usually unavailable in proprietary LLMs [6] 44 Additionally, they are often model-specific, necessitating readjustment or retraining when the underlying LLM changes 44 An alternative approach that avoids these limitations involves sampling multiple responses for the same prompt and deriving confidence based on response consistency [22]. While this method is compatible with proprietary LLMSs and is model-agnostic, its primary limitation is the increased computational cost due to repeated inference [47
[4]
In this work, we focus on verbalized confidence estimation, where the LLM is prompted to directly express its confidence as part of their output, either through linguistic phrase (e.g.. "Highly likely") or numerical probability (e.g., "0.95") [7]. Verbalized confidence estimation has emerged as a widely studied approach as compatible with proprietary LLMs, model-agnostic, and computationally efficient [40 45 47 While verbalized confidence estimation offers ease of use and broad applicability, its accessibility also makes its limitations more consequential. Overconfidence persistent issue: L.Mss tend to express high certainty even when they are incorrect, posing serious risks for real-world deployment of LLMs 45 50 Our results suggest that reasoning models and their use of slow thinking hold strong potential to mitigate this issue.
Reasoning Models Increasing compute at inference time, or inference- time scaling, has become a focus for improving the reasoning performance of LLMs (37]. One effective approach to inference- time scaling is training models to generate longer CoTs, as demonstrated by models like OpenAI's ol [30 and DeepSeek-RI [4), which are commonly referred to as reasoning models. The key factor that distinguishes the CoT of reasoning models from that of non-reasoning, instruction-tuner models beyond just length their slow thinking 224, which drives their moobeem-solving ability. Slow thinking mirrors human cognitive behaviors, characterized by non-linear reasoning traces such as exploring alternative approaches, verifying answers, and backtracking [6]. During slow thinking, reasoning models also frequently produce epistemic markers that signal uncertainty (e.g., "I think", "maybe") which are rarely observed in non-reasoning models [50). Structurally, reasoning models typically enclose their CoT within special tokens such as <think>... </think> to isolate the reasoning phase, in contrast to non non-reasonin reasoning models.
In this paper, we investigate whether reasoning models can more accurately express their confidence by leveraging their slow thinking behaviors. A concurrent study also examines the calibration of reasoning models 48, but focuses training an external probe over hidden states to optimize CoT generation. In contrast, our study centers on verbalized confidence estimation and provides extensive analysis to uncover how reasoning models are able to better express their confidence.
Reasoning models better express their confidence
In this section, we demonstrate that reasoning models express their confidence more accurately than non-reasoning models, first detailing the experimental setup (Section (.II) and then presenting the results (Section 3.2). Full results and additional experiments are provided in Appendix A and further
setup details are provided Appendix B
3.1 Experiment setup
Datasets We evaluate LLM calibration across two types of datasets: knowledge-focused and reasoning-intensive. We use the knowledge-focused datasets, TriviaQA and NonambigQA [55 23 to provide a more controlled setting for comparing the calibration of reasoning and non- reasoning models, as both achieve similar accuracy given that CoT does little to help solve these tasks [38]. Therefore, this setup ensures that any differences in calibration are not simply due to reasoning models being better at solving the task. In addition, since reasoning models are primarily intended for complex reasoning tasks, we also include maasoning-intensive datasets, MMLU-Pro and SuperGPQA [2 21,, to ensure our findings generalize. These datasets are challenging multiple- choice benchmarks centered on knowledge-drie -diven reasoning. We use two subsets of each reasoning dataset: a Math subset focused on arithmetic reasoning, and a Non-Math subset covering other types of reasoning. Due to our broad range of experiments, we uniformly sample 1,000 examples from each dataset or subset to keep the compute manageable. To assess the variability of this choice, we perform bootstrapping over multiple resampled subsets and report the standard deviation in Appendix A.2.4
3

---

Models We benchmark diverse set of six reasoning models derived from four different 32B- scale LLMs against their non non-reasoning counterparts Specifically, we evaluate: RI-Distill-Qwer Qwen. QwQ and OR1 Prr-iiee reasoning models 4 35 against Qwen2.5-Instruct [33 GLM--Z-04414
against GLMM--04 14 [39] EXAONE-Deep 20 against EXAONE-335-
[19]; and Qwen3 with
Thinking Mode (abbreviated as Qwen3 Thinking) against Qwen3 with Non-thinking Mode (Qwen3 Von-thinking) [34]. Unlike the other pairs, which consist of separate model checkpoints, Qwen3 is a hybrid model that supports both reasoning-style CoT (Thinking Mode) and non-reasoning-style CoT WNon-Thinking Mode) via prompting
Inference procedure In a single turn of conversation, we instruct the models to perform three steps using CoT: (1) SOLUTION REASONING, where the model reasons step-by-step to arrive at an answer to the given question; (2) CONFIDENCE REASONING, where it evaluates its own confidence for that answer step-by-step and (3) CONFIDENCE VERBALIZATION, where it maps their confidence in one of ten bins, ranging from "Almost no chance (0-0.1)" to "Almost certain (0.9-1.0)" Each bin includes both a linguistic descriptor (e.g.. "Almost certain") and its corresponding numerical probability (e.g.. "0.9-1.0"), which are inspired by the approaches used in prior work [40] 45). To ensure fair comparison, we use the same instructions for both reasoning and non-reasoning models. The full prompt we use is included in Appendix B.2
For reasoning models, we expect all three steps to be carried out within the thinking process sthink?...<tthinkke </think>, However, we observe that some reasoning models (specifically RI-Distill, ORI-Preview and GLM-ZI) rarely engage in CONFIDENCE REASONING To address this, we force these models to include CONFIDENCE REASONING within their thinking process by generating up to the </think> token, and replacing the token with "Okay, now let's 's assess my overall thinking process so far step-by-step second round of inference. We ablate the effect of this choice in Section 4.2 where we observe that it has negligible impact on calibration.
need to evaluate how likely my answer is correct." and performing a
As part of the instruction, models are asked to format their final response "Answer: ANSWER Conf idence: CONF CONFIDENCE" after completing all three steps. We apply rule-based filtering using regular expressions to extract the predicted answer and confidence. Answers are matched against the ground truth using GPT-4o mini 288) leveraging the prompt from OpenAl's simple-evale codebase 33.. We describe the full procedure we use to ensure that all model responses contain extractable answers and confidence predictions in Appendix B.2
Evaluation metries We use commonly adopted calibration metrics from prior work 7
40 45)
Expected Calibration Error (ECE) (26) measures the average discrepancy between accuracy and predicted confidence within each confidence bin, weighted by the number of samples in each bin. While ECE is an intuitive metric for measuring absolute calibration, it fails to capture calibration at the individual prediction level and does not account for the model's discriminative ability, which refers to how well the model assigns higher confidence correct predictions over incorrect ones 77]. In contrast, AUROC [1 captures the model's discriminative ability by computing the probability that a randomly chosen correct prediction is assigned higher confidence than a randomly chosen incorrect one, but it does not measure absolute calibration. Brier Score (2) quantifies the mean squared difference between predicted confidence and the true binary outcome, capturing both absolute calibration at the individual level and the discriminative ability.
Alternative prompting strategies and setups To ensure the robustness of our findings, we exper- iment with wide range of alternative prompting strategies and setups, and provide the results in Appendix A.2 (1) We explore different confidence expression styles such as providing only a linguis- tic descriptor without a probability, or outputting a numerical probability directly without binning. (2) Since some reasoning models perform two rounds of inference due to the forced CONFIDENCE REASONING, we also evaluate similar two-step sequential prompting setup for non-reasoning
'We discuss the results on different of model sizes Section
"The Non-thinking mode is enabled by injecting an empty thinking block, <think></think> at the beginning of the model's response.
'To elaborate, their thinking process mostly only contains SOLUTION REASONING, although they consistently do output their CONFIDENCE V VERBALIZATION outside of the thinking termination token </think> without COT.

---

Table I: Benchmark result knowledge- focusse datasets. Reasoning models highlighted in blue, and the best performance within each backbone LLM group is shown in bold. Lower ECE and Brier Scores, and higher AUROC, indicate better calibration. Accuracy included for reference.
TriviaQA
NonambigQA
Model
Acc.
FCF.
Brier
AUROCT
Acc.
FCR.
Brier
AUROCT
Qwen2.5-32B
Qwen2.5-Inst
0.718
0.129
0.176
0.769
0.517
0.297
0.303
0.720
RI-Distill-Qwer
0.727
0.042
0.157
0.782
0.491
0.195
0.241
0.749
ORI-Preview
0.738
0.052
0.150
0.795
0,470
0.219
0.247
0,759
QwQ
0.768
0.063
0.137
0.807
0.535
0.226
0.250
0.757
GLM-4-32B-0414
GLM-4-0441
0.814
0.084
0.137
0.675
0.640
0.246
0.269
0.643
GLM-Z1-0414
0.824
0.029
0.120
0.777
0.570
0.209
0.251
0.721
EXAONE-335:32B
EXAONE-3.5-Inst.
0.715
0.130
0.178
0.749
0.511
0.302
0.302
0.721
EXAONE-Deep
0.687
0.104
0.175
0.763
0.452
0.288
0.289
0,743
Qwen3-32B
Qwen3 Non-thinking
0.711
0.207
0.230
0.650
0.511
0.403
0.403
0.572
Qwen3 Thinking
0.768
0.063
0.137
0.807
0.535
0.226
0.250
0.757
models. (3) We evaluate more advanced prompting strategies for non-rcasoning models, as suggested by prior work, such Top-K and Multi-step prompting [40 45 (4) While main experiments use greedy decoding to avoid randomness and ensure reproducibility, we also test sampling- based decoding to confirm that results generalize across decoding strategies. Across all setups, findings remain consistent: reasoning models exhibit better calibration than their non-reasoning counterparts.
3.2 Experiment result
We observe that reasoning models consistently outperform their non-reasoning counterparts across all calibration metrics on knowledge-focused datasets (Table Notably, reasoning models exhibit superior calibration even in cases where they underperform non-reasoning models in task accuracy. For example, GLM-Z1-0K
achieves 0.07 lower accuracy than GLM-4-0414 on NonambigQA, yet obtains a 0.037 lower ECE and 0.078 higher AUROC. This indicates that accurate confidence estimation is not merely byproduct of better task performance. Additionally, this highlights potential motivation for using reasoning models even on simple factual question-answertin tasks: although they may not necessarily achieve higher accuracy, they communicate their confidence more reliably.
Interestingly, Qwen3 exhibits substantial difference in calibration depending on whether engages in CoT through Thinking or Non-thinking Mode, As Qwen3 a single checkpoint, this suggests that the calibration observed in reasoning models primarily stems from their slow thinking CoT, rather than some other inherent differences between the models. We further support this claim in non-reasoning models improve calibration by engaging in slow thinking through in-context learning.
Section 4.3 where we show that
Figure 2 (left) illustrates the superior calibration of 11-Distill-Qwen over Owen2.55Innsrru. on TriviaQA. A. Surprisingly, for confidence levels above 55% Φ RI-Distill- exhibits near pear-perfect calibration, where its estimated confidence closely matches the actual accuracy (i.e., points lie near the line y x, where y is accuracy and æ is confidence). Plotting the sample frequency of the confidence bins (right of Figure 2 reveals that Qwen2.5-Instrucc Instructt is highly overconfident, estimating either the 85% or 95% confidence bin for more than 80% of the total samples. In contrast, RI RI-Distill- Qwen produces estimations across a relatively diverse range of confidence levels, indicating less tendency to be overconfident. Nonetheless, there is room for improvement for RI-Distill -wen as well, as rarely estimates confidence lower than 55%.
555% corresponds to the "Better than even (0.5 0.6)" bin. Throughout the paper, we denote each bin by the average value of its numerical confidence range for brevity.
5

---

#wen2.5-32B-Instree
R1-Distill- Distill-Qwen332
Qwen2.5-32B-Insteeee
Frequency
400-
80
200
Accuracy (%)
60
R1- Al-Distill- ---DDitillQwee---a
40
Frequency
400
20
200
%
20
60
80
100
40
60
100
20
40
60
80
100
(%)
Confidence
Confidence (%)
Figure 2: Accuracy (left) and sample frequency (right) across confidence bins for Qwen2.5-32B-
Instruct and H1-Diisill-Qwen-B33 on TriviaQA.
Reasoning models are also better calibrated on reasoning- intenssie datasets (Table 2 showing that our findings generalize to more complex tasks. However, we observe a few cases where non-reasoning models outperform their reasoning counterparts in AUROC; specifically, GLM-4-0414 on both Math subsets, and EX AXXI--33.S-Issrricc on the Non-Math subset of SuperGPQA. Upon closer inspection, find that the root cause lies in the multiple-choice format of MMLU-Pro and SuperGPQA, which implicitly provides models with cues about their correctness based on the available options. Non- reasoning models, exhibiting overconfidence, tend to select the 95% confidence bin when their answers appear among the multiple-choice options, but when they do not, the models select the closest available option and assign lower confidence of 85%. Relying bins, reinforced by implicit cues from the multiple-choie options, gives non-reasoning models an unearned advantage in discriminative power, which inflates their AUROC scores. Nonetheless, the non-reasoning models receive worse Brier Scores, as the metric, unlike AUROC, directly penalizes mismatches between predicted confidence and actual outcomes.
just two confidence
Table 2: Benchmark results reasoning-ine inttenive datasets. The full result with ECE and Accuracy
available in Appendix A.3
SuperGPQA
MMLUU-Pr
Math
Non-Mall
Math
Non-Math
Model
Brier 4 AUROC1 Brier
AUROCT
Brier
AUROCT Brier | AUROC 1
Qwen-2.5-32B
Qwen2.5-
0.295
0.621
0.416
0.522
0.170
0.806
0.283
0.636
RI RI-Distill-Qwe -wenn
0.218
0.677
0.305
0.537
0.121
0.842
0.213
0.654
ORI ORI-Preeeww
0.212
0.647
0,275
0.593
0.107
0.824
0.200
0.679
QwQ
0.217
0.664
0.314
0.581
0.094
0.871
0.222
0.687
GLM-4-32R-0414
GI-M4-0444
0.256
0.681
0.459
0.516
0.131
0.794
0.282
0.626
GLM-Z1-0414
0.216
0.633
0.348
0.572
0.095
0.783
0.222
0.648
EXAONE-335-32B
EXAONE-3.5-Inst.
0.345
0.590
0.441
0.567
0,243
0.676
0.324
0.590
EXAONE-Deep
0.261
0.645
0.385
0.542
0.110
0.776
0.261
0.648
Qwen3-32B
Qwen3 Non-thinking
0.296
0.588
0.440
0.552
0.155
0.708
0.285
0.632
Qwen3 Thinking
0.217
0.664
0.314
0.581
0.094
0.871
0.222
0.687
4 Analysis: slow thinking enables accurate confidence adjustments
In this section, we present an in-depth analysis showing that the enhanced calibration of reasoning models stems from slow thinking, which enables LLMs to dynamically adjust their confidence throughout the course of reasoning, as illustrated Figure First, we show that reasoning models gradually produce more accurate confidence estimates as their CoT unfolds, whereas non-reasoning models exhibit no such trend (Section 4.1 We then perform an ablation study, systematically removing components of the CoT, and identify that slow thinking behaviors, specifically exploring

---

A Qwen2.5 Inst.
GLM4 A Qwen3 No Think
R1 Distill
GLM
Qwen3 Think
Relative Brier Score (4)
0.2
0.0
-0.2
-0.4
100
200
300
400
500
Average CoT Progress (In tokens)
Figure 3: Relative change in Brier Score CoT progresses on NonambigQA. Non-reasoning models are represented by triangles (A), and reasoning models by circles (o), with each model pair shown in matching colors.
alternatives and backtracking, are key contributors to improved calibration (Section 4.2) Finally, we demonstrate that simply prompting con-reasoning models to perform slow thinking through in-context learning yields similar benefits, supporting the notion that these gains are driven by slow thinking itself rather than other inherent differences (Section
4.1 (Only) Reasoning models gradually get better calibrated CoT progresses
reasoning models are truly capable of dynamically adjusting and correcting their confidence throughout the CoT process, would expect to observe steady, gradual increase (rather than random or inconsistent patterns) in calibration over the course of reasoning. To test this, we collect the CoTs of both reasoning and non-reasoning models, and divide each instance into 11 cumulative segments, where each segment includes the first 0%, 10%, up to 100% of the total number of generated tokens. We then prompt the models with each cumulative segment, appending tokens to trigger early termination of reasoning in order to collect the model's answer and its confidence expression at that point. We use the knowledge-focusee datasets, TriviaQA and NonambigQA, as the models' accuracy remains relatively stable even when the reasoning process is terminated early.
Figure shows the relative change in Brier Score compared the initial value O token length (i.e., before any CoT reasoning) on NonambigQA. We observe that all three reasoning models show progressively better calibration with the slope trending downwards. Surprisingly, for non-reasoning models, the Brier Score is the lowest without any CoT, and calibration worsens by the end of the CoT, with consistent trend observed in between.
To quantify this trend across both datasets and all three metrics, we a linear model to each calibration metric over CoT progress and report the slope and its statistical significance across both datasets (Table 3). For reasoning models, we observe strong linear relationships in most cases (except for Qwen3 Thinking's ECE) with fitted slopes indicating steadily improving calibration over the course of the COT. Meanwhile, non-reasoning models generally fail to yield statistically significant linear trends, and in the some cases where statistically significant fit is observed, the slope surprisingly indicates worsening calibration.
4.2 Ablation study: exploring alternatives and refining matters the most
To determine which aspects are the most responsible for calibration gains, we analyze RI-Distill- Qwen-32B by removing individual components from its COT. Specifically, we use the model's original CoTs on the knowledge-fooce focused datasets, TriviaQA and NonambigQA, and re-prompt the model with targeted components removed. We ablate the following:
Confidence Reasoning: We retain only the portion of the CoT that appears before the CONFIDENCE REASONING force prompt ("Okay, now let's assess my overall thinking process... ...") described in Section 3.] thereby ablating the effect of explicit reasoning about confidence.
Epistemic Markers: Reasoning models frequently generate epistemic phrases that signal uncer- tainty, such as "I think" or "maybe", which could affect the confidence estimation. Therefore, we
//think AnAnswer: for reasoning models and AnAnswer: for non-reasoning models.
7

---

Table 3: Slopes of calibration metrics measured over CoTT progress. Bold Blue indicates steady improvement in calibration as CoT progresses, and Bold Red indicates steady degradation. Bold colors are shown only when the trend is statistically significant (p < 0.05).
TriviaQA
NonambigQA
Model
Acc.
ECE
Brier
AUROC1
Acc.
ECE
Brier J AUROC 1
Qwen2.5-32B
Qwen2.5-Inst.
0.000
0.004
0.000
0.002
0.002
0.002
0.001
0.003
R1-Distill-Qwen
0.003
-0.007
-0.004
0,008
0.003
-0.015
-0.010
0.010
GLM-4-32B-0414
GLM-4-0414
0.007
-0.003
-0.003
0.000
0.005
0.002
-0.001
0.003
GLM-Z110414
0.009
-0.016
-0.011
0.026
0.009
-0.022
-0.020
0.022
EXAONE-3,5-32B
EXAONE-3.5-Inst.
-0.001
0.001
0.000
0.002
-0.001
0.004
0.003
-0.001
EXAONE-Deep
0.006
-0.016
-0.011
0.026
0.002
-0.016
-0.016
0.024
Qwen3-32B
Qwen3 Non-thinking
0.003
0.004
0.001
-0.001
0.004
0.002
0.002
-0.004
Qwen3 Thinking
0.004
-0,00S
-0.004
0.005
0.004
-0.005
-0.004
0.005
remove or paraphrase such phrases, while preserving the rest of the content as closely as possible to the original.
Non-linear Reasoning: Reasoning models follow complex non-linear reasoning paths, including behaviors such as exploring alternativss, refining, and backtracking. We prune these non-linear traces, retaining only the parts that directly support the model's final answer and thus making the reasoning path linear.
We also report the No CoT setting lower bound for reference, in which the model's reasoning is terminated immediately by prompting "</think>".
For both Epistemic Markers and Non-linear Reasoning removal, we use GPT-4.1 29 guided by detailed instructions and three-shot demonstrations created by the authors. Only the portion of the CoT before the forced CONFIDENCE REASONING prompt is provided. For all three settings, the authors manually inspect 100 examples using predefined criteria, observing that more than 90% meet the intended requirements for each setting. Additional details on the experimental setup, including the full prompts used with GPT-4.1, the manual inspection process, and before-and and-r-fed- examples of the CoTs, are provided in Appendix B.3
Table 4: Ablation study the CoT of RU-Distill-Qwen--33 corresponding component removed from the original COT.. The No CoT setting is included as a lower-bound reference.
Qwenn--QeenBBB.... The "_" symbol indicates that the
TriviaQA
NonambigQA
Method
Acc.
ECE.
Brier
AUROC1
Acc.
ECE.
Brier
AUROC1
Original
0.727
0.042
0.157
0.782
0.491
0.195
0.241
0.749
Confidence Reason.
0.722
0.063
0.166
0.763
0.494
0.187
0.247
0.718
Epistemic Markers
0.726
0.109
0.154
0.837
0.500
0.262
0.263
0.784
Non-lincur Reason.
0.731
0.161
0.179
0.734
0,500
0.341
0.320
0.728
No CoT
0.697
0.150
0.206
0.689
0.457
0.359
0.365
0.618
Table presents the ablation results. First, we observe that Confidence Reasoning has only a minor effect on calibration, suggesting that the model's ability express confidence accurately primarily stems from slow thinking about the question itself, rather than explicitly reasoning about its own confidence. Interestingly, removing Epistemic Phrases leads noticeable degradation in ECE, but results in improved AUROC. Upon further investigation, we find that the model becomes overconfident, yet surprisingly retains its ability to discriminate between correct and incorrect answers. Specifically, on TriviaQA, the model tends to predict either 95% or 65% confidence almost exclusively, as opposed to the more varied distribution in the original (see right of Figure Despite this overconfidence, predictions at 95% are generally correct, while those at 65% are not, preserving the model's i discriminative capacity. Restricting predictions to just two confidence bins, instead of distributing across wider range, results in an advantage over the original model in AUROC.

---

Finally, our results show that Non-linear Reasoning has the greatest impact on calibration. When the reasoning path is constrained a linear trajectory, all three metries degrade, indicating decline not only in absolute calibration but also in the model's s ability to discriminate between correct and incorrect answers. Overall, our ablation results align with the observations Section 4.1] indicating that as the CoT unfolds, the model gains more opportunity to express epistemic phrases, consider alternatives, and refine its reasoning which all contributes to improved calibration.
4.3 Non-reasoning models also better express their confidence with slow thinking
So far, have shown that reasoning models are better calibrated (Section 3, that their calibration improves as the CoT unfolds (Section4..) and identified components in slow thinking that contribute to this improvement (Section 4.2). Yet an important question remains: Are calibration gains from slow thinking exclusive to reasoning models that went through extensive fine-tuning or reinforcement learning with specialized reasoning data?
In this section, we demonstrate that non-reasoning models also "better express their confidence" by simply engaging in slow thinking via in-context learning without any parameter updates. Specifically, we collect three random examples of RI 21-Distill-Qwen- slow thinking on held-out TriviaQA samples, and use them few-shot exemplars prompt non-reasoning models to reason in a similar manner.
Tshows that non-reasoning models also consistently exhibit improved calibration across both datasets and all three metrics when prompted to engage in slow thinking. This highlights that the calibration gains in reasoning models, stem from the act of slow thinking itself rather than from inherent properties exclusive to reasoning models. This claim is further supported by our findings with Qwen3 in Section [3] where the model demonstrates significantly better calibration when performing CoT in Thinking mode compared to Non-Thinking mode.
Table 5: Benchmark results on non-reasoning models prompted engage in slow thinking via in-context learning.
TriviaQA
NonambigQA
Model
Acc.
BCE +
Brier $ AUROC1
Acc.
ECE
Brier | AUROC1
Owen2.5-32B-Ins
0.709
0.135
0.192
0.714
0,511
0.309
0.321
0.658
Slow Thinking
0.709
0.091
0.167
0.784
0.481
0.269
0.290
0.696
GLM-4-2B-0014
0.783
0.150
0.180
0.631
0.596
0.338
0.347
0.573
Slow Thinking
0.800
0.029
0.126
0.796
0.622
0.126
0.218
0.715
EXAONE-32B-In
0.728
0.118
0.183
0.712
0.519
0.326
0.331
0.670
Slow Thinking
0.708
0.096
0.165
0.798
0,500
0.279
0.282
0.759
5 Discussion
Does forcing longer CoT lead improved calibration? Our findings in Section 4.1 suggest that reasoning models become better calibrated the longer they engage in CoT. This prompted us to test whether calibration can be further improved by forcing reasoning models slow think longer. (25) by appending "Wait," to the end of the CoT and conducting
Specifically, we apply budget additional rounds of inference. Figure (left) of Appendix A. shows that budget forcing does not necessarily lead to further improvements in calibration, suggesting that calibration gains likely stem from the quality rather than the quantity of slow thinking.
Calibration scales better with model size in reasoning models Since our experiments are con- ducted on 32B-scale models, we further examine how the effects of slow thinking vary across different model scales. Figure 4 (right) of Appendix A. shows the relative calibration gain of reasoning models over their non-reasoning counterparts across various model scales. We find that the calibration gap widens as model size increases, suggesting that the benefits of slow thinking become more pronounced in larger, more capable I.LMs. This trend is encouraging, as it suggests that slow thinking will become increasingly effective mechanism for calibration as LLMs continue to improve.

---

Room for improvement even for reasoning models Generally, we observe that reasoning models still prefer to express high confidence, assigning values below 55% infrequentl, as shown in Figure (right). This tendency is further reflected in the calibration metrics: the lower task accuracy on NonambigQA leads to notably higher ECE and Brier Scores compared TriviaQA (Table [T. While these observations suggest that the challenge of expressing uncertainty remains, our overall results demonstrate that slow thinking meaningfully improves calibration and represents a valuable direction for future model development.
6 Conclusion
In this work, we present a comprehensive study of reasoning models' ability to express confidence through their own words. Across diverse datasets and model families, we show that reasoning models are consistently better calibrated than their non-reasoning counterparts. Through detailed analysis, we trace these calibration gains to slow thinking behaviors which allow models dynamically adjust Overall, our findings reveal that slow thinking
their confidence throughout the offers more than just improved problem-solving: it also enhances the trustworthiness and reliability of L.LMs by enabling them to better "know what they know."
Acknowledgments
This work was supported by the InnoCORE program of the Ministry of Science and ICT (N10250156, 50%) and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) 20222022-000111 Developing a Sustainable Collabo- rative Multi-mode Lifelong Learning Framework, 25%; No.RS 112221-1122266 Artificial Intelligence Innovation Hub, 25%).
References
[1] Andrew Bradley. The of the area under the curve in the evaluation of machine learning algorithms. Pattern recognition, 30(7):1145 1997.
[2] Glenn Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
[3] Arslan Chaudhry, Sridhar Thiagarajan, and Dilan Gorur. Finetuning language models to emit linguistic expressions of uncertainty. arXiv preprint vrXiv:2409.12180 2024.
[4] Dee.Seek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[5] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma. Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint MxXi:2301.0022:3 2022.
[6] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, four habits of highly effective stars. arXiv preprint arXiv:2503.013077 2025.
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. survey of contidence estimation and calibration large language models. Kevin Duh. Helena Gomez, and Steven Bethard, editors, P Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pages 6577-6595 Mexico City, Mexico, June 2024, Association for Computational Linguisties.
[8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modem neural networks. In International conference machine learning, pages 1321-1330. PMLR, 2017.
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024.
[10] Juuzhou Han, Wray Buntine, and Ehsan Shareghi. Towards uncertainty aware language agent. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages (662-6683 Bangkok, Thailand, August 2024. Association for Computational Linguistics.
10

---

[11] Jujie He. Jiacai Liu, Chris Yuhao Liu. Rui Yan. Chaojie Wang, Peng Cheng, Xiaoyu Zhang. Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li. Liang Zeng, Tianwen Wei, Cheng Cheng, Yang Liu, and Yahui Zhou. Skywork open reasoner series.
41c. noc.notion. site/ 2025. Notion Blog.
Slywork-Opens Reaonser Series 10bb99ee222238044464666606603330
[12] Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas. Shiyu Chang. and Yang Zhang, Decomposing uncertainty for large language models through input clarification ensembling. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JogR.org. 2024.
[13] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems, 43(2):1- 55, 2025.
[14] Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 196997992, Singapore, December 2023. Association for Computational Linguistics.
[15] Mandar Joshi, Funsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen, Kan, editors. Proceedings of the 55th Annual Meeting the Asvociation for Computational Linguistics (Volame I: Long Pupers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.
[16] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint 2rXiv:2207.052211 2022.
[17] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wes Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics. 7:452 466, 2019.
[18] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu. Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 61 611-62 2023.
[19] LGG AT Research. Exaone 3.5: Series of large language models for real-world use cases, 2024.
[20] LG Al Research. Exaone deep: Reasoning enhanced language models, 2025.
[21] M-A-P Team. Supergpqa: Scaling I1m evaluation across 285 graduate disciplines, 2025.
[22] Potsawee Manakul, Adian Liusie, and Mark Gales. Sel[CheckGPT: Zero-resource black-box hallucination detection for generative large language models. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceddiggs of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004 9017, Singapore, December 2023, Association for Computational Linguistics.
[23] Sewon Min, Julian Michael, Hannanch Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering am- biguous open-domain questions. In Bonnie Webber, Trevor Cohn, Yulan He. and Yang Liu, editors. Proceedings the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP). pages 5783-5797, Online, November 2020. Association for Computational Linguisties.
[24] Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng. Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:2412.00 09413 2024.
[25] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint X55i 19393, 2025.
[26] Mahdi Pakdaman Nacini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference artificial intelligence, 2015.
[27] Jeremy Nixon, Michael W Dusenberry, Linchuunn Zhang, Ghassen Jerfel, and Dustin Tran. Measuring calibration in deep learning. In CVPR workshops, 2019.

---

[28] OpenAL GPT-4o mini: advancing cost-efficient intelligence. https://openai com/ ctm/:i.ooiiiiiiiiiiei index/
gpt-4o-ming aat 2024. Accessed: 2023-05-16.
[29] OpenAL Introducing GPT-4.1 I in the API. https:///opoaal.coo///eee////// 2024. Accessed:
2025-05-16 16.
[30] OpenAI. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
[31] OpenAL. simple-evals: Evaluation framework for language models. httpp://githabb.com/ppeaaa Simple-evall 2024. Accessed: 2025-05-16.
[32] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-halt: Medical domain halluci- nation test for large language models. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 3 314-334, 2023.
[33] Qwen Team. Qwen2.5 technical report, 2025.
[34] Qwen Team. Qwen3, April 2025.
[35] Qwen Team. Qwq-32b: Embracing the power of reinforvement learning, March 2025.
[36] Vaishnavi Shrivastava, Perey Liang, and Ananya Kumar. Llamas know what gpts don't 't show: Surrogate models for confidence estimation. arXiv preprint arXiv:231 1.08877, 2023.
[37] Charlie Victor Snell,Jaehooe Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference Learning Representations, 2025.
[38] Zayne Sprague, Fangcong Yin. Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Yc, Kyle Mahowald, and Greg Durrett. To cot not to cot? chain-of-thouggh helps mainly on math and symbolic reasoning. arXiv preprint arXiv:2409.12/1
2024.
[39] Team GLM. Chatglm: A Family of large language models from glm- glm-130b to glm-4 all tools, 2024.
[40] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafacl Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategics for cliciting calibrated confidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433-5442, Singapore, December 2023. Association for Computational Linguistics.
[41] Neeraj Varshney, Wenlin Yao, Hongming Zhang. Jianshu Chen, and Dong Yu. A stitch in time saves nine: Detecting and mitigating hallucinations of Iims by validating low-confidence generation. arXiv preprint arXiv:2307.0 03987, 2023.
[42] Yubo Wang. Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMI.U-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. tf-hughh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824- 2283-24877 2022.
[44] Liangru Xie, Hui Liu, Jingying Zeng, Xianfeng Tang, Yan Han, Chen Luo, Jing Huang, Zhen Li, Suhang Wang, and Qi He. A survey of calibration process for black-box Ilms. arXiv preprint 1427 2024.
[45] Miao Xiong. Zhiyuan Hu, Xinyang Lu, YIFEI LL, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024.
[46] Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang. Yangyi Chen, and Jing Gao. Sayself: Teaching ilms to express confidence with self-reflective rationales. In Proceedingy of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5985-5998, 2024.
[47] Daniel Yang, Yao Hung Hubert Tsai, and Makoto Yamada. On verbalized confidence scores for 1ls.. arXiv preprint arXiv:2412. 2024.
12

---

[48] Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. Reasoning models know when they'r re right: Probing hidden states for self-verification. arXiv preprint wrXiv:2504.05419 2025.
[49] Yue Zhang. Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang. Enbo Zhao, Zhang, Yulong Chen, et al. Siren's song in the ai ocean: survey on hallucination in large language models. arXiv preprint arXiv:2309.0122 2023.
[50] Kaitlyn Zhou, Jena Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 3623 3643, 2024.
13

---

A Additional Experiment Results and Details
A.1 Experiment Results in Discussion
this section, present the results referenced in the Discussion (Section 5p. Specifically, Figure shows shows the effect of forcing reasoning models to produce longer CoT's (left) and the effect model scale on calibration (right).
R1 Distill
GLM Z
- Qwen3 Think
EXAONE Deep
Relative Brier Score Improvement
0.4
0.04
0.3
0.02
0.00
0.1
0.0
-0,04
0.06
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
5
10
15
20
25
30
Budget
Parameters (Billion)
Figure 4: Relative change Brier Score on NonambigQA under budget forcing (left) and across different model scales (right).
A.2 Expanded results for robustness
In this section, explore alternative setups and conditions the robustness of our findings.
A.2.1
Alternative Prompting Method for Non-reasoning Models
Table 6: Benchmark results with alternative prompting method for non-reasoning models.
TriviaQA
NonambigQA
Model
Acc.
ECE
Brier
AUROC
Acc.
ECE
Brier | AUROC1
Qwen2,5-32B
Qwen2.5-Inst.
0.718
0.129
0.176
0.769
0.517
0.297
0.303
0.720
+Two-turn
0.709
0.135
0.192
0.714
0.511
0.309
0.321
0.658
+Top-K
0.725
0.141
0.168
0.780
0.504
0.327
0.320
0.698
+Multi-step
U.734
0.107
0.177
0.735
0.524
0.278
0.305
0.659
RI-Distill-Quas
0.727
0.042
0.157
0,782
0.491
0.195
0,241
0.749
GLM-4-32-0014
GI-M4400414
0.814
0.084
0.137
0.675
0.640
0.246
0.269
0.643
+Two-turn
0.783
0.150
0.180
0.631
0.596
0.338
0.347
0.573
+Top-K
0.764
0.139
0.167
0.737
0.554
0.319
0.318
0.695
Multtisttp
0.798
0.105
0.153
0.652
0.667
0.228
0,262
0.589
GIM-Z.1-014
0.824
0.029
0.120
0.777
0.570
0.209
0.251
0.721
EXAONE-3.5-32B
FXAONE-33.5-Ins.
0.715
0.130
0.178
0.749
0.511
0.302
0.302
0.721
Two-tur
0.728
0,118
0.183
0.712
0.519
0.326
0.331
0.670
+Top-K
0.705
0.160
0.187
0.771
0.487
0.372
0.360
0.693
+Multi-step
0.685
0.115
0.186
0.728
0.501
0.269
0.286
0.698
EXAONF-Deep
0.687
0.104
0.175
0.763
0.452
0,288
0.289
0.743
Qwen3-32B
Qwen3 Non-thinking
0.711
0.207
0.230
0.650
0.511
0.403
0.403
0.572
+Two-turn
0.713
0.202
0.226
0.674
0.500
0.402
0.397
0.620
+Top-K
0.701
0.219
0.236
0.637
0.502
0.415
0.410
0.590
+Multi-slep
0.711
0.175
0.208
0.737
0.501
0.369
0.367
0.634
Qwen3 Thinking
0.768
0.063
0.137
0.807
0.535
0.226
0.250
0.757
We apply alternative, more advanced prompting strategies to reasoresnning models to give them further advantage. Specifically:
14

---

Table 7: Benchmark results when providing only the linguistic descriptor.
TriviaQA
NonambigQA
Model
Acc.
BCE
Brier
AUROCT
Acc.
ECE
Brier
AUROCT
Qwen2.5-32B
Qwen2.5-Inst.
0.714
0.099
0.170
0.756
0.516
0.259
0.279
0.705
RI-Distil--Qwen
0.707
0.059
0.169
0.762
0.480
0.219
0.255
0.736
GLM-4-32B-0414
GLM-4-0414
0.820
0.077
0.131
0.713
0.639
0.222
0.252
0.672
GLM-Z1-0414
0.833
0,080
0.118
0.777
0.584
0.143
0.226
0.719
EXAONE-3.5-32B
0.722
0.1
0.165
0.760
0.514
0.295
0.293
0.729
EXAONE-Deep
0.689
0.108
0.161
0.805
0.457
0,302
0.287
0.768
Qwen3-32B
Qwen3 Non-thinking
0.717
0.169
0,213
0.673
0.510
0.370
0.376
0.600
Qwen3 Thinking
0.742
0,079
0.147
0.778
0.516
0.246
0.271
0.710
Table 8: Benchmark results when directly outputting numerical probability without binning.
TriviaQA
NonambigQA
Model
Acc.
ECE.
Brier
AUROC1
Acc.
ECE
Brier
AUROCT
Qwen2.5-32B
Qwen2.5-Instruct
0.720
0.195
0.219
0.735
0.524
0.374
0.387
0.672
RI-Distill-Ween
0.721
0.184
0.202
0.806
0.477
0.380
0.359
0.770
GLM-4-32B-0414
GLM-4-0414
0.808
0.103
0,148
0.782
0.650
0.217
0.285
0.702
GIM-ZZ1-0414
0.832
0.080
0.124
0.832
0.572
0.306
0.303
0.784
EXAONE-3.5-32B
EXAANNE-3.-Instrucc
0.735
0.163
0.194
0.688
0.526
0.357
0.367
0.647
EXAONF-Deep
0.708
0.133
0.176
0.841
0,452
0.330
0.329
0.796
Qwen3-32B
Qwen3 Kon-Thinking
0.707
0.238
0.264
0.739
0.509
0.379
0.448
0.659
Qwen3 Thinking
0.742
0.169
0.182
0.808
0.517
0,380
0.364
0.725
Two-turn:
Since some reasoning models perform rounds of inference due the forced CON- FIDENCE REASONING (Section 3). we also evaluate a similar two-step sequential prompting setup for non-reasoning models. Specifically, we prompt the non-rcasoning models to engage in So- LUTION REASONING in the first turn, followed by CONFIDENCE REASONING and CONFIDENCE VERBALIZATION in the second.
Top-K:
Following prior work, we prompt the model to generate K candidate answers (with K 4. used in previous studies), each accompanied by confidence estimate. The final answer then selected as the one with the highest confidence.
Multi-step: As suggested by prior work, prompt the model to split its reasoning into K steps (with 4), assessing confidence after each step, and use the final confidence the output.
Even when non-reasoning models are given additional support through alternative prompting methods, reasoning models still consistently outperform them (Ta
A.2.2 Different Confidence Expression Styles
this section, we investigate the effect of confidence expression styles. Specifically test, (1) providing only a linguistic descriptor without a probability ("Almost Certain"), and (2) outputting a numerical probability directly without binning ("0.95").
Jandes/7andBpppeseeeee benchmark results when using only linguistic descriptors and when using numerical prob- abilities without bins, respectively. For using linguistic descriptors, reasoning models outperform non-reasoning models all metrics and both datasets. In contrast, when models prompted output numerical probability directly, we observe general degradation calibration for both reasoning and reasoning models. This finding is consistent with previous work suggesting that continuous probability expressions are less desirable for confidence estimation in LLMs [3]. Despite the inadequacy of this setting for reliable confidence estimation, reasoning models still generally outperform non-reasoning models.
15

---

A.2.3
Sampling Instead of Greedy Decoding
Table 9: Benchmark results with sampling for RI-Distill-Qwen Qwen-32B. Result of Srren2.5-32B-Insttuue provided for reference. We conduct five runs and report the average and standard deviation of each metric.
TriviaQA
NonambigQA
Model
Acc.
ECE J
Brier AUROCT
Acc.
ECE
Brier & AUROC1
Qwen2.5-32B 5-325-nstt.
0.718
0.129
0.176
0.769
0.317
0.297
0.303
0.720
RI RI-Distill- RI-Disiil--wenn332
Average
0.720
0.070
0.170
0.767
0.474
0.194
0.252
0.716
Standard Deviation
0.002
0.001
0.001
0.001
0.001
0.001
0.000
0.001
While main experiments greedy decoding reduce randomness and minimize computational cost, reasoning models are often recommended be used with sampling. We therefore test whether they retain their superior calibration under these recommended decoding settings.
Table[shoow that I.-Disiil-Qwen--21 continues to outperform Qwen2.5-32B 5-------II-ntttc in terms of ECE and Brier Score when using its recommended sampling configuration (lemperature 0.6). While its AUROC is slightly lower than that of Wween.5-32B-Instraae further analysis reveals that this drop is due to RI-Distill- Qwen-32B producing a more diverse range of confidence values unliie "wen2.5533B-Instruue which mostly outputs just two bins. This diversity imposes disadvantage in AUROC despite stronger absolute calibration performance.
A.2.4 Bootstrapping
Table 10: Benchmark results under stochastic sampling (five bootstrap resamples). We report the average and standard deviation of each metric.
TriviaQA
NonambigQA
Model
Acc.
ECE
Brier
AUROC1
Acc.
ECE
Brier AUROC1
Qwen2.5-32B-Inns
Average
0.503
0.306
0.308
0.719
Standard Deviation
0.019
0.017
0.012
0.022
0.010
0.007
0.005
0.010
RI-Distill-Qwen-3
AAeeaage
Standard Deviation
0.010
0.014
0.006
0.019
0.017
0.021
0.010
0.009
To variability, perform hootstrapping by generating five resampled subsets of 1,000 examples for cach dataset and report standard deviation across runs (Table[To.. Due the scale of our experiments, we not run this analysis across all models and datasets, as doing so would incur unreasonable computational cost.
A.3 Full Results on SuperGPQA and MMLU-Pro
this section, we provide the full benchmark results on SuperGPQA (Table II and MMLU-Pro (Table 12). including both accuracy and ECE.
A.4 Full CoT of the Qualitative Analysis
Figure
presents the full, untruncated CoT from R1 -istill -istilll-Qwenn Qwen-32B shown Figure
16

---

Table 11: Full benchmark results on SuperGPQA.
Math
Non-Math
Model
Acc.
ECE
Brier
AUROCT
Acc.
ECE
Brier AUROCT
Qwen2.5-32B
Qwcn2.5-Inst.
0.485
0,230
0.295
0.621
0.333
0.431
0.416
0.522
RI-Distil-Qweer
0.592
0.115
0.401
0.245
0.305
0.537
ORI-Preview
0,642
0.088
0.212
0.647
0,428
0.186
0,275
0,593
QwQ
0.658
0.118
0.217
0.664
0.448
0.256
0.314
0.581
GLM--32B-0414
GLM-4-0414
0.609
0.208
0.256
0.681
0.402
0.460
0.459
0.516
GLM-Z110414
0.669
0.112
0.216
0.633
0.402
0.327
0.348
0,572
EXAONE-3.5-32B
EXAONB-335-Inst
0.343
0.336
0.345
0.590
0.238
0.494
0.441
EXAONF-Deep
0.571
0.205
0.261
0.645
0.309
0.396
0.385
0.542
Qwen3-32B
Qwen3 Non-thinking
0.604
0.257
0.296
0.588
0.412
0.442
0.552
Qwen3 Thinking
0.658
0.118
0.217
0.664
0.448
0.256
0.314
0.581
Table 12: Full benchmark results on MMLU-Pro
Math
Non-Math
Model
Acc.
BCE
Brier
AUROC1
Acc.
ECB
Brier
AUROC f
Qwen2.5-32B
Qwen2.5-Inst.
0.669
0.094
0.170
0.806
0.560
0.230
0.283
0.636
RI-Distill-Qwen
0.785
0.105
0.121
0.842
0.646
0,039
0.213
0.654
ORI-Preview
0.820
0,086
0.107
0.824
0.668
0.030
0.200
0.679
QwQ
0.833
0.047
0.094
0.871
0.648
0.137
0.222
0.687
GLM-432B-0414
GLM-4-0414
0.802
0.067
0.131
0.794
0.637
0.250
0.282
0.626
GLM-7Z1-0414
0.861
0.072
0.095
0.783
0.669
0.124
0.222
0.648
EXAONE-3.5-32B
EXAONE-3.5-Inst.
0.598
0.156
0.243
0.676
0.498
0.285
0.324
0.590
EXAONE-Deep
0.809
0,036
0.110
0.776
0.580
0.191
0.261
0.648
Qwen3-32B
Qwen3 Non-thinking
0.787
0.083
0.155
0.708
0.626
0.249
0.285
0.632
Qwen3 Thinking
0.833
0,047
0.094
0.871
0.648
0.137
0.222
0.687
17

---

100
Confidence (%)
90
(2)
80
(4)
70
(1)
(3)
0
100
200
300
400
CoT Progress (in tokens)
Question
Who's the Athenian reformer that set the council of 500?
Model CoT
Okay, so I need to figure which Athenian reformer up the council of 500. Hmm, I remember that Athens had several important reformers, like Solon, Cleisthenes, and Pericles. Let me think about each of them. Solon was an early reformer, right? He introduced some democratic elements, like the council of 400, think. But wait, the question about the council of 500. So maybe wasn't Solon.
there's Cleisthenes. recall that was a significant figure in development Athemian democracy. overthrow
reorganized the political system local administrative units, and increased power of the Assembly. (I) Oh, and he also created the council of 500, which was called the Boule. Each deme would send a certain number of representatives to this council. which would meet regularly and handle day-to-dee affairs. That makes because the council of 500 was a key part of democratic reforms in Athens,
tyrants. He introduced concept of demes, which
(2) Pericles came later, and he more known for his leadership during the Golden Age of Athens, including who putting together, Cleisthenes the reformer set up council of 500, I pretty confident about this because it's a well-known part of his reforms. don't 'I think I'm mixing up any details here, The council of 500 major institution in the Athenian democrucy, and Cleisthenes is credited with its creation as part of his efforts to decentralize power and make the govermment more representative.
construction the Parthenon and the expansion democracy, (3) but don't think he
established the council of 500. That was already in place hefore his time.
Figure 5: Full CoT version of Figure !
18

---

B Experimental Setup Details
B.1 Experiments Compute Resources
We conduct experiments machines equipped with either Nvidia A6000 48GBs A100 80GBs GPUs. evaluating 32B-scale models, we use two GPUs. Leveraging vLLM [8] efficient inference, evaluation of a 32B reasoning model on single datasct takes approximately one hour on two A6000s.
B.2 Additional Details Inference Procedure
The full prompt given the models to perform the three steps- SOLUTION REASONING, CONFIDENCE REASONING, and CONFIDENCE VERBALIZATION-all with required answer and confidence formatting (* : ANSWER Confidence: CONFIDENCE") is provided in Listing
Listing Prompt used throughout our experiments
First, reason through the question step by step to arrive answer.
Then, thoroughly assess your confidence in that answer by evaluating your thinking process so far. Finally, classify your confidence into one of the following classes based on how likely your answer is to be correct:
"Almost no chance" (0.0-0.1)
"Highly unlikely" (0.1-0.2)
"Chances are slight" (0.2-0.3)
"Unlikely" (0.3-0.4)
"Less than even" (0.4-0.5)
"Better than even" (0.5-0.6)
"Likely" (0.6-0.7)
"Very good chance" (0.7-0.8)
"Highly likely" (0.8-0.9)
"Almost certain" (0.9-1.0)
Each category reflects the probability that your answer is correct.
At the very end of your output, format your answer and confidence * **Answe***; SANSWER
*Confidence** $CLASS
where CLASS is one of the names (only the names without the probability ranges) of the classes above.
During our preliminary experiments, we occasionally observed cases- cases-though though rare-where where models failed to produce the answer and confidence in the required format. To ensure robust extraction, we conduct an additional round of inference: for non-reasoning models, request only the answer and confidence to be re-output in the specified format; for reasoning models, first generate up to the </think> token, then replace with s/think>>nswer:: : and perform second round inference.
both reasoning and non-reasoning models, we use greedy decoding, with the maximum token length set to 4096 for knowledge-focused datasets and 8192 for reasoning-intensse datasets. The effect of using sampling is examined in Appendix A.2
B.3 Additional Details on Ablation Study
this section, we detail the process for assessing the quality the CoT's used in the ablation study Section 4.2 along with before-and-afier examples (Listing 2). and full prompts given to GPT-4.1 (Listing 3 and Listing 47. For quality check, we do the following:
The authors manually inspect 100 CoT examples sampled from both datasets we find that only four examples contain explicit reasoning about the model's own confidence.
Confidence Reasoning:
Epistemic Phrases: The authors inspect 100 samples collected from both datasets and evaluate them based two criteria: (1) whether epistemic phrases are absent, and whether the remaining content stays true to the original. We that 91 out of 100 examples satisfy both criteria.
Non-linear Reasoning:
We again provide detailed instructions along with three-shot examples. The authors then manually inspect 100 samples and evaluate them based the content supporting final answer remains, and (2) whether the remaining content stays faithful to the original. We find that all 100 examples meet both criteria.
criteria: (1) whether only
19

---

Listing 2:
Before-and-after examples of CoTT
[Original (No Confidence Reasoning) Okay, so need to ligure out which country the Order of the Elephant is the highest order of. I'm not too familiar with this, but I'll try to reason it out.
First, remember that orders are usually honors or decorations given by countries to recognize people for their contributions. The Order of the Elephant sounds like a very prestigious award. think it's associated with Denmark because I've heard of the Order of the Elephant in the context of Danish honors. But I'm not entirely sure.
Wait, maybe it's from another country. Let me think. know the Order of the British Empire from the UK, and the Order of Canada is from Canada. The Order of the Elephant doesn't 't ring a bell as being from those places. think Denmark has few orders, and the Order of the Elephant might be one of them. I'm pretty it's Danish order, but I' I'm not 100% certain.
should also consider there are other countries with similar-sounding orders. Maybe Sweden or Norway? I don't It recall any orders named Elephant from those countries. Denmark seems more likely because I've heard it mentioned in the context of their royal family or state honors.
So, putting all together, think the Order confident, but I'm not 100% sure because don't have direct knowledge. might be mixing it with another order, but can't 't think of any others right now.
Elephant the highest order Denmark. I'm fairly
[No Epistemic Phrases]
Okay, so need figure out which country Order of the Elephant is highest order of. I'll try to reason it out.
First, orders are usually honors decorations given by countries to recognize people for their contributions. The Order of the Flephant sounds like prestigious award. It is associated with Denmark, and the Order of Elephant appears
context Danish honors.
Wait, maybe it's from another country. The Order of the British Empire is from the UK, and the Order of Canada is from Canada. The Order of the Elephant does not appear to be from those places. Denmark has few orders, and the Order of the Elephant
them. is Danish order.
Consideration can be given to other countries with similar-sounding orders, such as Sweden or Norway. However, there are orders named Elephant from those countries. Denmark is more likely, as it is mentioned in the context of their royal family state honors.
So, the Order of the Elephant the highest order of Denmark.
[No Non-linear Reasoning] Okay, so I need to figure out which country the Order of the Elephant is the highest order of.
Order Elephant associated with Denmark. It's mentioned in the context the Danish royal family or state honors.
Orders are honors given by countries,
So, the Order of the Elephant the highest order of Denmark.
20

---

Listing 3:
Prompt given to GPT-4.1 to remove Epistemic Phrases
will give you a model's thinking process to question. want you to remove explicit expressions of uncertainty confidence (like "I'm not sure". "I think", "maybe", "I a bit confused", "I m pretty confident", etc.) in the model's thinking process.
Here are the details: Preserve the original thinking process and structure - do not delete any logical steps even if they seem mistaken or redundant.
Rephrase uncertain sentences into neutral equivalents whenever possible. For example, change "T not sure if he
all three books or if someone clse took over" into "He either wrote all three books or someone else took over."
Preserve "Wait," whenever appears. It signals natural shift or reevaluation, not uncertainty. Don't shortcut the reasoning or fix factual errors.
Keep the output close as possible to the original wording, only editing where necessary uncertainty.
the original thinking degenerates into repetition, or fails to reach a conclusion, stop the rewrite at the point where the reasoning becomes incomplete or broken. Do not invent, infer, or complete missing reasoning that not present in the original.
Strictly repeat the first sentence the model's thinking process the output.
Use double linebreaks between paragraphs.
Here some examples:
Listing 4: Prompt given to GPT-4.1 to remove Non-lincar Reasoning
will give you a model's 's thinking process to a question. want you to rewrite it into a concise and linear version, meaning the reasoning should move step-by-step directly toward the final answer without backtracking, unnecessary side exploration. Also remove any expressions of uncertainty confidence (like "I'm sure". "I think", "maybe", "I'm bit confused", "T'm pretty confident", etc.).
Here are the details: Linearize the reasoning remove backtracking. "Wait," moments, or diversions.
Keep only the details examples directly supporting the final answer make concise. Remove any explicit expressions of uncertainty or confidence (such as "maybe", not sure", think",
'I'm pretty confident", etc.).
Do not fix any factual errors.
Keep the tone casual and natural, like someone logically talking to themselves.
the original thinking degenerates into repetition, or fails reach conclusion, stop the rewrite at the point where the reasoning becomes incomplete or broken. Do not invent, infer, or complete missing reasoning that was not present in the original.
Strictly repeat the first sentence of the model's thinking process in the output.
Use double linebreaks between paragraphs.
Here are some examples:
21

---

C Licenses for existing assets
C.1
Models
RI-Distill 4 MIT
QwQ 35 : Apache-2.0
ORI-Preview
[
Apache-2.0
Qwen-2.5 Apache-2.0
Qwen-3
++ Apache-2.0
GLM-4-0414 39: MIT
GLM-ZZ1-0414
39 MIT
EXAONE-3.5
19 EXAONE
EXAONE-Deep
19
EXAONE
C.2
Datasets
SuperGPQA
21
MMLU-Pro
42
MIT. http:///waagguaaaa..//////a//aaa
NonambigQA 17
23
TriviaQA
13
22

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claim that reasoning models are better calibrated, and slow thinking enables this capability. This claim is supported throughout Section B and Section 4
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: While we have demonstrated that reasoning models hetter express their confidence, we acknowledge and discuss that there's 's still room for improvement in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, those are not discussed the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (c-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g.. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting.
speech-to- seech-to-tex- text system might not used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms how they scale with dataset size.
applicabll, the authors should discuss possible limitations their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and wross-referenced.
23

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed explanation regarding our experiment setting in Section 3.1 and Appendix B
Also our code is available for access.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution is a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct
model (e.g.. large language model), then there should either be
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (e.g.. to registered users), hut it should be possible for other rescarchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justification: Our code is available at the anonymous link mentioned at the first page of the paper. as well as in the supplementary material.
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (bttpp::/nips.cc////
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines httpp////nia.ccppuaabpeeee for more details.
Suadee//oodeSubmisssooooo
The authors should provide instructions data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our full experimental setting and details are explained throughout Section 3.1 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: To support our main claim that reasoning models are better calibrated, we perform
bootstrapping by generating five resampled subsets and report the standard deviation across runs at Appendix(A.22 A222.4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be elearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for caleulating the bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% if the hypothesis of Normality of errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: each experiment, does paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We provide details on our compute resources in Appendix B.
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Fthies
http://arrrppp.eepppppp////aaaaa [Yes]
Answer:
Justification:
The authors have reviewed the NeurlPS Code Ethics.
Guidelines:
The answer means that authors have not reviewed the NeurlPS Code of Fihics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special considcration duc to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work does not have negative socictal impacts. Throughout the paper, we mention that our findings can lead to development of trustworthy and reliable LLM systems.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimatc point out that improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it not needed point out a generic algorithm optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
26

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been seraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all the assets we use in Section 3.1 and list the licenses in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., ., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are relcased, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppppppphseeeoooopppppp thcode. com/ datasete has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should provided.
this information not available online, the authors are encouraged reach out to asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Iustification: do not release new assets other than the experiment code. The documentation code is provided the README. file which can he accessed via the anonymized URI. or from the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations. etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
27

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other
labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval. you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, not include any information that would break anonymity applica- ble), such as institution conducting the review.
16. Declaration of LLL usage
Question: Does the paper describe the usage of LLMs it an important, original, or aon-standard component of the core methods in this research? Note that if the LLM used only for writing, editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Answer: [Yes]
Justification: We leverage LLMs evaluate model responses and to create the ablation data, mentioned in Section 3.1 and Section 4.2
Guidelines:
The answer NA means that core method development this research not involve LLMs as any important, original, or non-standard components.
Please refer our L.LM policy (https (https://neurr should or should not be described.
for what
28


Reviews:
[{"review_id": "cX4ccrdcyI", "reviewer": "Reviewer_VJFc", "summary": "This paper studies reasoning models' behavior on confidence and find that they not only perform better but also give more reliable confidence. They test six models on six tasks and see that reasoning models are better at expressing their confidence in almost all cases. The authors further identified that this is because reasoning models take time to think and try different ideas, which helps them adjust their confidence. This paper finally pointed out that the confidence gets worse for reasoning models without slow thinking, while non-reasoning models can improve confidence if they are guided to think step-by-step.", "questions": "Please see Weakness 2-4.", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\n1. Exploring confidence of reasoning models is an important problem.\n\n2. Comprehensive experiments and ablations to obtain the conclusion.\n\n3. Writing is good and clear.\n\nWeakness:\n\n1. The finding that increased reasoning (CoT) correlates with higher confidence is not novel to this paper. Extensive prior research has established that CoT typically leads to enhanced confidence [1-3].\n\n[1]. Wang, Xuezhi, and Denny Zhou. \"Chain-of-thought reasoning without prompting.\" arXiv preprint arXiv:2402.10200 (2024).\n[2]. Wang, Zezhong, et al. \"Chain-of-Probe: Examing the Necessity and Accuracy of CoT Step-by-Step.\" arXiv preprint arXiv:2406.16144 (2024).\n[3]. Foodeei, Darius, Simin Fan, and Martin Jaggi. \"Semantic uncertainty in advanced decoding methods for LLM generation.\" arXiv preprint arXiv:2506.17296 (2025).\n\n2. Examining solely the relationship between reasoning and confidence may be superficial. The connection between improved confidence and enhanced correctness remains unclear. Previous studies [4-5] have suggested that while long CoT reasoning can boost confidence, it may simultaneously reinforce biases and hallucinations, potentially leading to incorrect conclusions.\n\n[4]. Lu, Haolang, et al. \"Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models.\" arXiv preprint arXiv:2505.13143 (2025).\n[5]. Cheng, Jiahao, et al. \"Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation.\" arXiv preprint arXiv:2506.17088 (2025).\n\n3. Reasoning models may perform significantly worse than non-reasoning models on some knowledge-dense benchmarks such as SimpleQA. It would be interesting to investigate that how reasoning could impact confidence and correctness on these benchmarks.\n\n4. It would be beneficial to conduct more controlled training experiments to establish robust conclusions. For instance, since QwQ and R1-Distill-32B underwent post-training on large-scale and diverse datasets, directly comparing them with Qwen2.5-32B-Instruct may not constitute a fair experimental setup.", "was_revised": true, "final_justification": "After reading the rebuttal and considering the discussions, I am updating my assessment based on the following:\n\nW1: The authors differentiate their focus on verbalized confidence, not logit-based, and show that \"slow thinking\" CoT brings new insights into calibration. \n\nW2: The authors claimed that calibration does not imply correctness. \n\nW3: Additional experiments on SimpleQA support their claims that reasoning models do not underperform and show better calibration.\n\nW4: They demonstrate that comparisons within the same checkpoint control for training differences.\n\nThis paper offers additional contributions to understanding confidence calibration in reasoning models.", "rebuttal": "Thank you Reviewer VJFc for your time and effort in reviewing our work. \n\nWe greatly appreciate you acknowledging the importance of our topic and the value of our comprehensive experiments and ablations.\n\nWe address your “Weakness” in (W1-W4).\n\n---\n\n### **(W1)**\n\n> The finding that increased reasoning (CoT) correlates with higher confidence is not novel to this paper. Extensive prior research has established that CoT typically leads to enhanced confidence.\n\nWe believe that this concern mostly arises from a misunderstanding.\n\nWe first want to kindly clarify that traditional CoT (as opposed to the slow thinking CoT of reasoning models) does not enhance calibration for verbalized confidence, which is the setting of our paper. \n\nWhile the works you have shared does indeed show that traditional CoT can lead to higher confidence, their focus has been on logit-based confidence estimation. In contrast, our work centers on verbalized confidence estimation which is conceptually distinct from logit-based confidence, in that it prompts LLMs to assess and express their confidence directly in their output and does not leverage internal states such as logit-based probabilities. Not only does the methodologies fundamentally differ, verbalized confidence estimation suffers from its own unique challenge: the overconfidence of LLMs [2].\n\nCrucially, according to prior research [1, 2], traditional CoT does not mitigate such overconfidence or enhance calibration for verbalized confidence. Our findings in Section 4.1 are consistent with this: we observe that non-reasoning models do not gain improved calibration with CoT. In contrast, we find that reasoning models get increasingly better calibrated with their slow thinking CoT. Therefore, our finding challenges the prior consensus that CoT does not enhance verbalized confidence calibration, contrary to your concern.\n\nFurthermore, we would like to kindly emphasize that the slow thinking CoT studied in our work is not just simply “increased CoT” already studied by prior works. The content of slow thinking differs considerably from traditional CoT, with complex behaviors such as backtracking or exploring alternatives [3]. The effect of such behaviors on calibration is not well explored by previous research.\n\nFinally, while reasoning models are becoming increasingly popular, their calibration remains underexplored compared to non-reasoning models. Our work is, to the best of our knowledge, the first to suggest that reasoning models exhibit superior calibration, providing a foundation for future research in this area.\n\nTo summarize, we address your concern on the novelty as:\n- We demonstrate that slow thinking CoT enhances verbalized confidence calibration, challenging the prior understanding that CoT does not enhance verbalized confidence calibration (Section 4.1).\n- We provide one of the first comprehensive experimental analyses on the impact of slow thinking behaviors on calibration, highlighting their positive effect (Section 4.2).\n- To our knowledge, our work is the first to suggest that reasoning models exhibit enhanced calibration compared to non-reasoning models in verbalized confidence estimation (Section 3).\n\nWe appreciate you sharing the related works. We will cite them as Related Work and clearly convey how our findings offer novel contributions beyond them.\n\n---\n\n### **(W2)**\n\n> Examining solely the relationship between reasoning and confidence may be superficial. The connection between improved confidence and enhanced correctness remains unclear. Previous studies [4-5] have suggested that while long CoT reasoning can boost confidence, it may simultaneously reinforce biases and hallucinations, potentially leading to incorrect conclusions.\n\nFirst, we respectfully ask you to consider that these findings only emerged recently and were not available at the time of our submission. Specifically, the two preprints you kindly shared were uploaded to arXiv after NeurIPS submission deadline (May 15th) and are not \"previous studies\". As such, we could not have reasonably incorporated or addressed them in our work. Given this, we hope our work will be considered within the scope of the knowledge available at the time of submission.\n\nThat said, we are glad to discuss the point you’ve brought up in more detail.\n\nWe would like to clarify that increase in calibration does not directly correlate to increase in correctness. Calibration (or confidence), in nature, is not intended as a means to enhance correctness; rather, it measures how well a model knows how likely they will be correct [4]. In other words, a well-calibrated model is better at estimating the correctness of their answer, but it is not necessarily better at actually arriving at the correct answer.\n\nTherefore, the findings of the preprints you cited regarding biases and hallucinations in reasoning models do not challenge our conclusion. A model may be better calibrated even if it (allegedly) is less accurate. To elaborate, even if a model frequently gives incorrect answers, it can still achieve high calibration if it precisely acknowledges that it might be wrong.\n\nNonetheless, investigating why reasoning models allegedly produce more hallucinations despite their increased confidence is an interesting direction for future work. The fact that our work is already helping to raise such follow-up research questions when coupled with more recent works highlights its potential impact on shaping future research directions.\n\nPlease let us know if we have misunderstood your point, we would be happy to further clarify or address it.\n\n\n\n### **(W3)**\n\n> Reasoning models may perform significantly worse than non-reasoning models on some knowledge-dense benchmarks such as SimpleQA. It would be interesting to investigate that how reasoning could impact confidence and correctness on these benchmarks.\n\nUnfortunately, we were unable to find any references that support your comment that reasoning models may perform significantly worse on benchmarks such as SimpleQA. The official SimpleQA preprint reports comparable performance between the reasoning and non-reasoning models [5].\n\nIn our paper, we have already evaluated reasoning and non-reasoning models on knowledge-based benchmarks, specifically TriviaQA and NonambigQA (Section 3.2). These are simple factoid QA tasks, similar to SimpleQA. On these datasets, we also did not observe that reasoning models perform significantly worse than non-reasoning models; in fact, both model types achieve comparable accuracy, while reasoning models demonstrate superior calibration.\n\nNonetheless, we appreciate your inquiry and have conducted additional experiments on SimpleQA. \n\n| Model           | Accuracy |   ECE   |  Brier  | AUROC |\n|----------------|----------|---------|---------|-------|\n| Qwen 3 No Think | 0.064    | 0.787   | 0.709   | 0.536 |\n| Qwen 3 Think    | 0.067    | **0.499** | **0.335** | **0.672** |\n|                |          |         |         |       |\n| GLM 4           | 0.097    | 0.676   | 0.599   | 0.619 |\n| GLM Z1          | 0.11     | **0.502** | **0.367** | **0.687** |\n\nConsistent with our results on TriviaQA and NonambigQA, we find that reasoning models do not necessarily exhibit lower accuracy than non-reasoning models, yet they demonstrate significantly better calibration.\n\n---\n\n### **(W4)**\n\n> It would be beneficial to conduct more controlled training experiments to establish robust conclusions. For instance, since QwQ and R1-Distill-32B underwent post-training on large-scale and diverse datasets, directly comparing them with Qwen2.5-32B-Instruct may not constitute a fair experimental setup.\n\nWe agree that directly comparing reasoning models like R1-Distill to models such as Qwen2.5-Instruct may involve confounding factors, as we do not fully know how these models differ in post-training.\n\nHowever, we would like to emphasize that we already provide experiments with this confounding factor fully controlled, since these experiments draw comparisons within a single checkpoint.\n\n- In our main experiments, we compare Qwen3 Thinking mode with Non-thinking mode, where we consistently observe that the Thinking mode is better calibrated (Section 3.2, L:177). Qwen3 is a hybrid model that supports both reasoning (Thinking) and non-reasoning (Non-thinking) modes.\n\n- In our analysis (Section 4.3), we further demonstrate that non-reasoning models can improve their calibration by engaging in slow thinking through in-context learning. Since the model checkpoint remains identical and the only difference is the presence of slow thinking, we believe this setup fully addresses the concern you raised.\n\nTherefore, while we acknowledge that a confounding factor may be present in certain settings, in the experiments where this factor was fully controlled, we observed consistent results. This demonstrates the robustness of our main findings and indicates that the confounding factor is not substantial enough to undermine them.\n\nWe will revise the write-up to clearly convey this point.\n\nThank you again for reviewing our work. Please let us know if any part of your concern remains unaddressed. We are more than happy to discuss further.\n\n---\n\n[1] Tian, Katherine, et al. \"Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.\" EMNLP 2023\n\n[2] Xiong, Miao, et al. \"Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.\" ICLR 2024\n\n[3] Gandhi, Kanishk, et al. \"Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.\" arXiv preprint arXiv:2503.0130\n\n[4] Geng, Jiahui, et al. \"A Survey of Confidence Estimation and Calibration in Large Language Models.\" NAACL 2024\n\n[5] Wei, Jason, et al. \"Measuring short-form factuality in large language models.\" arXiv preprint arXiv:2411.04368"}, {"review_id": "Q2weQrdxLt", "reviewer": "Reviewer_ofVM", "summary": "This work presents a study showing how reasoning models are better calibrated. The study is conducted on multiple LLMs and four different datasets. When explicitly asked to reason about confidence in their thoughts, the reasoning models show better calibration ability according to three metrics (ECE, Brier, and AUROC) despite having similar accuracy. This study also contains ablations showing what aspects of the thoughts are helpful, and shows non-reasoning models can also demonstrate similar improvements by using the thoughts as in-context examples.", "questions": "1. How do you design the prompts that are used in this study? Have you tried multiple different prompts? Is the model sensitive to these changes?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. Understanding how reasoning models (or thinking models) change their behaviors compared to other models is an important topic.\n2. I like the ablations studying what specific parts of the thought help calibration. The analysis is well-designed and insightful.\n3. The experiment is comprehensive, covering multiple models and multiple datasets.\n\n## Weaknesses\n1. I'm fairly convinced that the results in this study are generalizable, but it would still be great to show multiple ways of confidence expression in this work. Currently, this work only shows results on verbalized confidence. It would be interesting to see if the same trends also hold if you sample the models multiple times and compute the confidence from samples. Additionally, I think with greedy decoding, it's probably very hard to sample low-confidence answers, and that's why in Figure 2, the lower part of that figure has very few samples.\n2. This is a minor point. I'm slightly concerned about using \"Wait\" as the token to enforce longer CoT in Line 299. I don't feel \"Wait\" is semantically neutral and it can lead to unintended changes, hence hurting calibration. It would be better if the same study were implemented by just avoiding the tokens that end the thinking process.", "was_revised": true, "final_justification": "This is overall a good empirical analysis paper. I don't see major concerns for this study. The main concern I had was on the sensitivity of the results w.r.t decoding methods and prompt designs, which the authors claim to be robust. I don't see other major concerns from the review/rebuttal from other reviewers, so I keep my original score.", "rebuttal": "Thank you Reviewer ofVM for your positive review of our work.\n\nWe especially appreciate your recognition of the importance of our topic and your acknowledgement of the thoroughness of our experiments and ablations.\n\nWe address your \"Weaknesses\" in (W1) and (W2), and answer your question in (Q1).\n\n---\n\n### **(W1)**\n> I'm fairly convinced that the results in this study are generalizable, but it would still be great to show multiple ways of confidence expression in this work. Currently, this work only shows results on verbalized confidence. It would be interesting to see if the same trends also hold if you sample the models multiple times and compute the confidence from samples. Additionally, I think with greedy decoding, it's probably very hard to sample low-confidence answers, and that's why in Figure 2, the lower part of that figure has very few samples.\n\nWhile there are multiple different approaches to assess an LLM’s confidence (including the consistency based approach you have mentioned), each approach has its own specific challenges. \n\nThe challenge of verbalized confidence stems from the inability of LLMs to accurately judge their own correctness (mostly due to overconfidence). In our work, we have shown the potential of slow thinking in overcoming this specific challenge. Consistency based confidence, on the other hand, does not face the same challenge as the LLM does not judge its own correctness. Rather the challenge rises from increased computation due to repeated sampling.\n\nWe agree that exploring other confidence estimation approaches for reasoning models would be valuable future exploration. However, in this work, similar to prior studies that focus exclusively on verbalized confidence [1, 2], we chose to narrow our scope in order to tackle its unique challenges and provide a focused, in-depth analysis.\n\nRegarding the effect of greedy decoding, we also test sampling and report the average calibration scores with standard deviations in Appendix A.2.3, where our findings are consistent: reasoning models show better calibration even when using sampling.\n\n---\n\n### **(W2)**\n> This is a minor point. I'm slightly concerned about using \"Wait\" as the token to enforce longer CoT in Line 299. I don't feel \"Wait\" is semantically neutral and it can lead to unintended changes, hence hurting calibration. It would be better if the same study were implemented by just avoiding the tokens that end the thinking process.\n\nThank you for raising an interesting point. \n\nWe have chosen to use “Wait” to enforce longer CoTs following a previous study [3]. Specifically, the previous work has demonstrated “Wait”’s effectiveness over other alternatives. Indeed, it’s possible that the semantics of “Wait” may have some influence on calibration. However, we believe investigating the severity of this effect and further designing an alternative that both effectively forces longer reasoning and eliminates such influence is complex on its own and warrants a separate study.\n\n---\n\n### **(Q1)**\n> How do you design the prompts that are used in this study? Have you tried multiple different prompts? Is the model sensitive to these changes?\n\nOur prompt design follows prior work on verbalized confidence. Specifically, for our main experiments we have referenced the CoT prompting settings of seminal works [1, 2].\n\nAs stated in L:157, we tested several alternative prompting strategies and reported the results in Appendix A.2. While we observed some calibration sensitivities, our main finding that reasoning models are better calibrated than non-reasoning models remains consistent across prompts.\n\nThank you again for reviewing our work. Please let us know if anything is still unclear, we’d be happy to follow up.\n\n---\n\n[1] Tian, Katherine, et al. \"Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.\" EMNLP 2023\n\n[2] Xiong, Miao, et al. \"Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.\" ICLR 2024\n\n[3] Muennighoff, Niklas, et al. \"s1: Simple test-time scaling.\" arXiv preprint arXiv:2501.19393 (2025)."}, {"review_id": "6DhGQgzO6p", "reviewer": "Reviewer_PPaK", "summary": "This paper investigates the confidence calibration of reasoning and non-reasoning language models and finds that reasoning models consistently demonstrate better confidence calibration across a variety of datasets.\nFurther analysis shows that improvement stems from exploring alternatives and backtracking.\nBased on this finding, this work propose a simple but effect few-shot method to calibrate the non-reasoning model.", "questions": "1. TriviaQA is a free-form QA dataset. How is answer correctness determined in this paper? Is it based on exact match, or model-based scoring? The method used to assess correctness should be clearly specified", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengthness\n1. The observation that reasoning models express confidence more effectively is  novel. The analysis provided is intuitive and has the potential to influence future developments in both confidence calibration and reasoning-model.\n2. The analysis in Section 4.2 carefully disentangles multiple factors within the reasoning process that may affect calibration, ultimately identifying _non-linear reasoning_ as the dominant contributor. This insight is valuable and directly informs the design of the simple but highly effective calibration method proposed in Section 4.3.\n\n\n### Weakness\n\n1. Prompt Is Very Important! The same prompt is used for both reasoning and non-reasoning models, it appears to be specifically designed for reasoning models. [1] shows prompting non-reasoning models to output CoT or express confidence linguistically can actually degrade calibration. It is possible that the observed calibration advantage of reasoning models is prompt-specific. The authors should consider testing multiple prompt variants for non-reasoning models, including the most effective ones reported in [1].\n2. The calibration gap are relatively small in some cases. For instance, in Table 10 (TriviaQA), the Brier score difference is only 0.005 and AUROC differs by 0.015. These are modest gains and may fall within the range of variation caused by prompt sensitivity alone.\n\n\\[1\\] Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback", "was_revised": true, "final_justification": "The rebuttal addressed my concerns on the prompt.", "rebuttal": "Thank you Reviewer PPaK for your positive review of our work.\n\nWe appreciate your recognition of its novelty, potential impact, and the value of the analysis.\n\nWe address your \"Weaknesses\" in (W1) and (W2), and answer your question in (Q1).\n\n--- \n\n### **(W1)**\n> Prompt Is Very Important! The same prompt is used for both reasoning and non-reasoning models, it appears to be specifically designed for reasoning models. [1] shows prompting non-reasoning models to output CoT or express confidence linguistically can actually degrade calibration. It is possible that the observed calibration advantage of reasoning models is prompt-specific. The authors should consider testing multiple prompt variants for non-reasoning models, including the most effective ones reported in [1].\n\nWe respectfully believe that your concern rises from a misunderstanding.\n\nWe would like to first clarify that we did not specifically design the prompt for reasoning models. Our prompt used in the main experiments was designed after the prompts used in previous seminal works (back when there were no reasoning models) [1, 2], and is generalizable for both reasoning and non-reasoning models. We will revise our manuscript to clearly convey this point.\n\nWe would also like to clarify that we have already investigated the robustness of our findings across multiple different prompts, which includes the points that you have mentioned. This is stated in L:157 and in Appendix A.2.\n\nSpecifically, \n- In Appendix A.2.1 we test advanced prompting strategies (Top-K and Multi-step) suggested by the work you have mentioned [1]. \n- In Appendix A.2.2, we tested the effect of different confidence expression styles (linguistic, numeric, and linguistic + numeric).\n\nIndeed, calibration does show some sensitivity to prompt variations, as you noted. Yet, across all tested settings, our main finding remains consistent: reasoning models are better calibrated than non-reasoning models. Given the breadth of prompting strategies we evaluated, we believe our conclusion is well-supported and robust. \n\nIf there are any additional aspects of prompt robustness you'd like us to explore, we're happy to provide further results or clarifications.\n\n---\n### **(W2)**\n> The calibration gap are relatively small in some cases. For instance, in Table 10 (TriviaQA), the Brier score difference is only 0.005 and AUROC differs by 0.015. These are modest gains and may fall within the range of variation caused by prompt sensitivity alone.\n\nWe hope that we have sufficiently addressed the prompt sensitivity concern in (W1). Here we explain the reason behind the small calibration gap in TriviaQA in a few cases. In a nutshell, this is because LLMs achieve relatively high accuracy on TriviaQA (around ~70% accuracy), which makes the overconfidence and poor calibration of non-reasoning models less pronounced. In fact, on a more challenging dataset like NonambigQA (with ~50% accuracy) the calibration gap between reasoning and non-reasoning models widens, leading to a Brier score gap of 0.083 (Table 10).\n\nTo elaborate, non-reasoning models tend to be overconfident, frequently assigning high confidence levels (e.g., 85% or 95%) regardless of correctness (see Figure 2). When task accuracy is also high, this overconfidence appears less problematic, as the confidence roughly aligns with accuracy. However, in lower-accuracy settings, this “unintended alignment” breaks, as the overconfidence is no longer aligned with the accuracy. \n\nOn the other hand, reasoning models are able to generate relatively accurate confidence levels. This strength is more pronounced on low-accuracy tasks, where non-reasoning models do not have the unearned benefit of the “unintended alignment”.\n\nAll in all, we argue that the small calibration gap in TriviaQA doesn’t necessarily suggest that reasoning and non-reasoning models show similar levels of calibration. Rather, the small gap reflects limitations of the calibration metrics in capturing overconfidence and poor calibration when accuracy is high.\n\nTo further support our point, we provide additional results on SimpleQA [3], a factoid-based QA dataset similar to TriviaQA and NonambigQA but substantially more challenging. \n\n| Model           | Accuracy |   ECE   |  Brier  | AUROC |\n|----------------|----------|---------|---------|-------|\n| Qwen 3 No Think | 0.064    | 0.787   | 0.709   | 0.536 |\n| Qwen 3 Think    | 0.067    | **0.499** | **0.335** | **0.672** |\n|                |          |         |         |       |\n| GLM 4           | 0.097    | 0.676   | 0.599   | 0.619 |\n| GLM Z1          | 0.11     | **0.502** | **0.367** | **0.687** |\n\nAs shown in the Table above, the models achieve low accuracy, which leads to a massive difference in calibration score between reasoning and non-reasoning models (0.374 and 0.232 Brier score difference for Qwen 3 and GLM, respectively), clearly highlighting the superior calibration of reasoning models.\n\nThank you for raising this point. We will make sure to include this explanation in the Discussion section of the paper.\n\n---\n### **(Q1)**\n> TriviaQA is a free-form QA dataset. How is answer correctness determined in this paper? Is it based on exact match, or model-based scoring? The method used to assess correctness should be clearly specified\n\nFollowing prior work on verbalized confidence [1, 2], we also use an LLM to match the generated answer to the ground truth, as noted in L:143.\n\nThank you again for reviewing our work. If we have overlooked any part of your concern, please let us know. We are happy to engage further.\n\n---\n[1] Tian, Katherine, et al. \"Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.\" EMNLP 2023\n\n[2] Xiong, Miao, et al. \"Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.\" ICLR 2024\n\n[3] Wei, Jason, et al. \"Measuring short-form factuality in large language models.\" arXiv preprint arXiv:2411.04368"}]

Author rebuttal:

Meta-review: This paper investigates the confidence calibration of reasoning versus non-reasoning large language models (LLMs). This work finds that reasoning models, especially those exhibiting “slow thinking” behaviors such as backtracking and exploring alternatives, tend to demonstrate superior calibration of verbalized confidence, even when accuracy is similar. The paper further shows that non-reasoning models can partially benefit from in-context reasoning demonstrations, providing a practical path to improving calibration.

The reviewers agree that the topic is important and timely, as calibration is central to building trustworthy LLMs. The experiments are comprehensive, covering multiple models and datasets, and include ablations that disentangle aspects of reasoning behavior contributing to calibration. The paper is clearly written, and the rebuttal usefully clarified the distinction between verbalized confidence (the paper’s focus) and logit-based confidence (common in prior work). This helps establish novelty. Additional experiments (e.g., SimpleQA) strengthened claims by addressing concerns about reasoning models underperforming on knowledge-dense benchmarks.

There are also some weaknesses identified during the review and discussion.
- Some reviewers noted that the initial framing risked overstating novelty, since prior work has studied CoT and confidence. However, the rebuttal clarified that this paper’s contribution is specifically on verbalized confidence calibration in reasoning models, which is distinct and less explored.
- Concerns remain about prompt sensitivity and the modest effect sizes on certain datasets. While the authors provided robustness checks and additional explanations (e.g., why gaps are smaller on high-accuracy datasets like TriviaQA), the issue of generalizing across prompting strategies and decoding methods could be explored further.
- One reviewer suggested exploring multiple modes of confidence expression (e.g., sample-based consistency) beyond verbalized confidence, which the authors acknowledge as important but out of scope for this work.

The reviews reflect a progression from initial skepticism (borderline reject) to greater confidence in the paper’s contributions after rebuttal and discussion. While not all concerns are fully resolved—particularly around robustness and generalization—the paper makes a substantive empirical contribution by demonstrating that reasoning models are better calibrated in verbalized confidence, and by identifying mechanisms (slow thinking behaviors) that explain this effect. This adds useful insight for both the reasoning model and calibration communities.
Decision: Accept (poster)