Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervision

## Abstract

Abstract Uncertainty calibration is essential for the safe deployment of large language models (LLMs), particularly when users rely on verbalized confidence estimates. While prior work has focused on classifiers or short-form generation, confidence calibration for chain-of-thought (CoT) reasoning remains largely unexplored. Surprisingly, we find that supervised fine-tuning with scalar confidence labels alone suffices to elicit self-verification behavior of language models, without any explicit reasoning supervision or reinforcement learning-based rewards. Despite being trained only to produce a verbalized confidence score without any self-verifying examples, the model learns to generate longer and self-checking responses for low-confidence queries while providing more concise answers for high-confidence ones. We further propose a simple rethinking method that boosts performance via test-time scaling based on calibrated uncertainty. Experiments on GSM8K and held-out reasoning tasks such as MATH-500 and ARC-Challenge show that our confidence-aware fine-tuning improves both calibration and accuracy, while also enhancing interpretability by aligning the model’s reasoning path with its confidence.

## 1 Introduction

Large language models (LLMs) demonstrate strong performance not only in natural language generation but also in complex reasoning and decision-support tasks across diverse domains (achiam2023gpt; guo2025deepseek). Their application to high-stakes settings such as medical diagnosis and personalized financial analysis has drawn increasing attention (goh2024large; qiu2024llm; takayanagi2025generative), with the potential to reduce expert workload and accelerate decision-making.

Nonetheless, ensuring the reliability of LLMs remains a critical challenge. Models frequently produce incorrect outputs with high confidence, and such overconfident errors can lead to harmful decisions when left undetected by users (zhou2024relying). These hallucinations pose risks that go beyond factual inaccuracies, with implications for healthcare, law, and finance (du2024haloscope). To mitigate these risks, LLMs must be able to quantify and communicate their uncertainty in a human interpretable manner, such as through *verbalized confidence*.

Figure: Figure 1: Generation length and self-verification rate across confidence bins on GSM8K using the CSFT-trained LLaMA-3.2-3B-Instruct model. Lower-confidence bins yield longer outputs and higher self-verification rates, suggesting a learned fallback behavior. Overall, 20% of generations showed self-verification (measured using GPT-4.1), compared to under 1.5% in the zero-shot setting, which is omitted. Representative examples are shown in [Figure 2](https://arxiv.org/html/2506.03723v1#S4.F2).
Refer to caption: x1.png

While a few recent studies train models to explicitly verbalize their confidence (band2024linguistic; kapoorlarge; stengel2024lacie; jang2024calibrated), most rely on complex procedures such as reinforcement learning or classifier probing. Moreover, they report little evidence of generalization in zero-shot settings or under chain-of-thought (CoT) reasoning. At the same time, state-of-the-art LLMs increasingly tackle challenging problems by generating explicit CoT traces (zhou2023leasttomost; chen2023progofthought; wang2023selfconsistency). The reliability of verbalized confidence within such CoT reasoning, however, remains largely unexplored.

In contrast, this work shows that even a simple confidence-supervised fine-tuning (CSFT) approach, under suitable conditions, can yield well-calibrated verbalized confidence in CoT reasoning. Furthermore, we demonstrate that the model can autonomously adjust its response length and exhibit emergent self-verification behavior as a function of its uncertainty. Specifically, without reasoning supervision, the model learns to modulate its reasoning process while being trained to generate an answer, followed by a verbalized confidence score. As illustrated in [Figure 1](https://arxiv.org/html/2506.03723v1#S1.F1), low-confidence predictions result in longer outputs with self-check phrases such as “recalculate” or “let me double-check”, whereas high-confidence responses are shorter and more decisive. This phenomenon emerges even without CoT reasoning guidance in training, and is consistently observed across GSM8K, MATH-500, and ARC-Challenge. These findings suggest that verbalized confidence can serve not only as a calibration target but also as an effective training signal that encourages more deliberate chain-of-thought generation, ultimately leading to improved reasoning accuracy.

Our contributions are as follows:

- •
We propose CSFT, a simple confidence-supervised fine-tuning method using the problems and the corresponding synthetic self-confidence labels that enables reliable verbalized confidence in CoT reasoning tasks.
- •
We demonstrate that CSFT elicits emergent self-verification behavior without requiring reasoning supervision, manifesting as a systematic relationship between confidence and output length.
- •
We analyze how prompting style, regularization strength, and reasoning depth affect this phenomenon, and demonstrate generalization to held-out reasoning tasks.

These results provide a scalable path toward building uncertainty-aware LLMs using standard SFT pipelines, without architectural modifications or post-hoc correction. CSFT not only improves calibration but also guides model behavior toward safer and more interpretable reasoning.

## 2 Related Works

### 2.1 Confidence Calibration in LLMs

Calibration in LLMs has been studied from various perspectives. Likelihood-based methods estimate uncertainty using token-level entropy, sequence probabilities, or generation variance (desai2020calibration; nguyen2024semantic; kadavath2022language). These methods are helpful for model-side diagnostics, but they do not yield human-readable confidence statements.
Verbalized confidence, where models explicitly articulate how sure they are, has emerged as a more user-friendly and interpretable alternative. (band2024linguistic; stengel2024lacie). However, most existing approaches focus on short-form declarative QA and require manual scalar labels (lin2024confidence) or classifier-based tuning (kapoorlarge; jang2024calibrated), without an understanding of the dynamics of reasoning.
In the context of CoT reasoning, recent work has reported that instruction-tuned or reasoning-supervised models (yang2025qwen3; guo2025deepseek) exhibit better calibration under zero-shot inference (yoon2025reasoning). However, these findings remain observational and do not examine how calibration can be systematically induced or controlled in reasoning tasks.
In contrast, our work provides the first direct evidence that fine-tuning with weak, self-derived confidence labels, obtained through consistency across sampled answers, can induce improved calibration and emergent self-verification in reasoning, even in models with no prior exposure to CoT supervision.

#### 2.1.1 Self-Verification and CoT Optimization

Self-verification has emerged as a desirable property for LLMs, with prior work demonstrating that models capable of revisiting and refining their reasoning tend to achieve higher accuracy and robustness, especially on complex tasks (wang2023selfconsistency).
Accordingly, a growing body of research has focused on optimizing the structure of CoT outputs, either by making them more concise (nayab2024concise; team2025kimi) or by generating longer and more reflective traces (guo2025deepseek).
However, these approaches typically target a fixed generation style, without conditioning on the model’s internal uncertainty.
In reality, effective reasoning should adapt to the model’s confidence: when confidence is low, the model should elaborate and verify; when confidence is high, a brief and decisive answer may suffice.
Most existing methods do not capture this dynamic.
Reinforcement learning-based approaches train models to favor verifiable traces (chen2023progofthought; zhao2025learning; shafayat2025can), but require reward shaping and large-scale tuning.
We show that a single round of fine-tuning on scalar confidence labels derived from self-consistency induces CoT behaviors that adaptively reflect the model’s uncertainty.

## 3 Confidence-Supervised Fine-Tuning

We introduce Confidence-Supervised Fine-Tuning (CSFT), a simple yet effective method for calibrating verbalized confidence in LLMs under reasoning scenarios, without requiring explicit supervision of the reasoning process. CSFT fine-tunes the model to calibrate its verbalized confidence, while also producing a CoT reasoning trace and final answer, so that the reported confidence more accurately reflects the model’s belief in the correctness of its answer.

Given a question $q$, the decoder is trained to generate a structured response consisting of: (i) a CoT reasoning trace $r$ and final answer $a$, enclosed within <think> …  </think> and <answer> …  </answer> tags, respectively; followed by (ii) a suffix confidence prompt (see [Appendix C](https://arxiv.org/html/2506.03723v1#A3)), which elicits a discrete confidence score $c\in\{0,10,\dots,100\}$ expressed in <confidence> …  </confidence> tags. Only the confidence score $c$ is supervised during training; both the reasoning $r$ and the answer $a$ remain unconstrained.

In this section, we used Low Rank Adaptation (LoRA; hu2021lora) method to fine-tune the LLM model $f_{\theta_{0}}$. Here, we denote $\theta_{0}$ as fixed pre-trained parameters and $\theta$ as fixed pre-trained parameters with additional learnable LoRA weights.

##### Self-Confidence label.

To compute the confidence label, we first sample $K$ full generations $\{(r^{(i)},a^{(i)})\}_{i=1}^{K}\sim f_{\theta}(\cdot\mid q)$ and estimate the empirical accuracy as

$$ $\displaystyle\hat{p}(q)=\frac{1}{K}\sum_{i=1}^{K}\mathbbm{1}\bigl{[}a^{(i)}=a^ {\star}\bigr{]},$ (1) $$

where $a^{\star}$ denotes the gold answer. To determine whether each answer $a^{(i)}$ matches the gold answer $a^{\star}$, we first parsed the value between <answer> …  </answer> from the LLM response to extract $a^{(i)}$, and then checked whether it exactly matched the gold answer.
Then, the self-confidence label was obtained by discretizing the accuracy: $c=\lfloor 100\cdot\hat{p}(q)\rfloor$.

##### Training objective.

Let $T_{c}$ denote the token positions corresponding to the entire confidence span, including the <confidence> …  </confidence> tags.
CSFT minimizes the masked cross-entropy loss over these positions:

$$ $\displaystyle\mathcal{L}_{\text{CSFT}}=-\sum_{t\in T_{c}}\log p_{\theta}(y_{t} \mid y_{<t},q),$ (2) $$

where $p_{\theta}(y)$ indicates the predicted probability of token $y$ from LLM $f_{\theta}$. And optionally, we add a KL regularization term over the CoT and answer spans (including their respective tags), where the corresponding token positions are denoted by $T_{\text{KL}}$, to encourage the model to remain close to the pretrained distribution:

$$ $\displaystyle\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{CSFT}}+\lambda\sum_ {t\in T_{\text{KL}}}\mathrm{KL}\!\left(p_{\theta}\;\|\;p_{\theta_{0}}\right),$ (3) $$

where $\lambda$ is a weighting hyperparameter, and $p_{\theta}$ and $p_{\theta_{0}}$ represents $p_{\theta}(\cdot\mid y_{<t},q)$ and $p_{\theta_{0}}(\cdot\mid y_{<t},q)$, respectively. Unless otherwise noted, we set $\lambda=0$.

##### Calibration effect.

Because the target $c$ reflects the empirical accuracy of the model’s own generations, CSFT encourages alignment between predicted confidence and actual correctness. This leads to improved calibration, as measured by standard metrics such as expected calibration error (ECE; ece) in CoT reasoning tasks.

## 4 Experiments

##### Experimental Setup.

We construct our training corpus by sampling $K=10$ CoT traces and their corresponding answers for each question $q$ from the GSM8K (cobbe2021training) training split, and assigning a self-consistency label based on the proportion of sampled answers that match the gold answer. Using this signal, we fine-tune LLaMA3.2-3B-Instruct (grattafiori2024llama) and Qwen2.5-1.5B-Instruct (yang2025qwen3), and evaluate accuracy, Area Under the Receiver Operating Characteristic curve (AUROC), and calibration metrics (ECE, Brier Score) on the GSM8K test set as well as on the held-out reasoning benchmarks MATH-500 (lightman2023let) and ARC-Challenge (clark2018think). Further experimental details are provided in [Appendix A](https://arxiv.org/html/2506.03723v1#A1).

### 4.1 Main Experiments

**Table 1: Calibration results on the in-distribution GSM8K and two held-out reasoning benchmarks. For datasets, ✓: ‘seen’ during CSFT, ✗: ‘unseen’ during CSFT. For the metrics, $\downarrow$: lower is better and $\uparrow$: higher is better.**
| Dataset | Model | Method | AUROC ($\uparrow$) | ACC ($\uparrow)$ | ECE ($\downarrow$) | BS ($\downarrow$) | Avg. Len. |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GSM8K (✓) | LLaMA3.2-3B-Instruct | Pre-trained | 50.57 | 68.68 | 0.2065 | 0.2549 | 226.66 |
| CSFT | 81.25 | 71.34 | 0.0568 | 0.1450 | 288.71 |  |  |
| Qwen2.5-1.5B-Instruct | Pre-trained | 49.59 | 67.85 | 0.1928 | 0.2915 | 250.21 |  |
| CSFT | 67.67 | 69.63 | 0.0552 | 0.2285 | 291.70 |  |  |
| MATH-500 (✗) | LLaMA3.2-3B-Instruct | Pre-trained | 49.57 | 41.20 | 0.4730 | 0.4800 | 416.70 |
| CSFT | 62.97 | 56.60 | 0.1776 | 0.3059 | 559.33 |  |  |
| Qwen2.5-1.5B-Instruct | Pre-trained | 59.91 | 55.00 | 0.3786 | 0.2978 | 444.86 |  |
| CSFT | 60.27 | 56.40 | 0.2590 | 0.2629 | 477.96 |  |  |
| ARC-Challenge (✗) | LLaMA3.2-3B-Instruct | Pre-trained | 53.89 | 65.36 | 0.2251 | 0.2738 | 210.50 |
| CSFT | 72.58 | 69.45 | 0.0647 | 0.1853 | 293.13 |  |  |
| Qwen2.5-1.5B-Instruct | Pre-trained | 54.08 | 52.07 | 0.1660 | 0.2680 | 105.55 |  |
| CSFT | 61.63 | 56.82 | 0.1107 | 0.2584 | 114.85 |  |  |

##### Evaluation on GSM8K dataset.

[Table 1](https://arxiv.org/html/2506.03723v1#S4.T1) show results on the GSM8K test set, where our model was trained using CSFT with the GSM8K training dataset. Our method consistently improves all calibration metrics and accuracy over the pre-trained baseline, not only on LLaMA3.2-3B-Instruct but also on Qwen2.5-1.5B-Instruct. Beyond better alignment between predicted confidence and correctness, we observe that CSFT induces desirable reasoning behaviors, such as internal error checking and more deliberate output construction—hallmarks of self-verification, particularly in low-confidence cases. Refer to [Fig. 2](https://arxiv.org/html/2506.03723v1#S4.F2) to see an example. And in such low-confidence cases, responses based on self-verification can serve as a mechanism for users to view the predicted model confidence as more reliable.

##### Unseen CoT Tasks.

To evaluate the generalization capability of the model trained with CSFT, we test it on two reasoning benchmarks—MATH-500 and ARC-Challenge—that are structurally, topically, and cognitively distinct from GSM8K. These benchmarks allow us to assess how well the learned reasoning patterns transfer to the unseen domain and more challenging problem distributions.
As shown in [Table 1](https://arxiv.org/html/2506.03723v1#S4.T1), CSFT improves generalization to unseen CoT tasks for both models. On MATH-500, LLaMA3.2-3B-Instruct achieves a +37$\%$ accuracy gain and 63$\%$ ECE reduction, while Qwen2.5-1.5B-Instruct yields a +2.5$\%$ accuracy gain and 32$\%$ lower ECE. On ARC-Challenge, LLaMA improves accuracy by 6.3$\%$ and ECE by 71$\%$, and Qwen shows 9.1$\%$ higher accuracy and 33$\%$ better calibration.
These results indicate that our self-consistency-based calibration transfers beyond the training task, improving both confidence alignment and problem-solving ability on diverse reasoning challenges.

### 4.2 Self-Verification Behaviour

In this section, we analyze how training with CSFT induces self-verification behavior and how this behavior correlates with predicted confidence levels. We further support our analysis with concrete examples. First, [Figure 1](https://arxiv.org/html/2506.03723v1#S1.F1) presents two key relationships: (1) the average CoT token length as a function of predicted confidence, and (2) the proportion of answers that trigger self-verification across different confidence levels.

The results in [Figure 1](https://arxiv.org/html/2506.03723v1#S1.F1) show a clear trend: CSFT-trained models generate significantly longer outputs when their confidence is low. In particular, for the lowest-confidence bin (0), the average output length is nearly five times longer than that of the zero-shot baseline. This indicates that the model compensates for low confidence by engaging in extended reasoning, suggesting that self-verification is an emergent behavior tied to uncertainty. This increase in length is closely accompanied by a high self-verification rate, with nearly all corrected answers involving explicit verification behaviors. Moreover, this self-verification behavior under low-confidence scenarios can serve as a valuable mechanism for end users relying on LLM responses. It provides an implicit signal that the model is uncertain and is actively working to validate its answer, thereby enhancing the perceived reliability of the predicted confidence from the user’s perspective. As shown in [§ B.1](https://arxiv.org/html/2506.03723v1#A2.SS1), both the length and self-verification patterns generalize beyond the training distribution, emerging similarly on unseen CoT tasks as well.

##### Qualitative Examples.

Figure: Figure 2: Confidence reshapes reasoning behavior. Left: In the low-confidence case (top), the CSFT model generates a long reasoning trace with explicit self-verification, eventually arriving at the correct answer, while the zero-shot model fails. Right: In the high-confidence case (bottom), both models give the correct answer, but the CSFT response is significantly more concise, reflecting confidence-aware brevity.

In [Figure 2](https://arxiv.org/html/2506.03723v1#S4.F2), we provide qualitative examples comparing responses generated by the pre-trained baseline and the model fine-tuned with CSFT. The results highlight that CSFT-trained models dynamically adjust both the length and structure of their reasoning based on the predicted confidence.
Low-confidence cases elicit longer, reflective traces with internal correction, whereas high-confidence responses tend to be brief and decisive.

### 4.3 Ablation Studies

Figure: Figure 3: Test accuracy and ECE on GSM8K using CSFT-trained LLaMA3.2-3B-Instruct, evaluated under varying KL weights. Prefix performance declines without KL, whereas suffix remains stable or improves.
Refer to caption: x2.png

##### Prompt Position.

The suffix setting, in which the confidence prompt is appended after the model’s answer, is described in [§ 3](https://arxiv.org/html/2506.03723v1#S3).
In addition, we evaluate a prefix variant where the confidence prompt is inserted immediately after the question and before any reasoning begins (see [Appendix C](https://arxiv.org/html/2506.03723v1#A3)).
This placement allows the model to condition its reasoning on anticipated confidence, which may affect both generation and calibration.
As shown in [Figure 3](https://arxiv.org/html/2506.03723v1#S4.F3), the prefix setting consistently yields lower accuracy than the zero-shot baseline, with performance sharply degrading when KL regularization is removed (i.e., $\lambda=0$).
This suggests that without constraints on the CoT and answer spans, the model may overfit to expressing uncertainty rather than reasoning accurately, effectively learning to be confidently wrong or confidently uncertain.
In contrast, the suffix setting shows more favorable behavior. When KL regularization is removed, both calibration and accuracy improve compared to the zero-shot baseline.
These results highlight a key trade-off: prefix prompting influences the generation process and requires regularization to remain effective, while suffix prompting is more stable because it does not interfere with the model’s reasoning.

##### KL Regularization.

[Figure 3](https://arxiv.org/html/2506.03723v1#S4.F3)
We investigate the effect of varying the KL regularization weight $\lambda$ on model performance, focusing on its impact near $\lambda=0$ (see [Figure 3](https://arxiv.org/html/2506.03723v1#S4.F3), Zoom-in figure).
The motivation for this analysis is to examine whether performance gains at low $\lambda$ are stable or merely an artifact of tuning.
For the prefix setting, we observe that performance rapidly deteriorates as KL regularization is removed. This suggests that, without a constraint to preserve the pretrained distribution over CoT and answer spans, the model may exploit the freedom to optimize ECE at the expense of actual reasoning quality. In effect, the model becomes well-calibrated but confidently incorrect.
In contrast, the suffix setting remains stable or even improves in the absence of KL regularization. Since confidence is predicted independently after the full generation, removing the KL constraint does not impair reasoning quality, and may in fact allow for better post hoc alignment of confidence with correctness.
These results highlight the importance of controlling model behavior when confidence supervision is introduced at generation time (prefix), as opposed to after-the-fact (suffix).

##### Label Quality and Confidence Question.

**Table 2: Ablation analysis on the impact of confidence label quality and the inclusion of confidence prompt. Results are reported as differences relative to CSFT on GSM8K using LLaMA3.2-3B-Instruct.**
| Variant | ACC | ECE | Avg. Len. |
| --- | --- | --- | --- |
| w/o Correct label | -2.14 | +0.05 | -49.36 |
| w/o Conf question | Training collapsed |  |  |

To evaluate the role of confidence supervision and the design of the confidence prompt in enabling stable training and achieving strong performance, we conduct two ablation studies, as summarized in [Table 2](https://arxiv.org/html/2506.03723v1#S4.T2).
First, we randomly assign confidence labels within the <confidence> tag, breaking the link between prediction quality and label supervision.
Second, we remove the explicit confidence prompt, instead asking the model to generate a scalar confidence directly after the final answer using a repeated <answer> tag.
In the first setting, performance significantly degrades across all metrics, confirming that accurate supervision is critical for learning calibrated confidence.
In the second setting, the model fails to train altogether, suggesting that without an explicit instruction to predict confidence, the model cannot ground the meaning of the scalar and collapses.

##### Impact of CoT Visibility on the Confidence.

Figure: Figure 4: Reliability curves of CSFT-trained LLaMA3.2-3B-Instruct on GSM8K, comparing predicted confidence when elicited before vs. after CoT generation. The broadly similar calibration profiles suggest that the model’s confidence reflects internal uncertainty rather than simply CoT length.
Refer to caption: x3.png

To test whether the model’s confidence relies on observing the length or content of the generated CoT, we compare calibration when confidence is elicited before versus after CoT generation.
As shown in [Figure 4](https://arxiv.org/html/2506.03723v1#S4.F4), the two reliability curves are broadly similar, suggesting that the model does not depend heavily on CoT visibility and instead bases its confidence on internal uncertainty.

### 4.4 Confidence-Guided Reasoning Path Refinement

**Table 3: Manual rethinking improves accuracy in low-confidence bins on GSM8K test set using LLaMA3.2-3B-Instruct ($\Delta$ACC).**
| Bin | 0 | 10 | 20 | 30 |
| --- | --- | --- | --- | --- |
| $\Delta$ACC | +0.5625 | +0.5524 | +0.3158 | +0.2069 |
| Count | 16 | 143 | 38 | 29 |

When a model is well-calibrated, its verbalized confidence serves as a trustworthy signal for downstream decision-making and control. As demonstrated previously, the CSFT-trained model is capable of accurately predicting its confidence even before generating the full CoT. This capability opens up the possibility of guiding the reasoning trajectory from the very beginning.

We exploit this by manually redirecting low-confidence samples toward alternative reasoning paths. Specifically, if the model expresses low confidence in its initial output(elicited via the prefix prompt shown in [Figure 9](https://arxiv.org/html/2506.03723v1#A3.F9)), we initiate a new reasoning attempt with an altered or more structured prompt (as shown in [Figure 14](https://arxiv.org/html/2506.03723v1#A3.F14)), without waiting for failure. This preemptive rethinking mechanism enables selective refinement with minimal additional cost.

As shown in [Table 3](https://arxiv.org/html/2506.03723v1#S4.T3), this confidence-aware rethinking strategy substantially improves accuracy in the low confidence bins. For example, in the 0–10 confidence range, accuracy improves by over 55 percentage points. This result underscores the utility of confidence not just for post hoc calibration, but also for guiding efficient and cost-aware reasoning-time improvement.

## 5 Conclusion and Future Work

To develop well-calibrated LLMs capable of expressing trustworthy verbalized confidence in CoT reasoning, we propose a simple yet effective fine-tuning method called CSFT. For training, we construct a synthetic dataset based on the GSM8K training set, where each problem is paired with a self-generated confidence label.

Empirically, we demonstrate that LLMs fine-tuned with CSFT achieve substantial improvements across multiple evaluation metrics—including accuracy and ECE—on both the GSM8K test split and out-of-domain benchmarks. Remarkably, we also observe that CSFT elicits the emergence of self-verification behavior, particularly in low-confidence scenarios, despite not providing any explicit supervision related to reasoning strategies during training. Moreover, this self-verification behavior in low-confidence scenarios can act as a valuable mechanism for end users interacting with LLMs. It serves as an implicit indicator that the model recognizes its own uncertainty and is actively attempting to validate its response.

Looking forward, several directions emerge.
First, since we observe that confidence can be elicited prior to reasoning, it may be possible to predict the downstream cost of a reasoning trajectory (e.g., output length or compute usage) from the initial confidence.
This opens up opportunities for confidence-conditioned inference policies that balance accuracy and efficiency. Second, while self-verification is desirable under uncertainty, we find that some low-confidence generations enter excessively long or redundant reasoning loops, potentially reflecting local minima in the generation dynamics. Third, one could explore confidence-aware steering of CoT trajectories, or use latent confidence signals to trigger rethink-style interventions without incurring full-generation overhead.

## Appendix A Experimental details

In this section, we provide detailed information on the models and datasets used in our experiments, along with formal definitions of the calibration metrics employed for evaluation. Specifically, we describe the two instruction-tuned language models used: LLaMA3.2-3B-Instruct and Qwen2.5-1.5B-Instruct. For datasets, we include:

- •
GSM8K (cobbe2021training); a dataset of 7.47k grade-school math word problems designed to test step-by-step reasoning. We use 10% and 20% of the original training set (0.75k and 1.49k examples, respectively) for training and validation, and the full test set of 1.32k examples for evaluation. Available at [huggingface.co/openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k).
- •
MATH-500 (lightman2023let); a subset of the MATH dataset consisting of 500 diverse high school level problems covering algebra, geometry, calculus, and more. Used solely for evaluation. Available at [huggingface.co/HuggingFaceH4/MATH-500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500).
- •
ARC-Challenge (clark2018think); a multiple-choice science and commonsense QA benchmark containing 1.17k test questions that require reasoning beyond surface-level cues. We use the test set for evaluation. Available at [huggingface.co/allenai/ai2_arc](https://huggingface.co/datasets/allenai/ai2_arc).

### A.1 Model and datasets

**Table 4: Training hyperparameters used for CSFT fine-tuning across both models.**
| Hyperparameter | Value |
| --- | --- |
| Batch size | 1 |
| Gradient accumulation | 16 |
| Learning rate | [1e-5, 1e-04] |
| Optimizer | AdamW |
| Weight decay | 0.0 |
| Warmup ratio | 0.0 |
| Max sequence length | 1024 |
| KL regularization ($\lambda$) | 0.0 |
| Training steps | 2500 |
| Checkpoint selection | Best dev loss |
| LoRA configuration |  |
| LoRA rank ($r$) | 128 |
| LoRA alpha | 32 |
| LoRA dropout | 0.1 |
| LoRA target modules | $q_{\text{proj}}$, $v_{\text{proj}}$ |

#### A.1.1 Calibration metrics

- •
Expected Calibration Error (ECE; ece):
$\text{ECE}=\sum_{m=1}^{M}\frac{|B_{m}|}{n}\left|\text{acc}(B_{m})-\text{conf}(
B_{m})\right|$
where $B_{m}$ is the set of predictions in bin $m$, $\text{acc}(B_{m})$ is the accuracy, and $\text{conf}(B_{m})$ is the average confidence of the predictions in that bin. ECE measures how well the model’s predicted probabilities are calibrated.
- •
Brier Score (BS; brier):
$\text{BS}=\frac{1}{N}\sum_{i=1}^{N}(f_{i}-y_{i})^{2}$
where $f_{i}$ is the predicted probability and $y_{i}$ is the true label. BS combines both the accuracy and confidence of the predictions, penalizing overconfident and underconfident predictions.

## Appendix B Additional Results

### B.1 Length Analysis on Held-out CoT Tasks

Figure: Figure 5: Output length across confidence bins on Math-500, using LLaMA3.2-3B-Instruct fine-tuned with CSFT. The model generates significantly longer responses when confidence is low, while high-confidence predictions tend to be more concise.
Refer to caption: x4.png

Figure: Figure 6: Output length across confidence bins on ARC-Challenge, using LLaMA3.2-3B-Instruct fine-tuned with CSFT. The same trend of confidence-conditioned length modulation emerges, despite the task being unseen during training.
Refer to caption: x5.png

[Figure 5](https://arxiv.org/html/2506.03723v1#A2.F5) and [Figure 6](https://arxiv.org/html/2506.03723v1#A2.F6) present an analysis of model outputs on Math-500 and ARC-Challenge—two held-out CoT tasks not seen during CSFT training. As shown in both figures, output length increases in low-confidence bins. In the case of Math-500, there is also a clear trend toward more concise responses in high-confidence bins. These results demonstrate that the length modulation effect reported in the main paper is not restricted to the training distribution but generalizes to unseen tasks. In other words, CSFT enables the model to internalize the ability to adjust response length based on its uncertainty, suggesting a deeper transformation in its reasoning behavior.

## Appendix C Prompt Examples

Figure: Figure 7: Base prompt format used to elicit step-by-step reasoning and final answer generation.

Figure: Figure 8: Suffix confidence prompt used to elicit model confidence after the full reasoning and answer generation. Since confidence is predicted post hoc, this setup allows for calibration without interfering with the model’s original reasoning or final answer.

Figure: Figure 9: Prefix confidence prompt used to elicit model confidence before reasoning begins. By conditioning the generation on anticipated confidence, this prompt not only guides the model’s uncertainty expression but also influences the reasoning path and final answer.

Figure: Figure 10: Prompt used to extract final answers from model-generated responses on GSM8K.

Figure: Figure 11: Matching prompt for evaluating exact answer agreement on Math500. Designed to assess correctness by comparing model output with the ground truth in a strict yet interpretable format.

Figure: Figure 12: Parsing prompt for multiple-choice answer extraction on ARC-Challenge. The rules prioritize explicit letter selection, with fallback to semantic string matching.

Figure: Figure 13: Prompt format for eliciting a model’s self-assessed confidence prior to generating CoT response.

Figure: Figure 14: Prompt used when the model reports low confidence, encouraging it to pause and reconsider its reasoning path to generate a more confident response.

This section presents the prompt templates used in our experiments for eliciting model reasoning, answers, and confidence scores.
We include both the prefix prompt, where the confidence is generated before reasoning begins, and the suffix prompt, where confidence is predicted after the final answer.
All prompts follow a standardized format to ensure consistent supervision during CSFT and reliable evaluation during inference.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 3
questions: Can the authors please explain what is their main contribution and how it is differs from Lin et al.?
soundness: 3
strengths: - The paper is well-written and easy to follow and understand.

- Some of the experiments are novel and interesting, such as the cross-domain generalization that suggests that the learned  verbalized confidence transfers beyond the original reasoning tasks.
summary: The paper introduces a simple supervised fine-tuning method where a language model outputs both an answer and a verbalized confidence score. The model is trained using self-generated labels derived from its own sampling-based accuracy estimates, discretized into 10% confidence bins. The authors claim their method improves calibration (ECE, Brier Score) and accuracy.
weaknesses: The proposed method was already introduced by Lin, Hilton, and Evans (2022) in “Teaching Models to Express Their Uncertainty in Words”. That paper presented the exact same method of training a model to output natural-language confidence statements using the model's own empirical accuracy as labels. The only difference here is that the authors round the model’s empirical confidence to the nearest 10% instead of using continuous estimates.

Despite this near-identity, the authors do not cite or discuss Lin et al., nor do they explain how their work differs conceptually or methodologically. Given the overlap in both framing and implementation, the novelty of CSFT is extremely limited.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: n/a
soundness: 2
strengths: 1. This paper shows a major conceptual insight, when trained to verbalize confidence, models could self-check low-confidence responses, which contributes to the understanding the relationship between verbalized uncertainty and model's reasoning ability.
2. This papers demonstrates strong empirical results by achieving consistent improvements on both reasoning like MATH, GPQA and non-reasoning tasks like GSM8K, ARC, HellaSwag.
summary: This paper investigates verbalized confidence in LLMs which is the ability to state how confident they are in the answers. The authors propose confidence-supervised fine-tuning (CSFT) to train LLMs to generate both an answer and a corresponding confidence statement. 
Rather than relying on external clasisifers or RL, CSFT uses uses direct supervision on confidence tokens which enables the model to self-calibrate its uncertainty. Experiments on reasoning and non-reasoning benchmarks show that CSFT improves both calibration and accuracy.
weaknesses: 1. The idea of introducing confidence into LLM training is not new. There are several relevant studies: [1-3]. Especially, the paper misses the reference to those relevant studies.
2.  The evaluation lack deeper qualititative analysis. Espeically more real-world scenarios are not explored. Current reasoning models are quite strong on more complex tasks even for a 4B model like Qwen3-4B. More evaluation should be included on more complex tasks like AIME, CodeLiveBench.
3. Since only the confidence token span is supervised, the model may learn superficial correlations rather than genuine uncertainty calibration. For example, the model may learn how to output stylistic confidence phrases.

[1] R-Tuning: Instructing Large Language Models to Say `I Don't Know'

[2] TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning

[3] UAlign: Leveraging Uncertainty Estimations for Factuality Alignment on Large Language Models

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Great paper. Really worth reading it. I was still wondering how you get percentage confidence scores while the feedback is binary.

I wonder if you train on the traces, I guess the improvment might leak over to a correct answers also if you only provide the confidence? However, you get also benefits on tasks you do not train on? Is thsi perception correct that you might have indirekt leakage?
soundness: 3
strengths: The method provides confidence along the answers whiout altering the model too much due to KL-based normalization. 

The method improves on other tasks ie. generalizes

The mothod seems to improve answer quality 

I enyoed reading the paper and was informative too me, good method.
summary: The paper suggests a method to finetune including information on confidence and output answers and confidence about the result. They train only on reasoning tasks but the trained method generalizes to other tasks too. Two main methods are known to output directly confidence information or to use a meta network to predict this information by probing the internal states. They claim their method goes beyond those as it generalizes to unseen tasks. In addition, they show that it improves downstream tasks results too.
weaknesses: I could not figure out imediatley how they get a spectrum of confidence while the feedback is binary? Correct vs incorrect? Could you write this more clear pls?

I wonder how much we loose actualy of the models abilities even which could be assessed. Maybe, missed this point. 

Missed out on some literature on attribution to a source like retrieval in a loop, e.g. attributed question answering which has similar aims which provides evidence via retrieval.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: - **Confidence supervision**: Why is empirical accuracy used as a proxy for confidence? This approach marginalizes out the reasoning trace, which is often key to determining confidence (e.g., models may realize mid-reasoning that a solution is correct).

- **Relation to classifiers**: How does this method differ from training a standard classifier that predicts correctness given a reasoning trace? If the model simply outputs a confidence score without reasoning about it, the setup seems equivalent to classification fine-tuning.

- **Novelty**: How does this work differ from prior studies (e.g., [1]) that fine-tuned verbalized confidence using accuracy supervision? How exactly is this different from training a classifier?

- **Accuracy Improvement**: Given that no loss is applied to reasoning traces or answers, why does accuracy improve? A deeper investigation of why this is happening is important, as this is an unintended by-product. Please clear my understanding if I am wrong, but since there is no optimization over the reasoning traces and final answer, and there is a KL to the original model, then optimizing such an objective should not alter the reasoning traces or accuracy at all. Also, the claim of accuracy improving does not seem to be generally substantiated. The accuracy improvements in table 1 for the Qwen model are negligent, suggesting that this trend does not hold generally. 

- **Confidence variance**: What is the variance of predicted confidence across different reasoning traces or answers for the same question? 

- **Confidence-Guided Reasoning Path Refinement**:  Please provide more details on this. Is this method only applicable for the prefix confidence variant? If a model is prompted for confidence directly after the question, then how will resampling when it outputs low confidence increase accuracy, even if the prompt is different? The resampling prompt asks it to consider an alternative approach but based on my understanding, it has not considered any approach yet (since the confidence was directly after the question). What does alternate refer to in such a scenario? Instead, a better alternative might be to do sequential sampling with the suffix prompt. If the confidence turns out to be low, reprompt the model with its original attempt and predicted confidence, and ask it to generate a new answer with higher confidence.  

- **Figure 1:** Fig 1 is not a good way to present results. The bars in the graph represent different metrics and thus the deltas are not easily comparable. Table 1 is a better way to summarize these results. 

- **Baselines**: Please add more baselines to allow placing this work relative to existing work: 
     - *Shared Classifier*: Train a classifier on top of the reasoning model. This baseline should also have the same accuracy as the pre-trained model, but it uses representations of the model which is generating the answers.
     - *Separate Classifier*: Train a separate classifier model conditioned on generations from the reasoning model. This baseline should have same accuracy as the pre-trained model.

[1]: Lin, S., Hilton, J., & Evans, O. (2022). Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334.
soundness: 2
strengths: - **Simple, relevant and well-motivated**: The proposed method is simple yet tackles the important problem of model calibration. Calibration is increasingly critical for deploying reasoning models in real-world applications. 

- **Interesting results**: Despite its simplicity, the method appears to produce notable and somewhat surprising empirical results (e.g., accuracy improvements), which merit deeper investigation.
summary: This paper presents confidence-supervised fine-tuning (CSFT), a training paradigm where models are trained to express verbalized confidence. The ground truth confidence for a particular question is approximated as the empirical accuracy of the model on that task. The model is then optimized to output well-calibrated confidence statements. Optimizing with this simple objective improves both accuracy and calibration, and leads to emergent CoT self-checking reasoning.
weaknesses: - **Novelty**: It is unclear what the novelty of this framework is. There have been previous works from as long as 3 years ago [1] which use a very similar setup. 
- **Experimental Design and Strength**: No baselines (except a simple pre-trained model) are presented in the results, making it difficult to identify the strengths of the method relative to existing works in the field. Some results presented in this paper are also surprising, and merit deeper analysis. In particular, it is unclear to me how this method improves accuracy without optimizing for it in any way. 
- **Writing**: The writing of the paper is informal at times and can be significantly improved. For example “we basically do not apply loss to the reasoning trace” (Line 185) makes it unclear whether a loss to the reasoning trace is being applied or not. 

[1]: Lin, S., Hilton, J., & Evans, O. (2022). Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334.