Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty
Authors: Mehul Damani
Isha Puri
Stewart Slocum
Idan Shenfeld
Leshem Choshen
Yoon Kim
Jacob Andreas
Venue: ICLR 2026 Poster (2026)
Keywords: RL
calibration
reasoning
uncertainty
Primary area: foundation or frontier models, including LLMs

Abstract: When language models (LMs) are trained via reinforcement learning (RL) to generate natural language “reasoning chains”, their performance improves on a variety of difficult question answering tasks. Today, almost all successful applications of RL for reasoning use binary reward functions that evaluate the correctness of LM outputs. Because such reward functions do not penalize guessing or low-confidence outputs, they often have the unintended side-effect of degrading calibration and increasing the rate at which LMs generate incorrect responses (or “hallucinate”) in other problem domains. This paper describes RLCR (Reinforcement Learning with Calibration Rewards), an approach to training reasoning models that jointly improves accuracy and calibrated confidence estimation. During RLCR, LMs generate both predictions and numerical confidence estimates after reasoning. They are trained to optimize a reward function that augments a binary correctness score with a Brier score—a scoring rule for confidence estimates that incentivizes calibrated prediction. We first prove that this reward function (or any analogous reward function that uses a bounded, proper scoring rule) yields models whose predictions are both accurate and well-calibrated. We next show that across diverse datasets, RLCR substantially improves calibration with no loss in accuracy, on both in-domain and out-of-domain evaluations—outperforming both ordinary RL training and classifiers trained to assign post-hoc confidence scores. While ordinary RL hurts calibration, RLCR improves it. Finally, we demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration via confidence-weighted scaling methods. Our results show that explicitly optimizing for calibration can produce more generally reliable reasoning models.

Published as a conference paper at ICLR 2026
BEYOND BINARY REWARDS: TRAINING LMs TO REA- SON ABOUT THEIR UNCERTAINTY
Mehul Damani"
Isha Puri
Stewart Slocum
Idan Shenfeld
Leshem Choshen
Yoon Kim
Jacob Andreas
Massachusetts Institute of Technology
ABSTRACT
When language models (LMs) are trained via reinforcement learning (RL) to gen- erate natural language reasoning chains, their performance improves on variety of difficult question answering tasks. Today, almost all successful applications of RL for reasoning use binary reward functions that evaluate the correctness of LM outputs. Because such reward functions do not penalize guessing or low-confieence outputs, they often have the unintended side-effect of degrading calibration and increasing the rate at which I.Ms generate incorrect responses (i.e. "hallucinate") in other problem domains. This paper describes RLCR (Reinforcement Learning with Calibration Rewards), approach to training reasoning models that jointly improves accuracy and calibrated confidence estimation. During RLCR, LMs generate both predictions and numerical confidence estimates after reasoning. They are trained to optimize reward function that augments a binary correctness score with Brier score-a scoring rule for confidence estimates that incentivizes cali- brated prediction. We first prove that this reward function (or any analogous reward function that uses a bounded, proper scoring rule) yields models whose predictions are both accurate and well-calibrated. We next show that across diverse datasets, RLCR substantially improves calibration while maintaining strong accuracy on both in-domain and out-of -ut-of-domain evaluations outperforming both ordinary RL training and classifiers trained to assign post-hoc confidence scores. While ordinary RL hurts calibration, RLCR improves it. Finally, we demonstrate that verbalized confidence can be leveraged at test time to improve accuracy and calibration via confidence- confidennce-eighted scaling methods. Our results show that explicitly optimizing for calibration can produce more generally reliable reasoning models.
INTRODUCTION
Many recent advances in research on language models (LMs) have been driven by reasoning mod- els- els-LMs trained via reinforcement learning (RL) to "think out loud' in natural language before answering questions, achieving ofate-of-the-art performance on challenging tasks like math and programming (Guo et al., 2025).
The standard approach to reasoning training (often referred to as reinforcement learning with verifi- able rewards, or RLVR) performs RL with a simple binary correctness reward: Reorreeness (3) y²) Ly=y", where = checks whether the model's output y matches ground-truth answer y² While simple and effective for improving accuracy, this reward comes with critical limitation: it rewards models equally whether they are confidently correct or merely guessing, and penalizes identically whether they abstain or produce incorrect answers. This incentivizes overconfident guessing.
Consistent with this concern, studies have shown that even when initially well-calibrated, models tend become overconfident following RL training (Choshen al., 2019; Leng et al., 2025). Reasoning models, in particular, tend to exhibit worsened calibration and increased hallucination rates compared to base models, especially when trained with reward signals that emphasize only correctness (Kirichenko et al., 2025; Yao et al., 2025; OpenAI, 2025). This is a critical limitation in high-stakes domains such as healthcare or law, where models must not only be accurate but also communicate uncertainty when appropriate (Omar et al., 2024).
Equull Contribution. Correspondence mehul t.

---

Published as a conference paper at ICLR 2026
(x) Example Output
(b) In-distribution Evalustion
Ont-of-distribution Evaluation
<think- The question usks
with
70
60-
which Lalo esentedd
1969
REVR
RLVR-Clostler
Purovision Soug Contest.
recall
50-
the speeifie UK 1969. <ithink>
To With Love clanswar
Lulu perfonned for the
Accurzyy
comectuuss
RLCK
Accurayy
88-
RECR
suualysis> There high level uncestainty this because while Luke represent the UK the 1969 Eurevision performed known easily recalled. [...] without specific information. difficult pinpoint the exact cimaayysss
Reiee (Qurs) -
cocrections confidence comectuess?
RENR Base
REN ClVee-Caveilarr
the specifie
widely
sconfidenge 0.3 <lcunfidemce-
0.5 Calioration Score (L-ECE)
0.55 Calibration Scote (I-BCIE)
0.75
Figure 1: (a): Sample chain-of-thought from a model trained with RLCR, using <think>, <answer>, <analysis>, and <confidence> tags. (b) On in-domain evaluation tasks, RLCR improves standard reasoning training (RLVR) and even slightly outperforms combination of RLVR and dedicated classifier trained to predict RLVR correctness. (c) When evaluating generalization novel tasks, RLCR improves both accuracy and calibration, while other methods leave accuracy unchanged and sometimes harm calibration. All results shown are for HotpotQA, see Section 4 for more results.
This paper aims to address these limitations by answering two questions:
(1) Can reasoning models be optimized for both correctness and calibration?
(2) Can the contents of reasoning chains themselves improve calibration?
We approach these questions through the lens of statistical decision theory, specifically the theory of proper scoring rules. Given a predictor that produces an output y and a confidence q. a proper scoring rule is minimized when q reflects the true probability that y will agree with ground-truth outcome y" (Gneiting & Raftery, 2007). A canonical example is the Brier score (Brier, 1950):
RBrer (V. q: q.v") v²) -q - 1 > )² Proper scoring rules are widely used in forecasting (Waghmare & Ziegel, 2025), but have seen little application in training LLMs with RL.
Our approach, RLCR (reinforcement learning with calibration rewards), involves a modified version of reasoning training that encourages models to reason about both task correctness and uncertainty. To do so, we simply train models to output both answers y and (verbalized) confidence scores optimizing a combined reward function:
RRLCR (v, q, y=) = Reomectness (v. y²) + RBBer (V. q, y²)
=yygy" (g - 1 )
We show that this approach has several appealing theoretical and empirical properties:
RLCR provably incentivizes both accuracy and calibration: RRECR is maximized when mod- els output the answer most likely to be correct, along with a calibrated estimate of their prob- ability of success. In other words, RRECR is maximized by LM outputs (v.9) for which ply = y²), where denotes the true underlying proba-
y arg max,' p(y' = y²), and bility distribution over correctness labels. More generally, we show that an analogous objective can be constructed whenever a bounded, proper scoring rule used for the calibration term. Notably, log-likelihood, though a proper scoring rule, is unbounded and can incentivize models to output incorrect answers (Section 3).
In experiments on factual question answering and mathematical reasoning tasks, RLCR matches the task accuracy of RLVR while substantially improving calibration, on in-domain problems, reducing expected calibration error from 0.37 - 0.03 on HotpotQA (Yang et al., 2018) and 0.26 0.10 on a collection of math datasets (Section 4).
RLCR improves calibration generalization to new domains, RICR improves calibration, outperforming the RLVR model, base model and predictor equipped with a second model fine-tuned only output confidence scores.
tasks: where RLVR substantially worsens calibration
Verbalized confidence can be incorporated into test-time scaling, improving ensembling and best- of-N methods: This may be attributed to the fact that RLVR also improves the coherence of model predictions across samples: when multiple reasoning chains and predictions are generated for a given question, RLCR reduces the variance in confidence scores across reasoning chains that
2

---

Published as a conference paper at ICLR 2026
lead to the same answer, and reduces the frequency with which models assign high confidence to contradictory answers (Section 4.4).
Together, these results show that existing reasoning training methods be straightforwardly modified to additionally optimize for calibration, and that this in turn improves their accuracy, robustness, and scalability.
2 PRELIMINARIES
Let Te be a language model that maps from prompts E X to outputs E Y, perhaps preceded by natural language reasoning chain, with æ, y and reasoning chains all represented as strings. Given a dataset of prompt-output pairs D = {(04, v[)) (e.g. questions and ground-truth answers) and reward function R Y Y R that compares predicted ground-truth outputs, our goal is to improve LM outputs by optimizing:
arg max ((x,"))D y-mo(-)a)) R(y.y')
Reinforcement learning with verifiable rewards (RLVR) When training reasoning models, standard choice of R is the binary correctness reward:
Reorrectnes (v, y") = Lyzv"
(3)
where 1ym {0, 1 is the indicator function that evaluates whether y is correct, i.e. equivalent (perhaps modulo formatting details) to y"
Proper scoring rules measure of confidence in this answer.' ! A scoring rule measures the quality of a confidence estimate. In the case of modeling binary outcomes (e.g. our confidence that given answer y is correct), scoring rule is function S R x {0, 1} + I that maps a confidence estimate and an outcome scalar score. scoring rule is called proper if its expected value is minimized by confidence scores that match the true outcome probability:
Often, want predictors that output only answer y. but some scalar
Ea-p(a) S(p(a), S Euppla) S(g, a) Vg.
Here, p denotes the true underlying probability distribution over correctness labels. Perhaps the most familiar example of proper scoring rule is the log-loss:
Logarithmic score:
S(q a) = -a -alogg - (1 a) log(1 - ④)
(5)
But many other examples exist, including
Brier score:
S(q. a) = (a - (u))
(6)
S(q. a) qa + (1)
Spherical score:
(7)
Vq² 9)2
What all these scores have in common the property that they are minimized when confidences match the true probability p(a 1).
3
METHOD
The main idea behind our approach is to train language models via reinforcement learning with reward that incentivizes both correctness and calibration, by combining standard correctness reward with a reward based on the Brier score. In this approach, models are first prompted to produce reasoning chains that produce both answers and confidences estimates (as in Fig. la). They are then trained to optimize:
RRECR (y. 4.3")= 1y=s* - (g Ly=v")²
Intuitively, this reward incentivizes correctness but penalizes models when they output incorrect answers with high confidence or correct answers with low confidence.
'I is sometimes even more useful to models that can place complete distribution over a large set of possible answers But for very large answer spaces or expensive predictors like language models performing chain-of thought reasoning enumerating and scoring all possible answers is generally impractical. This paper mainly focuses models that generate one answer and one confidence score, though see Section 4.6 for one way of using this approach generate and multiple answers.

---

Published as a conference paper at ICLR 2026
It is not immediately obvious that this reward func- tion incentivizes desired LM wehavior-because involves tradeoff between accuracy (the first term) and calibration (the second), we might worry that models will learn output answers certain to be wrong in order to obtain a small calibration loss. But in fact, the calibration term in Eq. (8) comes at no cost in accuracy:
RI VR Reward Rmmrettees
1.0
0.5
Reward
0.0
(answer correct)
Theorem 1. Suppose, for any prediction that the success indicator yy= is distributed as Bernoull(py.). Then BRICR in Eq. (8) satisfies two properties:
(answer wrong)
-100
0.0
0.2
U.6 Verbalized confidence q
Ud
RLCR Reward Reatibrated (Ours)
Calibration incentive. expected reward E1 EI, RRICH(y. q: (.."") is maximized when q = Py-
any yb the
1.0
0.5
Revard
2. Correctness incentive. Among all cali- brated predictions (y-Pg), expected reward is maximized by the prediction whose suc- cess probability Py is greatest."
0.0
-0.
(answer correct)
(answer wrong)
Proof given in Appendix
0.0
0.2
04
Verbalized confidence q
An important property of Theorem 1 is that we cannot replace the Brier term (q 1 )² with any proper scoring fule-for example the log loss Ty=p- log q + (1 1l y= ) log(1 - 9) does not in- centivize correctness. However analogous version of Theorem exists for any bounded proper scoring rule satisfying S(p, S(p, 0) À for some X.
Figure 2: (a): RLVR focuses solely on cor- rectness, which incentivize guessing. (b): RLCR uses a calibrated reward that jointly optimizes for correctness and calibration.
4 EXPERIMENTS
Our main experiments aim to evaluate how RLCR empirically changes the accuracy and calibration of LMs, both in "in-domain" evaluations on the task used for RL, and "out-of-domain" evaluations on other ceestion-anssering tasks. Additional experiments evaluate interactions between RLCR and other test-time reasoning paradigms, and the extent to which RLCR causes LM predictions to become more coherent across predictions.
4.1 EXPERIMENTAL. SETUP
Training Details We Group Relative Policy Optimization (GRPO) (Shao et al., 2024) the base RL algorithm with some modifications (see Section B.2). We use the Qwen2.5-7B base model, Gandhi al., 2025). Following recent work on RL training for LM reasoning (Hu et al., 2025; Guo et al., 2025), we initialize RL from the base model and do use any KL regularization.
part of the Qwen family popularly used in RL tasks (Hu al., 2025;
Methods
We evaluate the following methods:
1. Base: The base pre-trained model. We use Qwen2.5-7B Base in our experiments. We prompt the model output both answers and confidences. detailed in Section B.4.
2. RLVR:
Initialized from the base model and trained using Reorretness with <think> and <answer> tags. During evaluation, the model is also prompted to output a verbalized confidence.
Nott that statement of the problem does not distinguish between epistemic and aleatoric uncertainty about success. Obviously, once has been predicted, the outcome of evaluation is fully determined, and the objective probability that y" is either 0 or 1. But an information- computation-constran predictor may still possess subjective uncertainty.
4

---

Published as a conference paper at ICLR 2026
3. RLVR BCE Classifier: A confidence classifier trained outputs from the RLVR model. Specifically, given problems, solution CoTs (from RLVR), and correctness labels (z, y, y=y)), we train confidence classifier fo (z, y) using the binary cross-entropy (BCE) loss:
LECE (0) (( y_ log fe(a, (r v) + (1 ,y==") log(1 fo(a, v))l
(9)
The classifier is initialized from Qwen2.5-7B Base and is thus highly expressive. This approach is expensive, as it requires training and inference with two large models.
4. RLVR + Brier Classifier: Instead of using binary cross- roos-entropy (BCE) loss to train a classifier, we use mean squared error (MSE), which allows more direct optimization of the Brier score:
Crier (0) (z.g) ( (fo(z.)) v) -y-=-
(10)
5. RLVR + Probe: Given final-layer embedding Φ(x of the RLVR model, we train a linear probe predict confidence. In Eq. (9), we replace the fine-tuned LM with a linear model: fx, y) = log (0 Φ(2, y)), where a(-) denotes the sigmoid function.
Answer Probability: We generate outputs using RLVR, extract tokens enclosed within <answer> tags, and compute their average probability: AnswerProb(y) 지 EIEA Po(Ut V)),
Here, the set A S (1,...,T) denotes the token positions that appear between the <answer> tags. Po(ye V)),,)) represents the model's probability of generating token ye. 7. RLCR (ours): Initialized from base model and trained using RRLCR-
Evaluation Metries We use the following evaluation metrics:
1. Accuracy (t): A measure of performance.
2. Area under ROC curve (AUROC) (t): Measures how well confidence scores distinguish correct from incorrect answers, treating correctness as a binary label and averaging TPR/FPR (true and false positive rates) over all thresholds. AUROC So TPR(FPR (APPR--(A) (t)) dt
3. Brier Score (L): Squared difference between confidence and ground truth.
Brier Score ス/ E.-n(a )
4. Expected Calibration Error (ECE) (4): Calibration metric that groups confidences into bins and computes difference between the average correctness and confidence. ECE =
Em .M
[Bm. |acc( Bm)) conf( (m)). where M is the number of bins, B, is the set of samples in bin m, and N is the number of samples. We use M = 10.
Evaluation datasets We evaluate RLCR on benchmarks highlighting distinct sources of uncertainty,
including ambiguous evidence, obscure facts, and complex reasoning. HotPotQA (Yang al., 2018) tests calibration under incomplete or distracting evidence, while SimpleQA (Wei et al., 2024) and TriviaQA (Joshi al., 2017) probe overconfidence on obscure factual knowledge: GPQA (Rein et al., 2024), Math500 (Hendrycks et al., 2021), GSM8K (Cobbe et al., 2021), and Big-Math (Albalak et al., 2025) assess calibration in complex, multi-step or scientific reasoning, where uncertainty accumulates across steps. CommonsenseQAA (Talmor et al., 2019) examines confidence in ambiguous, implicit reasoning scenarios.
4.2 HOTPOTQA
Dataset We modified HotPotQA distractor dataset (Yang al., 2018) with multi-hop ques- tions and 10 paragraphs (2 relevant, 8 distractors). To test uncertainty reasoning, HolPotQA-Modiffed removes 0, or both relevant paragraphs, creating varying information completeness. See Ap- pendix M for an example. The dataset is evenly split across these conditions, with 8 paragraphs per example. We train on 20,000 examples and use exact string match compute correctness.
Results On the in-distribution HotpotQA distractor dataset (Table 1), RL-trained models outperform off-the-shelf baselines in multi-hop accuracy, with RLCR being comparable to RLVR in accuracy, showing that the calibration term does not hurt performance. While both the base and RLVR remain overconfident and poorly calibrated, our method and the classifiers achieve substantially better calibration, with RI RLCR slightly ahead. The Answer Probability baseline performs poorly, as the
5

---

Published as a conference paper at ICLR 2026
(a) Models Trained on HotpotQA
Method
HotpotQA
O.O.D, Averaged
Ace.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
(T)
(1)
(1)
(L)
(1)
(1)
(L)
(1)
Base
39.7%
0.54
0.53
0.53
53.3%
0.54
0.41
0.40
RLVR
0.50
0.37
0.37
0.50
0.46
0.46
RLVR BCE Classifier
0.66
0.22
0.07
0.58
0.27
0.24
RLVR Brier Classifier
63.0%
0.65
0.22
0.09
53.9%
0.60
0.32
0.33
RLVR Probe
0,55
0.24
0.10
0,53
0,38
0.38
Answer Probability
0.72
0.36
0.36
0.60
0.42
0.42
RLCR (ours)
62.1%
0.69
0.21
0.03
56.2%
0.68
0.21
0.21
(b) Models Trained on Big-Math
Method
Math
O.O.D. Averaged
Acc.
AUROC
Brier (L)
ECE
Acc.
AUROC
Brier
ECE
(f)
(f)
(4)
(t)
(4)
(4)
Base
56.1%
0.56
0.40
0.39
47.8%
0.53
0.46
0.45
RLVR
0.47
0.28
0.26
0.52
0.49
0.49
RLVR BCE Classifier
0.78
0.10
0.34
0.33
RLVR Brier Classifier
72.9%
0.78
0.15
0.10
52.5%
0.57
0.28
0.27
RLVR Probe
0.65
0.19
0.13
0.53
0.33
0.30
Answer Probability
0.52
0.26
0.26
0.52
0.44
0.43
RLCR (ours)
72.7%
0.67
0.17
0.10
50.9%
0.60
0.28
0.25
SFT+RLCR (ours)
72.2%
0.78
0.14
0.08
43.8%
0.66
0.24
0.18
Table Accuracy and calibration metries for models trained HotpotQA and Big-Math. Best
values bolded. Dataset-specific results in Appendix K.
(a) Performance on HotpotQA and 6 out-of-distribution (O.O.D.) datasets. RLCR achieves competitive accuracy and significantly outperforms all baselines in calibration, especially on O.O.D. datasets, demonstrating benefits of jointly optimizing accuracy and calibration.
(b) Performance on Math and 5 out-of tnatof--iitributionn (O.O.D.) datasets. Math results are averaged over datasets: Math-500, GSMSK and Big-Math. SFT+RLCR variant achieves the best calibration across both in-distribution and O.O.D. settings. However, this comes at the cost of reduced general- ization accuracy, possibly due to catastrophic forgetting. RLCR offers a stronger trade-off O.O.D. settings, maintaining competitive accuracy while improving calibration.
model typically commits to an answer during CoT reasoning, inflating output confidence. Fig. 11 shows that both correctness and calibration reward for RLCR increase smoothly during training.
Next, we evaluate experiments across out-o--distributioo datasets (TriviaQA, SimpleQA, MATH500, GSM8K, CommonsenseQA. GPQA). We find that RL training HotpotQA does not improve accuracy OOD, with the base model performing comparably to RL-trained models. However, RLVR worsens calibration relative to the base model, whereas RLCR achieves substantial gains over all baselines across calibration metrics while maintaining (or slightly improving) on task accuracy. We hypothesize that better calibration generalization of RLCR could be due to:
1. Chain-of-thougee reasoning about uncertainty Reasoning about uncertainty can improve calibration by allowing reflection confidence, in line with recent work (Yoon et al., 2025).
Training dynamics RL During RL training, the model's confidence analysis and scores have to constantly adapt to the model's improving task performance. This non non-stationarity might lead to more robust learning and better generalization.
3. Shared representations Using single model for both solution generation and calibration allows calibration task to leverage internal representations used by the solution generating process.

---

Published as a conference paper at ICLR 2026
0.214
Centie food Mig Wale
Vigery lile
0.315
0.55
Vas Liiihood
Accuracy
Brier Score
6,12
0.58
6:211
0.53
0.310
0.206
0.55
2' 2'
2
N (Number of Samples)
K (Ensemble Size)
(a)
(b)
Figure 3: Test-time scaling curves. (a) Accuracy vs Number of Samples (N). Accuracy improves for all methods with increasing compute. Sonfidence-weighhted majority vote outperforms both vanilla majority vote and max-confidence, highlighting complementary benefits of combining voting with confidence scores. (b) Brier Scores vs Ensemble Size (K). Here we evalute the effect of applying test-time scaling to confidence estimation alone, resampling multiple analyses (blue text in Fig. 1). Calibration improves (although modestly) as the size of the analysis ensemble grows.
4.3
MATH
Training Details We use a subset of the Big-Math (Albalak al., 2025) dataset, containing 15,000 problems selected using criteria defined in Section B.1. We compute correctness using math-verify." To enhance the quality of uncertainty reasoning, we also train variant with a lightweight SFT warmup phase. We generate solutions from the base model on 500 examples and use Deepseek-RI 1 to produce uncertainty analyses for them. Further details in Section B.2.
Results On Math benchmarks (averaged over GSM8K, Math, and Big-Math), all RL methods improve accuracy significantly over the base model. SFT+RLCR achieves the best calibration, slightly surpassing the classifiers, while base and RLVR remain poorly calibrated. (TriviaQA. SimpleQA, CommonsenseQA. GPQA, HotPotQA), the accuracies of RLCR and RLVR are marginally better than the base model, but surprisingly the accuracy of the SFT+RLCRR model drops significantly, possibly due to catastrophic forgetting induced by SFT warmup. Despite this, SFT+RLCR achieves the strongest calibration. Overall, RLCR offers a stronger trade-off in O.O.D. settings, maintaining accuracy while matching or outperforming all baselines on calibration.
distri-iittiitttinn
4.4 CAN VERBALIZED CONFIDENCES BE USED FOR TEST-TIME SCALING?
We next evaluate whether confidence scores from RLCR can be incorporated into test-time scaling algorithms to yield improvements in both accuracy and calibration.
Accuracy Test-time scaling methods such as best-of-N or majority vote are widely used to boost model performance by aggregating multiple responses. Given N responses yı, JUN and reward model r(x. y). best-of-N selects the response with highest reward: Jehosen arg max (xpp( R( yi) Similarly, majority vote selects the most frequent answer among N responses. Prior work has also explored weighted majority vote, where the voting weighted using reward model scores.
Our key insight is that the verbalized confidence q output by the model can serve as an effective proxy reward (Taubenfeld et al., 2025). If a model is well-calibrated, then selecting responses with higher confidence will naturally lead to an increase in performance. This insight leads two simple algorithms: select the response with the highest verbalized confidence from the set of N candidates (max-confidence), or weight each vote by its confidence score confidence-weighted majority vote). These algorithms are the confidence-scored analogues to Best-of-N and weighted majority vote. Importantly, both approaches do not require any additional supervision or external reward models.
chttps://gittee http://ggiuuuuccco/eeeee http////gggggigaaaaaa//attteeeee
7

---

Published as a conference paper at ICLR 2026
RIAR
Sun Coefirrre Hamppe
Expecie
Propertion of Sammpts
0.12
0.10
0.08
0.00
0.00
0
0.65
0,10
0.15
0.20
025
EorporDA
Mrth
Test
Stanard Deviation Coniidence
(a)
Figure 4: (a): Distribution of standard deviation in confidence across multiple uncertainty reasoning chains for the same solution/answer. Most samples exhibit low deviation, indicating that the model's confidence estimates are self-consistent. (b) Swarm plot of confidence sums across 3 datasets. RLCR consistently remains closer to the ideal sum of Nonetheless, overconfidence remains, suggesting room for further improvement.
To contextualize these methods against likelihood based selection, which is always computable for any open model, we additionally compare against two likelihood- based baselines max generation likelihood, which performs Best-of-N using average sentence likelihood R p(y|=)/ll) as the reward score, and generation likelihhoddweeighted majority vote, which weights votes by these likelihood scores. We evaluate these approaches using the RLCR model (trained Hotpot) for gener- ation, plotting average accuracy across the 7 datasets used in Table 1. As shown in Fig. 3a, accuracy consistently improves with more samples, with confidence weighted majority vote outperforming vanilla majority voting, max-con maxxconfiieece,, and both likelihood baselines. These gains highlight how calibration can directly underpin test time scaling better calibrated confidence estimates lead to more accurate aggregated predictions.
Calibration In our structured RLCR COT, models first output solution, followed by an analysis and confidence score. To improve confidence scores for fixed answer y, we sample K analysis CoTs ; ~ p(- x, y). each producing a verbalized confidence qu. We then ensemble these confidences to obtain the aggregated confidence estimate
E; 9t- Figure 3b plots Brier score (averaged over 7 datasets) as a function ensemble size K. We observe that calibration improves as the ensemble size grows, though the absolute gains relatively modest. This reflects the fact that for most questions, there is low "uncertainty about uncertainty" so averaging does not substantially alter the estimate. Nonetheless, ensembling provides lightweight mechanism for reducing residual noise, especially on harder questions where confidence variability is greater.
4.5 ARE VERBALIZED CONFIDENCES SELF-CONSISTENT?
Intra-solution coherence desirable property of uncertainty-avvar reasoning is that a model should assign consistent confidence estimates when generating multiple uncertainty reasoning chains for the same answer. That is, the model should have low "uncertainty about uncertainty". Given fixed answer y. we sample K analysis CoTs 21,..., 2K p(- x, y). where each chain produces verbalized confidence score qı Ideally, these confidence scores should exhibit low variability.
Figure 4a plots the standard deviation across seven datasets, using analysis CoTss generated by the RLCR model trained on HotpotQA. Most samples have low standard deviation, suggesting that the model's confidence estimates are generally consistent.
Inter-solution consistency For tasks where answers are mutually exclusive-i.e. only one answer is correct per instance- is desirable that the model distributes its confidence across distinct answers such that the total confidence is less than or equal to 1, with equality holding when the set of answers exhaustive. Let model generate N responses N = with associated confidence scores {q: LV where E [0. 1]. Let A = {aI,... ax) denote the set of K < N unique answers among the yi- The mean confidence assigned to answer 0g
Σ, 1
(11)
Σ
1 y E
8

---

Published as a conference paper at ICLR 2026
Models Trained on HotpotQA
Method
HotpotQA
O.O.D. Averaged
Acc.
Tokens
Brier
ECE
Acc.
Tokens
Brier
ECE
(1)
(U)
(t)
(L)
(1)
(L)
(U)
(4)
RLCR (ours)
62.1%
0.21
0.03
56.2%
300
0.21
0.21
RLCR w/o Analysis
61.7%
113
0.23
0.09
56.5%
179
0.26
0.26
RLVR w/ Analysis
62.3%
224
0.35
0.34
52.6%
221
0.41
0.39
RLVR
63.0%
92
0.37
0.37
53.9%
142
0.46
0.46
Table 2: Calibration ablation evaluating the contributions of (i) calibration-aware reward learning and (ii) explicit uncertainty reasoning. Adding uncertainty analysis improves calibration for both RLCR and RLVR, while removing analysis from RLCR still preserves significant calibration benefits. Together, these results indicate that both components independently contribute to improved calibration, and the strongest performance is achieved when both are used together.
For model to give consistent answers, desire: Σ 9k S 1.
Fig. 4b shows a swarm plot of predicted confidence sums across three representative datasets. On the in-distribution HotpotQA dataset, RLCR's confidence well-calibrated belief distribution. For out-of-distribution datasets, both RLCR and RLVR exhibit overconfidence, with sums exceeding I, though RLCR remains significantly closer to the ideal. This reflects RLCR 's improved calibration, yet highlights room for improvement, particularly OOD.
cluster tightly around 1, indicating
4.6
WHAT DRIVES CALIBRATION GAINS?
RLCR differs from RLVR along two key axes: (1) RLCR's reward function directly incentivizes calibrated confidence estimates, and (2) RLCR models are prompted to explicitly reason about uncertainty. To isolate the contribution of these components, ablate the uncertainty-reasoning portion of the CoT prompt and evaluate the following four variants on the HotporQA-trainel models:
1. RLCR: Vanilla RLCR, as presented in Table 1.
RLCR w/o Analysis: The RLCR model evaluated without uncertainty reasoning. At inference time, the model is instructed to output only <think>, <answer>, and <conf confidenne>..Tee The model instructed to not perform any uncertainty reasoning.
3. RIVR: Vanilla RLVR, as presented in Table 1.
RLVR w/ Analysis: The RLVR model evaluated with uncertainty reasoning, using identical analysis prompt used for training/evaluating RLCR (see long RLCR prompt in Appendix C).
The results in Table 2 reveal several key findings.
(1) Explicit uncertainty reasoning improves calibration for both RLCR and RLVR. Across both training and OOD settings, the variants with uncertainty reasoning achieve lower Brier and ECE than their no-analysis counterparts. This consistent with prior work (Yoon al., 2025) and is further corroborated by the classifier ablations in Appendix G.
(2) heward-basedd calibration is substantially more effective than prompting alone. Although RLVR w/ Analysis improves over vanilla RLVR, remains far behind both RLCR variants, including RLCR w/o Analysis. Thus, prompting model to reason about uncertainty provides only modest gains compared to explicitly training with a calibrated reward.
(3) RLCR w/o Analysis nearly matches RLVR in accuracy and token cost, while dramatically improving calibration. On HotpotQA, RLCR w/o Analysis uses similar number of tokens as RLVR (113 vs. 92) and achieves comparable accuracy (61.7% vs. 63.0%), yet its calibration is far superior (ECE: 0.09 vs. 0.37).

---

Published as a conference paper at ICLR 2026
Overall. Both components expplnntteeepliiitttte analysis and calibration-aware reward learning- contribute to improved calibration. When efficiency is paramount, RLCR w/o Analysis offers a drop-in alternative that preserves accuracy and token efficiency while significantly improving calibration over RLVR.
RELATED WORK
Building reliable reasoning models requires not only high accuracy but also calibrated uncertainty a property that standard RI. objectives often erode (Achiam et al., 2023), leaving models over- confident (Xiong al., 2024) and prone to hallucination (Jaech al., 2024). We survey four strands of prior work on confidence estimation in LLMs:
Post-hoc verbalizations prompt models state their confidence after answering (Xiong et al., 2024; Yang et al., 2024; Tanneru et al., 2024; Lin et al., 2022). Lin et al. (2022) fine-tune GPT-3 to predict confidence given question and answer, using empirical accuracy the target label. Xiong al. (2024) find that LMs exhibit overconfidence when verbalizing their confidence. Tian et al. (2023) finds that RLHF models' verbalized confidence scores are better calibrated than their conditional probabilities. Mei et al. (2025) find that even reasoning models also suffer from overconfidence and can become even more overconfident with deeper reasoning. Similarly, Kirichenko et al. (2025) introduce AbstentionBench and show that LMs struggle to abstain appropriately, with reasoning fine-tuning often degrading abstention performance. Positively, Yoon et al. (2025) and Mei al. (2025) find that reasoning models can improve calibration by introspection and slow thinking.
Sampling-based based methods use response agreement (c.g., majority vote or best-of-N) as a proxy for confidence, but costly and require clear ground truth (Kang et al., 2025). Aichberger et al. (2025) estimate uncertainty by generating diverse, plausible responses and measuring their consistency. Kuhn et al. (2023) propose semantic entropy, sampling-basee based method that leverages linguistic invariances to better estimate uncertainty in natural language generation.
Internal probing extracts confidence from model features like token probabilities (Gupta al., 2024), offering fine-grained scores but lacking generality. Kadavath et al. (2022) prompt language models to output "true" or "false" and use their probability as a proxy for the model's confidence. Mielke et al. (2022) train a LM to to generate responses conditioned on confidence estimates provided by an external probe. Fadceva et al. (2024) propose a token-level uncertainty method that leverages internal model signals to fact-check claims and detect hallucinations. Azaria & Mitchell (2023) train a classifier on hidden layer activations detect the truthfulness of statements based on the LLM's internal state. Orgad et al. (2025) show that internal representations encode rich, token-level truthfulness signals, which can detect and categorize errors beyond what reflected the output.
RL-based methods train models to output calibrated verbal confidences with RL. Stengel-Eskin et al. (2024) introduce a speaker-listene listener framework, where the speaker's rewards are determined by the listener's inferred confidence in the speaker's responses. Leng et al. (2025) introduce verbalized confidence scores in reward model training. Most closely related to our work, Xu et al. (2024) and Stangel et al. (2025) train LMs with RL using proper scoring rules reward functions. Xu et al. (2024) adopt the Brier score, while Stangel et al. (2025) use a clipped version of the log loss. While effective, these approaches optimize solely for calibration, which can inadvertently harm task in larger models that may reward hack by outputting deliberately incorrect answers with zero confidence to achieve perfect calibration rewards. Furthermore, both methods are developed and evaluated exclusively on non-reasoming tasks. In contrast, our method incentivizes both correctness and calibration, and evaluated on both reasoning and non-reasoning benchmarks.
ecuracy-particulal
6 CONCLUSION
We show that incorporating proper scoring rules into RL, via an objective we call RLCR, enables reasoning models improve both accuracy and calibration. Our approach trains models to reason about and verbalize uncertainty, preserving task performance while significantly improving calibration in- and out-of-distribution. We demonstrate that reasoning about uncertainty improves calibration, and that our method improves the self-consistency of confidence, and improves with test-time scaling. However, there remains significant room for improvement even after RLCR, oft-of-domain calibration error is often high in absolute sense, and models may still assign high confidence to multiple contradictory answers. Nevertheless, these results suggest path toward reasoning systems that are not only accurate, but reliably reason about and communicate uncertainty.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt4, arXiv preprint arXiv:2303.08874, 2023.
Lukas Aichberger, Kajetan Schweighofer, Mykyta lelanskyi, and Sepp Hochreiter. Improving uncertainty estimation through semantically diverse language generation. In The Thirleenth International Conference on Learning Representations, 2025.
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale, high-quality math dataset for reinforcement learning in language models. arXiv preprint arXiv:2502. 17387, 2025.
Amos Azaria and Tom Mitchell. The internal state of Ilm knows when it's lying. Empirical Methods in Natural Language Processing Findings, 2023. URL https: / http///xxxvvv.o// org. abs//304. 133473
Glenn W Brier. Verification forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don't 't always say what they think, 2025.
URL https: /ttp///axvvvv///////aa/eee//aaar org/ abs/ 05410.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses reinforcement learning for neural machine translation. In International Conference on Learning Representations 2020, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, 11//2100.14168 2021. URL
14168.
Ekaterina Fadeeva, Alcksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the output of large language models via token-level uncertainty quantification. Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 1367-9385, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653vv3/ 1/2024 /2t.222..innnnn-all.555555 URL httpt:tts///ate hacatho//allathhllogyyy orw.:/gg/2222 2224. findings-ache acl. ac1.558/
Kanishk Gandhi, Ayush K Chakravarthy,, Anikait Singh, Nathan Lile, and Noah Goodman. Cognitive behaviors that enable self- ieff-iprooiing reasoners, or, four habits of highly effective STars. In Second Conference on Language Modeling, 2025. URL http::::///pppeeveeeeeeeeefo//oe QGJ9ttXLTy.
Tilmann Gneiting and Adrian Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-33) 2007.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Pciyi Wang, Xiao Bi, et al. Deepseck-r1: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.12949 2025.
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum. Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. The Twelfth International Conference Learning Representations, 2024. URL //pter///pperrviie.
nett/forum?idcKgaaszZZ11
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL

---

Published as a conference paper at ICLR 2026
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly su- pervised challenge dataset for reading comprehension. Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10. 18..18/3/vvv 1/P17- 1147, URL
hwwphooogg.ggggg/1/11/ org/P17-11477
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer EI Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. Language models (mostly) know what they know. CoRR, abs/2207 002222007.052,,,,, 2022. URL https: //w.///oii.oo org/10 /0// 48550/ ariiv. hteddi..og//////
2207.05221.
Zhewei Kang, Xuandong Zhao, and Dawn Song. Scalable best-of-n selection for large language models via self-certainty. In 2nd Al for Math Workshop @ ICML 2025, 2025. URL https:
Polina Kirichenko, Mark Ibrahim, Kamalika Chaudhuri, and Samuel J. Bell. Abstentionbench: Reasoning LLMs fail on unanswerable questions. In ICML 2025 Workshop on Reliable and Responsible Foundation Models, 2025. URL htppttttt////ereee htwppppeereeeeeeeeeeeeeeeeeeeeee net/forummidieeeeeeeee net//oorum?ddkkbboossA3))
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023. URL https: h/tter///operrvieew nett/forummid-VD--VVtPOO
Jixuan Leng. Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in LLMs: Reward calibration in RLHF, In The Thirleenth International Conference on Learning Representations, 2025. URL https:///peerewee
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022. ISSN 1835- URL https: //tpenreview.net.ooo hww.eeeweeetoorrrriddsssssu
Zhiting Mei, Christina Zhang, Tenny Yin, Justin Lidard, Ola Shorinwa, and Anirudha Majumdar. Reasoning about uncertainty: Do reasoning models know when they don't know? arXiv preprint vrXiv:2506.18188 2025.
Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Trunsactions of the Association for Computational Linguistics, 10:857-872. 2022. doi: 10. 20.1162/tal URL httphtts///aalaatooloooo org/ 2022. tac1-1.50/.
Mohamad Amin Mohamadi, Tianhao Wang, and Zhiyuan Li. Honesty over accuracy: Trustworthy language models through reinforced hesitation, 2025. URL https::/arxxi.orrg/bbb////// 11500.
Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement learning finetunes small subnetworks in large language models. arXiv preprint arXiv:2505.11711 2025.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo furious. arXiv preprint arXiv:2501.000 000666 2024.
12

---

Published as a conference paper at ICLR 2026
M Omar, BS Glicksberg, GN Nadkarni, and E Klang. Overconfident ai? benchmarking Ilm self- assessment in clinical scenarios. medRxiv, 2024.
OpenAl. Openai o3 and 04-mini system card. 2025.
Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. The Thirteenth International Conference on Learning Representations, 2025. net/forunf worfoormmmii wowforrmmnnd-nnnnnmennmmm d=KRnssSEm33..
URL
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proot q&a benchmark. In First Conference on Language Modeling, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.04255 2025.
Linxin Song, Taiwei Shi, and Jieyu Zhao. The hallucination tax of reinforcement finetuning. arXiv preprint arXiv:2505. 2025.
Paul Stangel, David Bani-Haroun Chantal Pellegrini, Ege Özsoy, Kamilia Zaripova, Matthias Keicher, and Nassir Navab. Rewarding doubt: A reinforcement learning approach to confidence calibration of large language models. CoRR, abs/2503.02623,, March 2025. URL https: org/10. 48550/ aariv/arivvi.... 2503. 02623.
Elias Stengel-Eskin, Peter Hase, and Mohit Bansal, LACIE: Listener-aware finetuning for calibra- tion in large language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: /tttt::s///ppeeeiiieeee htte//orrmmmiinnyyyysoeeoo
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense@A:: A question answering challenge targeting commonsense knowledge. Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149 4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/N11914422 URL https:t/aclaanaii
www.g..rgg.... org/N19-1422)
Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju. Quantifying uncertainty in natural language explanations of large language models. In International Conference on Artificial Intelligence and Statistics, pp. 1072-1080. PMLR, 2024.
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein, Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in lims. arXiv preprint arXiv:2502.06233, 2025.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https:///ppeereviee.. net/
Benjamin Turtel, Danny Franklin, Kris Skotheim, Luke Hewitt, and Philipp Schoenegger. Outcome- based reinforcement learning to predict the future. arXiv preprint arXiv:2505. /7989, 2025.
Kartik Waghmare and Johanna Ziegel. Proper scoring rules for estimation and forecast evaluation. arXiv preprint arXiv:2504.07788 2025.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality large language models. arXiv preprint arXiv:2411.0m 1.0368. 2024.
13

---

Published as a conference paper at ICLR 2026
Zhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang, Rulin Shao, Sean Chen, Mohammad Kachuee, Teja Gollapudi, Tony Liao, Nicolas Scheffer, et al. Truthrl: Incentivizing truthful Ilms via reinforcement learning. arXiv preprint arXtv:2509.25760, 2025.
Changyi Xiao, Mengdi Zhang, and Yixin Cao. Bnpo: Beta normalization policy optimization. arXiv preprint wrXiv:2506.02864 2025.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference Learning Representations, 2024. URL https://openreview net/forum?id=gjeoKK xFp
Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao.
Sayself: Teaching ilms to express confidence with self-reflectiv rationales. In Proceedings the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5985-5998, 2024.
Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. On verbalized confidence scores for Ilms, 2024. URL
httpaaawiii..orab//////aa// 14737.
Zhilin Yang, Peng Qi, Saizheng Zhang. Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse. explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2360,,380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/1/DD8--1D0 1259.. URL /iKLtttts////aaaanoologg w
Zijun Yao, Yantao Liu, Yanxu Chen, Jianhui Chen, Junfeng Fang, Lei Hou, Juanzi Li, and Tat-Seng Chua. Are reasoning models more prone to hallucination? arXiv preprint arXiv:2505.2366 2025.
Dongkeun Yoon, Seungone Kim, Sohee Yang. Sunkyoung Kim, Soyeon Kim, Yongil Kim, Funbi Choi, Yireun Kim, and Minjoon Seo. Reasoning models better express their confidence. arXiv preprint ar 4484992505.14448, 2025.
14

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREM
We consider a slightly more general family of reward functions than in the main paper body. As above, assume that predictors produce response y, confidence q [0. 1]; now assume that scores depend additionally on an arbitrary binary correctness signal c, distributed according to some Py : p(c y).
We consider all reward functions of the form:
Ríc, Q) = Xc S(q,
where S(g, c) is scoring rule and À > 0 is scalar reward for producing the correct answer.
We define the expected reward of choosing a confidence q for a given response y as:
(g E. R(c, Apy S,, 1) (1 Py) 0)] Lemma (Calibration incentive). For any response y, the expected reward V (yy, q) maximized at q = Py if and only if S(g, e) a proper scoring rule.
Proof. The correctness term Xpy in the reward function does not depend on q, so
arg max Vyy q) = arg max -Py + (1-))) 0) = arg min E, S(g,c).
But by a scoring rule is proper by definition if Bew-emoolli(py)) (q, e) is minimized by q = Py, so the statement of the lemma follows immediately.
Lemma 2 (Correctness incentive). Consider y' with associated success probabilities Py 2 Py- Then (y, Py) 2 Vy if and only if
S(p.1) - ((p.1)-((p,0))) S for all p [0,
all
Proof. First, define:
W()) Ap pS(p, 1) p))(p.
Note that (yy. = W (py), and W (p)) represents the maximum reward attainable for any y with associated success probability yp- Thus, to verify the statement of the lemma, suffices to show that (p) W(p') if and only if 2 p', which in tum is equivalent to showing that W (p) is nondecreasing in p.
We first compute the derivative of W:
('pp = - S(p, 1) + S(p, -1-
From the Savage-Dawid Dawid representation (Gneiting & Raftery, 2007) proper scoring rules, there exists some non-negative weight function w(t) such that:
S',p, 1) -1 w()), S' =p p- w(p)
Substituting this in:
W (')p) S(p 1) S(p, 0) + - w(p) - (1 - p1) w(p)
S(,, S(p,
Thus the derivative of W is non-ntgative (W is nondecreasing) if and only if S(p, 1) - S(p, S for p [0, 1).
Then the main theorem statement in Section 3 follows immediately from these two lemmas:
Proof of Theorem 1. First observe that ARLCR satisfies the conditions of Lemma 2 with À = 1 and S(g. 1 Lyzy (g Lymy' )²2 we have max, S(p. 1) S(p, 0) 1. Then condition (calibration) follows from Lemma 1 and the fact that the Brier score is a proper scoring rule, and condition (correctness) follows from Lemma and the boundedness of S(p. 1) S(p. 0).
Corollary 1. Let S(g, c) be strictly proper scoring rule.
15

---

Published as a conference paper at ICLR 2026
1. S(, 1)-51p-ppppppp is bounded, then there exists finite X > 0 such that the reward function R(c,g) S(q. Sq, c) satisfies the correctess condition:
S(p. 1) S(p. 0) S X for all p E [0,
and thus jointly incentivizes calibration and correctness.
2. If S(p, 1) S(p, 0) unbounded, then for any finite X O, there may exist some y y such that W (py) < (Py'), ), and BRECR prefers (y', Pyy to (v.Py).
Examples. The Brier score is bounded: S(p, 1) p) S(p p², so:
S(p, 1) S(p, 0) 2p < 1 for all p [0, 1]
Thus, the condition holds for 1. In contrast, the logarithmic score is unbounded:
1 11-P)
(p, 1) log S(p,0) 0) log(1-)), 1) 0) log
as p
p
So no finite À can satisfy the condition.
16

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL SETUP
B.1 TRAINING DATASETS
HotpotQA--Modinee We use modified version of the HotPotQA distractor dataset, which contains factual questions requiring multi-hop reasoning. (Yang et al., 2018). Each example in this setting presents ten paragraphs, only two of which contain the information necessary to answer the question; the remaining cight paragraphs include closely related but irrelevant details. Consequently, solving this task requires the model to identify and reason over the pertinent passages. To more strongly develop uncertainty reasoning capability, we construct new dataset, HotPorQA-Modifter in which we systematically remove either 0, I, or both of the key paragraphs required to answer each question. This modification introduces varying levels of informational completeness that the model must reason over. We distribute questions across three equal groups: one-third have no relevant paragraphs (0/8), one-third have I relevant paragraph (1/7), and one-third have both relevant paragraphs (2/6). Each question consistently contains 8 total paragraphs. Our training dataset consists of 20,000 examples. We measure correctness using exact string match.
Big-Math Digits: We use Big-Math (Albalak et al., 2025), a large, curated training dataset for RL containing over 250,000 math problems, including questions from benchmarks such as Math and GSM8K. To ensure an appropriate range of difficulty, we retain problems for which the LLaMA-8B solve rate (provided in the dataset) is between 0-70%. We also found that verifier noise can be significant in Math datasets and can cause training instability. To reduce verifier noise, we further restrict the dataset to problems with numerical answers, enabling near-perfect automatic verification. Our final training set consists of 15,000 problems. We compute correctness using math-verify.
B.2 ADDITIONAL TRAINING DETAILS
Following Turtel et al. (2025), we remove the standard deviation division the advantage, which might help with learning on examples where there are extreme miscalibrations. We use the BNPO loss function, which aggregates token level losses using the number of active tokens in the local training batch (Xiao et al., 2025). We generate 32 responses per prompt with a temperature of 0.7. and use an effective batch size of 2048. Experiments were conducted on both NVIDIA A100 and H100 GPUs (and we observed consistent results across hardware types). We use a constant learning rate with warmup of le-06 for HotpotQA and 5e-06 for Math. We use warmup ratio 0.1. For Hotpor, we set a maximum completion length of 1536 while for Math, we use a complction length of 4096. Hotpot requires significantly less reasoning. and using smaller completion length helped improve training time. We do epoch of training. For training RLCR, we use the Long RLCR system prompt for Hotpot and the Simple RLCR prompt for Math (the long version did not provide additional benefit on Math). We use the Simple Generation prompt for RLVR. All prompts in Appendix C.
Format Reward: We use format reward to encourage adherence the structured format shown in Fig. 1. In RIVR, models must format their output in <think and <answer> tags. In RICR, in addition to <think> and <answer> tags, we require <analysis> tag enclose uncertainty reasoning and soonfidene>>io tag for verbalized confidence. A valid response must contain all these tags in the correct order. Both format and calibration rewards are weighted equally.
SFT Warmup in Math: While RL directly on the base model improves calibrated reward, the uncertainty analyses produced in Math remain qualitatively generie, often lacking reasoning tied to specific solution steps (See Appendix M for an example). To improve their quality, we train a variant with lightweight SFT warmup phase before RL to obtain higher quality uncertainty analyses. We generate solutions from the base model on 500 examples and prompt Deepseek-R 1 with the Expert SFT Prompt to produce uncertainty analyses for them. We then perform SFT with the <think> and <answer> obtained from the base model, appended with the <analysis> obtained from Deepseek-RI Note that we do not ask Deepseek-BR RI to output confidence scores.
B.3 EVALUATION DATASETS
We evaluation on large number of datasets:
17

---

Published as a conference paper at ICLR 2026
1. HotPotQA (Distractor): We use 1000 validation examples from the original HotpotQA distractor dataset. We slightly modify the dataset and remove 2 non-re non-relevant paragraphs from each question. Thus, each question has paragraphs with both supporting paragraphs present. We measure correctness using exact-match (Yang et al., 2).
2. HolPotQA-Moodifeed We evaluate on 500 held-out validation examples. We measure correctness using exact-match.
3. TriviaQA: We use 2000 examples from the validation set of the TriviaQA dataset (Joshi et al., 2017). We use the no-context split to purely test factual accuracy, We evaluate using LLM--e---jjdgge
4. SimpleQA: We use the full SimpleQA dataset consisting of 4326 factual questions (Wei et al., 2024). We evaluate using LLM--s-a-judge.
5. Math-500 We use the popular MATH-500 dataset, which contains a subset of problems from the original MATH dataset (Hendrycks et al., 2021). We evaluate using math-verify, mathematical expression evaluation system relcased by huggingface.
6. GSM8K: We use the test set (1319 problems) of the popular Grade School Math 8K dataset (Cobbe al., 2021). We evaluate using math-verify.
7. Big-Math-Digiis: We evaluate on 1000 held-out validation examples. We evaluate using meth-verify.
8. Qo We use the validation set (1220 problems) of the CommonsenseeA dataset (Talmor et al., 2019), a multiple-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers. We evaluate using LLM--as---uddge
9. GPQA: We use the GPQA main dataset containing 448 multiple-choice questions written by experts biology, physics, and chemistry (Rein al., 2024). We evaluate using LLM-- as-a-judge.
B.4 EVALUATION DETAILS
All models are evaluated with temperature O. For all datasets except Math and GSM8K, we maximum token budget 4096. The system prompt for evaluation and the pipeline to extract answer and confidence scores varies slightly based on the method we are evaluating:
1. RLCR (ours): RLCR models <think>, <answer>, <analysis> and <confidence> tags. They are evaluated with the same system prompts they are trained on. We extract their answer from <answer> tag and their confidence from sconfidencer tag.
2. RLVR: RLVR models use the <think> and <answer>. It evaluated with the same system prompt and we extract their answer from the <answer> tag. To obtain their verbalized confidence, we append "Thinking time ended. My verbalized confidence my answer as number between 0 and 100 is equal to" to their generated output.
3. Classifier/Probe: Both methods are conditioned the question and the RLVR model's generation (solution and answer). These methods thus use RLVR model as a generator and their reported accuracies in the result tables are equal.
4. Base: The base model is not good at instruction following and is prompted with a simpler system prompt (Simple Confidence Prompf) that guides it to use <think>, <answer> and <confidence, tags. In case no valid confidence can extracted, we append "Thinking time ended. My verbalized confidence in my answer number between O and 100 is equal to" to their output and call them again to extract confidence.
For all methods, we are unable to extract valid answer from the <answer> tags, we append "Thinking time ended. My final answer is" to their output and call them again. The main goal of these custom pipelines is to be able to fairly extract an answer and a confidence level and minimize cases where incorrect formatting adversely affects performance. Note that because they are trained with format rewards, both the RL-trained models are nearly perfect in adhering to the desired format and require minimal interventions. However, the base model benefits from this full extraction pipeline.
18

---

Published as a conference paper at ICLR 2026
Importantly, once answers and confidences extraced, all methods evaluated identically and based on the dataset, exact-match, LLM-as-a-judge or math-verify is used.
LLM-as-a-judge: We Llama-3.1-88 --8-----snstt with temperature set to 0 provided with the question, the ground truth answer and the answer extracted from the evaluation pipeline. It is prompted respond with "YES" or "NO" based on the correctness of the answer. As the datasets we evaluate have short and objective answers, we do not condition the judge on the thinking traces which can add biases.
judge. The judge
C SYSTEM PROMPTS
Long RLCR Prompt
"A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind, provides the user with the final answer, then analyzes its confidence about the solution and then provides the user with its confidence level. The confidence level a number between and (inclusive) enclosed within <conf (ennceddd>>>>> </confidence> tags. The final answer is enclosed between <answer> </answer> tags. The analysis about confidence and uncertainty is enclosed within <analysis> </analysist tags. The assistant should reason about its confidence in the solution and its uncertainty in the solution within these tags. Here are some guidelines for the analysis: 1. Your task is point out things where the model could be wrong in its thinking, or things where there might be ambiguity in the solution steps, or in the reasoning process itself.
2. You should not suggest ways of fixing the response, your job is only to reason about uncertainties.
3. For some questions, the response might be correct. In these cases, It is also okay to have only a small number of uncertainties and then explicitly say that I unable to spot more uncertainties.
Uncertainties might be different from errors. For example, uncertainties may arise from ambiguities in the question, or from the application of particular lemma/proof.
If there are alternate potential approaches that may lead to different answers, you should mention them.
List out plausible uncertainties, do not make generic statements, be as specific about uncertainties as possible.
Enclose this uncertainty analysis within <analysis> </analysis> tags.
The final format that must be followed </think> <answer> final answer here </analysis> < <analysis> analysis about confidence and uncertainty here </analysis> <confidence> confidence level here (number between and 1) </ c/coffic (confidence>)
<think> reasoning process here
Simple RLCR Prompt
conversation between User and Assistant. The user asks a question, and the Assis- tant solves it. The Assistant first thinks about the reasoning process in the mind, pro- vides user with the final answer, then analyzes confidence about the solution and provides the user with its confidence level. The confidence level is number between and 1 (inclusive) enclosed within <confidence> //confidencer tags. The final an- swer is enclosed between <answer> </answer> tags. The analysis about confidence and uncertainty is enclosed within <analysis> </analysis> tags. The Assistant should reason about its confidence in the solution and its uncertainty in the solution within these tags. The final format that must be followed is: <think> reasoning process here </think><<nswer>> final answer here </ annser><analysss> analysis about confidence and uncertainty here </analysis><com wew.nniiifiieee dence>> confidence level here (number between 0 and 1) <<c/ooffi cisonfidence>
19

---

Published as a conference paper at ICLR 2026
Simple Confidence Prompt
conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and analyzes its confidence about the solution and then provides the user with the final answer as well as its confidence level. The confidence level is number between 0 and (inclusive) enclosed within <confidence idence> </conf idence> tags. The final answer is enclosed between <answer> </answer> tags. The final format that must be followed is <think> reasoning process here <//think><<nswer>> final answer here </answer> (number between and 1) s/confidences
confidence level here
RLVR Prompt
conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here <ansink>> answer here < </answer>.
Expert SFT Prompt
You are given question and a solution to it. You have to verify if the solution is correct and enclose your verification reasoning within <analysis> </analysis> tags. Your analysis should be a minimum of 300 characters and should sequentially through the thinking solution step by step. Here are guidelines for your analysis: Your analysis should also be in 'I' form as if you wrote the solution and are now verifying it.
1.
2. Your goal is not to solve the problem but instead to verify if the steps in the presented
solution are correct.
3. If there ambiguities in the solution steps or if step introduces uncertainty, you should mention it in the analysis.
Go through the solution sequentially in step-by-step manner.
5. The analysis should be 300 characters minimum.
Enclose this uncertainty analysis within <analysis> </analysis> tags.
D LLM USAGE
The authors made limited of ChatGPT to refine wording for clarity. It was not used for research ideation, related work retrieval, or substantive content generation.
20

---

Published as a conference paper at ICLR 2026
E COMPARING LOG SCORE AND BRIER SCORE
E INTUITION
Logarithmic score and Brier scores are both widely used proper scoring rules. Lemma showed that combining proper scoring rules with correctness rewards preserves the calibration incentive. Models are incentivized to output true correctness probability & = Py for both log and Brier scores. Using this, when the model outputs answer y and honestly reports confidence q Py:
Expacted Reward Correctress Prubabiltty
1.-
Brier Sane
Expected Roward
0.8
Log Score
write out the expected reward
04
02
-02
EErrier [[(y]] = Pu (1- + (-))
(12)
Py
Blog [R(y)] (1 log Py) (1 (1- Pb) log(1 Py).
Figure 5: Expected Reward vs correct-
(13) ness probability Py-
Fig. plots the expected reward as a function of py, While the Brier-based expected reward is strictly increasing in Py- the expected reward under the log score is not. This means that, when using log score combined with correctness rewards, there exist ranges of Py where the model receives higher expected reward by reporting answers with lower true correctness probability. Intuitively, the log score's 's calibration term can outweigh the correctness bonus, creating a non-monotonic reward landscape that can discourage the model from preferring more accurate answers.
E.2 Toy EXPERIMENT
Environment Setup. To empirically validate this phenomenon, we construct simple K-arm prediction task. For each instance, Nature samples a probability vector
K
Pk)),
O
1
i=1
which is unknown to the model. A sequence of N IID draws
21. EN ~ P. It {I....,K}, K}.
revealed the model in the prompt. The value of is drawn uniformly from
N ~ 1,,
The model's task is to predict the next draw
EN+1
by outputting an arm index ğ E (1...., K) along with a confidence value E [0, 1]. The model is free output invalid answer -1 with q = 0 if it wishes to abstain. We use K 5 &trouuhhuu. sample task instance is shown below.
Toy Arm Task Instance
There are 5 arms, each arm has a different probability of being sampled. You are given some draws from this 5-arm distribution. Your task is predict which arm the next draw will be from. You are also free to output -1 with confidence 0 you are really unsure.
Observed draws (in order): 1, 0.
Number of observed draws shown: 3
Answer with the arm index (0-4) that you predict for the draw.
21

---

Published as a conference paper at ICLR 2026
Since the observation sequence is short and varies arbitrarily across tasks, this setting induces substantial aleatoric uncertainty. The Bayes-optimal prediction conditioned on the observed data is the arm with highest empirical frequency,
N 1| i}.
1" arg VE
This environment is intentionally designed so that the model often highly uncertain about the correct arm. Referring to Fig. 5, we observe that under the logarithmic scoring rule, the expected reward in such low-confidence regimes is maximized by outputting an intentionally incorrect answer with confidence 0, rather than attempting good-faith prediction.
Training Setup. We train RLCR-Brit Brier and RLCR-Log on 10, 000 examples from the above dataset. We use the Qwen-2.5-77 model and use the same training configuration as Section B.2. For simplicity, we do not ask the model to do any uncertainty reasoning.
Results.
Fig. presents evaluation results on the Toy Arm task. Sample outputs from the two models are also shown below. The empirical findings closely match theoretical predictions. Under the logarithmic scoring rule, the model rapidly converges to a de- generate policy: outputs the invalid arm (-1) with reported confidence This strategy yields the maximum possible expected reward under the log score in the low-P,, Py regime, and consequently the model achieves an almost per- fect calibration metric. However, because the model never predicts valid arm, its accuracy on the underlying task collapses to O.
Method
Acc.
Brier
ECE
RLCR-Brier (ours)
34.4%
0.22
0.02
RLCR-Log
0.00
0.00
Figure 6: Performance the Toy Arm Task dataset. RICCR-Log collapses to degenerate so- lution, while RLCR-Brics achieves non-trivial accuracy and calibration.
for nearly all tasks.
In contrast, RLCR-Brics behaves qualitatively differently. Because the expected Brier reward is strictly increasing in the true correctness probability, collapsing to the invalid low-confidence answer strongly suboptimal. Models trained with RLCR-Bric therefore continue make genuine predictions about the next arm, achieving both non-trin trivial accuracy and stable calibration.
RI.C--rrie Sample Output
<think> Analyzing the frequency of each arm </think><<nswer>> <confidence> 0.25 </confidence:
</answer>
RLCR-Log Sample Output
<think> Based the given information, there is no data to predict the next draw. </think><<asserr>-1 </answer> cooniidene>> st/confidence
E.3 EXPERIMENTS ON HOTPOT
We also evaluate the two scoring rules in a realistic QA setting. To do so, we train an RLCRR-Log model using the same training configuration the RLCR models used in main HotpotQA results (see Table 1). Table reports a direct comparison between RLCR-Brin and RLCR-Logg the RLCR-Brier and RLVR numbers copied over from the main results table.
Overall, RLCR-Log performs slightly worse than RLCR-Bried on both accuracy and calibration. However, unlike the toy bandit setting, do not observe any evidence of reward hacking: the RLCR-Log model maintains reasonable accuracy and does not collapse to degenerate predictions. This suggests that though using log score can lead reward hacking, for many datasets hacking
22

---

Published as a conference paper at ICLR 2026
Method
HotpotQA
O.O.D Averaged
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
RLVR
63.0%
0.50
0.37
0.37
53.9%
0.50
0.46
RI.RR-Brier (ours)
62,1 1%
0.69
0.21
0.03
56.2%
0.68
0.21
0.21
RLCR--oog
59.5%
0.68
0.22
0.07
53.6%
0.67
0.22
Table 3: Comparison of RLCR-Brier and RLCR-Log on HotpotQA and 6 out-of-distributio (O.O.D.) datasets. RLCR-BaR -Bier marginally outperforms RLCR-Log in both accuracy and calibration.
might not happen practice. We believe the emergence hacking is dependent on both the data distribution well the model size.
F GENERALIZATION TO OTHER MODELS
OLMo-2
Method
HotpotQA
O.O.D Averaged
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
16.4%
0.55
0.78
0.79
47.8%
0.56
0.48
0.48
RLVR
61.7%
0.51
0.38
0.38
50.8%
0.53
0.48
0.48
RLCR (ours)
61.3%
0.58
0.24
0.09
49.3%
0.65
0.22
0.20
Table 4: Results of RLCR and RLVR training using OlMo-2-7B-Insrruct RLCR has comparable accuracy to RLVR, while significantly outperforming it in calibration.
Setup: To assess how well the RLCR framework generalizes across model families, we train RICC and RIVR variants starting from the O1Ms-2-7B-lnnsrruct model (OLMo et al., 2024) on our
HltpotQQAMModifer dataset.
Results: Table reports the final performance. The overall trends are consistent with our main
findings: RLCR and RLVR achieve similar accuracy, and both substantially improve accuracy over the base model. RLCR also delivers markedly better calibration than both RLVR and the base model.
F.2 QWEN3
Method
HotpotQA
O.O.D Averaged
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
61.1%
0.53
0.35
0.34
62.8%
0.59
0.28
0.28
RLVR
62.7%
0.53
0.36
0.36
65.5%
0.62
0.28
0.29
RLCR (ours)
61.8%
0.58
0.28
0.23
65.6 %
0.71
0.17
0.17
Table 5: Results of RLCR and RLVR training using Qwen-3-8B. RLCR maintains accuracy while substantially improving calibration both in-distribution and OOD settings.
We further assess performance on the newly released Qwen3 model family. Under in-distribution evaluation, RLCR matches RLVR in accuracy and surpasses it on AUROC, Brier. and ECE. When averaged across our OOD benchmarks, RLCR continues to outperform RLVR on AUROC, Brier. and ECE, with accuracy remaining effectively equivalent.
23

---

Published as a conference paper at ICLR 2026
G DOES REASONING IMPROVE CALIBRATION?
Recent work has shown that CoT reasoning can be unfaithful, with generated CoTTs that do not influence their final answers (Chen al., 2025). This raises the possibility that uncertainty analysis may not meaningfully inform the verbalized confidence score. To test this, we train two classifiers on HotPorQA- -MModiiied:
1. Baseline classifier: Trained RLVR outputs (these contain uncertainty analysis).
2. Analysis classifier: Trained on outputs of RLCR with confidence scores (present within <confidenced tags) removed to prevent direct hacking.
As both RL models have comparable task accuracy, differences in classifier performance would indicate that the RLCR-trained model's reasoning chains contain information specifically useful for calibration. We train classifiers for 3 different model sizes of the Qwen-base model: 0.5B, 1.5B and 7B. Figure 7 shows Brier and ECE scores on HotPotQA-Modiine 7B classifiers perform similarly, the analysis classifier outperforms the baseline at smaller sizes, suggesting classifier capacity is key. For sufficiently expressive classifier (as with the 7B model), is possible infer menfidence-relevaan features directly from the solution. In contrast, smaller classifiers can make better use of RLCR reasoning chains. We believe that broader questions about relationship between classifier capacity and CoT contents are an important topic for future work.
Interestingly, while
Baseline Classifier
0.25
Arslycis Chooilfcr
Analysis Classifier
0.20
Btier Sooro
0.125
0.100
0.05
0.025
0.00
0.000
0.5H
1.5H
78
0.5B
1.5B
Classifier Size (a)
Classifier Size (b)
Figure 7: Brier scores (a) and ECE (b) of baseline / analysis classifiers HotPotQA-Modiffed across three model sizes. Analysis classifiers outperform baselines at smaller sizes, suggesting that uncertainty CoT is essential for better calibration when capacity limited.
24

---

Published as a conference paper at ICLR 2026
H CONFIDENCE DISTRIBUTIONS
H.1 CONFIDENCE DISTRIBUTIONS ACROSS DIFFERENT INPUTS
(a) RLCR
(b) RLVR with Analysis
(c) RLVR + BCE Conf. Classifier
34
Contdarc
30
Canfearc
08
Figure 8: In-Distribution HotpotQA - calibration charts overlaid with confidence-frequency his- tograms for RLCR, RLVR, and RLVR+BCE Confidence Classifier.
Here, present calibration charts overlaid with confidence-frequeenyy frequency histograms. For an ideally calibrated model, the accuracy-calibration curve would lie close to the orange dashed line. The histograms directly address whether model's confidence values are genuinely diverse or instead clustered around a narrow range.
In Figure 8a, show results for In Distribution evaluation for HotpotQA. For RLCR, the histogram shows that the model uses the full confidence range, with substantial mass in mid-confidence bins (0.4-0.8) and non-trivial usage of both lower and higher bins. This indicates that RLCR produces input-dependent confidence scores rather than collapsing toward a single accuracy-like value. The accuracy-per cerruucyyy-e----iie per-bin curve also closely tracks the ideal diagonal, demonstrating that these varied confidence levels correspond to meaningful differences in correctness likelihood.
In contrast, RLVR (Figure 8b) concentrates almost all predictions in the 0.9-1.0 range, with very little representation of lower bins. This clustering suggests that RLVR's 's outputs are largely uniform and overconfident, and that its confidence does not meaningfully vary across inputs.
Finally, when we use an a-trind model as the base for BCE confidence classifier (Figure 8c), the accuracy-per-bin curve aligns more closely with the ideal line compared to the RLVR plots, suggesting that the classifier is also effective. However, it still underperforms RLCR.
Coafitene
Cosfidede
(a) RLCR
(b) RLVR+Analysis
(c) RLVR+BCE Classifier
Figure 9: Out-of-Distribution averaged 6 datasets calibration charts overlaid with confidence- frequency histograms for RLCR, RLVR, and RLVR+BCE Confidence Classifier.
We observe similar pattern in our out-of-distributt evaluations. Results are shown in Figure 9, RLVR (Figure 9b) continues to produce confidence values that collapse to 1.0 regardless of accuracy, and its accuracy-per-bin curve deviates substantially from ideal calibration- for example, predictions with verbalized confidences of 0.8-0.9 0.9 achieve only around 0.3 accuracy on average. Training a BCE confidence classifier top of the RLVR model (Figure 9c) mitigates this behavior some extent, yielding slightly more distributed spread of confidence values and an accuracy-per-bil curve that moves bit closer the ideal line. However, RLCR (Figure 9a) still performs best: it exhibits
25

---

Published as a conference paper at ICLR 2026
healthy spread of confidence values across the full range (0-1) and uccuracy-per per--eerbie per-bin eurve that remains much closer to ideal calibration than either RLVR or its BCE classifier variant.
H.2
PER INPUT CONFIDENCE DISTRIBUTION
Antwer indepencent
16
14-
10
0.00
6.55
0.10
0.15
0.20
0.25
Standard Deviatien of Confisience
Figure 10: Distribution of Standard Deviation of Answer-Condittoned and Answer-Independent confidence scores
To analyze how the model's confidence values are distributed for a given question, we conduct an analysis of per-input confidence variability. For each question, we compute the standard deviation of the model's confidence scores across sampled generations, and the plot shows the distribution of these per-question standard deviations.
plot two distributions:
Answer-Conditioned Distribution: p(c|x, y, a). For a given question, solution, and final answer, sample N 16 analysis/CoT trajectories and compute the standard deviation their confidence scores. This reflects how stable the model's confidence is for a given reasoning trajectory and answer. Intuitively, high standard deviation here would be undesirable, as would indicate that the model is internally inconsistent or "confused" about its confidence for a given solution.
2. Answer- lnneer-nneeendeett Distribution: p(c|z). We sample = 16 full CoTs, each with unique reasoning paths. These trajectories correspond to a variety of distinct reasoning paths (and may include multiple different answers). We then compute the standard deviation of the confidences over these trajectories for the same question, capturing how confidence varies across different solutions produced by the model for particular question.
As shown in Figure 10, the answer-independen distribution (orange) has significantly more mass the right, indicating noticeably higher variability in confidence across different reasoning trajectories. In contrast, the unswereconditioned distribution (blue) is narrower reflecting reduced noise once the answer is fixed but importantly, it still displays non-trivial spread rather than collapsing near zero.
Together, these findings show that when generating multiple reasoning trajectories for the same question, the confidence scores have noticeably higher variance compared to generating multiple confidence scores for fixed reasoning trajectory.
26

---

Published as a conference paper at ICLR 2026
I COMPARISON TO RL-CALIBRATION BASELINES
Method
HotpotQA
O.O.D Averaged
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Calibration RL (Full Seq)
0.00
N.A
0.00
0.00
0.00
N.A
0.00
0.00
Calibration RL (Analysis + Conf)
62.0%
0.53
0.24
0.08
$2.8%
0.57
0.27
0.25
Abstention-Re RL
62.1%
0.61
0.32
0.31
54%
0.59
0.35
0.35
RLVR
63.0%
0.50
0.37
0.37
53.9%
0.50
0.46
0.46
RLCR (ours)
62.1%
0.69
0.21
0.03
56.2%
0.68
0.21
0.21
Table 6: Results of RL-calibration baselines. All baselines significantly outperform RLVR on calibration, but underperform against RLCR on both accuracy and calibration. Training only with calibration rewards leads to degenerate solutions. This collapse can be avoided by masking loss over the the answer and think tokens. Training models to abstain never explicitly teaches confidence estimation and can also suppress exploration.
Baselines. We compare RLCR against carefully adapted variants of the most relevant RL-for- calibration baselines. Several prior approaches focus primarily calibration and do not explicitly optimize accuracy. To ensure fair comparison, we adapt these methods to the reasoning setting by initializing these baselines from RLVR model, which has already been trained for accuracy.
Calibration RL (Full-Sequence) (Stangel et al., 2025; Xu al., 2024): We train a variant that optimizes only the Brier-score reward, without any securacy-depentee term. We apply the loss over the entire generation (think, answer, analysis, confidence. This represents the most direct application of proper-scoring-rule reward to the reasoning-LLMM setting. We initialize from the RLVR model.
2. Calibration RL Inalysis+Confiddnnc Only). Prior work (Stangel et al., 2025) demon- strates that applying calibration rewards only to the analysis and confidence portions can stabilize training and preserve task accuracy. We therefore implement a stronger, accuracy- preserving variant of the above baseline by restricting the reward to the analysis and confi- dence spans while taking no loss over the thinking/answer. Similar to the above baseline, we initialize from the RLVR model.
3. Abstention-RL. (Wei et al., 2025; Mohamadi et al., 2025; Song et al., 2025): Recent papers propose ternary rewards that give +1 for correct answers, 0 for incorrect answers, and an intermediate reward X for explicit ahstentions (e.g.. "I don't know"). We adopt this family of methods with À 0.5, standard midpoint value. Because abstention models do not produce confidence scores, we evaluate calibration by using a test-time prompt that instructs the model never abstain. As this method directly optimizes for accuracy well, initialize from the standard Qwen2.5-7B model.
Results. Table 6 reports the full results. All baselines significantly outperform vanilla RLVR on calibration, but underperform against RLCR on all metries.
Calibration RL (ulll-Squeene) collapses to near-zero accuracy: with no reward shaping for cor- rectness, the model rapidly converges degenerate but wwwaddmmaxmiizin behavior, outputting empty or trivial answers with confidence 0. This yields perfect calibration under the Brier score and is essentially the optimal policy with a calibration-only reward. While Kl.rregularization can potentially reduce this collapse, there is constant pressure reduce task accuracy with this variant.
Calibration RL [Analysis+Conffddeee Confidence Only) prevents collapse and maintains accuracy comparable to RLCR, but calibration remains noticeably weaker. We hypothesize two contributing factors. First, the RLVR models might not be good starting points for further RL optimization. They might have reduced entropy very different output distribution compared to the base model. Second, jointly optimizing accuracy and calibration, as RLCR does, might provide complementary gradient signals that reinforce one another and enable more effective learning of confidence estimation.
27

---

Published as a conference paper at ICLR 2026
Abstention-Rl RL underperforms RLCR calibration. Because the abstention reward only teaches whether the model's internal confidence exceeds the threshold X, never learns fine- anee-grined confi- dence estimation. Morcover, rewarding abstention learns to abstain on difficult questions, it may no longer attempt them, limiting both reasoning improvement and calibration learning. Abstention rewards are better suited for settings where the primary goal is not to improve accuracy, but rather teach model the skill to abstain.
also suppress exploration: once the model
J ANALYZING SFT+RLCR
Method
HotpotQA
0.0.D Averaged
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
56.1%
0.56
0,40
0.39
47.8%
0.53
0.46
0,45
RLVR
72.9 %
0.47
0.28
0.26
52.5%
0.52
0.49
0,49
RLCR
72.7%
0.67
0.17
0.10
50.9%
0.60
0.28
0.25
SFT+RLCR (original)
72.2%
0.78
0.14
0.08
43.8%
0.66
0.24
0.18
SFT+RLCR (tweaked prompt)
72.5%
0.75
0.15
0.09
49.8%
0.62
0.25
0.21
Table Results of SFT+RLCR with simple prompt change. Adding a single line to the prompt boosts O.O.D accuracy from 43.8% to 48%.
The results in Table showed that our SFT+RLCR model experienced substantial drop in O.O.D. accuracy. While RLVR and vanilla RLCR achieve O.O.D. accuracies of 52.5% and 50.9% respec- tively, SFT+RLCR reaches only 43.8%. To investigate this degradation, we manually examined SFT+RLCR's generations and uncovered consistent abnormality unique to this model: the model often identifies the correct answer during its reasoning, dom number inside the answer tags. This behavior suggests that extended RL training on Math-heavy data may induce overfitting or domain-specinf bias in how the model formats its final answer.
then places an incorrect, seemingly ran-
To test whether this issue reflects catastrophic forgetting the evaluation with minor modifications to the prompt. We added a single clarifying instruction:
more superficial misalignment, reran
"Be careful about what you put in the answer tags. Do not arbitrarily put numbers there if the question has nothing to do with Math." "
Remarkably, this simple change improves O.O.D. accuracy from 43.8% 49.8%, shown in Table This indicates that the degradation is not solely due to catastrophic forgetting; rather, most of the failure arises from formatting biases learned during Math-focused RL training, which straightforward to reverse.
We hypothesize that introducing small amount of KL regularization, training more diverse RI. dataset beyond Math, would mitigate these effects. Prior work has also observed that SFT can induce more forgetting than RI. (Shenfeld al., 2025; Mukherjee al., 2025), and the remaining performance gap may indeed reflect residual forgetting- to a much lesser extent than we initially suspected.
28

---

Published as a conference paper at ICLR 2026
K НОTPOT TRAINING RESULTS
Fig. 11 shows the training curves for RLCR and RIVR. Both the correctness and calibration reward for RLCR increase smoothly, indicating that the model is able to jointly improve accuracy and calibration. Notably, the completion lengths of our method gradually increase during training as uncertainty reasoning improves.
250
Completion Length (tellems)
0.7
Rewerd
0.6
2004
0.5
150-
RLCR (Comeviness Reward)
RLCR (Brier Reward)
RLCR
RLVR
RIVR
10
50
100
200
300
20
50
100
200
300
Step
(b)
Figure 11: (a) Reward curves for RLCR (ours) and RLVR. Both correctness and calibration rewards improve under method, demonstrating simultaneous gains in correctness and calibration. The Brier reward is shifted upward by for clarity. (b) Completion lengths during training. The completion lengths of our method gradually increase during training uncertainty reasoning improves.
29

---

Published as a conference paper at ICLR 2026
L FULL RESULTS: MODELS TRAINED ON HOTPOTQA
Discussion Below we present the full dataset-specific results. Although RLCR outperforms base- lines on average, there can be considerable variance in calibration on individual O.O.D datasets. A particularly illustrative case is CommonsenseQA, where all methods achieve roughly 90% accuracy. RLVR is highly overconfident and consistently predicts 85-100% confidence across all questions and datasets. On CommonsenseQA this overconfidence happens to coincide with the dataset's 's high accu- racy, producing deceptively strong calibration. Importantly, this alignment a spurious correlation arising from RLVR's uniformly inflated confidence rather than genuine uncertainty modeling.
At the same time, we acknowledge that all methods, including RLCR, have significant room for improvement in O.O.D. calibration. We believe that extending RL training and incorporating more diverse training dataset can further strengthen robustness and calibration.
Method
SimpleQA
Trivia
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
13.5% 1.0
0.50 + 0.01
0.77 + 0.01
0.81
57.8% + 2.2
0.51 + 0.01
0.38 + 0.02
0.37
RLVR
0.00
0.88
0.00
0.38 0.02
RLVR+BCE Classifier
0.48 0.03
0.53 0.01
0.64
0.57 1: 0.03
0.26 0.01
0.15
RLVR+Brier Classifier
12.4% + 1.0
0.60 = 0.03
0.11 = 0.01
0.06
62.2% + 2.1
0.57 = 0.03
0.37 0.01
0.37
RIYR+Prohe
0.51 0.03
0.14
0.12
0.03
0.43 +
Answer Probability
0.42 + 0.03
0.83 = 0.01
0.85
0.50 -: 0.03
0.37 1 0.02
0.36
RLCR (ours)
12.1% + 1.0
0.60 + 0.02
0.24 + 0.01
0.34
60.8% +: 2.1
0.73 1: 0.03
0.20 0.01
0.06
Table 8: Performance on SimpleQA and Trivia datasets. Values indicate the mean with error margins given as half-widths of the 95% bootstrap confidence intervals. Best values bolded.
Method
CommonsenseQA
GPQA
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
88.6% + 1.8
0.62 + 0.05
0.10 + 0.01
0.00
39.5% 4.4
0.49 + 0.05
0.50
0.04
0.51
RLVR
0,50 + 0.00
0.09 + 0.02
0.09
0.50 = 0.00
0.60 = 0.05
0.60
RLVR+BCE Classifier
0.06
0.12
0.52
0.02
0.16
RLVR+Brier Classifier
90.8% + 1.6
0.65 + 0.05
0.26 = 0.01
0.42
39.5% 4.6
0.52 0.06
0.29 0.03
0.21
RLVR+Probe
0.50 + 0.06
0.75 = 0.01
0.81
0.50 = 0.05
0.33 = 0.04
0.29
Answer Prob
0.06
0.08 0.01
0.03
0.53 0.05
0.54 0.04
0.54
RLCR (ours)
91.3% + 1.6
0.73 1: 0.06
0.17 1 0.01
0.30
41.5% +: 4.7
0.55 0.05
0.27 0.01
0.16
Table Performance on Commonsense and GPQA datasets. Values indicate the mean with error margins given as half-widths of the 95% bootstrap confidence intervals. Best values bolded.
Method
MATH-500
GSMSK
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
46.8%
4.30
0.59 +: 0.05
0.04
0.49
73.5%
2.50
0.52 0.03
0.24 + 0.02
0.22
RLVR
0.50 + 0.01
0.61 + 0.04
0.61
0.50 0.00
0.20 = 0.02
0.20
RLVR + Classifier
0.70 + 0.05
0,26 + 0.02
0.22
0.57 + 0.04
0.16 = 0.02
0.07
RLVR Classifier
4.20
0.58
0.03
80.1% 2.10
0.04
0.55 0.02
0.63
RLVR Probe
0.67 + 0.05
0.24 : 0.03
0.15
0.53 - 0.04
0.41 = 0.01
0.48
Answer Probability
0.79 + 0.05
0.51 + 0.04
0.55
0.78 0.02
0.16 + 0.02
0.17
RLCR (ours)
45.4%
4.50
0.72 0.05
0.25 0.02
0.19
86.3%
1.75
0.74 0.04
0.14 0.01
0.20
Table 10: Performance on Math-500 and GSM8K datasets. Values indicate the mean with error margins given half-width of the 95% bootstrap confidence intervals. Best values bolded.
30

---

Published as a conference paper at ICLR 2026
Method
HotpotQA
HotporQA-MMoiites
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
Base
39.7% +: 3.10
0.54 t 0.02
0.53 0.03
0.53
30.4%
4,30
0.59 +: 0.04
0.57 0.04
0.59
RLVR
0.50 0.00
0.37 0.03
0.37
0.50 + 0.00
0.34 +: 0.05
0.54
RIVR +BCE Classifier
0.66 + 0.04
0.22 + 0.01
0.07
0.77 + 0.08
0.21 + 0.02
0.13
RLVR +Brier Classifier
63.0% + 3.05
0.65 0.04
0.22 0.02
0.09
46.0% 4.30
0.79 + 0.05
0.20 0.02
0.12
RLVR +Probe
0.55 1 0.04
0.04
0.10
0.57 1: 0.05
0.26 1: 0.01
0.12
Answer Prob
0.72 = 0.04
0.36 = 0.03
0.36
0.61 = 0.05
0.32 + 0.04
0.53
RLCR (ours)
62.1% + 3:05
0.69 0.04
0.21 0.01
0.03
44.4%
4.20
0.80 + 0.05
0.19 0.02
Table 11: Performance on H and HotpotQA-Modiiice datasets, Values indicate the mean with error margins given as half-widths of the 95% bootstrap confidence intervals. Best values bolded.
M FULL RESULTS: MODELS TRAINED ON MATH
Method
SimpleQA
TriviaQA
Aec.
AUROC
Brier
Acc.
AUROC
Brier
ECE
Base
13.5% 1.01
0.50 = 0.01
0.77 0.01
57.8% 2.25
0.51 0.01
0.38 + 0.02
0.37
RLVR
0.53 = 0.02
0.83 0.01
0.84
T
0.50
0.43 : 0.02
0.43
RIVR+BCE Classifier
0.45 + 0.02
0.57 0.01
0.64
0.57 + 0.03
0.29 + 0.01
0.22
RLVR+Brier Classifier
15.2% 1.05
0.49 + 0.02
0.15 0.01
58.3% 2.20
+ 0.01
0.25
RLVR+Probe
0,44 1 0.02
0.58 0.01
0.66
0.56 0.03
0.30 0.01
0.23
Answer Prob
0.45 = 0.02
0.80 = 0.01
0.81
0.48 = 0.02
0.40 + 0.02
0.38
RLCR (ours)
0.52 0.02
61.0%
0.67
4: 0.01
0.10
RICRRSFF (ours)
11.4%
0.94
0.60 + 0.03 0.29 0.01
0.40
55.6%
2.20
0.72 0.02 0.21 + 0.01
0.06
Table 12:
Performance on SimpleQA and TriviaQA datasets. Values indicate the mean with error margins given as half- widths of the 95% bootstrap confidence intervals. Best values bolded.
Method
Commonsense(A
GPQA
Áec.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
88.6% + 1.76
0.62 0.05
0.10 + 0.01
0.00
39.5% + 4.35
0.49 + 0.05
0.50 + 0.04
0.51
RLVR
0.55
0.13 0.02
0.13
0.50
0.53 0.04
0.53
RLVR+BCE Classifier
0.61 0.05
0.23 + 0.01
0.34
0.51 + 0.05
0.33 4 0.03
0.27
RLVR+Brier Classifier
89.3% + 1.88
0.60 : 0.05
0.30 0.01
0.45
50.0% + 4.91
0.53 1 0.05
0.33 + 0.03
0.28
RIYR+Prohe
0.57 0.05
0.09
0.28
0.50
0.02
0.19
Answer Prob
0.56 1: 0.05
0.10 0.02
0.09
0.53 1: 0.06
0.44 1: 0.04
0.40
RLCR (ours)
90.1%
1.68
0.62 1: 0.05
0.21 1 0.01
0.34
43.3% + 4.46
0.57 1: 0.05
0.26 1: 0.02
0.10
SFT+RLCR (ours)
77.6%
2.17
0.73
0.22 + 0.02
0.25
32.6% + 4.35
0.60 0.06
0.23 + 0.02
0.08
Table 13:
Performance on C and GPQA datasets. Values indicate the mean with margins given half widths of the 95% bootstrap confidence intervals. Best values bolded.
31

---

Published as a conference paper at ICLR 2026
Method
MATH-500
GSM8K
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
ECE
Base
46.8% +: 4,20
0.59 + 0.05
0.48 0.04
0.49
73.5% +: 2.46
0.52 d: 0.03
0.24 + 0.02
0.22
RLVR
0.45 + 0.04
0.44 0.04
0.43
0,47 0.05
0.09
0.05
RIVR +BCE Classifier
0.77 + 0.05
0.22 + 0.03
0.18
0.77 + 0.05
0.08 + 0.01
0.06
RLVR +Brier Classifier
59.2% + 4.10
0.79 = 0.05
0.22 0.03
0.18
90.6% + 1.57
0.76 0.05
0.08 + 0.01
0.05
RLVR +Probe
0.67 1 0.05
0.26:10..3
0.20
0.56 0.05
0.11 1: 0.01
0.14
Answer Prob
0.54 = 0.05
0.39 0.04
0.39
0.48 0.02
0.09 = 0.02
0.09
RLCR (ours)
59.8% + 4.20
0.04
0.23
0.12
89.6%
1.71
0.03
0.10 0.01
0.12
SFT+RICR (ours)
55.8% + 4.40
0.81 0.05
0.19
0.03
0.16
90.4% + 1.67
0.73 + 0.04
0.08 + 0.01
0.06
Table 14: Performance on MATH-500 and GSM8K datasets. Values indicate the mean with error margins given as half-widths of the 95% bootstrap confidence intervals. Best values bolded.
Method
Big-Math Digits
HotpotQA
Acc.
AUROC
Brier
ECE
Acc.
AUROC
Brier
Base
48.1% 3.10
0.56
= 0.03
0.47
39.7% 3.10
0.54 + 0.02
0.03
0.53
RLVR
0.50 0.03
0.32 + 0.03
0.30
0.50 + 0.01
0.55 + 0.03
0.55
RLVR+BCE Classifier
0.81 = 0.04
0.16 = 0.02
0.05
0.58 + 0.03
0.28 0.01
0.20
RLVR+Brien-Clasiffe
0.79
0.02
49.7%
0.60 1:
0.00
RLVR+Probe
0.72 0.04
0.19 = 0.02
0.07
0.58 + 0.04
0.28 0.01
0.16
Answer Prob
0.53 : 0.04
0.30 = 0.03
0.30
0.60 1 0.03
0.47 0.03
0.47
RLCR (ours)
68.7% 2.75
0.73
0.18 0.01
0.05
48.3%
3.00
0.61 0.04
0.27 0.01
0.18
RLCR+SFT (ours)
70.5%
2.70
0.79 1 0.04
0.15 + 0.02
0.03
41.6% 1: 3.15
0.65 1: 0.04
0.24 +: 0.01
0.09
Table 15:
Performance on Big- Digits and HotpotQA. Values indicate the mean with error margins given as half-widit of the 95% bootstrap confidence intervals. Best values bolded.
32

---

Published as a conference paper at ICLR 2026
SIMPLEQA EXAMPLE
Question
Who was awarded the Oceanography Society's ' Jerlov Award 2018? (Ground Truth Answer: Annick Bricaud)
RLCR (ours)
The question asks was awarded the Oceanography Society's Jerlov Award 2018. The Jerlov Award is given by the Oceanography Society, and it is named after the late Professor Gunnar Jerlov, pioneer in the field of ocean optics. To find the winner of the 2018 award, would need to look up the most recent information the Oceanography Society's website other reliable sources. </think>
<think>
<answer> Dr. David W. Schiedek </answer>
<analysis> The Oceanography Society's website or other reliable sources do not provide information on the winner of the 2018 Jerlov Award. It is possible that the information is nol readily available that the award was not given in 2018. Additionally, the name "Dr. David W. Schiedek" is not sell-known figure in the field of occanography, which raises uncertainty about the accuracy of this answer. </analysis>
<confidence> 0.25 </conidence> idence>>
RLVR
person was awarded the Oceanography Society's Jerlov Award 2018. The Jerlov Award is given by the Oceanography Society to recognize outstanding contributions to the field of oceanography. The recipient for 2018 was Dr. Laurence C. Smith. </think>
<think> The question
<answer> Laurence Smith </answer>
<confidence> 100. </confidence>
Qwen-7B-Base
<think To find out who was awarded the Occanography Society's 's Jerlov Award in 2018, I will search for the most recent information the award winners from the Occanography Society's official website reliable source that covers scientific awards. </think>
<answer> The Oceanography Society's Jerlov Award in 2018 was awarded to Dr. Lisa Levin. </answer>
<confidence> 0.95 </confidence>
33

---

Published as a conference paper at ICLR 2026
HOTPOTQA-MODIHED EXAMPLE
All supporting information from the given question has been removed. The only way to answer correctly if model remembers the information in weights.
Question and Supporting Information
Question: Jacques Sernas, actor Fugitive in Trieste, was of what nationality? (Ground Truth Answer: Lithuanian-bon French)
Your answer will be verified with exact match score. To ensure correct verification, only provide the reasoning process within the <answer> </answer> tags.
within the <answer> </answer> tags. Do put any sentences or
Supporting Information:
Paragraph 0
Man From 1997 is a time travel episode of the 1956-5 anthology television series Conflict directed by Roy del Ruth, produced by Roy Huggins, written by James Gunn from story by Alfred Bester, and starring Jacques Sernas, Charles Ruggles, Gloria Talbott and James Garner. The music was written by David Buttolph and the cinematographer was Ted D. McCord. The show was originally telecast on November 27, 1956 and a kinescope of the broadcast currently exists.
Paragraph 1
Altair is a 1956 Italian romantic drama film directed by Leonardo De Mitri and starring Franco Interlenghi, Antonella Lualdi and Jacques Sernas.
Paragraph 2
Dieci canzoni d' d'amore da salvare (English title: "Ten Love Songs") is a 1953 Italian film directed by Flavio Calzavara. The plot concerns songwriter, played by Jacques Sernas, who leaves his sweetheart and publisher when he learns that he going blind. Supporting Sernas were Brunella Bovo, Franca Tamantini, and Enrico Viarisio.
Paragraph 3
Loving You My Sin (Italian: Amarti il mio peccato) is a 1953 Italian melodrama film directed by Sergio Grieco and starring Jacques Sernas, Luisa Rossi and Elisa Cegani.
Paragraph 4
It's All Adam's Fault or In Six Easy Lessons (French: C'est la faute d' d'Adm)) is a 1958 French comedy film directed by Jacqueline Audry and starring Dany Robin, Jacques Seraas and Mijanou Bardot.
Paragraph
Balearic Caper (Spanish: "Zarabanda bing bing", Italian: "Baleari Operazione Oro", French: "Barbouze chérie", also known Operation Gold) is 1966 Spanish- Spanish-Italian-Frem Italian-French heist-Eurospy comedy film written and directed by José María Forqué and starring Jacques Sernas, Daniela Bianchi and Mireille Dare. It was shot in lbiza.
Paragraph 6
Finishing School (Italian: Fanciulle di lusso) is 1953 French- comedy film directed by Bernard Vorhaus and starring Susan Stephen, Anna Maria Ferrero and Jacques Sernas. was made at Cinecittà with sets designed by the art director Franco Lolli. It is also known by the alternative title of Luxury Girls.
Paragraph 7
Jump into Hell is 1955 war film directed by David Butler. The film stars Jacques Sernas (billed as "Jack Sernas") and Kurt Kasznar. The first contemporary Hollywood war film of the war in Indochina, the story fictionalized account of the Battle of Dien Bien Phu.
34

---

Published as a conference paper at ICLR 2026
RLCR
<think> The question asks for the nationality of Jacques Sernas, the actor in Fugitive in Trieste. The information provided mentions Jacques Scrnas in several films, but docs not specify his nationality. However, does mention that he starred in Altair, 1956 Italian romantic drama film, and that he was an actor several French and Italian films. This suggests that Jacques Semna might be French or Italian nationality, as he starred in films produced those countries. However, there is no direct information about his nationality in the given text. </think>
<answer> French </ </answer>
<analysis> The information provided mentions Jacques Semas in several French and Italian films, and he starred in Altair, 1956 Italian romantic drama film. However, there is no direct information about his nationality. While is possible that could be French or Italian nationality, the lack of specific information makes it uncertain. Additionally, the fact that he starred in both French and Italian films could suggest that he might be of mixed nationality have connections to both countries. Due to the ambiguity and lack of direct information, the confidence in this answer relatively low. </analysis>
<confidence> 0.4 </confidence?
RLVR
<think> The question asks for the nationality of Jacques Sernas, the actor in Fugitive in Trieste. Jacques Sernas American actor. </think>
<answer> American </answer>
Qwen-7B-Bas Base
<think> To determine Jacques Sernas' nationality, will analyze the information provided the supporting information.
From the information given, can see that Jacques Sernas has appeared in various films from different countries, such as Italy, France, and Spain. However, there specific mention of his nationality in the provided information.
Since there is no direct information about Jacques Sernas' nationality, will analyze the context of the question. The question is about Jacques Sernas' nationality in the context of the film "Fugitive in Trieste." Trieste
city in Italy, and the film Italian romantic drama.
Based on this context, is reasonable to assume that Jacques Sernas is of Italian nationality. However. without any direct information, cannot be 100% confident in this assumption. </think>
<answer> Italian </ /naswer>>
<confidence> 0.8 </confidence
35

---

Published as a conference paper at ICLR 2026
MATH EXAMPLE
An example where the analysis from the vanilla RLCR is generic, while the SFT+RLCR model produces much more specific evaluation that directly references the reasoning process. Although the RLCR model achieves reasonable calibration on Math, the SFT+RLCR model outperforms it, likely because it has learned to generate more effective uncertainty-aware chain-of-thought reasoning.
Question
Dalton, Hunter, Alex, Bella, and Chris formed a Superhero Fan Club. They set goal to watch as many Superhero movies as they could in one summer. Dalton watched 15 movies, Hunter watched 19, Alex watched 25, Bella watched 21, and Chris watched 11. They all watched superhero movies together, while Dalton, Hunter, and Alex watched an additional 3 movies only among themselves, and Bella and Chris watched 2 more movies only with each other. Every other movie they watched was different. How many different movies did they see? (Ground Truth Ans: 63)
RLCR (ours)
<think> To determine the total number of different movies watched by Dalton, Hunter, Alex, Bella, and Chris, we need to account for the movies they watched together and the additional movies they watched among themselves with each other.
First, let's caleulate the total number of movies watched if didn't account for any overlaps: Dalton watched 15 movics. Hunter watched movics. Alex watched 25 movics. Bella watched 21 movics. Chris watched 11 movics.
The sum of these is:
15 + 19 25
Next, we need to subtract the overlaps. They all watched 5 movies together, so we subtract 4 overlaps (since the 5 movies are counted times in the initial sum):
91 5 = 91 20
Now, we need to account for the additional movies watched among specific groups: Dalton, Hunter, and Alex watched 3 additional movies only among themselves. These movics are already subtracted the previous step, so we don't 't need subtract them again. Bella and Chris watched 2 additional movies only with each other. These movies are also already subtracted the previous step, so we don't t need to subtract them again.
Since all the overlaps have been accounted, the total number of different movies they saw is 71.
<answer> 71 </answer>
<analysis> The solution provided involves reasonable approach accounting for the overlaps in the number of movies watched. However, there uncertainty in the interpretation of problem statement, particularly the part about "Every other movic they watched was different."
This phrase could be interpreted in two ways: 1. That the movies watched together. the 3 additional movies watched by Dalton, Hunter, and Alex, and the 2 additional movies watched by Bella and Chris are all distinct from other from other movies watched by the individuals. 2. That the total number of movics watched by each individual, minus the overlaps, results in set of distinct movics. Given the information provided, the first interpretation seems more plausible, but it introduces an element of uncertainty. The problem does not clearly state that all movies watched are distinct, only that "Every other movie they watched different." which could he interpreted subset of the total movies watched.
Therefore, the confidence level in the solution set at 0.6, acknowledging the potential for misinterpre- tation and the lack of clear information in the problem statement. </ /analysis>
<confidenced 0.6 </ s/confidence
36

---

Published as a conference paper at ICLR 2026
SFT+RI.R (ours)
<think> To determine the total number of different movies the Superhero Fan Club members watched, we need to account for the movies watched individually and in groups. ensuring we don't double-count any movies.
First, let's sum the total number of movies watched by each member without considering overlaps: Dalton watched 15 movics. Hunter watched 19 movies. Alex watched 25 movies. Bella watched 21 movies.- Chris watched 11 movies. Adding these gives:
15 + 19 + 25 21 + 91
Next, need to subtract the overlaps. They all watched movies together, subtract 4 times (since the 5 movies are counted times in the initial sum, once for each person):
91 - 4 5 = 91 20 = 71
Now, we need account for the additional movies watched smaller groups. Dalton, Hunter, and Alex watehed more movies only among themselves. These movics are already included in the initial counts for Dalton, Hunter, and Alex, but we need to ensure they are not double-counted in the all- all-together group. Since the 3 movies are only among these three, they not part of the 5 movies watched together, don't need to subtract them again. They already included in the individual counts and group so add them back the total after the initial subtraction:
71 + 3 = 74
Bella and Chris watched 2 more movies only the initial counts for Bella and Chris, but we need to ensure they are not double-counted. Since the movies are only between Bella and Chris, they are not part of the movies watched together or the movies watched by Dalton, Hunter, and Alex, we add them back to the total after the initial subtraction:
each other. These movies are already included
+ 2
At this point, have accounted for all the overlaps, and the total number of different movies they is 76. To verify. let's consider the structure of the problem: The movies watched together counted 5 times initially but subtracted times, so they are correct. The 3 movies watched by Dalton,
Hunter, and Alex are included in their individual counts and the group of 5, so they are correct. The 2 movics watched by Bella and Chris are included in their individual counts, so they are correct. The final count of 76 different movics is consistent with the problem constraints. Therefore, the total number different movies they saw 76, </think>
<answer> 76 </anwer>>
<analysis> Upon reviewing the solution, notice that the reasoning process involves summing the total movies watched individually, subtracting the overlaps, and then adding back the croup-ppecffie movies. The initial sum of 91 is correct, and the subtraction of 20 for the movies watched together is appropriate, resulting 71. However, the addition of the 3 movies watched by Dalton, Hunter, and Alex (3) and the movies watched by Bella and Chris (2) is problematic. These movics are alrcady included in the individual counts and the group of 5, so adding them back is redundant and unnecessary. The reasoning does not clearly explain why these movies are added back, and the final total of 76 does seem to be derived from clear, step-by-step logical flow. The explanation about the movies being "only among these three" and not part of the confusing, as the movies are part of the individual counts and the group of 5, so they should not be added back. The 2 movies watched by Bella and Chris are also problematic, as they are only between Bella and Chris, but the reasoning does not clearly address how they are added back without double-counting. The final answer of 76 is not supported by the reasoning, and the steps are convoluted, making the solution unclear and potentially sncorret.<//nnlysiis.
<confidence> 0.3 </conf ience>
37

---

Published as a conference paper at ICLR 2026
RLVR
<think> To determine the total number of different movies watched by Dalton, Hunter, Alex, Bella, and Chris, we need to carefully account for the movies they watched together and the movies they watched exclusively among themselves and with cach other.
**Total movies watched by each individual:** Dalton: 15 movies Hunter: 19 movies Alex: 25
movies- Bella: movies Chris: movies
**Movies watched together:** They all watched movies together.
3. **Moie watched exclusively among themselves:** Dalton, Hunter, and Alex watched an additional movies only among themselves. Bella and Chris watched 2 more movies only with each other.
*Caacculating the total number of movics watched: ** First, sum the total movies watehed by each
individual:
15 + 19 25 21 + 11 91
Subtract the movies watched together (since they are counted multiple times in individual totals):
91 5 86
Add back movies watched together (they should only counted once):
86 5 = 91
Subtract the movies watched exclusively among themselves with each other (since they are also counted multiple times): Dalton, Hunter, and Alex watching 3 movies only among themselves are already subtracted when we subtracted the 5 movies watched together. Bella and Chris watching 2 movies only with each other already subtracted when subtracted the 5 movies watched together. Therefore, do not need make any further adjustments for these exclusive movies they already accounted for the initial sum and subtraction.
5. **Final calculation:** The total number of different movies they saw is the sum of all individual movies watched, minus the movies watched together (to avoid double-counting). plus the movies watched together again (to include them once), and further adjustments needed for the exclusive movies:
5 91
Thus, the total number of different movies they saw is:
91
</think>
<answer> </answer>
<confidence> 90 </confidence>
38


Reviews:
[{"review_id": "tmd8cgyAoW", "reviewer": "Reviewer_ffd5", "summary": "- This paper proposes a reward function that simultaneously incentivizes both answer correctness and probabilistic calibration, moving beyond binary correctness signals traditionally used in RLHF. By introducing a smooth, bounded Brier-score term, the method encourages models to be not only accurate but also appropriately uncertain. The proposed approach, RLCR (Reinforcement Learning with Calibrated Rewards), addresses common issues like reward hacking found with log-likelihood and unbounded scoring rules. The authors demonstrate that RLCR improves both calibration and output quality across reasoning tasks. Extensive experiments show that RLCR outperforms binary reward baselines in terms of calibration error, and answer diversity. Notably, the method avoids overconfident hallucinations and penalizes low-confidence correct answers, promoting reliable confidence estimates.", "questions": "- **Definition of p** Lines 92 and 134 use $p$ without definition; it should be mentioned that $p$ is the true distribution. In line 145, I doubt that the true probability $p(a=1)$ should be $p(a)$.\n- **GRPO acronym** Line 202 should spell out GRPO and include a citation.  \n- **Confidence scale inconsistency** Appendix B.4 defines confidence in [0, 100] at first, then [0, 1]. I wonder why this is the case.\n- **Table 1 labeling** Upper panel uses “OOD” while the lower uses “OOD averaged”; these two should match.  \n- **Dataset choice** Why use *hotpotqa-distracted*? Was the original HotpotQA too easy (near-perfect accuracy/calibration)?  \n- **ECE binning** What happens if the number of bins increases (e.g., 20)?  \n- **AUROC definition** Clarify whether AUROC is computed from verbalized confidence or token-level probabilities.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles a timely and practically-relevant problem supported by a fair amount of experiments. \n- Overall, the paper is clearly written and easy to follow.", "weaknesses": "- **Model coverage** The paper only tests Qwen2.5-7B, and additional models, maybe a non-reasoning model, should be included to demonstrate that RLCR generalizes beyond a single model. I must state that this is not a mere superficial comment rather one of my major concerns.\n- **Fair evaluation** Page 27 notes that RLCR produces shorter `<think>` sections than RLVR. It is unclear whether the results were evaluated under equal inference budgets, and this should be explicitly specified to ensure fairness. Further, I wonder whether reasoning length correlates with uncertainty. \n- **Missing hyperparameters** Key training details are absent, including GPU type, learning rate, and GRPO training configuration.\n- **Oracle-based labeling concern** For Math datasets, the uncertainty labels are generated using DeepSeek-R1, a far larger model. This could transfer oracle-level knowledge and confound the evaluation.  Please correct me if I understood something wrong.\n- **Inconsistent results and missing discussion** While Table 1 reports averaged results, some findings contradict this trend. For example, in Appendix E.3 (Table 3), RLVR achieves an ECE of 0.09 whereas RLCR records 0.30 with nearly identical accuracy. The paper should analyze and discuss such failure cases. Likewise, lines 356–357 state that RLCR maintains accuracy, but it clearly lags behind RLVR.  \n- **Formatting confound (analysis tag)** RLCR uses an `<analysis>` tag that allows longer uncertainty reasoning, whereas baselines do not. This formatting difference must be ablated by testing RLCR without the analysis section so that the only change is the reward function. This is crucial because (1) RLCR incurs higher inference costs (see p. 24, 26), and (2) prior work (Yoon et al., 2025) already shows that extended uncertainty reasoning improves calibration. Further, a fair comparison should thus include *RLVR + uncertainty analysis* as baselines. \n- **Missing baselines** The paper omits comparisons with prior works that enhance verbalized uncertainty. Even if those do not jointly optimize accuracy and calibration via RL, they should be included for context. Moreover, instead of training separate RLVR classifiers, a more natural baseline would be SFT with uncertainty labels (e.g., oracle- or self-consistency-based) alongside existing RL-calibration methods. \n- **Failure cases for unbounded scoring rules** Table 1 shows BCE classifiers outperforming Brier ones, yet there is no empirical support for the theoretical claim that unbounded proper scores lead to overfitted low-confidence answers. Experiments using other scoring rules such as logarithmic (Eq. 5) or spherical (Eq. 7) are needed.  \n- **Catastrophic forgetting and missing regularization** In Table 1(b), SFT + RLCR performs worse, and the authors blame this on catastrophic forgetting. It is unclear why KL-regularization—a rather straightforward fix—was not applied.  \n- **Figure 3 interpretation** Figure 3 indicates (1) some incorrect answers appear with high confidence and (2) majority voting consistently underperforms confidence-weighted voting by a small margin. Clarify: (a) how majority voting handles ties; and (b) include plots of both accuracy and frequency (y-axis) versus confidence bins (x-axis) to visualize calibration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K0UN7Vzzcl", "reviewer": "Reviewer_j9oB", "summary": "The paper identifies a key limitation of reinforcement learning with verifiable rewards (RLVR): the binary correctness reward penalizes equally whether the model abstains or produces an incorrect answer. Consequently, models that are initially well-calibrated tend to become overconfident after RL training. The authors propose reinforcement learning with calibration rewards (RLCR), which augments the binary correctness reward with a Brier score term based on proper scoring rules. This modification explicitly incentivizes calibrated confidence estimation in addition to correctness. RLCR substantially improves calibration both in-domain and out-of-domain without sacrificing accuracy, compared to RLVR and other post-hoc confidence baselines. Moreover, combining RLCR with confidence-weighted test-time scaling methods (e.g., weighted majority voting, best-of-N) further enhances accuracy.", "questions": "1. Could you report or comment on RLCR without confidence analysis (i.e., without `<analysis>` reasoning, only outputting numerical confidence)?\n2. Could the authors clarify how RLCR differs from or complements traditional Best-of-N sampling (using external reward model or sentence likelihood), and whether its advantages persist when compared under identical computational budgets (number of samples)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a well-known yet underexplored issue in RL-based reasoning: overconfidence induced by correctness-only rewards. This concern has also been raised in recent works such as *“Why Language Models Hallucinate” (OpenAI, 2025)*, but practical algorithmic remedies have been lacking.\n\n- The proposed modification is extremely intuitive—adding a proper scoring rule (Brier score) term to the RL objective—and is supported by clean theoretical analysis showing that it jointly optimizes accuracy and calibration.\n\n- Furthermore, the analysis of verbalized confidence self-consistency (examining both intra- and inter-solution coherence) meaningfully connects calibration to reasoning coherence.", "weaknesses": "1. **(Minor) Inconsistency in the definition of proper scoring rules.**\n    - Equation (4) defines a proper scoring rule as one whose expected value is minimized.\n    - However, the examples mix loss and reward conventions:\n        - The Brier score (Eq. 6) is defined as a loss (minimized).\n        - The logarithmic (Eq. 5) and spherical (Eq. 7) scores are utility functions (maximized).\n    - The statement that “all these scores... are maximized” (line 144) contradicts both Eq. (4) and the Brier example. \n    - This confusion between “loss” and “score” formulations creates unnecessary ambiguity and should be clarified for consistency.\n        \n2. **Lack of ablation on the necessity of “confidence analysis.”**\n    - The paper claims that *chain-of-thought reasoning about uncertainty* improves calibration. However, this structured reasoning step (the `<analysis>` tag) adds considerable inference and training cost compared to simply predicting a scalar confidence value.\n    - The results suggest that SFT+RLCR achieves the lowest ECE but also suffers from a substantial accuracy drop, attributed to catastrophic forgetting.\n    - Therefore, it remains unclear whether the observed calibration improvement stems from enhanced uncertainty reasoning or from task-specific overfitting introduced by SFT.\n    - Critically, the paper does not include RLCR results without confidence analysis (i.e., without the `<analysis>` tag), making it difficult to isolate the contribution of verbalized uncertainty reasoning to calibration performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2G0LQ9CqE", "reviewer": "Reviewer_xB9Y", "summary": "This paper proposes Reinforcement Learning with Calibration Rewards (RLCR), which improves the reward function of RLVR by combining with a term based on the Brier score over verbalized confidence estimates through reasoning, incentivizing both accuracy and calibration. Empirical evaluations across multiple QA benchmarks in factual, math, and long-form reasoning confirm its effectiveness in achieving better calibration while maintaining or improving accuracy, compared with standard RLVR and RLVR with additional classifier or probe. The paper also demonstrates that RLCR improves the self-consistency of confidence and the verbalized confidence scores of RLCR can be incorporated into test-time scaling to yield improvements in both accuracy and calibration.", "questions": "- How sensitive are the results to the reasoning template (e.g. the analysis section, ordering)?\n- For SFT warmup in Math, why not use Deepseek-R1 to output confidence scores?\n- For GRPO, why remove the KL regularization?\n- Are there other working scoring rules suitable for capturing uncertainty, besides Brier score?\n- Is it possible to weigh the correctness and the calibration reward terms differently, and explore the tradeoff?\n- Does this method show potential when there is no clear ground-truth?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The method is simple and intuitive, and shows potential and effectiveness in improving both task accuracy and calibration.\n* The reward design and the choice of score rule are well-motivated and theoretically justified.\n* Uncertainty-aware reasoning and RL training for calibration is a timely and relatively under-explored topic.", "weaknesses": "- The paper should make it clearer the contributions and benefits compared with previous RL-based calibration methods cited in related works [1,2,3]. Given the overlap in scope, it would be very useful to include these as baselines potentially stronger than the RLVR variants. Also, given the recent debate on the contamination issue of Qwen-family models in RLVR, it would be helpful to test on other models as well.\n- It would be good to isolate / disentangle the role of the Brier term and uncertainty-style CoT reasoning in improving calibration in both in-distribution and out-of-distribution settings. Is the gain mostly coming from the reward formulation (unique contribution of this paper) or simply a better confidence estimate from uncertainty reasoning?\n- There is still much room to improve for the OOD settings. Given that SFT warmup is also widely-adopted, the degraded generalization performance of SFT+RLCR is a bit concerning. More insights and analyses on this would be very helpful.\n\n---\n\n[1]  Taming overconfidence in LLMs: Reward calibration in RLHF\n\n[2]. LACIE: Listener-aware finetuning for calibration in large language models.\n\n[3]. Sayself: Teaching llms to express confidence with self-reflective rationales", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1slezGVif", "reviewer": "Reviewer_bCz3", "summary": "This paper introduces RLCR (Reinforcement Learning with Calibration Rewards), which augments standard binary-reward RL for reasoning LMs with a Brier score term to encourage calibrated confidence. Unlike conventional RL that promotes overconfidence, RLCR jointly optimizes accuracy and uncertainty estimation. The authors also provide some theoretical results showing that RLCR does not induce the model to generate uncertain and wrong answers. Experiments with Qwen2.5-7B on reasoning benchmarks show that RLCR maintains accuracy while improving calibration.", "questions": "See above section", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths\n\n- The paper is well written and easy to follow, with clear organization and motivation.\n\n- Although using proper scoring rules for calibration is a well-established idea, the work’s contribution lies in proposing an RL objective that maintains accuracy while improving calibration in reasoning LMs, which is still an important direction to explore.\n\n- The authors provide comprehensive empirical validation, showing that the proposed method consistently yields better-calibrated reasoning models without degrading accuracy across diverse tasks.", "weaknesses": "Weaknesses and Questions\n\n- I would like to see a deeper analysis of the confidence distribution. For instance, are the confidence values truly diverse across different inputs, or do they mostly cluster around a narrow range regardless of the question type? Understanding this would clarify whether the observed calibration improvements reflect genuine uncertainty modeling or simply uniform confidence outputs near accuracy.\n\n- While the paper theoretically explains why adding NLL instead of the Brier score breaks the correctness incentive, it would be informative to include an empirical comparison showing whether this theoretical difference actually affects performance or calibration in practice by conducting experiments with objectives that include NLL.\n\n- The reported AUROC scores (around 0.5–0.6) seem relatively low; it’s unclear whether such values reflect meaningful discrimination ability. Some qualitative analysis or normalized baselines could help interpret these numbers more convincingly.\n\n- It would be useful to examine how RLCR behaves in larger-scale or instruction-following settings, beyond the QA and math reasoning benchmarks. This would test whether the proposed calibration benefit generalizes to more complex reasoning domains.\n\n- I’m also curious how the model behaves when generating multiple samples for the same question—does it assign noticeably different confidence scores to different (possibly contradictory) answers, or are the confidences largely similar across samples? This would give insight into how well the model captures epistemic uncertainty across reasoning trajectories.\n\n- In Figure 4(a), the standard deviation of confidence across reasoning chains appears to be roughly 0.05–0.1, meaning that on a 0–100 scale, confidence typically fluctuates by 5–10 points, or up to 10–20 points across extremes. Doesn’t this suggest that the model’s confidence is still quite unstable across samples, rather than genuinely consistent? Some further interpretation of this behavior would help clarify the robustness of the claimed self-consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)