Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models
Authors: David Bani-Harouni
Chantal Pellegrini
Paul Stangel
Ege Özsoy
Kamilia Zaripova
Nassir Navab
Matthias Keicher
Venue: ICLR 2026 Poster (2026)
Keywords: Confidence Calibration
Uncertainty Estimation
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose Rewarding Doubt, an RL-based approach that models confidence estimation as a betting game, optimizing LLMs for calibrated confidence in factual answers.

Abstract: A safe and trustworthy use of Large Language Models (LLMs) requires an accurate expression of confidence in their answers. We propose a novel Reinforcement Learning approach that allows to directly fine-tune LLMs to express calibrated confidence estimates alongside their answers to factual questions. Our method optimizes a reward based on the logarithmic scoring rule, explicitly penalizing both over- and under-confidence. This encourages the model to align its confidence estimates with the actual predictive accuracy. The optimal policy under our reward design would result in perfectly calibrated confidence expressions. Unlike prior approaches that decouple confidence estimation from response generation, our method integrates confidence calibration seamlessly into the generative process of the LLM. Empirically, we demonstrate that models trained with our approach exhibit substantially improved calibration and generalize to unseen tasks without further fine-tuning, suggesting the emergence of general confidence awareness. Our code is available at https://github.com/pasta99/RewardingDoubt.

Published as a conference paper at ICLR 2026
REWARDING DOUBT: A REINFORCEMENT LEARNING APPROACH TO CALIBRATED CONFIDENCE EXPRES- SION OF LARGE LANGUAGE MODELS
David Bani-Harounj12- Chantal Pellegrini
Paul Stangel!
Ege Özsoy
Kamilia Zaripova'.
Nassir Navab
Matthias Keicher
'Technical University of Munich
Munich Center for Machine Learning david. wwabaaniiaae -hnnuni, chantall.elleggee..
de
ABSTRACT
A safe and trustworthy of Large Language Models (LLMs) requires accu- rate expression confidence in their answers. We propose novel Reinforcement Learning approach that allows directly fine-tune LLMs to express calibrated confidence estimates alongside their answers to factual questions. Our method op- timizes reward based on the logarithmic scoring rule, explicitly penalizing both over- and under-confidenee This encourages the model to align its confidence es- timates with the actual predictive accuracy. The optimal policy under our reward design would result in perfectly calibrated confidence expressions. Unlike prior approaches that decouple confidence estimation from response generation, our method integrates confidence calibration seamlessly into the generative process of the LLM. Empirically, we demonstrate that models trained with our approach ex- hibit substantially improved calibration and generalize to unseen tasks without fur- ther fine-tuning seee-tuning, suggesting the emergence of general confidence awareness. Our code is available at hulhttps:/// /tttps://github. com http.//asstttt//eeaaaaaggggooooo
INTRODUCTION
In human intelligence and inter-hummat interaction, the ability to understand our own uncertainty and communicate our doubts to others is fundamental for effective decision-making collaboration, and learning (Cosmides & Tooby 1996 Xiong et al., 2024). Similarly, for Large Language Models (LLMs) to be safely used in real-worid applications, especially when humans and AI systems work together, they must not only generate accurate information but also communicate their confidence that information. While L.LMs have demonstrated impressive capabilities in natural language un- derstanding, question answering and text summarization (Touvron et al. 2023 Chiang et al.) 20233 Achiam al., 2023), LLMs still face significant limitations, such as their tendency to generate inac- curate information, often referred to hallucinations (Hadi et al. 2023). This raises concerns about their reliability, particularly in real-world applications where trustworthiness is essential. Especially in high- high-stakes enviromments such medical diagnosis. where LLMs starting to become support tools for professionals (Moor et al. 2023 Pellegrini et al, 2025 Tu et al. 2024 Bani- et al.) 2024], overconfident predictions including factual errors hal lulunatiionns could have serious sequences for patient health. Also, in customer service or legal consultation (Shi et al.) 2024 Sun al. 2024). LLMs need express uncertainty and defer complex queries to human representatives when unsure to avoid misinformed decisions. Reliable confidence estimation and expression would enable these systems flag uncertain outputs for human review, ensuring that crucial decisions not made based on uncertain LLM outputs. To allow risk estimation while using LLM ILM-generated output, model confidence should be calibrated, meaning that the expressed numerical confidence should be equal to the probability of the model's 's answer being correct.
"Equal contribution

---

Published as a conference paper at ICLR 2026
What is capital city France? Provide an answer and confidence!
High confidence
Medium confidence
Low confidence
Answer: Paris
Answer: Paris
Answer: Paris
Correct
Confidence: 10
Confidence:
Confidence: 0
answer
Answer: Lyon
Answer: Lyon
Answer: Lyon
Incorrect
Confidence: 10
Confidence:
Confidence: 0
answer
cC
What confidence maximises reward, considering possible punishment?
Figure Illustration of our reward design: The model is rewarded for high confidence if it is correct and punished if it is incorrect. To maximize the reward, the model needs to learn when to predict higher lower confidence, considering a possible higher punishment. Our reward function is designed so that the best reward is given when the confidence and the epistemic probability of being correct are the same, thus incentivizing the expression of calibrated confidences.
Many previous methods for confidence estimation lack in calibration performance as they do not train the model and instead infer the confidence from the internal state in zero-shot setup (Huang et at 2023 Kuhn et al. [2023] Duan et al., (0044). Additionally. this does not give models an inherent awareness of confidence. Other trained methods in this area decouple the uncertainty estimation
from the text generation process (Azaria & Mitchell, [2023] Kapoor et al. 2024). This approach optimizes for calibrated confidence estimation but does not enable the uncertainty-awer awareeesss and expression in the model itself.
Targeting these limitations, we propose a novel cewtioorreemet-eeeammin (RL) approach for teaching L.LMs to express their calibrated confidence, encouraging granular, accurate estimation of the con- fidence level the training objective. For this, we model confidence estimation as a betting game: high-confidence answer would warrant larger bet, reflecting a strong belief in its correctness, while a lower confidence score would suggest caution. Central to our method a reward function based on the logarithmic scoring rule, strictly proper scoring rule. We are the first to optimize this func- weantt-eeaannnn--aeeeee based policy optimization, leveraging its calibration properties for directly and seamlessly training confidence calibration in LLM generations. This reward func- tion captures the fundamental risk-reward tradeoff in probabilistic decision-making, as illustrated in Figure It increases the reward when correct answer is given with high confidence, simulating the higher potential return of big bets. Conversely, it penalizes incorrect answers more when they made with high confidence, discouraging overconfidence. This ensures that both uncertainty and confidence are appropriately factored into the reward system. As a proper scoring rule, optimizing the reward function trains the model align its predicted confidence with the accuracy of its output, encouraging granular and calibrated confidence scoring. A calibrated confidence estimation will provably result in the highest reward during training. This not only improves the trustworthiness of LLMs in collaborative Auman-AI scenarios but also helps users better assess when Al tools should be trusted, checked, or deferred to human expertise.
tion through reinforcement-learming
RELATED WORKS
2.1 CONFIDENCE ESTIMATION IN LLMS
Confidence estimation and calibration have long history in machine learning and natural language processing (Wang 2024). With the rise of L.Mss, research has focused on adapting and extending these ideas to modern architectures. Broadly, methods fall into black-box and white-box approaches Geng et al. 2024).
Black-box methods Black-box methods rely only on model outputs. Linguistic prompting meth- ods ask the model to verbalize its confidence, sometimes aided by of-tn-of-thought reasoning (Xiong al. 2024 Wei et al. 2022). Consistency-based approaches estimate confidence by mea-
2

---

Published as a conference paper at ICLR 2026
suring agreement across multiple generations. with high variance indicating uncertainty (Manakul lt al. 2023 Wang et al., 2022). Recently, [Zhou et al. (2025) proposed SteerConf, which does multiple inference passes where the LLM is prompted to use different levels of caution in its fidence expression. The resulting verbalized confidences are aggregated based on confidence and answer consistency
overall confidence prediction. Black-box methods are valuable for their simplicity, ease-of-use and universality, however generally lack behind white-box methods in their calibration performance.
White-box methods White-box methods exploit internal model states. Logit-based techniques estimate confidence from token probabilities entropy (Huang et al.] 2023] Kuhn al. 2023 Duan et al. 2024), assuming that high probability tokens correspond to high confidence predictions. Self-evaluation methods let the model judge the truth of its own answers (Kadavath et al. 2022 They prompt the model to provide an answer followed by a judgment whether its own answer is "true" or "false". They then compare the probability of the "true" "false" token to calculate a confidence estimation. External probing approaches train classifiers on hidden states to predict correctness (Azaria & Mitchell 2023). While some of these methods achieve good confidence estimation results, they do not teach the model to express clear confidence values itself but depend on some auxiliary estimation mechanism.
2.2 FINETUNED CONFIDENCE EXPRESSION
A growing line of work integrates confidence estimation into instruction tuning. These methods typically follow two-step paradigm: First, they estimate model confidence using various methods, e.g., self-consistency (Cheng al. [2024] Yang al. 2024 Han al. 2024), token probabilities
(Chen et al. 2024). trained probes (Mielke et al. [2022] empirical accuracy (Zhang et al. 2024 Lin
ct al. 2022 Ulmer et al.] (024), or topic unfamiliarity (Wa et al., [2244] Kang et (004). Second,
they construct finetuning datasets that either replace uncertain answers with refusals (Zhang et al.)
2024
Cheng et al. 2024: Yang et al. all,2024 2024 Wan et al. 2024) or append the estimated uncertainty as
an additional supervised signal (Han et al. 2024 Chen et al. 2024 Mielke et al. 2022 Lin et al.)
2022
Ulmer et al. 2024).
The key limitation of this approach is that the model's expressed confidence is bounded by the quality of the constructed ground-truth estimates. Additionally, while the underlying confidence estimation method might optimize for perfect calibration (e.g. in the case of the trained probe), this theoretical guarantee is lost when performing supervised finetuning on these constructed ground truths to reproduce these scores.
2.3 REINFORCEMENT LEARNING FOR CONFIDENCE EXPRESSION
Reinforcement Learning from Human Feedback (RLHF) has proven effective for aligning LLMs with human preferences (Ouyang et al., 2022), 2, and has also been explored for agentic interaction in textual environments (Zhou et al. [023] Carta al. 2023). Only recently have researchers be- gun applying RL directly to confidence estimation. Tao et al. (2024) adapt RLHF by designing rewards that align verbalized confidence with preference ratings, but this requires human-annotated preference data and does not address factual calibration. Leng et al. (2024) identify that standard reward models in RLHF are biased toward high verbalized confidence, rating answers with high confidence expressions with a high reward. To counteract this, they introduce two reward model training paradigms, PPO-M and PPO-C, which fine-tune the reward model to reward answers where correctness and confidence expression are aligned. [xu et al. (2024) propose RI. from Knowledge Feedback (RIKF) to encourage refusals outside the model's 's knowledge scope, reducing hallucina- tions but without quantifying confidence. (Stengel-Eskin et al. (2024) propose LACIE, a DPO-based approach that simulates an interaction between a speaker and a listener model, rewarding accurate and honest confidence expression by aligning it with the listener's interpretation of confidence cues rather than with fact- based numerical calibration.
In contrast to previous works, method directly optimizes for factual calibration using a theo- retically grounded, proper scoring rule as the reward signal, enabling the model to develop intrin- sic uncertainty awareness without requiring external preference models, knowledge supervision,
3

---

Published as a conference paper at ICLR 2026
confidence
CONFAJONCE
contoence
question
judge
reward function
reward
Figure 2: Overview of our reinforcement learning framework: The LLM is prompted answer question and provide the confidence in this answer. The answer is checked for correctness by a judge function and the reward is calculated based on the correctness and the confidence. Correct answers with high confidences rewarded highly, but also penalized heavily when incorrect.
post-hoc calibration techniques, while the same time seamlessly integrating calibrated confidence expression into the LLMs response generation.
3 REWARDING DOUBT
We propose a novel cenfforcemen--learmin approach, that improves an LLM's ability to verbalize an accurate numerical confidence in previously generated answer. The L.M functions as agent in a simulated environment as shown in Figure[2] that poses challenging question-answering scenarios. prompted with task queries such factual questions and asked predict both an answer to the query as well as a confidence score. Based on the correctness of the answer, and the expressed confidence, we reward the model, incentivizing it to express a calibrated confidence.
Formally, let the model be provided with textual question or request 9. resulting in an answer- confidence pair (a. as response, where is a textual answer with binary correctness value, and 0 S < 1 is numerical confidence score representing the subjective probability the model assigns to answer a being correct. We train this subjective probability assessment align with the true epistemic probability p", which represents the actual likelihood of correctness given the model's s internal knowledge state. If ý and p" are aligned the model perfectly calibrated, meaning the probability of correctness P(j(a) = 1) always equals the expressed confidence:
P(j(a)
5 = x) Ir Va [0, 1],
where j(-) is correctness judging function that is answer a is correct, and otherwise.
The true epistemic probability y" is not directly observable, thus supervised learning of calibration only possible by constructing an artificial ground truth approximate p" Instead, we model this task Markov Decision Process (MDP) defined by the tuple (S, A, T, R), where the model learns to generate calibrated confidence scores through reinforcement learning. The MDP defined by the following components:
State space (S): A state & E S consists of a natural language question g, the model's predicted answer a, and the partial sequence of confidence tokens predicted so far, if any- That is, = (4, ct.r-1), where C represents the previously generated confidence score tokens.
Action space (A):: The action space consists of selecting the next token in the confidence estimation process from the LLM vocabulary, including numerical tokens (e.g., represent- ing percentages probability values) and special end-of- of--suen-eucce token that finalizes the prediction.
Transition function (T(s++1 st, a))): The environment transitions deterministically based on the language model's autoregressive token generation process. Given state
s, (q. a, c1+-1) and an action e,, the next state is defined as $t+1 (g. a, c1+). Once the end-of-sequence token is generated, the episode terminates.
Reward function (R): The reward R(a, c, is computed based on the final confidence score sequence c (ct,..-, Cr) and the correctness of the answer j(a).
4

---

Published as a conference paper at ICLR 2026
1.0
Normalized reward
0.5
-0.5
Reward for correct answer
Reward for incorrect answer
-1.0
2
3
8
Confidence
Figure 3: The rewards for each confidence value for correct and incorrect answers. The closer the confidence is to ten 7ero, respectively, the higher is the reward. At the same time, the possible punishment increases to a greater extent. The model has to learn when the trade-off between those two possibilities is worthwhile.
To promote accurate confidence estimation, the model's expected reward must fulfill the requirement of being maximized when p", i.e. when the predicted confidence aligns with the probability of correctness. The model should receive high reward when correctly predicts an outcome with high confidence or when it incorrectyy predicts an outcome with low confidence. Conversely, the reward should be low when incorrect predictions are provided with high confidence or correct predictions are provided with low confidence. This approach incentivizes the model to express high confidence only in cases where certainty is warranted while expressing doubt in ambiguous situations. By penalizing both overconfidence and underconfidence, the model is encouraged to calibrate its confidence accurately, effectively balancing the trade-off between reward maximization and penalty avoidance. Note, that through this design our method focuses exclusively improving calibration while keeping task performance stable.
We design our reward logarithmic scoring function:
(log(p), R(a,p.5) (log(1 p). if {(a) (incorrect)
if ja) = (correct)
(1)
This function fulfills the requirement described above as we show in the following proposition:
Proposition (Optimality implies Calibration). The expected reward R[ P.j) maximized for each sample when p = p" and the optimal policy under the reward design is thus perfectly calibrated.
The proof of Proposition ! is analogous the proof that the logarithmic scoring rule is a proper scoring rule. We provide it in full in Appendix B and discuss the influence of the clipping on the optimality of the reward function.
Since the logarithm of zero is undefined, we introduce small positive constant e as clipping value for numerical stability. Concretely, we clip the lower and upper limit of the confidence f to e and 1 respectively. The clipped reward function is provided in Appendix C The normalized and clipped reward for correct and incorrect answers for each confidence visualized Figure
4 EXPERIMENTAL SETUP
We evaluate our method in both Single-Answer and Multiple-Answer settings. We prompt the model to provide a confidence for each answer as an integer between and 10, which we normalize for the reward calculation. A confidence of zero is defined the model being certain that the answer incorrect, while ten is defined as the model being certain the answer is correct. We normalize the reward function to the range of [-1,1].
5

---

Published as a conference paper at ICLR 2026
In the Single- Answer setting train the model on the TriviaQA dataset (Joshi al. 2017), which contains question- answer-evidence triplets, from which we only use the questions and answers. For generalization experiments, evaluate our method on CommonsenseQAA (Talmor et al. 2019) and MedQA (Jin et al.) 2020), which are multiple-choice question datasets in the commonsense and medical domain, respectively. For the Multiple-Answer setting, train the QAMPARI dataset (Amouyal et al. 2023), which contains questions with multiple-answers as well as evidence, again only using the questions and answers.
In the Single-Answer setting we compare our approach on the TriviaQA dataset against the following (Xiong et al., 2024). Top-K K (Tian et al., 2023), Surrogate Token (Ka- (Wang et al. 2022] as zero-shot methods, LACIEE (Stengel-Eskin et al.J 2024). which uses DPO for optimizing confidence expression and Trained Probe (Azaria & Mitchell| (023), which employs supervised training of an external probe for estimation model. We also compare to the non-finetuned base model in a zero-shot manner, using the same prompt our Rewarding Doubt method and refer to this setup as Verbalize. In the Multiple-Answer setting we compare to Trained Probe and Se- quence probability, those methods are the best performing zero-shot and trained baselines in the Single-Answer setting. LACIE does not report results for this dataset, thus we can only compare on TriviaQA.
methods:
davath et al.) (022), Sequence Probability (Huang et al., (223) and Self-
We report our results using the Expected Calibration Error (ECE) and the Area Under the Receiver Operating Characteristic Curve (AUROC) metrie. Additionally, visualize the calibration with calibration curves, where a well-calibrated model lies close to the 45 line and large deviations show high miscalibration.
Response Generation To calibrate and reward the model only on the confidences and not the answers we separate generation in two steps during training: Answer and confidence generation. Answers are generated first and afterwards treated fixed inputs alongside the question, while the confidence is generated in separate generation step and considered as sole target for optimization. Like this, we ensure that answer generation is disentangled from the optimization process, ensuring the answer correctness is not affected by our confidence calibration training.
Correctness Assessment For the multiple-choice datasets MedQA and Commonsense we evaluate correctness using the exact string matching between the model's response and the ground truth answer. For the TriviaQA and QAMPARI datasets, we use the Fl score of word overlap to measure the similarity between the model's response and the ground truth candidates. The FI score is calculated for each candidate and the maximum score is considered the final score. We consider an answer as correct its score exceeds a threshold of 0.5.
Implementation Details We optimize the reward function using the Proximal Policy Optimiza- tion (PPO) algorithm (Schulman et al., 2017). Unless stated otherwise, we use Meta-Llamat Instruct (Grattafiori et al.) (202)) as base model for our experiments. We employ the 4-bit quantized performance-optin -ottiiiie model version by Unsloth Al (Han et al.) 2023) and apply LoRA fine-tuning (Hu et al. 2022). For the Single-Answer setting we train the model for two epochs with a learning rate of le-5. For the Multiple-Answer setting, due to the size of the training dataset and the fact that each question yields multiple facts, the model is trained for a limited amount 24,000 steps with a batchsize of eight and learning rate of le-5 and multiply the reward with 5 to increase its spread. All models are trained on one Nvidia A40 with each training run taking seven days. On average the model generated approximately 3.4 answers per fact. If the model fails to generate answer in the specified format, it is penalized with an out-of-format reward of -3. Detailed implementation choices for the baselines are provided in Appendix D
RESULTS AND DISCUSSION
This section presents and discusses the key findings of our experiments for both Single and Multiple- Answer tasks and the generalization to oft-of-domain datasets.
To assess how well our approach improves calibration, we compare against the zero-shot LLM baseline (Verbalize) and several established methods in both Single-Answer and Multiple-Answer

---

Published as a conference paper at ICLR 2026
Table Comparison of methods on the TriviaQA dataset in the Single- Answer setting with 95% CIs
in brackets. Results are from the original paper (Stengel-Eskin et al. 2024 and include standard
error.
Method
ECE (U)
AUROC (1)
Accuracy (1)
Verbalize
0.3459 (0.337500.5433
0.5858 [0.5778.0.5936]
0.6310 [0.6222,0639]]]
Chain-of-Thought
0.3065 [0.2981.0.315]]
0.6379 [0.628400.6455]
0.6273 [0.6181,06663]]
Top-K
0.1611 I [0.1529,0.1698
0.6673 [0.658000.6668]
0.6023 [0.5936.0.611]]
Surrogate Token
0.3686 [0.3595,0.3783]
0.5923 [0.5818,0.6027]
0.5933 [0.5844.0.016|
Sequence Probability
0.3156 [0.3074.0.3237]
0.7804 (0.7725,0.7876]
0.5955 [0.5864.0.604]]
Self-Consistency
0. 134 [0. (0.1066.0.122
0.8213 (0.8129,0.8298)
0.6224 [0.6131,0.637]]
PPO-M
0.3262 [0.3173,0.3366]
0,5274 [0.5227,0.5319)
0.5749 [0.5662,0.5835]
PPO-C
0,3607 (0.3524.0.3697)
0.5439 (0.5384.0.5491)
0.5258 [0.5164.0.5358]
LACIE*
0. 1200 =0.02
0.7200 #0.02
n/a
Trained Probe
0.0189 [0.0147,0002755]
0.8173 00.8099.088200
0.5925
[0.5834.0.6017]
Rewarding Doubt (ours)
0.0226 (0.0176,000302]
0.8592
[0.8523,0.8664]
0.6309 (0.6222,0.6399)
question -anwering tasks. Results for the Single-Answer setting on TriviaQA are presented in Ta- ble [ and those for the Multiple-Answer setting on QAMPARI appear in Table 2 Across both tasks, Rewarding Doubt substantially improves the model's ss confidence calibration over zero-shot verbalization.
In the Single-Answer setting on TriviaQA, Rewarding Doubt achieves an ECE of 0.0226 and an AU- ROC of 0.8592, clearly outperforming all zero-shot baselines as well as LACIE, which is based on DPO-based optimization. The second fine-tuned method, Trained Probe, which relies on supervised fine-tuning, reports a slightly lower ECE (0.0189), both methods achieve near-perfect results. Fur- ther the AUROC of Rewarding Doubt is notably higher, suggesting that although both methods offer strong calibration, Rewarding Doubt better discriminates between correct and incorrect answers. In the Multiple-Answer setting on QAMPARI, Rewarding Doubt also outperforms baselines, achieving an ECE of 0.0816 and AUROC of 0.6947. In comparison, Verbalize, Sequence Probability, and Trained Probe perform notably worse. Our findings support the claim by |Azaria & Mitchell (2023) that a model's internal state encodes information about the truthfulness of statements, which can serve as an indicator of uncertainty. However, without fine-tuning, the model struggles to utilize this internal information effectively. Our approach enables the model make use of this correlation and translate it into an accurate expression of the probability that a given answer is correct.
The calibration curves in Figure 4 further illustrate these improvements. For both TriviaQA and QAMPARI, the fine-tuned model's confidence much more closely aligns with the ideal 45" line than the zero-shot Verbalize baseline. Additionally, we observe shift in the confidence distribution after fine-tuning. As shown in Figure in zero-shot setting the LLM (Verbalize) predominantly assigns high confidence scores (8 or above), reflecting overconfidence. a pattern also noted by Xiong et al. (024),, who attribute it to supervised pretrriinng that favors confident expressions. After fine-tuning with Rewarding Doubt, the model's 's confidence scores (shown in Figure (5b) span a wider range, including lower values, indicating a more nuanced expression of uncertainty. This shift suggests that fine-tuning mitigates overconfidence and better aligns the model's confidence with its actual performance.
Table 2: Comparison of methods the QAMPARI dataset in the Multiple-Answer setting with 95% CIs in brackets.
Method
ECE (L)
AUROC (t)
Accuracy (t)
Verbalize
0,5319 [0.5172.0:5461]
0.6047 (0.58370062667
0.2550 (0.2410.026698]
Sequence probability
0.5324 [0.5225,0.5432]
0.5942 (0.5775.066110]
0.1928 (0.1629,0.202]]
Trained probe
0.1117 [0.0997,0.1222]
0.6481 [0.6241.0.6266]
0.2233 20.2094.0.2384)
Rewarding doubt (ours)
0.0816
[0.0723.000051]
0.6947
[0.6776.0.1133]
0.2480
(0.2348,0.2099]

---

Published as a conference paper at ICLR 2026
1.0-
Ideal
Frequency of correct answers
1.0
Ideal
TriviaQA
TriviaQA
0.8
QAMPARI
0.8
QAMPARI
Frequency correct
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.6
0.8
1.0
0.0
0.4
1.0
Predicted confidence
Predicted confidence
(a) Base model
(b) Fine-tuned by Rewarding Doubt
Figure 4: Calibration curves of the zero-shot base model (Verbalize) and the model fine-tuned by Rewarding Doubt.
Freguency for dataset
0.8
TriviaQA
Frequency for dataset
MedQA
0.20
MedQA CommonsenseQA
Commonsense@A
0.15
0.4
0.2
0.05
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Predicted confidence
Predicted confidence
(a) Base model
(b) tine-tuned by Rewarding Doubt
Figure 5: Histograms of predicted confidences of the zero-shot base model (Verbalize) and the model fine-tuned on the TriviaQA dataset.
To test the consistency of our method across different models, we perform an ablation study across diverse LLM architectures and sizes. Specifically, we apply Rewarding Doubt to Qwen-2.5 (3B and 7B) and Gemma-2 (9B) models, in addition LLaMA-3.1-BB Table 3 reports performance for each model before and after fine-tuning with our method. Despite architectural and pretraining differences, Rewarding Doubt consistently reduces calibration error and improves AUROC across all models, without degrading downstream accuracy.
Stability of Answer Correctness Training confidence calibration with our method only targets uncertainty estimation abilities and does not aim to alter the responses of the model. This achieved by only rewarding the model its expressed confidence, while the answer is generated beforehand independently from the model update step. Our results show a stable accuracy for all experiments without notable differences in accuracy between the base model (Verbalize) and the model adapted with Rewarding Doubt, showing that confidence calibration training with Rewarding Doubt does not affect task performance.
Generalization Capabilities To assess the generalization abilities Rewarding Doubt, we eval- uated the model trained on TriviaQA in out-of-domain settings using the CommonsenseeA (Talmor jet al. 2019) and MedQA (Jin et al. 2020) datasets. Results are shown in Table 4 On MedQA, Rewarding Doubt significantly outperforms Verbalize in both metrics, while CommonsenseQA, achieves comparable ECE, however paired with a much higher AUROC. This discrepancy high- lights limitation of relying solely on ECE for evaluating calibration. ECE does not reflect how
8

---

Published as a conference paper at ICLR 2026
Table 3: Calibration and accuracy of Verbalize vs. Rewarding Doubt across different LLMs with 95% CIs in brackets.
Model
Method
ECE (4)
AUROC (1)
Accuracy (1)
LLaMA-3.1-8B
Verbalize
0.2771 [0.2689.0.8622]
0.6766 [0.6667.0.6863]
0.6662 [0.6577.06645]
Trained probe
0.0152 [0.0118000235]]
0.8495 [0.8420.0.8567]
0.6231 [0.6143.0.6322]
Rew. Doubt
0.0256 (0.0209.00022]]
0.8793
[0.8729.0.8860]
0.6497
[0.6407,0.6583)
Qwen-2.5-3B
Verbalize
0,5330 [0.5252.0.5435)
0.5981
[0.5927.0.6035]
0.4185 [0.4085,0:4247]
Trained probe
0.0186 [0.0134.0.0268]
0.7975
[0.7880,0.806) 8006)
0.2540 (0.2463,0.2624]
Rew. Doubt
0.1483 [o. 10.1445,0.1144
0.9065
[0.9012,0.9122]
0.4193
[0.4097,0-4283]
Qwen-2.5-7B
Verbalize
0.3619 [0.3530.0.3755]
0,5818
[0.5762.0.5899]
0.5239 [0.5148.0.5331]
Trained probe
0.0989 [0.0920.0. 1057]
0.8737
[0.8676,088797]
0.4793
[0.4696,0.4881]
Rew. Doubt
0.1298 1237,0.1360
0.8928
[0.8866,0.8988]
0.5283
[0.5193.0.5368]
Gemma-2-9B
Verbalize
0.3206 [0.3122,0.3288]
0,5615
[0.5548,0.5682]
0.6690 [0.6603.0.6773]
Trained probe
0.0301 [0.0253,000173]
0.8694
[0.8629,0.8799]
0.6464 [0.6380.0.551]]
Rew. Doubt
0.0922 [0.08)1.000994]
0.8649 [0.8570.08725]
0.6832 [0.6743.0.691]
Table 4: Comparison of generalization results on CommonsenseQA (CsQA) and MedQA, trained on the TriviaQA dataset with 95% CIs in brackets.
Method
ECE (1)
AUROC (1)
Accuracy (t)
MedQA CsQA
Verbalize
0.2820 [0.2206.0.3422]
0.5425 (0.4740.0.6069)
0.6860 (0.6277.0.7444)
Trained Probe
0.4819 [0.4655.0.51300]
0.5374 [0.5021.0.5788]
0.7108 (0.6847.0.7355]
Rewarding doubt (ours)
0.2930 [0.2693.0.3199]
0.6385
(0.6065.0.6715)
0.7163
[0.6918.0.7410]
Verbalize
0,4480 [0.420000047533
0.5075 4803,0.53388
0.5067 [0.4784.0.5350]
Trained Probe
0.2099 [0.1881.0.2439]
0.5513 [0.5207.0.5844)
0.5051 [0.4792.0.5318]
Rewarding doubt (ours)
0.1145 [0.0893.0.1408]
0.6649 [0.6355.0.6944]
0.5161 [0.4586,0.5420]
well a model discriminates between correct and incorrect predictions across different confidence levels. A model consistently assigning moderate confidence values could appear well-calibrated un- der ECE, yet fail to offer meaningful distinetions between uncertain and certain cases. AUROC, in contrast, directly measures this discriminative ability. Thus, the substantial improvements in AU- ROC underscore that Rewarding Doubt produces more useful and actionable confidence estimates. Compared to the Trained Probe, the best-performing baseline, Rewarding Doubt consistently out- performs, showing a stronger ability to generalize to new datasets.
We also explore generalization across experimental settings in Table by applying a model trained in a Single-Answer setting to a Multiple-Answer task. Although under-performing a model trained specifically for that task, it still outperforms the base model considerably, demonstrating transfer- ability of the learned confidence estimation patterns. This suggests promising applications for im- proving confidence estimation in more complex less structured scenarios, such as fact verification and calibration in free-text generation, even when specialized training data is unavailable.
Our current experiments focus on settings where answer quality can be evaluated via exact rule- based metrics, yielding binary correctness signal, the Rewarding Doubt framework could be ex- tended work with correctness signals provided by an LLM ju-aude system, reward model trained on human preferences or continuous NL. metrics.
Overall, our experiments show that Rewarding Doubt provides a robust and efficient way to enhance calibration, while generalizing across tasks, and maintaining stable task performance, making it an effective approach for accurate confidence calibration and expression in LLMs. Beyond improve- ments in calibration, our method also offers practical advantages. While fine-tuning requires an initial training investment, inference remains highly efficient, as only a small, constant number of tokens need to be generated express confidence. In contrast, zero-shot methods like Chain-of- Thought and Self- toff-Consistency have substantial computational overhead during inference by requir- ing lengthy reasoning chains multiple generations. Rewarding Doubt introduces such over-

---

Published as a conference paper at ICLR 2026
Table 5: Comparison of the base and fine-tuned model on the Qampari dataset in different settings with 95% Cls in brackets.
Training
Evaluation
ECE (4)
AUROC (1)
Base model
Single fact
0.5875
[0.5597,0665]]
0.5787 [0.5408,0.6125]
Single fact
Single fact
0.1536
[0. [0.1320001813]
0.7240
(0.6889,0.7577]
Base model
Multi fact
0.5319
[0.5172,0.561]]
0.6047 [0.5837,0.6267]
Single fact
Multi fact
0.1777 (0.1679.0.1890)
0.6617
[0.6468.0.6779)
Multi fact
Multi fact
0.1061 [0.0935,0.1 1206]
0.7268
[0. 77065,077468
head, does not rely on an additional model, and directly provides actionable confidence estimates through verbalization directly by the LLM, making it highly suitable for real-world deployment.
Limitations Due to computational constraints, we only tested Rewarding Doubt on models with sizes ranging from 3B to 9B parameters. While we expect similar effectiveness on larger models, empirical validation on such scales would valuable.
CONCLUSION
In this work, we propose Rewarding Doubt, novel approach that enables LLMs to express confi- dence in their answers more accurately using natural language. We leverage reinforcement learning with reward function based on the logarithmic scoring rule that incentivizes well-calibrated confi- dence expressions. Fine-tuning with our method significantly improves the model's ability to esti- mate a calibrated confidence, effectively reducing the overconfidence patterns commonly observed in LLMs. This not only enhances the trustworthiness in Al-generated responses but also lays the groundwork for more reliable human-Al collaboration, where models can transparently communi- cate uncertainty, essential step toward safer and more accountable AI systems.
ACKNOWLEDGEMENTS
The authors gratefully acknowledge the financial support by the Bavarian Ministry of Economic Affairs, Regional Development and Energy (StMWi) under project ThoraXAI (DIK-2302-0002), and the German Research Foundation (DFG, grant 469106425- NA 620/51-1).
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility, we describe implementation details of Rewarding Doubt as well as the used baselines in Section 4 and Appendix Further, Appendix providees the exact prompts used for different experiments. Lastly, we included our code in the submission and will publish it upon acceptance.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv, 2023.
Samuel Joseph Amouyal, Tomer Wolfson, Ohad Rubin, Ori Yoran, Jonathan Herzig, and Jonathan Berant. Qampari: An open-domain question answering benchmark for questions with many an- swers from multiple paragraphs, 2023. URL https: aaaiiiiiivv/////////aaaaaa/aaava wttpk//.or//bb///200000 12665
Amos Azaria and Tom Mitchell. The internal state of an Ilm knows when it's lying, 2023. URL https://arxivv www.loorgggree.. http://bb//33.....
10

---

Published as a conference paper at ICLR 2026
David Hari-Harouni, Nassir Navab, and Matthias Keicher. Magda: Multi-agent guiieline-driven diagnostic assistance. In International workshop on foundation models for general medical Al, pp. 172. Springer, 2024.
Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. International Conference on Machine Learning, pp. 3676-3713. PMLR, 2023.
Lida Chen, Zujie Liang, Xintao Wang, Jiaqing Liang, Yanghua Xiao, Feng Wei, Jinglei Chen, Zhenghong Hao, Bing Han, and Wei Wang. Teaching large language models express knowl- edge boundary from their own signals. arXiv preprini arXiv:2406. 10881, 2024.
Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang. Zhangyue Yin, Shimin Li, Linyang Li, Zhengfu He, Kai Chen, and Xipeng Qiu. Can ai assistants know what they don't 't know? arXiv preprint urXiv:2401.13211 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
Leda Cosmides and John Tooby. Are humans good intuitive statisticians after all? rethinking some conclusions from the literature on judgment under uncertainty. cognition, 58(1):1-77 1996.
Jinhao Duan, Hao Cheng. Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5050-5063, 2024.
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration large language models, 2024. URL https:
7/
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al. The llama 3 herd of models, 2024. URLhttts://aart /ttp///xxwwv..rr/aaa/a//7/1/1
21783
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Daniel Han, Michael Han, et al. Unsloth. 2023. URL http:://ittubb..cmmmms..... unsloth
Haixia Han, Tingyun Li, Shisong Chen, Jie Shi, Chengyu Du, Yanghua Xiao, Jiaqing Liang, and Xin Lin. Enhancing confidence expression in large language models through learning from past experience. arXiv preprint arXiv:2404. 10315, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference on Learning Representations, 2022.
Yuheng Huang, Jiayang Song, Zhijie Wang. Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. Look before you leap: An exploratory study uncertainty measurement for large language models, 2023. URL ptttpp::/aaxiv.orgge abs 2307 10236
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What dis- ease does this patient have? large-scale open domain question answering dataset from medical exams, 2020. URL /tRp:ttts////axi.orr//
13081
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics, Vancouver, Canada, July 2017. Association for Computational Linguistics.
11

---

Published as a conference paper at ICLR 2026
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson. Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know, 2022. URL httts:://aaaat org/ abs/2207. 05221
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate. arXiv preprint arXiv:2403.05612 2024.
Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. Calibration-tuningg Teaching large language models to know what they don't know. In Raál Vázquez, Hande Celikkanat, Dennis Ulmer, Jörg Tiedemann, Swabha Swayamdipta, Wilker Aziz, Barbara Plank, Joris Baan, and Marie-Catherine de Marneffe (eds.), Proceedings of the Ist Workshop on Uncertainty-Aware NLP (UncertaiNLP 2024), pp. 1-14, St Julians, Malta, March 2024. Association for Computational Linguisties. URL hctps: www.eraiinppl1..1..
//aclanchology.ore
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664, 2023.
Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfiidece in llms:
Reward calibration in rlhf. ArXiv, abs/2410.09724, 2024. URL URL/httsss//arxvv..rr////.
2410. 09724
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. 14334, 2022.
Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallu- cination detection for generative large language models. arXiv preprint arXiv:2303.08896, 2023.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022.
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelli- gence. Nature, 616(7956):259-2666 2023.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Chantal Pellegrini, Ege Özsoy, Benjamin Busam, Benedikt Wiestler, Nassir Navab, and Matthias Keicher. Radialog: Large lision-language models for x-ray reporting and dialog-drive assis- tance. In Medical Imaging with Deep Learning, 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (tt::::777.66444 2017.
Jingzhe Shi, Jialuo Li, Qinwei Ma, Zaiwen Yang. Huan Ma, and Lei Li. Chops: Chat with cus- tomer profile systems for customer service with 1lms, 2024. URL https://arxiv.org/
abs/2404.0134 01343
Elias Stengel-Fskin, Peter Hase, and Mohit Bansal. Lacie: Listener-aware finetuning for confidence calibration in large language models. arXiv preprint arXi:: 1028, 2024.
Jingyun Sun, Chengxiao Dai, Zhongze Luo, Yangbo Chang, and Yang Li. Lawluo: A multi-
agent collaborative framework for multi-round chinese legal consultation, 2024. URL https:
//arxiv..rg/abbaee 16252
12

---

Published as a conference paper at ICLR 2026
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseCQ:: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 4149-4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. N..18633/11/1191444 1421, URL
1421
Shuchang Tao. Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. When trust Ilms: Aligning confidence with response quality. arXiv preprint arXiv:2404. 17887, 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.14975. 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv, 2023.
Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, al. Towards generalist biomedical ai. NEJM AI, 1(3):A1oa2300138, 2024.
Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh. Calibrating large language models using their generations only. arXiv preprint arXiv:2403.05977 05973, 2024.
Fanqi Wan, Xinting Huang. Leyang Cui, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge verification to nip hallucination in the bud. arXiv preprint arXiv:2401. 2024.
Cheng Wang. Calibrationn in deep learning: survey of the state-of-the-art, 2024. URL https:
//axxiv.org/abs.. a20/aa//////../ 00.3388.12200 01222
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 11171, 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-thouuht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24822 2022.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms, 2024. URL
htttps://arxiv. org/abs///////oo// /23/a//s.///...../ 13063
Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, and Kai Yu. Rejection im- proves reliability: Training ilms to refuse unknown questions using rl from knowledge feedback, 2024. PRRLhttps:/// arxiv. orq/
Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty. Advances in Neural Information Processing Systems, 37:63565-63598 2024.
Hamning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say 'i don't 't know'. In Proceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 7106-7132, 2024.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluat- ing text generation with bert. arXiv preprint arXiv: 190409994 2019.
Wei Zhou, Xiangyu Peng, and Mark Riedl. Dialogue shaping: Empowering agents through npc interaction. arXiv preprint arXiv:2307. /5833, 2023.
Ziang Zhou, Tianyuan Jin, Jieming Shi, and Qing Li. Stecrconf: Steering 1lms for confidence elicitation. arXiv preprint urXiv:2503.02860 2025.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROMPTS
For all the question-answering settings, the model is directly prompted to answer question without preceding example or context. For our method the model was prompted to answer the question and additionally provide verbalized confidence. For the other baselines that do not need verbalized confidence but infer it indirectly, the model is prompted to only give the correct answer. The specifics for multiple-choice are slightly changed but hold mostly the same meaning. The exact prompts for each method can be questions. The prompts for each Multi-Answer method can be seen in Table We decided not to give the model a role like "expert" in the system prompt but keep it neutral, role we give the model affects the verbalized confidence.
in Table for open questions and Table 7 for multiple-choice
observed that the
Table The prompts for each method for the open short form questions.
Method Verbalize/ Re-
Prompt
You will get questions. Answer with the correct answer. Additionally pro- vide confidence between O, I, 2, 3, 4, 5, 6, 7, 8, 9, 10, of how sure you are the answer is correct. A value close to 0 means you think there is high probability that the answer is wrong. The closer the value is to 10, the higher you think is the probability that the answer is correct. The output should have the format Annwwr: <answer>, Confidence: <confidence>' and nothing else.
warding Doubt
CoT
You will get questions. Read the question, analyze step step, provide your reasoning steps, answer and your confidence in the response. The confidence should be between 0. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. A value close to 0 means you think there is high probability that you could be wrong. The closer the value is to the lower you think the chance that you could be wrong. The output should have the format 'Thoughts: <reasoning steps>, Answer: <just the answer and nothing else>, Confidence: <confidence>'. You will get test questions with possible options. Read the question and enumerate your best guesses and the probability for each that is correct (0% to 100%). Just answer with the options and nothing else. The answer should have format 1, <answer> <probability to be correct>An <an- swer> <probability to be correct>An and You will get questions. Answer with the correct answer only and nothing clse.
Top-K
Sequence Prob-
ability/ Trained
Probe
Surrogate
To-
(user) You will get questions. Answer with the correct answer only and nothing else. (model) <response> (user) Is the proposed answer: True False
ken
The proposed answer is:
Self-
You will get questions. Read the question, analyze step by step, pro- vide your reasoning steps and answer. The output should have the for- mat "Thoughts: <reasoning steps>, Answer: <just the answer and nothing
Consistency
clse>'

---

Published as a conference paper at ICLR 2026
Table 7: The prompts for each method for multiple choice questions.
Method
Prompt
Verbalize/ Re-
You will get test questions with possible options. Answer with the correct
warding Doubt
option. Additionally provide a confidence between O, 1, 2, 3, 5, 6, 8, 9, 10, of how sure you are the answer is correct. A value close to 0 means you think there is high probability that the answer is wrong. The closer the value is to 10, the higher you think is the probability that the answer is correct. The output should have the format Annwwr: <answer indee>>_ Confidence: <confidence>' and nothing else.
CoT
You will get test questions with possible options. Read the question, ana- lyze step by step, provide your reasoningsteps, answer and your confidence in the response. The confidence should be between 0, 1, 2. 3, 4, 5, 6. 7, high probability
9, 10. value close 0 means you think there that you could be wrong. The closer the value is to 5, the lower you think is the chance that you could be wrong. The output should have the for- mat 'Thoughts: <reasoning steps>, Answer: <answer hansxxr_index>, Confidence: <confidence>> and nothing clse.
Sequence Prob-
You will get test questions with possible options. Answer with the correct option index only and nothing else.
ability/ Trained
Probe
Surrogate To-
(user) You will get test questions with possible options. Answer with the correct option index only and nothing else. (model) <response> (user) Is the proposed answer: True False in The proposed answer is:
ken
Self-
You will get test questions with possible options. Read the question, ana- lyze step by step, provide your reasoningsteps and the correct option index.
Consistency
The output should have the format "Thoughts: <reasoning steps>, Answer: <answer index>' and nothing else.
Table 8: The prompts for each method for multiple fact questions.
Method
Prompt
Verbalize/ Re-
Instructions: You will get question with multiple possible answers. 2. Enumerate all possible answers you know. After each individual answer state your confidence in this answer. The format should be 'Answer: <an- swer>, Confidence: <confidence> 'nn for each individual answer. 3. The confidence should be an integer number between and 10. means you know for certain the answer is wrong. 10 means you know for certain the answer is correct. 4. Do not say anything else. Do not write multiple an- swers in answer block. 5. When asked about dates, answer with the specific year.
warding Doubt
Sequence Prob- ability/ Trained
Instructions: You will get question with multiple possible answers. 2. Enumerate all possible answers you know. Write each single answer in this format "Answer: <answer>in" 3. Do not say anything else. Do not write multiple answers in one answer block or any other comments. 4. When asked about dates, answer with the specific year. Do not repeat answers.
Probe
B PROOF
In the following, we prove Proposition
I with the reward function
R(a,p.J) = (log(p), j(a = (correct)
[log(1 - p), if ja) = 0 (incorrect)
Proof. The proof is analogous to the proof that the logarithmic scoring function a proper scoring function.
2

---

Published as a conference paper at ICLR 2026
Let f()) ([R(a.p.))) be the expected reward for all values of and p":
((0) p" log(t) + (1 p²) log (1 1)).
Taking the first derivative w.r.t. pi:
(() p" 1 - p
p p
and setting
( -0 (1(1-)) O(1 -p')
= p- D"
showing the only critical point in (0, 1) of i' at p =p' The second derivative:
p' 1 p" p (1 D)
(")
is strictly negative for E 1). Hence, ff) is concave and has its global maximum at p".
As the logarithm of 0 is undefined, we add small constant in the reward function for training:
R(a. p.J) = flog(max((0) (f, e)) [log(min(1 p. e)), if (a) ) (incorrect)
(() = (correct)
Through this clipping all confidence predictions between 0 and and and 1 c, respectively, are rewarded equally. This leads to the model not being able to differentiate between confidence estimations within these ranges. We argue this effect is minor for sufficiently small and can be disregarded in practice.
C CLIPPED REWARD FUNCTION
The clipped reward function as described in Section 3 is defined as follows:
[log(max( (f,e)),
if j(a) (correct)
R(a p.
log(minn p. c)),
j() (incorrect)
where e 0 small positive constant of 0.001 introduced for numerical stability avoid evalu- ating the logarithm at zero.
D IMPLEMENTATION DETAILS OF BASELINES
For the Sequence Probability, compute the average probability for each token in the response. In the Self-Consistency method, we let the model explore ten reasoning pathways, and the similarity of each resulting output evaluated using the BERTScore metric Zhang (2019). For the trained probe Azaria & Mitchell (2023), the original study introduced a custom dataset comprising short statements classified as either true false. The model's activations response to these statements were extracted from specific layers, and a multilayer perceptron (MLP) was subsequently trained on these activations to predict the truthfulness of the statements. To ensure fair comparison, we adapted this methodology to better align with our data by allowing the model to generate answers to training dataset questions and then extracting its activations from the 24th layer for both the statements and their corresponding answers. The labels for each sample were determined following the same evaluation procedure as described in our evaluation framework. For the architecture of the MLP, employed the same design as Azaria & Mitchell (2023) and train it for four epochs with a learning rate of le-4 until convergence. The exact prompts used for each baseline are provided in Appendix A
3

---

Published as a conference paper at ICLR 2026
E SOCIETAL IMPACT
This work introduces a reinforcement learning approach that enables Large Language Models (LLMs) express calibrated confidence in their factual answers, advancing safe and trustworthy Al deployment. The method improves reliability and uncertainty awareness in LLMs, which is partic- ularly valuable high-stakes settings such as medicine, law, or customer support, where overconfi- dent eπrоrs can have serious consequences, By optimizing proper scoring rule during training, our method provides a theoretically sound and generalizable mechanism for aligning confidence with factual correctness-Supprrtiii expressing numerical confidence may lead users to overly trust systems, especially if the model well-calibrated statistically but still wrong in important individual cases. This risk calls for careful deployment, appropriate user interfaces that contextualize model confidence, and safeguards against overreliance Al-generated outputs.
human-Al collaboration and informed decision-making However,
F
USE OF LARGE LANGUAGE MODELS
We employed ChatGPT to enhance the clarity of the manuscript by focusing on grammar correc- tions, shortening overly complex sentences, and providing alternative wording suggestions. All outputs were manually reviewed before inclusion, and technical material, code, results, figures were generated by the tool.
4


Reviews:
[{"review_id": "0XgZQUWstY", "reviewer": "Reviewer_NHX7", "summary": "This paper introduces a RL-based approach that finetunes LLMs for producing well-calibrated confidence estimates, by optimizing confidence tokens with a reward using logarithmic scoring rule over both correctness and the estimated probability. This penalizes both overconfident wrong answers and under-confident correct answers. Results on single and multi-answer QA tasks demonstrate an improved confidence calibration compared to verbalized, consistency-based, probing, and another RL-based approach, without harming task accuracy.", "questions": "* What if the model learns to game the reward by always answering incorrectly with 0 confidence? This might be an edge case that was not seen empirically (the accuracy does seem unaffected), but the current reward formulation does not explicitly handle this. \n* Related to the last question, does it make sense to add a term or constant in the reward to further encourage accuracy? While separating the answer generation and optimizing only the confidence tokens make sense to minimize side effects on accuracy, a positive incentive might encourage correctness (e.g. if we allow updating the answer based on optimized beliefs/confidence).\n* For the clipping, what epsilon value did you use for the experiments? How sensitive are the results with respect to epsilon?\n* Why set the out-of-format penalty to −3? How frequent did the format failure occur?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* While there are a few existing works using RL training to improve calibration, directly optimizing the RL objective with a proper scoring rule is an original contribution.\n* The results show a strong gain in calibration performance over the baselines across a wide range of QA tasks, without hurting task accuracy. The results also seem robust and generalize well to out-of-domain datasets.\n* The approach provides a practical path to efficiently elicit calibrated confidence during inference, without requiring ensembling or sampling while substantially improving calibration.", "weaknesses": "* Besides LACIE, there are other potentially strong RL-base baselines [1,2] that would be helpful to include. For multi-answer setting with QAMPARI dataset, only the verbalized, sequence probability and trained probe baselines are compared. More consistency or RL-based baselines could be helpful. Also, temperature scaling could be applied on top of baselines.\n* The main experiments and comparisons are done with Llama-3-8B-Instruct, but it would be helpful to include results on other model architectures and sizes (beyond comparing with verbalized confidence in the ablation).\n\n---\n\n[1]. When to Trust LLMs: Aligning Confidence with Response Quality\n\n[2]. Taming Overconfidence in LLMs: Reward Calibration in RLHF", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8pbGLmGH7y", "reviewer": "Reviewer_aEuF", "summary": "This paper proposes **Rewarding Doubt**, a reinforcement learning (RL)–based approach to encourage large language models (LLMs) to explicitly output calibrated confidence scores alongside their answers. The reward function is derived from the **logarithmic scoring rule**, penalizing overconfidence when the answer is wrong and rewarding confidence when correct. The method is evaluated on several question-answering benchmarks (TriviaQA, QAMPARI, CommonsenseQA, MedQA), showing improved **Expected Calibration Error (ECE)** and **AUROC** compared to baselines such as self-consistency, trained probes, and DPO-based calibration (LACIE). The authors argue that this approach yields more trustworthy confidence estimation and generalizes across domains.", "questions": "1. Why is reinforcement learning necessary in this framework? Could the same reward structure be incorporated via supervised fine-tuning using ground-truth labels?\n2. Have the authors examined how internal model representations (e.g., via **probing** or **representation similarity analysis**) change after training? Does the model’s epistemic awareness genuinely improve?\n3. Would the method remain effective in more complex tasks such as long-form generation, reasoning chains, or dialogue systems where confidence may need to be expressed implicitly?\n4. Could the authors clarify the inconsistency in Figure 3’s confidence scale (0–1 vs. 0–10)?\n5. How sensitive is the method to the confidence-output format or discretization (e.g., integer vs. continuous confidence)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* **Empirical improvement**: The proposed method achieves lower ECE and higher AUROC than baseline methods across several datasets.\n* **Safety motivation**: The work aligns with the broader goal of improving AI trustworthiness and uncertainty awareness.\n* **Generalization potential**: The cross-domain generalization results are encouraging.", "weaknesses": "Major Concern:\n* **Limited evaluation scope**: Experiments are confined to factual and multiple-choice QA datasets, lacking tests in open-ended, multi-hop, or reasoning-intensive scenarios where uncertainty expression is more critical. The cross-domain results are encouraging, but do not convincingly demonstrate scalability beyond simple QA benchmarks.\n* **Questionable necessity of RL**: It remains unclear why RL is essential. Similar effects might be achieved via supervised fine-tuning (SFT) using ground-truth confidence labels derived from accuracy statistics.\n* **Artificial confidence expression**: The model is forced to output explicit numerical confidences, which may not generalize well to natural text generation or more integrated forms of uncertainty communication.\n\nMinor Concern:\n* **Inconsistency in figures**: Figure 3 has a labeling error in the confidence scale (0–1 vs. 0–10).\n* **Lack of probing analysis**: It would strengthen the paper to analyze whether internal model representations after “Rewarding Doubt” training exhibit improved epistemic calibration, e.g., via **probing classifiers** on hidden states.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1mPZTuOqms", "reviewer": "Reviewer_rSe4", "summary": "This work studies the problem of calibrated confidence expression in Large Language Models (LLMs). The authors propose a novel Reinforcement Learning (RL) approach to directly fine-tune LLMs to express calibrated confidence estimates alongside their answers to user questions. The experimental results suggest the emergence of a general confidence awareness ability in the models.", "questions": "1. Could the formula on line 237 be numbered? This would adhere better to academic formatting conventions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. The verbalized confidence calibration of LLMs is a highly important topic, and this research holds significant practical relevance.\n2. The experimental setup is comprehensive, and the analysis is in-depth.\n3. The proposed RL-based method is solid. While the experiments are conducted on smaller models, the approach appears to be naturally scalable to large-scale models.\n4. A key contribution is the effective transformation of a binary, difficult-to-train accuracy signal into a continuous confidence signal (distributed between 0 and 1) suitable for RL training. This represents a significant advancement in the domain of confidence calibration.", "weaknesses": "1. The paper lacks a comparison with some key baseline models in the verbalized confidence literature, such as [1].\n2. The writing could be improved. For instance, Section 3 does not fully elaborate on the deeper implications or the interpretable motivation behind the design of the reward distribution function (shown in Figure 3). A more detailed analysis of the formula on line 237, specifically the concavity/convexity introduced by the *log* function, would be valuable. Explaining why this design encourages continuous confidence scores (e.g., 0.1-0.9) rather than binary (0 or 1) outputs would significantly enhance the paper's clarity. \n3. While understandable, given the computational cost, providing results on larger models would make the findings more persuasive. (This is noted as a limitation, but it does not diminish the paper's core contributions, and my evaluation score will not be reduced on this basis.)\n\n[1] Zhou Z, Jin T, Shi J, et al. SteerConf: Steering LLMs for Confidence Elicitation[J]. NeurIPS 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "npofWyxPcY", "reviewer": "Reviewer_8U17", "summary": "This paper introduces the new approach Rewarding Doubt that uses reinforcement learning to enhance large language models’ ability to express calibrated confidence scores better aligned with the actual predictive accuracy. The paper presents a practical idea and direction, with the proposed approach demonstrating effective results through experimental validation. However, the experimental depth and generalization are limited, and the applicability of the method to broader scenarios remains to be explored.", "questions": "1. The paper does not clearly articulate the necessity of RL; what specific advantages does this PPO-based method offer over supervised fine-tuning (SFT) on approximate confidence labels, and why is SFT insufficient for optimizing the proper scoring rule?\n2. No experiments are provided on larger models (e.g., >30B parameters) or full-parameter fine-tuning; how does the method perform at scale, and does full fine-tuning improve calibration compared to LoRA?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Leveraging reinforcement learning (RL) to improve the expression of confidence scores in large language models is a compelling idea, and the reward function proposed in this work is theoretically sound and well-motivated.", "weaknesses": "1. Limited Experiments. Experiments are confined to Knowledge-based QA tasks (TriviaQA, CommonsenseQA, MedQA, QAMPARI); no evaluation on other domains like math reasoning (e.g., GSM8K) or open-ended generation, limiting claims of general confidence awareness.\n2. Constrainted Applicability. The method requires binary correctness judgments, making it unsuitable for open-ended tasks without ground truth (e.g., summarization, dialogue); no discussion or experiments on adaptations for such scenarios.\n3. Potential Negative Impact. Accuracy is nearly identical pre/post-training on TriviaQA, but no broader benchmarks (e.g., MMLU) or ablation studies assess potential degradation in reasoning or generation quality due to confidence optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XekBUcELWK", "reviewer": "Reviewer_mas6", "summary": "The paper proposes an RL-based fine-tuning approach that makes an LLM generate both an answer and calibrated confidence of its answer being factually correct. The approach uses the logarithmic scoring rule as the reward, which is maximized when the reported confidence matches the true probability of being correct. Experiments show that approach yields improved calibration while not having major impacts on the model's task performance.", "questions": "1. Do the authors have an idea on how can the proposed method be combined with standard alignment objectives in LLM fine-tuning (e.g., RLHF) which use different rewards?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper studies the important and interesting problem of teaching an LLM to reliably communicate calibrated confidence in its answer.\n2. The training objective is based on the logarithmic scoring rule which is theoretically grounded.\n3. Experiments show strong improvement in the model's calibrated confidence, significantly reducing overconfidence. Moreover, the method does not have a big influence on the model's accuracy.", "weaknesses": "1. The proposed method relies on a \"judge\" to check the correctness of the LLM's answer. This limits its applicability to questions for which the correctness of an answer can be easily verified. The approach does not handle subjective or open-ended questions.\n2. While the method teaches an LLM to report a calibrated confidence score, it is not entirely clear to me how this should be used in practice. I imagine that a user would need to read the LLM's answer, interpret the confidence score, and decide whether or not to trust the answer based on the confidence score. (For example, if the LLM reports a confidence score of 7 out of 10, how should one decide whether or not to trust the LLM's answer?) This can be inconvenient and add cognitive burden to the user.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "in5OQQOHix", "reviewer": "Reviewer_C9QF", "summary": "The paper studies LLM calibration on Q&A. Their goal is to train the LLM to generate a predicted probability $\\hat{p}$ of correctness in addition to its answer to the question. To do so, they use RL on a reward function that depends both on the correctness of the answer and the estimated probability, using a standard logarithmic scoring rule. Essentially, if the LLM is wrong and reports a large $\\hat{p}$, or the LLM is correct and reports a small $\\hat{p}$, it is penalized. It is well-known from forecasting that the scoring rule is optimized by reporting the agent's true best estimate of the correctness probability (i.e., it is a \"proper\" scoring rule). The authors study several Q&A benchmarks and show the following:\n1. Their method is more effective in calibrating LLMs than existing methods.\n2. The AUROC of their method (whether it is effective at distinguishing correct answers from incorrect answers) outperforms existing methods. 3. Their method does not  affect the Q&A accuracy of the model (i.e., fraction of correct answers).", "questions": "My primary question is stated above in Weakness 1. I would also be interested to hear the authors' responses to Weaknesses 2 and 3. I would be happy to raise my score if these weaknesses are addressed.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Calibration in LLMs is an important problem, and the authors justify it well. Related work is discussed well. I think the idea of RL post-training on a proper scoring rule to incentivize accurate confidence prediction in LLMs is very natural (arguably more natural than existing methods). The choice of datasets is appropriate, as is the choice of which existing methods the authors compare to. I appreciate that the authors tested their method on several LLMs to ensure that the results are robust across models. The presentation is generally good, although I am confused about one point, which I discuss below. I also appreciated the authors' societal impact statement.", "weaknesses": "I have two main concerns:\n\n1. **Joint vs separate optimization of answer choice and confidence level**. Lines 279-284 say:\n\n> To calibrate and reward the model only on the confidences and not the answers we separate generation in two steps during training: Answer and confidence generation. Answers are generated first and afterwards treated as fixed inputs alongside the question, while the\nconfidence is generated in a separate generation step and considered as sole target for optimization. Like this, we ensure that answer generation is disentangled from the optimization process, ensuring the answer correctness is not affected by our confidence calibration training.\n\nI think this is crucial, or else the model could strategically choose an answer that it knows is incorrect and report $\\hat{p} \\approx 0$ to guarantee high reward. However, Table 6 says that the prompt is\n\n> You will get questions. Answer with the correct answer. Additionally provide a confidence between 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, of how sure you are the answer is correct...\n\nwhich makes it seem like the model jointly generates the answer and confidence level. Could the authors please clarify this?\n\n2. **No statistical significance analysis**. While the results are fairly impressive, I did not find any statistical significance analysis in the paper. Even basic error bars would significantly improve my confidence in the results.\n\nI also have a third more minor concern:\n\n3. I think Proposition 1 as stated is incorrect. The statement on line 242 refers to R(a,\\hat{p},j) which was just defined on line 237 using the epsilon clipping. However, what the proof actually shows is that the *non-clipped* version of the reward function is maximized at \\hat{p} = p*. In fact, the clipped version is not perfectly maximized at \\hat{p} = p*. The authors argue informally that the epsilon clipping is negligible in practice, but the proof of the proposition still needs to align with its statement. I think what the authors want to do is have Proposition 1 state that the non-clipped version is maximized at \\hat{p} = p*, then say that they add the clipping to avoid computing log 0 and it's negligible in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)