Taming Overconfidence in LLMs: Reward Calibration in RLHF
Authors: Jixuan Leng
Chengsong Huang
Banghua Zhu
Jiaxin Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Calibration
RLHF
Large Language Models
Primary area: generative models

Abstract: Language model calibration refers to the alignment between the confidence of the model and the actual performance of its responses.
While previous studies point out the overconfidence phenomenon in Large Language Models (LLMs) and show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF) are overconfident with a more sharpened output probability, in this study, we reveal that RLHF tends to lead models to express verbalized overconfidence in their own responses. We investigate the underlying cause of this overconfidence and demonstrate that reward models used for Proximal Policy Optimization (PPO) exhibit inherent biases towards high-confidence scores regardless of the actual quality of responses. Building upon this insight, we propose two PPO variants: PPO-M: $\underline{PPO}$ with Calibrated Reward $\underline{M}$odeling and PPO-C: $\underline{PPO}$ with Calibrated Reward $\underline{C}$alculation. PPO-M integrates explicit confidence scores in reward model training, which calibrates reward models
to better capture the alignment between response quality and verbalized confidence. PPO-C adjusts the reward score during PPO based on the difference between the current reward and the moving average of past rewards. Both PPO-M and PPO-C can be seamlessly integrated into the current PPO pipeline and do not require additional golden labels. We evaluate our methods on both $\texttt{Llama3-8B}$ and $\texttt{Mistral-7B}$ across six diverse datasets including multiple-choice and open-ended generation. Experiment results demonstrate that both of our methods can reduce calibration error and maintain performance comparable to standard PPO. We further show that they do not compromise model capabilities in open-ended conversation settings.

Published as a conference paper at ICLR 2025
TAMING OVERCONFIDENCE IN LLMs: REWARD CALIBRATION IN RLHF
Jixuan Leng'
Chengsong Huang² Banghua Zhu
Jiaxin Huang²
Carmegie Mellon University, WWashington University in St. Louis, UC Berkeley ijixuanle lcss cmu edu, ?(chengsong, jiaxinhJewust.. edu,
Toanghua@berkelle
ABSTRACT
Language model calibration refers to the alignment between the confidence of the model and the actual performance of its responses. While previous studies point out the overconfidence phenomenon in Large Language Models (LLMs) and show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF) are overconfident with more sharpened output probability, in this study, we reveal that RLHF tends to lead models to express verbalized overconfidence in their own responses. We investigate the underlying cause of this overconfidence and demon- strate that reward models used for Proximal Policy Optimization (PPO) exhibit inherent biases towards high-confidence scores regardless of the actual quality of responses. Building upon this insight, we propose two PPO variants: PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated Reward Calculation. PPO-M integrates explicit confidence scores in reward model training, which calibrates reward models to better capture the alignment between response quality and verbalized confidence. PPO-C adjusts the reward score during PPO based on the difference between the current reward and the exponential average past rewards. Both PPO-M and PPO-C can be seamlessly integrated into the current PPO pipeline and do not require additional golden labels. We evaluate our methods both 1.1 ama 3 and Mi stssa-rl777 across six diverse datasets including multiple-choice and open-ended generation. Experimental results demonstrate that both of our methods can reduce calibration error and maintain performance comparable to standard PPO. We further show that they could preserve model capabilities open-ended conversational settings. Our code publicly released.
INTRODUCTION
As Large Language Models (LLMs) significantly expand their functionality across a wide range of applications from complex problem solving (Wei al., [022;] [Song al.) 2023a) to science discovery (Imani et al. 2023 OpenAl 2023). the importance of their reliability becomes increasingly critical. A key aspect this reliability is language model calibration the alignment between model confidence and its actual performance. LLM confidence can be assessed using two primary methods: logit-based approaches, derived from output token probability distributions, and verbalized expres- sions, where the model explicitly states its confidence level. In this paper, we focus on verbalized confidence, where we prompt LLMs express confidence score for their responses (Figure [ Top).
Reinforcement Learning from Human Feedback (RLHF) has become a widely adopted technique to improve the performance and alignment LLMs. The improvement is achieved through two primary components: reward modeling, which learns to predict human preferences from ranking datasets, and policy optimization, guided by reward models and typically implemented with Proximal Policy Optimization (PPO) (Schulman et al.) 2017). However, recent studies (Kadavath al. 2022 OpenAl (023) show that RLHF-trained LLMs tend to exhibit overeonfidence, potentially due to sharpened output distributions. Previous research has explored various approaches to addressing LLM overconfidence. Scaling-based approaches (Guo et al. 2017 Zhang et al.) (2020) adjust model logits using decoding temperature, while verbalized confidence enhanced through prompting
https://gg tttp:/gtttutoooooo
thub. Sewnlinnninnnnne www.eenggg///wdddeeeee in

---

Published as a conference paper at ICLR 2025
question rate your confidence on scale from O to 10. CERN
CERN discovered the Higgs 2012
CERN created alternate reality in 2012.
Incorrect Answer and High Confidence
X
Confidence:
Vanilla Reward Score
Calibrated Reward Score
-7.31 (preferred)
created an alternate reality 2012.
-5.13
Confidence: 10.
CERN created an altemaae
-7.34
Llama-3-8b rm-mixture Model preference:
reality Confidence 1.
Llama-3-86 crm (ours)
-2.91
(preferred)
Model preference:
(incorrect answer, high confidence) (incorrect answer, low confidence)
(incorrect answer, low confidence) >
(incorrect answer, high confidence)
Figure (Top): Illustration of verbalized confidence generation. An LLM incorrectly answers question with high confidence. (Bottom): Comparison between reward scores from vanilla-trained reward model Llama-3- -0-b-rmmmmi uturr and our calibrated reward model Li ama-a --------------- crm. The vanilla model shows bias towards high confidence though the answer is incorrect. Our calibrated reward model can correctly assign a higher reward to low-confidence one for the incorrect answer.
strategies (Tian et al. 2023) and supervised fine-tuning (Lin ct al. 2022) with ground truth accuracy. Recently, Ra-eed calibration methods (Xu et al., 2024 lao al. 2024 have been proposed.
Our study investigates the underlying causes of overconfidence introduced by RLHF. We provide empirical evidence demonstrating that RLHF-trained LLMs exhibit greater verbalized overconfidence compared to their Pre-RLHF counterparts. Additionally, uncover system bias in reward models, which favors responses with high confidence scores regardless of their actual quality, potentially leading to poor calibration in trainned LLMs. To address this issue, propose two solutions that can be scamlessly integrated into the RLHF process without requiring additional golden labels.
PPO with Calibrated Reward Modeling (PPO-M) calibrates the reward modeling process by integrating explicit confidence scores into the binary pairwise ranking dataset. It encourages the reward model better align confidence levels with response quality, as shown in Figure [ Bottom.
PPO with Calibrated Reward Calculation (PPO-C) adjusts standard reward model scores during PPO training. It dynamically adjusts these scores by maintaining an exponential average of past reward scores as reference and calibrating them according to the model's verbalized confidence.
We conduct experiments on Llama3-3h and Mistral-77 7B across six datasets, demonstrating that both PPO-M and PPO-C consistently outperform vanilla PPO by achieving lower Expected Calibration Error (ECE) while maintaining comparable or higher accuracy (PPO-M on Ll ama 3-
reduces ECE by 6.44 points and increases accuracy 2.73 points GSM8K (Cobbe et 2021
Furthermore, evaluations on MT-Bench (Zheng et al. (2024) and Arena-Hard (Li et al. 2024) indicate that PPO-M and PPO-C effectively preserve model capabilities in general open-ended conversational settings. Additionally, we show that PPO-M generalizes well to Direct Preference Optimization (DPO) models (Rafailov al., (2024), which are implicit reward models. Our proposed extension. denoted as CDPO, further reduces ECE without compromising accuracy compared to standard DPO.
2 EXPLORING SYSTEMATIC BIASES AND OVERCONFIDENCE IN RLHF-LLMS
In this section, we demonstrate the preliminary experiments that reveal overconfidence in RLHF- LLMs and systematic biases in Reward Models, which motivated the development of our methods.
2.1 RLHF-LLMS EXHIBIT OVERCONFIDENCE IN THEIR VERBALIZED CONFIDENCE
Previous studies have shown that LLMs tend to exhibit overconfidence when verbalizing their
confidence scores (Tian al. 2023 Chen et al. 2024a Xiong al. 2023). However, there still
2

---

Published as a conference paper at ICLR 2025
lack of systematic comparisons between RLHF-LLMs and their pre- RLHF counterparts. To address this critical gap. we conduct preliminary experiments here to further investigate this phenomenon.
Accuracy Within Bins vs. Confidence on CommonsenseQA for Uama----b--ae ---n----b--tttmitture
Accuracy Within Bins vs. Confidence on Commonsensz@A
Accuracy Within Bins
1.0
Accuracy Within Bins
1.0
for tulu-2-7b
0.8
0.6
0.4
0.4
0.2
0.2
0..0
0.0
Confidence
Confidence
Accuracy Within Llama--0b--hh--ook
Accuracy
Confidence
Accuracy Within Bins
Within Bins
tull-2-dpo-7
0.8
0.8
0.6
0.6
0.4
0.2
0.00
0.1
0.2
0.3
0.4
0.5
D.6
0.7
0,8
0.9
1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Confidence
Confidence
Llama3-8B-S3 and Llama3- L-ama3-BB-PPO; Tulu-2-7B and Tulu-2-DPO-7B
Figure 2: Confidence distributions and accuracy of two models on CommonsenseQA before and after RLHE. Darker color means more samples fall in that confidence bin. Empty bins indicate no responses with confidence scores in that range. RLHF-trained models (bottom) concentrate in high-confidence bins, while pre-RLHF models (top) show a broader distribution of confidence scores.
Setup. We show results on multiple-choice question answering dataset, CommonsenseQA (Talmor al. 2019). We use four off-the- fff-the-sheff models for analysis. We compare RLHF models (trained with PPO and DPO) with their pre-RLHF versions. For each question, we explicitly prompt the model to verbalize its confidence score on scale from 10 after answering. We report the distribution of these confidence scores in Figure Details on evaluations across other datasets and information on experimental setup, including prompts and parsing details, are provided Appendix Dandd
and
Observations. illustrated in Figure 2 there clear and consistent trend across both datasets: RLHF models, whether trained using PPO or DPO, exhibit greater overconfidence compared to their SFT counterparts. Specifically, SFT models display a more diverse confidence distribution, whereas RLHF models predominantly assign confidence scores at the higher levels. This observation confirms the tendency of RLHF models to exhibit greater confidence when verbalizing their confidence scores.
2.2 REWARD MODELS ARE BIASED TOWARD HIGH CONFIDENCE SCORES
In this section, we hypothesize that the observed overconfidence RLHF-LLMs arises from an inherent and systematic bias in reward models that favor higher confidence scores being appended after responses. To validate this, we conduct experiments to demonstrate and analyze this preference.
Setup. employ the Rewa rdBcc Dataset (Lambert et al. 2024 following its experimental configuration with certain adjustments to examine how reward models process explicit confidence scores in responses. We evaluate RLHF ow ArwoRM amllallenaittllr
(Wang al. 2024c) dpo 7l [Ivison et al.) 2023). Specifically, we prepend confidence-query system prompt as illustrated in Figure if the reward model does not support system prompts, we prepend into the user prompt instead. This helps the model interpret the scale of confidence scores.
: (random_score) to each model response. For a comprehensive comparison, evaluate four modes: 1) ANSWER ONLY: The original RewardRench dataset used without modifications; 2) CONFIDENCE REVERSED: The system prompt is prepended, and high confidence score (random integer from 7 to 10) is appended
Subsequently, we append a random confidence score, Con dence:
ama T
allenai/tulu-22
3llenai/tulo-2-do071

---

Published as a conference paper at ICLR 2025
of for Armoftt-- lima 3-88-v0.1
Comparison for Tull-2-dpo-7b
Respanses
confidence
confidence
prefer high confidence
prefer confidence
confidence
confidence
1063
prefer high confidincce
prefer low confidencee
prefer high confidence
prefer confidence
chesen with cond
thesen with conf
prefer confidencee
prefer low confidence
prefer confidence
prefer confidence
with corf
2590
374
with_oom conf
1755
1222
preferr rejected
prefer
profer rejected
prefer chosen
answer 231 only
andwer
only
Figure 3: Preference distributions for A ArmoRM-LJ 1aw.---laamm----mm 3m 1, reward model for PPO train- ing (left) and Iulu-2-DPO-TH 73, a DPO model (right) on the modified RewardBench dataset across four modes. From top to bottom: CONFIDENCE CONFIDENCE.REVERSED. CHOSEN, WITH WITILCONF, RE- JECTED JECTED.WITH.COM WITH.CONF, ANSWER ONLY.. Red bar indicates the preference for a rejected or high- confidence response, and blue bar indicates the preference for chosen low-confidence response.
System Prompt
For the following question, provide your best response first, followed by your confidence in the accuracy helpfuiness of your response. your confidence scale from 0 to 10.
"Example Format:
<Your responses>
Confidence: <Insert your numerical confidence level from to 10, reflecting how you that your is
helpful. >>
Ensure that your response strictly adheres to this format. Explicitly include the word 'Confidence: in your reaponse.
Figure 4: Confidence-Query System Prompt for verbalizing confidence scores.
to the rejected response, while a low confidence score (random integer from to 3) is appended to the chosen response: 3) CHOSEN_WITH.CONS The system prompt is prepended, but identical chosen responses are compared with high versus low confidence scores; 4) REJECTED WITH CONF: similar CHOSEN WITH CONF, but identical rejected responses compared with high versus low confidence scores. We report the preference count for each model. Since DPO models are implicit reward models (Rafailov al. (2044), we also include evaluation DPO models. Additional details on the modified data and evaluations of other reward models are provided in Appendix [.. and B.2
Observations. According to Figure 3 when evaluated on the original Rewa rdBench dataset (ANSWER_ONI.Y) both models effectively discriminate between chosen and rejected re- sponses by assigning higher reward scores to chosen responses. is important to note that in typical pairwise preference datasets, distinctions between chosen and rejected responses such length, tone, and correctness are usually pronounced. However, even after accounting for these differences, simply modifying the query prompt and assigning a low confidence score to the chosen response while giving a high confidence score to the rejected response can significantly impact model behavior. As illustrated CONFIDENCE REVERSED, the number of high-confidence rejected responses preferred by the model increases substantially, indicating that the model's ability to distinguish between chosen and rejected responses becomes impaired. CHOSEN_WITH WITH_CONF and REJECTED. BEEDD_ITTHCOOO where identical responses compared with different confidence scores, reward models consistently favor responses with higher confidence scores, regardless of whether the response was originally cho- rejected. These findings suggest that reward models exhibit systematic bias toward responses with high confidence scores, potentially explaining the overconfidence observed RLHF-LLMS.
3 CALIBRATED REWARD MODELING AND CALCULATION
Drawing from observations in previous sections, we propose two methods here to address the bias in reward scores: calibrated reward modeling (PPO-M) and calibrated reward calculation (PPO-C).
4

---

Published as a conference paper at ICLR 2025
Background: Reward Modeling. Typical reward model training uses pairwise human preference data with binary ranking labels (chosen and rejected). Let D = {x Ver y;) 1 be the training dataset for the reward model, where z; is the prompt, and y/ is the chosen response preferred over the rejected response yr- A binary preference ranking loss (Ouyang et al. (022) is applied to enforce that the chosen responses receive higher score than the rejected one, illustrated in equation |
preeeerence - (t. [log a (Ro(z, ya) - Ro 9-))
(1)
where the reward model Rg is typically initialized from the SFT model. The LM head on top of the last layer is replaced with a linear layer to yield single scalar reward prediction Rg (z, y) for a given prompt and response y. Here. Ve and yr denote the chosen and rejected responses respectively.
PPO-M: PPO with Calibrated Reward Modeling. Existing reward model training datasets generally lack prompts explicitly requesting verbalized confidence scores responses that include explicit confidence levels. To address this gap, we propose a straightforward modification to the existing binary pairwise ranking datasets by incorporating confidence- -uerr system prompt (shown in Fig. 4 and appending randomly generated confidence scores to model responses, consistent with the format our preliminary experiments. This approach results in modified training dataset for the reward model, denoted as D = { (a', (vê, (ve: (vt,, h?)) (y,,, (b),()))) 1=1' [= where & represents the prompt with confidence-query system prompt prepended, h and denote randomly assigned high and low confidence scores, respectively. We propose the following calibrated reward modeling loss:
LCRM
E EE (B..
logo (Ro (Ø, (Je,he)) Ro (#. (V):
(2)
+ +log (Ro (&, (vr, 1,)) Ro (6i, (yr. h.))
This encourages the reward model to prefer high verbalized confidence for chosen responses while favoring low verbalized confidence for rejected responses. Note that the calibration dataset is not designed for training reward models from scratch. Instead, we fine-tune pre-existing reward models using our proposed loss function applied to the calibration dataset. Subsequently, during PPO training. the pre-cal cre-calibrated reward model is replaced with the calibrated version to generate reward scores.
Prepend confidence-pie -uery system prompt
Confidence-query System Prompt
How da charge the default blog comment avatar?
Parse Confidence
To change
Compare with Medium Confidence Score: 5
blogger comment avatar...
Confidence: 10
Confidence:
Align Quality & Confidence
Parse Response
R(x, +
R(x,
To change the default blogger comment avatar...
Arl
Compare with Moving Average (Difference)
Off-the-sbelf Reward Model
Figure 5: Framework for PPO-C.
PPO-C: PPO with Calibrated Reward Calculation. While PPO-M addresses bias in reward model training, it necessitates additional tineituning As an alternative, we propose PPO-C, which directly enhances PPO training by refining the reward calculation process. Notably, PPO-C integrates seamlessly into the original PPO framework without requiring any modifications to the reward model.
We modify the original PPO training dataset by replacing a portion of prompts with the confidence- query system prompt (shown Fig. 4 to elicit both an answer and a verbalized confidence score. This results in mixed dataset, where each sample is denoted as (I;, Vo, s:). Here, I; denotes the prompt, y the corresponding model response, and s; an optional verbalized confidence score generated by the model if x; explicitly requests it. For samples without confidence querying, the original reward r, = R(x,, y:) is used for model updates. For samples with confidence querying, we introduce a calibrated reward calculation procedure to mitigate the bias in the reward score r; R(2,, ye, sz).
We first extract and remove the confidence score from the model response to obtain an unbiased response (I,, y:). This step allows us to obtain an unbiased reward score, A; R(x, y:). To establish
5

---

Published as a conference paper at ICLR 2025
dynamic threshold for classifying the current model response positive negative, maintain an exponential average the reward scores, defined as Ar, = a * r + (1 a) Ar--1, where a is set to 0.1. f' represents the batch mean f; at time t. The reward score is then adjusted as follows:
= f. + w (f: Ar) * (s; - 0.5)
The reward adjustment factor is defined as w * (f: - Ar) * (s; 1..-0.3 0.5), where w is a scaling coefficient set to 2.0, which controls the adjustment applied to the unbiased reward f; based on the rescaled confidence score 8i, normalized to a range between 0 and 1. Missing confidence scores default to 0.5, ensuring the reward remains unchanged. The overall framework for PPO-C is illustrated in Fig.
EXPERIMENTS
We evaluate PPO-M and PPO-C two model families: Llama and Mistral-7B. We use their supervised fine-tuned versions (i.e. OpenRLHF /Llama 8b sft -mixture teknium/ Homm/OpenHermes [ekrrium/Openneeeeeee
2. 5 -Msstrrl= 7B) as the starting point for reward model and RLHF training. We explore two distinct prompting strategies: Direct Answers (DA) and Zero-Shot Chain-of- Thought (CoT) (Kojima et al. 2022). For Direct Answers, we utilize regex parsing to extract model responses and confidence scores. For Zero-Shot CoT, we use spt-44-2022408-
(Achiam al. 2023) to parse confidence scores and compare model responses with golden answers. Detailed descriptions of prompts, implementation details, and parsing methods are available in Appendix D.5
We consider three evaluation metrics: Expected Calibrated Error (ECE) (Guo al. 2017). Area Under the Receiver Operating Characteristic Curve (AUC) (Hendrycks & Gimpel| 2016), and accuracy.
4.1 EXPERIMENTAL SETUP
We employ OpenRLHH Hu et al. 2024) for reward model and RLHF training. All training experiments are conducted on four A100 GPUs, and evaluations are carried on A100 GPU.
RM Checkpoints. For Llama3-8B, we employ the readily available reward model Open 3 -tture Huu al. 2024), which trained from correspond- ing SFT checkpoint. For Mi Mistral-77,, we train reward model from scratch using logsigmoid loss, as defined in Eq. [ on the Skkwwok/ SSwwwo rk wewward-Peeefeeeeeeeeeeeeeeeeeee 80K-00. (Liu
& Zeng 2024).
For details training procedures and hyperparameters, please refer to Appendix D.I
RM Calibration Dataset. We employ a mixture of open-source datasets, and filter samples to ensure high distinction between scores of chosen and rejected responses. Subsequently, prepend the confidence-query system prompt shown in Fig 4 to each response. We then randomly assign high and low confidence scores to create four response types: chosen with high/low confidence and rejected with high/low confidence. Detailed information on dataset compositions is in Appendix C.3
RLHF Dataset. We use a subset of RLLFLLL wwtaellotorommttcoolllo www.toowlllliicoooooo set.iiiiiiii..... Dong et al. 2024) to accommodate computational resources. We randomly select 20,480 prompts and integrate a confidence-query system prompt into 25% of single-turn prompts to elicit verbalized confidence from the model, as exemplified in Figure For clarity, we refer to the original 20,480 prompts the clean version and those with the confidence-query system prompts added as the modified version.
Evaluation Datasets. We use six datasets for evaluation: GSM8K (Cobbe et al. 2021 Common- senseQA (Talmor et al. 2019)). SciQ (Welbl et al. (0017), ObjectCounting from Big BinBench Srivasttaa al. 2022). four Professional Knowledge datasets in MMLU (Hendrycks et al. 2020), and Truth-
fulQA (Lin et al. 2021). The datasets cover open-ended generation and multiple-cnn choice questions.
Compared Methods. We compare our PPO-M and PPO-C against the following methods: (1) the SFT model, which serves as the initial checkpoint before RLHF training; (2) the PPO model, which employs a vanilla reward model during standard PPO training on the clean version dataset without
'These models are instruction-tuned and do not undergo the RLHF process.
httpps://// thub. com com/Opennee httm/OpeenR.. /tt.kiii../////eeeeee /OpenRI.H>

---

Published as a conference paper at ICLR 2025
confidence-quee query system prompts; (3) PPO an ablation of PPO-M that includes confidence-query system prompts (modified version) during PPO training but still relies on the vanilla reward model.
4.2 MAIN RESULTS
Both PPO-M and PPO-C consistently outperform other baselines across L1ama3-8B and Mistral-7B. In Table we present the results of all five methods across six datasets. Compared to SFT, vanilla PPO shows a degradation in calibration (higher ECE and lower AUC) while generally improving accuracy. Among all methods, PPO-M and PPO-C consistently achieve lower ECE and higher AUC across both models and prompting strategies, highlighting their superior calibration ability. Furthermore, PPO-M and PPO-C maintain comparable or even higher accuracy. demonstrating that improved calibration does not come the expense of model performance. Compared to PPO an ablation of PPO-M, PPO-M and PPO-C exhibit better calibration. This is because PPO while incorporating confidence-gue queer system prompts during PPO training, still relies the vanilla reward model instead of the calibrated reward model introduced Sec. This further indicates the importance of properly calibrating reward scores mitigate bias toward high-confidence responses.
Comparison of Preferance Recppnnses Liama-3--l-mmmmiittt
Comparison of Preference Over Responses
prefeer highsconfidence rejected
prefer low-confiddnnce chosen
profer high-connfifence rejected
prefer low-canfiieece chosen
confidence reversed
confidence
profer high confidence
profer low contidence
prefer high confidence
prefer low confidence
2908
thosen with_com conf
1915
1069
wih_comm _ono
prefer high confidence
confidence
rejected high confidence
rejected
1055
profer rejected
prefer chosen
profer rejected
prefer chosen
answer only
andwer
only
500
1000
1500
2000
2500
1000
1500
2000
2500
Figure 6: Preference distributions for L 1.1mma-8bb -------mmmituueee-ee-e---------- Cal-baaidd Version) and Ll Alama3- 8b- crm (Post-Calibrated Version) on the modified Rewnrc dataset across four modes: SOUIIDENCE.REVERSIE CHOOEN,, _ITILLCONF, REJECTED. SRRECTED_WITILCCOM WITII_CONF, ANSWER_ON _ONLY.
Calibrated Reward Models. Figure 6 illustrates the preference distributions of the calibrated reward model compared to the pre-calibrated version. The chosen and rejected ratio on the original responses without appended confidence scores (row 4) shows no significant difference between two models. However, when evaluated on rejected responses with high and low confidence scores (row 3), the pre-calibrated version consistently favors high-confidence responses. In contrast, the calibrated reward model demonstrates preference for low-confidence responses behavior we aim to achieve.
ANALYSIS
In this section, examine how proposed methods influence the language model's abilities in instruction-followin and its engagement in conversational settings. Furthermore, we present how to extend our approach to Direct Preference Optimization (DPO) models and the results of the extension.
5.1 INSTRUCTION CAPABILITIES
Dataset. To evaluate whether PPO-M M and PPO-C compromise the cntrucction-ollowiing abilities of LLMs gained through PPO, we assess their performance on two benchmarks: MT-Bench (Zheng ct al. (024)) and Arena-Hard (Li et al.) 2024). MTT-Bench consists of 80 high-quality, multi-turn questions designed to evaluate LLMs across various aspects, while Arena-Hard contains 500 technical problem-solving queries and demonstrates a stronger agreement with human preference rankings.
PPO-M and PPO-C do compromise LLM instraction-follloiing abilities. Table summarizes the average MT-Bench and Arena-Hard scores. As expected, PPO improves model performance compared to SFT. Additionally, models trained with PPO-M and PPO-C achieve scores comparable to or even slightly higher than those trained with standard PPO, highlighting that our calibration methods effectively preserve instruction-followinn abilities.
7

---

Published as a conference paper at ICLR 2025
GSM8K
SeiQ
CommonsenseQA
Methods
ECE
AUC
ACC
ECE +
AUC
ACC
ECE
AUC
ACC1
Llama3-8B
SFT
0.8608
0.5184
0.1221
0.0931
0.6067
0.873
0.2075
0.5889
0.7183
PPO
0.8843
0.5021
0.1099
0.0683
0.6507
0.911
0.1729
0.5815
0.7641
PPO
0.8954
0.5
0.1046
0.0958
0.5047
0.904
0.2222
0,5113
0.7748
PPO-M
0.8393
0.57
0.119
0.0267
0.6115
0.898
0.1206
0.5568
0.7707
PPO-C
0.8025
0.5343
0.1046
0.0319
0.5892
0.906
0.0457
0.5835
0.7699
SFT
0.4369
0.5138
0.5481
0.0944
0.65
0.856
0.1928
0.6155
0.7101
PPO 0.2566
0.5229
0.7392
0.0862
0.6763
0.879
0.1767
0.6287
0.7363
CoT
PPO
0.2553
0.5044
0.743
0.1265
0.5452
0.868
0.2654
0,5615
0.7191
PPO-M
0.1909
0.5499
0.7703
0.0392
0.6635
0.877
0.1555
0.579
0.7346
PPO-C
0.1546
0.5579
0.7635
0.0183
0.6473
0.868
0.1166
0.6049
0,7191
Mistral-7B
SFT
0.8628
0.5747
0.0902
0.0952
0.5877
0.882
0.1634
0.56
0.774
PPO 0.8675
0,583
0.097
0.0973
0,5497
0.89
0.1772
0,5594
0.7748
DA
PPO1
0.8851
0.5464
0.0877
0.1117
0.5439
0.885
0.1848
0,5674
0.7756
PPO-M 0.7963
0.5055
0.1016
0.0108
0.5090
0.888
0.1163
0,5303
0.7625
PPO-C
0.8161
0.534
0.0849
0.0399
0.5791
0.887
0,1311
0.5426
0,7592
SFT
0,4124
0.5277
0.5785
0.1124
0.6238
0.872
0,1908
0.6205
0.7518
PPO 0.4146
0.5228
0.58
0.1126
0.5794
0.877
0.1867
0,6238
0.7699
CoT
PPO
0.3932
0.5096
0.6035
0.1044
0.5693
0.885
0.2056
0.6135
0.7518
PPO-M
0,3379
0.5974
0,5982
0.0388
0.6584
0.886
0.1157
0.6118
0,7666
PPO-C
0.377
0.5641
0.6065
0.0848
0.6951
0.886
0.1311
0.6367
0.774
TruthfulQA
Object Counting
Professional Knowledge
Methods
ECE
AUC
ACC1
ECE 1
AUC
ACC
ECE
AUC
ACC1
Llama3-8B
SFT
0.4613
0.5506
0.4113
0.5054
0.5212
0.483
0.4308
0.5175
0.4798
PPO 0.425
0.5443
0.4651
0.508
0.4988
0.491
0.4078
0.4944
0.5046
DA
PPO1
0.5477
0.5246
0.4406
0.497
0.5
0.503
0.4951
0.4975
0.5009
PPO-M
0.3991
0.5813
0.47
0.4789
0.5227
0.505
0.3848
0.4926
0.502
PPO-C
0.3486
0.4856
0.4455
0.4405
0.5309
0.509
0.3318
0,5263
0.4798
SFT
0.4436
0.5745
0.4174
0.4545
0.5102
0.54
0.4644
0.5571
0.4242
PPO 0.4726
0.5851
0.4113
0.3651
0.5023
0.634
0.4309
0.5606
0.4635
PPO1
0.5535
0.5921
0.4076
0.337
0.5
0.663
0.5496
0,5219
0,4316
PPO-M
0.4283
0.5674
0.437
0.2863
0.5341
0.703
0.4329
0.5422
0.4424
PPO-C
0.3285
0.5193
0.4676
0.2525
0.5253
0.696
0.3798
0.5971
0.4353
Mistral-7B
SFT
0.3307
0.5755
0.5704
0.5083
0.4989
0.491
0.4134
0.5018
0.5031
PPO 0.3335
0.5567
0.5826
0.5008
0.5
0.499
0.4303
0.4889
0.4994
PPO1
0.3233
0.5651
0.601
0.5119
0.499
0.488
0.4571
0.4919
0.4872
PPO-M
0.245
0,5568
0.6071
0.4248
0.5067
0,483
0,3716
0.489
0,502
PPO-C
0.2679
0.5456
0.5887
0.4947
0.5242
0.484
0.3693
0.51
0.505
SFT
0.3657
0.6067
0.5398
0.4862
0.5072
0.512
0.4863
0.5369
0.4554
PPO 0.3677
0.5911
0.5581
0.4599
0.4991
0.54
0.4783
0.5275
0.4761
CoT
PPO*
0,3657
0.6089
0,5594
0.455
0.5022
0.543
0.4735
0,5215
0,4865
PPO-M 0.3142
0.6399
0.541
0.4134
0,5496
0.56
0.4090
0,5526
0.4579
PPO-C
0.3213
0.6108
0.5545
0.4344
0.5095
0.563
0.4248
0.5588
0.4731
Table 1: Performance comparison of SFT, PPO, PPO+, PPO-M and PPO-C across six datasets using Llama3- 3-BB and Mistral-7B. SFT denotes Supervised Fine-Tunel checkpoints, serving as the starting points for all methods. PPOf denotes an ablation of our PPO-M method which uses vanilla reward model in PPO training but on our modified dataset (with confidence-query system prompts).
8

---

Published as a conference paper at ICLR 2025
In contrast, PPOf exhibits inferior performance compared to both PPO and our proposed meth- ods. We hypothesize that this decline is primar- ily due to the reduced prompt diversity caused by the repetitive inclusion of confidence-quett system prompts during training. To validate this hypothesis, we conduct additional experiments analyzing the impact a higher proportion of iden- tical system prompts (See Appendix [..6p. No- tably, our analysis reveals that as the fraction of repeated system prompts increases, MT-Bench scores tend to decrease. These results consis- tently confirm a negative correlation the propor- tion of confidence- queryyyee-uuyuuueeeeee system prompts used in training and model performance on MTT-Benc.
Table Results on MT-Bench and Arena-Hard.
Model
Method
MT-
Arena-Hare
SFT
7.34
10.0
PPO
8.00
14,6
Llama3-8B
PPO
7.81
13,4
PPO-M
8.05
14.1
PPO-C
7.87
13.7
7.65
9.2
PPO
7.84
10.5
Mistral-7B
PPO
7.83
11.7
PPO-M
7.95
9.9
PPO-C
7.92
11.4
5.2 EXTENSION TO DPO
Setup. The CRM loss in Eq. 2 which calibrates the reward model using an augmented binary pairwise dataset, can naturally be extended to DPO training, as DPO models function as implicit reward models (Rafailov et 2024). We define this extension as Calibrated DPO (CDPO) in Eq.
Ccuro(To: Teet) = 2. -0..
llg (r( ye) - r V+)) +w(log (r(2, (yer ht) r(t, (yer ())) + log ( (yr: ()) - r (yr. h))))
where r(x; v) = log [ represents the implicit reward defined by model my and its reference model Tel- In this context, (Ye: h) and (yr,)) denote the model responses paired with high and low confidence, respectively, with subseripts indicating whether it is chosen or rejected response. represents the prompt prepended with confidence-uuee -uerry system prompt. w is the scaling coefficient.
=a(u(z)
The first rely subtle probability differences to effectively distinguish between chosen and rejected responses.
Eq. preserves the original DPO objective, preventing forgetting, since DPO models
We use the aattall7770 DPO version (i.e., waekknium//ppeeeeeeeree the reference model and NousResea wwarrchhoouusso Nous He reess s- 2 st.rall7B--10 as the DPO version) for the experiment. We fine-tune the DPO model on our RM calibration Dataset using Eq.
5- st
as
Results. As shown in Table CDPO sig-
Model
Method
MT-Bench t Arena-Hard
nificantly improves model calibration across all six datasets, achieving consistently lower ECE and higher AUC compared to other meth- ods. Notably, CDPO reduces ECE by over 50% on TruthfulQA, CommonsenseeAA, and Profes- sional Knowledge datasets. Although slight de- cline in performance is observed between CDPO and DPO CDPO still achieves performance comparable to the original DPO checkpoint, af- firming that calibration does not compromise overall model capabilities. Results on MT-Bench and Arena-Hart Hard presented in Table For Mistral-7E training additional data improves both MT-Bench and Arena-Hard scores, and CDPO further amplifies these gains compared to standard DPO the calibration dataset (DPOf). Results for Ll alaaa 8B provided in Appendix E.11
SFT
7.65
9.2
DPO
7.83
13.4
Mistral-7B
DPO
7.83
14.3
CDPO
7.85
15.9
Table 3: Comparison of DPO and CDPO on MT. Bench And Arena-Hard scores for Mistral-7B.
RELATED WORKS
LLM Calibration. Model Calibration aims to align model's confidence with its accuracy, Recent
studies show that LLMs often exhibit overconfidence (Tian et al. 2023 Chen et al. 2024a Xiong
et al. 2023 Achiam al. 2023). Previous studies have explored methods such as scaling based (eng
ct al. 2023 Guo et al. 2017:1 Zhang et al., 2020) approaches and nonparametric methods, such as
binning (Zadrozny & Elkan 2001). Recent work has introduced verbalized confidence (Lin et al.

---

Published as a conference paper at ICLR 2025
GSM8K
SciQ
CommonsenseQA
Methods
ECE
AUC f
ACC1
ECE 4
AUC 1
ACC
ECE
AUC
ACC1 1
SFT
0.8628
0.5747
0.0902
0,0952
0.5877
0.882
0.1634
0.56
0.774
DPO
0.8704
0.5916
0.0887
0.0845
0.581
0.892
0.177
0.5744
0.7682
DA
DPO
0.8057
0,5409
0.0826
0.0149
0,5215
0.884
0.1157
0,5491
0.7772
CDPO
0.6767
0.6163
0,0781
0.0967
0.7236
0.89
0.0513
0.6165
0.7666
SFT
0,4124
0,5277
0,5785
0.1124
0,6238
0.872
0.1908
0,6205
0.7518
DPO
0.4184
0.5253
0.5716
0.094
0.5837
0.896
0.1849
0,6145
0.7625
C6T
DPO
0.3456
0.5953
0,5989
0.0214
0.6687
0.898
0.0916
0.6553
0.7764
CDPO
0.1889
0.7178
0.6164
0.0553
0.7623
0.883
0.0676
0.6498
0.7633
TruthfulQA
Object Counting
Professional Knowledge
Methods
ECE +
AUC
ACC
ECE 4
1
ACC A
ECE +
AUC
ACC 1
SFT
0.3307
0,5755
0,5704
0,5083
0.4989
0.491
0,4134
0,5018
0,5031
DPO
0.2912
0.5725
0.6181
0,5149
0.501
0.485
0,4321
0.4967
0.4913
DA
DPO 0.2124
0.5674
0.6487
0.4336
0.5436
0.485
0.3649
0.5208
0.5091
CDPO 0.104
0.6225
0.661
0.3955
0.5304
0.491
0.2574
0.5451
0.4972
SFT 0.3657
0.6067
0.5398
0,4862
0.5072
0.5120
0.4863
0.5369
0.4554
DPO
0.3251
0.629
0.6022
0,4581
0,5003
0,5430
0,4950
0.5314
0.4609
C6T
DPO 0,2169
0.6176
0.6377
0.4037
0.5585
0.539
0.3679
0.5587
0.4961
CDPO
0.1756
0.685
0.6193
0.322
0.5139
0.553
0.2917
0.614
0.4817
Table 4: Performance comparison of SFT, DPO, DPO and CDPO across six datasets using Mi Mistral-7B. SFT and DPO denote the reference and trained DPO models, respectively. DPO and CDPO initiate from the trained DPO checkpoint; DPO applies standard DPO on the calibration dataset, focusing on chosen and rejected pairs to assess the impact of training with additional data.
2022). where models are prompted to directly output confidence scores. Most studies focus on pre-trained and instruction-tuned LLMs (Lin al., 2022 Han et al., 2024), while other studies examine RI RLHF-trained LLMs, proposing calibration through prompting strategies (Xiong et al.) 2023 Tian al., 2023). More recent work leverages Reinforcement Learning for calibration (Xu al. 2024 Tao et al. 2024). which aligns closely with our study. Our study contributes by identifying the potential cause for overconfidence in RLHF-LLMMs and proposing calibration of the reward models or reward score calculations to be seamlessly integrated into the existing PPO framework. In addition, our approach does not compromise the model's generalization capabilities in open-ended generation.
LLM Alignment And Reward Modeling. Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022; Christiano et al. 2017 Bai et al.] 2022) has been widely applied to align I.LMs with human preferences. This pipeline typically involves Supervised Fine-Tuning (SFT), reward modeling, and policy optimization using Proximal Policy Optimization (PPO) (Schul- man et al. 2017). Recent studies have explored variations of this pipeline to address noisy human preferences (Hong et al. 2022 Wang et al.) (024a) and to improve training efficiency by eliminating the need for separate reward model with Direct Preference Optimization (Rafailov al. 2024).
A comprehensive discussion of related works, including detailed analysis, is provided in Appendix A
7 CONCLUSION
This paper addresses the issue of overconfidence in RLHF-LLMs by identifying a systematic bias in reward models that favors high-confidence responses, regardless of their actual quality. To mitigate this bias, we propose PPO-M, which calibrates reward modeling by aligning confidence levels with response quality, and PPO-C, which adjusts standard reward model scores during PPO training. Both methods integrate seamlessly into the RLHF framework. Extensive experiments various benchmarks demonstrate the effectiveness of our approaches in reducing expected calibration error while maintaining accuracy and robust instruction followwng capabilities in open-ended generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
We thank the anonymous reviewers for their valuable and insightful feedback, which contributed to improving this work. This research supported in part by the NVIDIA Academic Grant Program.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, provide detailed information the datasets used (see Appendix C implementation details (see Appendix D and supplementary results and analysis (see Appendix
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.055 303.08774, 2023.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
Alvaro Bartolome, Gabriel Martin, and Daniel Vila.
Notus.
https://github. com/
argillaa Trgiilaaio/notuun 2023.
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, et al. Stable Im 2 1.6 technical report. arXiv preprint arXiv:2402.
2024.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(344)3224-45,, 1952.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08923 2024.
Lihu Chen, Alexandre Perez-Lebel, Fabian Suchanek, and Gaël Varoquaux. Reconfidencing Ilms from the grouping loss perspective. arXiv preprini arXiv:2402.009557 2024a.
Zhaorun Chen, Zhuokai Zhao, Zhihong Zhu, Ruiqi Zhang, Xiang Li, Bhiksha Raj. and Huaxiu Yao. Autoprm: Automating procedural supervision for multi-step reasoning via controllable question decomposition. North American Chapter of the Association for Computational Linguistics (NAACL), 2024b.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni. Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utrafeedback: Boosting language models with high-quality feedback. arXiv preprint ar arXiv:231000137. 2023.
Luigi Daniele and Suphavadeeprasit. Amplify-instrucs Synthetically generated diverse multi- turn conversations for efficient IIm training. arXiv preprint arXiv:(coming soon), 2023. URL
https::/huuginggfaee http:////...aaaaaacaeaaaaaaaaaaa wttp://cctaaaeeeaaaaaaaaaaa

---

Published as a conference paper at ICLR 2025
Ailin Deng, Miao Xiong, and Bryan Hooi. Great models think alike: Improving model reliability via inter-model latent agreement. arXiv preprint arXiv:2305.01481, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ArXiv preprint, abs/2304.06761 2023a. URL https:://aaxiyy:..sss///./
org/ abs / 2304 06767
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling online rlhf. arXiv preprint 2405::2405.078666 2024.
Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. Steerlm: Attribute conditioned sft as an (user-steerable) alternative to rlhf. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 11275-11288, 2023b.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988 6008. PMLR, 17-23 Jul 2022.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization, 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Haixia Han, Tingyun Li, Shisong Chen, Jie Shi, Chengyu Du, Yanghua Xiao, Jiaqing Liang, and Xin Lin. Enhancing confidence expression in large language models through learning from past experience. arXiv preprint arXiv:2404. 10315, 2024.
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and disttriiditti examples in neural networks. arXiv preprint arXiv:1610.02130 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024.
Joey Hong, Kush Bhatia, and Anca Dragan. On the sensitivity of reward inference to misspecified human models. arXiv preprint arXiv:2212.04711, 2022.
Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and bigh-performance rlhf framework. arXiv preprint arXiv:2405.11143 2024.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. ArXiv preprint, abs/2303.05398, 2023.
abs/
2303. 05398
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in changing climate: Enhancing Im adaptation with tulu 2, 2023.
Jiaming Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a human- peefeeeeeference dataset. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint ar trXiv:220..0522. 2022.
W Bradley Knox, Stephane dro Allievi. Models of human preference for learning reward functions. arXiv preprini arXiv:2206.000 022311, 2022.
Serena Booth, Scott Niekum, Peter Stone, and Alessan-
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213 2022.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprini arXiv:2302.0966 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Openorea: An open dataset of gpt augmented flan reasoning traces. bttps://https:// 2023.
huggingface..c co htt.a//e....ceeeeeeeorr//////aaa
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human 2021.
falsehoods. arXiv preprini
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv: 143344 2022.
Chris Yuhao Liu and Liang Zeng. Skywork rewaadd reward model series, httpsssstssssggggtttteteeeee co/ Skywork September 2024. URL http::ts///hggggggggeeee co/Skywork
co/
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly adversarial regularizer. arXiv preprint arXiv:2405. 16436, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference- free reward. arXiv preprint arXiv:2405. 14734, 2024.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucie. Revisiting the calibration of modern neural networks. Advances in 2021.
Neural Information Processing Systems,
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr and Puneet Dokania. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 5288--1115 2020.
Xavier Murias. Simple-math: 2+2=4 4-1=3, https://haggingfacceco// cottttssetssstssseeeaaaasasseee
fblgit/simple-ee 2024.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators, 2024.
13

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2998-3009, 2023a.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. ArXiv preprint, abs/2306. 17492, 2023b. URL /
17492
Aarohi Srivastava, Abhinav Rastogi. Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Carriga-Alonss et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.046655 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense@A: question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/N11-14222 1421. URL https www.laantholooo..or///3333
Shuchang Tao, Liuyi Yao, Hanxing Ding, Yuexiang Xie, Qi Cao, Fei Sun, Jinyang Gao, Huawei Shen, and Bolin Ding. When to trust Ilms: Aligning confidence with response quality. arXiv preprint arXiv:2404. 17287, 2024.
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305.1497. 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099: 0928889888 2023.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401 240000000 2024a.
Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of lims for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.185771 2024b.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406. 12444, 2024c.
Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi-atrrbbte helpfulness dataset for steerlm, 2023.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 24-337, 2022.
14

---

Published as a conference paper at ICLR 2025
Johannes Welbi, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint 1eti:::777.060000 2017.
Martin Weyssow, Aton Kamanda, and Houari Sahraoui. Codeultrafecdacck: An llm-as-a-judge dataset for aligning large language models coding preferences, 2024.
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis- Philippe Morency. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. arXiv preprint arXiv:2210.047144 2022.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian Hc, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306. 13063, 2023.
Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. Sayself: Teaching Ilms express confidence with self-reflective rationales. arXiv preprint arXiv:2405.20974, 2024.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, Zhenghao Liu, Bowen Zhou, Hao Peng. Zhiyuan Liu, and Maosong Sun. Advancing IIm reasoning generalists with preference trees, 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. ArXiv preprint, abs/2304.05302, 2023.
/ttp:::///xaiiii org wtt.ab///3330 05302
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Ieml, volume 1, pp. 609-616. 2001.
Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration deep learning. In International conference on machine learning, pp. 11117-11128 PMLR, 2020.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. ArXiv preprint, abs/2305. 2023.
URL https: /RRL/htts://aaax /tapss///rxivv org/ abs/ 10425
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond meeepreference-for-all Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023.
15

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
LLM Calibration. Model Calibration aims to align model's s confidence with its accuracy.
has been observed that modern neural networks. including Large Language Models (LLMs), often
exhibit overconfidence, suggesting poor calibration (Tian et al., 2023 Chen et al. 2024a Xiong
er al.
2023 Achiam et al., (023). Previous studies have explored methods like scaling-based (Dng
cct al., 2023; Guo ct al. [l.||201]] Zhang et al., 2020) approaches and nonparametric methods such as
binning Zadrozny & Elkan 2001). Among these, temperature scaling (Guo et al. [017 Zhang et al.
2020) has been proven to be effective when combined with large pre-trained L.M.s (Kadavath et al.
2022 Xiao al., [022 Kuhn et al. 2023). However, previous evaluations focus probabilities
derived from model logits (Hendrycks et al. 2020 Mukhoti et al.) [2000] Guo et al. 2017 Minderer
al. 2021>, ). which can sometimes be inaccessible proprietary models and unclear to human users.
Recently, verbalized confidence has been introduced (Lin et al. 2022), prompting models to directly output confidence scores alongside responses. While most studies focus on calibrating pre-trained LLMs through supervised fine-tuning (Lin et al. [0222 Han et al., 2024), which typically involves sampling responses and calculating average accuracy as the estimation for ground truth confidence scores, other studies have examined verbalized confidence in instruction fine-tume tne-tuned and RLHF -tainnedd
LLMs, and propose calibration through prompting strategies (Xiong et al. 2023
Tian et al. 2023).
More recent work leverages Reinforcement Learning for calibration (Xu et al. 2024 Tao et al. 2024)
which closely aligns with the focus of our study. We contribute by identifying a potential cause of overconfidence in RL-HF-trained LLMs and proposing calibration of the reward models or reward score calculations to mitigate this issue. The proposed methods can be seamlessly integrated into the existing PPO framework. Unlike supervised fine-tuning (SFT) methods, which require datasets with ground truth labels for accuracy calculation - limiting their applicability to open-ended generation tasks approach does not compromise the model's generalization capabilities in such settings.
LLM Alignment And Reward Modeling. Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al. 2022; Christiano et al. 2017 Bai et al. 2022) has been widely applied to align LLMs with human preferences. This pipeline typically comprises three steps: Supervised Fine-Tuming (SFT), the collection of pairwise ranking data and the development of a reward model, and optimization of the policy model obtained from the first step using Proximal Policy Optimization (PPO) (Schulman et al., 2017). The effectiveness of PPO depends heavily on the accuracy and robustness of the reward model. Following traditional Bradley-Tery reward models (Bradley & Terry 1952). training typically utilizes binary pairwise dataset. However, Nluman-labeled preferences are often noisy or exhibit conflicting signals (Hong et al., 2022, Knox et al.) 2022, Wang et al., 2024 To address these challenges, several methods have been proposed, including introducing margin to guide the reward model in assigning greater weight to more distinguishable comparison pairs ( Touvron et al. 2023 Wang et al. 20244)). and employing multi-objective reward modeling that
considers joint preference, such as "hel "fulnesss correctness, coherence". (Dong et al.] 20235 Zhou
al. 2023 Wang et al. 2024b; Chen et al. 2024b Chakraborty al. 2024 Wang et al. (024c).
Although the RLHF pipeline has proven effective in aligning LLMs with human preferences, Proxi- mal Policy Optimization (PPO) presents several challenges, including reward hacking, sensitivity to hyperparameters, and substantial computational resource demands, which complicate its implementa- tion and practical use. To address these challenges. various alternatives have been proposed (Dong
ct al.
2023al Yuan et al. 2023; Zhao et al., [203] Rafailov et al.) (2024) Song et al.) 2023b] Azar
lt al.
2023 Ethayarajh et al.] 2024 Hong et al. 2024 Liu et al.) [224]] Meng et al. (024). Among these, Direct Preference Optimization (DPO) has gained significant adoption (Rafailov et al. 2024
Dubey et al. 2024). DPO defines the preference loss as a direct function of the policy model, thereby eliminating the need for a separate reward model. However, despite these advancements, limited research has examined how reward models contribute to the confidence calibration of LLMs. In this study, address this gap by highlighting the vulnerability of reward models trained through different approaches, which can be casily biased by simply adding confidence scores. Furthermore, we propose two methods calibrate these models and effectively reduce overconfidence in RI trained L.L.M..
16

---

Published as a conference paper at ICLR 2025
B LIMITATION AND BROADER IMPACT
B.1 LIMITATION
While we demonstrate that directly applying CRM to DPO training can effectively reduce ECE and increase AUC, thereby improving model calibration, this method is not explicitly optimized for DPO. Our observations indicate some degree performance degradation, highlighting the need for future work to explore hyperparameter tuning or development of more specifically designed dataset.
B.2 BROADER IMPACT
Our work emphasizes model calibration, offering two methods that can be applied across a wide range of domains requiring well-calibrated language models. Improved model calibration enhances the reliability, trustworthiness, and safety of general Al systems, thereby benefiting the communities.
C DATASETS
In this section, we provide detailed descriptions of the datasets utilized in this study, including those used for the preliminary experiments, reward modeling, reward model calibration, and PPO training.
C.I PRELIMINARY EXPERIMENTS DATASET
In Section 2.2
demonstrate the inherent bias in reward models toward responses with high confidence scores by modifying modifying/allenall www.eaaleeeaward-bbaaataae
dataset. This modification involves prepending a confidence-query system prompt and appending random confidence scores to the original model responses. For this purpose, we assign scores ranging from 0 to 3 to indicate low confidence and 10 signify high confidence. Figure provides an example from the modified dataset on REJECTED. CONE, showing rejected response with appended low confidence scores.
Modified Reward Bench Example
Prompts:
For the following question, provide your best response first, followed by your confidence in accuracy helpfulness your response. Rate your confidence scale from to 10.
"Example Format:
<Your responses>
Confidence: <Insert your numerical confidence level from to 10, reflecting how helpful,>
you that your
your response strictly to Explicitly 'Cofiidence: in response.
User Prompt:
do car?
Response:
Datailing car involves washing the exterior and interior of the car, well polishing and waxing the exterior. Interior detailing typically involves vacuuming, cleaning the upholstery and vents, polishing the dashboard and console, and dusting, Polishing and waxing the exterior will depend on the condition of paint, but typically involvas applying polish and make it shine. Confidence:
Figure 7: example from the Modified Rewa roBench mode: REJECTED WITH CONF.
17

---

Published as a conference paper at ICLR 2025
C.2 REWARD MODEL TRAINING DATASETS
For Mistral-Tae -BB, Zeng 2024), an open-source pairwise binary dataset and train the reward model from scratch.
1 LLuu
C.3 REWARD MODEL CALIBRATION DATASETS.
In order compile the dataset for calibrating reward models, filter samples from multiple open-source datasets. Table the datasets utilized and the thresholds applied for each in detail.
Initially, filter out samples that multi-tum have tokenized length exceeding 8192, as multi- turn formats are unsuitable for assigning confidence scores, and truncation should be avoided. The threshold represents the preference strength (Wang et al. (024a), defined as the difference between chosen and rejected scores. In datasets such as RLHF Low Argi glggillaa Mat -etthh--P--a- st andard, preference strength below 1 often indicates that both chosen and rejected responses yield the same answer via different reasoning paths. Our objective is to calibrate the reward model to assign higher scores to high- high-confidence chosen responses and lower scores to high-confidence rejected responses, while reversing this pattern for low-confidence responses. However, when both responses produce the same mathematical solution through different reasoning, it is inappropriate and misleading for low- confidence rejected responses to receive higher scores. Consequently, exclude these ambiguous samples and retain only those with a sigmificant discrepancy between chosen and rejected responses. To balance computational resources, dataset. For datasets lacking specific chosen and rejected scores, we randomly select 2,500 samples.
threshold retain approximately 2,500 samples per
Dataset
Threshold
uugjlldlitilllllccccmmmmmn---/aa (Daniele & Suphavadeeprasit 2023)
RLIFFoowCCcodUnunaFccllassssaadd (Weyssow al.
2024)
Bartolome et
2023
3.5
Wang et
2023
2.5
RLHFFoow/tcllpaccr/ssssaaddddd Wang et al. 2024d
RLIIFow/OOncaddsiiblllessssaaaa Lian et al. 2023
2.0
RLHHUow/SHP-standdrd   (Ethayayra et al. 2022)
50
RLIIFoow/HHHRRLHHHHHllppppsssss (Bai et
2022
NA
RLITTowwAAgglaaaaatt-DD--taaaaa
(Ji et
2024
NA
NA
blgit/simple-mat--DOO Murias 2024)
NA
Table Dataset compositions.
C.4 PPO DATASETS
For PPO training, we filter out prompts with a tokenized length exceeding 8192 to prevent truncation and randomly select 20,480 prompts from RLHFlooe wwwpptprppp...o co wewpett--olleeeooooeoeee
Dong et al.) system
2024). To elicit verbalized confidence from the model, we integrate a confidence-quee prompt into single-turn prompts. The system prompt is included in 25% of the single-turn prompts for main results. Figure illlustrates an example from the dataset that incorporates this system prompt.
18

---

Published as a conference paper at ICLR 2025
PPO Prompts Example
System Prompts:
For the following question, provide your best response first, followed by your confidence in the accuracy helpfulness of your response. Rate your confidence on scale from to 10.
* EEammpe Format:
<Your generated response>
Confidence:
your numerical confidence level
10, reflecting that your answer is accurate helpful.>"
certain
that your response strictly adheres to this format. Explicitly include the word 'oofiiaace:: in your response. cell full
User Prompt:
Write me an excel function to sum the values in the cells in column to the left
coll
Figure 8: PPO Prompt Example.
C.5 EVALUATION DATASETS.
We examine six datasets encompassing six distinct categories: Arithmetic Reasoning. Common- sense Knowledge, Symbolic Reasoning, Truthful Reasoning, and Professional Knowledge Col- lectively, these datasets include mix open-ended generation tasks and multiple-choice questions.
GSM8K (Cobbe et al., 2021): This dataset contains high-quality, linguistically diverse grade school math word problems. We utilize the test split, which contains 1319 samples.
Commonsense TTlmor et al., 2019): This dataset features a multiple-choiee question format requiring commonsense knowledge. We use the test split, containing 1,221 samples.
TruthfulQA Lin et al., 2021 5 This dataset contains 817 questions designed test whether the model can generate truthful responses while recognizing false beliefs and misconceptions. We utilize the multiple-choice format of the dataset and consider one single target answer. To ensure the correct label is not predictably the first option, we randomly shuffle the answer options and corresponding true labels. We format the questions as lettered multiple-choiee and instruct the model to select the best answer from the options provided.
SeiQ (Welbl et al. 2017) This dataset contains crowdsourced science exams. We use the test split for evaluation, which includes 1000 examples. It is multiple-choice dataset, with each question offering four answer options. Similar to TruthfulQA and Commonsense(A. we assign a letter each answer option and request the model output the answer letter.
Object Counting in BigBench (Srivastava et al. 2022): BigBench is collaborative benchmark encompassing over 200 tasks. For Symbolic Reasoning, we focus on one subset, Object Counting, which includes 1000 samples. This open-ended generation task evaluates whether models can accurately determine the number of objects mentioned in the questions.
Professional Knowledge in MMLU (Hendrycks et al., 2020): MMLU multitask benchmark that includes multiple-choice format questions from diverse knowledge domains. For the Professional Knowledge category, we combine the test sets from four subsets: Pro- fessional Accounting, Professional Law, Professional Medicine, and Professional Teaching.
D IMPLEMENTATION DETAILS
In this section, we describe the implementation details for all experiments.
/wvve//vv//////// ewer/
1 wiippppchhicce
19

---

Published as a conference paper at ICLR 2025
D.1 REWARD MODEL TRAINING
This study utilizes two reward models. For Ll ama 3-m8, 8B, we use an off-the-shelf checkpoint from OpenRLHF mim For Mistral-77 the reward model is trained from scratch using www.eeeinnmmaaeeeeeeeee omeemiiiemm/peenerrmeeessss -2 2.2....isi Mistrall 7B, referred to as Mistral -B--RM
D.1.1 HYPERPARAMETERS
Parameter
Mistral-7B
Train Batch Size
512
Micro Batch Size
1
Learning Rate
2e-6
Max Length
8192
LRR Scheduler
cosine with min Ir
Warmup Ratio
0.03
Optimizer
AdamW
Weight Decay
0.01
Epochs
2
Table 6: Hyperparameters for Reward Modeling.
We list the detailed hyperparameters used for training wwtlsssral7B---- in T reference.
D.2 REWARD MODEL CALIBRATION
As stated in Section 3
we assume that reward models used for calibration are already trained beforehand and generally perform well. To this end, we utilize trained RM checkpoints, OpenRLHF/LLane 3- --------m-ii-uue-e-------------- and Mistral-7B- Battrrl-7BBBa for calibration. The calibrated versions of these models are referred L.l ama ---88---rn and Mistra B-crm, respectively.
D.2.1 HYPERPARAMETERS
Hyperparameters for calibrating a.l.mm3-8b-eee 1 are provided in Table
Parameter
Llama3-8B-crm
Mistral-7B-crm
Train Batch Size
256
256
Micro Batch Size
Learning Rate
9e-6
5e-6
Max Length
8192
8192
Scheduler
cosine_with.ngg wri
cosine Desine_with.mi..1
Warmup Ratio
0.03
0.03
Optimizer
Adam
Adam
Epochs
2
Table 7: Hyperparameters for Calibrating L1 ama 3-83 1---8B-rrr and Mistral -ereaal-----cr
D.3 PPO TRAINING
Following the standard RLHF pipeline, we initialize the policy model using corresponding super- vised fine-tuning
for L1 3- 3-88. 88B, and tekni um/ OpenHermes-22.. 2-M..55MMstaaa-777 for Mistral-7B. For standard PPO and PPO-C. we utilize the pre-calibrated reward models, specifically (OennLHFF /Llammaa ama3- 1----b-- wm-mixt---tt mixture and Mi stral 1 -B-RM In standard PPO, the reward score is obtained on EOS token of the sequence.
20

---

Published as a conference paper at ICLR 2025
For PPO-C, apply our proposed calibrated reward calculation method (see Section 3 for details). For PPO-M, we leverage Llama3 1- -- Bb-orm and Mistral- 7B-crm to calculate reward scores.
D.3.1 HYPERPARAMETERS
For each model (Ll 3 8B and Mistral-7B) we employ consistent set of hyperparameters across PPO, PPO-M, and PPO-C to ensure fair comparisons and reproducibility, as detailed in Table g
Parameter
Llama3-8B
Mistral-7B
Train Batch Size
64
64
Micro Batch Size
2
2
Mícro Rollout Batch Size
4
4
Rollout Batch Size
512
512
Prompt Max Len
1024
1024
Generate Max
1024
1024
Actor Leamning Rate
5c-7
1c-7
Critic Learning Rate
9e-6
le-6
Actor Weight Decay
0.0
0.01
Critic Weight Decay
0.0
0.0
Initial KL. Confidence
0.01
0.05
LR Scheduler
cosine_ hiiin_.with_mn....
Cosine_with_min
Warmup Ratio
0.03
0.03
Optimizer
Adam
Adam
Epochs
Table 8: Hyperparameters for PPO Training.
D.4 DPO TRAINING
In Section 5.2 we extend calibrated reward modeling (PPO-M) to DPO training using Eq. 4 Follow- ing the approach used for calibrating reward models, we leverage pre- trained DPO checkpoints.
For 1. 3 we utt.iiaeeeeeeeetoorrr///mmmmmmee checkpoint and ott.://eeetoorll////aaa--------- as the reference model. For as the DPO checkpoint, with 2ec.iiumnoogeenlerrmsse 2.5 serving as the reference model.
as the DPO
Mistral-7B, we use www.oo/sssarrc///ooooseeeeesssss
D.4.1 HYPERPARAMETERS
We list the hyperparameters used for DPO Llama- -3 Base 8B SFT-DPO in Table 9 The same set of hyperparameters is applied to both DPO and CDPO. However, is important note that the scaling coefficient utilized in DPO.
and
D.5 EVALUATION AND PARSING
In this section, we provide a detailed overview of the generation configuration, prompting and parsing strategies. All evaluations are performed on single Nvidia A100 80GB GPU with a batch size of 8.
D.5.1 GENERATION CONFIGURATION
We use consistent settings for both preliminary and main experiments: temperature at 1.0, top-p at 1.0, top-k at 50, with maximum token limit of 16 for direct answers and 256 for zero-shot CoT.
21

---

Published as a conference paper at ICLR 2025
Parameter
Llama3-8B
Mistral-7B
Train Batch Size
128
128
Micro Batch Size
1
Max Length
4096
4096
Learning Rate
3e-7
3e-7
Beta
0.01
0.01
Weight Decay
0.0
Scheduler
cosine min
cosine with min Ir
Warmup Ratio
0.03
0.03
Optimizer
Adam
Adam
Epochs
1
Zero Stage
2
Adam Offload
True
False
w (scaling coefficient)
1.0
0.5
Table 9: Hyperparameters for DPO and CDPO Training.
D.6 EVALUATION PROMPTS
Following the format described in Tian et al. (2023), modify the prompt to improve clarity and simplify the interpretation of the results. We consider two prompting strategies for evaluation: Direct Answer and Zero-Shot CoT (Kojima al. 2022). The exact prompt is shown in Fig 9 and Fig IO which also include a model response from GSMSK. For answer_tyse for maltiple-choice questions and number for open-ended math problems. For demo: we use (A) for multiple-choie questions and for open-ended math problems. Prompt formatting leverages the chat template the tokenizer. Instructions are placed the system prompt, and the question placed in the user prompt. For models like Tulu-2 (Ivison et al. (023). which lacks a system prompt section the tokenizer chat template, we append the question after the instruction as the user prompt.
we use opt letter
Prompt for Direct Answers
System Prompts:
the following question, provide your including only the (answer type) first, followed by your confidence in the accuracy or helpfulness of your your Please respond only with your answer and numerical confidence score. Do not include any additional text, characters, explanations. below for your response.
response,
demonstrated
'Example Format:
Answer: <Insert only the (answer type) here (a.g., [dama))>
Confidence: <Insert your numerical confidence level from to 10, reflecting how you
your
contain only (answer_type) and the confidence Explicitly include the words 'Answer:' and 'Confidance:" in your response.
your response strictly
User Prompt: :
Question: James decides
sprints times week. Be 60 meters each sprint. How many total maters does he run weak?
Model Output:
Answer: 540
Confidence: 10
Figure 9: Direct Answers Prompt.
22

---

Published as a conference paper at ICLR 2025
Prompt for Zero-Shot Co'T
System Prompts:
For the following question, provide a step-by-step explanation of your thought process and rate your confidence in the accuracy helpfulness of each respense on a scale from to 10. your response.
first, then offer your best
the format demonstrated
Example Format:
Explanation:
detailed explanation here, outlining how you
at your
answer.>
Answer: <Insert your concise answer here, which should include (answer_type) (e.g., (damo))>
Confidence: <Insert your numerical confidence level from to 10, reflecting how your answer correct.>
certain you
Ensure that your reaponse strictly adheres to this format. Explicitly include the words 'Explanation:', 'Answer:' and 'Confidence:" in your response.
User Prompt:
Question: James decides to run 3 sprints times week. He runs 60 meters each sprint. How many total meters does he week?
Model Output:
Explanation: James runs sprints of 60 meters each, and he does this 3 times week. Therefore, he
sprints 60 maters per sprint 180 maters por Since he does this 3 times a waak, he runs total of 180 maters/session sessions
meters
Answer: 540 meters
Confidence: 10
Figure 10: Zero-Shot CoT Prompt.
D.7 PARSING DETAILS
Regex Parsing Details. To parse the confidence score from model-generated responses, we im- plement a stopping criterion that triggers only when numeric digits directly follow the phrase "Confidence:" ". For responses that are initially unparseable, we set retry limit up to ten attempts. If parsing failures persist, we manually append "Confidence: to the model's response and resubmit it for completion, allowing the model to generate the missing score. This approach enables to achieve nearly 100% success parsing all responses. the rare instances where parsing ultimately fails,
empty string the default answer. Instead of assigning arbitrary confidence score of 5.0 - which could introduce bias and artificially inflate ECE - we use the most frequently observed confidence score from successfully parsed responses the default value. This approach ensures the assigned score could remain representative of the model's 's behavior and minimizes the risk of bias.
GPT-4o Evaluation Prompt
System Prompt:
specialized evaluator designad assess responses against golden for various tasks and extract model confidence. Output your evaluation in JSON format.
User Prompt:
Evaluate the semantic aquivalence between the given model response and the provided -. golden answer. Determine if they convey the same meaning. model response accurately
the golden answer (i.e., model response is correct), assign score of 1. If the model response does not match the golden answer, assign score Additionslly, extract the confidence
from the model response. If the model response does not explicitly state a confidence score, return -100.
Provide your in the following J8OM format: ('correctness"; 'cofiidene': X.X)
Figure 11: Prompts for GPT4-o Evaluation.
23

---

Published as a conference paper at ICLR 2025
GPT-40 Evaluation Details. We use gpt-4o-2024-08-06 evaluate zero-shot CoT results. Leveraging GPT's structured output feature, we configure the model to generate results in JSON format, enabling straightforward and efficient parsing. The prompt used for this is shown in Figure I1
E
MORE RESULTS AND ANALYSIS
E.1 OVERCONFIDENCE IN RLHF-LLMS
In this section, we present additional results from our preliminary experiments, demonstrating overconfidence in RLHF--tained LLMs across five datasets, shown in Figure [2 [6 These results show that RLHI--rained LLMs consistently exhibit verbalized overconfidence across datasets.
Accuracy Within Corfidence BigBench
Accuracy Bins Confidence on BigBench
1.0
Acturary Withi Bint
Accuracy Within
0.8
0.6
0.6
0.4
80.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0
0.00
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Confidence
Confidence
Accuracy Within vs, Confidence
Accuracy tull-2-0po-70
Confidence BigBench
1amme3--0o-Ihf-100
Accuracy Within Bins
Accuracy Within
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.00
0.1
0.2
0.3
0.4
0.6
0.7
0.9
1.0
0..0
0.2
0.4
0.5
0.6
0.7
0.8
0.9
Confidence
Confidence
Llama3-8B-S3 and
Tulu-2-7B and Tulu-2-DFO-77
Figure 12: Confidence distributions of models on ObjectCount before (top) and after (bottom) RLHF.
Accuracy Within Bins vs. Confidence on GSMBK for (lama-3-9--S--mixttue
Accuracy Within Bins vs. Confidence on GSMBK for tulu-2-76
2.0
Accuracy Within Bins
1.0
Accuracy Within
0.8
0.2
0.2
0.0.0
0.1
0.2
0.3
0.4
0.6
0.7
0.9
1.0
8..0
0.1
0.2
0.3
0.4
0.5
0.8
0.9
Confidence
Confidence
Accuracy Within Bins vs. Confidence on GSMBK for Llama-3-fo-lhf-1100
Accuracy Withnn Bins vs, Confidence on GSMBK for tull-2--po-77
1.0
Accurary Within Birs
1.0
Vithin
0.8
0.8
0.2
0.2
0.0.0
0.1
0.2
0.3
0.4
0.5 Confidence
0.6
0.7
0.8
0.9
1.0
8..00
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Confidence
Llama3 8B-SFT and Llama3 8B 8B-PPO;
Tulu-2-78 and Tulu 2-02--DDPP---
Figure 13: Confidence distributions of models on GSMSK before (top) and after (bottom) RLHF.
E.2
REWARD MODELS BIASED TOWARD HIGH CONFIDENCE SCORES
Following Section
2.2
present additional results to further substantiate the observed phenomenon.
24

---

Published as a conference paper at ICLR 2025
Accuracy Within Bins vs. Confidence professional for Llame-3--b--tteeeee tere---
Accuracy Within Bins vs. Confidence on professional for tulu-2-76
1.0
Accuracy Within Bins
1100
0.8
0.4
0.4
0.2
0.2
8.0
0.1
0.3
0.6
0.7
1.0
0.00
0.1
0.2
0.4
0.5
0.8
0.9
Confidence
Confidencce
Accuracy Within Bins vs. Canfidence on professional for -lama-3-Bb-rhf-100k
Accuracy Within Bins vs. Confidence on professional for tull-2-dpo-78
1.0
Accuracy Within Birs
1.0
Vittin
0.8
0.2
0.2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
88.3
0.1
0.2
0.3
0.4
0.5 Confidence
0.6
0.7
0.8
0.9
1.0
Confidence
L1 Ulama3- 3-8B---TTTTT and 1mama3 3- 8B-PPO;
Tulu-2-7B and
Figure 14: Confidence distributions of models on Frowl before (top) and after (bottom) RLHF.
Accuracy Llama-3
Confidence SclQ
Accuracy Bins Confidence SciQ
1.0
Acturary Within Bing
Within
0.8
0.8
0.6
0.6
0.4
00000
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Confidence
Confidence
Accuracy -lamma---0--ih--1000
Confidence
Accuracy Confidence Sciq
Within Blns
Bint
tulu-2-dpo-7b
0.8
0.6
0.6
0.4
Accuracy
0.2
00.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.00
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Confidence
Confidence
1.lama3-8B-SF and L. amaaa3--mmmmm -88--88PPP---
Tulu-2-78 and Tulu-2-DPO-7R
Figure 15: Confidence distributions of models on SciQ before (top) and after (bottom) R.HF.
Accuracy Within Bins Confidence TruthfulQA for Llama-3-98 sit-mtttiiitture
Accuracy Within Bins Confidence TruthfulQA for tulu-2:7b
Accurtty Within Bins
Accuracy Couracy/oo Wein
0.6
0.6
0.4
0.4
0.2
0.2
0.0.0
8.00
Accuracy Within Bins Confidence on TruthfulOA
Accuracy Within Bins vs. Confidence TruthfulQA for tulu-2- 2ull-2--pp-75
1.0
for Llama-3-8h riht-.kk-
Accurary within Birs
Accuraty Within
0.8
0.4
0.4
0.2
0.2
0.0
0.3
0.6 Confidence
0
0.2
0.5 Confidence
and L. 1llama3-880 -ama------PP---
Tulu-2-7B and Tulu-2-DPO-7B
Figure 16: Confidence distributions of models on TruthfulQA before (top) and after (bottom) RLHE.
25

---

Published as a conference paper at ICLR 2025
Comparioo Preferance Over Responsess for Ammoit--ilaaa-BBBoo
Compar lion Pruferunce Over Responses
for Ar Armmil---aamaa--e 3-80mm8-0....
prefer high-confideene rejected prefer low onfidence choven
prefer Hightcorfider rejected prefer lom-corffderee choset
confidence reversed
1754
13 98
confidince
122
2852
reversed
prefer high confidarce
profer low confidiecce
profer high confidence
profer low confidence
chosen with cont
2884
80
chosen with ctor
1736
1098
preter confidence
confidence
prefer high confidence
preter confidence
with conf
2598
374
with cont
1530
1358
prefer rejected
choser
peofer rejected
answer 231
2737
answer, 2311
27:37
5ō0
1900
1500
2000
2500
500
1000
1800
2000
2500
(a) RIAK ow ow/ArmoRM 1. amm.3 8 8--00
Wang et al.
2024c with (left) and w/o (right) conf.-query
prompt.
Comparieon of Preference Dver Responses
Comparison of Preference Over Resporses for STAM Qwen2
STRM Qwen 0613
prefer high-conliderce rejected prefer Iow-cortidence chosen
prefer high-colieene rejected prefer low-contiience chosen
confidence reversed
1842
1142
confidence reversed
1156
1827
profer high confidance
profor low confidence
prefer high confidence
prefer low confiidncc
th cont
chesen.
360
with with_cort _cort
chosen
2242
742
prefer high confidence
prefer low confidence
prefer high confidence
prefer low cor fidence
with conf
rejected
2392
922
rejected with: .com
2225
756
profer rejected
profer chesen
prefee rejected
profer chosen
721
2264
721
2264
only
only
50g
1000
1500
2000
2500
500
1000
1500
2000
2500
(b)
CIR AMS//MM/RA / BB////TM____U QQwen__7b_00 261333 with (left) and w/o (right) conf. prompt.
Comparison Preference Responses Burus-1M4-75
Comparison Preference Resporses for Buru--MM-7
prwfer high-confiderce rejected prefer los-corfidence chosen
prefer High-cnnfidereee rejected
prefeer loo-condiidence chosen
confidence
reversed
1566
1438
confidence reversed
577
2407
prefer confidarce
confidencce
prefor high confiidncce
profer fidence
with
chosen
2765
219
with
choson
2069
909
prefer high confider-ce
prefer low confidencce
prefer high confidence
prefer fidence
rejected
2542
442
rejected with_cort
1859
1:14
prefer rejected
prefer chosen
prefer rejected
prefer chosen
aniser
answer cnly
ony
500
1000
1500
2000
2500
1000
2000
(c) openbmb/ Eurus- RM-7b Yuan et al.
2024 with (left) and w/o (right) conf-query prompt.
Comp arison Preference Over Responses
aomparisondr of Preference Over Resporses for FstiiX-LLMMAA-AMe
for MMMX--LLMMA--M--O..
preter high contidence rejected prefer low confidence chosen
prefer High high-contideeeeeee confideneedee rejected prefer low con--coniidecce chosen
cond coniidence
confidence reversed
reversed
prwfer High confidence
prefer low confidence
profer high con fideidence
prefer low con Sidence
choser. aith_ cont
285
chosen. with with_cort _cort
2381
593
profer high confidence
profer low confidence
prefer high confiidecc
profer low cor fidence
relerted with conf
2593
392
rejected with, conf
2055
907
prefer rejected
preter chosen
prefer rejected
prefer choson
only
303
2592
only
303
2592
550
1000
3500
2000
2500
500
1000
1500
2000
2500
(d)
Dong et al. 2023a with (left) and w/o (right) conf--query
prompt.
Comparison Proference Over Responses
Comparison Proference Over Resporses for lleme-S-fe-mm-1ode
profer high confidence rojected profer confiderce chosen
profer high confidence rejected
profor confidence choson
confidence
1445
confidence
584
2392
profer hrgh confidence
profer low confidence
profer high fidence
prefer confidince
choser
chosen
2078
with
with conf
proter high confidence
profer low confidence
prefer high con confidence
prefer low confidonce
with_ cont
rejected
rejected with. with_cort _cort
2063
B20
prefer rejected
prefer choven
peefer rejected
prefer chosen
answer
450
2531
ansswer only
450
2531
only
500
1000
2000
1000
2000
(c) OpenRLKF/ LLaaaa (aaaaaa--800 3-8b- rm- 700k
Hu ct al. 2024 with (left) and w/o (right) conf. conf--query prompt.
Figure 17: Preference Distributions for various reward models across four modes (Part 1). The left follows the same setting in preliminary experiments, while the right represents the setting where all confidence-query system prompts are removed, and only random confidence scores appended.
26

---

Published as a conference paper at ICLR 2025
Comparison of Praference Over Buspories
Comparison Preferunce Over Nesporses
for weworr--eearr--llmaee 3.188
for Skyaork-fwearr-Laa Sww.ar--laaaaa...... 3.188
prefee ligh-confiderce rejected prefer bencorfiddnce chosen
preferr Highsconfidence rejected prefier loo-confiience olvee
confidence reversed
1540
1445
confidince reversed
YM
2650
prefer high confidarcce
profor low confidencce
prefor high confidencc
profer low cor fidencc
chosen with cont
2716
267
chosen with .oot
1949
1031
prefer confidencce
confidencce
preter high confidence
prefer can coneiiencce
with conf
2437
545
with cont
1760
1206
profer rejected
profer rejected
prefer
answer 198
2786
answer 198
2786
500
1000
1500
2000
2500
500
1000
15:00
2000
2500
(a) Sk Skywork / RewaRdwwarddloeee ama 3.. 1-81 query prompt.
[Li & Zeng 2024) with (left) and w/o (right) conf.-
Comparison of Preference Duer Responses stableim 1221122--0 chat
Comparison Preference Over Resporses stab i0l.lemm22. 126
prefer high-contiserce rejected prefer lov-confidenne chosen
preter High-conience rejected prefer low noe-coniiencce chosen
confidence
801
confidence reversed
reversed
profer high confidorcce
profor low confidence
prefer high confidence
prefer low confidence
chosen.
2188
788
chosen with with_cort _cort
th cont
prefer high confidencc
prefer low card ieence
prefer high confidence
prefer low confitence
with conf
rejected
1992
982
rejected with: .con
1147
1810
profer rejected
profer chesen
prefer rujected
prefer chesen
only
570
2412
only
570
2412
500
1000
1500
2000
2500
500
1000
1500
2000
2500
(b) sttabiiiiii www.lllttyitsttatte Im 2 t [Bellagente et al.
2024) with (left) and w/o (right) conf.-
query prompt.
Comparisor Preference Dvet Besponses for Iulo-2-ppo-7
Comparison Preference Over Resporses for tulo-2-ppo-26
high-confiderce rejected
lne-corfidenre
High-confide-nee
lso-confidence
confidence
1063
confidence
938
2047
prefer High confiderice
profer low confidence
profer high con confidence
prefer Edence
chosen.
1205
chosen
1851
1134
profer high confiderce
profor low confidence
prefer high confidence
prefer low conficence
rejectd
rejecte
with
with conf
prefer rejected
prefer chosen
prefer rejected
prefer chosen
answer
2240
answer only
22:40
only
5ón
1000
1500
2000
2500
1000
15500
2000
(c) alll
7b
(Ivison et al.
2023) with (left) and w/o (right) conf.-query prompt.
Comparissn Preference Responses for UttrRM-133
Comparison Preference Resporses for JitraRMM 13b
prefer high cont-ittncceeee rejected
preter low confidence chosen
proter high confidence rejected
profer low confidence chosen
confidence
reversed
1028
1955
contidence reversed
936
2042
prefer high confidercce
prefer low corfidence
prefer high confidence
prefer low confidence
chosen with cont
1610
1289
chosen with .comt
1490
1388
prefer confidencce
prefer low confidence
prefer high confidence
prefer confisence
with ronf
1615
1301
with .corl
1389
1498
profer rejected
profeer
prefer rejected
chosen
answer
985
2014
answer
965
2014
500
1000
15:00
2000
2500
500
1000
1500
2000
2500
(d)
open.bbb /ttttuttrarrs 13b Cui et al.
2023 with (left) and w/o (right) conf. -uery prompt.
Comparison of Preference Over Responses
Comparison Proference Over Resporses
profor high confidencc rejected
profer low confidence choson
profer high confidence rejected
profor confidence di choson
confidence
1550
1435
confidence
2510
profer high confidarice
profer low confidancc
prefer high fidencc
profer Sidence
choser
2673
312
chosen with conf
2026
profer high confiderce
profor low confidence
prefor high confidence
prefer low con fodence
rejected with_ cont
2437
545
rejected with. with_cort _cort
1894
1099
prefer rejected
prefer chosen
peefer rejected
preferchosen chosen
answer 247
2734
ansswer
247 only
2734
only
500
1500
2000
1000
2000
(c) LazGordon/ /URM- LLaMa-3 3. 1-88 -B Cui et al. 2023) with (left) and w/o (right) confi-query prompt.
Figure 18: Preference Distributions for various reward models across four modes (Part 2). The left follows the same setting in preliminary experiments, while the right represents the setting where all confidence-query system prompts are removed, and only random confidence scores appended.
27

---

Published as a conference paper at ICLR 2025
Comparicon Preference Over Responses for Liama-3 (aam----b---mmaae --b--mmiature
Compar koo Preferunce Over Responses for Liana-3----r--iee
prefer higheoonfidenc dence rejected prefer low-cord idence onennceee chasen
prefer Highsconfidence rejected
prefer Ine-corffderce chosen
confidence reversed
1542
1440
confidince reversed
2211
profer high confidence
profer low confidence
prffer high confidence
profer low confidence
chosen with cont
2908
76
chosen with .com
2731
223
preter confidence
prefer confidence
prefer high confidence
preter confidence
with conf
2785
177
with cont
2539
285
prefer rejected
prefer chosen
profer rejected
chowen
answer
514
2460
answer
514
2460
500
1000
1500
2000
2500
500
1000
1500
2000
2500
(a) Open RIRHE/IHHIIIII /1.1 ama 3- 8b 1 rm xture
Hu et al.
2024 with (left) and w/o (right) conf.qqeery prompt.
Comparisor Preference Dver Resporses Unmaa- D-te----tteettia------
Camparison Preference Over Resporses hiw.s--ftttBBssAA---.a
prefer high-confiderce rejected
inw-cortidence chosen
prefer nigh-conlidenne rejected prefer
chosen
confidence
120
1163
confidencee
449
2535
profer High confidercce
profer low confidancea
profer hight cor fidance
prefer fidence
with_cont
Chosen.
2781
204
with_
chosen.
profor high confidance
profor low confidence
profor high confidencc
prefer low con fidence
with conf
rejected
rejected with corf
1876
1083
profer rejected
profer chosen
profoo rojucted
prefer choson
answer
2709
answer only
27:00
only
500
1000
2000
1000
2000
(b) NCSOFT/ Llama-3----- 3-offfotBifasssss-------- RM-8B
Park et al. 2024 with (left) and w/o (right)
prompt.
Comparisar Preference
Comparisan Preference calll-mm-2 2-ad--m---1--pyttiee L-pyttia-i.A--pppeee Ab-eeiibbbbbbhhee b-2.5
for -assll-m-2. J-tylhia-l L.bb-ppoch-25.
prwfer high- oofidheeeeeeeeeeee rejected prefer Ins-conficerce chosen
prefer High-confidence rejected
prefer low-c lon-ccnfideenc choser
confidence reversed
1347
1633
confidence reversed
1230
17:69
profer confidarice
profor confidence
profer confidence
profer confidence
with
chooen
2184
795
with
choson
2213
754
prefer high confidercce
prefer low dence
prefer high confidence
profer confidence
rejected
2260
rejected
2276
with_
with with_cort
prefer rejected
trefer chosen
prefer rejected
prefer chosen
ariswer
answer cnly
ony
500
1000
1500
2000
2500
1000
1500
2000
(c) OpenAss www.sssssttttttttto aacan///aaaee wesaassttrm--..0 rm- 2tt.-tt----1111------------ 4b 4b-epoch -2
with (left) and w/o (right) conf.-
query prompt.
Comparisse Preference Responses for Ztya LLaMA RB-Re
Comparison Preference Over Resporses
for Zlya-LLaMA- 7B Reward
prefer high-contid con--denniddee rejected
preter low confidence chosen
prefer high confidence rejected prefer low confidence cf ostn
confidence reversed
1555
1428
contidence reversed
1145
1839
prefer high confidercce
profer low confidence
welfer high con cenfidence
prefer con fidence
chosen with with_com conn
2689
295
chosen with .comt
1821
1134
prefer confidence
prefer confidence
prefer high confidence
prefer fidence
with ronf
2669
315
with .cor
1922
1141
prufer rejected
chosen
prefer rejected
prefer
answer
1026
1961
answer
1026
1963
500
1000
1500
2000
2500
500
1000
15:00
2000
2500
(d) IDEA-CCNNL LLLaa--LLMM---. -Rewaa Cui al.
2023 with (left) and w/o (right) conf.-query prompt.
Compartpon Preforence Over Responses intemmm2--00-
Comparison Proference Over Resporses (0000-0-----wweee
proferr high- confidencee rojected
profor confidence cheson
profer high confidence rejected
profor confidencc di ossn
confidence
1151
1033
confidence
2408
prefer high confideacce
profer confidunce
prefer high fidence
prefer Sidence
choser
2829
chosen
with
with conf
prefer high confiderce
prefer low confidonce
prefor high confidence
prefer low confidence
with_ cont
rejected
2467
with. with_cort _cort
rejected
2220
760
prefer rejected
prefer chosen
peefer rejected
chosen
answer
only
48
2736
ansswer 248
only
2736
500
1000
1500
2000
1000
2000
(c) internlm/ inteenneeelmm Internim/internnn2
rewadd with (Ieft) and w/o (right) conf.-query prompt.
Figure 19: Preference Distributions for various reward models across four modes (Part 3). The left follows the same setting in preliminary experiments, while the right represents the setting where all confidence-query system prompts are removed, and only random confidence scores appended.
28

---

Published as a conference paper at ICLR 2025
A concern arises that the reward model may be influenced by the inclusion of the confidence- queryy system prompt, which designed to ensure the model verbalizes its confidence level. To investigate the impact of this system prompt, conduct additional experiments with and without its inclusion.
As shown in Figure I7 and 19 the plots on the left follow the configuration outlined in prelimi- nary experiments, where confidence-query system prompt is prepended and random confidence scores are appended to model responses. These plots clearly demonstrate that tested reward models exhibit biased preference towards high-confidence responses, with the degree of bias varying across models. On the right, we evaluate four modes, but this time without the confidence-query system prompts, and only random confidence scores are appended to the model responses. For example, in REJECTED WITH CONF, the comparison involves the same chosen responses with high confidence score versus a low confidence score. The results reveal similar phenomenon, although the bias more subtle in this setting, indicating the potential influence of the confidence-ques system prompt.
E.3 CALIBRATED REWARD MODELS
highlights the preference distributions of our calibrated reward model compared to the pre-calibrated version for 1-ama3-88 complete set results and extend the analysis to include the Mistral- model, providing more comprehensive evaluation of the calibrated reward models' performance across different architectures.
Section 4.2
In this section, present the
Comparison of Preference Over Chosen Responses with High/Low Confidence Scores
Comparison of Preference Over Rejected Responses with High/Low Confidence Scores
Uama- 1amna-3-8o-m--a -nn--m-mmixua
Llama-3-8ol 1-mma-3---mmmmtturr Llama3-0b-cm (ours)
Uame3-8b-crm (ours)
2000
Count
Count
1000
Prefer High Confidence (High Low)
Prefer Low Confidence (Low High)
Prefer High Confidence (High Low)
Prefer Low Confidence (Low High)
WHOSEN_WITH_COMM
WIIECTEDWITTHCONT
Figure 20: Comparison of preference distributions between the calibrated reward model
L. a.lama- -0b--8b-- 8b-88b-rrr rm and the pre-calibrated version 1.1 amlam--i - 8---8----------- xture on two modes.
Comparison Preference Over Chosen Responses with High/Low Confidence Scores
Comparison of Preference Rejected Responses
with High/Low Confidence Scores
mistral-72 7o-hemmes---sskyeeree
missrrl-7h-heemee-nn htter/th-hee------eee
mistral- Jb.cmm cmm lours)
2000
mistral- 7b.cmm (ours)
2000
Count
Count
1000
1000
Prefer High Confidence (High Low)
Prefer Low Confidence (Low High)
Prefer High Confidence (High Low)
Prefer Low Confidence (Low High)
CHOSEN CHOSEN_WITH_COO WITH.CONF
REJECTED WITH
Figure 21: Comparison of preference distributions between the calibrated reward model Mistral-7B-cree and the pre-calibrated version Mistral-7B-RM on two modes.
As shown in Figure 20 and 21 both calibrated models exhibit similar trend. When evaluated on chosen responses with high and low confidence scores, the calibrated reward models are less certain than their pre-calibrated counterparts. Additionally. when evaluated rejected responses with high and low confidence scores, both calibrated models show a preference for low cow-confidence responses, indicating improved capability of calibrated models in identifying overconfident model responses.
29

---

Published as a conference paper at ICLR 2025
E.4 VISUALIZATION OF THE CONFIDENCE DISTRIBUTION
Confidence Hiltogram
Confidence Distribution
Samples scale!
Ulama3 Llama3-Bp Iruchfullw
Llama-3
cdpo TruthfulQA
Lamaa 8b
L'ma Liama-3-0-ccpp
Liama3-b-spp-m
10-1
10
20-4
10
6.
0.5
0.6
0.9
0.7
Confidence
Confidence
PPO and PPO-M
DPO and CDPO
Figure 22: Confidence distributions of PPO and PPO-M (left) and DPO and CDPO (right).
In Figure 22 we present the confidence distributions of the PPO and PPO-M models on the left, and the DPO and CDPO models on the right. Notably, the confidence distribution for PPO-M is slightly shifted the left relative to PPO, indicating a reduction high-confidence scores (e-g., confidence level 10, representing a highly overconfident state) and an increase in lower-confiee confidence categories. For CDPO, this phenomenon even more pronounced; compared to DPO, the confidence distribution of CDPO is more dispersed across categories, with noticeable increase in lower-confidence levels.
E.5 MODEL LOGITS FOR CONFIDENCE SCORES
presents the density distribution of numbers 0 to 10 based on the log probabilities extracted from model responses on the TruthfulQA dataset for both PPO and PPO-M models. Specifically, we forward the model responses and examine the log probabilities at the position corresponding to the original confidence score within the response. We then analyze the log probabilities of other numbers at the same position. The figure reveals that certain numbers exhibit notably high density, For instance, the PPO model exhibits high density for the number 10, while the PPO-M model favors the number 9. This non-uniform distribution of log probabilities indicates that the model does not generate numbers randomly at the confidence score position but instead favors specific numbers.
Figure 23
Distribution
11ama3-8b-190-4
Probabilties Numbers TruthfulAA
for Numbers 1-10 TruthfulGA
0.5
1.0
15
10
Probabiity
0.2
22
15
10
PPO
PPO-M
Figure 23: Density Plot of LogProb for Confidence Scores for PPO and PPO-M on TruthfulQA.
E.6 PARAMETER SENSITIVITY
In Eq. 3 we introduce a reward adjustment factor y, defined as 7 w (f: Art) (s; 0.5). Here w represents a scaling coefficient set to 2.0 in our main results. To evaluate the impact of w, we conduct a hyperparameter sensitivity study, detailed in this section. The results, presented in Table TO reveal clear positive correlation between calibration performance and w, and negative correlation between model cestruction-oollwinn performance and w. This demonstrates trade-off' between calibration effectiveness and model instruction-folloovnng capabilities w increases. Increasing from 0.5 to 2.0 significantly enhances calibration performance, as indicated by a decrease in ECE. However, this improvement is accompanied by slight reduction in MT-Bench and Arena-Har scores. Based on our primary focus on confidence calibration, we select w = 2.0 for the main results.
30

---

Published as a conference paper at ICLR 2025
GSM8K
SciQ
CommonsenseQA
MT/Arena-Hard
ECE 4
AUC
1
ECE L
AUC
ACC
ECE +
AUC
ACC f
0.5
8.03 14.7
0.8792
0.521
0.1099
0.0703
0.6031
0.896
0.1552
0.5678
0.7674
1.0
7.91 13.8
0.8238
0.4937
0.119
0.0087
0.578
0.898
0.1153
0.585
0.7625
2.0
7.87 / 13.7
0.8025
0.5342
0.1046
0.0319
0.5892
0.906
0.0457
0.5835
0.7699
TruthfulQA
Object Counting
Professional Knowledge
MT/Arena-Hard
ECE +
AUC
1CC
ECE J
AUC
ACC
ECE + AUC
ACC f
0.5
8,03 14.7
0.4428
0.5549
0.4553
0.4856
0.5036
0.512
0.4286
0.5027
0.4906
1.0
7.91 13.8
0.4104
0.515
0.4492
0.4774
0.5118
0,496
0.383
0.509
0.4902
2.0
7.87 / 13.7
0.3486
0,4856
0.4455
0.4405
0.5309
0.509
0.3318
0.5263
0.4798
Table 10: Performance of PPO-C with different coefficient on Llama 3 3-BB. Prompts: DA.
GSM8K
SciQ
Commonsense
MT-Bench
ECE 4
AUC
ACC +
ECE 4
AUC +
ACC
ECE 4
AUC 1
ACC
7.97
0.8832
0.1168
0.0967
0.5244
0.902
0,2251
0.5111
0.7715
0.1
7.87
0.8025
0.5343
0.1046
0.0319
0.5892
0.906
0.0457
0,5835
0.7699
1.0
7.97
0.8658
0.5009
0.1114
0.0373
0.6426
0.905
0.0821
0.5646
0.7756
TruthfulQA
Object Counting
Professional Knowledge
MT-Bench
ECE +
AUC
ACC
ECE 4
AUC
ACC
ECE 4
AUC
ACC
0
7.97
0.5502
0.5332
0.437
0.4947
0.501
0.505
0.4877
0.4985
0.5072
0.1
7.87
0.3486
0.4856
0.4455
0.4405
0.5309
0.509
0.3318
0.5263
0.4798
1.0
7.97
0,3846
0.524
0.4443
0.4899
0.4985
0.506
0,381
0.52
0.4728
Table 11: Difference-Battn PPO-C with different for Ar on Llama3-8B. Prompts: DA.
GSM8K
SciQ
Commonsense
MT-Beech
ECE 1
AUC
ACC f
ECE |
AUC
ACC1
ECE
AUC
ACC f
7.79
0.8833
0.5034
0.116
0.1056
0.5238
0.891
0,2178
0.5568
0.7649
0.1
8.05
0,8638
0,516
0.1031
0.0282
0.6513
0.904
0.1286
0,5621
0.7756
1.0
8.03
0.8827
0.5112
0.1145
0.0849
0.5493
0.907
0.1992
0.5632
0.7625
TruthfulQA
Object Counting
Professional Knowledge
MTT-Bench
ECE +
AUC
ACC
ECE 4
AUC
ACC
ECE
AUC
ACC
0
7.79
0.5185
0.5655
0.4394
0.4948
0.498
0.505
0.4753
0.5119
0.5024
0.1
8.05
0,4426
0.5303
0.4431
0.4839
0.5178
0,503
0,3949
0.4902
0.502
1.0
8.03
0.4965
0.5595
0.4333
0.4797
0.5011
0.52
0.4614
0.4968
0.4935
Table 12: Threshold-Based PPO-C with different for Ar am -B. Prompts: DA.
GSMSK
SciQ
Commonsense
Percentage
MTT-Bench
ECE 4 AUC f
ACC
ECE 4. AUC
ACC 1
ECE 4 AUC
ACC f
0.25
8.05
0.8393
0.57
0.119
0.0267
0.6115
0.898
0.1206
0.5568
0.7707
0.5
7.88
0.86
0.5185
0.1031
0.0389
0.5829
0.896
0.134
0,5399
0.7682
1.0
7.74
0.8608
0.5065
0.1243
0.0471
0.7165
0.898
0.074
0.6341
0.7658
TruthfulQA
Object Counting
Professional Knowledge
Percentage
MT-Bench
ECE 4
AUC f
ACC 1
ECE
AUC
ACC
ECE 4. AUC ACC t
0.25
8.05
0.3991
0.5813
0.47
0.4789
0.5227
0.505
0.3848
0.4926
0.502
0.5
7.88
0.4453
0.5283
0.4357
0,5119
0.5413
0.473
0.3988
0,5221
0.4935
1.0
7.74
0.3438
0.5737
0.4786
0.5087
0.5052
0.487
0.3501
0.5184
0.502
Table 13: Performance of PPO-M on downstream tasks using Prompt Dataset with various percentage of single-turn prompts prepending confidence-query system prompts on L1 ama 3- 3-8B, Prompts: DA.
31

---

Published as a conference paper at ICLR 2025
Tables 11 and 12 present ablation studies on the decay factor for the exponential average, for both ciference-baseee and threshold-base PPO-C. This parameter controls how quickly the exponential average adapts to new data and reflects recent model performance. For the main results, set = 0.1, a commonly used value for exponential averages, as it halances stability with filtering out short-term variability. We compare this to 1.0, where the exponential average is updated to match the batch mean at each iteration, and a = 0.0, where it remains fixed at its initial value (in this case, it initialized the reward mean on the evaluation set when the reward model is trained). As shown in the tables, = 1.0 leads to a notable overll decline in calibration performance and slight increase in the MT-Bench score for difference- based PPO-C. Similarly, consistently inferior performance compared to a 0.1 in both calibration and MTT-enchh scores.
0.0 results in
E.7 IMPACT OF CONFIDENCE EEEHIENNEE-UURRY SYSTEM PROMPTS
For the main experiments, we select 25% of the single-turn prompts to prepend a confidence-query system prompt. Here, we present study the effect of varying the percentage of single-turn prompts with this system prompt. As shown in Table [3] the impact on calibration does not show a consistent trend; however, observe decrease in MT-Bench scores as the percentage increases. Given our primary goal to maintain model capability while improving calibration, we opt for 25%.
E.8 IMPACT OF COMBINING EQ.
I AND
Given that Eq. does not inherently enforce the preference for chosen responses over rejected ones. In this section, we compare models trained using the combined loss from Eq [ and Eq against those trained solely with 2 is important to note that we are not training the reward model from scratch; instead, we fine-tune it using the calibration dataset. As shown in Figure [4] the model trained exclusively with Eq exhibits similar ability to distinguish between chosen and rejected responses the model trained with the combined loss. Furthermore, Table I4 shows that PPO-M, when using the reward model trained with the combined loss does not yield better calibration results.
MM Mon you
Figure 24: Training Details of reward model with Eq. 2 alone (orange) and in combination with Eq. (red). Left column: reward of chosen rejected responses. Middle column: reward of chosen responses with high confidence / reward of rejected responses with low confidence. Right column: reward of chosen responses with low confidence / reward of rejected responses with high confidence.
E.9 COMPARING THRESHOLD- -BSSED Vs. REWARD-AVERAGE DIFFERENCE APPROACHES
While PPO-C has demonstrated effectiveness, shown in Table I it important explore alternative methods for adjusting reward scores to provide a broader perspective and facilitate comprehensive comparisons. In this section, we introduce a threshold-based variant of PPO-C for evaluation. Specifically, we use the reward exponential average as threshold and employ the absolute value of the reward as a scaling factor for adjustment. The final reward in this approach is then calculated as:
f +
if Art
ri
(5)
f. - if < Ar.
32

---

Published as a conference paper at ICLR 2025
GSM8K
SciQ
Commonsense(A
Loss
MTT-enchh
ECE
1UC
ACC
ECE +
AUC t
ACC f
ECE 4
AUC
ACC 1
Eq
8.05
0.8638
0.516
0.1031
0.0282
0.6513 0.5186
0.904
0.1286
0.5621
0.7756
7.75
0.8891
0.4974
0.1107
0.1043
0.894
0.2286
0.528
0.7584
TruthfulQA
Object Counting
Professional Knowledge
Loss
MTT-Bench
ECE 4
AUC f
ACC
ECE +
AUC 1
ACC f
ECE
AUC
ACC 1
Eq
8.05
0.4426 0,5006
0,5303
0.4431
0.4839 0.518
0.5178 0.5
0.503
0.3949
0,4902
0,502
Eq
7.75
0.564
0.4565
0.482
0,4786
0,4964
0.5061
Table 14: PPO-M with the reward model trained using two losses on Llama3 8B. Prompts: DA.
where 7 = [ (8; 0.5). As shown in Table 15 we refer to this new threshold-based PPO-C variant as Threshold and the original PPO-C as Difference in the table. The threshold-based PPO-C demonstrates promising results across six datasets. It also exhibits similar trade-off trand between calibration and model instruction- instruction-follooing following capabilities as w increases. These results suggest that threshold-basee approach may serve viable alternative for calibrating reward scores during PPO.
GSMSK
SciQ
CommonsenseQA
Method
W
MT
ECE +
AUC t
ACC
ECE
AUC
ACC
ECE.
AUC
ACC
Threshold
0.5
8.05
0.8638
0.516
0.1031
0.0282
0.6513
0.904
0.1286
0,5621
0.7756
Threshold
1.0
7.76
0.8261
0.501
0.1092
0.0075
0.5641
0.903
0.1025
0.5076
0.7805
Difference
0.5
8.03
0.8792
0.521
0.1099
0.0703
0.6031
0.896
0.1552
0,5678
0.7674
Difference
1.0
7.91
0.8238
0.4937
0.119
0.0087
0.578
0.898
0.1153
0.585
0.7625
TruthfulQA
Object Counting
Professional Knowledge
Method
MT
ECE 4
AUC f
ACC t
ECE +
AUC
ACC 1
ECE 4
AUC f
ACC f
Threshold 8.05
0.4426
0.5303
0.4431
0.4839
0.5178
0.503
0.3949
0.4902
0.502
Threshold 1.0
7.76
0.4271
0.5207
0.4345
0.4709
0.5318
0.505
0.388
0.5069
0,4883
Difference 8.03
0.4428
0.5549
0.4553
0.4856
0.5036
0.512
0.4286
0.5027
0.4906
Difference
1.0 7.91
0.4104
0.515
0.4492
0.4774
0.5118
0.496
0.383
0.509
0.4902
Table 15: Comparison of Threshold-Baaas and PPO-C Llama a- Prompts: DA.
E.
CAN PPO-M AND PPO-C BE COMBINED?
GSM8K
SciQ
Commonsense(AA
MT-Bench Arena-Hard
ECE J AUC ACC ECE
AUC
ACC
ECE J AUC ACC
DA
7.82
14.7
0.8774
0.6199 0.0538 0.104
0.5834
0.879
0.1774
0,5837 0.7617
COT
7.82
14.7
0.2123
0.5317 0.7794 0.0909
0.6641
0.884
0.1957
0.6335
0.7297
TruthfulQA
Object Counting
Professional Knowledge
MT-Bench Arena-Hard
ECE
AUC
ACC
ECE
AUC
ACC
ECE
AUC
ACC
DA
7.82
14.7
0.4654
0.5178
0.4345
0.4927
0.5
0.507
0.5005
0.5287 0.4216
CoT
7.82
14.7
0.4561
0.5656
0,4419
0.2843
0.5
0.715
0.4425
0.5793
0.4439
Table 16: Performance of PPO-Combine Llama across six datasets.
Since PPO-M and PPO-C operate independently, this section explores the potential of combining these methods. Specifically, the calibrated reward models using Eq. 2 are employed in conjunction with the calibrated reward calculation from PPO-C to generate reward scores. The results, presented in Table 16 indicate that the combined approach does not outperform the individual methods and, in
33

---

Published as a conference paper at ICLR 2025
GSM8K
SeiQ
CommonsenseQA
Methods
ECE
AUC
ACC1
ECE +
AUC
ACC
ECE
AUC
ACC1
SFT
0.8783
0.5292
0.0773
0.1681
0.5253
0.801
0.3913
0.5294
0.5528
DPO
0.904
0.5381
0.0834
0.1085
0.561
0.886
0.3011
0.535
0.6871
DA
DPO
0,8861
0.5203
0.097
0.1103
0,5626
0.881
0.3004
0,5409
0.683
CDPO
0.5664
0.5389
0.1024
0.0143
0.6497
0.877
0.1697
0.5815
0.6912
SFT
0.6473
0,5508
0.326
0.1699
0,5816
0.803
0.3293
0,588
0.579
DPO
0.4159
0.5452
0.577
0.113
0.6376
0.858
0.2621
0,6295
0.6593
CoT
DPO
0,452
0,5456
0.539
0.0964
0.6614
0.876
0.235
0,5973
0.6749
CDPO
0.3313
0.6054
0.5277
0.0386
0.7036
0.86
0.1269
0.6685
0.6798
TruthfulQA
Object Counting
Professional Knowledge
Methods
ECE
AUC
ACC
ECE 4
AUC f
ACC f
ECE
AUC
ACC 1
SFT
0.592
0,5388
0,3256
0.5964
0.4938
0.395
0.5109
0,5189
0.4127
DPO
0.6126
0.5581
0.3525
0.5848
0.4996
0,415
0,4764
0.4992
0.495
DPO 0.5647
0.5886
0.3856
0.5999
0.5008
0.4
0.467
0.5153
0.4939
CDPO
0.4022
0.6194
0.3929
0.4662
0.5262
0.422
0.3525
0.5581
0.4898
SFT 0.5259
0.5698
0.3782
0.5388
0.5126
0.45
0.5091
0.5457
0.4068
DPO
0.5188
0,5822
0,4088
0.3520
0.5000
0.6480
0,4289
0.5700
0.4831
C6T
DPO 0.4931
0.6111
0,4113
0.3783
0.5018
0.621
0,4312
0.562
0.4694
CDPO
0.3651
0.634
0,4345
0.3488
0.5286
0.567
0.3349
0.6303
0.4609
Table 18:
Performance comparison of SFT, DPO, DPO;, and CDPO across six datasets using Ll 1--8. SFT and DPO denote the reference and trained DPO models, respectively. DPO1 and CDPO initiate from the trained DPO checkpoint; DPO applies standard DPO on the calibration dataset, focusing on chosen and rejected pairs to assess the impact of training with additional data.
some cases, leads to a decline in performance. We hypothesize that this outcome arises because the calibrated reward model is trained specifically responses incorporating confidence scores, which are optimized to produce unbiased rewards. Consequently, removing these confidence scores to estimate rewards based their difference from exponential average dynamic may be inappropriate.
E.11 EXTENSION To DPO
we present the results of extend- ing PPO-M DPO training on Mistral-73. In this section, we include additional results for Llama3-88. As shown in Table [8] and 17 CDPO effectively reduces ECE and in- creases AUC, mirroring the trend observed with Mistral-7B, while maintaining performance on MT-Bench. However, we observe a slight performance degradation Arena-Hard using either DPO or CDPO. This issue may arise from insufficient hyperparameter tuning in- herent limitations in the structure of the calibra- tion dataset, which leave for future research.
In Section 5.2
Model
Method
MT-Bench
Arena-Hard
SFT
6.44 (6.6)
3.1 (3.3)
DPO
7.67 (7.7)
15.9 (15.9)
Llama3-8B
DPO
7.52
15.2
CDPO 7.68
14.7
Table 17: Comparison of DPO and CDPO on MT- Bench And Arena-Hard for Llama3-8B Num- bers in parenthesis are from Meng et a (2024).
34


Reviews:
[{"review_id": "uQnTkqJj1O", "reviewer": "Reviewer_Wbvf", "summary": "In instruction-following for language models, the paper studies the task of reporting a confidence score in the helpfulness and accuracy of the model's own response. The paper shows that existing reward models are biased in favor of high-confidence scores, and proposes two mitigation methods. The first method involves data augmentation for the reward model, and the second method adds a customized term to the reward in PPO. The methods show improved calibration with no noticeable performance degradation.", "questions": "I didn't understand the paragraph at Lines 239-244. It sounds like you are saying you didn't train the reward model yourself for PPO-M, but I don't see how that could be, since you need a reward model trained on your new calibration dataset. Are you saying that you fine-tuned an existing reward model on the new calibration dataset? Please clarify.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper studies an important and interesting topic, that of verbalized calibration in instruction-following for language models. It is easy to follow and clearly demonstrates the existence of the problem and the success of the proposed mitigation methods. The methods do not require new data to be collected, making them straightforward to use, and yet the improved calibration transfers from reward model scores to accuracy on multiple-choice datasets.", "weaknesses": "Although the paper studies a broadly important topic and is well-executed, I think the strength of the contribution itself is somewhat limited:\n- From a scientific perspective, the methods do not offer much insight: there is no distinction between different confidence levels other than \"high scores\" and \"low scores\", and the methods just increase the reward for high scores for high quality responses and low scores for low quality responses. So it is not really surprising that they improve calibration.\n- This wouldn't be a problem if the methods had a lot of practical value. But from a practical perspective, the methods don't seem that useful either: if the user wanted a confidence score in the quality of the response, you could just display the reward model score (or some prompt-calibrated version) to the user. It's true that it's nice to have this distilled into the model, but there are many alternative ways in which one could do this, such as distilling the reward model score into a special token's logit. And if you just want a calibrated confidence score in the accuracy of a short-form response, there are many existing alternatives available.\n\nIn spite of this weakness, I consider the paper to be above the acceptance threshold because of the careful analysis the paper provides.\n\nI also have a more minor issue with the definition of gamma for the PPO-C method (Line 279): the factor of |r_i (hat)| doesn't make sense to me. Unless the reward model is trained with an L2 penalty on the rewards or similar, the location of the reward is meaningless, since the loss uses only the difference between rewards, so taking an absolute value doesn't make sense. Even if the reward model is trained with an L2 penalty, I don't see why there should be a bigger adjustment for rewards that are far from 0 – or if there is some reason, it may be good to discuss it. Instead, to ensure that gamma is correctly scaled, it would make more sense to me to use some measure of spread of the rewards, such as standard deviation.\n\nA couple typos:\n- Line 134: \"Dand\" -> \"D and\"\n- Line 1046: \"calirbation\" -> \"calibration\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7F214ENQey", "reviewer": "Reviewer_i6YC", "summary": "This paper focuses on LLM calibration: the alignment between the confidence a model expresses and its actual performance. Previous research has demonstrated the phenomenon of overconfidence in LLMs, especially those trained using RLHF, which tend to have a sharpened output probability. The systematic bias in reward models used for RLHF is one of the key causes of LLM overconfidence.\n\nThe paper proposes two methods to address the issue of verbalized overconfidence in LLMs trained with RLHF:\n\n(1) **PPO-M**: a reward model re-training objective that aims to fine-tune existing BT-based reward models. PPO-M achieves this by augmenting existing pairwise ranking datasets with verbal confidence scores. The author filtered samples from various open-source datasets, prepended a confidence-query system prompt to each, and randomly assigned high and low confidence scores, creating a modified training dataset.\n\n(2) **PPO-C**: a reward-shaping method that dynamically adjusts reward scores during PPO training based on a moving average of past rewards and the model's verbalized confidence level. PPO-C differs from vanilla PPO in that it incorporates a confidence-query system prompt into a portion of the training prompts to elicit verbal confidence scores from the model. The reward function promotes higher confidence for responses exceeding the moving average and lower confidence for those falling below.\n\n---\n*Nov. 26 update*: most of my concerns are resolved, satisfied with the added evaluations, and raised the rating from 6 ---> 8 while maintaining my confidence of 3.", "questions": "- I understand that just for the sake of an evaluation of how the current chat models are conditioned on the given verbal confidence, appending a random confidence score to the RewardBench dataset may make sense. Why not simply prompt the chat models to self-annotate? Asking models to generate their own confidence scores could yield a more genuine assessment of how they internally gauge response certainty.\n\n- Figure 2 primarily highlights instances of overconfidence, but are there scenarios where the SFT model consistently exhibits low confidence? It would be interesting to see if the proposed methods can help address underconfidence in addition to overconfidence, particularly if SFT models tend to understate confidence levels.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is very well-written and easy to follow. The motivation is well-supported by evidence, including visualizations of confidence distributions and accuracy, that highlight the overconfidence phenomenon.\n- The proposed methods are straightforward, yet backed by a sufficient amount of evaluation on different families of open-weights models and on diverse benchmarks. The thorough evaluation, along with the authors' transparency in presenting both the strengths and limitations of their approach, makes their findings trustworthy - small improvements in the factuality-focused benchmarks as expected (e.g. TruthfulQA benchmark) and almost no improvement in chat performance (e.g. MT-Bench, Arena-Hard).", "weaknesses": "- Referring to Eq. 2 as the \"PPO-M\" loss could be confusing, as it is not directly related to the standard PPO objective. This equation is, in fact, a re-training objective for the reward model rather than a modification of PPO itself. The labels \"PPO-M\" and \"PPO-C\" might suggest that the paper centers on enhancing PPO objectives, when the proposed methods primarily aim at co-training the reward model with verbal confidence. This distinction could benefit from clarification to prevent misunderstanding.\n\n- While not introduced in this work, the use of verbal confidence to measure a model's confidence in its response may not be ideal. Language models, especially those under 70B parameters, may simulate confidence without true understanding due to limitations in reasoning capability. Have you examined the logits for the 0–10 range when asking for verbal confidence? Observing how these tokens are distributed might provide insight into whether the model is indeed producing a typical probability distribution or simply outputting values. Given these concerns, more concrete metrics, such as those derived from entropy in token-space, might be more accurate indicators of true confidence.\n\n- The confidence metric presented in Figure 2 lacks clarity, and the \"perfect calibration\" line does not contribute meaningful differentiation between a model's overconfidence and cases where the dataset may simply be too easy. Maybe Llama-3-8B-PPO's performance on CommonsenseQA could indicate that the questions are too easy leading to high confidence in correct answers.\n\n- Given that Figure 2 does not reflect the effects of PPO-M and PPO-C, it would be valuable to include a distribution of confidence scores after tuning with these proposed methods. This would enable a direct comparison of the effects of the calibration techniques on model confidence.\n\n-  Figure 3 would benefit from results that combine the answer and confidence score to more clearly illustrate how the proposed methods affect the chosen/rejected ratio.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zFdknuA4bz", "reviewer": "Reviewer_sUqe", "summary": "The paper tackles the problem of calibration in LLMs that went through RLHF, and specifically calibration through verbalized confidence. Based on prior work that showed that RLHF hurt the model calibration, the authors point to the reward model as the source of the problem. They demonstrate that off-the-shelf reward model have a bias towards answers with high-confidence. As a solution, they propose two different methods that should overcome this problem - one focus on the reward model training and the other fix the rewards given during policy optimization.", "questions": "- Regarding PPO-M, is it intended to be an additional objective to equation 1? Because there is nothing in equation 2 that will make the reward of the chosen answer higher than the rejected one.\n\n- Line 270: what do you mean by “biased response”? Why does asking for confidence create bias?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper has a clear structure, beginning with analysis experiments, moving to the proposed method, and concluding with experiments that showcase its effectiveness.\n- The experiments are comprehensive, covering multiple datasets and showing clear improvement in calibration metrics.\n- The problem addressed is relevant and of interest to the community.", "weaknesses": "- The conclusion drawn from Figure 2 doesn’t align with the plots. It’s not that the model after RLHF isn’t overconfident, but rather that the model before RLHF is also overconfident. Nearly all the bins fall below the diagonal line, indicating overconfidence. Additionally, why are the data points so sparse? How many prompts were used in this experiment?\n\n- In Figure 3, since the reward models haven’t been trained with this type of prompt, I would suggest running a control experiment—appending the prompt with a random confidence score to both chosen and rejected answers (similar to PPO-M training). With random confidence scores, we would expect the reward model, on average, to still prefer the chosen answer. If it doesn’t, this would indicate that the prompt is so far out of distribution that the reward model isn’t applicable (which highlights its brittleness, though that’s a separate issue).\n\n- While I understand the algorithm used in PPO-C, the intuition behind it isn’t clear to me. Why would the average reward across the dataset serve as a threshold for determining if a model’s response is correct? Couldn’t it be that most responses are either correct or incorrect, depending on the dataset and model? This approach seems like a heuristic that would only work on datasets where the model is correct about 50% of the time. \nAdditionally, why is $\\Delta r$ defined as a running average of $r$ rather than $\\hat{r}$? On a related note, wouldn’t an off-the-shelf reward model struggle to assign rewards to answers that include confidence scores (i.e., $r$) since it wasn’t trained on such data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FGZob1UGxe", "reviewer": "Reviewer_BbUH", "summary": "This paper identifies a systematic bias in reward models that favors high-confidence responses regardless of their actual quality. They propose two solutions for this bias issue: PPO-M, which calibrates reward modeling by aligning confidence levels with response quality, and PPO-C, which adjusts standard reward model scores during PPO training. Empirical results show their methods can reduce expected calibration error without compromising accuracy or instruction-following ability in open-ended generation.", "questions": "1. Why not add something similar to (1) onto (2) to enforce the preference of $(y_c,h_c)$ over $(y_r,l_r)$?\n2. Why do PPO-M and PPO-C often result in lower ACC?\n3. How does $\\alpha$ affect the results of PPO-C?\n4. Are you assuming that $\\hat{r}(y_c)>\\Delta r$ and $\\hat{r}(y_r)<\\Delta r$?\n\nOverall, It is a well-motivated work presented in a structured manner. However, the contributions lack substantial impact. The proposed algorithms resemble engineering techniques rather than rigorous, principled approaches.\n\n## Lower confidence after discussion\nI maintain my critique of the heuristic algorithm's lack of rigorous design but am less certain if this empirical research meets ICLR standards. I've reduced my confidence to 3. \n\nTo be honest, the algorithm's design and naming seem somewhat inelegant and naive. While empirical results are solid, nowadays it has become increasingly difficult to determine whether their success stems from effective code-level implementation, a limited dataset, or a genuinely strong algorithm.\n\nSince other reviewers have given higher ratings, I am open to its acceptance as a poster. That said, I would be deeply disappointed if it were elevated to the level of a spotlight or oral presentation, as I believe this would set a poor precedent for ICLR's standards.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. Good motivation to identify the systematic bias and propose intuitive solutions.\n2. The presentation is clear and easy to follow, with a logical and well-reasoned train of thought.\n3. Sufficient empirical results to justify the effectiveness of proposed methods", "weaknesses": "1. Regarding the proposed PPO-M, there is no guarantee that the model would prefer $(y_c,h_c)$ over $(y_r,l_r)$. This could be a potential reason why the calibrated model results in lower ACC metrics than the uncalibrated PPO shown in the tables. Although the systematic bias may cause the model to prefer $h_c$ over $l_r$, PPO-M lacks an explicit mechanism to enforce this preference. Have you considered jointly optimizing (1) and (2) to enforce $y_c$ over $y_r$ as well?\n\n2. Since PPO-C depends on a running average reward for confidence calibration, the $\\alpha$ affecting the moving average could be an important factor. However, there seems no evaluation of how $\\alpha$ affects PPO-C. I suggest experiments to show how varying $\\alpha$ influences reward magnitude and performance. Moreover, are you assuming that $\\hat{r}(y_c)>\\Delta r$ and $\\hat{r}(y_r)<\\Delta r$? If so, would be helpful to explicitly state the assumption and analyze its appropriateness.\n\n3. The design and evaluation of the proposed algorithms, as noted above, lack clarity or rigor. The formulations would benefit from a more detailed and persuasive explanation. Ideally, theoretical support should be provided to justify that the approach achieves debiasing without compromising model performance. At present, the methods appear to rely heavily on heuristics rather than principled algorithm design, which diminishes the significance of the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)