Taming Overconfidence in LLMs: Reward Calibration in RLHF

## Abstract

Abstract Language model calibration refers to the alignment between the confidence of the model and the actual performance of its responses.
While previous studies point out the overconfidence phenomenon in Large Language Models (LLMs) and show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF) are overconfident with a more sharpened output probability, in this study, we reveal that RLHF tends to lead models to express verbalized overconfidence in their own responses.
We investigate the underlying cause of this overconfidence and demonstrate that reward models used for Proximal Policy Optimization (PPO) exhibit inherent biases towards
high-confidence scores regardless of the actual quality of responses.
Building upon this insight, we propose two PPO variants: PPO-M: PPO with Calibrated Reward M odeling and PPO-C: PPO with Calibrated Reward C alculation.
PPO-M integrates explicit confidence scores in reward model training, which calibrates reward models
to better capture the alignment between response quality and verbalized confidence.
PPO-C adjusts the reward score during PPO based on the difference between the current reward and the moving average of past rewards. Both PPO-M and PPO-C can be seamlessly integrated into the current PPO pipeline and do not require additional golden labels. We evaluate our methods on both Llama3-8B and Mistral-7B across six diverse datasets including multiple-choice and open-ended generation.
Experiment results demonstrate that both
of our methods
can reduce calibration error and maintain performance comparable to standard PPO.
We further show that they do not compromise model capabilities in open-ended conversation settings. 1 1 1 Our code is released at https://github.com/SeanLeng1/Reward-Calibration .

## 1 Introduction

As Large Language Models (LLMs) significantly expand their functionality
to a wide range of applications from complex problem solving (Wei et al., ; Song et al., ) to science discovery (Imani et al., ; OpenAI, ),
the importance of their reliability becomes increasingly critical.
A key aspect of this reliability is language model calibration – the alignment between model confidence and its actual performance.
LLM confidence can be assessed through two primary methods: logit-based approaches, derived from output token probability distributions, and verbalized expressions, where the model explicitly states its confidence level.
In this paper, we focus on verbalized confidence, where we prompt LLMs to express a confidence score for their responses (Figure 1, Top).

Figure: Figure 1: (Top): Illustration of verbalized confidence generation. An LLM incorrectly answers a question with high confidence. (Bottom): Comparison between reward scores from a vanilla-trained reward model Llama-3-8b-rm-mixture and our calibrated reward model Llama-3-8b-crm. The vanilla model shows bias towards high confidence though the answer is incorrect. Our calibrated reward model correctly assigns a higher reward to the low confidence for the incorrect answer.
Refer to caption: extracted/5922513/sections/calibration-framework.png

Reinforcement Learning from Human Feedback (RLHF) has significantly improved LLM performance through two key components: reward modeling, which learns to predict human preferences from ranking datasets, and policy optimization guided by reward models, typically implemented with Proximal Policy Optimization (PPO) (Schulman et al., ). However, recent studies (Kadavath et al., ; OpenAI, ) show that RLHF-trained LLMs tend to exhibit overconfidence, potentially due to sharpened output distributions.
Previous works address LLM confidence through various methods. Scaling-based approaches (Guo et al., ; Zhang et al., ) adjust model logits with decoding temperature, while verbalized confidence is enhanced via prompting strategies (Tian et al., ) and supervised fine-tuning (Lin et al., ) using ground truth accuracy. More recently, RLHF-based calibration methods (Xu et al., ; Tao et al., ) are proposed.

Our study investigates the underlying causes of overconfidence introduced by RLHF. We demonstrate empirical evidence that RLHF-LLMs elicit verbalized overconfidence compared to their pre-RLHF counterparts. We then reveal a system bias in reward models favoring responses with high confidence scores, regardless of the actual response quality, which potentially causes the poor calibration in RLHF-LLMs.
We propose two novel solutions to be seamlessly integrated into the RLHF process without additional golden labels.

- •
PPO with Calibrated Reward Modeling (PPO-M) calibrates the reward modeling process. We augment the binary pairwise ranking dataset with explicit confidence scores, and encourages the reward model to align confidence levels with response quality (Figure 1, Bottom).
- •
PPO with Calibrated Reward Calculation (PPO-C) adjusts standard reward model scores during PPO training. It maintains a running average of past reward scores as a dynamic threshold to classify responses, and adjusts the reward scores based on model expressed verbalized confidence.

We conduct experiments on Llama3-8B and Mistral-7B across six datasets, and demonstrate that both PPO-M and PPO-C consistently outperform vanilla PPO, achieving a lower Expected Calibration Error (ECE) while maintaining comparable or higher accuracy
(PPO-M on Llama3-8B reduces ECE by 6.44 points and increases accuracy by 2.73 points on GSM8K (Cobbe et al., )).
Moreover, evaluation on MT-Bench (Zheng et al., ) shows that PPO-M and PPO-C preserve model capabilities in general open-ended conversation settings. We further demonstrate that PPO-M generalizes well to Direct Preference Optimization (DPO) models (Rafailov et al., ), which are implicit reward models. Experiments show that our extension, denoted as CDPO, reduces ECE without sacrificing accuracy compared to standard DPO.

## 2 Exploring Systematic Biases and Overconfidence in RLHF-LLMs

In this section, we demonstrate the preliminary experiments that reveal overconfidence in RLHF-LLMs and systematic biases in Reward Models, which motivated the development of our methods.

### 2.1 RLHF-LLMs Exhibit Overconfidence in Their Verbalized Confidence

Previous studies have shown that LLMs tend to exhibit overconfidence when verbalizing their confidence scores (Tian et al., ; Chen et al., ; Xiong et al., ).
However, there is a lack of systematic comparisons between RLHF-LLMs and their pre-RLHF counterparts.
To address this gap, we conduct preliminary experiments here to further investigate this phenomenon.

Figure: Llama3-8B-SFT and Llama3-8B-PPO; Tulu-2-7B and Tulu-2-DPO-7B
Refer to caption: x1.png

##### Setup.

We show results on a multiple-choice question answering dataset,
CommonsenseQA (Talmor et al., ).
We use four off-the-shelf models
(^2^22 [OpenRLHF/Llama-3-8b-sft-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-sft-mixture) [OpenRLHF/Llama-3-8b-rlhf-100k](https://huggingface.co/OpenRLHF/Llama-3-8b-rlhf-100k) [allenai/tulu-2-7b](https://huggingface.co/allenai/tulu-2-7b) [allenai/tulu-2-dpo-7b](https://huggingface.co/allenai/tulu-2-dpo-7b))
for our analysis.
We compare RLHF models (trained with PPO and DPO) with their pre-RLHF versions.
For each question in the dataset,
we explicitly prompt the model to verbalize its confidence score on a scale from 0 to 10 after providing its answer to the question. We report the distribution of these confidence scores in Figure 2.
Evaluations on other datasets and detailed information on the experimental setup, including prompts and parsing details, are provided in Appendix D and E.1.

##### Observations.

As illustrated in Figure 2, there is a clear trend on both datasets that the RLHF models, whether trained using PPO or DPO, exhibit greater overconfidence compared to their SFT counterparts.
Specifically, the SFT models display a more diverse confidence distribution, while RLHF models show confidence scores predominantly at the higher levels. This confirms the phenomenon that RLHF models tend to be more confident when verbalizing their confidence scores.

### 2.2 Reward Models are Biased Toward High Confidence Scores

In this section, we hypothesize that the observed overconfidence in RLHF-LLMs stems from an inherent bias in reward models favoring a higher confidence score appended after responses. We conduct experiments to demonstrate this preference.

Figure: Figure 3: Preference distributions for ArmoRM-Llama3-8B-v0.1, a reward model for PPO training (left) and Tulu-2-DPO-7B, a DPO model (right) on the modified RewardBench dataset across four modes. From top to bottom: confidence_reversed, chosen_with_conf, rejected_with_conf, answer_only. Red bar indicates the preference for a rejected or high-confidence response, and blue bar indicates the preference for a chosen or low-confidence response.
Refer to caption: x5.png

Figure: Figure 4: Confidence-Query System Prompt for verbalizing confidence scores.

##### Setup.

We employ the  [RewardBench](https://huggingface.co/datasets/allenai/reward-bench)Dataset (Lambert et al., ), following its experimental configuration with certain adjustments to examine how reward models process explicit confidence scores in responses. We evaluate [RLHFlow/ArmoRM-Llama3-8B-v0.1](https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1) (Wang et al., ) and [allenai/tulu-2-dpo-7b](https://huggingface.co/allenai/tulu-2-dpo-7b) (Ivison et al., ). Specifically, we prepend a confidence-query system prompt as illustrated in Figure 4; if the reward model does not support system prompts, we prepend it into the user prompt instead.
This helps the model interpret the scale of confidence scores.

Subsequently, we append a random confidence score, Confidence:{random_score}, to each model response. For a comprehensive comparison, we explore four modes: 1) answer_only: applying the original RewardBench dataset without modifications; 2) confidence_reversed: the system prompt is prepended. A high confidence score (random integer from 7 to 10) is appended to the rejected response, and a low confidence score (random integer from 0 to 3) to the chosen response; 3) chosen_with_conf: the system prompt is prepended, but we compare identical chosen responses with high versus low confidence scores; 4) rejected_with_conf: similar to chosen_with_conf, but comparing high and low confidence scores for identical rejected responses. We report the preference count for each model. Given that DPO models are implicit reward models (Rafailov et al., ), we also include evaluation on DPO models. We provide more details on the modified data and evaluations on other reward models in
Appendix C.1 and  E.2

##### Observations.

According to Figure 3, when evaluated on the original RewardBench dataset (answer_only), both models effectively discriminate between chosen and rejected responses by assigning higher reward scores to chosen responses. It is important to note that in typical pairwise preference datasets,
distinctions between the chosen and rejected responses – such as length, tone, and correctness – are usually pronounced. However, even accounting for these differences, simply modifying the query prompt and assigning a low confidence score to the chosen response and a high confidence score to the rejected response can significantly alter model behavior. As depicted in confidence_reversed, the number of preferred rejected responses with high confidence largely increases, indicating that the model’s ability to distinguish between chosen and rejected response is distorted.
In chosen_with_conf and rejected_with_conf where identical responses are compared with different confidence scores, we observe that reward models clearly prefers responses with higher confidence scores, regardless of whether the response is originally chosen or rejected.
These observations suggest that reward models exhibit a systematic bias towards responses with high confidence scores, and potentially explain the overconfidence of RLHF-LLMs.

## 3 Calibrated Reward Modeling and Calculation

Drawing from observations in previous sections, we propose two methods here to address the bias in reward scores: calibrated reward modeling (PPO-M) and calibrated reward calculation (PPO-C).

##### Background: Reward Modeling.

Typical reward model training uses a pairwise human preference data with binary ranking labels (chosen and rejected). Let $\mathcal{D}=\{(x_{i},y_{c}^{i},y_{r}^{i})\}_{i=1}^{n}$
be the training dataset for the reward model, where $x_{i}$ is the prompt, and $y_{c}^{i}$ is the chosen response preferred over the rejected response $y_{r}^{i}$.
A binary preference ranking loss (Ouyang et al., ) is applied to enforce that the chosen responses receives a higher score than the rejected one, as illustrated in Eq. 1.

$$ $\mathcal{L}_{\text{preference}}=-\mathbb{E}_{(x,y_{c},y_{r})\sim\mathcal{D}} \left[\log\sigma\left(R_{\theta}(x,y_{c})-R_{\theta}(x,y_{r})\right)\right]$ (1) $$

where the reward model $R_{\theta}$ is typically initialized from the SFT model. The LM head
on top of the last layer
is replaced with a linear layer
to yield a single scalar reward prediction $R_{\theta}\left(x,y\right)$ for a given prompt $x$ and response $y$.
Here, $y_{c}$ and $y_{r}$ denote the chosen and rejected responses respectively.

##### PPO-M: PPO with Calibrated Reward M odeling.

Existing reward model training datasets typically do not include prompts requesting verbalized confidence scores or responses including explicit confidence levels.
To address this gap, we propose a straightforward modification to the existing binary pairwise ranking dataset by incorporating a confidence-query system prompt (as shown in Fig. 4) and appending random confidence scores to model responses, consistent with the format in our preliminary experiments. This results in a modified training dataset for the reward model, denoted as $\mathcal{\hat{D}}=\left\{\left(\hat{x}^{i},\left(y_{c}^{i},h_{c}^{i}\right),
\left(y_{c}^{i},l_{c}^{i}\right),\left(y_{r}^{i},h_{r}^{i}\right),\left(y_{r}^
{i},l_{r}^{i}\right)\right)\right\}_{i=1}^{n}$, where $\hat{x}^{i}$ represents the prompt with confidence-query system prompt prepended, $h$ and $l$ represent random high and low confidence scores, respectively.
We propose the calibrated reward modeling loss as follows:

$$ $\displaystyle\mathcal{L_{\text{PPO-M}}}=$ $\displaystyle-\mathbb{E}_{\left(\hat{x},\left(y_{c},h_{c}\right),\left(y_{c},l _{c}\right),\left(y_{r},h_{r}\right),\left(y_{r},l_{r}\right)\right))\sim \mathcal{\hat{D}}}\Big{[}\log\sigma\left(R_{\theta}\left(\hat{x},(y_{c},h_{c}) \right)-R_{\theta}\left(\hat{x},(y_{c},l_{c})\right)\right)$ (2) $\displaystyle+\log\sigma\left(R_{\theta}\left(\hat{x},(y_{r},l_{r})\right)-R_{ \theta}\left(\hat{x},(y_{r},h_{r})\right)\right)\Big{]}$ $$

This encourages the reward model to prefer high over low verbalized confidence for chosen responses and prefer low over high verbalized confidence for rejected responses.
Note that the calibration dataset is not intended for training reward models from scratch. Instead, considering the availability of various off-the-shelf reward models, we assume reward models for calibration are already trained beforehand and generally perform well.
Then in PPO training, we replace the pre-calibrated reward model with this calibrated one to generate reward scores.

Figure: Figure 5: Framework for PPO-C.
Refer to caption: extracted/5922513/sections/method-framework.png

##### PPO-C: PPO with Calibrated Reward C alculation.

While PPO-M addresses bias in reward model training, we propose an alternative approach, PPO-C, to directly improve PPO training by adjusting the reward calculation process.
PPO-C can be seamlessly integrated into the original PPO procedure without modifying the reward model.

We modify the original PPO training set by replacing a certain amount of prompts with the confidence-query system prompt (shown in Fig. 4) to ask for both the answer and verbalized confidence. This results in a mixed dataset with each sample denoted as $(x_{i},y_{i},s_{i})$ where $x_{i}$ and $y_{i}$ are the prompt and corresponding model response, and $s_{i}$ is an optional verbalized confidence generated by the model if $x_{i}$ asks for a confidence score.
For samples without confidence querying, we simply use their original reward $r_{i}=R(x_{i},y_{i})$ for model updating.
For samples with confidence querying, we propose a calibrated reward calculation procedure to mitigate the bias in the reward score $r_{i}=R(x_{i},y_{i},s_{i})$.

We first parse and remove the confidence score from the model response to obtain an unbiased response $(x_{i},y_{i})$.
This allows us to obtain an unbiased reward score $\hat{r}_{i}=R(x_{i},y_{i})$.
We maintain a moving average of the reward scores, defined by $\Delta r_{t}=\alpha*r_{t}+(1-\alpha)*\Delta r_{t-1}$, where $\alpha$ is set to 0.1, to serve as a dynamic threshold for classifying the current model response as positive or negative. We then adjust the reward score as follows:

$$ $\displaystyle r_{i}=\begin{cases}\hat{r}_{i}+\gamma&\text{if }\hat{r}_{i}\geq \Delta r_{t}\\ \hat{r}_{i}-\gamma&\text{if }\hat{r}_{i}<\Delta r_{t}\\ \end{cases}$ (3) $$

where $\gamma$ is the reward adjustment factor, defined as $\gamma=w*|\hat{r}_{i}|*(s_{i}-0.5)$. $w$ is a scaling coefficient set to 0.5. This coefficient adjusts the extent of the adjustment applied to the unbiased reward $\hat{r}$, based on the rescaled confidence score $s_{i}$, which is normalized from 0 to 1.0.
We handle missing confidence scores by assigning a default confidence of 0.5, to keep the reward score unchanged.
The overall framework for PPO-C is illustrated in Fig. 5.

## 4 Experiments

We evaluate PPO-M and PPO-C on two model families: Llama3-8B and Mistral-7B. We use their supervised fine-tuned versions(^3^33These models are instruction-tuned and do not undergo the RLHF process.) (*i.e.*, [OpenRLHF/Llama-3-8b-sft-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-sft-mixture), [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B)) as the starting point for reward model and RLHF training.
We explore two prompting stategies: Direct Answers (DA) and Zero-Shot Chain-of-Thought (CoT) (Kojima et al., ). For Direct Answers, we utilize regex parsing to extract model responses and confidence scores. For Zero-Shot CoT, we use gpt-4o-2024-08-06 (Achiam et al., ) to parse confidence scores and compare model responses with golden answers. Detailed descriptions of prompts, implementation, and parsing methods are available in Appendix D.5.

We consider two evaluation metrics: Expected Calibrated Error (ECE) (Guo et al., ) and Area Under the Receiver Operating Characteristic Curve (AUC) (Hendrycks & Gimpel, ). ECE bins model predictions by confidence and measures the prediction accuracy within each category. AUC measures the ability of the model to distinguish between correct and incorrect responses.

### 4.1 Experimental Setup

We employ OpenRLHF(^4^44https://github.com/OpenRLHF/OpenRLHF) (Hu et al., ) for reward model and RLHF training. All training experiments are conducted on 4$\times$A100 GPUs, and evaluations are carried out on one A100 GPU.

##### RM Checkpoints.

For Llama3-8B, we employ the readily available reward model [OpenRLHF/Llama-3-8b-rm-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-rm-mixture) (Hu et al., ), trained from the corresponding SFT checkpoint. For Mistral-7B, we train a reward model using logsigmoid loss, as shown in Eq. 1, from scratch on [Skywork/Skywork-Reward-Preference-80K-v0.1](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1) (Liu & Zeng, ).
For additional training details, please refer to Appendix D.1.

##### RM Calibration Dataset.

We employ a mixture of open-source datasets, and filter samples to ensure a high distinction between scores of chosen and rejected responses.
Subsequently, we prepend the confidence-query system prompt shown in Fig 4 to each response. We then randomly assign high and low confidence scores to create four response types: chosen with high/low confidence and rejected with high/low confidence.
For additional information on dataset compositions, the predetermined threshold, and other details, please refer to Appendix C.3.

##### Evaluation Datasets.

We use six diverse datasets for model evaluation: GSM8K (Cobbe et al., ) for Arithmetic Reasoning, CommonsenseQA (Talmor et al., ) for Commonsense Knowledge, SciQ (Welbl et al., ) for Science Knowledge, ObjectCounting from BigBench (Srivastava et al., ) for Symbolic Reasoning, four Professional Knowledge datasets (Professional Medicine, Professional Law, Professional Accounting and Professional Psychology) in MMLU (Hendrycks et al., ),
and TruthfulQA (Lin et al., ) for Truthful Reasoning. The datasets include both open-ended generation and multiple-choice questions.

##### RLHF Dataset.

We use a subset of [RLHFlow/prompt-collection-v0.1](https://huggingface.co/datasets/RLHFlow/prompt-collection-v0.1) (Dong et al., ) considering computational resources. We randomly select 20,480 prompts and integrate a confidence-query system prompt into 25% of single-turn prompts to elicit verbalized confidence from the model, as exemplified in Figure 4. For clarity, we refer the original 20,480 prompts as the clean version and those with the confidence-query system prompts added as the modified version.

##### Compared Methods.

We compare our PPO-M and PPO-C with the following methods: (1) the SFT model, which is the initial checkpoint before RLHF training; (2) the PPO model, which uses vanilla reward model in standard PPO training on the clean version dataset without confidence querying in system prompt; (3) PPO$\dagger$, an ablation of our PPO-M, which includes confidence-query system prompts (modified version) in PPO training, but still uses the vanilla reward model.

### 4.2 Main Results

**(a) Performance comparison across various methods on six datasets. SFT: Supervised Fine-Tuned checkpoints, serving as the starting points for all methods. PPO$\dagger$: an ablation of our PPO-M method which uses vanilla reward model in PPO training but on our modified dataset (with confidence-query system prompts).**
|  | Methods | GSM8K | SciQ | CommonsenseQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ |  |  |
| Llama3-8B |  |  |  |  |  |  |  |  |  |  |
| DA | SFT | 0.8608 | 0.5184 | 0.1221 | 0.0931 | 0.6067 | 0.873 | 0.2075 | 0.5889 | 0.7183 |
| \arrayrulecolor<br>gray\arrayrulecolorblack | PPO | 0.8843 | 0.5021 | 0.1099 | 0.0683 | 0.6507 | 0.911 | 0.1729 | 0.5815 | 0.7641 |
|  | PPO$\dagger$ | 0.8954 | 0.5 | 0.1046 | 0.0958 | 0.5047 | 0.904 | 0.2222 | 0.5113 | 0.7748 |
|  | PPO-M | 0.8393 | 0.57 | 0.119 | 0.0267 | 0.6115 | 0.898 | 0.1206 | 0.5568 | 0.7707 |
|  | PPO-C | 0.8638 | 0.516 | 0.1031 | 0.0282 | 0.6513 | 0.904 | 0.1286 | 0.5621 | 0.7756 |
| CoT | SFT | 0.4369 | 0.5138 | 0.5481 | 0.0944 | 0.65 | 0.856 | 0.1928 | 0.6155 | 0.7101 |
| \arrayrulecolor<br>gray\arrayrulecolorblack | PPO | 0.2566 | 0.5229 | 0.7392 | 0.0862 | 0.6763 | 0.879 | 0.1767 | 0.6287 | 0.7363 |
|  | PPO$\dagger$ | 0.2553 | 0.5044 | 0.743 | 0.1265 | 0.5452 | 0.868 | 0.2654 | 0.5615 | 0.7191 |
|  | PPO-M | 0.1909 | 0.5499 | 0.7703 | 0.0392 | 0.6635 | 0.877 | 0.1555 | 0.579 | 0.7346 |
|  | PPO-C | 0.2252 | 0.5662 | 0.7422 | 0.0403 | 0.6383 | 0.889 | 0.1581 | 0.5713 | 0.7396 |
| Mistral-7B |  |  |  |  |  |  |  |  |  |  |
| DA | SFT | 0.8628 | 0.5747 | 0.0902 | 0.0952 | 0.5877 | 0.882 | 0.1634 | 0.56 | 0.774 |
| \arrayrulecolor<br>gray\arrayrulecolorblack | PPO | 0.8675 | 0.583 | 0.097 | 0.0973 | 0.5497 | 0.89 | 0.1772 | 0.5594 | 0.7748 |
|  | PPO$\dagger$ | 0.8851 | 0.5464 | 0.0877 | 0.1117 | 0.5439 | 0.885 | 0.1848 | 0.5674 | 0.7756 |
|  | PPO-M | 0.7963 | 0.5055 | 0.1016 | 0.0108 | 0.5090 | 0.888 | 0.1163 | 0.5303 | 0.7625 |
|  | PPO-C | 0.8702 | 0.634 | 0.0811 | 0.0865 | 0.5758 | 0.891 | 0.1608 | 0.5913 | 0.7723 |
| CoT | SFT | 0.4124 | 0.5277 | 0.5785 | 0.1124 | 0.6238 | 0.872 | 0.1908 | 0.6205 | 0.7518 |
| \arrayrulecolor<br>gray\arrayrulecolorblack | PPO | 0.4146 | 0.5228 | 0.58 | 0.1126 | 0.5794 | 0.877 | 0.1867 | 0.6238 | 0.7699 |
|  | PPO$\dagger$ | 0.3932 | 0.5096 | 0.6035 | 0.1044 | 0.5693 | 0.885 | 0.2056 | 0.6135 | 0.7518 |
|  | PPO-M | 0.3379 | 0.5974 | 0.5982 | 0.0388 | 0.6584 | 0.886 | 0.1157 | 0.6118 | 0.7666 |
|  | PPO-C | 0.3653 | 0.5312 | 0.624 | 0.1006 | 0.6425 | 0.892 | 0.1767 | 0.6228 | 0.7633 |

##### Both PPO-M and PPO-C consistently outperform other baselines across Llama3-8B and Mistral-7B .

Figure: Figure 6: Preference distributions of our calibrated reward model Llama-3-8b-crm and the pre-calibrated version Llama-3-8b-rm-mixture on rejected_with_conf.
Refer to caption: extracted/5922513/sections/eval_graphs/comparison_rejected_Llama-3-8b-rm-mixture.png

In Table 4.2 we show the results of all five methods on six datasets. If we compare vanilla PPO and SFT, vanilla PPO indeed shows a degradation in calibration (higher ECE and lower AUC) though generally improves in accuracy.
Among all methods, PPO-M and PPO-C consistently demonstrate lower ECE and higher AUC than all the baselines across both models and prompting strategies, indicating their improved calibration ability. Meanwhile, PPO-M and PPO-C preserve comparable and even higher accuracy, showing that their calibration ability is not compromised by model performance.
Compared to PPO$\dagger$, an ablation of PPO-M, PPO-M and PPO-C demonstrate better calibration ability.
This is because while PPO$\dagger$ includes confidence-query system prompts in PPO training, it still uses the vanilla reward model instead of the calibrated reward model introduced in Sec. 3.
This further indicates that properly calibrating the reward score could mitigate the bias towards high-confidence responses.

##### Calibrated Reward Models.

Figure 6 presents the preference distributions of the calibrated reward model compared to the pre-calibrated version. When evaluated on rejected responses with high and low confidence scores, unlike pre-calibrated version that consistently prefers the high-confidence responses, the calibrated reward model tends to favor those with low confidence – a behavior we aim to achieve. Further results for chosen_with_conf are detailed in Appendix E.3.

## 5 Analysis

In this section, we explore how our proposed methods affect language model abilities in instruction-following and engagement in conversational settings. Additionally, we also present an extension of our approach to Direct Preference Optimization (DPO) models.

### 5.1 Instruction-Following Capabilities

##### Dataset.

To assess whether PPO-M and PPO-C compromise the instruction-following abilities of LLMs gained through PPO, we evaluate their performance on two benchmarks: MT-Bench (Zheng et al., ) and Arena-Hard (Li et al., ). MT-Bench consists of 80 high-quality, multi-turn questions designed to evaluate LLMs across various aspects, while Arena-Hard contains 500 technical problem-solving queries and demonstrates a higher agreement with human preference rankings.

##### PPO-M and PPO-C do not compromise LLM instruction-following abilities.

**Table 1: Results on MT-Bench and Arena-Hard.**
| Model | Method | MT-Bench $\uparrow$ | Arena-Hard $\uparrow$ |
| --- | --- | --- | --- |
| Llama3-8B | SFT | 7.34 | 10.0 |
| \arrayrulecolorgray\arrayrulecolorblack | PPO | 8.00 | 14.6 |
|  | PPO$\dagger$ | 7.81 | 13.4 |
|  | PPO-M | 8.05 | 14.1 |
|  | PPO-C | 8.05 | 14.1 |
| Mistral-7B | SFT | 7.65 | 9.2 |
| \arrayrulecolorgray\arrayrulecolorblack | PPO | 7.84 | 10.5 |
|  | PPO$\dagger$ | 7.83 | 11.7 |
|  | PPO-M | 7.95 | 9.9 |
|  | PPO-C | 7.98 | 11.8 |

### 5.2 Extension to DPO

##### Setup.

As PPO-M calibrates the reward model using an augmented binary pairwise dataset, we can naturally extend Eq. 2 to Direct Preference Optimization (DPO) training. This is because DPO models are implicit reward models (Rafailov et al., ). We denote our extension as Calibrated DPO (CDPO), and illustrate the loss function in Eq. 4.

$$ $\displaystyle\mathcal{L_{\text{CDPO}}(\pi_{\theta};\pi_{\text{ref}})}=$ $\displaystyle-\mathbb{E}_{\left(x,y_{c},y_{r},\hat{x},\left(y_{c},h\right), \left(y_{c},l\right),\left(y_{r},h\right),\left(y_{r},l\right)\right))\sim \mathcal{D}}\left[\log\sigma(r(x,y_{c})-r(x,y_{r}))\right.$ (4) $\displaystyle\left.+w(\log\sigma(r(\hat{x},(y_{c},h))-r(\hat{x},(y_{c},l)))+ \log\sigma(r(\hat{x},(y_{r},l))-r(\hat{x},(y_{r},h))))\right]$ $$

where $r(x,y)=\beta\log\frac{\pi_{\theta}\left(y\mid x\right)}{\pi_{\text{ref}}\left(
y\mid x\right)}$ represents the implicit reward defined by model $\pi_{\theta}$ and its reference model $\pi_{\text{ref}}$.
$(y_{c},h)$ and $(y_{r},l)$ denote the model responses paired with high and low confidence respectively, and the subscripts indicate whether it is a chosen or rejected response. $\hat{x}$ represents the prompt with confidence-query system prompt prepended. $w$ is the scaling coefficient.

The first term in Eq. 4 maintains the original DPO objective to prevent forgetting, since DPO models rely on subtle probability differences to distinguish between chosen and rejected responses.

We use the Mistral-7B DPO version (*i.e.*, [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B)) as reference model and the DPO model [NousResearch/Nous-Hermes-2-Mistral-7B-DPO](https://huggingface.co/NousResearch/Nous-Hermes-2-Mistral-7B-DPO)
for the experiment, and fine-tune the DPO models on our RM calibration Dataset using Eq. 4.

##### Results.

**Table 2: Comparison of MT-Bench And Arena-Hard Performance for Mistral-7B.**
| Model | Method | MT-Bench $\uparrow$ | Arena-Hard $\uparrow$ |
| --- | --- | --- | --- |
| Mistral-7B | SFT | 7.65 | 9.2 |
| DPO | 7.83 | 13.4 |  |
| \arrayrulecolorgray\arrayrulecolorblack | DPO$\dagger$ | 7.83 | 14.3 |
|  | CDPO | 7.85 | 15.9 |

## 6 Related Works

##### LLM Calibration.

Model Calibration aims to align a model’s confidence with its accuracy. Recent studies show that LLMs often exhibit overconfidence (Tian et al., ; Chen et al., ; Xiong et al., ; Achiam et al., ). Previous studies have explored methods such as scaling-based (Deng et al., ; Guo et al., ; Zhang et al., ) approaches and nonparametric methods such as binning (Zadrozny & Elkan, ).
Recent work has introduced verbalized confidence (Lin et al., ), where models are prompted to directly output confidence scores. Most studies focus on pre-trained and instruction-tuned LLMs (Lin et al., ; Han et al., ),
while other studies examine RLHF-trained LLMs, proposing calibration through prompting strategies (Xiong et al., ; Tian et al., ).
More recent work leverages Reinforcement Learning for calibration (Xu et al., ; Tao et al., ), which aligns closely with our study. Our study contributes by identifying the potential cause for overconfidence in RLHF-LLMs and proposing calibration of the reward models or reward score calculations to be seamlessly integrated into the existing PPO framework. In addition, our approach does not compromise the model’s generalization capabilities in open-ended generation.

##### LLM Alignment And Reward Modeling.

Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ; Christiano et al., ; Bai et al., ) has been widely applied to align LLMs with human preferences. This pipeline typically involves Supervised Fine-Tuning (SFT), reward modeling, and policy optimization using Proximal Policy Optimization (PPO) (Schulman et al., ).
Recent works have explored variations of this pipeline to address the challenge of noisy human preferences (Hong et al., ; Wang et al., )
and to improve computational efficiency with Direct Preference Optimization (DPO) (Rafailov et al., ; Dubey et al., ) to eliminate the need of a separate reward model.

We leave a comprehensive discussion of related works in Appendix. A.

## 7 Conclusion

This paper addresses overconfidence in RLHF-LLMs by identifying a systematic bias in reward models that favors high-confidence responses regardless of their actual quality. We propose two solutions, PPO-M, which calibrates reward modeling by aligning confidence levels with response quality, and PPO-C, which adjusts standard reward model scores during PPO training. Both methods can be seamlessly integrated into the RLHF process. Extensive experiments across various benchmarks demonstrate the effectiveness of our methods in reducing expected calibration error without compromising accuracy and the instruction following ability in open-ended generation. Future research directions include extending our study to logit-based confidence, as well as exploring confidence calibration methods for open-ended questions.

## Appendix A Related Works

##### LLM Calibration.

Model Calibration aims to align a model’s confidence with its accuracy. It has been observed that modern neural network, including Large Language Models (LLMs), often exhibit overconfidence, suggesting poor calibration (Tian et al., ; Chen et al., ; Xiong et al., ; Achiam et al., ). Previous studies have explored methods such as scaling-based (Deng et al., ; Guo et al., ; Zhang et al., ) approaches and nonparametric methods such as binning (Zadrozny & Elkan, ). Among these, temperature scaling (Guo et al., ; Zhang et al., ) has been proved to be effective when combined with large pre-trained LLMs (Kadavath et al., ; Xiao et al., ; Kuhn et al., ). However, previous evaluations often focus on probabilities derived from model logits (Hendrycks et al., ; Mukhoti et al., ; Guo et al., ; Minderer et al., ), which can be inaccessible in proprietary models and are unintuitive to human users.

Recently, verbalized confidence has been introduced (Lin et al., ), prompting models to directly output confidence scores. While most studies focus on pre-trained LLMs and calibrate them through supervised fine-tuning (Lin et al., ; Han et al., ), which typically involves sampling responses and calculating average accuracy as the ground truth confidence score, other studies have examined verbalized confidence in instruction-tuned and RLHF-trained LLMs, proposing calibration through prompting strategies (Xiong et al., ; Tian et al., ).

More recent work leverages Reinforcement Learning for calibration (Xu et al., ; Tao et al., ), which aligns closely with our study. Our study contributes by identifying the potential cause for overconfidence in RLHF-LLMs and proposing calibration of the reward models or reward score calculations to reduce this issue. The proposed methods can be seamlessly integrated into the existing PPO framework and, unlike supervised fine-tuning (SFT) methods that require a dataset with ground truth labels for accuracy calculation—limiting their use in open-ended generation—our approach does not compromise the model’s generalization capabilities in such settings.

##### LLM Alignment And Reward Modeling.

Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ; Christiano et al., ; Bai et al., ) has been widely applied to align LLMs with human preferences. This pipeline typically involves three steps: Supervised Fine-Tuning (SFT), the collection of pairwise ranking data and the development of a reward model, and optimization of the policy model obtained from the first step using Proximal Policy Optimization (PPO) (Schulman et al., ). The effectiveness of PPO relies on the accuracy and robustness of the reward model. Following traditional Bradley-Terry reward models (Bradley & Terry, ), training typically utilizes a binary pairwise dataset. However, human-labeled preferences can be noisy or present conflicting indications (Hong et al., ; Knox et al., ; Wang et al., ). Several methods have been proposed to address this challenge, such as introducing a margin to guide the reward model in assigning a greater discrepancy to more separable comparison pairs (Touvron et al., ; Wang et al., ), and empolying multi-objective reward modeling that considers joint preference such as “helpfulness, correctness, coherence”, etc. (Dong et al., ; Zhou et al., ; Wang et al., ; Chen et al., ; Chakraborty et al., ; Wang et al., ).

While RLHF pipeline has been proven effective in aligning LLMs with human preferences, PPO involves several challenges, including reward hacking, sensitivity to hyperparameters, and substantial computational resource requirements, which complicate its implementation and usage. Several variants have been proposed (Dong et al., ; Yuan et al., ; Zhao et al., ; Rafailov et al., ; Song et al., ; Azar et al., ; Ethayarajh et al., ; Hong et al., ; Liu et al., ; Meng et al., ), and among these, Direct Preference Optimization (DPO) has been widely adopted (Rafailov et al., ; Dubey et al., ). DPO defines the preference loss as a function of the policy model directly, thereby eliminating the need for a separate reward model.

Despite these advancements, there remains a lack of research on how reward models contribute to the confidence calibration of LLMs. In this study, we bridge this gap by demonstrating the vulnerability of reward models trained using various methods, which can be biased by a simple confidence score. Additionally, we explore methods to calibrate these models to reduce overconfidence in RLHF-LLMs.

## Appendix B Limitation and Broader Impact

### B.1 Limitation

Although we demonstrate that directly applying PPO-M style training loss to DPO training can reduce ECE and increase AUC, thereby enhancing calibration, this method is not specifically tailored for DPO. We observe some performance degradation, suggesting that future work could explore specific hyperparameter tuning or a more specially designed dataset.

### B.2 Broader Impact

Since our work focuses on model calibration, it can be applied to various fields in need of well-calibrated language models to improve reliability for general AI systems.

## Appendix C Datasets

In this section, we provide detailed descriptions of the datasets used in this study.

### C.1 Preliminary Experiments Dataset

In Section 2.2, we demonstrate the inherent bias in reward models toward responses with high confidence scores by modifying [allenai/reward-bench](https://huggingface.co/datasets/allenai/reward-bench). This modification involves prepending a confidence-query system prompt and appending random confidence scores to the original model responses. We assigned scores ranging from 0 to 3 for low confidence and 7 to 10 for high confidence. Figure 7 displays an example from the modified dataset on rejected_with_conf, featuring a rejected response with low confidence scores appended.

Figure: Figure 7: An example from the Modified RewardBench in mode: rejected_with_conf

### C.2 Reward Model Training Datasets

For Mistral-7B, we employ [Skywork/Skywork-Reward-Preference-80K-v0.1](https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.1) (Liu & Zeng, ) to train the reward model from scratch.

### C.3 Reward Model Calibration Datasets.

To compile the dataset for calibrating reward models, we filtered samples from open-source datasets. Below, we list the datasets utilized and the thresholds set for each in Table 4.

Initially, we filter out samples that are multi-turn or have a tokenized length exceeding 8192, as multi-turn formats are unsuitable for assigning confidence scores and we aim to prevent truncation. The threshold indicates the preference strength (Wang et al., ), defined as the difference between chosen and rejected scores. Notably, in datasets such as [RLHFlow/Argilla-Math-DPO-standard](https://huggingface.co/datasets/RLHFlow/Argilla-Math-DPO-standard), a preference strength less than 1 often means that both chosen and rejected responses yield the same answer through different reasoning paths. The goal is to calibrate the reward model to assign higher scores to to high-confidence chosen responses and lower scores to high-confidence rejected responses, while doing the opposite for low-confidence responses.
However, when both responses yield the same mathematical solution through different reasoning, it is inappropriate for low-confidence rejected responses to get higher scores. Consequently, we exclude these samples and retain those with a significant discrepancy between chosen and rejected responses. We set the threshold to retain about 2,500 samples per dataset considering computational resources. For datasets without specific chosen and rejected scores, we randomly select 2,500 samples.

**Table 4: Datset compositions.**
| Dataset | Threhold |
| --- | --- |
| [argilla/distilabel-capybara-dpo-7k-binarized](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized) (Daniele & Suphavadeeprasit, ) | 1 |
| [RLHFlow/CodeUltraFeedback-standard](https://huggingface.co/datasets/RLHFlow/CodeUltraFeedback-standard) (Weyssow et al., ) | 3 |
| [argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned) (Bartolome et al., ) | 3.5 |
| [RLHFlow/Helpsteer-preference-standard](https://huggingface.co/datasets/RLHFlow/Helpsteer-preference-standard) (Wang et al., ) | 2.5 |
| [RLHFlow/Helpsteer2-standard](https://huggingface.co/datasets/RLHFlow/Helpsteer2-standard) (Wang et al., ) | 2 |
| [RLHFlow/Orca-distibalel-standard](https://huggingface.co/datasets/argilla/RLHFlow/Orca-distibalel-standard) (Lian et al., ) | 2.0 |
| [RLHFlow/SHP-standard](https://huggingface.co/datasets/RLHFlow/SHP-standard) (Ethayarajh et al., ) | 50 |
| [RLHFlow/HH-RLHF-Helpful-standard](https://huggingface.co/datasets/RLHFlow/HH-RLHF-Helpful-standard) (Bai et al., ) | NA |
| [RLHFlow/Argilla-Math-DPO-standard](https://huggingface.co/datasets/RLHFlow/Argilla-Math-DPO-standard) | 1 |
| [RLHFlow/PKU-SafeRLHF-30K-standard](https://huggingface.co/datasets/RLHFlow/PKU-SafeRLHF-30K-standard) (Ji et al., ) | NA |
| [CyberNative/Code_Vulnerability_Security_DPO](https://huggingface.co/datasets/CyberNative/Code_Vulnerability_Security_DPO) | NA |
| [fblgit/simple-math-DPO](https://huggingface.co/datasets/fblgit/simple-math-DPO) (Murias, ) | NA |

### C.4 PPO Datasets

For PPO training, we filter out prompts with a tokenized length exceeding 8192 to prevent truncation and randomly select 20,480 prompts from [RLHFlow/prompt-collection-v0.1](https://huggingface.co/datasets/RLHFlow/prompt-collection-v0.1) (Dong et al., ). We integrate a confidence-query system prompt into single-turn prompts to elicit verbalized confidence from the model. The system prompt is used in 25% of the single-turn prompts for main results. Figure 8 illustrates an example from the dataset that incorporates this system prompt.

Figure: Figure 8: PPO Prompt Example.

### C.5 Evaluation Datasets.

We examine six datasets encompassing six distinct categories: Arithmetic Reasoning, Commonsense Knowledge, Symbolic Reasoning, Truthful Reasoning, and Professional Knowledge Collectively, these datasets include a mix of open-ended generation tasks and multiple-choice questions. The specific datasets are detailed below:

- •
GSM8K (Cobbe et al., ): This dataset contains high-quality, linguistically diverse grade school math word problems. We utilize the test split, which contains 1319 samples.
- •
SciQ (Welbl et al., ) :
This dataset contains crowdsourced science exams. We use the test split for evaluation, which includes 1000 example. It is a multiple-choice dataset, each question offering four answer options. Similar to TruthfulQA and CommonsenseQA, we assign a letter to each answer option and request the model to output the corresponding answer letter.
- •
CommonsenseQA (Talmor et al., ): This dataset comprises a multiple-choice question-answering format that requires commonsense knowledge. We utilize the test split, which contains 1221 samples.
- •
TruthfulQA (Lin et al., ):(^5^55[https://huggingface.co/datasets/truthfulqa/truthful_qa/viewer/multiple_choice](https://huggingface.co/datasets/truthfulqa/truthful_qa/viewer/multiple_choice)) This dataset contains 817 questions designed to test whether the model can generate truthful responses without failing to recognize false beliefs and misconceptions. We utilize the multiple-choice format of the dataset and consider one single target answer. We randomly shuffle the answer options and corresponding true labels to ensure that the correct label is not predictably the first one. We format the questions as lettered multiple-choices and instruct the model to select the best answer from the options provided.
- •
Object Counting in BigBench (Srivastava et al., ): BigBench is a collaborative benchmark encompassing over 200 tasks. For Symbolic Reasoning, we focus solely on one subset, Object Counting, which includes exactly 1000 samples. This open-ended generation task evaluates whether models can correctly identify the number of objects mentioned in the question prompt.
- •
Professional Knowledge in MMLU (Hendrycks et al., ): MMLU is a multitask benchmark that includes multiple-choice questions from diverse knowledge domains. For the Professional Knowledge category, we combine the test sets from four subsets: Professional Accounting, Professional Law, Professional Medicine, and Professional Teaching.

## Appendix D Implementation Details

In this section, we describe the implementation details for all experiments.

### D.1 Reward Model
Training

This study utilizes two reward models. For Llama3-8B, we use an off-the-shelf checkpoint from [OpenRLHF/Llama3-8b-rm-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-rm-mixture) . For Mistral-7B, the reward model is trained from scratch using [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) as the starting point, referred to as Mistral-7B-RM.

#### D.1.1 Hyperparameters

We list the hyperparameters used for training Mistral-7B-RM in Table 5.

**Table 5: Hyperparameters for training Mistral-7B-RM.**
| Parameter | Mistral-7B |
| --- | --- |
| Train BS | 512 |
| Micro Train BS | 1 |
| Learning Rate | 2e-6 |
| Max Length | 8192 |
| LR Scheduler | cosine_with_min_lr |
| Warmup Ratio | 0.03 |
| Optimizer | AdamW |
| Weight Decay | 0.01 |
| Epoch | 2 |

### D.2 Reward Model Calibration

As stated in Section 3, we asssume reward models for calirbation are already trained beforehand and generally perform well. Therefore, we utilize trained RM checkpoints, namely [OpenRLHF/Llama3-8b-rm-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-rm-mixture) and Mistral-7B-RM for calibration. The calibrated versions are referred to as Llama3-8b-crm and Mistral-7B-crm.

#### D.2.1 Hyperparameters

We list the hyperparameters used for calibrating [OpenRLHF/Llama3-8b-rm-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-rm-mixture) and Mistral-7B-RM in PPO-M in Table 6.

**Table 6: Hyperparameters for Training Llama3-8B-crm and Mistral-7B-crm.**
| Parameter | Llama3-8b-crm | Mistral-7B-crm |
| --- | --- | --- |
| Train BS | 256 | 256 |
| Micro Train BS | 1 | 1 |
| Learning Rate | 9e-6 | 5e-6 |
| Max Length | 8192 | 8192 |
| LR Scheduler | cosine_with_min_lr | cosine_with_min_lr |
| Warmup Ratio | 0.03 | 0.03 |
| Optimizer | Adam | Adam |
| Epoch | 1 | 2 |

### D.3 PPO Training

Following the standard RLHF pipeline, we initialize the policy model using supervised fine-tuning checkpoints: [OpenRLHF/Llama3-8b-sft-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-sft-mixture) for Llama3-8B, and [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) for Mistal-7B.

For standard PPO and PPO-C, we utilize the pre-calibrated reward models, namely [OpenRLHF/Llama3-8b-rm-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-rm-mixture) and Mistral-7B-RM. For standard PPO, we input the entire sequence into the reward model to retrieve the score. For PPO-C, we perform our proposed calibrated reward calculation (see Section 3 for method descriptions).

For PPO-M, we use the calibrated reward models, Llama3-8b-crm and Mistral-7B-crm, to calculate reward scores.

#### D.3.1 Hyperparameters

For each model (Llama3-8B and Mistral-7B), we use a consistent set of hyperparameters across PPO, PPO-M, and PPO-C as shown in Table 7.

**Table 7: Hyperparameters for PPO Training.**
| Parameter | Llama3-8B | Mistral-7B |
| --- | --- | --- |
| Train BS | 64 | 64 |
| Micro Train BS | 2 | 2 |
| Micro Rollout BS | 4 | 4 |
| Rollout BS | 512 | 512 |
| Prompt max len | 1024 | 1024 |
| Generate max len | 1024 | 1024 |
| Actor Learning Rate | 5e-7 | 1e-7 |
| Critic Learning Rate | 9e-6 | 1e-6 |
| Actor Weight Decay | 0.0 | 0.01 |
| Critic Weight Decay | 0.0 | 0.0 |
| Init KL Conf | 0.01 | 0.05 |
| LR Scheduler | cosine_with_min_lr | cosine_with_min_lr |
| Warmup Ratio | 0.03 | 0.03 |
| Optimizer | Adam | Adam |
| Epoch | 1 | 1 |

### D.4 DPO Training

In Section 5.2, we extend calibrated reward modeling (PPO-M) to DPO training using Eq. 4. Similar to the calibration of reward models, we utilize trained DPO checkpoints.

For Llama3-8B, we use [Llama-3-Base-8B-SFT-DPO](https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT-DPO) as DPO checkpoint, and use [princeton-nlp/Llama-3-8B-Base-SFT](https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT) as reference model.

For Mistral-7B, we use [NousResearch/Nous-Hermes-2-Mistral-7B-DPO](https://huggingface.co/NousResearch/Nous-Hermes-2-Mistral-7B-DPO) as DPO checkpoint, with [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) serving as reference model.

#### D.4.1 Hyperparameters

We list the hyperparameters used for DPO training
[Nous-Hermes-2-Mistral-7B-DPO](https://huggingface.co/NousResearch/Nous-Hermes-2-Mistral-7B-DPO) and
[Llama-3-Base-8B-SFT-DPO](https://huggingface.co/princeton-nlp/Llama-3-Base-8B-SFT-DPO) in Table 8.
The same set of hyperparameters is applied to both DPO and CDPO; however, the scaling coefficient $w$ is not utilized in DPO.

**Table 8: Hyperparameters for DPO and CDPO Training.**
| Parameter | Llama3-8B | Mistral-7B |
| --- | --- | --- |
| Train BS | 128 | 128 |
| Micro Train BS | 1 | 1 |
| Max Length | 4096 | 4096 |
| Learning Rate | 3e-7 | 3e-7 |
| Beta | 0.01 | 0.01 |
| Weight Decay | 0.0 | 0.0 |
| LR Scheduler | cosine_with_min_lr | cosine_with_min_lr |
| Warmup Ratio | 0.03 | 0.03 |
| Optimizer | Adam | Adam |
| Epoch | 1 | 1 |
| Zero Stage | 3 | 2 |
| Adam Offload | True | False |
| $w$ (scaling coefficient) | 1.0 | 0.5 |

### D.5 Evaluation and Parsing

We detail the generation configuration, prompting strategies, and parsing strategies utilized in our evaluations.

#### D.5.1 Generation Configuration

We maintain consistent configuration settings across both preliminary and main experiments: temperature is set at 1.0, top-p at 1.0, and top-k at 50. The maximum token count is set at 16 for direct answers and 256 for zero-shot CoT.

#### D.5.2 Evaluation Hyperparameters.

Evaluations are conducted on a single A100 80GB GPU with a batch size of 8.

### D.6 Evaluation Prompts

Following the format described in  Tian et al. (), we modify the prompt to enhance clarity and simplify result interpretation. We consider two prompting stratgies for evaluation: Direct Answer and Zero-Shot CoT (Kojima et al., ). The exact prompt is shown in Fig 9 and Fig 10, which also include a model response from GSM8K. For answer_type: we use option letter for multiple-choice questions and number for open-ended math problems. For demo: we use (A) for multiple-choice questions and 1 for open-ended math problems. The prompt formatting utilizes the chat template in the tokenizer. Instructions are placed in the system prompt, and the question is placed in user prompt. For models like Tulu-2 (Ivison et al., ) that lacks a system prompt section in the tokenizer chat template, we append the question after the instruction as the user prompt.

Figure: Figure 9: Direct Answers Prompt.

Figure: Figure 10: Zero-Shot CoT Prompt.

### D.7 Parsing Details

##### Regex Parsing Details.

To parse the confidence score from model generation, we implement a stopping criterion that halts only when digits follow the phrase “Confidence: ”. For responses that are initially unparseable, we set a retry limit of ten attempts. If parsing failures persist, we manually append “Confidence: ” to the model response and resubmit it to the model for completion of the sentence and score generation.
This helps us achieve nearly 100% success in parsing all responses. If parsing fails, which is very rare, we use an empty string as the default answer. Instead of assigning an arbitrary confidence of 5.0,
which could introduce bias and decrease ECE,
we use the most frequently observed confidence scores from successfully parsed responses as the default value for these rare cases,
since they are the most representative confidence scores of the model.

Figure: Figure 11: Prompts for GPT4-o Evaluation.

##### GPT-4o Evaluation Details.

We use gpt-4o-2024-08-06 to evaluate zero-shot CoT results. Utilizing the structured output feature of GPT, we configure the model to output results in JSON format for easy parsing. The prompt is shown in Figure 11.

## Appendix E More Results and Analysis

### E.1 Overconfidence in RLHF-LLMs

In this section, we provide additional results for our preminlinary experiments, which demonstate overconfidence in RLHF-LLMs on five more datasets, as shown in Figure 12,  13,
14, 15,
and 16. The results indicate that RLHF-LLMs generally express verbalized overconfidence on various datasets.

Figure: Llama3-8B-SFT and Llama3-8B-PPO; Tulu-2-7B and Tulu-2-DPO-7B
Refer to caption: x7.png

Figure: Llama3-8B-SFT and Llama3-8B-PPO; Tulu-2-7B and Tulu-2-DPO-7B
Refer to caption: x11.png

Figure: Llama3-8B-SFT and Llama3-8B-PPO; Tulu-2-7B and Tulu-2-DPO-7B
Refer to caption: x15.png

Figure: Llama3-8B-SFT and Llama3-8B-PPO; Tulu-2-7B and Tulu-2-DPO-7B
Refer to caption: x19.png

Figure: Llama3-8B-SFT and Llama3-8B-PPO; Tulu-2-7B and Tulu-2-DPO-7B
Refer to caption: x23.png

### E.2 Reward Models are Biased Toward High Confidence
Scores

Figure: (a) [RLHFlow/ArmoRM-Llama3-8B-v0.1](https://huggingface.co/RLHFlow/ArmoRM-Llama3-8B-v0.1) (Wang et al., )
Refer to caption: x27.png

Figure: (a) [Skywork/Skywork-Reward-Llama-3.1-8B](https://huggingface.co/Skywork/Skywork-Reward-Llama-3.1-8B) (Liu & Zeng, )
Refer to caption: x37.png

Figure: (a) [OpenRLHF/Llama-3-8b-rm-mixture](https://huggingface.co/OpenRLHF/Llama-3-8b-rm-mixture) (Hu et al., )
Refer to caption: x47.png

Following Section 2.2, we present additional results from various reward models to further substantiate the observed phenomenon.

A concern arises that the reward model may be biased by the confidence-query system prompt, which
is included to ensure that the model verbalizes its confidence level.
To further investigate the impact of this system prompt, we conduct additional experiments both with and without the system prompt.
As shown in Figure 17,  18, and  19, the plots on the left follow the setting outlined in preliminary experiments, where a confidence-query system prompt is prepended and random confidence scores are appended to model responses. These plots demonstrate that every tested reward model exhibits a biased preference towards high-confidence responses to varying degrees. On the right, we still consider four modes, but this time without the confidence-query system prompts, and only random confidence scores are appended to the model responses. For instance, in rejected_with_conf, the comparison would involve the same chosen responses with a high confidence score against that with a low confidence score. The results reveal a similar, albeit more subtle, phenomenon.

### E.3 Calibrated Reward Models

Section 4.2 demonstrates the preference distributions of our calibrated reward model compared to the pre-calibrated version for Llama3-8B on rejected_with_conf. Here we present a complete result and also extend to the Mistral-7B model.

Figure: (a) chosen_with_conf
Refer to caption: extracted/5922513/sections/eval_graphs/comparison_chosen_Llama-3-8b-rm-mixture.png

Figure: (a) chosen_with_conf
Refer to caption: extracted/5922513/sections/eval_graphs/comparison_chosen_mistral-7b-hermes-rm-skywork.png

As shown in Figure 20 and  21, both calibrated models exhibit a similar trend. When evaluated on chosen responses with high and low confidence scores, the calibrated reward models do not exhibit the same level of certainty as their pre-calibrated counterparts. Additionally, when evaluated on rejected responses with high and low confidence scores, both calibrated models show a preference for the low-confidence responses, suggesting that our calibrated reward models are more capable at identifying overconfident statements.

### E.4 Parameter Sensitivity

In Eq. 3, we introduce a reward adjustment factor $\gamma$, defined as $\gamma=w*|\hat{r}_{i}|*(s_{i}-0.5)$. Here $w$ represents a scaling coefficient set at 0.5 for our main results.
We conduct a hyperparameter study in this section to investigate the impact of $w$. As shown in Table 9, when $w$ is increased to 1.0, the model demonstrates a lower ECE across all six datasets. However, it exhibits reduced AUC on SciQ and CommonsenseQA, possibly because the model becomes more conservative in its confidence yet lacks predictive power.
Additionally, the model scores lower on the MT-Bench benchmark, indicating a compromise in general capabilities.

**Table 9: Performance of PPO-C with different $w$ coefficient.**
| $w$ | MT-Bench | GSM8K | SciQ | CommonsenseQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ |  |  |
| 0.5 | 8.05 | 0.8638 | 0.516 | 0.1031 | 0.0282 | 0.6513 | 0.904 | 0.1286 | 0.5621 | 0.7756 |
| 1.0 | 7.76 | 0.8261 | 0.501 | 0.1092 | 0.0075 | 0.5641 | 0.903 | 0.1025 | 0.5076 | 0.7805 |
| $w$ | MT-Bench | TruthfulQA | Object Counting | Professional Knowledge |  |  |  |  |  |  |
| ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ |  |  |
| 0.5 | 8.05 | 0.4426 | 0.5303 | 0.4431 | 0.4839 | 0.5178 | 0.503 | 0.3949 | 0.4902 | 0.502 |
| 1.0 | 7.76 | 0.4271 | 0.5207 | 0.4345 | 0.4709 | 0.5318 | 0.505 | 0.388 | 0.5069 | 0.4883 |

### E.5 Impact of Confidence-Query System Prompts.

For the main experiments, we select 25% of the single-turn prompts to prepend a confidence-query system prompt. Here, we present our study on the effect of varying the percentage of single-turn prompts with this system prompt. As shown in Table 10, the impact on calibration does not show a consistent trend; however, we observe a decrease in MT-Bench scores as the percentage increases. Given our objective to maintain model capability while improving calibration, we opt for 25%.

**Table 10: Performance of PPO-M on Prompt Dataset with various percentage of single-turn prompts prepending confidence-query system prompts.**
| Percentage | MT-Bench | GSM8K | SciQ | CommonsenseQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ |  |  |
| 0.25 | 8.05 | 0.8393 | 0.57 | 0.119 | 0.0267 | 0.6115 | 0.898 | 0.1206 | 0.5568 | 0.7707 |
| 0.5 | 7.88 | 0.86 | 0.5185 | 0.1031 | 0.0389 | 0.5829 | 0.896 | 0.134 | 0.5399 | 0.7682 |
| 1.0 | 7.74 | 0.8608 | 0.5065 | 0.1243 | 0.0471 | 0.7165 | 0.898 | 0.074 | 0.6341 | 0.7658 |
| Percentage | MT-Bench | TruthfulQA | Object Counting | Professional Knowledge |  |  |  |  |  |  |
| ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ |  |  |
| 0.25 | 8.05 | 0.3991 | 0.5813 | 0.47 | 0.4789 | 0.5227 | 0.505 | 0.3848 | 0.4926 | 0.502 |
| 0.5 | 7.88 | 0.4453 | 0.5283 | 0.4357 | 0.5119 | 0.5413 | 0.473 | 0.3988 | 0.5221 | 0.4935 |
| 1.0 | 7.74 | 0.3438 | 0.5737 | 0.4786 | 0.5087 | 0.5052 | 0.487 | 0.3501 | 0.5184 | 0.502 |

### E.6 Can PPO-M and PPO-C be Combined?

As PPO-M and PPO-C function independently, we aim to explore whether these methods can be combined. Specifically, we apply the calibrated reward models from PPO-M while simultaneously using the calibrated reward calculation from PPO-C to generate scores. Table 11 demonstrates the performance of this combined method. We observe no improvement over individual methods, and sometimes even a decline in performance. We hypothesize that this is due to calibrated reward model being trained specifically on responses with confidence scores, which are trained to produce an unbiased reward. Thus, removing these confidence scores to estimate rewards based on comparison with threshold might not be appropriate.

**Table 11: Performance of PPO-Combine on Llama3-8B across six datasets.**
|  | MT-Bench | Arena-Hard | GSM8K | SciQ | CommonsenseQA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ |  |  |  |
| DA | 7.82 | 12.0 | 0.8948 | 0.5168 | 0.0978 | 0.1026 | 0.5094 | 0.896 | 0.2134 | 0.5415 | 0.774 |
| CoT | 7.82 | 12.0 | 0.2441 | 0.5108 | 0.7544 | 0.1434 | 0.5809 | 0.847 | 0.2553 | 0.631 | 0.7027 |
|  | MT-Bench | Arena-Hard | TruthfulQA | Object Counting | Professional Knowledge |  |  |  |  |  |  |
| ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ | ECE $\downarrow$ | AUC $\uparrow$ | ACC $\uparrow$ |  |  |  |
| DA | 7.82 | 12.0 | 0.533 | 0.575 | 0.4235 | 0.469 | 0.5 | 0.531 | 0.5256 | 0.5158 | 0.4535 |
| CoT | 7.82 | 12.0 | 0.5203 | 0.6263 | 0.4137 | 0.309 | 0.5 | 0.691 | 0.525 | 0.5573 | 0.4324 |

### E.7 Extension To DPO

**Table 12: Comparison over MT-Bench and Arena-Hard for Llama3-8B. The numbers in parenthesis are sourced from Meng et al. ().**
| Model | Method | MT-Bench $\uparrow$ | Arena-Hard $\uparrow$ |
| --- | --- | --- | --- |
| Llama3-8B | SFT | 6.44 (6.6) | 3.1 (3.3) |
| DPO | 7.67 (7.7) | 15.9 (15.9) |  |
| \arrayrulecolorgray\arrayrulecolorblack | DPO$\dagger$ | 7.52 | 15.2 |
|  | CDPO | 7.68 | 14.7 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: I didn't understand the paragraph at Lines 239-244. It sounds like you are saying you didn't train the reward model yourself for PPO-M, but I don't see how that could be, since you need a reward model trained on your new calibration dataset. Are you saying that you fine-tuned an existing reward model on the new calibration dataset? Please clarify.
soundness: 4
strengths: The paper studies an important and interesting topic, that of verbalized calibration in instruction-following for language models. It is easy to follow and clearly demonstrates the existence of the problem and the success of the proposed mitigation methods. The methods do not require new data to be collected, making them straightforward to use, and yet the improved calibration transfers from reward model scores to accuracy on multiple-choice datasets.
summary: In instruction-following for language models, the paper studies the task of reporting a confidence score in the helpfulness and accuracy of the model's own response. The paper shows that existing reward models are biased in favor of high-confidence scores, and proposes two mitigation methods. The first method involves data augmentation for the reward model, and the second method adds a customized term to the reward in PPO. The methods show improved calibration with no noticeable performance degradation.
weaknesses: Although the paper studies a broadly important topic and is well-executed, I think the strength of the contribution itself is somewhat limited:
- From a scientific perspective, the methods do not offer much insight: there is no distinction between different confidence levels other than "high scores" and "low scores", and the methods just increase the reward for high scores for high quality responses and low scores for low quality responses. So it is not really surprising that they improve calibration.
- This wouldn't be a problem if the methods had a lot of practical value. But from a practical perspective, the methods don't seem that useful either: if the user wanted a confidence score in the quality of the response, you could just display the reward model score (or some prompt-calibrated version) to the user. It's true that it's nice to have this distilled into the model, but there are many alternative ways in which one could do this, such as distilling the reward model score into a special token's logit. And if you just want a calibrated confidence score in the accuracy of a short-form response, there are many existing alternatives available.

In spite of this weakness, I consider the paper to be above the acceptance threshold because of the careful analysis the paper provides.

I also have a more minor issue with the definition of gamma for the PPO-C method (Line 279): the factor of |r_i (hat)| doesn't make sense to me. Unless the reward model is trained with an L2 penalty on the rewards or similar, the location of the reward is meaningless, since the loss uses only the difference between rewards, so taking an absolute value doesn't make sense. Even if the reward model is trained with an L2 penalty, I don't see why there should be a bigger adjustment for rewards that are far from 0 – or if there is some reason, it may be good to discuss it. Instead, to ensure that gamma is correctly scaled, it would make more sense to me to use some measure of spread of the rewards, such as standard deviation.

A couple typos:
- Line 134: "Dand" -> "D and"
- Line 1046: "calirbation" -> "calibration"

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - I understand that just for the sake of an evaluation of how the current chat models are conditioned on the given verbal confidence, appending a random confidence score to the RewardBench dataset may make sense. Why not simply prompt the chat models to self-annotate? Asking models to generate their own confidence scores could yield a more genuine assessment of how they internally gauge response certainty.

- Figure 2 primarily highlights instances of overconfidence, but are there scenarios where the SFT model consistently exhibits low confidence? It would be interesting to see if the proposed methods can help address underconfidence in addition to overconfidence, particularly if SFT models tend to understate confidence levels.
soundness: 2
strengths: - This paper is very well-written and easy to follow. The motivation is well-supported by evidence, including visualizations of confidence distributions and accuracy, that highlight the overconfidence phenomenon.
- The proposed methods are straightforward, yet backed by a sufficient amount of evaluation on different families of open-weights models and on diverse benchmarks. The thorough evaluation, along with the authors' transparency in presenting both the strengths and limitations of their approach, makes their findings trustworthy - small improvements in the factuality-focused benchmarks as expected (e.g. TruthfulQA benchmark) and almost no improvement in chat performance (e.g. MT-Bench, Arena-Hard).
summary: This paper focuses on LLM calibration: the alignment between the confidence a model expresses and its actual performance. Previous research has demonstrated the phenomenon of overconfidence in LLMs, especially those trained using RLHF, which tend to have a sharpened output probability. The systematic bias in reward models used for RLHF is one of the key causes of LLM overconfidence.

The paper proposes two methods to address the issue of verbalized overconfidence in LLMs trained with RLHF:

(1) **PPO-M**: a reward model re-training objective that aims to fine-tune existing BT-based reward models. PPO-M achieves this by augmenting existing pairwise ranking datasets with verbal confidence scores. The author filtered samples from various open-source datasets, prepended a confidence-query system prompt to each, and randomly assigned high and low confidence scores, creating a modified training dataset.

(2) **PPO-C**: a reward-shaping method that dynamically adjusts reward scores during PPO training based on a moving average of past rewards and the model's verbalized confidence level. PPO-C differs from vanilla PPO in that it incorporates a confidence-query system prompt into a portion of the training prompts to elicit verbal confidence scores from the model. The reward function promotes higher confidence for responses exceeding the moving average and lower confidence for those falling below.

---
*Nov. 26 update*: most of my concerns are resolved, satisfied with the added evaluations, and raised the rating from 6 ---> 8 while maintaining my confidence of 3.
weaknesses: - Referring to Eq. 2 as the "PPO-M" loss could be confusing, as it is not directly related to the standard PPO objective. This equation is, in fact, a re-training objective for the reward model rather than a modification of PPO itself. The labels "PPO-M" and "PPO-C" might suggest that the paper centers on enhancing PPO objectives, when the proposed methods primarily aim at co-training the reward model with verbal confidence. This distinction could benefit from clarification to prevent misunderstanding.

- While not introduced in this work, the use of verbal confidence to measure a model's confidence in its response may not be ideal. Language models, especially those under 70B parameters, may simulate confidence without true understanding due to limitations in reasoning capability. Have you examined the logits for the 0–10 range when asking for verbal confidence? Observing how these tokens are distributed might provide insight into whether the model is indeed producing a typical probability distribution or simply outputting values. Given these concerns, more concrete metrics, such as those derived from entropy in token-space, might be more accurate indicators of true confidence.

- The confidence metric presented in Figure 2 lacks clarity, and the "perfect calibration" line does not contribute meaningful differentiation between a model's overconfidence and cases where the dataset may simply be too easy. Maybe Llama-3-8B-PPO's performance on CommonsenseQA could indicate that the questions are too easy leading to high confidence in correct answers.

- Given that Figure 2 does not reflect the effects of PPO-M and PPO-C, it would be valuable to include a distribution of confidence scores after tuning with these proposed methods. This would enable a direct comparison of the effects of the calibration techniques on model confidence.

-  Figure 3 would benefit from results that combine the answer and confidence score to more clearly illustrate how the proposed methods affect the chosen/rejected ratio.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: - Regarding PPO-M, is it intended to be an additional objective to equation 1? Because there is nothing in equation 2 that will make the reward of the chosen answer higher than the rejected one.

- Line 270: what do you mean by “biased response”? Why does asking for confidence create bias?
soundness: 2
strengths: - The paper has a clear structure, beginning with analysis experiments, moving to the proposed method, and concluding with experiments that showcase its effectiveness.
- The experiments are comprehensive, covering multiple datasets and showing clear improvement in calibration metrics.
- The problem addressed is relevant and of interest to the community.
summary: The paper tackles the problem of calibration in LLMs that went through RLHF, and specifically calibration through verbalized confidence. Based on prior work that showed that RLHF hurt the model calibration, the authors point to the reward model as the source of the problem. They demonstrate that off-the-shelf reward model have a bias towards answers with high-confidence. As a solution, they propose two different methods that should overcome this problem - one focus on the reward model training and the other fix the rewards given during policy optimization.
weaknesses: - The conclusion drawn from Figure 2 doesn’t align with the plots. It’s not that the model after RLHF isn’t overconfident, but rather that the model before RLHF is also overconfident. Nearly all the bins fall below the diagonal line, indicating overconfidence. Additionally, why are the data points so sparse? How many prompts were used in this experiment?

- In Figure 3, since the reward models haven’t been trained with this type of prompt, I would suggest running a control experiment—appending the prompt with a random confidence score to both chosen and rejected answers (similar to PPO-M training). With random confidence scores, we would expect the reward model, on average, to still prefer the chosen answer. If it doesn’t, this would indicate that the prompt is so far out of distribution that the reward model isn’t applicable (which highlights its brittleness, though that’s a separate issue).

- While I understand the algorithm used in PPO-C, the intuition behind it isn’t clear to me. Why would the average reward across the dataset serve as a threshold for determining if a model’s response is correct? Couldn’t it be that most responses are either correct or incorrect, depending on the dataset and model? This approach seems like a heuristic that would only work on datasets where the model is correct about 50% of the time. 
Additionally, why is $\Delta r$ defined as a running average of $r$ rather than $\hat{r}$? On a related note, wouldn’t an off-the-shelf reward model struggle to assign rewards to answers that include confidence scores (i.e., $r$) since it wasn’t trained on such data?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. Why not add something similar to (1) onto (2) to enforce the preference of $(y_c,h_c)$ over $(y_r,l_r)$?
2. Why do PPO-M and PPO-C often result in lower ACC?
3. How does $\alpha$ affect the results of PPO-C?
4. Are you assuming that $\hat{r}(y_c)>\Delta r$ and $\hat{r}(y_r)<\Delta r$?

Overall, It is a well-motivated work presented in a structured manner. However, the contributions lack substantial impact. The proposed algorithms resemble engineering techniques rather than rigorous, principled approaches.

## Lower confidence after discussion
I maintain my critique of the heuristic algorithm's lack of rigorous design but am less certain if this empirical research meets ICLR standards. I've reduced my confidence to 3. 

To be honest, the algorithm's design and naming seem somewhat inelegant and naive. While empirical results are solid, nowadays it has become increasingly difficult to determine whether their success stems from effective code-level implementation, a limited dataset, or a genuinely strong algorithm.

Since other reviewers have given higher ratings, I am open to its acceptance as a poster. That said, I would be deeply disappointed if it were elevated to the level of a spotlight or oral presentation, as I believe this would set a poor precedent for ICLR's standards.
soundness: 2
strengths: 1. Good motivation to identify the systematic bias and propose intuitive solutions.
2. The presentation is clear and easy to follow, with a logical and well-reasoned train of thought.
3. Sufficient empirical results to justify the effectiveness of proposed methods
summary: This paper identifies a systematic bias in reward models that favors high-confidence responses regardless of their actual quality. They propose two solutions for this bias issue: PPO-M, which calibrates reward modeling by aligning confidence levels with response quality, and PPO-C, which adjusts standard reward model scores during PPO training. Empirical results show their methods can reduce expected calibration error without compromising accuracy or instruction-following ability in open-ended generation.
weaknesses: 1. Regarding the proposed PPO-M, there is no guarantee that the model would prefer $(y_c,h_c)$ over $(y_r,l_r)$. This could be a potential reason why the calibrated model results in lower ACC metrics than the uncalibrated PPO shown in the tables. Although the systematic bias may cause the model to prefer $h_c$ over $l_r$, PPO-M lacks an explicit mechanism to enforce this preference. Have you considered jointly optimizing (1) and (2) to enforce $y_c$ over $y_r$ as well?

2. Since PPO-C depends on a running average reward for confidence calibration, the $\alpha$ affecting the moving average could be an important factor. However, there seems no evaluation of how $\alpha$ affects PPO-C. I suggest experiments to show how varying $\alpha$ influences reward magnitude and performance. Moreover, are you assuming that $\hat{r}(y_c)>\Delta r$ and $\hat{r}(y_r)<\Delta r$? If so, would be helpful to explicitly state the assumption and analyze its appropriateness.

3. The design and evaluation of the proposed algorithms, as noted above, lack clarity or rigor. The formulations would benefit from a more detailed and persuasive explanation. Ideally, theoretical support should be provided to justify that the approach achieves debiasing without compromising model performance. At present, the methods appear to rely heavily on heuristics rather than principled algorithm design, which diminishes the significance of the contribution.

Meta-review: This paper investigates overconfidence in LLMs trained with RLHF, finding that reward models are biased towards high-confidence responses regardless of accuracy. To address this, the authors propose two heuristic methods: PPO-M, which calibrates reward models, and PPO-C, which dynamically adjusts rewards during training. These methods successfully reduce calibration error while maintaining performance comparable to standard PPO across various tasks. The paper's strengths lie in its clear identification of the problem, well-written presentation, and comprehensive evaluation. However, it also has weaknesses, such as limited scientific insight, reliance on verbal confidence, and a lack of theoretical justification. Despite these limitations, I am recommending acceptance based on paper's empirical results and transparent discussion of its limitations.

Decision: Accept (Poster)