Post-hoc Reward Calibration: A Case Study on Length Bias

## Abstract

Abstract Reinforcement Learning from Human Feedback aligns the outputs of Large Language Models with human values and preferences. Central to this process is the reward model (RM), which translates human feedback into training signals for optimising LLM behaviour.
However, RMs can develop biases by exploiting spurious correlations in their training data, such as favouring outputs based on length or style rather than true quality.
These biases can lead to incorrect output rankings, sub-optimal model evaluations, and the amplification of undesirable behaviours in LLMs alignment.
This paper addresses the challenge of correcting such biases without additional data and training, introducing the concept of Post-hoc Reward Calibration .
We first propose an intuitive approach to estimate the bias term and, thus, remove it to approximate the underlying true reward.
We then extend the approach to a more general and robust form with the Locally Weighted Regression.
Focusing on the prevalent length bias, we validate our proposed approaches across three experimental settings, demonstrating consistent improvements: (1) a 3.11 average performance gain across 33 reward models on the RewardBench dataset; (2) enhanced alignment of RM rankings with GPT-4 evaluations and human preferences based on the AlpacaEval benchmark; and (3) improved Length-Controlled win rate of the RLHF process in multiple LLM–RM combinations.
Our method is computationally efficient and generalisable to other types of bias and RMs,
offering a scalable and robust solution for mitigating biases in LLM alignment. Our code and results are available at https://github.com/ZeroYuHuang/Reward-Calibration .

## 1 Introduction

Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ; Kaufmann et al., ) is driving the success of Large Language Models (LLMs) (OpenAI, ; Anil et al., ; Dubey et al., ).
By integrating human preferences into the training loop, RLHF aligns models with desired behaviours and human values (Chakraborty et al., ), enabling LLMs to generate content that is conceptually appropriate (Sun et al., ), ethically sound (Hendrycks et al., ; Dai et al., ), and reflecting nuanced human judgements (Wu et al., ).
A key component in this process is the reward model (RM), which translates qualitative human feedback into a signal for optimising LLMs.
The primary purpose of RM training is to approximate human preferences, allowing it to evaluate the content generated by LLMs by assigning scores.
These scores (rewards) are then used to optimize the model, shaping its behaviour to better meet human expectations.

However, there is a risk that an RM may exploit spurious correlations in the training data that do not genuinely reflect the output quality (Stiennon et al., ; Gao et al., ; Singhal et al., ; Pang et al., ; Lambert & Calandra, ).
For instance, an RM might favour outputs based on length (Park et al., ) or style (Lambert et al., ; Dubois et al., ), leading to biased assessments that do not accurately represent human preferences.
The involvement of a biased RM in developing LLMs can have significant drawbacks.
Firstly, it may lead to incorrect rankings of outputs, favouring content that aligns with learned biases over genuinely high-quality outputs.
This issue is particularly problematic when RMs are used to rank different models during LLM evaluations, as a biased RM could produce misleading results, endorsing suboptimal models and impeding progress in model development. Moreover, when biased RMs are employed in RLHF, these biases can be further amplified (Zheng et al., ), with the model learning to exploit these biases rather than improving genuine task performance—a phenomenon known as reward hacking.

Recent advancements introduce various strategies to mitigate these biases and alleviate reward hacking during different stages of RLHF, from data handling (Park et al., ) and reward model engineering (Coste et al., ) to the final reinforcement learning phase (Chen et al., ). Most of these approaches, however, either require additional data collection, re-engineering of the RM, or modifications to reinforcement learning algorithms, which raises an intriguing question: is it possible to correct or mitigate biases in reward signals in a training-free manner?
Specifically, given a set of scored outputs from the RM and a specific bias that the model may exhibit, how can we calibrate these reward signals to reflect the true quality of the outputs more accurately?

This paper makes an initial attempt at answering this question by framing it as Post-hoc Reward Calibration.
Specifically, for a particular characteristic of the output (e.g., the length), we assume the biased reward from the RM can be decomposed into the sum of (1) the underlying calibrated reward and (2) a bias term that solely depends on that characteristic.
Under certain assumptions, we propose to use the local average reward to provide an estimation for the bias term, which can then be removed, thereby approximating the true reward.
We then extend it to be more general and robust using Locally Weighted Regression for bias estimation.
Among different biases (Zhang et al., ), a particularly prevalent one is the sequence length. Prior works shown that models generate significantly longer responses after RLHF without improving the genuine quality of the content (Kabir et al., ; Wu et al., ; ).
Singhal et al. () point out that even a
purely length-based reward can reproduce most downstream RLHF improvements of existing RMs, highlighting that current RMs fail to convincingly outperform simple heuristics and may be strongly biased by the length. The issue is also evident in powerful proprietary models like GPT-4, which is increasingly used as a judge, resulting in LLM evaluations that favour more verbose responses.
Focusing on length bias, we validate our method in three settings and observe consistent improvements:
(1) Benchmark performance: a 3.11 performance gain averaged over 33 RMs on the RewardBench;
(2) RM as LLMs evaluators: based on AlpacaEval, we utilise 8 open-source RMs to rank 184 LLMs.
After applying our calibration, the rankings correlate better with GPT-4 evaluations and human preferences.
(3) RM for LLMs alignment:
we assess calibrated rewards for RLHF process. Testing with four LLM–RM configurations, we observe consistent improvements in AlpacaEval2 Length-controlled win rates.

In addition to being computationally efficient (e.g., calibrating over 300k samples takes only 30 seconds with a single CPU), we empirically reveal that our method can generalise to other biases, such as the presence of markdown-style features in the outputs, and to pairwise GPT4-as-Judge models. Importantly, the calibration effect is strongly correlated with the RM’s preference for specific properties: for strongly biased RMs, the calibration yields greater improvements, while for weakly biased RMs, it only slightly alters its rewarding behaviour.
These findings suggest that our proposed Post-hoc Reward Calibration method enhances the reliability of RMs and offers a practical, scalable solution for mitigating biases in RLHF.
This approach could be instrumental in advancing RLHF methodologies and developing more robust and aligned LLMs.

## 2 Related works

#### Reward Models

Given a prompt $q$ and two corresponding responses $y_{1}$ and $y_{2}$, training an RM usually involves training a classifier to predict the human preference probability $p^{*}(y_{1}\succ y_{2}|q)$ of two responses $y_{1}$ and $y_{2}$ with the Bradley-Terry (BT) model (Bradley & Terry, ):

$$ $p^{*}(y_{1}\succ y_{2}|q)=\sigma(r^{*}(x_{1})-r^{*}(x_{2}))$ (1) $$

where $x_{i}=(q,y_{i})$ represents the prompt-response pair, $r^{*}$ is the latent true reward we cannot access, and $\sigma$ is the logistic function.
Then, given a reward modelling dataset $\mathcal{D}=\{(q^{i},y^{i}_{1},y^{i}_{2})\}_{i=1}^{N}=\{(x_{1}^{i},x_{2}^{i})
\}_{i=1}^{N}$, where $y_{1}^{i}$ is annotated as a better response to prompt $q_{i}$ compared to $y_{2}^{i}$, training a RM $r_{\theta}$ is to minimise the negative log-likelihood for a binary classification problem:

$$ $\mathcal{L(\theta)}=-\mathrm{E}_{(x_{1},x_{2})\sim D}[\log{\sigma\left(r_{ \theta}(x_{1})-r_{\theta}(x_{2})\right)}]$ (2) $$

In practice, RMs are often implemented by adding an extra linear layer to LLMs, which projects the representations of the last layers of LLMs to a scalar as the reward.
There are also two other types of RMs:
(1) DPO-based: Models optimised with Direct Preference Optimisation (DPO) (Rafailov et al., ) could function as an implicit RM.
(2) LLM-as-Judge: A generative LLM can also be prompted to function as an RM to generate feedback, usually consisting of a piece of chain-of-thought reasoning and a final score representing the overall quality (Zheng et al., ; Kwon et al., ; Kim et al., ).
In this paper, we validate the effectiveness of our methods across all three types of judge models. Specifically, we employ our method to calibrate BT-based and DPO-based RMs on the RewardBench dataset, and to calibrate the pairwise judge of GPT-4 with the AlpacaEval benchmark.

#### Bias Mitigation for RLHF

Recent works explore the bias mitigation, also termed as mitigate reward over-optimisation, for the RLHF process mainly from the perspective of (1) Data handling:  Park et al. () identifies six types of inherent bias in various judge models and employ LLMs to generate the de-biasing dataset for further fine-tuning.
Zhu et al. () propose to iteratively replace hard labels in data with soft labels during RM training, while
Rita et al. () rely on human expert demonstration to recalibrate the reward objective optimized with Reinforcement Learning.
(2) Reward Model Engineering:
Coste et al. (); Eisenstein et al. (); Sun et al. (); Zhang et al. () implement the reward ensemble;  Ramé et al. () first fine-tune multiple RMs and then average them in the weight space;
Dai et al. () decouple the reward model training for helpfulness and harmlessness and  Chen et al. () disentangle the length signal and the quality signal into two different reward heads.
Shen et al. (); Wang et al. (); Quan () exploit the idea of Mixture-of-Experts (Qiu et al., ; ) in RM training to capture different facets of human preference to avoid the potential conflict between them;
(3) Reinforcement Learning methods: Another research line
(Moskovitz et al., ; Sun et al., ; Zhou et al., ; Zhai et al., )
modifies the algorithm or optimisation objective during reinforcement learning with  Park et al. (); Zhu et al. (); Lu et al. () specifically focusing on the length bias.
Our work takes a different approach from previous methods by employing post-hoc reward calibration, i.e., using only a batch of scored prompt-response examples to mitigate the reward model’s bias, without intervening the preference data collection, RM training, and the RLHF phase.

## 3 Method

### 3.1 Problem Statement: Reward Calibration

The reward calibration problem could be formalised as follows:
given an input–output pair $x$ to be evaluated, the reward model $r_{\theta}$ assigns a scalar $r_{\theta}(x)$ to represent its quality.
However, having been trained on the human preference dataset, the reward model $r_{\theta}$ may learn shortcuts to approximate the human preference and thus have biases regarding some characteristics of the input–output pair $x$. For example, it tends to assign higher rewards to outputs with longer length or containing knowledge commonly encountered in the real-world data (Park et al., ).
Supposing a measurable characteristic of interest $c$ is a function that maps the input–output pair $x$ to a scalar $c(x)$:

$$ $\displaystyle{\rm c}:$ $\displaystyle{\mathbb{X}}$ $\displaystyle\mapsto$ $\displaystyle\quad{\mathbb{R}}$ $\displaystyle x$ $\displaystyle\rightarrow$ $\displaystyle\quad c(x)$ $$

We assume that the biased reward from the reward model $r_{\theta}(x)$ could be decomposed into two terms, an underlying calibrated reward $r^{*}_{\theta}(x)$ and a bias $b_{c}^{\theta}(c(x))$ with regards characteristic $c$:

$$ $r_{\theta}(x)=r^{*}_{\theta}(x)+b^{\theta}_{c}(c(x))$ (3) $$

The RM is usually used to calculate the margin between two outputs $(x_{1},x_{2})$ to predict the human preference:

$$ $\Delta_{r_{\theta}}(x_{1},x_{2})=r_{\theta}(x_{1})-r_{\theta}(x_{2})=r^{*}_{ \theta}(x_{1})-r^{*}_{\theta}(x_{2})+b^{\theta}_{c}(c(x_{1}))-b^{\theta}_{c}(c (x_{2}))$ (4) $$

Thus, the reward calibration problem is to estimate the reward difference owing to the bias term and subtract it to recover the underlying true reward margin $\Delta^{*}_{r_{\theta}}(x_{1},x_{2})=r^{*}_{\theta}(x_{1})-r^{*}_{\theta}(x_{2})$.

Post-hoc Reward Calibration:
The proposed reward calibration problem is applicable to various stages of RLHF. Depending on the focus, it can be tailored either toward collecting debiased preference data, refining the training of reward models, or even enhancing the alignment procedure by adjusting the alignment objectives.
In contrast, this paper focuses on the post-hoc setting, performing reward calibration after the reward assignment without interfering with any data collection or RM training stage. Formally, given $N$ already-rewarded data points $\{x_{1}^{i},x_{2}^{i},\Delta_{r_{\theta}}(x_{1}^{i},x_{2}^{i})\}_{i=1}^{N}$, we aim to find the calibrated reward margin $\hat{\Delta}^{*}_{r_{\theta}}(x_{1}^{i},x_{2}^{i})$ that is closer to the underlying oracle margin $\Delta^{*}_{r_{\theta}}(x_{1}^{i},x_{2}^{i})$ for every pair $(x_{1}^{i},x_{2}^{i})$.

#### Assumptions

We make the following hypothesis about the decomposition in Eq. 3.
(1) Independence of the biased characteristic: consider a general reward modeling dataset composed of prompts and corresponding responses pairs $\mathcal{D}=\{(q^{i},y^{i}_{1},y^{i}_{2})\}_{i=1}^{N}=\{(x_{1}^{i},x_{2}^{i})
\}_{i=1}^{N}$, the expectation of the underlying gold reward margin should be 0 and is independent of the biased characteristic $c$,

$$ $\mathbb{E}_{(x_{1},x_{2}\sim\mathcal{D})}\left[\Delta^{*}_{r_{\theta}}(x_{1},x _{2})\right|c(x_{1})=c_{1},c(x_{2})=c_{2}]=0$ $$

(2) Sufficient Density: the function values of the characteristic function $c$ is assumed to be sufficiently dense in its range $c({\mathbb{X}})$.
(3) Lipschitz Continuity: the bias term $b_{c}^{\theta}$ is a slow-varying function of characteristic $c$, which means if pairs $x_{1}$ and $x_{2}$ are close to each other regarding the characteristic $c$, their corresponding bias term in Eq. 3 should be close as well. Mathematically, given $c(x_{0})$ and a neighbourhood $\mathbb{V}(x_{0})=\{c(x)\mid\lvert c(x)-c(x_{0})\rvert<\epsilon\}$ of $x_{0}$, for all $c(x_{1}),c(x_{2})\in\mathbb{V}$, there exists a constant $K$ such that $\lvert b_{c}^{\theta}(c(x_{1}))-b_{c}^{\theta}(c(x_{1}))\rvert<K\lvert c(x)-c(
x_{0})\rvert$;

### 3.2 Bias Estimation

#### Uniform Averaging Approach

According to Eq. 4, the reward margin between $x_{1}$ and $x_{2}$ consists of (1) the underlying calibrated margin $\Delta^{*}_{r_{\theta}}(x_{1},x_{2})$ and (2) the biased “bonus reward” that solely depend on the characteristic $c(x_{1})$ and $c(x_{2})$. Intuitively, to estimate $b_{c}^{\theta}(c(x_{1}))-b_{c}^{\theta}(c(x_{2}))$, we ask:

Given the characteristic measures $c(x_{1})$ and $c(x_{2})$ for an arbitrary pair $(x_{1},x_{2})$, what would be their reward margin?

A straightforward estimation for this question could be

$$ $\mathbb{E}[r_{\theta}(x)\mid c(x)=c(x_{1})]-\mathbb{E}[r_{\theta}(x)\mid c(x)= c(x_{2})].$ (5) $$

Under the assumption of Lipschitz continuity and sufficient density, we can obtain a simple calibrated reward margin described in the Eq. 6:

$$ $\hat{\Delta}^{*}_{r_{\theta}}(x_{1},x_{2})=\Delta_{r_{\theta}}(x_{1},x_{2})- \left(\mathbb{E}[r_{\theta}(x)\mid|c(x)-c(x_{1})|<d]-\mathbb{E}[r_{\theta}(x) \mid|c(x)-c(x_{2})|<d]\right)$ (6) $$

where $d$ is a threshold distance that governs the neighbourhood size around $x_{1}$ and $x_{2}$.

#### Locally-Weighted-Regression Method

How do we choose the threshold distance $d$ in Eq. 6? (1) It must be sufficiently small to ensure that the bias term can be approximated as a constant function within the local neighbourhood, enabling the application of Eq.6; and (2) it must also be large enough to meet the sufficient density assumption, ensuring sufficient data points used for estimation. Satisfying these conflicting requirements may necessitate thorough hyper-parameter tuning of $d$ to employ the intuitive approach in practice. Furthermore, the density may not be uniform in practice, so that the optimal threshold distance $d$ may vary for different data points $c(x)$.
Therefore, a desirable approach should be able to estimate the expectations in Eq. 5 in a more general, robust, and practical way by (1) relaxing the constant-function assumption and (2) mitigating the sensitivity to the choice of neighbourhood size, possibly by incorporating certain adaptive mechanisms to optimise the neighbourhood size or the contribution for each datapoint dynamically. Both desiderata motivate us to employ Locally Weighted Regression (LWR) towards a more robust estimation.

Specifically, given a point of interest (e.g., a characteristic value $c(x)$ in our case), the Locally Weighted Regression (LWR) method first uses a bandwidth parameter $f$ to define the fraction of the dataset utilised for regression. Then the method assigns weights to neighbouring data points, giving higher weights to points closer to the target and lower weights to those farther away. This approach ensures that nearby points have a greater influence on the regression. LWR then fits a weighted linear regression to model the local behaviour of the target function, effectively approximating the weighted average within the specified local context.

LWR addresses the desiderata by offering a more adaptable approach to estimate the Eq. 5.
It relaxes the constant-function assumption by introducing the linear approximation to capture the subtle local variations in the data. This is reasonable because the bias terms are slow-varying functions and can be linearly approximated using Taylor’s expansion.
Furthermore, LWR mitigates sensitivity to neighbourhood size by using a distance-based weighting mechanism, which assigns greater importance to closer data points. This adaptive weighting scheme dynamically adjusts the influence of each data point, reducing reliance on a fixed neighbourhood size and enhancing the robustness and flexibility of the estimation process.
Formally, given a dataset consisting of the characteristic $c$ in outputs and corresponding rewards $D_{c}=\{(c(x_{j}),r_{\theta}(x_{j}))\}_{j=1}^{N}$ and a bandwidth $f$, our general method could be formulated as:

$$ $\hat{\Delta}^{*}_{r_{\theta}}(x_{1},x_{2})=\Delta_{r_{\theta}}(x_{1},x_{2})- \left(\hat{r}_{\theta}(c(x_{1}))-\hat{r}_{\theta}(c(x_{2}))\right)$ (7) $$

where $\hat{r}_{\theta}(c(x_{1}))$ and $\hat{r}_{\theta}(c(x_{2}))$ are the LWR prediction for the characteristic $c(x_{1})$ and $c(x_{2})$, respectively, given the dataset $D_{c}$.
To determine the preference, we use the sign of reward margin between $x_{1}$ and $x_{2}$: $\operatorname{sgn}(\hat{\Delta}^{*}_{r_{\theta}}(x_{1},x_{2}))$.
In practice, we employ the Robust Locally Estimated Scatterplot Smoothing (LOWESS), a robust version of LWR.
Based one LWR, it defines different weights $\delta_{j}$ for the point $(c(x_{j}),r_{\theta}(x_{j}))$ according to the size of residual $r_{\theta}(x_{j})-\hat{r}_{\theta}(c(x_{j}))$.
It assigns smaller weights to data points with large residuals and large weights to those with small residuals.
New fitted values are computed using LWR again but with $w_{j}$ replaced by $\delta_{j}w_{j}$. The computation of new weights and fitted values is repeated several times to converge. The full algorithm of using LOWESS for reward calibration is described with Algorithm 1.
We directly employ off-the-shelf library statsmodels.api and pass the dataset $D_{c}$ to it for our implementation.

#### Length Bias Estimation

This paper prioritises the prevalent length bias presented in the reward model. Thus, the characteristic of interest here is the character length of the output, i.e., $c(x)=|x|$. The dataset used for regression consists of the length of the output and its corresponding rewards.

## 4 Experiments

We test the performance of the proposed reward calibration method in three settings:
(1) RM Benchmark performance: we test the calibrated RM’s performance on the RewardBench dataset (Lambert et al., ).
(2) RM As LLMs Evaluator: Based on the AlpacaEval benchmark (Dubois et al., ), we employ RMs to label the output of different LLMs from AlpacaEval leaderboard and produce a ranking of LLMs. Then, we measure the correlation between the RM’s ranking and GPT-4 ranking (AlpacaEval2) or human preference ranking (ChatbotArena).
(3) RM for LLMs alignment: We utilise the calibrated rewards rather than the original reward to label the preference data, and then we conduct the Direct Preference Optimization (Rafailov et al., ) with the labelled preference pair. We evaluate the aligned model’s performance using AlpacaEval2 and eight popular benchmarks.
In all three settings, we are given a group of data points $D_{|x|}=\{x_{i},r_{\theta}(x_{i})\}$ for calibration. Because LOWESS requires calculating the residual for every fitted data point, we employ the entire $D_{|x|}$ for regression.
Therefore, for RewardBench, there are 2,985 test data points, so we employ all 5,970 data points to perform the regression (every test data is a pair of responses for one prompt).
Regarding AlpacaEval, there are 184 LLMs and 805 instructions for each model, resulting in 151k samples used for calibration.
For LLMs alignment, five responses are generated from the model for about 60k instructions, leading to about 300k samples used for bias estimation.
The proposed method is computation-efficient, and calibrating 300k samples only takes 30 seconds using a single CPU.

#### Compared Algorithms

For all three experimental settings, we try to compare the following algorithms:
(1) Original reward;
(2) Length penalty: a broadly adopted approach to mitigate the length bias in reward models. It adds a penalty term to the original reward (Singhal et al., ; Park et al., ; Liu et al., ; Dong et al., ), formalised as $\hat{r^{*}}(x)=r_{\theta}(x)-\alpha\times|x|$.
Following Dong et al. (), we use the character length of each output and set $\alpha=0.001$.
(3) RC-Mean: our proposed uniform averaging calibration method described in Eq. 6.
(4) RC-LWR: our proposed Locally Weighted Regression calibration method shown in Eq. 7.
(5) RC-LWR-Penalty: to see whether our method could be applied on top of the penalty-based methods, we first apply the length penalty to the original reward and then use the RC-LWR.
All hyper-parameters related to RC-Mean, RC-LWR and RC-LWR-Penalty are presented in Appendix B.

### 4.1 Length Calibrated Rewards on RewardBench

#### Setting and Metrics

We first validate our proposed calibration algorithms using the RewardBench dataset.
Each data point in RewardBench comprises one prompt $q$ and two LLMs-generated responses $y_{w}$ and $y_{l}$.
The final score is calculated as the averaged accuracy of reward models in identifying $y_{w}$ over four subsets: Chat, Chat Hard, Safety, and Reasoning.
We mainly investigate 33 BT-based RMs whose rewarding scores are officially available(^1^11[https://huggingface.co/datasets/allenai/reward-bench-results](https://huggingface.co/datasets/allenai/reward-bench-results)) and whose accuracy is above 50% (random guess) from the RewardBench Leaderboard as of Aug. 2024.

Figure: Figure 1: Results of different calibration algorithms on the RewardBench benchmark are shown in two charts. Left: the line chart demonstrates the RewardBench score before calibration (x-axis) and after calibration (y-axis) of different algorithms for different models. Right: the scatter plot highlights the performance gains achieved by different calibration methods, annotated with the maximum, average, and minimum values.
Refer to caption: x1.png

#### Results

The overall results for different calibration methods are illustrated in Fig. 1.
The line chart on the left showcases the calibrated performance (y-axis) w.r.t original performance (x-axis) of different calibration algorithms for different RMs. The scatter plot on the right highlights the performance gains for different calibration methods. Our empirical findings include: (1) The simple coarse-grained Length Penalty method already leads to a notable improvement (orange line v. blue line), to some extent indicating the prevalence and significance of the RM’s length bias. (2) The reward calibration techniques we introduce yield improvements for nearly all RMs. Among these techniques, the RC-LWR-Penalty achieves a 3.11 performance gain averaged over all 33 models. Though it could lead to performance degradation (5 models for RC-LWR and 4 models for RC-LWR-Penalty), the degradation is very slight and the worst case is only about -1 point for RC-LWR and RC-LWR-Penalty.
(3) More importantly, besides improving reward models with moderate performance, the proposed calibration method could also boost the performance of strong reward models. For example, after RC-LWR-Penalty calibration, nine reward models surpass the accuracy of GPT4-Turbo, and the best calibrated model even approaches the latest state-of-the-art (SOTA) LLM-as-Judge models.
We also calibrate DPO-based reward models with our proposed method and report results in the Appendix C.

### 4.2 Length Calibrated Reward as LLMs Evaluators

#### Settings

A critical application for RM is to serve as a judge to replace human evaluation for evaluating LLMs.
Most existing methods (Dubois et al., ; ; Li et al., ; Kwan et al., ) use strong generative LLMs as the judge model.
For instance, AlpacaEval (AE1) uses the GPT4’s output logits to calculate the preference probability of the evaluated LLM’s output, $y$, against the responses of the baseline LLM (GPT4-1106-preview), $y_{b}$, for each prompt $q$.
This preference probability is then averaged to determine the win rate to rank different LLMs.
AlpacaEval2 (AE2) further introduced a length-controlled win rate, a regression-based method to mitigate GPT4’s length bias.
In our setting, we primarily use BT-based RMs to score the LLM’s output.
Given a prompt $q$ and corresponding output $y$ from the evaluated LLM and output $y_{b}$ from the baseline LLM, the RM scores them as $r$ and $r_{b}$, respectively; then the Bradley-Terry model yields the preference probability $p(y\succ y_{b}|q)=\sigma(r-r_{b})$. We then compute the average of $p(y\succ y_{b}|q)$ across all prompts to determine the final win rate for ranking all evaluated LLMs.
We directly use the model’s output from the AlpacaEval official repo.(^2^22[https://github.com/tatsu-lab/alpaca_eval/tree/main/results](https://github.com/tatsu-lab/alpaca_eval/tree/main/results))
Additionally, we apply the length-controlled regression method for our BT-based RMs as one of our baselines, denoted as LC.
To further validate the generality of our proposed method, we also apply RC-LWR to AE1.
Since AE1 uses GPT4 to compare two responses pairwise and uses its output logits to compute the preference probability $p(y\succ y_{b}|x)$, following Eq. 1, we employ the inverse function of $\operatorname{sigmoid}$ to calculate the reward margin $r-r_{b}$. We then utilise the LWR to directly predict the biased reward margin given the length margin $|y|-|y_{b}|$.

#### Metrics

We look at the following metrics following Dubois et al. ():
(1) Gameability: this measures whether the win rate given by the evaluator is significantly affected by simply promoting the same model to be more concise and verbose. Specifically, five models are selected to be prompted three times: (a) normally, (b) to “give as much detail as possible”, (c) to “be as concise as possible while still providing all the necessary information to answer the question” and get three win rates. Then, the gameability is defined as the normalised variance between those three win rates and averaged across these five models.
Higher gameability indicates that the evaluator is more sensitive to the output’s length.
(2) Spearman correlation of reward model produced ranking with official AlpacaEval2 leaderboard over 184 models.
(3) Spearman correlation of reward model produced ranking with ChatbotArena.
We employ three versions of ChatbotArena (CA): CA Feb. is used by Dubois et al. (), we introduce it to ensure that we reproduce their results, CA Aug. is the latest one that is more recent and includes more models, and the CA. Length is the CA Aug. disentangled with length proposed by Chiang et al. ().

Figure: Figure 2: The Gameability $\downarrow$ and LengthBiasBench accuracy $\uparrow$ for selected BT-based reward models. The lower the Gamebility, the less sensible the win rate is to different prompt strategies. The higher the accuracy on the LengthBiasBench, the less preference the model has for output length. The Gameability of AlpacaEval1 (AE) and AlpaceEval2 (AE2) win rates are 11.05% and 27.9%, respectively, presented with the dashed line.
Refer to caption: x3.png

Figure: Figure 3: The heatmap demonstrates the Spearman correlation ($\uparrow$) between reward-models-produced rankings with the AlpacaEval2 (AE2) and ChatbotArena (CA). We also report the calibration results for GPT4-1106 on the right, with the Original Reward indicating AlpacaEval1, LC representing AE2, and RC-LWR referring to rankings calibrated with our proposed RC-LWR method. The x-axis represents different calibration methods. The two highest correlations in each row are marked with a dashed rectangle . We select three versions of CA. CA Feb. is used in the paper of AE2 and is selected to validate our reproduction of relevant results. CA Aug. is the latest ranking and contains more models. CA Length is the CA Aug. disentangled with the response length.
Refer to caption: x4.png

**Table 1: The Gameability $\downarrow$ of different calibrations methods across various reward models on AlpacaEval**
| Model | Length Penalty | LC | RC-Mean | RC-LWR | RC-LWR-Penalty |
| --- | --- | --- | --- | --- | --- |
| internlm2-7b-reward | 13.6% | 10.6% | 9.4% | 9.0% | 25.6% |
| GRM-llama3-8B-distill | 12.1% | 9.5% | 8.8% | 9.0% | 16.7% |
| FsfairX-LLama3-RM-v0.1 | 20.2% | 12.7% | 10.8% | 10.1% | 19.0% |
| BTRM-Qwen2_7b | 18.5% | 13.8% | 10.5% | 10.6% | 17.5% |
| GRM-llama3-8B-sftreg | 23.4% | 13.7% | 9.7% | 10.0% | 14.8% |
| RM-Mistral-7B | 20.5% | 14.2% | 10.8% | 10.5% | 19.8% |
| Mistral-RM-for-RAFT | 22.0% | 13.5% | 9.9% | 9.6% | 19.5% |
| Eurus-RM-7b | 37.8% | 18.3% | 12.2% | 11.8% | 11.8% |

#### Identifying RMs with strong length bias

Before calibration, we first evaluate the top-10 BT-based RMs from RewardBench leaderboard as of Aug. 2024 using the Gameability and the length bias subset of the EvalBiasBench dataset (Park et al., ).
The latter contains 17 test examples and is denoted as LengthBiasBench.
We aim to identify RMs demonstrating severe length bias, i.e., high Gameability and low accuracy on LengthBiasBench.
Their results are shown in Fig. 2.
We observe the prevalence of length bias for BT-based RMs. 8 of 10 models exhibit high Gameability and poor performance on LengthBiasBench.
The two metrics demonstrate consistency, i.e., the model with high Gameability usually achieves low accuracy on LengthBiasBench, achieving -0.879 Spearman correlation score.
As Llama-3-OffsetBias-RM and internlm2-20b-reward do not demonstrate severe length bias, we primarily report results for other 8 models.

#### Results

The Spearman correlations of RM-produced rankings with the AE2 and the ChatbotArenas are presented in the Fig. 3.
Our findings include:
(1) Overall, RC-Mean and RC-LWR provide effective calibrations, leading to a higher correlation with AE2 and ChatbotArena, achieving comparable calibration effects with LC on all eight RMs. Note that LC is specifically designed for the AE leaderboard and can not be straightforwardly employed for reward calibration because it requires the model ID and instruction ID for regression, which are not available for general reward calibration.
(2) Compared with AE2, RC-LWR calibrated AE achieves comparable Spearman correlations with Chatbot Arena, suggesting that our method can generalise to LLM-as-Judge results.
(3) The calibrated GRM-llama3-8B-distill achieves 0.975 with ChatbotArena Feb., and the calibrated FsfairX-LlaMA3-RM-v0.1 achieves 0.975 with AE2 and 0.951 with ChatbotArena Aug., demonstrating the potential of calibrated open-source RM to provide reliable LLM evaluations.
(4) The Length Penalty brings improvements in most cases. But it can also lead to degeneration (internlm2-7b-reward) or can be not effective at all (Eurus-RM-7b).
This is because different RMs have different reward scales, making it harder to employ the same penalty weight for different RMs. On the contrary, our proposed RC-LWR is more robust to different hyper-parameter choices, rendering it easier to use in practice.
(5) Unlike the results on RewardBench, the Length Penalty and RC-LWR-Penalty are less effective in this setting, likely due to differences in length margin ( denoted as $||x_{1}|-|x_{2}||$) distributions of two dataset. For instance, in the RewardBench, 60% of data points have a length margin below 600, while in the AlpacaEval setting, this figure is less than 40%. Consequently, the penalty term has a more significant impact on the AlpacaEval calibration. More relevant analyses are shown in the Appendix C. This underscores the practicality of our method over the penalty-based approach, as the latter may require extensive hyperparameter tuning for penalty weight. Such tuning is influenced not only by the reward model’s scale but also by the output’s length distribution.
(6) The rankings calibrated by LC, RC-Mean, RC-LWR are usually more aligned with CA Length compared with CA Aug., further validating the effectiveness of our proposed methods.
(7) As shown in Tab. 1, RC-Mean and RC-LWR effectively reduce the Gameability for various reward RMs, demonstrating the calibrated rewards are less sensitive to different prompting techniques and thus more reflective of the response quality.

### 4.3 Length Calibrated Rewards for LLMs alignment

#### Setting

In this setting, we investigate whether the calibrated rewards could serve as a better training signal for LLMs alignment.
Because the post-hoc reward calibration assumes a scored dataset used for calibration, which could be seamlessly integrated with the offline RL, we choose to first employ the RMs to score the on policy dataset and then perform DPO.
Our experiment pipeline could be described as follows: for each prompt $q$ in a prompt dataset, multiple responses $y_{i}$ are sampled from an LLM, then scored as $r_{i}$ by an RM. Among these responses, the response with the highest or lowest score is selected to create a preference pair $(p,y_{w},y_{l})$ for Direct Preference Optimisation (DPO) tuning.
Specifically, to validate the effectiveness of our method, we choose two instruct models: Meta-Llama-3-8B-Instruct (AI@Meta, ) and gemma2-9b-it (Team, ) and two reward models: FsfairX-LLaMA3-RM-v0.1 (Dong et al., ) and GRM-llama3-8B-sftreg (Yang et al., ), resulting in 4 LLM-RM configurations:
(1) Llama-3-8B-Fsfairx;
(2) Llama-3-8B-GRM;
(3) gemma2-9b-Fsfairx;
and
(4) gemma2-9b-GRM.
For each LLM–RM combination, we compare the Original Reward, Length Penalty, and RC-LWR.
We directly utilise the on-policy generations(^3^33[huggingface.co/datasets/princeton-nlp/llama3-ultrafeedback-armorm](https://huggingface.co/datasets/princeton-nlp/llama3-ultrafeedback-armorm) and [huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm](https://huggingface.co/datasets/princeton-nlp/gemma2-ultrafeedback-armorm)) of two models generated by Meng et al. (). They employ the 60k instructions from the UltraFeedback (Cui et al., ) dataset and sample 5 responses for each instruction from the model.
Then, we use those two reward models to score the dataset and to conduct DPO tuning.
All hyper-parameters related to DPO tuning are set following Meng et al. ().

#### Metrics

We evaluate the model with the Length-Controlled win rate from the AE2 for open-ended generation.
We also test the model on eight popular benchmarks: MMLU (Hendrycks et al., ),
GSM8K (Cobbe et al., ),
ARC-challenge (Clark et al., ), HellaSwag (Zellers et al., ), Winogrande (Sakaguchi et al., ),
IFEval (Zhou et al., ),
GPQA (Rein et al., ),
and BBH (Suzgun et al., )
to see how the proposed calibration affects the benchmark performance.
We employ the OpenCompass (Contributors, ) for benchmark evaluation and report the average score of these benchmarks.
All evaluation configurations are set as a default in OpenCompass.

#### Results

Results of employing calibrated rewards for LLMs alignment are demonstrated in Tab. 2, where the left and the middle focus on the LC win rate and the right on benchmark performance.
We observe that:
(1) Similar to the experimental results with the RewardBench, the Length Penalty serves as a simple yet effective baseline, highlighting the significant issue of length exploitation in LLM alignment (Meng et al., ).
(2) Without extra training and data annotation, the RC-LWR outperforms the Original Reward baseline on AE2 by about 9.5 for Meta-Llama-3-8B-Instruct-based DPO and about 7 points for gemma2-9b-it. It effectively imrpove the output quality while preserving the length.
(3) The DPO tuning leads to performance drops on our eight benchmarks.
The drop is more significant for gemma2-9b-it-based configurations (e.g., for gemma2-9b-fsfairx, the performance drops from 63.69 to 55.28). This may be because a larger learning rate of 5e-7 is set for gemma2-based experiments compared to 3e-7 for Llama3-based experiments following Meng et al. (). However, with the same hyper-parameters, both Length Penalty and RC-LWR methods could mitigate the performance drop, indicating that the length-regularised alignment not only controls the output length but also reserves the model’s benchmark performance during DPO alignment.

**Table 2: Results of using length-calibrated rewards for LLMs alignment. Calibration methods we compare include Original Reward (OR), Length Penalty (LP), and RC LWR. We apply these methods in four different LLM–RM configurations. We report the Length-Controlled win rate, the average character length on the AlpacaEval, and average performance on eight benchmarks. The Table illustrates that RC-LWR calibration achieves up to 10% LC win rate improvement and alleviates the performance drop on benchmarks.**
| Algorithm | no DPO | OR | LP | RC LWR | no DPO | OR | LP | RC LWR | no DPO | OR | LP | RC LWR |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Length-Controlled | Avg Character | Avg Performance |  |  |  |  |  |  |  |  |  |
|  | Win Rate | Length in Alpaca | on 8 Benchmarks |  |  |  |  |  |  |  |  |  |
| Llama-3-8B-Fsfairx | 23.91 | 41.82 | 49.98 | 51.37 | 1967 | 2404 | 1989 | 1867 | 62.69 | 60.45 | 61.75 | 61.94 |
| Llama-3-8B-GRM | 23.91 | 41.00 | 48.41 | 50.49 | 1967 | 2438 | 1996 | 1858 | 62.69 | 60.94 | 62.11 | 61.83 |
| gemma2-9b-Fsfairx | 46.96 | 62.79 | 67.46 | 69.54 | 1521 | 2365 | 1491 | 1798 | 63.69 | 58.88 | 60.52 | 60.79 |
| gemma2-9b-Grm | 46.96 | 63.21 | 66.28 | 70.45 | 1521 | 2106 | 1426 | 1780 | 63.69 | 55.28 | 57.28 | 58.31 |

## 5 Analysis

**Table 3: The average and variance of absolute correlation score for different methods on RewardBench.**
|  | Original Reward | Length Penalty | RC-Mean | RC-LWR | RC-LWR-Penalty |
| --- | --- | --- | --- | --- | --- |
| $|\rho|_{\text{avg}}$ | 0.2930$\pm$0.1836 | 0.2092$\pm$0.1706 | 0.0390$\pm$0.0369 | 0.0233$\pm$0.0223 | 0.0229$\pm$0.0223 |

#### Does the calibration mitigate the length preference of models?

One desideratum for an ideal calibration method is that the rewards should be weakly correlated with the bias characteristics of interest, thus cancelling the model’s preference for specific characteristics and preventing reward hacking in its downstream applications.
Therefore, we calculate the average and variance of the absolute Spearman correlation score over RMs for different methods on the RewardBench dataset and show them in Tab. 3.
Our observation includes:
(1) Length bias is significant for the board spectrum of RMs (0.293 for original reward).
(2) The Length Penalty cannot effectively mitigate the model’s preference for length. The added penalty term renders RM to prefer shorter outputs.
(3) In contrast to Length Penalty method, our proposed methods could effectively calibrate the RM’s preference on output length, i.e., the correlation score between output length and reward is approximately 0 for all RMs after calibration.

#### Calibration behaviours on models with different length preferences.

If an RM exhibits a weak preference for the characteristic of interest, an optimal calibration should not significantly alter its rewarding behaviour and vice versa.
Rewarding accuracy alone may not effectively capture this aspect, as a calibration might result in an equal number of correct and incorrect predictions, thereby leaving the overall accuracy unchanged. We thus count how many preference pairs are reversed by the calibration on the RewardBench dataset. We report the fraction of it over the entire dataset. The results are shown in the Fig. 4.
Our observations are as follows:
(1) Length Penalty demonstrates the moderate correlation between RM length bias and the preference overturned and is unstable. It does not calibrate any pairs for some models with strong positive length preferences. For models with a strong negative preference for length, it is ineffective as the model already prefers shorter outputs in general.
(2) Both RC-Mean and RC-LWR methods exhibit a strong correlation: the weaker the bias of the model, the fewer pairs are reversed by the calibration method, indicating that our calibration method will not severely affect the non-biased model.
The desirable feature may enhance the practicability of the proposed method.

Figure: Figure 4: The number of preferences reversed by the calibration (y-axis) for models with different length preferences (x-axis), measured by the Spearman correlation with length. To determine the correlation between x and y, we further report the Spearman Correlation between them. For RC-Mean and RC-LWR, as the plot is approximately symmetric, we report the correlation between absolute values of x and y.
Refer to caption: x5.png

#### Generalisation to other bias types: markdown features.

Figure: Figure 5: The calibrated w.r.t original performance of 33 markdown-feature-calibrated BT-based models.
Refer to caption: x6.png

To validate that our proposed method could generalise to other bias features, we apply it to calibrating the bias towards the markdown listing features in the output, motivated by the empirical findings of Dubois et al. ().
Observing that LLMs often utilise markdown style lists with bold subheadings as the output format, we define the characteristic function $c(x)$ as the number of headers, lists, and bold strings following  Chiang et al. ().
The calibration results for 33 BT-based RMs are shown in Fig. 5. We find that calibrating the number of lists in the output also yields improvement on the RewardBench, with a 1.81 averaged performance gain for RC-Mean and 1.86 for RC-LWR. Furthermore, similar to length bias calibration, improvements are observed across nearly all models, and the originally good reward models get slightly stronger after calibration.
The gains are not as big as the gains for calibrating length bias, which demonstrates, to some extent, that the length bias is more prevalent.

#### Further Analysis

In Appendix C, we show that the proposed method is robust to different bandwidth choices of LWR (Fig. 7). And its calibration effect can be smoothly adjusted by introducing a constant before the LWR prediction (Fig. 8 and Tab. 4). We also demonstrate that RMs with more substantial bias benefit more from our approach (Fig. 10). We further showcase the potential of the proposed method to calibrate bias from multiple characteristics simultaneously (Fig. 11). Moreover, a stress test evaluating the data efficiency of the method reveals that our method is not significantly constrained by the size of the datasets used for the calibration (Fig. 13).

## 6 Conclusion

This paper introduces a post-hoc reward calibration method to mitigate biases in RMs. Our approach effectively reduces the impact of prevalent biases, such as length bias, and has been validated across multiple experimental settings. The results consistently show substantial performance gains.
Other empirical findings confirm that our method satisfies several critical desiderata for optimal calibration:
(1) Our approach consistently mitigates both length and markdown biases in RMs, and can successfully calibrate LLM-as-Judge rewards, underscoring its generality.
(2) The method requires no additional data annotation or RM retraining and is computationally efficient.
(3) The calibrated rewards exhibit weak correlations with the characteristics of interest, thereby preventing reward hacking and improving the overall quality of the reward signals.
(4) For non-biased RMs, our method minimally alters the reward behaviour, whereas it substantially improves models with stronger biases, making the calibration impact proportional to the bias level.
(5) The method is robust to various hyperparameter choices and can be controlled smoothly by introducing a calibration constant before LWR prediction, enhancing its practical usability.
(6) The approach is not constrained by dataset size, as even a few hundred data points are sufficient to achieve effective calibration.
In conclusion, our approach offers a scalable, efficient, and generalisable solution for addressing biases in RMs, ensuring better alignment between LLMs and human preferences.

## Acknowledgements

IT is supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053).

## Appendix A Algorithms

The algorithm describing employing the LOWESS method for length bias calibration is shown in Algorithm 1.
The main idea is to first fit a locally weighted regression and then adjust the weight based on the residual between regression prediction and the ground truth label.
LOWESS is a non-parametric regression technique typically used for smoothing a scatterplot. It is particularly useful when no specific functional form is assumed. Key uses of LOWESS include: (1) Data Smoothing: LOWESS is used to smooth out noise in data, providing a clearer trend by creating a smooth curve that captures the underlying pattern.
(2) Data Analysis: By overlaying a LOWESS curve on a scatterplot, one can explore the potential relationship between variables without relying on predefined models or functions, and it’s particularly useful when the relationship between variables is complex and non-linear, as it adapts to the shape of the data.
Based on the typical usage of LOWESS, in this paper, we employ LOWESS to investigate the potential relationship between rewards from a biased RM and a characteristic in the output. By assuming that they are actually independent, we regard the prediction from the LOWESS as the bias term and remove it to recover the underlying true reward.

Figure: Algorithm 1 LOWESS for Length Bias Calibration

## Appendix B Experimental Details

#### Hyper-parameters

Two key hyper-parameters for our proposed method, RC-Mean and RC-LWR, are the threshold $d$ and the bandwidth $f$.
(1) Determine the threshold $d$: The way we use to determine the threshold distance $d$ in Eq. 6 is as follows: (1) Taking the length bias calibration as an example, given a dataset of data pairs to calibrate $\{(x_{1}^{i},x_{2}^{i},r_{\theta}(x_{1}^{i}),r_{\theta}(x_{2}^{i})\}$, we first calculate the average of $||x_{1}^{i}|-|x_{2}^{i}||$, noted as $\operatorname{AVG}(||x_{1}|-|x_{2}||))$, (2) then we set $d=\operatorname{AVG}(||x_{1}|-|x_{2}||))/4$, which is set by intuition. And if the number of data points within the neighborhood is smaller than 10, we do not calibrate that pair as the results may be overly affected by specific data points. This set of hyper-parameters is employed for all three different experimental settings.
(2) Determine the bandwidth $f$ for LOWESS: this hyper-parameter is closely related to the assumption. Regarding the experiment of RM as LLM evaluators and RM for LLM alignment, we consider the sufficient density assumption valid since there are 150k samples and 300k samples, respectively. We thus set the bandwidth as default as in statsmodels.api, $f=\frac{1}{3}$, and the number of iterations is 3.
(Though we use all scored data points for LOWESS regression, we show in Fig. 12
and Fig. 13 that the methods yield improvements with only hundreds of data points.)
Regarding the experiment on the RewardBench, there are only 2,985 data pairs, resulting in 5970 data points. Therefore, the sufficient density assumption may be invalid for some length intervals, so we choose a larger bandwidth $f=0.9$.
Importantly, all hyper-parameters relevant to the reward calibration haven’t been tuned for specific reward models.
For RewardBench experiments, hyper-parameters for all 33 BT-based reward models and 46 DPO-based reward models are the same, and for AlpacaEval calibration, the hyper-parameters used for 8 models are the same. We argue that this further validates the generality of our proposed methods.
In practice, one could conduct a more comprehensive hyper-parameter search for a specific reward model.

#### Computational Environments

All experiments about reward calibration are performed with a single CPU.
Experiments for getting the reward model’s score on AlpacaEval’s output are conducted using a NVIDIA H100 GPU.
Experiments regarding using DPO for LLM alignment use 8 NVIDIA H100 GPU and costs about 2-3 hours for the DPO training.

## Appendix C Extra Experimental Results

#### Calibration Results For DPO-based Reward Models

In addition to calibrating the traditional BT-based RMs, we calibrate length bias with DPO-based reward models.
Formally, given a pair of completions $(y_{1},y_{2})$ to the prompt $q$, the preference of the DPO-trained model $\pi_{\theta}$ could be determined by comparing the following log-ratios:
$\frac{\pi_{\theta}(y_{1}|p)}{\pi_{ref}(y_{1}|p)}$ and $\frac{\pi_{\theta}(y_{2}|p)}{\pi_{ref}(y_{2}|p)}$, where $\pi_{ref}$ is the reference model during the DPO training.
We directly employ the official rewarding results from the RewardBench leaderboard.
The reasons why we choose to present the results for DPO-based models in the appendix instead of the main text are twofold: (1) The rewarding results for DPO-based model are more noisy as some of their reference models are not available; (2) To the best of our knowledge, DPO-tuned models are more frequently used as chat models or instruct models, and are rarely employed to indicate the preference with the implicit reward.
The results are shown in Fig. 6.
Our observations are as follows:
(1) The overall reward performance of DPO-based models is inferior to that of BT-based reward models, as well as their improvements after calibration.
(2) The proposed methods, RC-LWR and RC-LWR, are effective in most cases, achieving average performance gains of 1.60 and 1.68, respectively.
(3) The method RC-Mean can lead to improvements for most models. However, it can also result in performance collapse. For instance, after calibration, the performance of Qwen1.5-72B-Chat decreases from 71.5 to 49.9.
(4) The Length Penalty shows no calibration effect for most DPO-based models because the absolute values of log ratios, used as implicit rewards, are usually high. This finding further emphasizes that the penalty weight requires more comprehensive research in practical applications and is highly dependent on the specific reward scales of the model and the length distribution of the dataset.

Figure: Figure 6: Results of different calibration algorithms on the RewardBench benchmark for DPO-based reward models are shown in two charts. Left: the line chart demonstrates the RewardBench score before calibration (x-axis) and after calibration (y-axis) of different algorithms for different models. Right: the scatter plot highlights the performance gains achieved by different calibration methods, annotated with the maximum, average, and minimum values.
Refer to caption: x7.png

#### Robustness to the choice of bandwidth

Figure: Figure 7: Calibration results, shown in the improvements, of RC-LWR (top) and RC-LWR-Penalty (bottom) with different bandwidth $f$ on the RewardBench dataset.
Refer to caption: x9.png

We investigate the extent to which the calibration effect is influenced by different choices of bandwidth $f$. The experiments are conducted using RewardBench because, for the RewardBench setting, there are only 6k data points, and the smaller number of total data points for calibration may increase sensitivity to variations in bandwidth. The overall results for different bandwidths for both RC-LWR and RC-LWR-Penalty are presented in Fig. 7.
Our empirical findings are as follows:
(1) Overall, the proposed methods demonstrate robustness across varying bandwidths, achieving similar average performance gains when the bandwidth ranges from 0.1 to 1.0. For RC-LWR (top), the average gain ranges from 2.68 to 2.96, while for RC-LWR-Penalty, it ranges from 2.84 to 3.11. A slight performance improvement is observed when $f$ increases from 0.1 to 0.9.
(2) Although the average performance gains remain relatively consistent, the maximum improvements and degradations exhibit more notable trends. Specifically, as the bandwidth increases, both the maximum improvements and degradations diminish. These results suggest that calibration can be controlled through bandwidth selection, with larger bandwidths yielding more stable calibration effects.

#### Controlling calibration effects

One critical assumption for our proposed method is that the underlying gold reward is independent of the biased characteristic. We also showcase that our method could reduce the correlation between rewards and the characteristic of interest to 0.
However, the independence assumption may be invalid if one focuses on a specific subset of instructions.
For example, the independence assumption is considered valid given a general prompt set. But if we add “generate contents as concise as you can” at the end of every prompt, thus the underlying true reward should be positively correlated with the output length for this prompt set.
Therefore, a more practical calibration method should be controllable for users in different application scenarios.
We show that our proposed method can also be controlled by introducing a hyper-parameter named calibration constant $\gamma$ to the Eq. 7:

$$ $\hat{\Delta}^{*}_{r_{\theta}}(x_{1},x_{2})=\Delta_{r_{\theta}}(x_{1},x_{2})- \gamma\times\left(\hat{r}_{\theta}(c(x_{1}))-\hat{r}_{\theta}(c(x_{2}))\right)$ (8) $$

We validate the effects of $\gamma$ using two experimental settings: RM on the RewardBench and RM for LLM alignment. The ablation results on the RewardBench dataset are presented in Fig.8
, while those for the LLM alignment setting are shown in Table 4
.
A noteworthy statistic for the RewardBench dataset is that simply selecting the shorter response yields a 60% accuracy. Thus, the ideal correlation between rewards and response length should be slightly negative. The results in Fig. 8 support this notion: if $\gamma=1$ results in zero Spearman correlation as indicated in the Table. 3, the optimal $\gamma$ should exceed 1. Our findings indicate that the best average performance gain occurs when $\gamma=1.4$.
More importantly, the results across both settings demonstrate that the calibration effect can be smoothly controlled through the calibration constant. This significantly enhances the practical utility of the proposed method, as one can adjust this constant to achieve superior rewarding performance if certain prior knowledge about the correlation between rewards and the characteristic of interest is known.

Figure: Figure 8: Ablation results of different calibration constant on the RewardBench for BT-based reward models.
Refer to caption: x11.png

**Table 4: Results of using Length-Calibrated rewards for LLMs alignment with different calibrate constant $\gamma$ in RC-LWR. We apply these methods with four different LLM–RM configurations. The top reports the Length-controlled win rate, and the bottom showcases the average length of generations on AlpacaEval dataset.**
| Algorithm | 0 | 0.2 | 0.4 | 0.6 | 0.8 | 1.0 |
| --- | --- | --- | --- | --- | --- | --- |
| Length-Controlled Win Rate |  |  |  |  |  |  |
| Llama-3-8B-Fsfaix | 41.82 | 42.59 | 46.39 | 48.12 | 50.25 | 51.37 |
| Llama-3-8B-GRM | 41.00 | 43.54 | 46.44 | 49.76 | 49.61 | 50.49 |
| gemma2-9b-Fsfaix | 62.79 | 63.96 | 62.69 | 67.57 | 68.87 | 69.54 |
| gemma2-9b-Grm | 63.21 | 67.23 | 66.95 | 67.27 | 68.27 | 70.45 |
| Average Character Length on Alpaca Benchmark |  |  |  |  |  |  |
| Llama-3-8B-Fsfaix | 2404 | 2365 | 2340 | 2194 | 2028 | 1867 |
| Llama-3-8B-GRM | 2438 | 2375 | 2303 | 2194 | 1986 | 1858 |
| gemma2-9b-Fsfaix | 2365 | 2321 | 2570 | 2028 | 1725 | 1798 |
| gemma2-9b-Grm | 2106 | 2117 | 1939 | 1860 | 1638 | 1780 |

#### Length distribution

Defining the length margin as the absolute difference in length between two completions, the cumulative distribution function of the length margin on four different datasets is shown in Fig. 9. As shown in the Fig. 9, different datasets demonstrate distinct length margin distributions. For Ultrafeedback-Llama3 and Ultrafeedback-Gemma2, about 80% of data pairs have a length margin that is less than 600, which is reasonable because the compared completions are sampled from the same model. And the RewardBench dataset demonstrates a similar pattern.
In contrast, the length margin on the AlpacaEval benchmark is a lot longer, with only about 40% of the data points having a length margin that is less than 600.
Notably, the simple Length Penalty method works well in calibrated RM on RewardBench and calibrated RM for LLM alignment settings. This may be because the length margin in those two settings is smaller, thus leading to moderate penalization. However, when length margins are more significant, more meticulous adjustments for penalty weight are required as they are more sensitive.
In summary, the baseline Length Penalty is already strong when reward models exhibit strong length bias. However, both reward scales and length margins should be considered to adjust the penalty weight, impairing its practicability. In contrast, the proposed method RC-LWR is robust to different hyper-parameter choices, yielding stable improvements in different settings.

Figure: Figure 9: The cumulative distribution function of the length on different datasets used in our experiments.
Refer to caption: x12.png

#### Reward models with more substantial bias are improved more.

Figure: Figure 10: Based on the Chat Hard subset of RewardBench, we plot how the performance (left) or improvement brought (right) by RC-LWR vary with the model’s length preference, which is represented by the Spearman score between output length and rewards on RewardBench.
Refer to caption: x13.png

Results in Fig. 4 motivate us to investigate how calibration gains vary for models with different length bias. Therefore, we selected Chat Hard, the most challenging subset of RewardBench, for our investigation.
Results are shown in Fig. 10.
We use the Spearman correlation to represent the reward models’ length preference on the x-axis; we then plot the model’s performance and RC-LWR improvements, different points in Fig. 10 refers to different models.
In Fig. 10 left, we observe that the model’s performance on Char Hard is negatively correlated with the model’s length preference, achieving -0.28 correlation. (Note that a rule-based model that always chooses the shorted sentence could achieve an accuracy of 90% on Chat Hard subset.)
In Fig. 10 right, we observe a strong positive correlation between the model’s length preference and its improvement on the Chat Hard subset, achieving 0.76 correlation score. Note that similar trends are found between length preference and overall performance improvement of RewardBench with 0.47 Spearman Correlation score, both indicating that Reward Models with stronger length preference are improved more by the proposed RC-LWR method.

#### Calibrate for more than one characteristic

Figure: Figure 11: Results of calibrating with output length and markdown features simultaneously.
Refer to caption: x14.png

One straightforward extension to calibrate with more than one characteristic is to employ multi-dimensional Locally Weighted Regression.
For example, if we consider calibrating two characteristics $c_{1}$ and $c_{2}$ simultaneously, the dataset employed to do the LOWESS regression could be described as $D=\{(c_{1}(x_{j}),c_{2}(x_{j}),r_{\theta}(x_{j}))\}_{j=1}^{N}$. Then, the Euclidean distance is employed to measure the distance between different data points. Based on this, we extend our method to directly calibrate rewards with output length and markdown features.
Because these two characteristics have very different scales, we first normalize their values by Z-score normalization.
The calibration results are shown in the Fig. 11.
We observe that: (1) the proposed method RC-LWR could directly generalize to calibration for multiple characteristics, yielding a 2.58 average performance gain,
(2) the gain is superior to solely calibrating markdown bias (1.86) but inferior to solely calibrating length bias (2.93).
Possible reasons for no synergy effect include: a. The two characteristics of interest here, length and markdown features, may correlate, and the method does not touch on thiat; b. the Euclidean distance used here considers two characteristics equivalent.

#### Data efficiency

LOWESS is a non-parametric method that requires ground truth labels to compute residuals during its robustifying iterations, making it logical to utilize all available data points for calibration. We demonstrate that even with a moderate dataset size (2,985 preference pairs in RewardBench), calibration yields significant improvements. To explore this further, we conduct a stress test on the number of data points used for calibration.
Focusing on the RewardBench dataset, our experimental procedure involves sampling a subset of the dataset, performing calibration on the selected subset, and using accuracy as the evaluation metric. It is important to note that accuracy is not directly comparable to the RewardBench score in our main experiments. The RewardBench score is the average accuracy across four subsets (Chat, Chat Hard, Safety, and Reasoning), which vary in size. We define the subset size as a fraction of the original dataset, ranging from 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, to 0.5. For each subset size, we run ten experiments with different random seeds and average the performance across these seeds.
The results for different subset size for RC-LWR-Penalty, RC-LWR, RC-Mean are shown in the Fig. 12, Fig. 13 and Fig. 14, respectively.
We observe that the intuitive method, RC-Mean, achieves increasing accuracy as the subset size grows, while RC-LWR and RC-LWR-Penalty exhibit more stable improvements across different sizes, demonstrating that our method is not heavily constrained by data density.

Figure: Figure 12: Calibration results with different subset size for RC-LWR-Penalty method
Refer to caption: x16.png

Figure: Figure 13: Calibration results with different subset size for RC-LWR method
Refer to caption: x17.png

Figure: Figure 14: Calibration results with different subset size for RC-Mean
Refer to caption: x18.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: How is the sensitivity of the length penalty hyperparameter to the performance compared to the proposed methods?
soundness: 2
strengths: **Originality**: The paper proposes a unique, post-hoc calibration approach to mitigate reward model biases related to LLM output length without requiring additional data or retraining.

**Efficiency**: This method is computationally efficient, applying bias correction as a post-processing step, making it easily applicable across tasks without modifying reward models.

**Empirical Validation**: The approach is thoroughly evaluated on three tasks, effectively demonstrating its capacity to mitigate length bias.
summary: This paper proposes a novel method, “Post-hoc Reward Calibration,” to mitigate biases in reward models (RMs), particularly focusing on LLM output text length bias in RMs for large language models (LLMs). The method removes this bias by calibrating the reward score without additional RM retraining, making it computationally efficient. The approach assumes independence between the true reward and text length, constructs a locally linear regression model from text length to (proxy) reward values using the Robust Locally Estimated Scatterplot Smoothing (LOWESS), and calibrates the (proxy) reward with this regression model. The authors evaluate the approach on three tasks—RewardBench, RM as LLMs Evaluator, and RM for LLMs Alignment—and demonstrate that the calibrated RMs better align with human evaluations by providing fairer, length-independent scoring.
weaknesses: **Assumption**:
The assumption of “Independence of the biased characteristic” may be overly restrictive in practical applications, especially given that this study focuses on text length as the biased characteristic. My understanding of this assumption (line 166) is that even when $x_1$ and $x_2$ are fixed at any arbitrary lengths $c_1$ and $c_2$, respectively, the true reward difference between $x_1$ and $x_2$ would be expected to be zero, implying no correlation between reward and text length. However, in practical applications, this assumption will often not hold. For example, in cases where prompts explicitly request brevity or detail, such as “respond concisely” or “provide a detailed explanation,” reward often depends on the response length. This assumption could thus limit the generalizability of the proposed method, especially for tasks where length naturally influences quality. A more flexible approach that adjusts for prompt-specific length expectations could enhance applicability.

**Baseline setting**:
The Length Penalty hyperparameter $\alpha$ seems to be fixed at 0.001, which may not be optimal across different reward models (RMs) and tasks due to variations in scale. While the proposed method benefits from adaptive $d$ tuning, the baseline might perform better with an α adjusted per RM or task, ensuring a fairer comparison. For instance, $\alpha$ could be proportional to the RM’s reward scale.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: With regards to the second weakness above, it would be helpful if the authors could plot the learned $\hat{r}_\theta(c)$ function for several reward models to see what kind of affect subtracting it has compared to the linear length bias $\alpha \\, c$. Also, it would be helpful to compare to a baseline of a linear length bias where the factor $\alpha$ is tuned per reward model.
soundness: 3
strengths: The proposed method is relatively simple and is well-motivated conceptually. The fact that it can be applied post-hoc means that retraining is not required, which is helpful. I am not very familiar with the literature in this area but to my knowledge it is also novel. The results are extensive and seem to generally support that post-hoc reward calibration effectively removes length bias.
summary: This paper focuses on correction of spurious correlations in LLM-based reward models learned from preference data. The authors assume that the spurious feature (e.g., length) biases the rewards in a consistent away. They propose to estimate this consistent bias via a locally-weighted-regression method. Then, the estimated bias can be subtracted from learned reward in order to achieve a model without the spurious correlation. The authors focus on length as the spurious feature and compare to a baseline which subtracts a constant factor times the length from the reward. They evaluate a variety of reward models with various adjustments to eliminate length bias across several metrics, including RewardBench scores, correlation with human rankings of LLMs in ChatbotArena, and length-controlled win rate in AlpacaEval2. In general, the proposed method appears to better remove length bias.
weaknesses: The main weaknesses of the paper include:

 * **Presentation:** I found the introduction of the method in Section 3 to be relatively clear, but the experiments in Sections 4 and 5 were much more difficult to understand. While I eventually could figure out the gist of the experiment design, it is often quite complicated and could be better presented. In particular, Section 4.2 is very dense. It describes multiple evaluation methods (AlpacaEval1, AlpacaEval2, ChatbotArena), which if I understand correctly in some cases used as both gold standard (AlpacaEval2) and as baselines (LC). The combination of many reward models, LLMs, calibration methods, and comparison rankings makes it quite important to carefully describe the purpose behind the experimental setup and why it is a good evaluation of the calibration methods. I think the current text requires several readings to understand these details. The rest of Sections 4 and 5 are also very dense and difficult to understand at first. It would be good to improve the presentation of these such that the results are easier to understand.

 * **Lack of insight into the result of reward calibration:** while the authors present many experiments on how reward calibration affects downstream performance, they do not present what the actual calibration process looks like for typical reward models. It would be helpful to see what the actual $\hat{r}\_\theta(c)$ function looks like over a range of values for $c$ for various reward models. This could help readers understand why RC-Mean and RC-LWR outperform length penalty, especially if the $\hat{r}\_\theta(c)$ function is quite non-linear. On the other hand, if it is close to linear, then it would raise the question of why not simply tune the constant $\alpha$ in the linear length penalty. Perhaps the length penalty simply does not use the right coefficient—in fact, it seems that it should always be possible to remove the length correlation measured in Table 3 by using a linear length penalty with the right coefficient.

 * Finally, the authors do not consider the extent to which the correlation between length and reward may not be entirely spurious—there may actually be some amount of weight that the reward models *should* place on length. 

Small issues:
 * "LLMs alignment" -> "LLMs' alignment"
 * Line 470 "board" -> "broad"

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.	Bias and Spurious Correlation: How does the characteristic bias $c$ lead to spurious correlations? Examples would facilitate a better understanding for the reader.


2.	Hyperparameter Selection: What guidelines are there for selecting hyperparameters in different scenarios, such as $\beta_0$ and $\beta_1$?

3.	Other Types of Bias: Are there experiments demonstrating bias beyond length and style?

4.	Fig. 5 x-axis: What does the x-axis represent in Figure 5? Providing clarity would help readers interpret the results more effectively.

5.	There are several methods that can mitigate length bias, particularly in DPO. Have you considered simple baselines such as simPO?
soundness: 2
strengths: 1.	The authors effectively identify common limitations in reward models and propose a method to enhance their reliability.

2.	The proposed method is claimed to be computationally efficient.

3.	Experimental results confirm the method's effectiveness.
summary: This paper focuses on the reward modeling process in Reinforcement Learning from Human Feedback (RLHF). The authors highlight the issues of bias and spurious correlations in the training data and propose a method called post-hoc reward calibration to address these limitations. Specifically, they assume the reward model can be decomposed into two components and use the evaluation set to estimate the bias term through locally weighted regression. The experimental results demonstrate the method's effectiveness.
weaknesses: 1.	Assumptions: The proposed method depends heavily on several assumptions, such as (1) decomposition of the biased reward, (2) independence of the bias characteristic, (3) sufficient density, and (4) Lipschitz continuity. These assumptions may not hold in all scenarios.

2.	Difficulty in Determining $d$: In some scenarios, determining dd is difficult. While LWR can estimate Equation 5, linear approximation may often be inaccurate, necessitating further analysis.

3.	Simple Baselines: The baselines used are relatively simple. Comparisons with more complex baselines, such as reward model engineering in reference, are needed.

4.	Demonstrations Needed: Additional examples should be provided to showcase the method's effectiveness in various situations.

6.	Bias Exploration: It would be beneficial to explore other biases beyond 'length' and 'style', given that significant improvements are mostly seen in 'length bias'.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Typo between lines 224-225 “Based one LWR…”
soundness: 4
strengths: - The proposed method, RC-LWR, is post-hoc and so does not require intervention during the training of the RM. It is also fast to compute. This means you can make many post-hoc corrections and sweep over hyper parameters all without needing to retrain. This is a huge advantage over adding a length penalty or other penalty term during RM training. This means you can try to correct for as many different biases as you want to do and try out combinations of them all post-hoc.

- RC-LWR is also robust to the curvature of the bias term due to using locally weighted regression, and only relies on the bias being locally close to linear.

- The paper is very clear and easy to follow.
summary: This paper proposes a calibration method for correcting bias (in particular length bias) in already trained LLM reward models (RM) on datasets. Assuming that the reward can be decomposed into an aligned reward and a bias reward term, the proposed method estimates the bias reward term using locally weighted regression (LWR) or a more robust variant (LOWESS). Then the correct reward is obtained by simply subtracting the estimated bias term. Experiments in RMBench, RM as Judge, and RM alignment show that the proposed method (RC-LWR) is able to effectively and robustly remove length bias compared with the common approach of adding a length penalty during RM training. Adding a length penalty requires a task-dependent penalty weight, whereas RC-LWR robustly adapts to different tasks and datasets.
weaknesses: - A slight weakness is the majority focus on length bias. The experiments pretty much all focus on length bias, with one generalization task to markdown features. 

- Another slight weakness is that only the combination of length penalty + RC-LWR was tested. A more interesting direction would be to test a combination of two biases, and whether you can apply the method multiple times to remove multiple biases.

Meta-review: This paper studies the length bias in reward modeling and RLHF, which is a well-known issue in learned reward models and language model preferences. They provide a method of "Post-hoc Reward Calibration" to address it and obtained promising empirical results.

Strengths:  
The studied problem is well-motivated. The proposed approach is simple and needs no extra training. Experiments also support the proposed argument.

Weaknesses:  
The proposed approach assumes independence between the true reward and text length, which appears too strong. The compared baselines are also not sufficiently challenging.

Most of the weaknesses have been addressed during the rebuttal period. All reviewers suggest acceptance for this paper, and I agree with their assessment.

Decision: Accept (Poster)