Post-hoc Reward Calibration: A Case Study on Length Bias
Authors: Zeyu Huang
Zihan Qiu
Zili Wang
Edoardo Ponti
Ivan Titov
Venue: ICLR 2025 Poster (2025)
Keywords: RLHF; Reward Model;  Length Bias
Primary area: foundation or frontier models, including LLMs
TL;DR: A simple yet effective reward calibration method to mitigate the bias in reward models without extra training and data.

Abstract: Reinforcement Learning from Human Feedback aligns the outputs of Large Language Models with human values and preferences. Central to this process is the reward model (RM), which translates human feedback into training signals for optimising LLM behaviour. However, RMs can develop biases by exploiting spurious correlations in their training data, such as favouring outputs based on length or
style rather than true quality. These biases can lead to incorrect output rankings, sub-optimal model evaluations, and the amplification of undesirable behaviours in LLMs alignment. This paper addresses the challenge of correcting such biases without additional data and training, introducing the concept of Post-hoc Reward Calibration. We first propose to use local average reward to estimate the bias term
and, thus, remove it to approximate the underlying true reward. We then extend the approach to a more general and robust form with the Locally Weighted Regression. Focusing on the prevalent length bias, we validate our proposed approaches across three experimental settings, demonstrating consistent improvements: (1) a 3.11 average performance gain across 33 reward models on the RewardBench
dataset; (2) improved agreement of RM produced rankings with GPT-4 evaluations and human preferences based on the AlpacaEval benchmark; and (3) improved Length-Controlled win rate (Dubois et al., 2024) of the RLHF process in multiple LLM–RM combinations. According to our experiments, our method is computationally efficient and generalisable to other types of bias and RMs, offering a scalable and robust solution for mitigating biases in LLM alignment and evaluation.

Published as a conference paper at ICLR 2025
POST-HOC REWARD CALIBRATION: A CASE STUDY ON LENGTH BIAS
'Zeyu Huang "Zihan Qiu Ziii Wang
Edoordo M. Ponti
Ivan Titov
University of Edinburgh AAlibaba Group zeyu, zeyu, huangeed.ddddddco aat.ledddc.... ac.uk
INF Technology
*University of Amsterdam
ABSTRACT
Reinforcement Learning from Human Feedback aligns the outputs of Large Lan- guage Models with human values and preferences. Central to this process is the reward model (RM), which translates human feedback into training signals for optimising LLM behaviour. However, RMs can develop biases by exploiting spurious correlations in their training data, such favouring outputs based on length or style rather than true quality. These biases can lead to incorrect output rankings, sub-optimal model evaluations, and the amplification of undesirable behaviours in LLMs' alignment. This paper addresses the challenge of correct- ing such biases without additional data and training, introducing the concept of Post-hoc Reward Calibration. We first propose to use the local average reward estimate the bias term and, thus, remove to approximate the underlying true reward. We then extend the approach to more general and robust form with the Locally Weighted Regression. Focusing on the prevalent length bias, we validate our proposed approaches across three experimental settings, demonstrating consis- tent improvements: (1) a 3.1 average performance gain across 33 reward models on the RewardBench dataset; (2) improved agreement of RM produced rankings with GPT-4 evaluations and human preferences based on the AlpacaEval bench- mark; and (3) improved Length-Controller win rate (Dubois et al., 2024) of the RLHF process in multiple LLM-RM combinations. According to our experiments, our method is computationally efficient and generalisable other types of bias and RMs, offering a scalable and robust solution for mitigating biases in LLM aligmment and evaluation.
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022; Kaufmann al., 2023) is driving the success of Large Language Models (LLMs) (OpenAI, 2023; Anil et al., 2023; Dubey et al., 2024). By integrating human preferences into the training loop. RLHF aligns models with desired behaviours and human values (Chakraborty et al., 2024), enabling LLMs to generate content that is conceptually appropriate (Sun et al., 2023), ethically sound (Hendrycks et al., 2021a; Dai et al., 2024), and reflecting nuanced human judgements (Wu et al., 2023). A key component in this process is the reward model (RM), which translates qualitative human feedback into a signal for optimising LLMs. The primary purpose of RM training is to approximate human preferences, allowing evaluate the content generated by LLMs by assigning scores. These scores (rewards) are then used to optimize the LLM, shaping its behaviour to better meet human expectations.
However, there a risk that an RM may exploit spurious correlations in the training data that do not genuinely reflect the output quality (Stiennon et al., 2020: Gao et al., 2023; Singhal et al., 2023; Pang et al., 2023; Lambert & Calandra, 2023). For instance, an RM might favour outputs based on length (Park et al., 2024a) or style (Lambert et al., 2024; Dubois et al., 2023a), leading to biased assessments that do not accurately represent human preferences. The involvement of a biased RM in developing LLMs can have significant drawbacks. Firstly, it may lead to incorrect rankings of outputs, favouring content that aligns with learned biases over genuinely high-quality outputs. This issue is particularly problematic when RMs used rank different LLMs during evaluations, biased RM could produce misleading results, endorsing suboptimal models and impeding progress in model development. Moreover, when biased RMs employed in RLHE, these biases can be further

---

Published as a conference paper at ICLR 2025
amplified (Zheng al., 2023b), with the LLM learning to exploit these biases rather than improving genuine task performance a phenomenon known as reward hacking.
Recent advancements introduce various strategies to mitigate these biases and alleviate reward hacking during different stages of RLHF, from data handling (Park et al., 2024a) and reward model engineering (Coste et al., 2024) to the final reinforcement learning phase (Chen et al., 2024b). Most of these approaches, however, either require additional data collection, re-engineering of the RM, or modifications reinforcement learning algorithms, which raises an intriguing question: is it possible to correct or mitigate biases in reward signals in a training-free manner? Specifically, given a set of scored outputs from the RM and specific bias that the RM may exhibit, how can we calibrate these reward signals to reflect the true quality of the outputs more accurately?
This paper makes an initial attempt answering this question by framing as Post-hoc Reward Calibration. Specifically, for a particular characteristic of the output (e.g., the length), we assume the biased reward from the RM can be decomposed into the sum of (1) the underlying calibrated reward and (2) bias term that solely depends that characteristic. Under certain assumptions, we propose to use the local average reward to provide an estimation for the bias term, which can then be removed, thereby approximating the true reward. We then extend it to be more general and robust using Locally Weighted Regression for bias estimation. Among different biases (Zhang et al., 2024b), a particularly prevalent one is the sequence length. Prior works showed that LLMs generate significantly longer responses after RLHF without improving the genuine quality of the content (Kabir et al., 2023; Wu et al., 2023; 2024b). Singhal al. (2023) point out that even purely length-based reward can reproduce most downstream RLHF improvements of existing RMs, highlighting that current RMs fail convincingly outperform simple heuristics and may be strongly biased by length. The issue is also evident in powerful proprietary models like GPT-4, which is increasingly used as a judge, resulting in LLM evaluations that favour more verbose responses. Focusing on length bias, we validate our method in three settings and observe consistent improvements: (1) Benchmark performance: a 3.11 performance gain averaged over 33 RMs on the RewardBench; (2) RM as L.M evaluators: based on AlpacaEval, we utilise 8 open-source RMs to rank 184 L.Mss. After applying our calibration, the rankings correlate better with GPT-4 evaluations and human preferences. (3) RM for LLMs' alignment: we assess calibrated rewards for RLHF process. Testing with four LLM-RM configurations, we observe consistent improvements in AlpacaEval2 Length-controln win rates.
In addition to being computationally efficient (e.g., calibrating over 300k samples takes only 30 seconds with a single CPU), we empirically reveal that our method can generalise to other quantifiable biases, such the presence of markdown-style features in the outputs, and pairwise GPT4-as- -uduu models. Importantly, the calibration effect is strongly correlated with the RM's preference for specific properties: for strongly biased RMs, the calibration yields greater improvements, while for weakly biased RMs, it only slightly alters its rewarding behaviour. These findings suggest that our proposed Post-hoc Reward Calibration method enhances the reliability of RMs and offers a practical, scalable solution for mitigating biases in RLHF. This approach could be instrumental in advancing RLHF methodologies and developing more robust and aligned LLMs.
2 RELATED WORKS
Reward Models Given a prompt q and two corresponding responses y1 and y/, training an RM usually involves training classifier to predict the human preference probability (v1 > 32l4) of two responses y/1 and y/2 with the Bradley- Terry (BT) model (Bradley & Terry, 1952):
p" (y) > y2/4) = or (a1) - r (x22)
(1)
where x; = (g. 3s) represents the prompt-response pair, r" the latent true reward we cannot access, and a is the logistic function. Then, given a reward modelling dataset DD {(g vi, bi) = {( J where vi is annotated as a better response to prompt compared to 1/z training a RM ra is to minimise the negative log- likelihood for a binary classification problem: problemmm
(()) -
(log (ra(#)) re(2))))
(2)
In practice, RMs are often implemented by adding an extra linear layer to LLMs, which projects the representations of the last layers of LLMs to a scalar as the reward. There are also two other types of RMs: (1) DPO-based: L.MMs optimised with Direct Preference Optimisation (DPO) (Rafailov
2

---

Published as a conference paper at ICLR 2025
et al., 2023) could function an implicit RM. (2) LLM-as-Judger A generative LLM can also be prompted to function as an RM to generate feedback, usually consisting of a piece of chain-of-thougee reasoning and a final score representing the overall quality (Zheng al., 2023a; Kwon et al., 2023; Kim et al., 2024). In this paper, we validate the effectiveness of our methods across all three types of judge models. Specifically, we employ our method calibrate BT-based and DPO-based RMs on the RewardBench dataset, and to calibrate the pairwise judge of GPT-4 with the AlpacaEval benchmark.
Bias Mitigation for RLHF Recent works explore the bias mitigation, also termed as mitigate reward over-optiimisatinn for the RLHF process mainly from the perspective of (1) Data handling: Park al. (2024a) identifies six types of inherent bias in various judge models and employ L.LMs to generate the de-biasing dataset for further fine-tuning. Zhu et al. (2024) propose to iteratively replace hard labels in data with soft labels during RM training. while Rita et al. (2024) rely on human expert demonstration recalibrate the reward objective optimised with Reinforcement Learning. (2) Reward Model Engineering: Coste et al. (2024); Eisenstein et al. (2023); Sun et al. (2024); Zhang et al. (2024a) implement the reward ensemble; Ramé et al. (2024) first fine-tune multiple RMs and then average them in the weight space; Dai et al. (2024) decouple the RM training for helpfulness and harmlessness and Chen al. (2024b) disentangle the length signal and the quality signal into two different reward heads. Shen et al. (2023); Wang et al. (2024); Quan (2024) exploit the idea of Mixture-of -ffxxere-f-- (Qiu et al., 2024a:b) in RM training to capture different facets of human preference to avoid the potential conflict between them; (3) Reinforcement Learning methods: Another research line (Moskovitz et al., 2024; Sun et al., 2024; Zhou al., 2024; Zhai al., 2024) modifies the algorithm or optimisation objective during reinforcement learning with Park et al. (2024b); Zhu et al. (2024); Lu et al. (2024) specifically focusing on the length, one of the most consequential sources of bias (More discussions on length bias are in Appendix. A). Our work takes a different approach from previous methods by employing post-hoe reward calibration, iee, using only batch of scored prompt-response examples to mitigate the RM's bias, without intervening the preference data collection, RM training, and the RLHF phase.
3 METHOD
3.1 PROBLEM STATEMENT: REWARD CALIBRATION
The reward calibration problem could be formalised follows: given onput- pair to be evaluated, the reward model ry assigns a scalar rx(r) to represent its quality. However, having been trained on the human preference dataset, the reward model re may learn shorteuts to approximate the human preference and thus have biases regarding some characteristics of the input-output pair æ. For example, tends to assign higher rewards to outputs with longer length or containing knowledge commonly encountered in the real-world data (Park et al., 2024a). Supposing a measurable characteristic of interest e a function that maps the input input-output pair æ to a scalar c(x):
C:
X
IR
c(z)
We assume that the biased reward from the reward model ra(x)) could be decomposed into two terms, an underlying calibrated reward (ar) and a bias (c(r)) with regards characteristic c:
ro(x) = (z()) + bg (c(x))
The RM is usually used calculate the margin between outputs (I1, to predict the human preference:
Arp (z1, = (2 - r5 - 1 (x2) + b% (c(x1)) - (c(x2))
(4)
Thus, the reward calibration problem is to estimate the reward difference owing the bias term and subtract it to recover the underlying true reward margin A; (z1, = r (x r (x2).
Post-hoc Reward Calibration: The proposed reward calibration problem is applicable to various stages of RLHF Depending on the focus, can be tailored either toward collecting debiased pref- erence data, refining the training of reward models,
enhancing the alignment procedure by adjusting the alignment objectives. In contrast, this paper focuses on the post-hoc setting, performing reward calibration after the reward assignment without interfering with any data collection or RM
3

---

Published as a conference paper at ICLR 2025
training stage. Formally, given N already- rewarded data points 12i. Aro (2), we aim to find the calibrated reward margin Ai, (4) that closer to the underlying oracle margin Ar, (ri,a;) for every pair x2).
Assumptions We make the following hypothesis about the decomposition in Eq. 3. (1) Inde- pendence of the biased characteristic: consider a general reward modelling dataset composed of prompts and corresponding responses pairs D {(q', yi, } {(xf 225 ( the expectation of the underlying gold reward margin should be and is independent of the biased characteristic c,
I (0r A (21, 72) c(x1) = eı, c(22) = 02] = 0
(2) Sufficient Density: the function values of the characteristic function e is assumed to be sufficiently dense in its range c(X). (3) Lipschitz Continuity: the bias term 6% is a slow-varying function of characteristic c, which means if pairs 21 and 22 are close each other regarding the characteristic c, their corresponding bias term in Eq. 3 should be close as well. Mathematically, given c(zo) and a neighbourhood ((zo) {((x) |e(x) c(zo)] <) of zg. for all c(x1), c(x2) V, there exists a constant K such that (c(x))) ig (c(x))) < K|e(x) c(xo))-
3.2 BIAS ESTIMATION
Uniform Averaging Approach According to Eq. 4, the reward margin between ø1 and ze consists of (1) the underlying calibrated margin A7, (22) and (2) the biased "bonus reward" that solely depend on the characteristic c(x1) and c(x2). Intuitively, to estimate (c(x1)) by. (c((22)), we ask:
Given the characteristic measures c(21) and c(222) for an arbitrary pair (æ1, 22), what would
be their reward margin?
straightforward estimation for this question could be
E(ro(x) c(x) c(x1)] )] E(ro(x) c(x) c(x2)).
Under the assumption of Lipschitz continuity and sufficient density, we can obtain simple calibrated reward margin described in Eq. 6:
Å (x1, x2) = Aro (2), - (B(re(z) [c(x) c(<1)) < d] E[ra(x) [e(x) - c(x2) < d))
(6)
where d is a threshold distance that governs the neighbourhood size around and æz.
Method How do we choose the threshold distance in Eq. 6? (I) It must be sufficiently small to ensure that the bias term can be approximated function within the local neighbourhood, enabling the application of Eq.6; and (2) it must also be large enough meet the sufficient density assumption, ensuring sufficient data points used for estimation. Satisfying these conflicting requirements may necessitate thorough hyper-parameter tuning of d employ the intuitive approach in practice. Furthermore, the density may not be uniform in practice, so that the optimal threshold distance d may vary for different data points c(x). Therefore, desirable approach should be able to estimate the expectations in Eq. 5 in a more general, robust, and assumption and (2) mitigating the sensitivity to
constant
practical way by (I) relaxing the the choice of neighbourhood size, possibly by incorporating certain adaptive mechanisms optimise the neighbourhood size or the contribution for each datapoint dynamically. Both desiderata motivate us to employ Locally Weighted Regression (LWR) towards a more robust estimation.
Specifically, given a point of interest (e.g., characteristic value c(x) in our case), the Locally Weighted Regression (LWR) method first uses a bandwidth parameter f to define the fraction of the dataset utilised for regression. Then the method assigns weights to neighbouring data points, giving higher weights to points closer to the target and lower weights to those farther away. This approach ensures that nearby points have greater influence the regression. LWR then fits weighted linear regression to model the local behaviour of the target function, effeetively approximating the weighted average within the specified local context.
LWR addresses the desiderata by offering a more adaptable approach to estimate the Eq. 5. relaxes the constant-function assumption by introducing the linear approximation capture the
4

---

Published as a conference paper at ICLR 2025
subtle local variations in the data. This is reasonable because the bias terms are slow-varying functions and can be linearly approximated using Taylor's expansion. Furthermore, LWR mitigates sensitivity to neighbourhood size by using distance-based weighting mechanism, which assigns greater importance to closer data points. This adaptive weighting scheme dynamically adjusts the influence of each data point, reducing reliance a fixed neighbourhood size and enhancing the robustness and flexibility of the estimation process. Formally, given a dataset consisting of the characteristic in outputs and corresponding rewards D. ro and a bandwidth f. our general method could be formulated as:
(11, x2) = Aro (21, (Fo(c(21)) Fo(((22)))
where 1)((((1)) and éa ()())) are the LWR prediction for the characteristic c(x1) and c(r2), respec- tively, given the dataset De. To determine the preference, we use the sign of reward margin between Ti and i2: sgn( (Ar 222). In practice, we employ the Robust Locally Estimated Scatterplot Smoothing (LOWESSS), robust version of LWR. Based on LWR, it defines different weights 6j for the point (c(æ,), rr((), according to the size of residual rr()))) - igg fo(((r)))) It assigns smaller weights to data points with large residuals and large weights to those with small residuals. New fitted values are computed using LWR again but with W, replaced by 6, k,y. The computation of new weights and fitted values is repeated several times converge. The full algorithm of using LOWESS for reward calibration is described with Appendix B Algorithm 1. We directly employ off-the-shelf library stat smodels s. api and pass the dataset D, to it for our implementation.
Length Bias Estimation This paper prioritises the prevalent length bias presented in the reward model. Thus, the characteristic of interest here is the character length of the output, i.e., c(x) = |æ]] The dataset used for regression consists of the length of the output and its corresponding rewards.
4 EXPERIMENTS
One key advantage of our proposed Post-hoc reward calibration is that does not need any extra data or training and thus be seamlessly applied various scenarios. So, test the proposed method in three settings: (1) RM Benchmark performance: we test the calibrated RM's performance on the RewardBench dataset (Lambert et al., 2024). (2) RM As LLMs Evaluator: Another critical application for RMM is to serve as judge to replace human evaluation when evaluating LLMs. We thus also explore whether the calibrated RM could provide better evaluation. Based on the AlpacaEval benchmark (Dubois et al., 2023a), we employ RMs to label the output of different LLMs from AlpacaEEvll leaderboard and produce a ranking LLMs.. Then, we measure the correlation between the RM's 's ranking and GPT-4 ranking or human preference ranking. (3) RM for LLMs' alignment: Finally, as discussed in our introduction, employing a biased RM risks amplifying the potential bias and thus leads to unexpected behaviours. So, we explore how proposed calibration method affects the LLM alignment results. Specifically, we use the calibrated rewards to label the preference data, and then conduct Direct Preference Optimization (Rafailov et al., 2023) with the labelled preference pair. We evaluate the aligned LLM's performance using AlpacaEval2 and eight popular benchmarks.
In all three settings, we are given group of data points D|= (2., rx.)))) } for calibration. Because LOWESS requires calculating the residual for every fitted data point, we employ the entire Di=| for regression. Therefore, for RewardBench, there are 2.985 test data points, so we employ all 5.970 data points to perform the regression (every test data is pair of responses for one prompt). Regarding AlpacaEval, there are 184 LLMS and 805 instructions for each LLM,, resulting in 151k samples used for calibration. For LLMs' alignment, five responses are generated from the LLM for about 60k instructions, leading to about 300k samples used for bias estimation. The proposed method is computation-efficien and calibrating 300k samples only takes 30 seconds using single CPU.
Compared Algorithms For all three experimental settings, we try to compare the following algorithms: (1) Original reward; (2) Length penalty: a broadly adopted approach to
mitigate the length bias in reward models. It adds penalty term to the original reward (Singhal et al.,
2023; Park et al., 2024b; Liu et al., 2024a; Dong et al., 2024), formalised as r*() = re(z) - a x |x]. Following Dong et al. (2024), we use the character length of each output and set
0.001. (3)
RC-Mean: our proposed uniform averaging calibration method described in Eq. 6. (4) RC-LWB our proposed Locally Weighted Regression calibration method shown in Eq. 7. (5) RC--IMRR-enaatte
5

---

Published as a conference paper at ICLR 2025
9.22
10 .iePtt-Tuutss 451
Lagg
Moo
100
RC-WW
-77
Mathoda
Figure I: Results of different calibration algorithms on the RewardBench benchmark are shown two charts.
Left: the line chart demonstrates the RewardBench score before calibration (x (x-axis) and after calibration (y-axis) of different algorithums for different models. Right: the scatter plot highlights the performance gains achieved by different calibration methods, annotated with the maximum, average, and minimum values.
to see whether our method could be applied top of the penalty- -basd methods, we first apply the length penalty to the original reward and then use the RC-LWR. All hyper- papee-parameterss related to RC- RC-Mean, RC-LWR and RC-LRR-PenaLE are presented in Appendix C.
4.1 LENGTH CALIBRATED REWARDS ON REWARDBENCH
Setting and Metrics Each data point in RewardBench comprises one prompt and two LLMs- generated responses Yuu and yt. The final score is calculated as the averaged accuracy of reward models in identifying Yeu over four subsets: Chat, Chat Hard, Safety, and Reasoning. We mainly investigate 33 BT-based RMs whose rewarding scores are officially available! and whose accuracy above 50% (random guess) from the RewardBench Leaderboard as of Aug. 2024.
Results The overall results for different calibration methods are illustrated Fig. The line chart the left showcases the calibrated performance (y-axis) w.r.t original performance (x-axis) of different calibration algorithms for different R.M.. The scatter plot on the right highlights the performance gains for different calibration methods. Our empirical findings include: (1) The simple coarse-grained Length Penalty method already leads to a notable improvement (orange line blue line), some extent indicating the prevalence and significance of the RM's length bias. (2) The reward calibration techniques we introduce yield improvements for nearly all RMs. Among these techniques, the RC-LWR-Penn Penatty achieves 3.11 performance gain averaged over all 33 models. Though it could lead to performance degradation (5 models for RC- and 4 models for RC-IWR-Pena 1yy)) ty), the degradation is very slight and the worst case only about point for RC-LWR and RC-L-RR-Pena 1 ty. (3) More importantly, besides improving reward models with moderate performance, the proposed calibration method could also boost the performance of strong reward models. For example, after RC- -WR PeRR-Penalts calibration, nine reward models surpass the accuracy of GFT4-Turbo, and the best calibrated model even approaches the latest state-of-the-ar (SOTA) LLM-as- Judge models. We also calibrate DPO-based reward models with proposed method and report results in the Appendix D.
4.2 LENGTII CALIBRATED REWARD AS LLMS EVALUATORS
Settings The typical LLM evaluation pipeline first constructs a prompt dataset, samples responses from different LLMs, and then calculates the win rate between different models. Based on this pipeline, AlaacaEEval (AEI) (Dubois et al., 2023b), with GPT4-11 111066rreve ew as the evaluator, fixes a baseline LLM and uses the win rates against the same baseline LLM to produce the ranking. To calibrate the length bias of GPT4 evaluation, AlpacaEval2 (AE2) (Dubois et al., 2024) introduced conttolonttllled regression to calibrate the win rate of AE1 and became popular generative LLM leaderboard. Based on the AEI prompt dataset, this section tests whether the open-source BT-baase RMs could provide more reliable evaluation for LLMS after calibration. Because the length- controlled technique proposed in AE2 can apply to BT-based RMs, we include it in our baselines, denoted LC. We also apply our method calibrate AEI to further compare with LC, More details about applying RM for AE and calibrating the produced ranking are in Appendix C.
Metrics Following Dubois et (2024), we look the following metrics to see whether the calibrated rewards could provide more reliable evaluation: (1) Gameability; this measures whether
httpps: hhttpsss/huggginggfae.o.///
Tenai rewa www.heenchheeeeeeeee

---

Published as a conference paper at ICLR 2025
82.4
Caneebilly
LonghhBiadtoonh
37.80%
58.8
10.0%
31.7% 58.5
32.5%
AB1-STO%%
21 1.22
16.0%
AKIHTUIN
Lare www.9-00eeSB0e00000 Wrayistb/00ueeee
SM
01S.-Tenai Ro-laid
RTBML Qumml, Th
Matui Matts-MM- MAFE
Figure 2: The Cameability + and Bencth accuracy t for selected Ba-b reward models. The lower the Gamebility, the less sensible the win rate is to different prompt strategies. The higher the accuracy on the Benneeh the less preference the model has for output length. The Gameability of AlpacaEvall
(AF) and AlpaceEval2 (AE2) win rates are 11.05% and 27. 27,9% respectivcly, presented with the dashed line.
intermin2-2. The
CRM- MRMM-llama--ao Badaiill
FaGai:--L Leaix---LLMM... LLMAA--M-O.
BIRM Qwecs 75 0613
AZZ 8
8.708
8.15--0 18.62
2.515
2.355 01969 luibe k052 0.53
2,1122
0.098
0,77 KOPT
1.500
E.00is 8555 8.50 ausls 1,k0 20133
GAres
0.819
1300 1.914 lone Konr 0.519
0.505
8.000
8.30 ausas Ruin 2044
0.896 B.00 1.00
18.86
8.911
9.942
0,50M
8.000
0.0 9,413
GPT4 105
(L837
1.947
8.677
Mistral- RMM So-------- L--GSSF---
RM-Mattul--
Eunu-RM-76
A12 6.977
8.9% EU07 BUAIT wi 8463
B368 8.60 jude aos 2012 1012
8368
0.97778777 8.71 3453
8.995
0.655
0.000 03571 ROM
8.766
0.766 0.000 our 2o1 2011
8557 8.59
0.51
8555
1.00 as52
8.74
0.855
(191) 1OR sans
0.311
0.573 0377 0,99 1591
8339 8.39 2.54 aum 1u54
jau54
8374
8.54
1363 a33s
8851
8.59
8,05 a
1373
8:59
8.79
2.307
0.e 1,80 ao00
Ameah Foads
Leppl Peady
Oagend Armsel
produced rankings with the AlpacaEval2 (AE2) and ChatbotArena (CA). We also report the calibration results for 1PTT4-100 66 on the right, with the Original Reward indicating AlpacaEvall, LC representing AE2, and RC-LMR referring rankings calibrated with our proposed RC-LWK method. The x-axis represents different calibration methods. The two highest correlations in each row are marked with (dashed reciangle We select three versions of CA: (i) CA Feb., from the AB2 paper, to validate our reproduction of the original results; (ii) CA Aug., the latest ranking which contains more LLMs; and (iii) CA Length, which is CA Aug. disentangled with respect to the response length.
Figure 3: The heatmap demonstrates the Spearman correlation (1) between reward-models models-produced
the win rate given by the evaluator is significantly affected by simply promoting the same model to be more concise or verbose. Like Dubois et al. (2024), five LLMs are selected to be prompted to generate the response normally, verbosely, and concisely. Then, the gameability is defined as the normalised variance between those three win rates and averaged across these five models. Higher gameability indicates that the evaluator is more sensitive to the output's 's length and could help us identify evaluators with length bias. (2) Spearman correlation with existing popular LLM leaderboard, including AE2 and ChatbotArena (CA). We employ three versions of (CA): (i) CA Feb., used by Dubois et al. (2024), to reproduce their results; (ii) CA Aug., the latest version that includes more LLMs; and (iii) CA. Length, namely CA Aug, calibrated with respect to length as proposed by Chiang al. (2024).
Identifying RMs with strong length bias Before calibration, we first evaluate the top-10 BT-based RMs from RewardBench leaderboard as of Aug. 2024 using the Gameability and the length bias subset of the EvalBiasBench dataset (Park et al., 2024a). The latter contains 17 test examples and is denoted as LengthBiusBench. We aim to identify RMs demonstrating severe length bias, f.e., high Gameability and low accuracy on Benchh. Their results are shown in Fig. 2. We observe the prevalence of length bias for BT-based RMs. 8 of 10 models exhibit high Gameability and poor performance Bengthh.. The two metrics demonstrate consistency, i.., the model with high Gameability usually achieves low accuracy on LengthBias Spearman correlation score. As Llama 3-Offset Bias RM-RM and internlnd 10.1mm2-002 do not demonstrate severe length bias, we primarily report results for other 8 models.
achieving -0 879
The Spearman correlations of RM-produced rankings with the AE2 and the ChathotArenas are presented in the Fig. 3. Our findings include: (1) Overall, RC-Mean and RC-LWR provide effective calibrations, leading to a higher correlation with AE2 and ChatbotArena, achieving compa- rable calibration effects with LC on all eight RMs. Note that LC is specifically designed for the AE leaderboard and can not be straightforwardly employed for reward calibration because it requires the LLM ID and instruction ID for regression, which not available for general reward calibration. (2) Compared with AE2, RC-LWR calibrated AE achieves comparable Spearman correlations with Chatbot Arena, suggesting that method can generalise to -.MM-as-Judge results. (3) The cali-
Results
7

---

related to Published as a conference paper at ICLR 2025
Table I: The Gameability J of different calibrations methods across various reward models on AlpacaEval
Model
Length Penalty
LC
RC Mean
RC LNR
RC LNR Penalty
13.6%
10.6%
9.4%
9.0%
25.6%
12.1%
9.5%
8.8 %
9.0%
16.7%
Tsfairx-Dlama--RM--
20.2%
12.7%
10.8%
10.1%
19.0%
STRM-QWen2_7b.
18.5%
13.8%
10.5%
10.6%
17.5%
GRV-LLana3--8B-aftrre
23.4%
13.7%
9.7%
10.0%
14.8%
RM-Mistral-78
20.5%
14.2%
10.8%
10.5%
19.8%
Mistral-RM-for-RAB
22.0%
13.5%
9,9%
9.6%
19.5%
Eurus RM-76
37.8%
18.3%
12.2%
11.8%
11.8%
achieves 0. 9.75 with ChatbotArena Feb., and the calibrated achieves 0 975 with AE2 and 0. 951 with ChatbotArena Aug., demonstrating the potential of calibrated open-source RM provide reliable LLM evaluations. (4) The Length Penalty brings improvements in most cases. But it can also lead to degeneration or can be not effective at all (Burus-RM-77) ent RMs have different reward scales, making it harder to employ the same penalty weight for different RMs. On the contrary. our proposed RC-LWR is more robust to different hyper-parameter choices, rendering it easier to use in practice. (5) Unlike the results on RewardBench, the Length Pena 1 ty and S0S-LMR-Pen-e 1 y are less effective in this setting, likely due to differences in length margin ( denoted as ||ا|| |22]] \) distributions of two dataset. For instance, in the RewardBench, 60% of data points have a length margin below 600, while in the AlpacaEval setting, this figure is less than 40%. Consequently, the penalty term has more significant impact the AlpacaEval calibration. More relevant analyses are shown in the Appendix D. This underscores the practicality of our method over the cenalty-bascll approach, the latter may require extensive hyperparameter tuming for penalty weight. Such tuning is influenced not only by the RM's reward scale but also by the output's 's length distribution.
brated
Aefairx-LLlMAA-RRII
Luntern_m22-7--ewaare
This is because differ-
(6) The rankings calibrated by LC, RC-Mean, RC-LWR are usually more aligned with CA Length compared with CA Aug., further validating the effectiveness of our proposed methods. (7) As shown in Tab. I, MC-Mean and LMR effectively reduce the Gameability for various reward RMs, demonstrating the calibrated rewards are less sensitive to different prompting techniques and thus more reflective of the response quality.
4.3 LENGTH CALIBRATED REWARDS FOR LLMS' ALIGNMENT
Setting In this setting, investigate whether the calibrated rewards could serve a better training signal for LLMs' alignment. Because the post-hoc reward calibration assumes a scored dataset used for calibration, which could be seamlessly integrated with the offline RL, choose to first employ the RMs to score the on policy dataset and then perform DPO. Our experiment pipeline could be described as follows: for each prompt q in prompt dataset, multiple re- sponses yi are sampled from an LLM, then scored as by an RM. Among these responses, the response with the highest or lowest score is selected create a preference pair (p. Nw y1) for Direct Preference Optimisation (DPO) tuning. Specifically, to validate the effectiveness of our method, we choose two instruct LLMs: yemma2-9b-iig it (Team, 2024) and two reward models:
(AI@ Meta, 2024) and 1 (Dong (Yang et al., 2024), resulting in 4 LLM-RM configu- rations: (1) Llama - 3 11--8B-FFF Fa rx; (2) --llamm---BB--a
et al., 2023) and 10w--11ama--BB----
(3) gemma 1at----F-aaaa Fos--Faaiix and (4) gemma - For each LLM-RM combination, we compare the Original Reward, Length Penalty. and RW-LWR We directly utilise the on-policy generations" of two selected LLMs generated by Meng et al. (2024). They employ the 60k instructions from the UltraFeed- back (Cui et al.. 2024) dataset and sample 5 responses for each instruction from the LLM. Then, use those two RMs to score the dataset and to conduct DPO tuning. All DPO tuning are set following Meng al. (2024).
Metrics We evaluate the LLM with the Length- win rate from the AE2 for open-ended generation. We also test the LLM on eight popular benchmarks: MMLU (Hendrycks et al., 2021b), GSM8K (Cobbe et al., 2021). ARC-challenge (Clark et al., 2018), HellaSwag (Zellers et al., 2019), Winogrande (Sakaguchi et al., 2021), IFEval (Zhou et al., 2023). GPQA (Rein et al., 2023), and BBH (Suzgun et al., 2023) to see how the proposed calibration affects the benchmark performance. We employ the OpenCompass (Contributors, 2023) for benchmark evaluation and report the average score of these benchmarks. All evaluation configurations are set as default in OpenCompass.
1p/ 1 a-mammmalll......mmm www.arrrrttaaaebaccceoooo. rmo and huggi nggace. http://e..eeecttteeeeeeeeeeeeeee wwwpa//aa///peeeeeeeeeeaa www.aaaeeeeac--eeeeeeeee rmorm
8

---

Published as a conference paper at ICLR 2025
Table 2: Results of using length-calibratee rewards for LLMs' alignment. Calibration methods compare include Original Reward (CR), Length Penalty (LP), and RC LAR. We apply these methods win rate, the average character length
four different LLM-RM configurations. We report the on the Ava, and average performance on eight benchmarks. The Table illustrates that RC-LWR calibration achieves up to 10% LC win rate improvement and alleviates the performance drop on benchmarks.
Algorithm
no
OR
LP
RC
no
OR
LP
RC
no
OR
LP
RC
DPO
LNR
DPO
LWR
DPO
LNR
Length- Controlled Win Rate
Avg Character
Avg Performance on 8 Benchmarks
Length in Alpaca
Llama-3-Ba 3-88---fai---
23.91 41.82 49.98
51.37
1967
2404
1989
1867 62.69 60.45 61.75 61.94
-lama-3-8B--n
23,91 41.00 48.41
50.49
1967
2438
1996
1858
62.69 60,94 62.11
61.83
jemma2-9b-Fsfse -b-Fsfairx
46.96 62.79
67 46
69.54
1521
2365
1491
1798
63.69
58.88
60.52
60.79
gemma2-9p- 9bmm
46.96 63.21 66.28
70.45
1521
2106
1426
1780 63.69 55.28 57.28 58.31
Results Results of employing calibrated rewards for LLMs' alignment are demonstrated in Tab. where the left and the middle focus on the LC win rate and the right on benchmark performance. We observe that: (I) Similar to the experimental results with the RewardBench, the Length Penalty serves as a simple yet effective baseline, highlighting the significant issue of length exploitation L.M alignment (Meng et al., 2024). (2) Without extra training and data anno- tation, the RC-LMR outperforms the Ori Original deta-Llama- 3- Ba-InBttttt rucntsruutttt.oo ruct based DPO and about points for gemma2-9b- t. It ef- fectively improves the output quality while preserving the length. (3) The DPO tuning leads to performance drops on our eight benchmarks. The drop is more significant for gemma2 -9b- it- based configurations (e.g., for gemma 1atmmaa-9b--fee fs fa rx, the performance drops from 63.69 to 55.28). This may be because a larger learning rate of 5e-7 is set for gemma2-bass 3e-7 for Llama3-based experiments following Meng al. (2024). However, with the same hyper- parameters, both Length Penal ty and RC-LWR methods could mitigate the performance drop, indicating that the length-regularised alignment not only controls the output length but also reserves the LLM's benchmark performance during DPO alignment.
Reward baseline on AE2 by about 9.5 for
experiments compared to
5 ANALYSIS
Original Reward
Length Penalty
RC-Kean
RC-LNR
BC-IMR-Fenalty
Iobas
0.2930:10.1836
0.2092:10.170
0.039010.0369
0.0233-10.0223 0.0229::0.0223
Table 3: The average and variance of absolute correlation score for different methods on RewardBench.
Does the calibration mitigate the length preference of RMs? One desideratum for an ideal calibration method is that the rewards should be weakly correlated with the bias characteristics of interest, thus cancelling the RM's preference for specific characteristics and preventing reward hacking in its downstream applications. Therefore, absolute Spearman correlation score over RMs for different methods on the RewardBench dataset and show them in Tab. 3. Our observation includes: (1) Length bias is significant for the broad spectrum of RMs (0. 293 for original reward). (2) The Length Penalty cannot effectively mitigate the RM's preference for length. The added penalty term renders RM to prefer shorter outputs. (3) In contrast to Length Fenalty method, our proposed methods could effectively calibrate the RM's preference on output length, i.e., the correlation score between output length and reward is approximately 0 for all RMs after calibration.
calculate the average and variance of the
Calibration behaviours on RMs with different length preferences. If an RM exhibits a weak preference for the characteristic of interest, an optimal calibration should not significantly alter its rewarding behaviour and vice versa. Rewarding accuracy alone may not effectively capture this aspect, as a calibration might result in an equal number of correct and incorrect predictions, thereby leaving the overall accuracy unchanged. We thus count how many preference pairs are reversed by the calibration on the RewardBench dataset. We report the fraction of over the entire dataset. The results are shown in the Fig. 4. Our observations are follows: (1) Length Penal ty demonstrates the moderate correlation between RM length bias and the preference overturndd and is unstable. It does not calibrate any pairs for some RMs with strong positive length preferences. For RMs with a strong negative preference for length, it is ineffective as the RM already prefers shorter outputs in general. (2) Both RC-Mean and RC-LMR methods exhibit strong correlation: the weaker the bias of the
9

---

Published as a conference paper at ICLR 2025
Speermnn comelation 0.50
Sp Spcerrn oindhiiin" 0.65
Speamon exeriation 0.54
Longh Poslty
RCA3ce
Figure 4: The number of preferences reversed by the calibration (y-axis) for RMs with different length preferences (x-axis), measured by the Spearman correlation with length. To determine the correlation between x and we further report the Spearman Correlation between them. For RC-Mean and RC-LMB, as the plot is approximately symmetrie, report the correlation between absolute values of and y.
RM, the fewer pairs reversed by the calibration method, indicating that our calibration method will not severely affect the non-biased RM. The desirable feature may enhance the practicability of the proposed method.
Generalisation to other bias types: mark- down features. To validate that our proposed method could generalise to other bias, we apply it to calibrating the bias towards the markdown listing features in the output, motivated by the empirical findings of Dubois et al. (2023a). Ob- serving that LLMs often utilise markdown style lists with bold subheadings as the output format, define the characteristic function c(x) as the number of headers, lists, and bold strings follow- ing Chiang et (2024). The calibration results for 33 BT-based RMs are shown in Fig. 5. We find that calibrating the number of lists in the output also yields improvement on the RewardBench, with 1 81 averaged performance gain for RC-Mean and 1. 8 for RC-LWR. Furthermore, similar to length bias calibration, improvements are observed across nearly all RMs, and the originally good RMs get slightly stronger after calibration. The gains are not as big the gains for calibrating length bias, which demonstrates, to some extent, that the length bias is more prevalent.
GF Getee 45
BOUKE
Figure 5: RewardBench score before (x-axis) and after (y-axis)
feature-calibration
Further Analysis In Appendix D, we show that the proposed method is robust different hyper- parameter choices of LWR (Fig. 7). For the cases when the independence assumption does not determine how much bias to
hold, we introduce a calibration constant before LWR prediction subtract, thus smoothly controlling the calibration effect (Fig. 8 and Tab. 4). We also demonstrate that RMs with more substantial bias benefit more from our approach (Fig. 10). We further showcase the potential of the proposed method to calibrate bias from multiple characteristics simultaneously (Fig. 11). Moreover, a stress test evaluating the data efficiency of the method reveals that our method is not significantly constrained by the size of the datasets used for the calibration (Fig. 13).
6 CONCLUSION
This paper introduces post-hoc reward calibration method to mitigate biases in RMs. The proposed approach effectively reduces the impact of prevalent biases, such as length bias, and has been validated across multiple experimental settings. The results consistently show substantial performance gains. Other empirical findings confirm that our method satisfies several critical desiderata for optimal calibration: (1) Our approach consistently mitigates both length and markdown biases in RMs, and can successfully calibrate LLM LLM-as-Judge rewards, underscoring its generality. (2) The method requires no additional data annotation or RM retraining and computationally efficient. (3) The calibrated rewards exhibit weak correlations with the characteristics of interest, thereby preventing reward hacking and improving the overall quality of the reward signals. (4) For non-biased RMs, our method minimally alters the reward behaviour, whereas it substantially improves RMs with stronger biases, making the calibration impact proportional to the bias level. (5) The method robust to hyperparameter choices, and calibration constant can be introduced to preserve some desirable bias towards the specific feature, enhancing its practical usability. by dataset size, even few hundred data points are sufficient to achieve effective calibration. In conclusion, our approach offers a scalable, efficient, and generalisable solution for addressing biases in RMs, ensuring better alignment between L.MMs and human preferences and LLMSs evaluation.
(6) The approach is not constrained
10

---

Published as a conference paper at ICLR 2025
REFERENCES
AI@ Meta. Llama 3 model card. 2024. URL blob/mas moob/main/MOLn www.iiim/MMLLCCO CARD mnd
com/ wttemmettaaaaaaaamaaeeaaaaaaaaee 11amaaaamaaaaa /1ama33
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillierap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras. Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312. 1805, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324- 345, 1952.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large lan- guage models with diverse human preferences. CoRR, abs/2402.08925, 2024. URL https:
//doi. 2tt.:/1...855///00
2402. 28925.
Changyu Chen, Zichen Liu, Chao Du Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham, and Min Lin. Bootstrapping language models with DPO implicit rewards. CoRR, abs/2406.09760, 2024a.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: disentangled reward mitigates hacking in RLHF. CoRR, abs/2402.07319, 2024b.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating Ilms by human preference, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/1803.00 880.05557, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110. 10.1110........ 2021.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. http://///ggihuubc.ooooooeaeeeee cww/open compass/ https//ppccccssssseeeeeee 2023.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization, In The Twelfth International Conference on Learning Representations, ICL. 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with sealed ai feedback. In Forty-first International Conference Machine Learning, 2024.
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: safe reinforcement learning from human feedback. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-1 1, 2024. OpenReview.net, 2024.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint 2rXiv:2304.06760 2023.
11

---

Published as a conference paper at ICLR 2025
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. CoRR, abs/2405.07863,, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu. Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jía, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, and Kevin Stone. The llama 3 herd of models. CoRR, abs/2407.21783. 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023b.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.044f 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, Dj Dvijotham, Adam Fisch, Katherine A. Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, and Jonathan Berant. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. CoRR, abs/2312.09244, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10835-10866. PMLR, 2023. URL https::///// pttppeeicceeegseeee mlr. wpt.sse//eeee v202/ gao2 3h. html.
Xiaobo Guo and Soroush fosoughi. Length does matter: Summary length can bias summarization metrics. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 10, 2023, pp. 15869-15879. Association for Computational Linguistics, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning Al with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021b.
12

---

Published as a conference paper at ICLR 2025
Zhenyu Hou, Yilin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng. Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, and Yuxiao Dong. Chatglm-rlhf: Practices of aligning large language models with human feedback. CoRR, abs/2404.00034, 2024.
Samia Kabir, David N. Udo-Imeh, Bonan Kou, and Tianyi Zhang. Who answers it better? an in-depth analysis of chatgpt and stack overflow answers to software engineering questions. CoRR, abs/2308.0231 2023.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. CoRR, abs/2312.14925, 2023, URL httpp:://det org/10. 18550/arxiv. 2312. 14925.
Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine- grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling. CoRR, abs/2403.13787, 2024.
URL https: /ttp///iiiio///////ooo org/ 48550/ arxiv 2403.13787
Nathan O. Lambert and Roberto Calandra. The alignment ceiling: Objective mismatch in reinforce- ment learning from human feedback. CoRR, 03a/2311.00016 2023.
Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang. Hansen Zhong, and Wanli Ouyang. Iterative length-regularize direct preference optimization: A case study on improving 7b language models to GPT-4 level. CoRR, abs/2406. 1 11817, 2024a. doi: 10.48555/ARRXII.246..111 https: /ooi. www.ooo...............o......... 48500 arxiv 2406. 11817.
URL
Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, and Xunliang Cai. Length desensitization in directed preference optimization. arXiv preprint arXiv:2409.0644 06411, 2024b.
Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. Eliminating biased length reliance of direct preference optimization via down-sampled KL divergence. CoRR, abs/2406. 10957, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. CoRR, abs/2405. 14734, 2024.
Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anea D. Dragan, and Stephen Mareus McAleer. Confronting reward model overoptimization with con- strained RLHF. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.ned 2024.
Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Pro- ceedings of the Third Conference Machine Translation: Research Papers, Brussels, Belgium, October 2018. Association for Computational Linguistics.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08874, 2023. URL https:://bl.orgg/// 48550/arXiv. 2303. 08774.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022.
13

---

Published as a conference paper at ICLR 2025
Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, and He He. Reward gaming in conditional text generation. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023. Toronto, Canada, July 9- 9-14, 2023, pp. 4746-4763. Association for Computational Linguistics, 2023.
Junsoo Park, Seungyeon Jwa, Meiying Ren, Dacyoung Kim, and Sanghyuk Choi. Offsetbias: Leveraging debiased data for tuning evaluators, 2024a.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 4998-5017. Association for Computational Linguisties, 2024b.
Zihan Qiu, Zeyu Huang, Shuang Cheng, Yizhi Zhou, Zili Wang, Ivan Titov, and Jie Fu. Layerwise recurrent router for mixture-of-expert. CoRR, ths/2408.06792 2024a.
Zihan Qiu, Zeyu Huang, and Jie Fu. Unlocking emergent modularity in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 2638 2660. Association for Computational Linguistics, 2024b.
Shanghaoran Quan. Dmoerm: Recipes of mixture-of ex-eerttss for effective reward modeling, Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August 1 11- 1--1,, 2024, pp. 7006-7028. Association for Computational Linguistics, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. URL httLhtttp///ppeeeeee nips oc htc//aper__fileee hwt.fiilssspppeee/eeeeeeemeeeeee /ttpp/aappppppee/eepeeeaaee 2023/ / 2023/hash/ hash/ a
html.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: on the benefits of weight averaged reward models. CORR, abs/2401.12187,, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. GPQA: A graduate- google-proof q&a benchmark. CoRR, abs/2311 120222 2023.
Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization LLM with demonstration guidedd reinforcement learning. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thai- land and virtual meeting, August I 11-16, 2024, pp. 12447-12472. Association for Computational Linguistics, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversar ial winograd schema challenge scale. Commun. ACM, 64(9):99-106, 2021.
Wei Shen, Rui Zheng, Wen Yu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pp. 2859-28733 Association for Computational Linguistics, 2023.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF CoRR, abs/2310.03716 2023.
14

---

Published as a conference paper at ICLR 2025
Julius Steen and Katja Markert. Bias in news summarization: Measures, pitfalls and corpora. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pp. 59622-983, Association for Computational Linguisties, 2024.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325,, 2020.
Haoran Sun, Yurong Chen, Siwei Wang. Wei Chen, and Xiaotic Deng. Mechanism design for LLM fine-tuning with multiple reward models. CoRR, abs/2405. 16276, 2024.
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu- Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented RLHF. CoRR, abs/2309. 11133099.4555... 14525, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether thain-of-thought can solve them. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (eds.). Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 13003-13051, Association for Computational Linguistics, 2023.
Gemma Team. Gemma. 2024. doi: 10.347700KAAGGLL///33000 URL https://www.kag// com/m/ 3301.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-expert. CoRR, abs/2406. 12845, 2024.
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with llm-as-a-metta-uddee CoRR, abs/2407.1959) 2024a.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. CORR, fas/2405.00675. 2024b.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for lms. arXiv preprint arXiv:2406. 102 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Unnertaintty-eenalize reinforcement learning from human feedback with diverse reward lora ensembles. CoRR, abs/2401. .00243, 2024.
Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble. CoRR, abs/2401.16635,, 2024a.
Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists emojis: How format bias affects model alignment. arXiv preprint arXiv:2409. /704, 2024b.
Zhuocheng Zhang, Shuhao Gu, Min Zhang, and Yang Feng. Addressing the length bias problem in decument-level neural machine translation. CoRR, abs/2311.11601, 2023.
15

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023a.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan. Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of RLHF in large language models part I: PPO. CoRR, 0is/2307.04964 2023b. URL https: http://oooooo/////////// .org/0//g 10. 48550/a 2a.00/raxvvv. 2307 04964.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follovinng evaluation for large language models. CoRR, abs/2311.0099) I, 2023.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang. Wanli Ouyang, and Yu Qiao. Beyond one- pwwference-fitsseeeeeeeeeeees--- the Association for Computational Linguistics, ACI. 2024, Bangkok, Thailand and virtual meeting, August I 11-16, 2024, pp. 10586-1066 10613. Association for Computational Linguistics, 2024.
alignment: Multiiobjective direct preference optimization. In Findings of
Banghua Zhu, Michael I. Jordan, and Jiantao Jiao, Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHE, CoRR, abs/2401.16335, 2024.
A LENGTH BIAS
Length bias emerged as a significant issue in Natural Language Processing (NLP), affecting a wide range of tasks and applications, including machine translation (Zhang et al., 2023; Murray & Chiang, 2018) and text summarization Guo & ffooughi (2023); Steen & Markert (2024). This bias occurs when models disproportionately prefer outputs of a certain length, which stands as of the most consequential sources of bias in NLP.
In LLM era, length bias (also termed the verbosity issue) continued to be pressing concern. One key process to develop such LLMs is RLHF, which employ human preference data to fine-tune LLMSs (directly or indirectly) to generate more human human-aligned outpus but also may unintentionally amplify the length bias in the preference dataset. Therefore, mitigating length bias become innvreasingly important, drawing significant attention. Prior works include: (1) works focusing on RMs training: Chen et al. (2024b) propose to jointly train two linear reward heads on top of the same model, one trained to correlate with length and the other trained to decorrelate with length, their experiments demonstrate that the approach eliminate the Pearson correlation to 0 (same Shen et al. (2023) jointly optimise a main expert RM and a length-bias only expert R.M, and exclusively use the main expert in the RL phase; Hou et al. (2024) devise Bucket-Based Length Balancing, that they divide the preference data used to train RM in to different bucket based on length difference, and within each bucket, they balance the number of examples where the better/worse response is more lengthy; (2) works on focusing on LLM alignment: Meng et al. (2024) leverages the eength-normmlised reward to prevent model from generating longer but low-quality responses; Lu et al. (2024) modify the DPO objective by down-sampling same amount of tokens from the chosen and rejected responses when calculating the implicit reward, similar with Liu et al. (2024b); Chen et al. (2024a); Liu et al. (2024a); Park et al. (2024b) add an extra penalty term to the implicit reward to penalise the verbosity; Wu et al. (2024a) set a threshold for top-tier response and select the shortest one as the best response when constructing the preferenc data; and (3) works focusing on the evaluation: Based on AlpacaFval benchmark, Dubois et al. (2024) propose use L.LM ID and instruction ID to do regression to estimate the bias; Chiang et al. (2024) employ the same method for Chatbot Arena.
approach); Similarly,
All these previous studies underscore the prevalence and significance of length bias across different subfields of LLMs, highlighting the pressing need for a unified approach to address this issue comprehensively. While our method was not specifically designed to target length bias, it has been empirically validated to effectively mitigate this problem in various contexts, demonstrating its broader applicability.
16

---

Published as a conference paper at ICLR 2025
B FULL ALGORITHMS
The algorithm describing employing the LOWESS method for length bias calibration is shown in Algorithm 1. The main idea first fit a locally weighted regression and then adjust the weight based on the residual between regression prediction and the ground truth label. LOWESS is non-parametric regression technique typically used for smoothing scatterplot. is particularly useful when specific functional form is assumed. Key uses of LOWESS include: (1) Data Smoothing: LOWESS is used to smooth out noise in data, providing a clearer trend by creating a smooth curve that captures the underlying pattern. (2) Data Analysis: By overlaying LOWESS curve on a scatterplot, one can explore the potential relationship between variables without relying on predefined functions, and it's particularly useful when the relationship between variables is complex and non-linear, as it adapts to the shape of the data. Based on the typical usage of LOWESS, in this paper, we employ LOWESS to investigate the potential relationship between rewards from biased RM and a characteristic in the output. By assuming that they are actually independent, we regard the prediction from the LOWESS as the bias term and remove to recover the underlying true reward.
Algorithm LOWESS for Length Bias Calibration
Input: Output length and corresponding rewards ro(;)) for 1,.2..... .. n. the band- width f to demterming the fraction of dataset used for regression, the number of iterations k
2: Output: Calibrated rewards 1 (ara)
3: for each x; do
4: Compute the distances d, ||æ |æ,ا|| for all j ..
5: Determine the d, as the distance to the q-th nearest neighbor, where & = [fn]
6:
Compute the tricube weights way
d:g
for di S di
7: Fit weighted linear regression near ur; arg gin9 E, (in(ro((;)) 9o 111
8: Compute the fitted value to ((x:)) 8. + B1|aal
9: end for
10: for t= 1 to 1: do
p. Robustifying iterations
Compute the absolute residuals A; = |ro(s;)) - (((a))))
12: Compute the median of absolute residuals: median A.)
13:
Calculate robustness weights 6; B
A c
where B(u) max(0, (0, =2
14:
for each T; do
15:
Combine weights: w. 8, Wi
Refit the weighted linear regression: arg min 91 E, u (re(z;) Bo =1,1,,1)2 Compute the fitted value fo( 1) AG 2;
17:
18:
Compute the calibrated rewards (z,)) = ro (x;) (o(1
19:
end for
20:
end for
21:
return Calibrated rewards i (x::
C EXPERIMENTAL DETAILS
Experiments with AlpacaEval AlpacaEval i. popular LLM benchmark ranking over 200 models. The official leaderboard employs GPT4-110 10-006-iiiiee judge. In our setting, we primarily use open-source BT-based RMs to score the LLM's output. Given prompt q and corre- sponding output from the evaluated LLM and output from the baseline LLM, the RM scores them as r and Ts, respectively; then the Bradley-Terry model yields the preference probability orr rb). We then compute the average of the perference probability p(y > vol9) across all prompts to determine the final win rate for ranking all evaluated LLMs. We directly use the ptoposed method to calibrate AEI. Since AE1 uses GPT4 to compare two responses pairwisely and uses its output logits to compute
p(y > y6 2644)
model's 's output from the AlpacaEval official repo. 3 We also
hhttps://gi thub. cowb..com/tart wwt.//ooottttoooeeeeeo 1ab/a 11paca_aeee /ttee//eeaaaeeeeeeeeee resuu 1
17

---

Published as a conference paper at ICLR 2025
the preference probability ply > yblz). Thus, following Eq. 1, employ the inverse function of sigmoid to calculate the reward margin r - rb. We then utilise the LWR to directly predict the biased reward margin given the length margin ly -1bl--
Hlyper-parametenn Two key hyper-parameters for our proposed method, RC-Mean and RC-LWR, are the threshold d and the bandwidth f. (1) Determine the threshold d: The way we use to determine the threshold distance d in Eq. 6 is as follows: (1) Taking the length bias calibration as
an example, given a dataset of data pairs to calibrate {2f 22 re(ai) 1,1,,, re (2²) we first calculate the
average of [zil - [zgil, noted as [[ - [x2l))). (2) then we set d AVG([ | /4. which is set by intuition. And if the number of data points within the neighborhood is smaller than 10, we do not calibrate that pair as the results may be overly affected by specific data points. This set of haperepasameters is employed for all three different experimental settings. (2) Determine the bandwidth f for LOWESS: this hyper-parameter is closely related to the assumption. Regarding the experiment of RM as LLM evaluators and RM for LLM alignment, we consider the sufficient density assumption valid since there are 150k samples and 300k samples, respectively. We thus set the bandwidth as default as in stat smodels api, f 3. and the number of iterations is 3. (Though we use all scored data points for LOWESS regression, we show in Fig. 12 and Fig. 13 that the methods yield improvements with only hundreds of data points.) Regarding the experiment on the RewardBench, there are only 2,985 data pairs, resulting in 5970 data points. Therefore, the sufficient density assumption may be invalid for some length intervals, so we choose a larger bandwidth f = 0.9. Importantly, all syper-parameters relevant to the reward calibration haven' 't been tuned for specific RMM. For RewardBench experiments, hyper-pate parameters for all 33 BTT-ased RMs and 46 DPO-based reward models are the same, and for AlpacaEval calibration, the hyper-paramert used for models are the same. We argue that this further validates the generality of our proposed methods. In practice, one could conduct a more comprehensive hyper-paramess search for specific R.M.
Computational Environments All experiments about reward calibration are performed with a single CPU. Experiments for getting the reward model's score on AlpacaEval's output conducted using a NVIDIA H100 GPU. Experiments regarding using DPO for LLM alignment use NVIDIA H100 GPU and costs about 2-3 hours for the DPO training.
D EXTRA EXPERIMENTAL RESULTS
Calibration Results For DPO-based Reward Models In addition to calibrating the traditional
BT-based RMs, we calibrate length bias with DPO-based reward models. Formally, given a pair of completions (y1, 32) to the prompt the preference of the DPO-trained model Ta could be
my(y1pp) determined by comparing the following log-ratios: (re and Tre p)* where Tref is the
=o(ys[p)
reference model during the DPO training. We directly employ the official rewarding results from
the RewardBench leaderboard. The reasons why choose to present the results for DaO-based models in the appendix instead of the main text are twofold: (1) The rewarding results for DPO-based model are more noisy some of their reference models not available; (2) To the best of our knowledge, DPO-tuned models are more frequently used as chat models or instruct models, and are rarely employed indicate the preference with the implicit reward. The results shown in Fig. 6. Our observations are as follows: (1) The overall reward performance of DPO-based models is inferior to that of ba-ba reward models, as well as their improvements after calibration. (2) The proposed methods, RC- LWR and RC-LWR, are effective in most cases, achieving average performance gains of 1.60 and 1.68, respectively. (3) The method RC-Mean can lead to improvements for most models. However, it can also result in performance collapse. For instance, after calibration, the performance of Qwen1 1015--77-hhhhee
decreases from 71.5 to 49.9. (4) The Length Penalty shows no calibration effect for most DPO-based models because the absolute values of log ratios, used as implicit rewards, are usually high. This finding further emphasizes that the penalty weight requires more comprehensive research in practical applications and is highly dependent on the specific reward scales of the model and the length distribution of the datasct.
Robustness to the choice of bandwidth We investigate the extent to which the calibration effect is influenced by different choices of bandwidth f. The experiments are conducted using RewardBench
18

---

Published as a conference paper at ICLR 2025
136
5.51
Largh
Man
BC-WW
BC-IWB-Naaly
4.40
21.6
Lengls Peedly
(MK Parlly
Figure 6: Results of different calibration algorithms on the RewardBench henchmark for DPO- reward models are shown in two charts. Left: the line chart demonstrates the RewardBench score before calibration (x-axis) and after calibration (y-axis) of different algorithms for different models. Right: the scatter plot highlights the performance gains achieved by different calibration methods, annotated with maximum, average, and minimum values.
10.09
9.77
9.36
9.00
9.01
Improvementt
2
2,85
2,83
2,88
2:90
2,94
2,97
2:96
2,03
2,24
-2.45
-2466
-2.22
-1.77
-1.39
-1.7
-1.03
-1.01
6.1
0,3
6.5
9.3
03
1.
Bandwidth
10
10:10
10:01
9.88
9.68
9.60
9.37
9.40
9.22
8.86
Ingrovemmntt
2,84
2,94
2,96
7,99
3,11
2,86
25
-1.58
-1.88
-1.52
-1.50
-1.02
-0.94
2.12
2.13
-2.25
-2.00
Bundwidth
Calibration results, shown in the improvements, of RC-LIRR (top) and RC-L.RR-Pena lyy (bottom) with different bandwidth f on the RewardBench datasct.
Figure 7:
because, for the RewardBench setting, there are only 6k data points, and the smaller number of total data points for calibration may increase sensitivity to variations in bandwidth. The overall results for different bandwidths for both RC-LWR and RC-LWR-Penal ty are presented in Fig. Our empirical findings are as follows: (1) Overall, the proposed methods demonstrate robustness across varying bandwidths, achieving similar average performance gains when the bandwidth ranges from 0.1 to 1.0. For RC-IWE (top), the average gain ranges from 2.68 to 2.96, while for -WW--Pnaltteeee it ranges from 2.84 to 3.11. A slight performance improvement observed when f increases from 0.1 0.9. (2) Although the average performance gains remain relatively consistent, the maximum improvements and degradations exhibit more notable trends. Specifically, as the bandwidth increases, both the maximum improvements and degradations diminish. These results suggest that calibration
19

---

Published as a conference paper at ICLR 2025
can be controlled through bandwidth selection, with larger bandwidths yielding more stable calibration effects.
Controlling calibration effects One critical assumption for our proposed method is that the underlying gold reward is independent of the biased characteristic. We also showcase that our method could reduce the correlation between rewards and the characteristic of interest to O. However, the independence assumption may be invalid if one focuses on a specific subset of instructions. For example, the independence assumption is considered valid given a general prompt set. But if we add "generate contents as concise as you can" at the end of every prompt, thus the underlying true reward should be positively correlated with the output length for this prompt set. Therefore, more practical calibration method should be controllable for users in different application scenarios. We show that our proposed method can also be controlled by introducing hyper-parameter named calibration constant y to the Eq. 7:
Li (21. = Ar, (2), y x (fo(c(=1)) - to (o(c(22)))
(8)
We validate the effects of y using two experimental settings: RM on the RewardBench and RM for LLM alignment. The ablation results on the RewardBench dataset are presented in Fig.8, while those for the LLM alignment setting are shown in Table 4 A noteworthy statistic for the RewardBench dataset is that simply selecting the shorter response yields 60% accuracy. Thus, the ideal correlation between rewards and response length should be slightly negative. The results in Fig. support this notion: if = 1 results in zero Spearman correlation as indicated in the Table. 3, the optimal y should exceed 1. Our findings indicate that the best average performance gain occurs when 7 = 1.4. More importantly, the results across both settings demonstrate that the calibration effect controlled through the calibration constant. This significantly enhances the practical utility of the proposed method, as one can adjust this constant achieve superior rewarding performance if certain prior knowledge about the correlation between rewards and the characteristic of interest is known.
be smoothly
9.58
10:00
10.26
10.17
8.49
9:01
8:99
7,81
7.22
6.57
Improvements
1-
+2
2.11
2
2:07
2
2,93
3102
3/13
3,08
3:14
3.09
1"
3
-0.7
41.90
-1.03
-1.36
-219
-2.99
-5.08
-577
Calibration Constant
Figure 8: Ablation results of different calibration constant on the RewardBench for BT-based RMs.
Length distribution Defining the length margin the absolute difference in length between two completions, the cumulative distribution function of the length margin on four different datasets is shown in Fig. 9. As shown in the Fig. 9, different datasets demonstrate distinct length margin distributions. For Ultrafeedback-Lllmaa and Ultrafeedback--Gemte length margin that is less than 600, which is reasonable because the compared completions are sampled from the same LLM. And the RewardBench dataset demonstrates a similar pattern. In contrast, the length margin on the AlpacaEval benchmark is a lot longer, with only about 40% of the data points having a length margin that is less than 600. Notably, the simple Length Penalty method works well in calibrated RM on RewardBench and calibrated RM for LLM alignment settings. This may be because the length margin in those two settings is smaller, thus leading to moderate penalization. However, when length margins are more significant. more meticulous adjustments for penalty weight are required as they are more sensitive. In summary, the baseline Length h Penalty y is already strong when RMs exhibit strong length bias. However, both reward scales and length margins should be considered to adjust the penalty weight, impairing its practicability. In contrast, the proposed method RC- is robust to different parameee choices, yielding stable improvements in different settings.
about 80% of data pairs have
20

---

Published as a conference paper at ICLR 2025
Table 4: Results of using Length- Calibrated rewards LLMs' alignment with different calibrate constant ?y in RC-IMR. We apply these methods with four different LLM RM configurations. The top reports the Length-controlles win rate, and the bottom showcases the average length of generations AlpacaEval dataset.
Algorithm
0.4
1.0
Length-Controid Win Rate
Llama-3-8B--stf
41.82
42.59
46.39
48.12
50.25
51.37
Llama-3-0B-GRM
41.00
43.54
46.44
49.76
49.61
50.49
gemma2- -
62.79
63.96
62.69
67.57
68.87
69.54
temma2-9b-Gas
63.21
67.23
66.95
67.27
68.27
70.45
Average Character Length on Alpaca Benchmark
2404
2365
2340
2194
2028
1867
11ama-3-8B-GGB
2438
2375
2303
2194
1986
1858
2365
2321
2570
2028
1725
1798
gemma 2ommma299--Gm
2106
2117
1939
1860
1638
1780
Cunulative Distribution Function
RewardBench
ALgacaEval
Ultrafeedback Llama3
Ultrafeedback-Grre Gemma2
de tbo0 1500 1700 1900 21901209 2eoo
marzin
Figure 9: The cumulative distribution function of the length different datasets used in experiments.
Sysemon comvleiee -0
Spuarmonn oxmebrien
Pefonurn
Calhnot
Sp Ouror Legg and Rowarss RewenBBonc
Sparmeee of Ougur Lorgh and Rowedd on RewxeBBooch
Figure 10:
Based Chat Hard subset of RewardBench. we plot performance (left) improvement brought (right) by RC-LNR vary with the RM's length preference, which is represented by the Spearman score between output length and rewards on RewardBench.
Reward models with more substantial bias are improved more. Results in Fig. 4 motivate us to investigate how calibration gains vary for RMs with different length bias. Therefore, we selected Chat Hard, the most challenging subset of RewardBench, for our investigation. Results are shown in Fig. 10. We use the Spearman correlation represent the RMs' length preference on the x-axis; we then plot the RM's performance and RC-LWR improvements, different points in Fig. 10 refers different RMs. Fig. 10 left, we observe that the RM's 's performance on Char Hard negatively correlates with the RM's s length preference, achieving - 2 correlation. (Note that rule-based RM that always chooses the shorted sentence could achieve an accuracy of 90% on Chat Hard subset.) In Fig. 10 right, we observe strong positive correlation between the RM's length preference and its improvement on the Chat Hard subset, achieving C. correlation score. Note that similar trends are found between length preference and overall performance improvement of RewardBench with 0. 47 Spearman Correlation score, both indicating that Reward Models with stronger length preference are improved more by the proposed RC-IWR method.
21

---

Published as a conference paper at ICLR 2025
SOTA: 92.7
8.76
GP::-Turbu:
Improvemets
Original Reward
RC-LWX
2,55
-1.12
93
Methods
Figure 11: Results of calibrating with output length and markdown features simultaneously.
Calibrate for more than one characteristic One straightforward extension calibrate with more than one characteristic is to employ culli-dimennsonal Locally Weighted Regression. For example, consider calibrating two characteristics 61 and C2 simultaneously, the dataset employed to do the LOWESS regression could be described as D { (#g) (12)),2((1)) ro(rr Then, the Euclidean distance employed measure the distance between different data points. Based on this, we extend our method to directly calibrate rewards with output length and markdown features. Because these two characteristics have very different scales, we first normalize their values by Z-score normalization. The calibration results are shown in the Fig. 11. We observe that: (1) the proposed method RC-LWR could directly generalize to calibration for multiple characteristics, yielding a 2.58 average performance gain, (2) the gain is superior solely calibrating markdown bias (1.86) but inferior to solely calibrating length bias (2.93). Possible reasons for no synergy effect include: a. The two characteristics of interest here, length and markdown features, may correlate, and the method does not touch on thiat; b. the Euclidean distance used here considers two characteristics equivalent.
Data efficiency LOWESS is non-parametric method that requires ground truth labels to compute residuals during its robustifying iterations, making it logical to utilize all available data points for calibration. We demonstrate that even with moderate dataset size (2,985 preference pairs in RewardBench), calibration yields significant improvements. To explore this further, we conduct a stress test the number of data points used for calibration. Focusing our experimental procedure involves sampling subset of the dataset, performing calibration on the selected subset, and using accuracy the evaluation metric. It is important note that accuracy is not directly comparable to the RewardBencch score in our main experiments, The RewardBench score the average accuracy across four subsets (Chat, Chat Hard, Safety, and Reasoning), which vary in size. We define the subset size as fraction of the original dataset, ranging from 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, to 0.5. For each subset size, we run ten experiments with different random seeds and average the performance across these seeds. The results for different subset size for RC-LMR-Penalts RC-LWR, RC Rn are shown in the Fig. 12, Fig. 13 and Fig. 14, respectively. We observe that intuitive method, RC-Mean, achieves increasing accuracy as the subset size grows, while RC-LWR and RC-LWR- Peeee-enattyyyy exhibit more stable improvements across different sizes, demonstrating that method is not heavily constrained by data density.
the RewardBench dataset,
Further analysis about the chosen length penalty baseline our main experiment results, we follow previous research works to set = 0.001, which may raise concerns that the a is not optimal for the selected tasks and reward models. Therefore, we first do ablation studies with different a values from {0. 1, 0.05, 0.02, 0.01, 0.005, 0.002, 0.001, 0.0005, 0.0002, 0.0001), }. To further decouple the effect of the fact that different reward models have different reward distributions, we propose to first z-normalize the reward and then apply the Length Penalty method. The results of these two methods are demonstrated in Fig. 15. According to Fig. 15, the chosen value a = 0.001 achieves the best performance compared with other values (Original line). The Z-normalization trick could slightly boost the performance to 2. 75, which is reasonable because alleviates the issue that different reward models have different distributions. However, both of them underperform the proposed
22

---

Published as a conference paper at ICLR 2025
4,00
4.02
3.84
3.56
3.66
Improvements
2:79
2.62
2.69
2.
2.68
5-8
2.01
1.14
0.74
0.33
0.
0.54
0.0
6.75
7.1
93
Subset size
Figure 12: Calibration results with different subset size for RC-LNR-Penalty method
3.52
3.50
3.84
3.73
3.33
3.22
Improvement
2.5
2:42
2.28
2.40
2,36
2,36
2.35
23-
1.3
1.31
1.54
1.2
0'54
0.44
0.42
0,37
0.46
Subset
Figure 13: Calibration results with different subset size for RC-LWR method
method. Furthermore, we argue that the optimal a is challenging to search for the following reasons: (1) both the reward distribution of the model and the length distribution of the task affect a, asking users to search different a per reward model and tasks repeatedly; (2) the method is very sensitive to the value of a, for some values the model may even collapse. On the contrary, our proposed method could robustly adapt to different reward models and tasks.
Reward calibration v.s. Reward Ensemble As discussed in our related work section, previous works can utilise ensemble techniques to mitigate the potential bias. We thus try to compare our proposed RC-LWR with the ensemble. Specifically, we use normalize-then-aveeie average to aggregate rewards from different reward models. We first select the top nine models from the RewardBench Leaderboard of Aug and then aggregate them. The results are shown in the Fig. 16. In Fig. 16, the number in the cell [i, j] denotes the ensemble performance of model and model j where and j are the model's 's rank on the RewardBench leaderboard. The last row is the calibrated performance of models. We find that the calibration outperforms most ensemble results except for ensembles with much stronger model. And intuitively, ensemble- methods may not address common bias across models. For example, these models overly prefer lengthy outputs, their ensemble model will also prefer lengthy output.
Visualization To better illustrate how our proposed method shifts the reward distribution of the model and removes the length bias, we present the kernel density plot (KDE) of rewards (y-axis) w.r.t length (x-axis) for different reward models, calibration methods and hyper-parameters, Specifically, we select ten different reward models (Fig. 17- 26). For each reward mode, we visualize four distributions based on RewardBench dataset: (a) The first (left) figure demonstrates the original distribution between the reward and character length of the example; (b) The second figure demon- strates the calibrated distribution of our proposed RC-LWR method: (c) The third figure demonstrates the calibrated distribution of the proposed variant of RC-LWR (Eq. 8) with y 1.25; (d) the last (right) figure illustrates the calibrated distribution of the Length-Penal:: baseline with 0.001. In each figure, the colour represents the density of data points. The deeper the colour,
23

---

Published as a conference paper at ICLR 2025
3-59
3.22
3.06
Improvementt
1.78
1:52
1,62
1.73
0.76
0.33
04
-0.10
-0.20
0.
-0.18
-0.30
-0.31
-0.8
1,00
302
6.b
2.1
Subset size
Figure 14: Calibration results with different subset size for RC-Mean
5.0
1.18
-25
-5.0
-7.5
10.0
-12.
Normalize
Alpha Values
Figure 15: The performance Length-Penalee Penalty baseline with different a values in the RewardbBnech setting. Two different variants are considered here. Original means that we directly add the penalty term to the Reward Model. Normalize means first z-normalise the rewards and then apply the Length-Pena. baseline.
90.26
92.25
90.09
89.66
89.56
89.56
89.66
88.93
88.31
92.25
89.74
91.43
89.56
89.08
89.29
89.57
88.71
89.03
90.09
91.43
87.85
89.00
88.65
88.70
87.68
87.16
87.22
89.66
89.56
89.00
87.00
86.77
86.72
86.17
86.59
86.14
86.77
89.56
89.29
88.70
86.72
86.23
8:4.7
85.70
85.60
84.82
87.68
88.93
88.71
87.16
86.59
85.86
85.60
85.36
82.78
84.48
88.31
89.03
87.22
86.14
85.74
844.82
84,48
82.70
Calleate
92.08
89.53
91.75
90.19
88.53
88.12
88.04
86.34
#5.67
Figure 16: Ensemble performance calibrated performance
24

---

Published as a conference paper at ICLR 2025
the denser the data points. The red dashed line illustrates the LOWESS smoothing for the KDE plot. According to these figures, we find that: (1) As discussed in our introduction, the length bias exists for nearly all selected ten reward models. (Note that the ideal correlation on RewardBench should be slightly negatíve.) And the correlation non-linear. (2) The red dashed lines become almost horizontal after RC-LWR calibration, indicating that method removes the reward's correlation with length smoothly and uniformly. (3) By setting 7 1.5, we can push the correlation of rewards and length be negative, showcasing the potential of method to inject desirable bias into reward models. (4) Regarding the Length Penalty baseline, though the Spearman correlation between reward and length decreases, the length bias is not removed. For models with no length bias (e.g.. NCSOFT / Ll ama 3- 3-0ff feeffeetiase ------M---- RM-8B), the length penalty method will push the reward to prefer shorter responses.
Spearma conclation: 0.37
Sparmmas cocreltion: -0.03
Spearman carekitoon 0.10
Spearmn conclation 0.02
Figure 17: Lat.emal//Anccrrnmmm------------
Speamun corrilationn-
Sparmmss cocrduiia: -0.44
Spearman carelitions .004
Spearmnn conclatio: .030
Figure 18: MCSOFT//llmaa330ffooooo
Spearmnn conclation 0.38
Spaammas coerduiinn -0.03
Spearman careltion .0.11
Spearman conclation 0.85
Figure 19: internlm/ Mentermln/intteenn22--11
25

---

Published as a conference paper at ICLR 2025
Syeamun corristion 0.33
Sparman oxniation -0.4
Sparmon cordi cior +1 14
Syeamion corrdttionn 0.20
-5
500
1500
200
300
280
Figure 20: Ray2333/GRM-11 3maa3 - strre
Speaaman conclation: 0.30
Sparmmas courdation 0.00
Spearmen surcehiim -007
Speanmun cvereation:
Figure 21: Ray2333/GRM-D -
Syeammn corriationn 0.15
Sparman ourciation -0.65
Speermn conciatius +0.12
Sperman comblttoo 0.0
-1
-15
3.0
1300
2000
200
1800
2000
Figure 22: sfairxc//aaaaltt-Laaaa
Syeamon corriltioon 0.3)
Sparman oxniation +0.67
Speemon cond 41
Sycammn corriation 6
3
7.0
1300
2500
200
1/56
250
Figure 23: CIR-AMS/ SATMM///EEMM..Oe2.2aaaaae 2_b__0613
26

---

Published as a conference paper at ICLR 2025
Speazzus ourclaton:
Speamann -enamann-ddddaaa---0000
Speurnzn concilaioon
Spearmas 0.31
100
Bid
Figure 24: DpennbbbEuurssRRR
Speanmus cocrelation: 0.4]
Sperman cocreltion: +002
Spearman committion 40.14
Speanmas cocreaaion 0,85
180
300
J00
1100
204
X6
X00
190
206
Figure 25: Nexussflow/Starlingg RRM--334
Speaman conclation:
Sparmes cocrdstion
Spearmen curchiiin
Spuaeman comelatonn -0.12
1550
2000
200
1550
250
3000
200
280
Figure 26: internllm/intern11222 8b-rewar
rd d
27


Reviews:
[{"review_id": "spv9Tz5TUI", "reviewer": "Reviewer_PP1Z", "summary": "This paper proposes a novel method, “Post-hoc Reward Calibration,” to mitigate biases in reward models (RMs), particularly focusing on LLM output text length bias in RMs for large language models (LLMs). The method removes this bias by calibrating the reward score without additional RM retraining, making it computationally efficient. The approach assumes independence between the true reward and text length, constructs a locally linear regression model from text length to (proxy) reward values using the Robust Locally Estimated Scatterplot Smoothing (LOWESS), and calibrates the (proxy) reward with this regression model. The authors evaluate the approach on three tasks—RewardBench, RM as LLMs Evaluator, and RM for LLMs Alignment—and demonstrate that the calibrated RMs better align with human evaluations by providing fairer, length-independent scoring.", "questions": "How is the sensitivity of the length penalty hyperparameter to the performance compared to the proposed methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Originality**: The paper proposes a unique, post-hoc calibration approach to mitigate reward model biases related to LLM output length without requiring additional data or retraining.\n\n**Efficiency**: This method is computationally efficient, applying bias correction as a post-processing step, making it easily applicable across tasks without modifying reward models.\n\n**Empirical Validation**: The approach is thoroughly evaluated on three tasks, effectively demonstrating its capacity to mitigate length bias.", "weaknesses": "**Assumption**:\nThe assumption of “Independence of the biased characteristic” may be overly restrictive in practical applications, especially given that this study focuses on text length as the biased characteristic. My understanding of this assumption (line 166) is that even when $x_1$ and $x_2$ are fixed at any arbitrary lengths $c_1$ and $c_2$, respectively, the true reward difference between $x_1$ and $x_2$ would be expected to be zero, implying no correlation between reward and text length. However, in practical applications, this assumption will often not hold. For example, in cases where prompts explicitly request brevity or detail, such as “respond concisely” or “provide a detailed explanation,” reward often depends on the response length. This assumption could thus limit the generalizability of the proposed method, especially for tasks where length naturally influences quality. A more flexible approach that adjusts for prompt-specific length expectations could enhance applicability.\n\n**Baseline setting**:\nThe Length Penalty hyperparameter $\\alpha$ seems to be fixed at 0.001, which may not be optimal across different reward models (RMs) and tasks due to variations in scale. While the proposed method benefits from adaptive $d$ tuning, the baseline might perform better with an α adjusted per RM or task, ensuring a fairer comparison. For instance, $\\alpha$ could be proportional to the RM’s reward scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dYN6xtPwAs", "reviewer": "Reviewer_Lf2Z", "summary": "This paper focuses on correction of spurious correlations in LLM-based reward models learned from preference data. The authors assume that the spurious feature (e.g., length) biases the rewards in a consistent away. They propose to estimate this consistent bias via a locally-weighted-regression method. Then, the estimated bias can be subtracted from learned reward in order to achieve a model without the spurious correlation. The authors focus on length as the spurious feature and compare to a baseline which subtracts a constant factor times the length from the reward. They evaluate a variety of reward models with various adjustments to eliminate length bias across several metrics, including RewardBench scores, correlation with human rankings of LLMs in ChatbotArena, and length-controlled win rate in AlpacaEval2. In general, the proposed method appears to better remove length bias.", "questions": "With regards to the second weakness above, it would be helpful if the authors could plot the learned $\\hat{r}_\\theta(c)$ function for several reward models to see what kind of affect subtracting it has compared to the linear length bias $\\alpha \\\\, c$. Also, it would be helpful to compare to a baseline of a linear length bias where the factor $\\alpha$ is tuned per reward model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed method is relatively simple and is well-motivated conceptually. The fact that it can be applied post-hoc means that retraining is not required, which is helpful. I am not very familiar with the literature in this area but to my knowledge it is also novel. The results are extensive and seem to generally support that post-hoc reward calibration effectively removes length bias.", "weaknesses": "The main weaknesses of the paper include:\n\n * **Presentation:** I found the introduction of the method in Section 3 to be relatively clear, but the experiments in Sections 4 and 5 were much more difficult to understand. While I eventually could figure out the gist of the experiment design, it is often quite complicated and could be better presented. In particular, Section 4.2 is very dense. It describes multiple evaluation methods (AlpacaEval1, AlpacaEval2, ChatbotArena), which if I understand correctly in some cases used as both gold standard (AlpacaEval2) and as baselines (LC). The combination of many reward models, LLMs, calibration methods, and comparison rankings makes it quite important to carefully describe the purpose behind the experimental setup and why it is a good evaluation of the calibration methods. I think the current text requires several readings to understand these details. The rest of Sections 4 and 5 are also very dense and difficult to understand at first. It would be good to improve the presentation of these such that the results are easier to understand.\n\n * **Lack of insight into the result of reward calibration:** while the authors present many experiments on how reward calibration affects downstream performance, they do not present what the actual calibration process looks like for typical reward models. It would be helpful to see what the actual $\\hat{r}\\_\\theta(c)$ function looks like over a range of values for $c$ for various reward models. This could help readers understand why RC-Mean and RC-LWR outperform length penalty, especially if the $\\hat{r}\\_\\theta(c)$ function is quite non-linear. On the other hand, if it is close to linear, then it would raise the question of why not simply tune the constant $\\alpha$ in the linear length penalty. Perhaps the length penalty simply does not use the right coefficient—in fact, it seems that it should always be possible to remove the length correlation measured in Table 3 by using a linear length penalty with the right coefficient.\n\n * Finally, the authors do not consider the extent to which the correlation between length and reward may not be entirely spurious—there may actually be some amount of weight that the reward models *should* place on length. \n\nSmall issues:\n * \"LLMs alignment\" -> \"LLMs' alignment\"\n * Line 470 \"board\" -> \"broad\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5IziiMdm6K", "reviewer": "Reviewer_fLWP", "summary": "This paper focuses on the reward modeling process in Reinforcement Learning from Human Feedback (RLHF). The authors highlight the issues of bias and spurious correlations in the training data and propose a method called post-hoc reward calibration to address these limitations. Specifically, they assume the reward model can be decomposed into two components and use the evaluation set to estimate the bias term through locally weighted regression. The experimental results demonstrate the method's effectiveness.", "questions": "1.\tBias and Spurious Correlation: How does the characteristic bias $c$ lead to spurious correlations? Examples would facilitate a better understanding for the reader.\n\n\n2.\tHyperparameter Selection: What guidelines are there for selecting hyperparameters in different scenarios, such as $\\beta_0$ and $\\beta_1$?\n\n3.\tOther Types of Bias: Are there experiments demonstrating bias beyond length and style?\n\n4.\tFig. 5 x-axis: What does the x-axis represent in Figure 5? Providing clarity would help readers interpret the results more effectively.\n\n5.\tThere are several methods that can mitigate length bias, particularly in DPO. Have you considered simple baselines such as simPO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe authors effectively identify common limitations in reward models and propose a method to enhance their reliability.\n\n2.\tThe proposed method is claimed to be computationally efficient.\n\n3.\tExperimental results confirm the method's effectiveness.", "weaknesses": "1.\tAssumptions: The proposed method depends heavily on several assumptions, such as (1) decomposition of the biased reward, (2) independence of the bias characteristic, (3) sufficient density, and (4) Lipschitz continuity. These assumptions may not hold in all scenarios.\n\n2.\tDifficulty in Determining $d$: In some scenarios, determining dd is difficult. While LWR can estimate Equation 5, linear approximation may often be inaccurate, necessitating further analysis.\n\n3.\tSimple Baselines: The baselines used are relatively simple. Comparisons with more complex baselines, such as reward model engineering in reference, are needed.\n\n4.\tDemonstrations Needed: Additional examples should be provided to showcase the method's effectiveness in various situations.\n\n6.\tBias Exploration: It would be beneficial to explore other biases beyond 'length' and 'style', given that significant improvements are mostly seen in 'length bias'.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PXC5GUBhyQ", "reviewer": "Reviewer_aRiJ", "summary": "This paper proposes a calibration method for correcting bias (in particular length bias) in already trained LLM reward models (RM) on datasets. Assuming that the reward can be decomposed into an aligned reward and a bias reward term, the proposed method estimates the bias reward term using locally weighted regression (LWR) or a more robust variant (LOWESS). Then the correct reward is obtained by simply subtracting the estimated bias term. Experiments in RMBench, RM as Judge, and RM alignment show that the proposed method (RC-LWR) is able to effectively and robustly remove length bias compared with the common approach of adding a length penalty during RM training. Adding a length penalty requires a task-dependent penalty weight, whereas RC-LWR robustly adapts to different tasks and datasets.", "questions": "Typo between lines 224-225 “Based one LWR…”", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed method, RC-LWR, is post-hoc and so does not require intervention during the training of the RM. It is also fast to compute. This means you can make many post-hoc corrections and sweep over hyper parameters all without needing to retrain. This is a huge advantage over adding a length penalty or other penalty term during RM training. This means you can try to correct for as many different biases as you want to do and try out combinations of them all post-hoc.\n\n- RC-LWR is also robust to the curvature of the bias term due to using locally weighted regression, and only relies on the bias being locally close to linear.\n\n- The paper is very clear and easy to follow.", "weaknesses": "- A slight weakness is the majority focus on length bias. The experiments pretty much all focus on length bias, with one generalization task to markdown features. \n\n- Another slight weakness is that only the combination of length penalty + RC-LWR was tested. A more interesting direction would be to test a combination of two biases, and whether you can apply the method multiple times to remove multiple biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)