Disentangling Length Bias in Preference Learning via Response-Conditioned Modeling
Authors: Jianfeng Cai
Jinhua Zhu
Ruopei Sun
Yue Wang
Li Li
Wengang Zhou
Houqiang Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Preference Modeling
Bradley-Terry Model
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Reinforcement Learning from Human Feedback (RLHF) has achieved considerable success in aligning large language models (LLMs) by modeling human preferences with a learnable reward model and employing a reinforcement learning algorithm to maximize the reward model's scores. However, these reward models are susceptible to exploitation through various superficial confounding factors, with length bias emerging as a particularly significant concern. Moreover, while the pronounced impact of length bias on preference modeling suggests that LLMs possess an inherent sensitivity to length perception, our preliminary investigations reveal that fine-tuned LLMs consistently struggle to adhere to explicit length instructions. To address these two limitations, we propose a novel framework wherein the reward model explicitly differentiates between human semantic preferences and response length requirements. Specifically, we introduce a $\textbf{R}$esponse-$\textbf{c}$onditioned $\textbf{B}$radley-$\textbf{T}$erry (Rc-BT) model that enhances the model's capability in length bias mitigating and length instruction following, through training on our augmented dataset. Furthermore, we propose the Rc-RM and Rc-DPO algorithm to leverage the Rc-BT model for reward modeling and direct policy optimization (DPO) of LLMs, simultaneously mitigating length bias and promoting adherence to length instructions. Extensive experiments across various models and datasets demonstrate the effectiveness and generalizability of our approach.

Published as a conference paper at ICLR 2026
DISENTANGLING LENGTH BIAS IN PREFERENCE LEARNING VIA RESPONSE-CONDITTONED MODELING
Jianfeng Cail
Jinhua Zhu
Ruopei Sun
Yue Wang
Li
Wengang Zhout
Hougiang Lit
University of Science and Technology of China *Zhongguancun Academy (xiaobaicai, teslazhu]@maii wwwtaaillsst....... edu. en
ABSTRACT
Reinforcement Learning from Human Feedback (RLHF) has achieved consider- able success in aligning large language models (LLMs) by modeling human pref- erences with learnable reward model and employing reinforcement learning algorithm to maximize the reward model's scores. However, these reward mod- els are susceptible to exploitation through various superficial confounding factors, with length bias emerging as a particularly significant concern. Moreover, while the pronounced impact of length bias on preference modeling suggests that LLMs possess an inherent sensitivity to length perception, our preliminary investiga- tions reveal that fine-tuned LLMs consistently struggle to adhere explicit length instructions. To address these two limitations, we propose a novel framework wherein the reward model explicitly differentiates between human semantic pref- erences and response length requirements. Specifically, we introduce a Response- conditioned Bradley-Terry (Rc-BT) model that enhances the model's capability in length bias mitigating and length instruction following, through training on our augmented dataset. Furthermore, we propose the Rc-RM and Rc-DPO algorithm to leverage the Rc-BT model for reward modeling and direct policy optimiza- tion (DPÓ) of LLMs, simultaneously mitigating length bias and promoting adher- ence to length instructions. A series of experiments across various models and datasets demonstrate the effectiveness and generalizability our approach.
INTRODUCTION
Reinforcement learning from human feedback (RLHF) (Christiano et al. 2017 Ziegler et al. 2019
Ouyang et al., 2022) has revolutionized the field of natural language processing, enabling advance-
ments in areas such as conversation (Bai et al. 2022a), code generation (Poesia et al.) 2022 Jiang
ct al. 2024). complex planning (Hao et al. 2023: Zhao et al. 2024), mathematical reasoning [mani
al. 2023 Luo al. 2023), and
Within this framework. preference learning plays a pivotal
role by using aegeeealizabll model as proxy for human preferences and optimizing large language models (LLMs) based this preference model. However, significant challenge in RLHF is the phenomenon termed reward overoptimization or hacking (Gao et al. 2023). This occurs when exces- sive optimization against the preference model undermines the attainment of the true objective. As result, LLMs may learn to exploit simpler criteria such as length, bullet points, or politeness (Ramé) al. 2024), rather than more causal and nuanced indicators to achieve higher reward.
Among these reward hacking phenomena. length bias stands out as both a prevalent and challenging issues (Dubois et al. 2024a). and we identify as a representative test bed, with the expectation that our approach can be readily extended to address other spurious correlations. To mitigate length bias in the RLHF pipeline, two primary strategies have emerged, which may be complementary: The first strategy focuses on rectifying the policy optimization process through techniques such as increasing the KL penalty coefficient, applying length penalty rewards (Singhal al. 2023).
"oorreponning authors.

---

Published as a conference paper at ICLR 2026
reward clipping (Chen al., 2024b), and various hyper- rrper-parameter tuning methods. The second strategy aims to disentangle length information from quality in reward modeling. This includes length balancing, reward data augmentation, confidence-based truncation in training data (Singhal et al. 2023), and using sophisticated objectives to separate length information (Chen et al. 2024b).
Despite these advances, observe several drawbacks in these methods. First, rectification methods for policy optimization are sensitive to hyperparameter changes and foundational model variations, and prior work demonstrates that their effectiveness is limited (Singhal et al., 2023 Chen et al.) 2024b). The second category shows superior performance compared to the first; however, over- parameterization of the two branches can cause unstable optimization (Kurach et al. 2011)), and regularization methods that encourage linear irrelevance may not guarantee true independence (Shi- mony 1993). Furthermore, ignoring the biased nature of the training data and forcing an irrelevancy or orthogonality between response quality and response length is not always reasonable. For in- stance, some instruction-follovinn may indeed be factor of response quality. Finally, these methods regard length information as harmful and aim to minimize its influence on the instruction-folloiing. However, whether we could leverage these length biases for better preference modeling remains underexplored question.
datasets, such as length instruction dataset, response length
Based on the aforementioned observations, we aim to improve preference modeling by leveraging response length information and propose novel approach that allows the preference model to plicitly differentiate between human semantic intent and response length instructions. Specifically, first develop augmented length instruction dataset derived from the original preference dataset. Subsequently, we introduce a Response-conditiones Bradley- -eeryy (Rc-BT) model, which improves the model's ability mitigate length bias and follow length instructions.
Given the widespread use of paired data preference modeling in RLHF, our method can be seam- lessly integrated into both the reward modeling and policy optimization stages. First, we empirically verify our method in the reward modeling task. The results demonstrate that our method not only enhances the reward model's ability to mitigate length bias, thereby improving its alignment with human semantic quality, but also strengthens its adherence to length instructions. Furthermore, applying our method to direct preference optimization (DPO) (Rafailov al., 2024). popular alignment algorithm, further validates its effectiveness.
Our contributions are summarized as follows: (1) We propose the Response-conditioned Bradley- Terry method to mitigate length bias and enhance the model's capacity to follow length instructions. (2) We show that our method can be integrated into reward modeling and policy optimization with minimal adjustments. (3) Experimental results illustrate the superiority and generalizability of our method across multiple models and datasets.
2 RELATED WORK
In this section, we briefly introduce the background of RLHF, reward hacking and length instruction following. A more detailed version is in Appendix B
Reinforcement Learning From Human Feedback. RLHF (Christiano et al. 2017) involves train- ing a reward model to approximate human preferences and optimizing the LLMs through reinforce- ment learning (RL) (Schulman al. (017). However, this approach suffers from training instability and requires careful tuning of numerous hyperaarameters. Recent direct preference alignment meth- ods, particularly DPO (Ratailov et al. 2024), offer a more stable alternative. By reformulating the reward function, DPO eliminates the need for an online reward model, enabling robust offline pref- erence learning (Hong et al.) 2024; Chen et al. 2024a| Etthyaaaajh et al. 2024).. Our method not only enhances RLHF by improving reward model but also seamlessly integrates into the DPO framework.
Reward Hacking. RLHF vulnerable to reward hacking, where L.MSs exploit inherent biases in
the proxy preference model to achieve higher scores (Pan et al. 2022 Casper et al.] 2023 Lambert & Calandra 2023). This phenomenon persists in DPO, primarily arising from task complexity, eval- uation limitations, and biased feedback (Dubois et al. 2024b). One common form of reward hacking is length bias (Singhal et al. 2023 Park et al. 2024). wherein models favor longer responses regard- less of semantic quality. Previous attempts to address this issue include ODIN's (Chen et al. 2024b) length regularization in reward modeling: moal- training with different learning rates (Shen et al. 2023); and DPO objective modification with length penalties (Park et al. 2024). In contrast to
2

---

Published as a conference paper at ICLR 2026
these methods that suppress length information, method enables models to distinguish between semantic intent and length instructions, preserving both aspects in a balanced manner.
Length Instruction Following, Current LLMs can respond to qualitative length descriptors like "concise" or "verbose" but struggle with explicit numerical constraints such as "150 words or less" (Yuan al. 2024). Although LIFT (Yuan et al. 2024) improves length instruction adherence through specialized datasets, this approach compromises semantic quality, resulting in degraded overall performance. Our method, in contrast, achieves effective length instruction following while simultaneously enhancing semantic quality.
3 PRELIMINARY EXPLORATIONS
In this section, we conduct several preliminary investigations into the ability of LLMs to perceive and process length information, with an emphasis the reward model. Following Yuan et al. (2024), we utilize the OpenAssistant dataset (Köpf et al. 2024). which is partitioned into three subsets: D Deft ft for supervised fine-tuning (SFT), Drm for reward model (RM) training, and Deval for RM evaluation. Additionally, we employ a range of models for subsequent experimental analyses, including Qwen2- 1.5B, Qwen2.5-7B, and 3lama-3.1-88. Additional details of the training and evaluation procedures are provided in Appendix D and summarize all notations in Appendix A to aid comprehension.
3.1 LENGTH BIAS INDERD EXISTS
To verify the presence of length bias in reward models trained on Drm = ((x(), yr , }} where (f) is the prompt and response (8) is preferred to yr according to human annotations, we design three additional evaluation datasets based on Deval: Each prompt (+() within Devat is replaced with an empty prompt Te "empty prompt", forming the empty evaluation dataset Devat's Each prompt (f()) is randomly replaced with z!1i (where i + j resulting in the random evaluation dataset D Dovat's building upon the dataset Dr evat randomly shuffling the token sequences of each response /e/ yr in the preference pairs to create a random sequence evaluation dataset D
Table Evaluation results reward models on different evaluation datasets D Donat Devef and Dr Dr (Accuracy: percentage of preference pairs correctly ranked by the reward model. Consistency: percentage of preference pairs ranked identically by the reward model across different datasets.)
Accuracy (%)
Consistency (%)
Model
Devat
De
Gval
D
De (Di Devat)
Qwen2-1.5B-BB Base
63.86
64.13
64.40
58.14
89.40 (87.34)
Qwven--1.5B-Inntruutt
62.77
65.22
60.87
57.88
92.12 (90.42)
Owen2.5-7B-Bel
57.07
60.05
60.33
56.97
88.32 (90.13)
Qwen2.5-7BBInnttmee
63.04
62.23
60.60
57.12
88.60 (89.85)
1.ama-3.1-8B-BB
56.25
60.05
61.41
56.43
89.40 (85.22 224
Llama-3. -aama-3.1-8B-Im -nS--nnttttt
57.88
58.15
56.25
57.07
88.59 (87 (87.26)
Table I reports the evaluation results. Despite the semantic mismatch between responses and prompts De Désat Divugt and D the reward model nevertheless attains relatively high accu- racy on these datasets. Specifically, nearly all models achieve an accuracy exceeding 60%, which is very close to the accuracy observed on the original dataset Dowat. This finding suggests that there is significant bias in the models towards preferring ye. even when both responses ye and ye are misaligned with the prompt.
Next, we examine the consistency of evaluation results across different reward models between De (or D and Devel. with a particular focus on the influence of varying prompts on the model's preferences. The results, as shown in Table [ reveal that the models exhibit the same preference for responses in over 85% of cases, despite different prompts. This suggests that the models' preferences not primarily driven by the prompts.
For clarity of notation, we omit the superscript (+) in subsequent discussions where no ambiguity exists.
3

---

Published as a conference paper at ICLR 2026
We further plot the relationship between response length and reward score for the reward model across D Deval. Degal Devgl' and De as shown in Figure | A strong positive relationship is observed, suggesting that the reward model heavily relies response length criterion for eval- uating response quality. These findings provide compelling evidence of length bias in the reward models. More detailed analysis be found in Appendix E.
RMM Store
ROY Store
PM
Lize-1 (ellitaan
30
200
150
24
200
208
196
56
700
100
26
400
Reggomn Length
Reggonoal Length
Reqgoran Tength
Beegormn Tength
(a) Deval
(b) Deval
(c) D
(d)
D anal
Figure I: The relationships between response lengths and scores of reward models (Baseline) trained with (wven--.5B-Instruue and Llama-3. 1-0---..------tttruu evaluated on different evaluation datasets.
3.2 LENGTH BIAS IN EVALUATION DATASET
Here, aim verify whether the initial evaluation dataset Denat exhibits length bias, which could hinder its ability to accurately evaluate the true performance of the reward models. Specifically, our analysis reveals that 59.78% of the chosen responses in Devol are longer than the corresponding rejected responses. Consequently, this length bias allows the model to attain an accuracy nearing 60% merely by favoring longer responses, resulting in an unreliable evaluation of the model.
To fairly and accurately assess the semantic quality of the reward model, we reconstruct the qual- ity evaluation dataset D S from Deval through an automated process. Specifically, we em- ploy GPT-4o (Hurst et al. 2024) to transform each triplet (x, Mr) in Devat into two distinct triplets (xr w: U,.) and (r., yo. v.). These newly generated triplets adhere to the length constraints lu-l > l/d and yfl < [Iv2.. while ensuring that y and v². well as y, and v². remain semantically similar to the original Ne and yr. The detailed generation process is in Appendix [D22
The reconstructed evaluation dataset D demonstrates significant improvement in mitigating the inherent length bias Devat, thereby facilitating a more objective comparison among diverse reward models. As evidenced in Table 4 under this less biased evaluation dataset, Baseline models exhibit notably reduced performance metrics, with the majority of accuracy scores not exceeding 60%. Further experimental analysis of D Deval is detailed in Appendix E.2
3.3 LENGTH BIASED MODEL SHOW LIMITED ADHERENCE TO LENGTH INSTRUCTIONS
The reward model demonstrates length bias, indicating its ability to perceive response length. How- ever, the question remains whether it apply this ability to follow length instruction Ii, where xi length constraint + x. For simplicity, we use the same length constraint as LIFT (Yuan et al.] 2024 which specified by word num, as detailed in Appendix D.4 To validate this, construct the length evaluation dataset Di based on Dr where both responses semantically satisfy æ in Tt. but only one adheres to the length constraint. Appendix D.peendxxD33pp for more construction details.
When evaluated on D (results shown in Table 10 Baseline), all models demonstrate the accuracy close 50%, no better than chance. This demonstrates that models with length bias unable to apply this ability to follow length instructions.
We further implemented simple heuristic evaluation serve as an upper bound. Specifically, for each triplet (z)(), yo "))). we replaced the original prompt ( with a length-only instruction ("Please generate response more/less than word numm words") and ensured that y² and yr were approximately balanced terms of satisfying or violating the constraint, generating heuristic eval- uation datasets D ! We then evaluated the Baseline models, based on Qwen2.5-77 B-BB In-Irsctcct and eval-
|yl denotes the length of given response y-
4

---

Published as a conference paper at ICLR 2026
Llama3. 1--8B-nnttruc,, on this dataset assess their compliance with length instructions. The sults as shown in 2 The results reveal that even under such simplified conditions, the Baseline models exhibit negligible ability to follow explicit length instructions.
Table 2: Evaluation results of reward models heuristic evaluation datasets D D
eval
Metrics
Qwen2.5-7B-Instruct Llama-3. 1-8--nrrrutt
heuristic (%)
56.73
54.12
This finding suggests that reward models unconsciously acquire length bias during preference learn- ing without explicit awareness length as a measurable attribute. Motivated by this observation, propose the following hypothesis:
Hypothesis 1. Explicit length- length-instrucn instruction learning enables language models to form clearer per- ception of target response length, thereby reducing undesired length bias.
Based on this, propose incorporating length constraints into the training process transform implicit length bias into explicit length understanding.
3.4 LENGTH INSTRUCTIONS ARE EASILY LEARNED
The most straightforward approach to explicit length instruction learning would be training directly on data formatted as {zr. Me. Yr }. To examine the effectiveness of this intuitive format, we ex- tend LIFT Yuaa et al. 2024) to create LIFT-plus by incorporating minimum length constraints ("or more" length instruction) alongside its original maximum length constraints ("or less" length instruction] We then construct three variant datasets: LIFT-plus, feverse which reverses the prefer- ence order between chosen and rejected responses based on length constraints; LIFT-plus which preserves the original preference ordering: and LIFT-plus emply where substitutes in 24 with an empty prompt Ze while maintaining the preference order, examines the impact of semantic prompts length instruction following. The complete construction process is in Appendix D.4
Empirical results presented in Table (Appendixx E.3 reveal that while models demonstrate notably higher accuracy on DÍ Divall they achieve diminished accuracy D Degat compared to the Baseline in Table [ This performance disparity suggests that reward models prioritize length instruction compliance the expense of semantic understanding. Furthermore, accuracy trajectories across training steps (Figure 5) Appendix E.3) indicates similar pattern: accuracy on Di exhibits rapid improvement, while accuracy on Dicai shows initial improvement followed by deterioration.
These results highlight the limitations of the { Ne: 3r format, which encourages overfitting to length instructions the expense of semantic capability, indicating the need for better methods.
DASPONSE-CONDITIONN
MODELING
In this section, we present our Response-conditioned Bradley-Terry model, with the overall frame- work depicted in Figure 2 In Section 4.1 we briefly review the Bradley-Terry (BT) model. In Section 4.2 we introduce the cesponse-conditioonl Bradley-Terry (Rc-BT) mode presenting its algorithmic details and mathematical formulation. Finally, we demonstrate the implementation in both Reward Model (Section 4.3) and DPO (Section 4.4) through mathematical derivations.
4.1 PRELIMINARY: BRADLEY-TERES MODEI.
In RLHF, given a human request supervised fine-tuned LLM denoted SFT is prompted with x to sample response pairs y1, 1/2 Human feedback on this pair is then collected and represented as yr- To model human response preferences, the Bradley-Tery
Ye Vr. where Ye is preferred
p During the improvement over LIFT, the training dataset always includes the original dataset Dym to a halanced exposure to both original and length-instraction formatted data.
*hhle primarily developed for length hias mitigation and length instruction adherence, the Rc-BT frame- work is theoretically applicable diverse instruction followiin--oloowingg tasks.
5

---

Published as a conference paper at ICLR 2026
constraint
longth constraint
Lenger constroint
Standand prefeerecce pain
LIFT preference pair
Chesen preference pain
Refected preference pair
Rc preference pair
(a)
(b)
(C)
Figure 2: To illustrate the distinct data formats across different methods, we present the maximum length instruction case: (a) The conventional RLHF (Ouyang al., 2022) with standard preference pair (x, ye yv) (b) LIFT with augmented format (2), Yr. Be); and (c) Our method (Rc-BT) with two preference pairs (x,, at. yo) and (af. H, yr). The term length constraint refers to maximum allowable length of the response specified in the length instruction 21, at or u?. The black dashed lines indicate the data utilized by each method.
(BT) (Bradley & Terry 1952) model is a prevalent choice (Ouyang et al., 2022). It models the underlining human preference distribution p" by assuming an underlying true reward model r*:
exp (r (x. 9c))
(ye > yz) exp(r"(<,, (1), + (xpp("((x (+))
(1)
where is not accessible or easily defined by several rules. Therefore, people try to parameterize reward model ra and optimize the parameters via maximum likelihood or leverage analytical mapping from reward function to optimal policy, to directly optimize LLM policy (Dubey et al. 2024). Given high quality preference dataset {(x, yhe, Ur)), the BT model has demonstrated sig- nificant efficacy in various preference modeling and policy optimization tasks (Ji et al. 2023)
4.2
MAALEE--TERY MODEL.
Response preference modeling employs Equ. I as a framework for response differentiation. How- ever, fundamental challenge arises in scenarios involving prompts with multiple instructions, where responses may only partially satisfy the given criteria. As previously discussed in Section 3.4 the conventional { Ne: Jr data format exhibits tendency to overfit to length instructions, consequently compromising the model's semantic comprehension capabilities. This limitation rep- resents inherent constraint in response preference modeling.
Therefore, we propose a response response-conditiones modeling framework, as illustrated in Figure Our approach proceeds as follows: given original chosen response Ve and prompt æ, construct length augmented instruction zj by incorporating a length constraint with the prompt æ, ensuring that Ye violates this constraint. We then formulate preference pair (a,, at Vc). where (I, is considered preferable to (z) Mc). This preference is then modeled using the BT model as follow:
exp (r" yo)) p' (a of Hue) (xp(r"(z, exp(r* (x) ve))
Similarly, for a given rejected response y, and original prompt I, we construct a length augmented instruction 27 such that y, satisfies the specified length constraint. We then formulate preference pair (x x, Mr), where (27 yr) is considered preferable to (z, wr). This preference structure is similarly modeled using the BT formulation:
exp (r (af v..)
(0) x/p-) (xp exp(r=
The final response-coe conponiinnod modeling preference dataset is defined as DRe ((x, 2/ va) U {(af, x. }. Through maximum likelihood estimation, we derive the cesponse-conditioned BT (Rc-BT) modeling objective function follows:

---

Published as a conference paper at ICLR 2026
Lite
E
logp" (x > xb(be)]
E (0g ....)--R.
Dog (2î > zlyx)].
(4)
The Rc-BT facilitates explicit comparisons between i and its length-augmented variants (x) and af enabling the model to systematically perceive response lengths and thus mitigating implicit length bias through explicit length understanding. To better contrast the BT model with our approach, Table 3 provides a detailed comparison between BT and our method Rc-BT.
Table 3: Comparison of Preference Modeling Methods (BT vs Rc-BT).
Method
Motivation
Capture preferences between responses
Cupture preferences between prompts (Zoz Br.
format
Mes D+) exply (k.wel) expp 13
Modeling oquation
(Be (-)
p (Zt ox)
eapLE" (eepp) #exp!" (Ry.87)
4.3
RESPONSE-CORE CONDITIONED REWARD MODEL
response preference modeling. reward model is initialized from SFT and augmented with linear projection layer that transforms the complete sequence representation into a scalar value (2 y). Given a preference dataset D {(x, yw }r]}. the reward model is optimized through maximum likelihood estimation according to Eqn.
Cry (D)
Dog (ro((, (o, v-))).
(5)
(s. We
where a--) is the sigmoid function. In response-conditione modeling, the architectural structure of the parameterized reward model re remains unchanged while the data format is transformed from prompt-conditioned to response-conditione. Similar to Eqn. [5] the model is optimized through maximum likelihood estimation based on Eqn. Jusing dataset DRo:
fro (DRc) = (sran) Doo
Yogo(rs(z; Ve) (o(att (.))l (e) (n
E
(rgo(ro(ui, vx) ro( Vr vr.))l
(6)
where À is used to balance the relative contribution of the pairs (I, at Mo) and (2 x, v.).
4.4 SEEPONSE-CCONITIONN DIRECT PREFERENCE OPTIMIZATION
In response preference modeling, DPO derives an alternative formulation from Eqn. g where the reward is expressed as function of the optimal policy. By substituting this formulation into the reward optimization objective specified in Eqn. [ we obtaín direct optimization approach for
policy training. Specifically, the policy can be optimized on dataset D using the following objective
nee (x)(
Ta (yr (x)
CDPo(To: Rnd)
[log o log
8 log
Terr(y-1z) (x) }}-
(7)
Tedt The (Vel=)
For the response-conditioned modeling, we follow an analogous derivation process to DPO. Specif- ically, we begin with the modified RI. objective (Eqn. T0). derive the reward model expression in terms of the optimal policy, and subsequently incorporate into the Re-BT modeling function (Eqn. 4 This derivation yields the Response-conditioned DPO (Rc-DPO) objective function:
To ye) B log
mo( Ue). }]
RC ( and) -
E (xrx) Me DDD
[log ( log
Tiz The (2,, yc)
Tiet ((z) Mo)
(8)
(1². 3r))
To yr) Teat (x, yr
[log ouga((3 (. log
3 log
Tret ( (17, yr)
For detailed and complete derivation, please refer to Appendix
For the remainder of this paper, unless explicitly stated otherwise, every instance of denotes mo x)).

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
This section presents experiments and ablation studies to demonstrate the effectiveness and vali- date the design of our Rc-BT' framework. We primarily demonstrate the results of our method in mitigating length bias, while its efficacy in length instruction following can be found in Appendix F
5.1 EXPERIMENTAL SETTINGS
Dataset and Models. Consistent with Section 3 for the dataset, we use Daft for SFT finetune the Base models. For RM and DPO, Rc-BT generates an augmented dataset D derived from Dym- Both reward models and DPO models are trained on the combined dataset Drm 'Dee referred to as Rc-RM and Rc-DPO, respectively. We employ three pretrained models as our base models: Qwen2-1.55 1.5B Qwen2.5-7B. and Llama-3.1-8F Gemma-2-90 2-BB and Qwen2.5-14B.
Training Details. For Rc-RM training, the learning rate is set to x 10 -5 followed by cosine learning rate schedule with an initial warmup of 10 steps and a batch size of 64. Each experiment is trained for 5 epochs. For Re-DPO training. the settings are identical to RM training, except the learning rate is x 10 All experiments are implemented based on DeepSpeed (Yao et al. 2023) and Transformers WWolf et al. 2020), and conducted on machine with 8 NVIDIA A100 GPUs.
Compared Methods and Evaluation. For RM evaluation, we assess the performance across meth- ods using two primary metrics: Quality Eval Acc and Length Eval Acc, which are accuracy on D Deval and D evat respectively. The former metric evaluates semantic quality with a focus length bias mitigation, while the latter measures the effectiveness of length instruction adherence.
For DPO evaluation, leveraging recent advancements in automated assessment approaches (Zheng et al. 2023 Dubois et al., 2024b), we employ a model-based evaluation framework to systemati- cally assess the quality of generated responses. First, construct the AlpacaEval-LI less and A AlpacaEval-LL--plu--moo
benchmarks, enhanced versions of AlpacaEval (Dubois et al. 2024b), designed to evaluate the model's ability to follow "or less" and "or more" length instructions, spectively. Then, we evaluate each DPO model's performance by comparing it with its correspond- ing SFT/Instruct model using two complementary metrics. The first metric, Length Acc, quantifies the model's 'ss adherence to length instructions by measuring how well the generated responses con- form to the specified length constraints in 2/- The second metric assesses the model's semantic instruction following capability through two comparative win ratios: Length Win Ratio represents the proportion of cases where the model generates semantically superior responses when prompted with length-augmented instructions (zz), while Quality Win Rutio measures the proportion of cases where the model demonstrates better semantic quality with original prompts (x). For comprehensive evaluation details, please refer Appendix F.I
We refer to the model trained on the D Drm as Baseline, For RM, we incorporate ODIN (Chen et al,, 2024b) and R-DA (Singhal et al. 2023) comparison method, with particular emphasis its Quality Eval Acc, owing to its demonstrated effectiveness in mitigating length bias. In addition, we compare our method with LIFT-plus on the Length Eval Acd For DPO, we include LIFT-plus. R-DPO (Park et al., 2024). Dr. DPO (Wu et al. 2024). SimPO (Meng et al. (0024), and CPO (Xu al. 2024a) as comparison methods. Unless otherwise specified, all compared methods are trained using the recommended settings from their papers.
5.2 THE RESULTS OF REWARD MODELS
Re-RM significantly alleviates length bias. The Quality Eval results in Table demon- strate the effectiveness of Rc-RM in mitigating length bias. Specifically, Rc-RM outperforms Base- line, R-DA, and ODIN across all models. Notably, on Qwen2-1 1.502-1 Base, Rc-RM exceeds Baseline by 10.41% and outperforms ODIN by 13.43%, respectively,
1----...-1--II-tucttt it surpasses Baseline by 16.85% and ODIN by 11.54%, and on Gemma-2-9B-ii. Rc-RM surpasses Baseline by 10.11% and ODIN by 7.71%. Additionally, Qwen2.5- 14B-Instruut also benefits significantly from Rc-RM It achieves a remarkable 81.70% Quality Eval Acc, substantially higher than ODIN, R-DA,
6 We also conduct experiments using the Qwen2.5-1.5B, which shown in Appendix F3
As discussed in Section 3 LIFT-plus harms semantic quality and does not mitigate length bias, we exclude it from the Quality Eval Acc comparison.
8

---

Published as a conference paper at ICLR 2026
Table 4: Quality Eval Acc of different reward models on quality (De evaluation datasets.
Model
Baseline
ODIN
R-DA
Rc-RM
Qwen2-1.55
59.14
56.12
60.17
69.55
60.75
63.56
61.27
71.47
Owen2.5-7B-Bass
54.26
60.90
63.46
70.74
Qwen2.5-7B-Instrucct
59.31
67.55
66.34
73.07
1--ma-3.1-8B-B
59.04
60.37
65.17
70.51
55.59
60,90
60.78
72.44
Gemma-2-9B-it Qwen2.5-14B-Instrun
53.45
55.85
55.21
63.56
65.57
76.22
75.14
81.70
and the Baseline, validating Rc-RM's robustness under model scaling. Overall, these results af- firm the generalizability and scalability of our Re-RM framework across diverse model families and sizes, reinforcing its effectiveness in mitigating length-bias.
We conducted further experiments by training a policy using the trained reward models via PPO. We trained the Qwen2-1.5B --nsrruu-- policy using the reward models of (iren2-1.5B-Instiie and 2wee2.5--B-Innsteee
from Table 4 The evaluation was carried out on AlpacaEval following the same metrics used for evaluating the DPO models. The results in Table [Tp (Appendix) F.2). It can be observed that the policy trained using Rc-RM exhibits high semantic quality, which significantly surpasses other competing methods, further demonstrating the effectiveness of our method.
NAI Squte Dilleennie
LiFTal
CEnT
833 Soore
Reggomer Lmugh Roage
Preepmee Langth Range
Respass Langth Ronge
Respumor Lengl Ronge
(a) Qwcn2-1,,8--nstrree (b) Llama-3. 1orma--..--B---e I-8-Insttctt (c) 1w5n2-1.5B-Instruct (d)
Figure 3: (a), (b): Analysis of reward scores across models on Dmm as function of response length, with smaller changes indicating reduced length bias. (c), (d): Comparison of score differences between LIFT-plus and Re-RM on Dinla under varying word_num constraints. An ideal reward model should show an initial increase in score difference followed by return its initial value.
Furthermore, for Devat, we rewrite each (1)(), (0) into multiple length-varied pairs Den {(x(9) ve ( (1,2,3), where lyc ( ly (1.2) (,o,,,)) while maintaining the semantic
consistency between y! (i) and ye (1-) Then we evaluate each reward model on Deni with the results shown in Figure where the reward model with less length bias exhibits smaller slopes in its scores. The results indicate that both Baseline and LIFT-plus show an upward trend in their scores, while ODIN fluctuates due varying degrees of length penalty. Rc-RM, however, demonstrates superior stability, clearly highlighting its effectiveness in mitigating length bias.
5.3 THE RESULTS OF DPO MODELS
Re-DPO effectively reduces response length and improves quality. Table 5 highlights the ef- fectiveness of Rc-DPO in mitigating length bias and enhancing response quality on AlpacaEval. Specifically, LIFT-plus negatively impacts semantic quality, leading
notable decline Qual- ity Win Ratio. Compared to R-DPO, Dr. DPO, SimPO, and CPO, on Qwen2.5-7B-Ban achieves a Quality Win Ratio 45.39%, outperforming the Baseline by 11.85% and R-DPO by 4.99%, while reducing the response length from 517.30 to 208.42. Similarly, Llama-3. 1-B- Instruct, Rc-DPO attains the highest Quality Win Ratio of 64.34%, surpassing SimPO by 6.21% and Dr. DPO by 12.16%. These results demonstrate that Re-DPO consistently achieves the best trade-off between response quality and length control across both Base and Instruct models, further validating the effectiveness and generalizability of our approach.
Rc-DPO

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation results of different DPO models on AlpacaEval (Duoois et al.
2024b).
Qwen2.5-7B-Bal
Metrics
Baseline
LIPT-plus
R-DPO
Dr.DPO
SimPO
CPO
Rc-DPO
Quality Win Ratio (%)
33.54 517.30
31.67
40.40
39.74
41.26
40.07
45.39
Response Length
184.17
583.18
311.49
286.54
254.86
208 42
Qwen2.5-7B IB-Inrtuett
Metrics
Baseline
LIFT- plus
R-DPO
Dr.DPO
SimPO
CPO
Rc-DPO
Quality Win Ratio (%) Respouse Length
28.43 261.32
25.69 195.80
34.16
36.12
37.16
35.72
44.63
235.39
247.19
281.26
257.95
228.24
Llama-3.1-8B-
Metrics
Baseline
LIFT-plus
R-DPO
Dr.PPO
SimPO
CPO
Re-DPO
Quality Win Ratio (%)
46.30 435.98
40.15 157.80
49.13
51.12
54.38
50.86
58.10
Response Length
465.72
347.29
309.86
288.64
202.01
Llama-33 1.-ma-3.1-8B--. --I-ruuut
Metries
Baseline
LIFT-plus
R-DPO
Dr.DPO
SimPO
CPO
Re-DPO
Quality Win Ratio (%)
42.52 247.74
47.88 153.77
42.64
52.18 229.17
58.13
54.67
64.34
Response Length
215.82
218.46
244.04
204.77
Table 6: Evaluation results of Rc-RM w/o Di and w/o De on D
evol and D
evol.
Qwen2- 1.5B-Bat
Qwen2- Instruut
Metrics
w/o Di
w/o D
w/o Di
w/o Di
Quality Eval Ace (%)
58.78
55.59
59.31 45.51
57.71 44.87
Length Eval Ace (%)
45.83
36.22
Llama-3.1-880 --sB-B
Llama-3.1-8B-Instre
Metrics
w/o Do
w/o D
w/o Dis
w/o Ro
Quality Eval Ace (%)
50.27
50.53
58.51
52.13
Length Eval Acc (%)
44.87
32.69
37.18
33.65
5.4 ABLATION STUDIES
Due the high of evaluating the DPO model, we focus on the reward model for ablation studies. For convenience, we denote x( } as DE, and {(x x, 31) } as D We report the key ablation results this section, with extended experiments provided in Appendix F.7
Dí and Di are complementary. We conduct ablation experiments by training with Drm U D (w/o D and Drm UD' (w/o Di.),, with the results shown in Table 6 As observed, when only Di or Dr. is used, the Quality Eval Acc of Rc-RM drops significantly. nearing the performance of Baseline Table 4 while its Length Eval Ace hovers around 50%, failing to learn length instruc- tions. Subsequently, we prepend a length constraint to each x in D enat. forming where ye satisfies
evgl shown in Figure g Specifically, Figure a(a) indicates that for the reward models trained with Drm. U D' the scores of (I1, ye) are consistently lower than those of the original (æ, ye), despite ye's 's's length satisfies zi. A similar phenomenon is observed in Drm U'Dr (Figure (b). Therefore, combining Dr. and Dr. is essential for preventing the model from developing new biases.
the length constraint, resulting in De.!
cval. The evaluation results both D cval and D
CONCLUSION
We propose Rc-BT, a method that separates human semantic intent from length instructions to mit- igate length bias while retaining length sensitivity, and provide mathematical extensions to RM and DPO. Extensive experiments demonstrate the effectiveness of Re-BT in reducing length bias and improving length instruction following, with potential for broader bias mitigation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China under Contract 623B2097, the Youth Innovation Promotion Association CAS. It was also supported by Merchants Union Consumer Finance Company Limited, and the GPU cluster built by MCC Lab of USTC & the Supercomputing Center of USTC
REPRODUCIBILIIY STATEMENT
We have made significant efforts ensure the reproducibility of our work. For theoretical contribu- tions, Section and A provides clear explanations and complete mathematical derivations of our proposed Re-BT framework. Regarding the experimental setup, Section 3 and [S] Appen- contain
dices (and]] contaiin a full description of the data processing procedures used to construct training and evaluation datasets. Together, these materials are intended facilitate accurate reproduction and verification of our results.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 08774, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 058662, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08077 2022b.
Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Loredana Caruccio. Stefano Cirillo, Giuseppe Polese, Giandomenico Solimando, Shanmugam Sun- daramurthy, and Genoveffa Tortora. Claude 2.0 large language model: Tackling a real-world classification problem with a new iterative prompt engineering approach. Intelligent Systems with Applications, 21:200336, 2024.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomek Korbak, David Lindner, Pedro Freire, Tony Tong Wang. Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip J.K. Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell Open problems and funda- mental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: http//ppnneeeeww.eee / Id=bx24kpJJEb, Survey Certification, Featured Certification.
forum?
Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. Noise contrastive align- ment of language models with explicit rewards. arXiv preprint arXiv:2402.05369, 2024a.
Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. In Forty-first International Conference on Machine Learning. 2024b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See http:://vicuna Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
11

---

Published as a conference paper at ICLR 2026
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators, arXiv preprint arXiv:2404.04475, 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Perey S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Yang Gu, Yuhu Cheng, C. L. Philip Chen, and Xuesong Wang. Proximal policy optimization with policy feedback. IEEE Transactions on Systems, Man, and Cybernetcs:: Systems, 52(7))4600- 4610, 2022. doi: 10.1 D0.119//TMC.22211338880
Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154-8173, Singapore, December 2023. Association for Computational Linguis- URLhttpa www.hlonnnooooy....o.ooooooooooe org 202.3.
tics. doi: 10.1 10.18653/v1/200 1ttp///2....m2mmma...a emn p nn. 507
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without refer- ence model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170-11111 11889, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. In Proceedings of the 61st Annual Meeting the Association for Computational Linguistics (Volume 5: Industry Track), pp. 37-42, 2023.
Hamish Ivison. Yizhong Wang. Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, and Hannanch Hajishirzi. Unpacking DPO and PPO: Disentangling best practices for learning from preference feedback. In The eiihth- Annual Conference Neural Information Processing Systems, 2024. URL htRpstts::/ /ttp:///pperevveeee net. formm? 1d= JMBWTlaaa
Jaming Ji, Mickel Liu, Josef Dai, Xuchai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via human- prefannnre dataset. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 24678- 24704. Curran Associates, Inc., 2023.
Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wen- pin Jiao. Self self-planning code generation with large language models. ACM Trans. Softw. Eng. Methodol, 33(7), September 2024. ISSN 1049-331X. doi: 10.1145/3672456. URL https://dof /ttp://....rr/0011111466/6622
12

---

Published as a conference paper at ICLR 2026
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations convrssaions-dmmorattinnngg large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to Segueence-o--equencee reinforcement learning. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1777-1788, Melbourne, Australia, July 2018. Association for Com- putational Linguistics. doi: 10.188633/1//18-11 1165. URL httpt::////// //as:://cllls a antho ogy. ogy.org/
P18-1165
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly, The GAN
landscape: Losses, architectures, regularization, and normalization, 2019. URL https://
@pw.reeveew net wwwp:/eee/oooomidc-mmmmm http://rrumm...kkkreeeeeoaoro
Nathan Lambert and Roberto Calandra. The alignment ceiling: Objective mismatch in reinforcement learning from human feedback. arXiv preprint urXiv:2311.00163 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint A4Iv:2403.13787 2024.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.0026: 00269900767,..... 2023.
Fei Liu al. Learning to summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Do Xuan Long, Hai Nguyen Ngoc, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F Chen, and Min-Yen Kan. Lims are biased towards output formats! systematically evaluating and mitigating output format bias of llms. arXiv preprint arxiv:2408. 248800640,0 2024.
Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, and Xing Sun. Eliminating biased length reliance of direct preference optimization via down-sampled kl divergence. arXiv preprint arXiv:2406. 10957, 2024.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qing- wei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint 2tt.:2308.0988.3 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122255, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang. Jana Thomp- son, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086-2105. Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.11863//11/0000 findings- 165. URL https: / antho wgw..org///22.. ndi http://00222..iinnnnsneeeeee.ee 1657
13

---

Published as a conference paper at ICLR 2026
Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. Synchromesh: Reliable code generation from pre-trained language models. In Interna- tional Conference Learning Representations, 2022. URL https::/ net/
forum? d=Kmt 9797343e
James Queeney, Yannis Paschalidis, and Christos G Cassandras. Generalized prox- imal policy optimization with sample reuse. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp. 11909-11919. Curran Associates, Inc., 2021. URL httpp::/ ngs. www.ips.crc.ccc.cc wwtppppr.ppppp...p/peee www.ffe/ppppe////00//// BLcc/BBcalaaab044000033553322222 Papp:e
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Ramé, Nino Vieillard. Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401. 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. arXiv preprint arXiv:2310.05199 2023.
Solomon Eyal Shimony. The role of relevance in explanation i: Irrelevance as statistical indepen- dence. International Journal of Approximate Reasoning, 8(4):281-324, 1993.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXi::2310.037/6 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv: 1805, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing:
system demonstrations, pp. 38-45, 2020.
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. arXiv preprint M:XXi:2407.073880 2024.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference Machine Learning, 2024a. URL
New. net. forum? i i 1wkkookk
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation, 2024b. URL htttss://aaxiii>> wrg. abs/2405
08417
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. In Forty-first International Conference on Machine Learning, 2024c. URL /tsttpp://openeviee..o
14

---

Published as a conference paper at ICLR 2026
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Am- mar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. Deepspeed- chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv preprint arXiv:2308.01320, 2023.
Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu, Following length constraints instructions. arXiv preprint arXiv:2406.1 17744, 2024.
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 04446595-46623 2023.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 19t.:19999008933,, 2019.
15

---

Published as a conference paper at ICLR 2026
A
SUMMARY OF NOTATIONS
Table provides summary of the notations used throughout this paper. The table is intended serve quick reference help readers understand the mathematical symbols and datasets introduced in the paper.
Table 7: List of notations and their definitions used throughout the paper
Notations
Description
The original prompt.
Te
The "empty prompt" string.
Prompt z augmented with length constraint.
2j
Variant of with specific length constraint.
Fe
Chosen prompt for y.
Ty
Rejected prompt for y- Chosen response for œ. Semantically equivalent rewrite yo. Rejected response for Semantically equivalent rewrite of yr- Training Dataset used for SFT.
Ve
ya
y
y
Dett
Drm
Training Dataset used for reward modeling and DPO. Augmented training dataset for Re-BT based on Drm. Subset of Dte containing Ve- Subset of Die containing Yr-
DRC
Dio
Denat
Original evaluation dataset for the reward model.
De Denat
Derived from Denat by replacing prompts with xe (empty evaluation dataset).
Dr
Derived from Devat by randomly matching prompts with responses from other samples (random evaluation dataset). Evaluation dataset for assessing semantic quality of the reward model (quality evaluation dataset).
eval
D
eval
D
Evaluation dataset for assessing length-following ability of the reward model (length evaluation dataset).
enal
TΦ
Reward model with parameters d.
USFT
Initialization LLM.
T
LLM with parameters 0.
Tref
Reference model used in GRPO/PPO/DPO.
B RELATED WORK
Reinforcement Learning from Human Feedback. Reinforcement Learning from Human Feed- back (RLHF) (Christiano al. 2017 Ziegler al.] (2019) has established itself as dominant paradigm for aligning LLMs with human preferences (Achiam et al. 2023) Team et al. [2023 Caruc-
cio al. 2024; Yang et al. 2024 Dubey et al. 2024). At core, RLHF trains models by optimizing
reward signals derived from human feedback, typically collected through preference comparisons or explicit ratings (Bai et al. 2022b Lee al., (023). This approach has demonstrated signifi-
cant efficacy in refining LLM behavior to align with human expectations across diverse complex
tasks (Kreutzer et al. 2018 Liu al. 2020 Ziegler et al. 2019 Ouyang et al. 2022).
However, RLHF necessitates training a reward model (RM) to approximate human preferences, followed by LLM alignment through reinforcement learning (RL) algorithms, notably Proximal al. 2017). This process presents significant challenges
Policy Optimization (PPO) (Schulman Gu Gu et al. (2022). Direct preference alignment methods, in
and stability issues (Queeney et al. 2021 contrast, provide more robust training approach (Ivison al. 2024 Hong et al. 2024 Xu et al.
2024b). Among these methods, direct preference optimization (DPO) (Rafailov et al.
2024 has
emerged leading technique, inspiring various derivative algorithms (Hong et al.
2024 Chen
16

---

Published as a conference paper at ICLR 2026
et al. 2024a Ethayarajh al., 2024). By reformulating the RLHF reward function, DPO eliminates
both the need for an online reward model and the instabilities inherent in RL algorithms, enabling stable offline preference learning.
Our method enhances RLHF by improving reward model robustness and seamlessly integrates into DPO, enabling more effective LLM training.
Reward Hacking.
Despite its promising performance, RLHF is vulnerable to reward hacking
the over-optimization of the reward model (Skalse et al., 2022 Pan et al. 2022 Casper al. 2023
Lambert & Calandra 2023). This phenomenon occurs when the policy exploits inherent biases in the reward model to maximize rewards without achieving intended objectives. Similar exploitation
patterns have been observed in oPo-trrined LLMs (Lambert et al. [2024] Xu et al. (024c).. The root
causes are multifaceted: task complexity and subjectivity (Casper al. 2023). evaluation criteria limitations (Parrish et al. 2022), and evaluator qualification constraints (Skalse et al. 2022). Conse-
quently, human preference data exhibit biases and inconsistencies (Dubois et al. 2024b), and reward
models struggle with preference approximation and out-of-distribution (OOD) generalization.
One of the most common forms reward hacking length bias (Shen et al. 2023 Singhal et al.) 2023 Park et al. 2024 Chen et al. 2024b), where the reward models tend to favor longer responses, assigning higher reward scores to longer responses even if their semantic contents are not of high quality. This tendeney is largely driven by human evaluators' preference for longer answers, which is casily exploited by the LLMs. As a result, LLMs often generate unnecessarily lengthy replies to appear more detailed or better formatted, even when the actual quality remains unchanged. To address this issue, several methods have been proposed. ODIN (Chen et al., 2024b) adds multi- ple length regularization terms the RM's training objective, which helps to distinguish response quality from length and effectively controls the LLM's dependency on response length. [Shen et al.] (002) involve using dual-reward model and different sets of learning rate hyperparameters, allow- ing different part of the reward model to learn distinct paradigms, thereby removing length bias from final reward scores. Length Regularized DPO (R-DPO) (Park al. 2024) modify the DPO training objective by adding length penalty term to prevent length bias exploitation.
However, the above methods assume that the model's sensitivity to length information is detrimental and, therefore, seek to eliminate the model's perception of length information. In contrast, our approach enables the model to explicitly distinguish between human semantic intent and length instruction, preserving its ability to perceive response length rather than completely eliminating the perception of length information.
Length Instruction Following. Current state-of-the-art (SOTA) LLMs, both open-source and closed-source, demonstrate a certain degree of implicit ability to follow length instructions YYuan al. 2024). For example, adding terms like "concise" "verbose" can influence the length of the model's 's outputs. However, when it comes to explicit length instructions, such as "Answer the following instruction using 150 words less.", these models often fail to adhere effectively. To address this, LIFT (Yuan et al., 2024) enhances the model's length instruction following capability by constructing explicit length instruction preference dataset. However, the dataset used by LIFT is built at the expense of semantic quality, resulting in a degradation of the model's 's output quality. In contrast, our model not only effectively learns length instructions but also maintains, even in many cases improves, the semantic quality of the model's 's outputs.
C DERIVATION OF THE DPO OBJECTIVE UNDER THE RESPONSE-COS CONDISTONNDDDD BRADLEY-TERRY MODEL
Derivation of RL Fine-Tuning Objective. During the RL phase of traditional RLHF, the trained reward model r3 serves as the feedback mechanism for policy optimization. The standard RL opti- mization objective is formulated as:
max E. yow.oooll (rolz, v)] fI. [TO(Y) x) || Trer(yy | x)]]
under the reference policy Thef and the parametrized policy To. In the response-condition sce- nario, where responses predetermined and guide prompt selection, the KI Kivdivereeece constraint
17

---

Published as a conference paper at ICLR 2026
in Eqn. 9 requires modification. Accordingly, reformulate the response-conditione RL opti- mization objective as:
max By RT [ro(1, v)] - BD [To v) [er [v]
(10)
70
For simplicity, we unify the notation of DRc- {(x, a/ ye)) U {(f x, y-)) as {Fre: Tr, y)). Next, similar to the derivation in DPO (Park et al. 2024), we can derive the partition function Z(y) from Eqn. 10
2 Σ Mee(( y) exp (gro(a.v))
(11)
Since the partition function is a function only of the response y and the reference policy Tres and does not depend on the prompt the parametrized policy my be optimized, we can reorganize Eqn. to obtain the following objective:
min EYnD (x)r)(z) log To(xly) log Z(v)
min EywD (KL( x (*(l))) log (v)),, me
. (x:ly)
me
(12)
Z(v) Teer( exp exp (Fra(e.w)).
" *" y)
(13)
1= By using Gibbs' inequality, the KL divergence is minimized to 0 if and only the two distributions are identical. Therefore, obtain the optimal solution:
where >"xx y) is a valid probability distribution which x² (x y) and <"<x
()) exp (Frale.)))
( v) T=" (x vv)
7(y) Teer
(14)
Finally, by taking the logarithm on both sides and performing some basic algebraic operations, we express the reward model ro(#, y) through its corresponding optimal policy r* y):
Tx(x y) t v)
ro(2,y) = 3 log
+ B log Z(v).
(15)
Derivation of Response-conditionee DPO Objective. Following Section 4.2 the Response- conditioned Bradley-Terry model is formulated as:
exp (r* (Ke, 91) ( > x,l8) exp(r*(Fe: v)) exp(r* exp((*(xrr,))
(16)
In the above derivation, demonstrated that the optimal reward r" y), parameterized as Ta y), can be expressed in term of its corresponding optimal policy n" (x y). By substitut-
ing Eqn. 15 into Eqn. a6d preforming simple simplifications, obtain:
16
18

---

Published as a conference paper at ICLR 2026
exp (3log (x,lv)) marlzalu)
+ B log 2(v)
D¹ (Ie xrl11)
exp
log a (2el)) _a((_ll)
llg (v)) + exp gog x (x, [y] T,((1,,1)))
B log 2
+ exp
(3log (x(lvv) (a((a>(M)
8 log x (m.1u) (ar
+ (3log ((y) 3 log (())
1+ exp
Boog (3 Tar(arlu)
$ log )ar(F.))))
x² (mcll))
c0($1gg =0 =o (
n²' (xcll) Blog (
n (2(xl1y)
(17)
Tee((Zr
Since we cannot directly access (<]y), we reparameterize using Bayes' theorem for conditional probability decomposition, transforming Eqn. 17 into joint probability formulation follows:
T (2ely)
x² (xrly)
pr xrly)-
log
B log
Ner (2ely)
Teet x (8)
(xr.:3)
(xr
B log
x (3))
log
* (3r)
Tar( Tar(y))
Tnr(3/)
π (Fe, y)
tT (Iir 8)
B log
| 3log
(18)
Teed (Zer y)
(er 8)
Finally, we derive the loss function for Rc-DPO for parameterized policy using the response- conditioned preference dataset Dr. = ((Tee Fr,
y)} wr,y))
by applying maximum likelihood estimation:
Ric C Tres) - -E(e,,ee. 9)) [log a log Th (Zer 3) 3 log
To Te(
(19)
Trer(Tee y) Trer(arr
D THE IMPLEMENTATION DETAILS OF PRELIMINARY EXPLORATIONS
D.1 TRAINING DATASET AND MODELS
Training Dataset. Similar to the data processing approach used by Yuan et al. (2024). we extract the first turn of English conversations from the OpenAssistant dataset (Köpf et al., (024)) and use it as our complete dataset. Based on their human annotated ranking, we label rank 0 as "chosen" (wo) and rank 1 as "rejected" (yr). resulting in the complete dataset D, that each example is a triple (6)
(z) Me yr ()))
For training of different models and subsequent experimental analysis, we first divide D into a train- ing dataset Dirain. which contains 90% of the data for model training, and an evaluation dataset Deval, comprising the remaining 10% for model evaluation. Next, we further divide Derain into two subsets: a supervised fine-tuning (SFT) training dataset Daft. containing 30% of Dtrain and reward model (RM) training dataset Drm- containing the remaining 70% of Dirain- For all reward models, we incorporate Dym into the training process to ensure the fundamental semantic com- prehension ability of the trained reward models. For example, in Section 3.4 each reward model is noreverse trained using the variant dataset (LIFT-plus, reverse LLIFT-plus por LIFTT-plus emptyy combined with Drm-
For the Base models, first apply SFT using Dast, followed by reward model training Drms For the Instruct models, we directly train the reward model using D, D,m without prior supervised fine-tuning.
19

---

Published as a conference paper at ICLR 2026
Training Models. In the preliminary explorations, employ three groups of models Qwen2- 1.5B-Base with Qwen2-1.5B. 1.nn2-1.B-IInstrueee Qwen2.5-7B- B-se with 1aa.n..5--B-Instruuue and Llama- 3.1 B.1-8B-Base with Llama-3. 1.mm-331-888 ----Instruct-- to verify the broad applicability of our analytical results. However, in Section 3.4 we exclude the Qwen2.5-775 Ba--aae and ( models from analysis for simplicity.
D.2 THE CONSTRUCTION PROCESS OF D
eval
To fairly and accurately evaluate the semantic comprehension capabilities of the reward model, we establish refined quality evaluation dataset D through an automated transformation of Deval, leveraging advanced language models (Dubois et al. 2004b) Zheng et al. [023 Chiang al. 2023 To ensure the reliability of the generated dataset, we did not directly adopt GPT-4o's outputs, as GPT-4o also exhibits length bias. Instead, we employed a rigorous algorithm that augments simple model generation with programmatic verification, thereby transforming the process into generation + verification.
Specifically, First, for each triplet (x:1), (6) Ye Ur D Devats we employ GPT-4o (Hurst et 2024
with the Response Rewriting Template (Figure 4 aa) to generate two alternative responses, ye"
[+).
and ye (0)2 based on the original prompt (8() and response y²") The semantic equivalence between
the generated response (0).1 (or ye (0),2 and the original response ye is verified using the Quality Consistency Verification Template in Figure
Subsequently, we process y through GPT-4o to generate two alternative responses, yf ( and Br (0)2 based on the original prompt zz((), 'hese generated responses must satisfy two criteria: (I) maintain semantic equivalence with 2/²², verified using the Quality Consistency Verification Template, and (2) exhibit specific length relationships with their counterparts, namely lyp (0
lye 1 and ( < lyfe 1 verified using programmatic evaluation. Given GPT-4o's limitations in direct length control, we implement a separate generation process utilizing the Response Expansion
Template and Response Compression Template (Figure b)) and 4()) to obtain (a),1 and (0),
respectively.
The resulting quality evaluation dataset is formulated DD coal
{(x(0,)6 (0.1 br 1 U
((a(1) ye (2,2 yr ). The complete prompting templates for response generation and quality con- sistency verification are presented in Figure
Suppras leartting Tepiic
Inaez Eegaster Toa
INSIC US CIRFAEEzO Tmplizz
callyy Graiatand KO Het Leston Iaplic
Cierr the (ollddi Inerrvettoe and neccus:
Slw filladre Inttrcttion respaner
taspert::
Cretructio::
statter snatin the widch wi
resperss soways
fillaws
ceat than the
the the ateee Seager
atate siate
waree siving eecinal mantine
resernes The reppoee shall wireeon
the Emppare 2% Reeeww better the
(a)
(b)
(c)
(c)
Figure 4: (a). Response rewriting template for the chosen response ye (0) (b). Response expansion
template for the rejected response yf (c). Response compression template for the rejected response
" (d). Quality consistency verification template for assessing the quality consistency between the rewritten and the original responses.
D.3 THE CONSTRUCTION PROCESS OF D' L
To accurately and efficiently evaluate the reward model's adherence to length instructions, estab- lish the length evaluation dataset Dt D Each triplet (zr²) y (.. (0,f) in DJ Di consists of a length
instruction 20()) length constraint+ and two semantically equivalent responses that differ only
20

---

Published as a conference paper at ICLR 2026
in length: ye" (4)d adheres the length constraint specified in 2 while yr (0)d violates To optimize computational efficiency, derive Di Deval from the previously constructed D eval- Specifically, for the responses in D we already ensure that Vc (6). and ye (6),2 maintain semantic similarity during the construction of DD eval' Additionally, the stochastic nature of GPT-4o's response
responns generation inher- and 1/ typically differing. Leveraging this property, we select these responses () and yo ()))) as candidates for x((). Specifically, we set the word nua parameter in zi within the range [lve (1vE))11, 1, 1v6 (assuming 1 21,
ently produces responses of varying lengths, with ly. (8).1
ai less length constraint + xz ). Consequently, we assign (0). ye (i),1 the chosen response
and yr (0), yc
(8).2 as the rejected response. Finally, the constructed length evaluation dataset is (5) (1). 33 ).
D! Devat = {(x]
D.4 THE EXTENSION PROCESS OF LIFT
In LIFT (Yuan al. 2024). the authors propose a straightforward method for constructing length instruction dataset. In Section 3.4 we will examine the limitations of this method. The LIFT (6) approach can be divided into two types: the first part LIFT, constructs length instruction 2 that both responses (y(") and ) satisfy the length constraint specified in z ( the second part LIFT, constructs length instruction xt that only one response (vé") yf satisfies. In cases where the rejected response vي" adheres to the length instruction but the chosen response (f) does not, this can lead to reversal of the original preference order, which fundamentally undermines the semantic quality of the data. Furthermore, LIFT only considers maximum length constraint, i.e., "less length constraint", without accounting for a minimum length constraint, i.e., "more length constraint". Therefore, made slight improvement to LIFT by adding "more length constraint", resulting in LIFT-plus:
LIFT LIFT, ULIFT,
{I u yf U (d ,/))!
(1)
Leess less length constraint + 200,,
(
(20)
LIFT-plus, more {( (more vo ( U (more v/t² ve (0))),
(1
(1)
Lmore more length constraint z(1),
(t)
LIFT-plus LIFT-pluS ULIFT-plus,more
where less length constraint is "Answer the following instruction using (word.rum)) words or less.", and more length constraint is "Answer the following instruction using { {word words more." In this article, we refer to 2i,iess the "or less" length instruction and ii,more the "or more" length instruction. Similarly, LIFT-plus can also be decomposed into two parts: LIFT-plus, and LIFT-plus, as follows:
(i)
LIFT-plus LIFT-plus, LIFT-plus, = ((x) (1) .v6 , ( 1
(21)
length constraint + (6)
where length constraint is less length constraint or more length constraint. In the first part LIFT-plus,, the length instruction x is satisfied to both responses. As a result, it may degrade into length-agnostic instruction, where the model disregards the added length constraint and fo- cuses solely the original prompt ( Therefore, the key component that effectively facilitates adherence to length instructions the second part, LIFT--luss,, where only one response (y or ( satisfies (
"hhe detailed definition of the length constraint be found in Appendix D.4
21

---

Published as a conference paper at ICLR 2026
yr _y²)), which reverses the original preference order between chosen and rejected responses due to the length con- straint in x7 and LIFT-plus order. In addition, based on LIFT-plus horetrr we construct a dataset purely centered on length in- structions, denoted LIFT-plus, (xc Mc
Furthermore, we divide LIFT-plus, into two subsets: LIFT-plus [
(6)
{(x(]) Vc
(t) (t) " which preserves the original preference
(9) (2) v where (6) length constraint e'mmptyy prompt"J. This dataset isolates length instructions from other semantic content and only focuses on the length instructions themselves. Here, LIFT-plus 5mpty is specifically designed to investigate the impact of semantic prompts length instruction following in Section 3.4
E THE DETAILED RESULTS AND ANALYSIS OF PRELIMINARY EXPLORATIONS
E.1 LENGTH BIAS INDEED EXISTS
Although the responses (1) and (2) in De Deval' D Divel and De engl semantically misaligned with their respective prompt, we observe that the reward models still achieve relatively high accuracies on these evaluation datasets. Specifically, the results for the three groups of models D Devai D De and evel as shown in Table 1 indicate that nearly all reward models achieve an accuracy of 60% or higher, which is very close to the accuracy observed on the original evaluation dataset Denat- This suggests that, even in the absence of the prompt, the reward models exhibit a significant bias toward the chosen responses.
To further investigate whether this bias primarily arises from response length, we first analyze the consistency of model evaluation results between D De (or Dewat) and Devat, focusing on how vary- ing prompts affect the selection of the same response. The results, presented in Table show that in over 85% of cases, the models select the same response regardless of the prompt. This reinforces the observation that the reward models' preference is not primarily driven by the prompts themselves but rather by other respense-related factors.
Next, we plot the relationship between the response length and the corresponding reward score of the reward models for Deval. Devat Dical and D as shown in Figure To facilitate comparison, normalize the reward scores of both models Dowen-1..B-Inntruue and Llama-3. --stttrruct) to the same range. The results reveal a strong linear correlation between the response length and the reward score across all three evaluation datasets. Specifically, response length increases, the model's 's reward score also rises, indicating that response length plays a significant role in the model's assessment of response quality.
We further conducted experiments that analyzing the relationship between the accuracy of reward models on Deval and the length difference between the chosen and rejected responses. The results are demonstrated in Table g It can be observed that as the length difference increases, the accuracy also increases. This clearly demonstrates the presence of length bias in the Baseline models.
Table 8: Accuracy of different reward models on different subsets of Deval partitioned by length difference between chosen and rejected responses
Model
S -100
-100 ~ -50
-50
~ 50
50 ~ 100
2 100
Qwen2-1.5B-Instee
9.10%
6.06%
38.96%
84.40%
89.79%
95.52%
Qwenn.5-7B-Inntrucct
18. 18%
15.16%
29.87%
81.65%
91.84%
88.46%
18.19%
9.10%
40.26%
67. 67.89%
79.59%
88.06%
E.2 LENGTH BIAS IN EVALUATION DATASET
We conducted further analysis on the generated evaluation dataset D Pgaal Specifically, we recruited
20 volunteers, each assigned 10 randomly sampled data from D eval The volunteers were instructed to focus solely on the semantic quality of the responses, disregarding factors such as length or
22

---

Published as a conference paper at ICLR 2026
format. We then collected their preferences regarding the data. Upon aggregating the results, found that human preferences are 90.5% consistent with D of our generated datasets.
eval" This strongly supports the validity
E.3 LENGTH INSTRUCTIONS ARE EASILY LEARNED
The results for the Qwen2-1.5B and Llama-3. 3.1ma-3..--BBB 1-8 models trained on the three variant datasets (LIFT-plus LJFF-plus and LIFT-plus, (mptrr are presented in Table As observed, most reward models achieve similar or even lower accuracy on D Deval compared to Baseline mod- els in Table 4 while their accuracy on Devaal remains relatively high. These results suggest that, although reward models are trained on diverse length instruction datasets, they all primarily learn to capture response length information rather than balancing adherence to length instructions with attention to the semantic content the prompt.
Table 9: Evaluation results of different reward models (LIFT-plus variants) on quality (D" (Deval and
length (D) emaz) evaluation datasets.
Model
Variant
Quality Eval Acc (%)
Length Eval Acc (%)
LIFT-plus, reverse
58.78
85.58
Qwen2-1.5B-
LIFT-plush horeserse
59.41
87.78
LIFT-plus Empty
58.33
89.71
LIFT-plus reverse
60.11
86.54
Qwen2-1.5B-Instruct
LIFT-plus hoorevreeee
horeverse
63.17
85.21
LIFT-plus emptyy
61.02
85.53
LIFT-plus, reverse
51.34
91.32
1lama-3.1-8B--a Base
LIFT-plus horewerse
57.71
88.14
LIFT-plus emply
50.27
88.78
LIFT-plus, J reverse
60.11
95.19
Llama-3.1-88 1-B..B.1-BBBBB 1-8B-Inntruct
LIFT-plus Rorewerse
56.65
92.31
LIFT-plus empty
57.97
90.71
To further illustrate how reward models quickly overfit length instructions during training. thereby impeding semantic learning, we visualize the accuracy trajectories of Qwen-2-1.. 2-10n-2-1-B-Insttii and 1ama--31-8B--nssee
on D and D! De as training steps increase (see Figure 5). The results show ceal that as training progresses, the model accuracy on D' D increases rapidly, whereas the accuracy on much slower rate. Moreover, once the accuracy on D' reaches its peak, the accuracy on D Peaa also begins to decline and gradually plateaus. This tendency strongly supports the conclusion that the reward models quickly prioritize learning length instructions, achieving high accuracy length-related tasks, but at the expense of semantic quality. The slow improvement the accuracy on D Di further suggests that the reward models struggle to learn the necessary semantic understanding, as they overly focused conforming to length constraints.
D4 eval improves
F ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
F.1 THE EVALUATION DETAILS OF DPO MODELS
Similar to Appendix D.4 due to the length instruction in the AlpacaEval-LLI benchmark provided by LIFT (Yuan et al. 2024 includes only the "or less" length instruction, is necessary to incorpo- rate the "or more" length instruction. Therefore, both the "or less" and "or more" length instructions. Specifically, AlpacaEval-LLI word_num is set based the shortest response length for the original prompt across three advanced models GPT-4 (Achiam et al.) (002),, Claude3-Opus, and Mistral Large - which we assume approximates the median response length for each prompt. This suggests that well-reasoned, detailed responses
extend the AlpacaEval-LL benchmark to include
23

---

Published as a conference paper at ICLR 2026
0.9
Quality Eval
0.5
Accuracy
0.8
Leugth Eval
Quality Eval Ace
0.4
Accuracy
0.8
Length Eval
(Smoothed)
0.4
0.7
(Smoothed)
Loss
0.7
0.2
0.6
0.1
0.5
200
600
Train Stop
Train Step
(a) 1.5n--I-5BInntrruet
(b) -.lama-3.--B--mmm -nS-tttrttttt
Figure 5: The trajectories of Quality Eval Acc, Length Eval Acc, and Training Loss (Smoothed) for different reward models trained on LIFT-plus, reverse across training steps.
may exceed word_num, while concise yet valid responses may fall below it. Therefore, we im- prove the AlpacaEval-LL benchmark by modifying the "or less" in Iı to "or more", specifically replacing "Answer the following instruction using (word num) or less" with "Answer the following instruction using (word_uum) more", while keeping all other aspects unchanged. This results in the expanded AlpacaBval-LL-plus-mon benchmark, and the original AlpacaEval-LI benchmark is referred to AlpacaEval-LI-plon --plu--s---- benchmark. In cases where there no ambiguity, refer to waapccvvvvl----uus----ea and Liaicaavv------------ee more benchmark as AlpacaEval-LL -va------l-llss bench- mark.
For the DPO evaluation metries, as described in Section 5.1 we use automated assessment by lever- aging GPT-4o (Hurst et al.] 2024) label each pair responses as win. sie, or lose based on semantic quality. The prompt used to guide GPT-4ooss evaluation is similar to the one provided in Appendix [.2], as shown Figure Specifically, we evaluate each DPO model's performance by comparing it with its corresponding SFT//nstruct model using three metrics: Length Acc refers to the accuracy of the model-generated response length that satisfies the length constraint specified in the given length instruction x; in AlpacaEval-LL-plus Length Win Ratio denotes the proportion of cases in which the model-generater response is labeled as win compared to the response generated by the SFT//nstruct models, under the given length instruction 2) Quality Win Ratio measures the proportion cases in which the model-generated response labeled as win compared to the re- sponse generated by the SFT//nstruct models under the given original prompt v. Length Win Ratio and Quality Win Ratio evaluate the semantic quality of responses generated by the DPO models under different prompts.
Respanse Semantic Quality Comparison Template
Given following instruction and candidate nesponses:
Instruction: ( Response ( Response 2: ()
Please evaluate which response better follows instruction based the following criteria: 1. Adherence to the instruction.
2. Overall semantic quality.
Conciseness: the response should avoid redundancy, unnecessary verbosity, or inclusion irrelevant information.
Based on your evaluation, respond in the format: "Response better than Response 2', Ressonnse 2 is better than Response 'Response equal to Response 2'.
Figure 6: Response comparison template for evaluating the semantic quality between responses.
24

---

Published as a conference paper at ICLR 2026
Table 10: Length Eval Acc of different reward models on length (Di (Devt) evaluation datasets.
Model
Baseline
LIFT-plus
Re-RM
@wen2-1.5B--Bas
52.41
87.18
86.22
Qwen2-1.5B-Insteee
51.77
86.54
84.61
Qwen2.5-7B-BB
52.24
84.94
88.14
Qwen2.5-7B.
#wen2.5-7B-Instruct
52.88
83.97
92.31
Llama-3 -,, 3.1-8B-Bass
51.92
91.32
88.46
Llama-3.1 3.1-8B-Innttrus
49.04
95.19
93.27
Gemma-2-9B-it Bwen2.5-14B-Insttie
51.28
91.66
90.91
54.13
95.25
95.34
F.2 THE RESULTS OF REWARD MODELS
Re-RM effectively follows length instructions. In addition to mitigating length bias, Rc-RM sig- nificantly enhances the ability to follow length instructions, as demonstrated by the Length Eval Acc results in Table TO Specifically, Rc-RM achieves accuracy comparable LIFT-plus and surpasses it in nearly half of the models, with gains of 3.2% on Qwen2.5-7B-BB B-Base, 8.34% on Qwen2.5-7B. Instruct, and 0.09% on Qwen2.5-14P Instruct.
representative experiment, we evaluate the score consistency of reward models with respect to length instructions by constructing sequence of length instruc- tions Ii (5.5) .j = 21, .... 8}. with increasing word.num. Specifically, we first select all instances (6) (1) Deval where ( | referring to this subset as Dless, For each x(() in (1()), Me 4/
Taking "or less" length instruction
Diess we then construct the following sequence of word num values (Lum) to generate the corre- sponding sequence of length instructions x (i.5), and refer to the entire constructed evaluation dataset Dmis.
Lun =[hw 2I, ww - T, Lw ww + L,ly + 2L, 4, 4 + , 4 27'], (i.j)
(22)
Dmlla = (x y yi )0) jE (
where si (i.j) "Answer the following instruction using {word_nwn = LG)) words or less." (0)
= 10, L = (4 l/3 3,2, L = 1ycl. and 4 = 1rl- For the sequence of (b.)) ye yr ).j (1, 8} in Dmil, the ideal performance of the reward model should be as follows: Initially, when word_rum Lw. the length constraint of (z(-.) is invalid for both ye and ( the difference in reward scores between (8) and yf primarily reflect the semantic difference in the original data (a(1), ye (6) ). When word.num L.. the length constraint of z (i.) aligns with y! but not v resulting in greater difference between (2) and (1) compared to their original semantic difference. When word.num 4, the length constraint of xi
(i.j) becomes valid for both yo and / and the difference reverts to those derived from the original semantics. Therefore, the difference
in the predicted scores the reward model for い and y!" should initially increase with the rise in word numn, then decrease, eventually returning to the original semantic difference.
The results, as shown Figure (c) and demonstrate that both LIFT-ppus and Rc-RM exhibit an initial increase followed by decrease in reward score differences word_nwm increases. However, LIFT-plus fails to return to the original difference after word increases, instead remaining at level higher or lower than that observed when word_num < |yel- This indicates that LIFT-plus overfits to the length instructions, thereby neglecting the original semantic instructions. In contrast, Re- RM accurately reflects the expected behavior: the score difference increases with the rise in word num, then decreases, eventually returning to the original semantic difference. Moreover, Rc- RM exhibits minimal score fluctuation both extremes when length constraints invalid for
25

---

Published as a conference paper at ICLR 2026
both responses and when they valid for both responses. This robust performance provides strong evidence for the effectiveness of our approach in simultaneously adhering to both length and original semantic instructions.
We conducted further experiments by training a policy using the trained reward models via PPO. We trained the Qwen2-1.5B-Ing Insstrcctt policy using the reward models of (ive22-..5BIInstrunl and Swven..5-7B-Insteen
from Table 4 The evaluation was carried out on AlpacaEval following the same metries used for evaluating the DPO models. The results are in Table [1] It can be observed that the policy trained using Rc-RM exhibits high semantic quality. Its performance significantly sur- passes other length- tannth-bias-minee mitigating methods, further demonstrating the effectiveness of our method.
Table 11: Evaluation results of different PPO models on AlpacaEval DDboos et al. 2024b)
Qwen2-1.5BB 1ree2-1.5B-Instrucc
Metrics
Bascline
ODIN
Rc-RM
Quality Win Ratio (%)
38.28
41.57
44.06
Response Length
671.49
342.85
308.16
Qwen2.5-7B-Instruct
Metrics
Baseline
ODIN
Rc-RM
Quality Win Ratio (%)
40.31
45.03
49.26
Response Length
597.27
336.15
277.15
We further conducted experiments with the GRPO algorithm. In line with the PPO experiments pre- sented above, we trained the Qwen2-1.5B- 1.n-structtt policy using the reward models of Qwen2-1.5B. Instruct and Qwen2.5-775 7 from Table The evaluation was again performed on AlpacaE- val, reporting Quality Win Ratio as the primary metric. The results are summarized in Table 12 As shown, the findings are consistent with those obtained using PPO: policies trained with Rc-RM sig- nificantly outperform other mangth-bias-mitiaane methods, further corroborating the effectiveness of our approach.
Table 12: Evaluation results of different GRPO models on AlpacaEval (Dubois et al. 2024b
Qwen2-1.5B-Insse
Metrics
Baseline
ODIN
Rc-RM
Quality Win Ratio (%)
37.69
43.14
45.89
Response Length
719.37
498.22
330.15
Qwen2.5-7B-Inntrucc
Metrics
Baseline
ODIN
Rc-RM
Quality Win Ratio (%)
41.23
47.25
51.60
Response Length
603.18
355.69
298.70
F.3 THE RESULTS OF QWEN2.5-L5 5-555BBBB.BB REWARD MODELS
This subsection mainly presents the RM experimental results for Qwen2.5-1.5B and some unreason- able experimental phenomena. As show in Table 13 for Qwen2.5-1 --5--Bas the results are similar to those to Section 5.2 Rc-RM not only achieves higher performance in Quality Eval Acc, being 7.69% higher than LIFT-plus and 6.41% higher than ODIN, but also performs almost equally with LIFT-plus Length Eval Acc. For 2wwen..5-1.B--nnttuue
Rc-RM still outperforms LIFT-plus by 3.21% and ODIN by 1.61% % in Quality Eval Acc. However, the results for LIFT-plus somewhat unreasonable: its Length Eval Acc is only 67.637 63%,, 12.5% lower than Rc-BT, while its Quality Eval Acc is 10.9% higher than Baseline.
Furthermore, we plot the Quality Eval Acc and Length Eval Acc trajectories for each method across training steps, as shown in Figure 7 It can be observed that in the early stages of training, LIFT-
26

---

Published as a conference paper at ICLR 2026
Table 13: Evaluation results of reward models on quality (D eval) and length (D eDeat) evaluation datasets (Qwen2.5-1.5B)
Qwen225-1L5BB -55-Bse
Soven2.5-1.58-nnatte
Metrics
Baseline
LIFT-plus
ODIN
Rc-RM
Baseline
L.FT-ppuus
ODIN
Rc: RM
Quality Eval Ace (%)
58.97
60.26
61.54
67.95 85,89
58.01
68.91
70.51
72.12
Length Eval Aoc (26)
51,28
87.82
55.45
53.53
67.63
50.00
80.13
0.8
Baxclin LIFT-plux
0.70
ODIN
0.7
Re-RM
0.65
Accuracy
Accuracy
0.60
0.6
0.55
0.50
0.5
LIPT-plus
0.45
ODIN
100
150
0
50
100
150
Train Step
Train Step
(a) Length Eval Acc
(b) Quality Eval Acc
Figure 7: The trajectories in Length Eval Ace and Quality Eval Acc of Qwen2.5-1..B-Insstit with training steps.
plus's Length Eval Ace gradually increases, while its Quality Eval Ace is worse than that Baseline, which aligns with our analysis in Section 3.4 However, after just 60 steps of training, LIFT-plus's Length Eval Ace suddenly drops drastically, while Quality Eval Acc begins to rise significantly. This ultimately results in Length Eval Ace stabilizing stabilizing at relatively high value. Since this is a single isolated example, we suspect it may be related the specific model type and will conduct a more detailed analysis in future work.
relatively low value and Quality Eval Acc
F.4 THE RESULTS OF DPO MODELS ALPACAEVAL
To further complement the results reported Section we conducted additional comparisons between method and SamPO (Lu et al.) 2024) by training the same datasets used in Sec- 5 Specifically, we focused on two representative backbone models: Qwen2.5-7B-Insttee The results are evaluated AlpacaEval, where report Quality Win Ratio as the primary metric. The results are summarized Table 14 models trained with Rc-DPO consis- tently maintain high semantic quality. More importantly, their performance substantially surpasses that of SamPO across both backbones, which further substantiates the robustness and effectiveness of our approach. These findings reinforce the conclusions drawn in Section [.. indicating that Re- DPO not only mitigates length bias but also enhances overall response quality in alignment-orienn fine-tuning.
and
Table 14: Evaluation results of different DPO models on AlpacaEval.
Owen2.5-7B-Insttree
Llama-3. 3.1-BB
1--8BB-Instrue
Metrics
SamPO
Rc-DPO
SamPO
Rc-DPO
Quality Win Ratio (%)
38.12
44.63
50.83
64.34
27

---

Published as a conference paper at ICLR 2026
F.5 THE RESULTS OF DPO MODELS IN ALPACAEVAL-LI-PLUS--AEE
Due to length bias, models tend to generate longer responses, leading to higher Length Acc on the AlpacaEval-LL--plus--ee
benchmark, even when the models have no awareness of length in- structions. This makes it difficult to fairly compare models' ability follow length instructions. Therefore, by examining the performance on the AlpacaEval-LI-plus-e assess how effectively each method adheres to the specified length limits while still maintaining a high level of semantic quality in the generated responses.
benchmark, we can better
The results on LaneeaEval-L--luu--Bee
are shown in Tabel [5] As observed, for the Base models, Baseline models are trained only on Daft and Drm, and do not demonstrate the ability to follow length instructions. On the other hand, the Instruct models inherently exhibit stronger ability to follow length instructions. Consequently, Baseline models trained on the Instruct models also acquire a certain degree of length instruction adherence. For R-DPO, since it was trained only on Dym. and according to the Response Length results Tables 5 15 and 17 it primarily reduces the response length across all instructions (include "or more" length instruction), without explicitly following the length constraints. This indicates that R-DPO does not specifically address length instruction adherence, but simply focuses on minimizing the response length overall. Thus, the main comparison method we focus on here is LIFT-plus.
Table 15: Evaluation results of different DPO models on AlpacaEva--LL-plus-less
Qwen2.5-7B 5-------B-BBB Base
Qwen2.5-7B-Instee
Metrics
Baseline
LIFT-plus
R-DPO
Rc-DPO
Baseline
LIFT-plus
R-DPO
Ro-DPO
Length Ace (%)
86.78 106.87
89.65
90.90
Response Length
$44.47
637.44
177.40
136.75
23.56
131.17
108.24
Length Win Ratio (18)
34.04
1.37
34.54
44.14
32.17
2.99
31.92
50.75
Llama-3. 1-8B-Base
Llame-3. 1lam--31188--Instee Instrrttt
Metrics
Baseline
LIFT-plus
R-DPO
Re-DPO
Baseline
LIFTT-plus
R-DPO
Rc-DPO
Longth Acc (%)
13.59
98.75
27.68
94,51
87.66
100
87.03
100
Response Length
424.47
6.62
518.74
118.50
150.43
35.01
151.39
89.75
Length Win Ratio (8)
35.41
0.25
47.88
64.96
43.89
1.00
43.77
64.71
It is evident that LIFT-plus suffers from severe "short bias" issue in Appendix F8 focusing ex- clusively on the "or less" length instruction 2i itself without considering the specified word num or the original prompt This leads to extremely short responses that strictly comply with the length instruction but completely disregard the semantic requirement of the original prompt. Specif- ically, On Qwen2.5-7B-Bass and Qwen2.5-7B-Instiee LIFT-plus are only 106.87 and 23.56, significantly below the average length constraint of 180.23 in AppacaEval-L--plus-lee on 1--ma-3.1-8B-Baar the average response length drops to a mere 6.62, showing a 111.8d difference compared Rc-BT. This behavior results in LIFT-plus achieving nearly 100% Length Acc. but with Length Win Ratio close to 0%, which fails to align with the desired semantic quality.
the average response lengths generated by
On L.a-a-3.1-8B- Instructt the average response length is 35.01, and
In contrast, Rc-DPO not only adheres to the "or less" length instruction but also considers the specified word_nwm and original prompt. It strives to maximize the semantic quality of the re- sponse within the length constraint. Therefore, compared the Quality Win Ratio in Table S Re- DPO maintains semantic quality even under length constraints. Notably, on Qwen2.5-7B Base and Llama- 1.1--- Base, Rc-BT achieves 82.04% and 94.51% Length Acc, respectively, while boost- ing the Length Win Ratio to 44.14% and 64.96%, further enhancing the semantic abilities of SFT and Llama-3.
Rc-BT achieves 100% Length Accuracy, while obtaining Length Win Ratio of 50. 75% and 64.71% respectively. This fully demon- strates the effectiveness of Re-DPO in balancing adherence to length instructions with high semantic quality.
models. For both Loven2.5-7B-Insteen
To provide a more detailed analysis of how well our DPO models follow length instructions, we fur- ther evaluate on AppacaEval-LL-plus-les For each length instruction z; in AlpacaEvaalLL-pluu--teee we adjust the original length constraint by +50 and 100 to create five benchmarks, denoted A 1010 A 50, Ag. Aso, A100- These benchmarks allow to assess each model's 's ability dynamically ad-
28

---

Published as a conference paper at ICLR 2026
just response length. The results are shown in the Table 16 (with the average length constraint of each benchmark indicated in parentheses).
Since some original length constraints are below 100, the corresponding values in A010 and even A-50 would become negative. We therefore remove these samples from 100 and A A-so. which results in the average length constraint of A-s0 (A-100) not being exactly 50 (100) less than that of A. As the baseline and R-DPO models perform poorly in following length instructions (see Table 15 let alone in dynamic adherence, we primarily compare against LIFT-plus. As the table shows, LIFT-plus fails to dynamically follow varying constraints and merely shortens responses indiscriminately. In contrast, our method effectively adjusts response length accordance with the changing length instructions, clearly demonstrating its effectiveness.
Table 16: Evaluation results of different DPO models on AlpacaEval-LIpllu--esss with different length constraints.
Owen2.5-7B-Instren
Metrics
LIFT-plus
Rc-DPO
LIFT-plus
Rc-DPO
A 100 (124.27)
23.01
78.29
30.14
65.73
A A- (149.42)
24.71
85.17
36.83
72.69
(180.24)
23.56
108.24
35.01
89.75
Aso (230.24)
25.78
130.50
37.19
121.14
A100 (280.24)
26.14
164.07
42.93
159.88
F.6 THE RESULTS OF DPO MODELS IN ALPACAEWAL-LI--PUU--NOOR
In this subsection, we present the experimental results of the DPO models from Section 5.3 on Al AppacaFval-LL-plus--mee
as shown Table 17 Similar to Section [53] since R-DPO was not method with LIFT-plus. First,
trained on length instruction data, we primarily compare in T due to the influence of length bias, Baseline model outputs longer responses, which means that it largely satisfies the "or more" length instruction limit without focusing on the length constraint itself. Moreover, a comparison with Table I7 further confirms it: after adding the "or more" length instruction limit, the length variation in the Baseline responses is significant (especially for the Instruct model). At the same time, its Length Win Ratio demonstrates that compared to the SFT / Instruct models, the Baseline's responses improving semantic quality, leading to no significant improvement in Length Win Ratio. In fact, for all four models, the Baseline's Length Win Ratio is lower than ours by 17.46%, 24.31%, 1%, 9.61 %, and 14.71% %, respectively.
more redundant and repetitive, rather than truly
For LIFT-plus, the results are similar to those in Appendix F8 where focuses only on the "or more" length instruction and disregards the semantic requirements of the original prompt and the specific length constraints of the "or more" length instruction. This causes the model to generate excessively long and unnecessary responses, resulting in a high Length Acc but decline in Length Win Ratio. Specifically, for the Instruct models, due to the strong foundational capability, the over-generation behavior of LIFT-plus is relatively mild, with response lengths gen- crally controlled within 200 ~ 400. Its Length Win Ratio is only lower than Re-DPO by 33.62%
cestruct-folloiig
and 26. 28% However, for the Base models, LIFT-plus's response length skyrockets above 600, causing severe semantic redundancy, and its Length Win Ratio significantly drops, being lower than Re-DPO by 39.53% and 35.91%. This result, when analyzed alongside the findings in Section 5.3 and Appendix [.8 fully illustrates the drawbacks of LIFT-plus's over-reliance on length instructions and the complementary nature of Rc-BT in terms of both length instruction and semantic quality.
F.7 ABLATION STUDIES AND EXTRA RESULTS
Due to the high cost of evaluating the DPO model, we primarily the reward model for ablation studies. First, we conduct ablation studies to validate the key design choices our approach. For convenience, we denote ((x, st, }o)} Dr, and { 3.)} Pre-
29

---

eval and D evat for Published as a conference paper at ICLR 2026
Table 17: Evaluation results of different DPO models Alpacatval-LLpplas-oooee
Qwen2.5-7B-Bal 5-7B-Bacc
Qwennn--7B-instruc
Metrics
Baseline
LIFT-plus
R-DPO
Rc-DPO
Baseline
LIFT-plus
R-DPO
Rc-DPO
Length Acc (%)
99.75
99.88
99.38
84.29
78.43
100
Response Length
680.80 84
550.27
682.66
319.05
249.99
303. 10
238.03
263.30
Longth Ratio (%)
27.93
5.86
35.16
45.39
35.79
26.48
26.93
60.10
Llama-3. 1-8B-Base
Llamn-3. 3--.--..-------- Instrrett
Metrics
Baseline
LIFT-plus
R-DPO
Ro-DPO
Baseline
LIFT-plus
R-DPO
Rc-DPO
Length Acc (%)
89.03
99.75
93.39
97.51
98.25
99.88
97.01
99.88
Response Length
448.77
664.20
540,34
429.69
279 04
413.42
246.73
279.56
Length Win Ratio (18)
36.03
9.73
47.76
45.64
45.14
33.67
47.01
59.85
Die and Dr Rtc are complementary. We begin by proving the necessity of Df and De To this end, we conduct ablation experiments using Drm U'ee (w/o Diee) and Drm U Piie (w/o Pite). with the results shown in Table 6 As observed, when only one type of augmented dataset is used, the RM's Quality Eval Acc drops significantly, approaching the Bascline results in Table 4 Simultancously, its Length Eval Ace hovers around 50%, indicating that fails to learn the length instruction. To further investigate, we prepend length instructions to each in D evat. forming 37, where ye satisfies
the length constraint in the instruction, resulting in D 1 coat- We then use the DPO models in Table to score the pairs (Ii, Ye) in D aval and (ic, Me) in D aavel to observe changes in the RM's predicted scores. In theory, for the pair (mg, Me), Ve), since Ye satisfies both the length constraint and original prompt of zt, it should outperform (x,, Ye). That is, the score of reward model for (z), Be) should be higher than that for (zz ye). However, the results, illustrated in Figure ()., reveal that for the RM trained on D U 'D., the scores of (x1, Me) with added length instructions are consistently lower than those of the original (xr, ye). regardless of whether the added length constraint matches ye length. This indicates that after training
Drm U Uiee the RM learns a length instruction bias. Specifically, in Drm UD for any (Iit, No), with an added length instruction, the reward model merely reduces the score compared to the original (x, Me) pair, without paying attention to the length instruction itself. similar phenomenon in Figture 88 is observed the R.M trained on D Drm U Di Therefore, the combination of D Rc and Rd is necessary to prevent the reward model from being exploited by this bias hacking.
16
-18
RM Score
RM Score
-20
-15
-22
Dad eval
De.
D
24
coal
200
400
200
400
600
Response Length
Respouse Length
(a) w/o Dioe
(b) w/o Dis
Figure 8: The relationship between response length and RM score evaluated D Rc-RM trained with Llama-3. 1iima--.1-8B-Insiiee -------nsttttt...
Effectiveness of different ratios of DRo- We conducted further experiments to demonstrate the efficiency of the augmented dataset DRc we constructed. Specifically, we gradually increased DRe the original training dataset Drm- with 10% increment, train the reward
from 0% to 100% models for Qwen2-1.. 1..22-1.B--ime -B--ntttuct and @wen2.5-78 5-7-n...----sstrrect The results of Quality Eval Acc (%) shown in Table 18 It can be observed that our method demonstrates superlinear performance
30

---

Published as a conference paper at ICLR 2026
improvements with smaller proportion of DRe- In other words, using only 40% of the augmented data yields upproximately a 70% performance improvement. This indicates that, even under tight computational constraints, method can achieve significant performance gains with the addition of a small amount of data, which fully demonstrates the high efficiency of our augmented dataset Dn and method.
Table 18: Quality Eval Ace of different reward models on quality (Da different proportions of DRc-
(Deat) evaluation datasets under
Model
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Qwen2-1 SB-Instrect
60.75
64,43
65.69
66.68
67.31
68.11
68.77
69.95
70.40
71.05
71.47
Qwen2.5 Instruct
59.31
61.92
64.26
66.33
68.12
69.63
70.87
71.83
72.52
72.93
73.07
Effectiveness of different ratios of Rc and Di mixtures. We also briefly explore the impact of different ratios Die and Die on the final RM results. Specifically, we fix Dym and D constant, and increase Díe from 0% to 100% with a 10% increment. The results are shown in Figure 9 (a) and (b). As observed, initially, the model is biased toward the one-sided data of Dite" leading to low Quality Eval Acc and Length Eval Acc, even falling below the Baseline models (denoted as green dashed line). As Dio increased, Length Eval Acc rises rapidly, with the growth rate slowing down around 30%, while Quality Eval Ace gradually increases. It effectively demonstrates the complementary effect of Re and Dio in mitigating length bias and enhancing adherence to length Rc
instructions.
0.9
0.8
Bmal Aee
0.7
0.6
0.6
0.5
0.5
Length Evad Ace Qudlity Ace
0.4
Length Eval Ace Quality
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Dr Ratio
Dr Ratio
(a) Brren2-1,5B-Intmeee
(b) 1.1ma-3.1-88-Iaaae
Figure 9: Variations in Rc-RM's Length Eval Ace and Quality Eval Acc as Dr dashed line denotes the Baseline Quality Eval Acc, while the blue dashed line represents the Length Eval Acc of LIFT-plus.
Aee increases. The green
Effectiveness of Lic- We further conducted experiments by training the reward model using only the Re-BT loss LR.. emoving the standard BT loss while keeping all other training settings identical to Section 5 The evaluation was performed on the quality evaluation set D eval (Quality Eval Acc) and the results are reported in Table 19 As shown, even without the BT loss, our method still out- performs the comparison methods, indicating that Rc-BT alone is sufficient for learning semantic preferences. These results further underscore the complementary nature Re-BT and BT, as com- bining them yields more consistent improvements in both semantic quality and length adherence.
Impact dataset size normalization. Since our method constructs two examples from each train- ing pair, (z, Me) and (I, Mr), it is essentially equivalent to duplicating the original dataset rather than enlarging it with new data. To ensure fairness, we instead upsampled the baseline dataset to Dwwn2.551..B--l IIsttrct and Qwen2.5-77 7--.5--7BBnstccct As shown in Table 20 our method still achieves higher Quality Eval Acc, significantly outperforming both the Baseline and other length-bias miti mitigation methods.
match method's size and conducted reward model experiments
31

---

Published as a conference paper at ICLR 2026
Table 19: Evaluation results of Rc-RM only CrC on quality (Deval evaluation datasets.
Qwen2.5-1.5B-Instt Instructt
Qwen2.5-7B --Istttcttttt
Metrics
Rc-RM (w/o BT)
Re-RM
Rc-RM (w/o BT)
Rc-RM
Quality Eval Acc (%)
66.13
71.47
69.89
73.07
Table 20: Evaluation results of Rc-RM same dataset size LR on quality (D evaluation datasets.
Qwen2.5-1.5B-Instrul
Qwen2.5--B-Instrucct
Metrics
Baseline
ODIN
Rc-RM
Baseline
ODIN
Rc-RM
Quality Eval Acc (%)
61.02
63.92
71.47
61.27
68.19
73.07
Effectiveness across different datasets. To further validate the generalizability of Rc-BT, we duet RMM training on the HH-RLHF (Bai et al. 2022a) datase while still using D eval and D eval for evaluation. The results are shown in Table 21 As seen, despite the HH-RLHF dataset being from different domain compared to the OpenAssistant dataset (Köpf et al. 2024), which can be consid- ered an out-of-distributioo (OOD) scenario, Re-RM still significantly outperforms other methods on De evall while achieving results similar to LIFT-plus on Di Specifically, for Qwen2-1.55 1..5B 5B-Instruct, Rc-RM outperforms Baseline by 2.57% and ODIN by 1.60%. For Qwen2.5-7B-Insst IInttruct, Rc-RM surpasses Baseline by 2.88% and ODIN by 0.32%. Notably, for Llama-3. 1.tnaa-..1-8-- 1-8B-Instruct, Rc-RM significantly outperforms Baseline by 6.09% and also surpasses ODIN by 5.13. further demon- strating the effectiveness of Rc-RM in mitigating length bias. Moreover. D &ppl Rc-RM achieves results comparable to LIFT, reaffirming its effectiveness in following length instructions. Finally, the consistent performance across different datasets provides strong evidence of the robustness and effectiveness of Rc-BT in diverse scenarios.
Table 21: Evaluation results of different reward models on quality (Drmat and length (Dí (vva)) eval- uation datasets (HH-RLHF).
Model
Variant
Quality Eval Ace (%)
Length Eval Ace (%)
Baseline
58.65
59.36
LIFT-plus
58.01
85.90
Qwen2-1.5B-Insten
ODIN
59.62
48.72
Rc-RM
61.22
85.26
Baseline
62.50
52.24
LIFT-plus
57.37
92.63
Qwen2.55-BBInntruct
ODIN
65.06
57.69
Rc-RM
65.38
91.03
Baseline
63.14
51.60
LIFT-plus
58.33
93.27
1lama-3.1-8B-Ime
ODIN
64.10
50.00
Rc-RM
69.23
91.03
Effectiveness across different bias. We used format bias as another bed, as it the another common type of bias observed in LLMs. We used the FormatBiasEval dataset (Long et al.) 22024), which consists of multiple-choice questions with specific formatting requirements, to train and eval- uate reward models. As the dataset comes with predefined split into training and evaluation sub- sets, we adhere to this split in our experiments. To adapt the training subset into a preference dataset, we designated the correct option as the chosen response and randomly selected incorrect option as
We Instruct models for direct RM training the HH-RIHF dataset.
32

---

Published as a conference paper at ICLR 2026
the rejected response. Then we constructed additional "rexponse-conditionee triples of the form (x, .ry, ye) and (x, .2y,,))), Ye), where Ye violates the format constraints in z (or z3) while yr satisfies it. The final augmented dataset is defined D {(z, 2/, yc) U ((r,4f.8e))
Due to the programmatically constructible nature of the format, we created semantic evaluation dataset based on the evaluation subset, referred DJ eeap. using a methodology similar to that used for constructing D gegls but without relying on GPT-4o. The process is as follows: For each triple (Ir, Nos V-) in the original evaluation dataset, we generated two complementary triples: (z. fya), yr) and (2r, Mes J'(x)) )), where (w)) denotes program that applies a randomly selected format pattern to the response y. Therefore, the D can effectively mitigate the impact of format bias and facilitate more fair evaluation of the reward model's semantic capabilities.
Table 22: The accuracy of different reward models on quality (D) (Démai) evaluation datasets.
Model
Baseline
Pe-Format
Rc-RM
Qwen2- 11--2-11.B--B- 1.5B-Instruct
76.25
75.41
79.17
Qwen2.5-7B- --et-tttttll
82.42
78.33
89.27
Llama-3.1 -SSttttttuutt
79.39
79.01
84.24
We conduct experiments Instruct models. And the final results are shown in Table 22 where refer the prompt-conditioned method, which is similar to LIFT, as Pe-Format. As shown, Re-RM remains more effective in mitigating format bias, achieving superior accuracy across all evaluated models. These results clearly highlight the strong generalization ability of our method.
Effectiveness of eneration + verification pipeline. We further employed another strong LLM, (f) (8)
Slaue-SSonne-44. to rewrite the triplets (1r Me (1 using the same pipeline described in Ap-
evaluation dataset D eva!" We then evaluated all methods this dataset. The results (Quality Eval Acc) are as shown in Table 23 The results demonstrate that the performance trends across different models remain consistent even when evaluated datasets rewritten by different LLMs, further validating the robustness of our evaluation data construction pipeline.
pendix D.2 generating
Table 23: Quality Eval Ace of different reward models quality (D evaluation datasets.
Model
Baseline
LIFT
ODIN
Rc-RM
Qween2.5-7B-Innsrruu Llama3. 1-8B-Instruct
61.74
57.32
68.15
74.89
58.13
58.67
64.34
72.21
Additionally, we incorporated two types of automatic metrics: BERTScore (BERTScore-FI) to quantify compression-retentin and n-gram Repetition Rate (n 3) to measure redundancy. Specif- ically, For each generation iteration, we computed these metrics on the rewritten responses. The results are shown in Table [4 Thanks to the strong capability of GPT-4o and our generation- verification pipeline, the initial BERTScores were already high while redundancy remained low. Further, low-quality candidates were consistently filtered out through iterative rejection, ensuring that the final responses were both semantically faithful and nondrndun..
Table 24: BERTScores and n-gram Repetition Rate of different turn of rewriting.
Model
First Rewrite
Second Rewrite
Third Rewrite
BERTScore
0.912
0.974
0.981 0.06%
n-gram Repetition Rate
0.27%
0.09%
We also include qualitative examples of discarded failure cases (Figure 10 in earlier rewriting rounds for reference. These results collectively demonstrate that, despite the potential risk of se- mantic stylistic drift in LLM rewrites, iterative generation pipeline effectively identifies and
33

---

Published as a conference paper at ICLR 2026
filters out low-quality responses, ultimately ensuring that the final outputs exhibit strong semantic retention and minimal redundancy.
query: What types of tests do we have in software development?
original response: Software testing generally elassified into two broad calegories: functional testing and non- functional testing. There is also another general type testing called maintenance testing, in'nFunctional testing involves the testing of the functional aspects of software application. When you're performing functional tests, you have to cach and every functionality. You need see whether you getting the desired results or not.'n'nnTerr are several types of functional testing, such as:'m'nUnit testinginlntegrution testing-nEndto-end testing'n Sanity lestingiiRegrresionn testinginAcceptanee lesting'minterfice functional uspects of application, such performance, reliubility, usability, security, and tests are not generally run manually. In fact, it's difficult to
testing is the testing of non-
Non-functional tests
are performed after the functional tests, perform this kind of tests mamually So these tests are usually exccuted using tools.'nn Mols:.nnnhhhrre There are several types of non- functional testing. such as/lniaPerforman testing'SSecurity testing'nl nn oad testing'n SetinngnFFailone Failover testing'anongg Compatibiliyy MaatinggUusabliigg Bestingsn't performing tests, measuring the effectiveness of the tests is also important, and test coveruge tells the effecctiveness of your tests. You' can use programs called linters to check if the code of your software project meets the specified coding convention rules. A linter actually saves you lot of time as manually checking the code written by several developers is a very time-consuming process.
first rewrite (discarded to semantic loss): Non-functional lesting focuses on aspects like performance, reliability, security, usability, and compatibility. These tests are usually automated and include performance, load, failover, and similar evaluations.
query: What's a good prompt generate photorealistic objects Stable Diffusion?
original response: There are several versions of Stable Diffusion which each respond differently to prompts. The best prompt is also likely to vary depending on the object in question and other context. For these reasons it would be difficult for me to provide an exuct prompt. In general would recommend including terms like \"photorealistic"". 1 "ultrarcalistio", ""phooorraph!" or '"HD cameral".
first rewrite (discarded and each one can react quite differently the same prompt. Because of this variation, well the that the ideal prompt depends heavily the specific object you want to generate und the brouder visual context you have mind, it's difficult give single, universally "best" prompt. In general, though, if your goal is create photorealistic objects, it usually helps include deseriptive keywords such "photorealistie" "ultrarealistic" " "high-detail photograph," " worll-world lighting." " or "shot with HD camera." " These kinds of terms tend to encourage the model generate images with more lifelike appearance and stronger photographic qualities.
redundancy):
There many different versions checkpoints Stable Diffusion,
Figure 10: Failure cases identified by the generation + verification pipeline.
F.8 THE "SHORT BIAS" IN LIFT-PLUS
From the perspective of Response Length in Table it might appear that LIFT-plus exhibits smaller length bias. This is not the case. Our results demonstrate that while LIFT-plus reduces response length, it does so at the expense of disregarding the balance between response length and seman- tic quality, resulting in suboptimal performance in both semantic quality and adherence to length instructions. Specifically, our results indicate that DPO models trained with the LIFT-plus method tend to exhibit "short bias" phenomenon, where the model prioritizes generating shorter responses without considering semantic quality, given the "or less" instruction xy or the original prompt & As shown in Table and 15] both for Quality Win Ratio and Length Win Ratio, LIFT-plus significantly underperforms compared to R-DPO and Rc-BT. Notably, for Qwen2.5-7B-Bal Instruct, LIFT-plus's 's Quality Win Ratio is lower than R-DPO by 8. 73% and 8.47%, and lower than Re-BT by 13. 13.72% and 18.94%, respectively. A similar trend is observed with 1--ma--..--8B---. Furthermore, we plot the distribution of response lengths generated by LIFT-plus trained on Llama-3. 1-BB. --SB-Instruct under the length instruction 2 in AlpacaEval--I-pliee less benchmark, as shown in Figure IT As observed, the mean response length specified by the length instruction x; in AlpacaEval-LI-ge -i-l-LL-ssss-esss is 1 180.23 while the response lengths generated by LIFT-plus are all below 100. Furthermore, more than 90% of the responses have length under 64. This clearly demonstrates that the DPO models trained with LIFT-plus focus solely on the "or less"
and Qwen2.5-7B-
and 1-ama-3.1-BB-Inste
34

---

Published as a conference paper at ICLR 2026
length instruction 27, disregarding the semantic requirements of the original prompt and even overlooking the specific length constraint (word_jum) of the 2t itself, resulting in the "short bias".
140
120
Reponse Num
80
40
20
20-30
80.90
Repoose Length Range
Figure 11: Response length range generated by LIFT-plus trained on Llama-3. 1lamm-3..1-8-- 1..e....--BBBBB --sttrrutttttt using length instruction 2; in AlpacaEval-LI-pluss benchmark. The red dashed line r = 64 represents the 90% threshold, indicating that 90% of the responses have length less than or equal 64.
G LLM USAGE
We made limited use of the LLM exclusively for minor linguistic refinements. Specifically, the LLM small number of sentences and phrases to improve clarity and readability. The LLM did not contribute to research ideation, experimental design, analysis, the substantive writing of the paper. All conceptual, methodological, and technical contributions are entirely the work of the authors.
was used to polish the wording
35


Reviews:
[{"review_id": "Coq53Jc5xa", "reviewer": "Reviewer_aP85", "summary": "This paper introduces a framework (response conditioned Bradely-Terry) to improve the utility of reward models for preference learning by reducing their susceptibility to length bias and ensuring models better follow length instructions. The approach is validated for multiple models and datasets, and the results suggest the approach is effective for improving the robustness to length bias.", "questions": "- Avoid using subjective qualifiers in writing. For example, the reader can decide how “extensive” the experiments are.\n- Table 1 caption: explain what, e.g., accuracy is computed on — is it the accuracy with which the actual preferred response receives a higher reward from the reward model?\n- I was surprised to not see the seminal paper Deep Reinforcement Learning from Human Preferences by Christiano et al. cited. Especially when introducing RLHF.\n- In the random evaluation set, this is where responses are swapped for one another arbitrarily.  Is there a case for a truly random set where the token sequences are random and so a longer response might be pure gibberish?\n- For Figure 1(b) why is there so much variation in response length when the prompt is always just “empty prompt”?\n- Throughout the paper the incorrect opening quotes are used.\n- Line 193: “demonstrates the length bias” > “demonstrates length bias”\n- Line 195: the subscript “l” is being reused. It is used here for length, and previously for loss (in the context of a preferred / dispreferred preference pair).\n- Line 203: What is stated as being a hypothesis is not actually written as a hypothesis.\n- Line 257: “preferred than” > “preferred over”.\n- Line 257: “to modeling human preferences” > “to model human preferences”\n- Line 259: “assuming a underlining true” > “assuming an underlying true”.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is an important problem to tackle as length bias continues to be problem, where longer responses tend to receive higher reward regardless of the content.\n- The experiments documented in the appendix provide an in-depth analysis of the approach.\n- The formulation of the preference pairs is a nice idea. I initially had a concern when reading the paper that optimizing to simply constrain length might cause the model hack by doing something like generate responses in a more compressed form (e.g., Chinese characters offer a more condensed form than English). The formulation of the preference pairs is a nice way to prevent this.", "weaknesses": "- Only two model families are considered.\n- The largest model sizes tested are on the smaller size by the standards of today.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k4K1WfekTF", "reviewer": "Reviewer_WdGS", "summary": "This paper investigates the issue of length bias in RLHF, where reward models tend to favor longer responses rather than true human preferences. To address this, the authors propose a Response-Conditioned Bradley–Terry (Rc-BT) model. Building on this model, they introduce algorithms designed to mitigate length bias. Numerical results across several models and datasets demonstrate that the proposed approach can reduce length bias.", "questions": "See Weaknesses above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is overall well written and effectively motivates the phenomenon of length bias in RLHF.", "weaknesses": "I agree that length bias is a common phenomenon in RLHF; however, I remain unclear about several points:\n\n- What causes the length bias? Is it due to the training dataset, the scale of the LLMs, or the training procedure (e.g., lack of length regularization)? For instance, does increasing the size of the reward or policy model reduce this effect? From my experience, larger models such as GPT5 seem less likely to have this issue.\n\n- A following weakness I feel is that, the paper lack of a principle way to address the length bias problem as they do not address the problem by fining the cause of the problem but rather simply use a post-hoc method to address the problem. \n\nThe new model lacks novelty, as it appears to be a trick based on the BT model.\n\nRegarding methodology, the notation involving $\\pi$ is somehow confusing. What does $\\pi_{\\theta}(x, y)$ represent, and how does it differ from $\\pi_{\\theta}(y|x)$? From a practical standpoint, how should the trained policy be used for testing after training?\n\nThe experimental section compares only a few baselines, even though several new algorithms have been proposed based on DPO. I recommend including additional recent works:\n\n- https://arxiv.org/abs/2310.03716\n\n- https://arxiv.org/abs/2403.19159\n\n- https://arxiv.org/abs/2407.07880", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "otLJceLtOk", "reviewer": "Reviewer_tJtt", "summary": "This work tackles length bias in DPO. It first uses four focused experiments to show that (i) length bias genuinely exists, (ii) the evaluation set itself carries length bias, (iii) an implicit preference for longer answers ≠ the ability to follow explicit length instructions, and (iv) explicitly teaching length is easy but harms semantic quality. The experimental designs are insightful. Building on these findings, the authors propose a method that converts “implicit length bias” into an explicit understanding of length and disentangles it from semantic preference. They apply this idea to both reward modeling (RM) and DPO, and demonstrate effectiveness and generalization across multiple base models.", "questions": "See weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Clear motivation and an important problem. The paper targets a widely observed and practically consequential issue in RLHF/DPO—length bias. The authors establish motivation via a “falsify first, then model” storyline, using evidence to demonstrate the problem before proposing a solution.\n\n2. The four small studies form a coherent chain: (existence) → (evaluation bias) → (implicit ≠ explicit) → (explicit hurts semantics). This progression is logically tight and highly readable, making the diagnosis convincing and the need for a new method evident.\n\n3.  The core idea that explicitly modeling and disentangling a spurious format constraint from semantic preference appears portable to other instruction/format biases (e.g., politeness, number of bullet points, templated structure). This suggests a promising research trajectory beyond length alone.", "weaknesses": "1. 3.3 (length-compliance eval) construct validity: The eval set is built from model-rewritten pairs assumed to be semantically equivalent. Any residual semantic or stylistic drift can confound the claim “implicit length bias ≠ explicit length control.” Add a simple length-only heuristic baseline (choose the candidate that satisfies the numeric constraint) as a sanity upper bound, and include human spot checks for equivalence.\n\n2. 3.4 (explicitly teaching length)  might result in overfitting vs. distribution shift: The {x_l, y_w, y_l} format makes length the most salient signal. The quality drop might reflect not only “overfitting to length,” but also a mismatch between synthetic length-instruction training data and the de-biased quality eval. \n\n3. RC-DPO is essentially in the DPO family. Comparing mainly to ODIN (more of an RM-regularization baseline) is insufficient. Please add strong DPO variants such as SIMPO, CPO, etc, and evaluate on standard leaderboards.\n\n4. Both the de-biased quality set and the length set depend on strong-LLM rewrites, risking style/domain shift and label noise. Strengthen with: (i) a human agreement study on a sampled subset; (ii) cross-source replication (another rewrite model or a small human-curated slice); and (iii) reporting failure cases where “meeting the numeric length” reduces information density (add redundancy/compression-retention metrics).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)