Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidance
Authors: Zhuo Li
Pengyu Cheng
Zhechao Yu
FeifeiTong
Anningzhe Gao
Tsung-Hui Chang
Xiang Wan
erchao.zec
xiaoxi jiang
guanjunjiang
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
RLHF
Reward Hacking
Debias
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Reward models (RMs) are essential in reinforcement learning from human feedback (RLHF) to align large language models (LLMs) with human values. However, RM training data is commonly recognized as low-quality, containing inductive biases that can easily lead to overfitting and reward hacking. For example, more detailed and comprehensive responses are usually human-preferred but with more words, leading response length to become one of the inevitable inductive biases. A limited number of prior RM debiasing approaches either target a single specific type of bias or model the problem with only simple linear correlations, *e.g.*,  Pearson coefficients. To mitigate more complex and diverse inductive biases in reward modeling, we introduce a novel information-theoretic debiasing method called **D**ebiasing via **I**nformation optimization for **R**M (DIR). Inspired by the information bottleneck (IB), we maximize the mutual information (MI) between RM scores and human preference pairs, while minimizing the MI between RM outputs and biased attributes of preference inputs. With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations, broadly extending the real-world application scenarios for RM debiasing methods. In experiments, we verify the effectiveness of DIR with three types of inductive biases: *response length*, *sycophancy*, and *format*.  We discover that DIR not only effectively mitigates target inductive biases but also enhances RLHF performance across diverse benchmarks, yielding better generalization abilities. The code and training recipes are available at https://github.com/Qwen-Applications/DIR.

Published as a conference paper at ICLR 2026
ELIMINATING INDUCTIVE BIAS IN REWARD MODELS WITH ANFORMATION-THEORETI GUIDANCE
Zhuo Li Pengyu Cheng*., Zhechao Yu', Feifei Tong', Anningzhe Gao", Tsung-Hui Chang", Xiang Wan Erchao Zhao', Xiaoxi Jiang', Guanjun Jiang'
1 *Qwen Large Model Application Team, Alibaba SShenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen) *Corresponding authors.
ABSTRACT
Reward models (RMs) are essential in reinforcement learning from human feed- back (RLHF) to align large language models (LLMs) with human values. How- ever, RM training data is commonly recognized as low-quality, containing indue- tive biases that can easily lead to overfitting and reward hacking. For example, more detailed and comprehensive responses are usually human- mremanrreeerred but with words, leading response length to become one of the inevitable inductive biases. A limited number of prior RM debiasing approaches either target a single specific type of bias or model the problem with only simple linear correlations, e.g., Pearson coefficients. To mitigate more complex and diverse inductive bi- ases in reward modeling, we introduce a novel information theoretic debiasing method called Debiasing via Information optimization for RM (DIR). Inspired by the information bottleneck (IB), we maximize the mutual information (MI) between RM scores and human preference pairs, while minimizing the MI be- tween RM outputs and biased attributes of preference inputs. With theoretical justification from information theory, DIR can handle more sophisticated types of biases with non-linear correlations, broadly extending the real-world application scenarios for RM debiasing methods. In experiments, we verify the effective- ness of DIR with three types of inductive biases: response length, sycophancy, and format. We discover that DIR only effectively mitigates target inductive biases but also enhances RLHF performance across diverse benchmarks, yield- ing better generalization abilities. The code and training recipes are available attps::/githnni com http:////eeaapppoiattoooooo///.
INTRODUCTION
Aligning Large Language Models (LLMs) (OpenAl 2024 Touvron et al. 2023 Yang al. 2024
with human values is a fundamental technique to guarantee the helpfulness and harmlessness of LLM responses, which has been widely applied in various open-domain scenarios (Oyang al. 2022b Kimi et al., 2025) Touvron et al. 2023) Gemini
2025 OpenAl 2024 Hu et al. 2025 Dai
al.] (025). Toward more human- preferedee LLM behaviors, reinforcement learning from human
feedback (RLHF) (Ouyang et al. 2022b Rafailov et al. 2024b DeepSeck-Al, (2025) has become the
mainstream approach, which first trains reward model (RM) on collection of human preference response pairs, then scores the LLM's responses with the learned RM as the rewards to conduct reinforcement learning (RL) (Ouyang al. 2022b). Although dominantly applied in LLM post- training (DeepSeek-Al) 2025 Touvron et al. 2023), RL.FF has continuously been criticized by its training instability, which can easily lead to LLM's training collapse and overfitting (Rafailov et al. 2024b
Zhu et al. 2024 Yu et al. 2025).
Among the many factors leading to the training instability of RLHF, the issue of reward model hacking is non-negligible: due to the low quality of human preferences (Zeng et al. 2024| Liu et al.) 2025:
Wang et al. 2025b Liu et al. 2024a) that contains massive preference conflicts and inductive
Work done during an internship Alibaba.

---

Published as a conference paper at ICLR 2026
biases, the reward model can easily be mislead by irrelative data attributes instead of targeting on the real content quality (Skalse et al., 2025; Gao et al. 2023 Amodei et al. 2016 Du et al. 2025 Li
et al. 2025a). For example, annotators are always instrccted to choose more informative responses, whereas more detailed responses usually have longer response lengths. Learning on this biased human feedback dataset can lead the reward model to ignore the true response quality and only favor responses with longer lengths (Singhal et al., (2023). Besides the response length bias, stylistic and format patterns (Zhang al., 2025), and sycophantic phrasing (Sharma et al.) 2023 Denison et al.) 2024] have also been recently recognized as the typical inductive biases in reward modeling, which are unrelated to response quality but strongly correlated with the preference annotations (Sharma)
et al.
2023 Liu et al. 2024c). Learning with inductive biases can easily disrupt RMs" learning targets
and critically hinders the reliability and the generalization ability of RLHF (Gao et al. 2023 Coste
et al.
2023).
To mitigate inductive biases in reward modeling, a limited number of recent studies have made some preliminary explorations. Bu et al. (2025); Chen et al. (2024); Zhang et al. (2025) consider Pearson Coefficient (Benesty et al. 2009) as the bias measurement, which is then jointly minimized with the reward modeling loss. However, the Pearson Coefficient only captures the simplest linear correlation between RM and the bias attributes, which are not sufficiently applicable in more general scenarios. Shen et al. (2023) adds another RM head predict response length score, which only applicable with scalar types of inductive biases and lacks theoretical justification. |Wang et al. (2025a) introduces overly restrictive external constraints, such as enforcing distributional invariance by minizmizing the Maximum Mean Discrepancy (MMD) (Gretton et al. 2012) between the chosen and rejected distributions. This approach risks distorting the reward landscape by inducing a collapse in the scores of functionally distinct response groups. In contrast, methods utilizing general compression strategies, such as the Information Bottleneck (Tishby et al.) 2000) used in InfoRM (Miao et al., (024). cannot guarantee the mitigation of inductive biases, since no explicit constraint is applied to the biased attributes within the optimization.
To uniformly eliminate inductive biases from reward modeling with theoretical guarantees, we propose an citormmation-theoree debiasing framework called Debiasing via Information optimization for RMs (DIR). Inspired by the information bottleneck methods (Tishby et al., 2000), we model the complicated inductive biases between irrelevant attributes and human preferences by the concept of mutual information (MI) from the perspective of information theory (Kullback 1997). The proposed debiasing method maximizes the mutual information between the content quality of response pairs and the ground-truth preference label, while simultaneously minimizing the mutual information between the RM's preference prediction and the irrelevant bias attributes. To tackle the intractable MI calculation (Poole et al.) 2019), we estimated the above objective with two variational bounds: the Barber-Agakov (BA) lower bound (Barber & Agakov| (2004) for the MI maximization and the contrastive log-ratio upper bound (CLUB) (Cheng et al. (020) for the MI minimization. To further extend the method's 's applicability, we design a comparative regularizer that operates on relative bias attributes between response pairs, rather than on the isolated attributes of individual responses. This modification allows DIR to generally handle diverse and complex types of biases without distorting the underlying reward landscape, yielding broader application scenarios. We conduct extensive experiments both LLM capability benchmarks (e.g., GSM8K (Cobbe et al. 1021 MMLU (Hendrycks et al., 202T), ArenaHard (Li ct al.) 2024), and MT-Bench (Zheng et al. 2023) and reward model benchmarks (e.g., RMMBench (Liu al. 2024c) and RewardBench (Lambert et al. 2024)), under multiple bias settings including response length, formatting style, and sycophancy.
The numerical results demonstrate that the proposed DIR method consistently outperforms existing debiasing baselines, yielding more robust and reliable alignment of LLMs.
2 PRELIMINARY
Reinforcement Learning from Human Feedback (RLHF) has become one of the essential
training processes to align LLMs with human values (Ouyang et al. 2022a). With a well-learned reward model (RM) r y) scoring the degree of human preference of generated response y E y given input prompt x E X,, RLHF optimizes the LLM policy Te Te(y(z) with the follow objective:
y (m)))))) (r(x, y) 3 KL| (y) e
(I)
where T x)) is the initial model policy served as a reference, B > 0 controls the strength of a Kullback-Leibler (KI.) divergence (Csiszár 1975) between the reference model Ter(yyl>) and the
2

---

Published as a conference paper at ICLR 2026
learning policy Te( gz)) To train LLMs with the above objective, Proximal Policy Optimization (PPO) (Schulman et al. (017)) has been recognized as the mainstream optimization approach (OpenAl 2024 Bai et al.] 2023] Rafailov et al. 2024b). Group Relative Policy Optimization (GRPO) (Shao et al. 2024) further removes the critic model in PPO and uses simplified group-related advan- tage approximation instead, which has shown competitive performance with practically simpler
infrastructures (DeepSeck-A] 2025 Yang et al. 2025).
Reward Modeling targets learning the human preference distribution via a parameterized reward model (RM) rs X' x y - R, where ras (a y) is the predicted reward score of the input prompt and the corresponding response y. For every input æ, given a pair of response (y. y). we can calculate the "preference" by comparing the reward scores: if rxx y) > r(x, y). then y is predicted as a more "preferred" response than y (denote as y > y) and vice versa. We use a binary indicator
1 ly-y represent the event of "human preference": lyzy 1, if y > y: and ly-y 0, if y < y- Then the RM predicting preference ly-g can be regarded as drawing a conditional Bernoulli (Chen & Liu 1997) random variable from:
y. 4o (1y89 = 2.
exp(ro(x, y))
= of -) r0( B)). (2)
exp(ro(2, + exp(ro(a, Ø))
where o()) is a Sigmoid function. Note that the ground-truth human preference distribution p" (1 x, y. ÿ) is unknown. To optimize the reward model, we instead maximize the log- likelhhooh of go with a group of human preference data Dree = {1x,, y," v()
N
(RM(Ø)= = Eyg--rp' [og g0 (Iw-ola v...)
[log g0 (v,"" ull y." v()]
N
1 N
log a (rol - (x, v()),,
(by equation
where each nd > y is annotated by human judgment with respect to the response content quality. Equation is commonly recognized as the Bradley-Terry ranking loss (Bradley & Terry 1952).
Information theorettc Methods optimize deep models from the perspective of information the- ory (Chen et al. 2016 Hjelm et 2019 Yuan et al. [ et al. (00T)) The core methodology of information-theoree
methods modeling the feed-forward process of neural networks as an information channel transmission, where the correlation between different neural embeddings is measured by mutual information (MI) as:
(x; 3) = (() log p( y) p()))(()
KL [pl, (p(ax:))ll<<))(()))
where p(r, y) is the joint distribution, and p(æ:) and p(y) are the marginal distributions. Due its general ability to capture arbitrary non-linear correlations, MI has achieved considerable success as a learning objective in various deep learning tasks (Chen al. 2016 Belghazi et al. 2018 Hjelm et 2019). However, due to the intractable expectation w.r.f. p(æ, y). the exact MI value in equation is challenging to compute, especially when only samples from p( y) are provided. To address this, several approximation methods have been proposed to estimate MI from samples using tractable variational bounds (Oord et al. (2018, Cheng et al., (020 [Belghzi et al.) 202T]. Barber-Agakov (BA) bound (Barber & Agakov| (2044) provides simple lower bound approximation of MI, by introducing a variational approximation qo ( mr):
I [x;; v) 2 Ep((a)) [lo go x)) H [p) TBx v):
where H is the entropy of the ground-truth distribution p(æc, y)- Besides, Cheng al. (2020)
propose a variational contrastive log-ratio upper bound (CLUB) also utilizing the variational approxi- mation qe (eyy [)::
I (æ; 3) I E Ep(a.w)loog [log go (y [=) (p(w)r())) [log go (3) [=)] =: ICLUB (æ;
(6)
By minimizing equation 6 the amount of information between x and y can be effectively reduced.
We provide the proof of BA bound and CLUB in Appendix B. B.2 A well-known application of
3

---

Published as a conference paper at ICLR 2026
information ttheorrticcc methods is the information bottleneck (IB) Tishby al. 2000), which aims to learn a compressed but informative representation h of an input æ to the output as a trade-off between two MI terms:
min / (z; h) A (h: h
where hyper-parameter
controls the balance between compressing the input and retaining relevant information for the prediction IB has been recognized as a powerful tool for representation
learning and widely applied to diverse deep learning scenarios (Saxe al. 2019 Wan al. 2021 Federici et al. 2020).
3 METHODOLOGY
We begin by revisiting reward modeling from a perspective of information theory. Given an input query æ E X and a pair of responses y. 3 E Y, we denote as a concerned bias attribute with respect to (z, y. y) Our debiasing target is to learn reward model r. (2, 3) that produces predictions 1y>0 highly correlated with the content quality of response pairs (y: f) while eliminating any indication of the pre-defined bias attribute b. Motivated by the information bottleneck method in equation 7 we model the debiasing objective as maximizing the mutual information between the input response content and the RM preference prediction, while minimizing the mutual information between the RM prediction and the bias attribute:
max (1
リr yy x.
11 b):
(8)
Preference Term
Debiasing Term
where X > O is a hyper-parameter balancing the trade-off between preference learning and debiasing. Ideally, minimizing equation (8 encourage the reward model rg capture the true performance signal from the input triplet (z, y. y), while decreasing the reliance on the bias attribute b.
However, directly optimizing the mutual information- based objective is computationally intractable due to the difficulty in estimating mutual information in high-dimensional spaces (Poole et al. (219). Hence, we follow the prior works (Oord al. 2018 Cheng et al. 202T) and utilize the variational mutual information bounds (as in equation 5 & 6 to estimate the preference term and debiasing term of equation separately.
Preference Term Estimation. Instead of directly enlarging 7 (1 (I x. y. y). maximize its lower bound approximation by applying the BA estimator as in equation 5
1 (1yrvi vi y. 7) 2 Epp (w. s) [log go (1,>> g|a y: 9) + Hy"",
where p²(x, (æ, y, 1 1 is the ground-truth joint distribution of human preference training data, and H [p" is the entropy of the data distribution p* as a constant to the learning parameters. By equation 3 the expectation term in the right-hand side of equation Dis exactly the commonly used
Bradley-Tore Terry ranking loss (Bradley & Terry| (1922) of reward modeling (Azar et al. 2024 Cheng et al. 2024). Hence, minimizing the RM ranking loss actually maximizes the mutual information between the preference prediction 1 ÿ and the input triplet (ær, y, 7). encouraging the reward model rs to output higher score to the preferred response y. Therefore, given batch of preference data Drref = [ax,, y!",, 1y > y! B we maximize the following RM ranking loss to approximate
the preference term in equation 8 instead:
B
Ppreference (0)
1
Σ
log (ra(a,, (a(
(10)
B
Debiasing Term Estimation. Since the response pairs (æ, y, V) contains sufficient information to determine the bias attribute b, we can conclude that the RM forward process (b -> (æc y. y + H - 1 1y-g) is Markov Chain (Shannon 1948), where H [h (z. y), (a 7) is the last hidden states of the RM's transformer backbone. According to the data processing incquality (Shannon 1948) and the CLUB upper bound (Cheng et al. 2020 in equation 6 we have
((1 yi b) (H: < TeLum (H:b),
(11)
4

---

Published as a conference paper at ICLR 2026
ar
Prroferencel (6)
(a 3/²
Score Head
Score Head
A
H
Ax'
Reward Model Backbone
Debias Head
Reward Model
Debissiin (0.1))
(xr, y
(xr, y
Figure 1: The proposed DIR framework. The architecture of the reward model is considered as a transformer backbone and RM score head. The original RM ranking loss Prreterence (x) is calculated based on the outputs of the score head between each preference pair. The last hidden states (h h²) of the backbone are collected as the representation H. The debiasing loss Ldebiusing (Ø, s)) is computed between the inductive bias label and the output the debiasing head with parameter th.
where ICLUB( ( H; b) can be practically caleulated with variational approximation network (e within the data batch Preet
{(x,, U. y b,) b > y.
B
- ICCLUB H:b)
1
Σ
log qu b H.)
Σ B
1
log (b, H.) Lieblasing (Ø, v)), (12)
where H, 1 h!] Iho y.") (2; vf)l. By minimizing Debissiing (0) v/) as a upper bound estimation of (1 ツン-r y; b), we effectively reduce the correlation between the biased attribute b and the RM hidden representation h (x. y). Hence, the output RM scores r (x, y). as a deterministic function of h.g (a:, y). can remain unaffected from the inductive bias attributes b.
As proved in Cheng et al. 20200, the better zo (b,l H.) approximates the ground-truth data distribution p" (b. H. the more accurate CCLUB serves as the MI upper bound estimator: Therefore, during the optimization of Debiasing (, (b) as in equation we simultancously maximize the log- likelioddooo of qy [b: H.) within the batch samples {Hi b. J to maintain the accuracy of the estimator:
LEstimater (1b = B DE log qu (b, H.).
(13)
Overall Objective. Based on the above discussion, the original RM debiasing objective in equa- tion [ converts the following learning loss, which is practically tractable:
min Pereference (x) + Dreiiasiig (6) v)
(14)
The illustration of the coss-calculation pipeline is shown in Figure 1 To make sure Lecbiasing (Φ, v)) constantly have an accurate estimation upper bound (1y>9i b). we iteratively updated r& (æ, y) and (b(H) within each training batch as shown in Algorithm We refer to the proposed method as Debiasing via Information optimization for RMs (DIR).
RELATED WORK
Reward Hacking of LLMs. Reward hacking occurs when policy model exploits spurious
correlations or misspecifications within the reward function to achieve high scores without fulfilling
the intended goal (Pan al.) 2022), which has emerged
critical challenge in the stable and
effective RL training of LLMs [Langosco et al. 2023 Hurst et al. 2024) Kaufmann et al. 2024
Skalse et al. 2025 Zhang et al. 20255 Li et 20256)
RLHF. if the RM inadvertently learns
inductive bias from the preference data (c.g., a bias towards more verbose (Singhal et al.) (023). or
sycophantic responses (Sharma al. 2023)), the LLM being optimized will learn exploit these flaws, leading to degradation in true performance (Hurst et al. 2024). Prior work has sought to
mitigate reward hacking by empowering RMs, including better data curation (Liu et al., 2024a Wang
al. 2025b Dubois et al. 2024), model scaling up (Wang et al. 2025b), model ensembling Wang
et al. 2024). reward host-hoc calibration (Huang et al. (024). causal inference (Shen et al. 2023
Wang et al. 2025a), and disentangled reward learning (Bu et al. 2025 Chen al. 2024). Close to
our method, InfoRM (Miao et al.) 2024) employs an information theoretic framework to compress the entire latent representation of the RM backbone, indirectly removing spurious information.
5

---

Published as a conference paper at ICLR 2026
Algorithm I:
The iterative training processes of ra x. y) and 9: (:(
Input:
Preference pairs with bias attributes Drier [ax; y vi. b.) b,) learning rates ck,, 02 > 0:
while each training iteration do
Sample batch of triplets ((x;, y." y. b:)) ~ Diar:
Encode (at; yt" and (xt, v!) into embeddings H, = (m yl") (m v(]]: for bias estimator updating steps do
Calculate the estimator loss Lessiato (v)
Σ log qu (b. H.):
Update approximation qu (bb H) with 17 t- v0- 02 V Eatimator (1));
end
Calculate RM preference loss LPretenence (6) = B Ei= llog (ro yt - ro (2r.
10
Calculate RM debiasing loss
Lietissing (0. v) B > [log (b:; H:i) $ -ラ= log qu (b, (b))H
11
Compute RM total loss Crom (6, 2) = Liveterace (6) + + Luesssing (0: v);;
12
Update reward model rs with + o on V VoLrealee v):
13 end
Debiasing Methods of Language Models. Debiasing methods seek to prevent models from
learning and amplifying undesirable biases inherent in training data (He et al., 2019 Nam et al. 2020: Blodgett al. (020). The development of debiasing methods in natural language models
has evolved from the word level (Caliskan et al. 2017 Kaneko & Bollegala| 2019 Manzini et al.
2019). to the sentence level (Liang et al. 22020] Cheng al. 2021). and has gradually extended
to generative LLMs (Wang et |2023 Gallegos et al. (025) most of which focus on essential
social biases, including gender (Kaneko & Bollegala| 2019 Fatemi al. 2023), race (Caliskan et
2017), and age (Liu et al. (2024b). Core strategies for language model debiasing include adversarial training (Nam et al. 2020), causal inference (Zhou et al. 2023a), and iefformation-theoree
methods (Tartaglione et al. [221]] Liu et al. [023 Wang et al. (023). Unlike generative LLM debiasing, the
reward model debiasing methods focus on inductive bias attributes such as response length (Singhal ctt al. 2023), format (Zhang et al. 2025), and sycophancy (Denison al. 2024). For instance, Chen
ct al. (2024); Bu et (2025) and Zhang et al. (2025) suppress length or format bias by penalizing the Pearson correlation between rewards and bias attributes, only capturing linear dependencies and missing higher-order interactions. [Shen et al. (2023) use a two-head architecture for length bias but relies heuristic disentanglement without explicitly modeling the preference-bias relationship. Wang et al. (2025a) enforce counterfactual invariance via MMD, which may over-constrain the reward model and distort its signal.
EXPERIMENT
We first evaluate the effectiveness of our DIR method under three practical debiasing scenarios: response length, sycophancy, and format as the inductive biases, respectively. Then, we explore whether our method can alleviate the concurrent multi-bias problem.
Relative Bias Attributes. In our DIR framework, to minimize the correlation between the biased attribute and the RM hidden representation H, the variational approximation of qu (b)HH) is required. However, when considering response length as the biased attribute, directly predicting the exact number of tokens each response only based on the compressed representation H is very challenging. Therefore, instead of predicting the absolute value of response length, we introduce the relative bias altributes, which only consider the difference between chosen and rejected responses. For response length, the relative bias 1 (1ength(y)) > length(g)) E {0, 1}. indicating whether the chosen response is longer than the rejected one or not. Thus, the variational approximation for qu (b) H) becomes a binary classifier indicating the label of the relative bias.
Implementation Details. Based on the above discussion, we have converted the response length bias into a relative binary bias indicator. Hence, under all three debiasing setups, the bias attributes can be represented by categorical label, e.g., "longer/shorter" for response length, and "sycophantic/in- sycophantic" for sycophancy. Therefore, in the experiments, we implement the variational network qu for bias estimation as lightweight two-layer categorical classifier: qu (b H) Softmax (attmaxxMMMP1)))

---

Published as a conference paper at ICLR 2026
Scale Density vs. Linear- Scale Binned Mean Skywork
BT
PoE
ALBM
OURS
Mean Reward Score Reward Score
Tuers the Scaees
Teaes Couti thee Stato
Teen Ownrt Hor Seato
Them One tire hooko
Crem Socks
Figure 2: Evaluation of length bias in RMs RM-Bench. We calculated the correlation between response length and reward score for RMs trained with different methods. Our approach yields the lowest Pearson correlation coefficient (r 0.468), proving its effective ability in assigning more uniform reward scores.
Moreover, respond to the relative bias, further consider the linear transformation to the hidden H = (ar, y² (x, y')l as the difference A hh- = ha y²) - h y²) to emphasize the representation difference of the distinct features of the two responses.
Training Setups. Among all three RM debiasing scenarios. we use Llama3. 1-BB Instruct (Grattafiori & Team 2024) as the reward model backbone and the initial checkpoint. Besides, we fully fine-tune the reward models with global batch size of 128. The initial learning rate on for RMs is 2e - 6, which decays with a Cosine scheduler. For the bias estimator head, its learning rate a2 is set to 1e- 3. We fine-tune the policy models using Low-Rank Adaptation (LoRA) (Hu et al.) 202T) with a global batch size of 128 and bíloat16 precision for one epoch. Both the actor and critic models use learning rate of le 5. The LoRA configuration employs a rank of 8 and an of 32. The maximum context length is set to 4096, and the maximum generation length is set to 2048. The generation temperature for rollouts is set to 0.7.
5.1 LENGTII DEBIASING
Datasets and Models. We conduct the response length debiasing experiments by training RMs on the Skywork- Skywork-Preeerencee-88 Preference-80K-v00 (SK) dataset (Liu et al. (024a). Then, we test the debiased RMs performance in RLHF training with Llama3. 1-8B--nstttuct and
(Dong
(et al. 2024) as the initial policy and PPO (Schulman et al. 2017) as the learning algorithm for one epoch. PPO training prompts are 20K samples from the Alpaca- Alpaca-GPT4-EN dataset (Peng et al. 2023
Baselines and Evaluations. We consider the following baselines for reproducibility: (1) Vanilla
BT Baseline and popular open-source RM
Rewoorr---eaaa--aaaaaaaa 1-8B-v0.2 (Liu et al. 2024a): ;
(2) Length Debiased RMs, including PoE (Shen et al. 2023) and ALBM (Bu et al. 2025): ) (3)
Length
Penalty that directly resharps the reward during PPO by f(x y) = r(x, y) 0.001 len y
(Dong
ct al. 2024); (4) InfoRM (Miao et al. 2024 that is also designed from the information theory
perspective. Our evaluation protocol utlize few-shot settings for GSM8K (4-shot) (Cobbe et al. 2021
Race (3-shot) (Lai et 20) and TriviaQA (5-shot) (Joshi et al. 2017). All other benchmarks.
including Hellaswag (Zellers et al. (2019), IFeval (Zhou et al. 2023b). MMLU (Hendrycks et al.
2021 ProcessBench (Zheng et al. 2025), BBH (Suzgun et al. 2022). and HumanEval (Chen et al. 2021
are a zero-shot setting. We report accuracy as the primary metric for all tasks.
Reward Model Evaluation, Results and Analysis. We first evaluate the inherent length bias of RMs by analyzing the correlation between their scores and response lengths on the RM-Bench (Liu ct al. 2024c). As visualized in Figure 2 the standard BT RM exhibits strong, undesirable positive correlation between length and reward (P (Pearson = 0.533), which implies that even without an explicit preference for length in the training date]] the model still learns a spurious "longer is better" heuristic, highlighting fundamental issue in standard BT: the objective itself is susceptible to capturing such simple, non-causal patterns. Our approach demonstrates an effective ability to mitigate the length bias by achieving a Pearson correlation of just 0.468, the lowest among all evaluated methods. The quantitative advantage is further illustrated in the binned mean reward plots, where our flatter curve demonstrates the success of mitigating RM preferring longer responses. By learning assign
Average token number of (æ, ))" in the SK training set is less than (z,, v') ones (622.86 vs. 707.24).
7

---

Published as a conference paper at ICLR 2026
Table I: RLHF performance based on different length-debiased RMs. Bold scores mean the best. Underline scores are the second-best. The A indicates the performance change relative to the respective Baseline,
Benchmark
Llama3.1-8B 1-8B-Instruct
LP
ALBB
InfoRM
GSMSK
83.93
$4.61
83.62
15.97
84.08
53.78
84.84
74.83
78.17
17.79
77.18
78.85
76.74
79.08
Hellaswag
77.21
76.42
77.08
73.15
77.21
76.78
77.33
72.51
74.76
72.51
72.51
74.63
72.12
74.52
IFevll
72.83
71.72
65.47
74.12
MMLU
72.31
72.33
71.97
63.13
72.55
72.22
72.64
54.45
52.40
54.72
54.45
55.25
$4.97
54.30
ProcesiRench
25.39
24.49
28.50
24.91
26.
26.
27.73
4,46
10:31
9.68
7.84
10.85
3.24
13,82
Race
66.50
53.89
60.03
78.90
59.00
65.20
62.02
79.21
78.82
81.39
80.30
80.69
78.72
80.32
BBH
64.52
60.50
64.84
66.13
67.27
61.20
68
62.69
62.28
61.10
61.62
62.99
HvmanEval
70.12
68.29
66.46
60.57
65.85
70.12
70.12
60.98
57.32
59.76
59.76
60.37
57.32
63.41
TriviQQA
32.64
49.01
48.41
47.20
$2.00
30.56
55.86
48.53
52.86
52.34
48.32
51.52
48.16
52.52
Avg.
62.83
61.36
63.92
66.20
57.85
57.72
59.25
A
0.48
7 0.31
1 1.47
7 11000
[ 0.03
3:37
1:26
22117
0.86
2.00
0..44
1 3:35
(a) Win Rate ve. Baselines
(b) Resporize Length Comparison
Rate (89
800
65
$1.27+ e Ho1.2
ALSM WCRM
Slywork
Pal
Avenge Tokeens per Reepomse
700
Skywerk
ALSM
PSt
Ours
670
722
13
000
Buacine
11.3
36.501 27.7
35
10D
401
OpenLJama3-8BB-FT vs. Liama..1 1-88-innttt
CPT40-0314
OpenLLaams3-88--
Llama3, 1-8B-Instuct
Figure RLHF Evaluation on ArenaHard-vO.J with different length-debiased RMs. (a) Head-to-head win rates. Policies are PPO fine-tuned from specified base models (from left to right: OpenLlama3- 8B-SFT, Llama3. 1 Instruuutt,, and Llama3. 8B-Instruct, respectively) using five different RMs, which then act as challengers against opponents. (b) Average response length comparison.
scores more uniformly across different lengths, our method produces more reliable RM, preventing the policy from being misguided into generating unnecessarily verbose outputs during subsequent fine-tuning. We report the performance R.M-Bench in Appendix C.T
PPO Evaluation, Results and Analysis. We report the RLHF performance based the above length-debiased RMs across several benchmarks in Table which demonstrates that mitigating length bias does not compromise, and ideally enhances, the policy model's core reasoning and knowledge capabilities. Using Llama3. -88-Innsruut as the initial checkpoint, our method achieves the highest average performance of 66.20, significantly outperforming strong baselines. Furthermore, the trend of improved performance is consistent across different base models, as our method also secures the top backbone, indicating that our fine-tuning strategy
average score on the OpenRLHF. HH-.m-a-----aaB-.... successfully enhances objective performance by mitigating the length bias. We also assess the user preference for policies fine-tuned using different reward models and compare average response length on the ArenaHard-vO.1 benchmark (Li et al. (024). Figure 3 shows the head-to-head win rates of these challenger policies against strong opponents, as judged by (wwen-223B--Ae ------222--5-- (Yang al. 2025). The policy trained with our method consistently demonstrates the highest win rate across all conditions. For instance, in Figure (a), when fine-tuned on Llama3.1 -SS-IInttruct, ours achieves a remarkable 54.3% win rate against the baseline and 41.9% against GPT-4o-0314, Crucially, Figure 3 (b) reveals that the improved preference is achieved with expected conciseness. The policy guided by our RM produces shorter responses (e.g.. 679 tokens on the Llama3. 1 base) compared to policies guided by other RMS like ALBM (722 tokens) and the verbose original baseline (754 tokens). The better trade-off of a relatively higher win rate and lower verbosity shows that DIR can successfully guide PPO to produce more efficient and human-alip aligned policy, effectively overcoming the common "longer better" bias. In addition, we report the Win Rate performance on MTTBBench (Zheng et al.. 2023) and Length-Contro Alpaca (Dubois et al., 2025) in Appendix C.T from which we observe that DIR can yield policies that are preferred more often.
Additional Experimental Results. We visualize the PPO training dynamics metrics, such as RLHF Rewards and KL divergences in Appendix C.I which demonstrates that our RM helps make PPO training more stable with higher rewards. We analyze the training cost Appendix C22 which shows that the significant performance improvements do not come at the expense of prohibitive computational costs. We provide a detailed ablation study on and representation difference in Appendix C where the performance demonstrates the trade-off between preference learning and debiasing, showing better effectiveness of representation difference than concatenation. We also provide case study in Appendix E
8

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation on ArenaHard-v0.1 for poli- cies
Table 3: Length debiasing performance via DPO training. Bold is
best, underline is second-best best.
fine-tuned with DPO, DPO+LC and DPO+Ours.
Meta-L Jama3. Mama-L 1-8B- --strret
OpenRLHF-Llumi 3-B--SIT
Benchmark
DPO
+LC
+Ours
DPO
+LC
+Ours
OpenkLHH--Lama---88---
GSM8K-4shots
82.11
81.43
82.56
75.89
76.35
77.26
Base
Win Rate (%)
Length
Hellaswag
74.84
75.17
75.10
66.56
66.41
73,91
DPO
38.63
436.55
Ievval
73.57
74.12
74.68
38.26
35.86
41.59
+LC
40.96
407.23
MMLU
71.36
71.58
71.54
48.92
48.92
57.01
+Ours
45.27
404.61
ProcessBench
26.75
26.70
27.60
4.28
4.95
6.93
Race- 3shots
69.98
70.14
70.29
79.27
78.68
79.97
Mata-I.
BBH
67.22
66.81
66.99
59.73
60,36
62.34
v8 Base
Win Rate (3t)
Length
HumanEval
62,80
65,24
69.51
59.15
60.37
59.15
TriviaQA Sshots
55.12
55.15
55,29
47.45
47.69
48.93
DPO
44.06
700.87
+LC
46.57
691.43
Avg.
64.86
65.04
65.95
53.28
53,29
60.12
+Ours 49.09
684.67
0.118
1.009
0.001
1.84
Combination with Direct Preference Optimization (DPO). In addition to PPO, Direct Preference Optimization (DPO) (Rafailov et al. 2024a) has emerged as a powerful post-alignment method that directly trains the policy to increase the log-probability of the preferred response relative to the rejected one. Here, we explore whether our DIR can be effectively combined with DPO. Conceptually, DIR operates at the reward modeling stage and should not modify the DPO objective: DPO still optimizes the standard log- sigmoid preference loss, and our method only modifies preference signals by making them less correlated with inductive bias. Specifically, we add De (Ø) 1/) to DPO loss. Empirically, we conduct the corresponding experiments, which show that our method can also improve DPO's 's performance with controlled length. We provide training details in Appendix C.4 Results in Table 2 indicate that our method leads to a final policy with both a better win-rate and more effective length control, effectively boosting vanilla DPO and also outperforming a specialized Length Controlled DPO (DPO+LC) variant (Park et al., 2024). Results in Table indicate that our method also leads to a final policy with performance gains, especially for the SFT model that has not undergone a preference alignment. In summary, experiments on Table 2 and Table 3 suggest that debiased reward signals from DIR interact smoothly with DPO and effectively remove spurious gradients induced by length bias.
5.2 SYCOPHANCY DEBIASING
Datasets, Models, and Baselines. Sycophancy bias occurs when an RM learns to favor responses that agree with or flatter the user, rather than prioritizing factual accuracy and helpfulness. Motivated by (Sharma et al. (0023) Wang et al. (2025a), we create semi-sycophantic dataset by partially contaminating the HelpSteer3 dataset (Wang et al.] 2025b). Specifically, we artificially inject a sycophantic prefix (i.e., "Yes, you are right.") into a proportion 7 (e.g.. Y 40%) of responses in the training dataset. Within the contaminated subset, the prefix appears in the chosen response with probability (e.g., a 70%) and in the rejected response with probability 1 a 30%, where the remaining 1 - = 60% of the dataset remains unmodified and contains no sycophantic phrases. This contamination process creates a challenging, the sycophantic phrase serves as a strong but unreliable reward signal. Since other debiasing methods are either mainly designed for length bias (e.g., PoE (Shen et al. 2023), ALBM (Bu et al. (025)) ) and Length-Penalty (Dong et al. 2024)) or are not open-sourced (e.g.. CRM (Wang et al. 2025a)), we primarily compare our method against two key baselines: a standard BT reward model (Bradley & Terry 1952) and InfoRM (Miao et al. 2024).
environment in which
Evaluation, Results, and Analysis. To evaluate the models' susceptibility to sycophancy, we conduct an adversarial test. We take clean evaluation and create two versions: a "natural" version and a "sycophantic" version where the undesirable prefix is added to the rejected responses. We then measure the model's accuracy in correctly identifying the preferred response in both scenarios. A robust model should maintain its accuracy. whereas a biased model's 's performance will degrade when faced with the "flattering but wrong" responses. As shown in Table 4 the performance of the reward models varies under different settings. The BT model shows vulnerability to the bias, as its accuracy on natural examples is generally the lowest, particularly under high contamination. While InfoRM shows a clear improvement and greater resilience, our method demonstrates the most consistent and robust performance, which frequently achieves the highest accuracy across natural, adversarial, and overall settings, even under high contamination ratios. In summary, Table indicates that our explicit debiasing mechanism is effective at mitigating the influence of sycophantic signals, enabling the model to focus more on the intrinsic quality of the response. In addition,
9

---

Published as a conference paper at ICLR 2026
Table 4: Preference accuracy of sycophaney-dehiasee RMs under different contamination settings.
Settings
All.
Nat.
Adv.
7
BT
InfoRM
Ours
BT
InfoRM
Ours
BT
InfoRM
Ours
20%
30%
86.6
89.4
90.2
85.5
88.9
89.8
91.0
91.2
93.6
20%
50%
85.6
89.8
88.7
85.7
90.3
88.2
84,9
87.9
90,9
20%
70%
84,8
86.1
87.1
85,2
86.0
87,5
83.1
86.6
85.1
40%
30%
87.4
89.0
90.9
86.0
88,1
87.4
88.9
90.3
93.9
40%
50%
86.1
87.9
88.7
87.0
87.7
89.8
84.8
88.3
89.1
40%
70%
83.6
86.6
88.0
84.4
86.3
87.4
82.6
87.2
88.6
80%
30%
89.0
90.4
91.3
82.3
89.5
88.0
90.7
91.9
92.2
80%
50%
85.5
87.2
88.1
86.3
86.3
86.2
85.3
87.5
90.3
80%
70%
81.2
84.5
86.2
86.4
86.4
87.2
79.7
84,0
86.2
we experimentally find that DIR can effectively handle multple concurrent biases as detailed in Appendix C.5
5.3 FORMAT DEBIASING
Datasets, Models, and Baselines Zhang et al. (2025); Long et al. (2024) have shown that format biases, such as the use of lists, emojis, and boldface, are prevalent in human annotations and strong preference models. Hence, we process a format-biased dataset following the data generation protocol ofan al. (025)) The base preference dataset consists of 71.6K response pairs, obtained by filtering U1t ra aacaraeeee Feedback (Cui et al.) 2024) to retain only pairs with a human score difference exceeding 1.0. To introduce format bias, we augment this clean dataset with two types of synthetically biased examples: (1) 0.7% of training pairs where response wrapped in bold formatting is spuriously labeled as preferred over an identical unformatted version, and (2) 1.4% of pairs where a formatedd response is similarly assigned a false preference label. The overall training set combines the clean and biased subsets. We compare against three baselines under the same experimental setup as (Zhang et al. 2025): (i) standard BT, (ii) BT trained after removing all format-biased examples from the training data (denoted fT and (ii) the Format Decoupling (FD) method (Zhang et al. 2025).
Evaluation, Results, and Analysis. As reported in Ta- ble the standard BT model exhibits pronounced format bias, achieving win-rates of 89.0% and 92.5% for responses in Bold and List formats, respectively, providing strong ev- idence that vanilla BT conflates superficial formatting cues with response quality. The BTf variant, while partially mit- igating this bias through data filtering, suffers a substantial RewardBench, indicat-
Table 5: RM performance on both Bold and List format debiasing.
Metric
BT
BTf
FD
Ours
Win-Rate (%) Bold
89.0
49.0
50,5
51.2
List
92.5
52.5
53.0
52.0
drop in downstream performance ing that naive removal of format-biased samples compro- mises the model's ability to learn robust reward signals. In contrast, both FD and our method successfully neutralize format bias, driving win-rates close to the ideal 50% thresh- old. Crucially, our approach outperforms FD on the more challenging subsets of RewardBench. demonstrating superior generalization in high-stakes domains, which underscores that our method achieves a more favorable trade-off between format debiasing and preference learning.
RewardBench
(Filtered)
Chat
98.3
92,2
97.2
93.0
Chat Hard
71.4
64.4
72.8
80.1
Safety
83.1
75.5
82.9
89.6
Reasoning
85.1
81.4
89.7
92.2
6 CONCLUSION
We introduce Debiasing via Information optimization of RMs (DIR), novel information-theoreti method, to address the pervasive issue of inductive biases in reward modeling for RLHF By max- imizing the mutual information between RM scores and genuine human preference signals while minimizing the mutual information between RM predictions and biased attributes, DIR effectively disentangles genuine human preference signals from spurious correlations, e-g., response length, sycophancy, and format. Equipped with variational bounds and MI estimation strategies, our method handles non-linear and complex bias structures with theoretical rigor and practical efficacy. Extensive experiments across diverse LLM and reward model benchmarks demonsirate that DIR not only mitigates a broad spectrum of biases but also enhances the generalization and alignment quality of reward models, leading to more robust downstream performance. We believe DIR offers principled, scalable, and widely applicable solution for building more reliable and balanced alignment systems, paving the way toward more human- value-consistent artificial intelligence.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENT
This work is supported by Longgang District Special Funds for Science and Technology Innovation under Grant LGKCSDPT2023002.
ETHICS STATEMENT
This work aims enhance the fairness and reliability of LLMs by mitigating format biases, pre- venting models from "gaming" evaluations based on style over substance. Our method encourages more accurate assessment of a model's true capabilities. We acknowledge that our method only addresses the specific format biases targeted during training and does not mitigate broader societal or demographic biases. Furthermore, our ablation studies show that an overly aggressive debiasing coefficient (A) can create trade-off, potentially harming performance use public models and datasets, we recognize they may contain their own inherent biases. We believe our contribution is positive step towards more robust and transparent AI alignment.
simpler tasks. While we
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, 2016. URL http://ss/aa..v...//////////////
06565
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4457-4455. PMLR, 2024.
Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL
fttt:s://axxv..orggge
2209. 16609
David Barber and Felix Agakov. The im algorithm: a variational approach information maximiza- tion. Advances in neural information processing systems, 16(320):201, 2004.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pp. 531-540. PMLR, 2018.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: Mutual information neural estimation, 2021. URL
https: /tttss:/aarxxn //tp///aavvvvv/vv org abs/ 110100000000000 .40662
Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pp. 1-4. Springer, 2009.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. Language (technology) is power: critical survey of "bias" in NLP. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5474,, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/2020.//0020.cl/// main.485. URL http::ttt//tt /tt./ttte::///aaaaaaaoooooooo/rp 4857
acl main
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.

---

Published as a conference paper at ICLR 2026
Yuyan Bu, Liangyu Huo, Yi Jing. and Qing Yang. Beyond excess and deficiency: Adaptive length bias mitigation in reward models for rlhf. In Findings of the Association for Computational Linguisties: NAACL 2025, pp. 3091-3098, 2025.
Aylin Caliskan, Joanna Bryson, and Arvind Narayanan. Semanties derived automatically from language corpora contain human-like like biases. Science, 356(6334):183-186 2017.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang,
Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf, 2024. URL / http://aaavvvv/.a UR.pptpps///arvv...c////////// /ttp///xx.i....///////.......... 2000000022000203099
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol. Alex Paino, Nikolas Tezak, Jie Tang. Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021. URL btttts:///attiiii wrt.//o.rr//aae hacp://b//22/ http://ppgaabb.///a////////// 03374
Sean Chen and Jun S Liu. Statistical applications the poisson-binomial and conditional bernoulli distributions. Statistica Sinica, pp. 875-892, 1997.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbeel. Info- gan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pp. 1779-1788, PMLR, 2020.
Pengyu Cheng, Weituo Hao, Siyang Yuan, Shijing Si, and Lawrence Carin. Fairfil: Contrastive neural debiasing method for pretrained text encoders. In International Conference on Learning Representations, 2021.
Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Tianhao Hu, Peixin Cao, Nan Du, and Xiaolong Li. Adversarial preference optimization: Enhancing your alignment via rm-llm game. Findings of the Association for Computational Linguistics, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiihhrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL fittps://arxiv.
bbss/210.. 14168
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprint urXiv:2310.02743, 2023.
Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pp. 146-1455 1975.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafcedback: Boosting language models with scaled ai feedback, 2024. URL httpst:s:///:
Chongyuan Dai, Jinpeng Hu, Hongchang Shi, Zhuo Li, Xun Yang, and Meng Wang. Psyche-rl: Towards reliable psychological Ilms through unified empathy, expertise, and reasoning, 2025. URL
https::/ www./ba...................o./.ee 2508 10848
DeepSeck-AL. Deepscek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL. wtt.htttt:/aarxtee http:///wxiiv..og/aaa///0//// 12948
12

---

Published as a conference paper at ICLR 2026
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, et al. Sycophancy to subterfuge: Investigating reward- tewarritampering in large language models. arXiv preprint 0rXiv:2446.10162, 2024.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang. Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf, 2024. URL hhtps::
/ttppp:://rrx/e //tp///axvvvv//..///a//ee.eeavva orgg ht///bb//2440 2405. 07863
Yuhao Du, Zhuo Li, Pengyu Cheng, Xiang Wan, and Anningzhe Gao. Atoxia: Red-teaming large language models with target toxic answers. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 3251-3266, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979- 8-89176-195-7, doi: 10.18653/v 1/2025.122 http:/..innnnnn........ naacl. URL htt.ttts///aaaatho...oo org/2025. findings naaci. 1797
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024. URL htLtttts///axxet org. hbs////b//aa 2305.14387
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025. URL URL battps://arxiv: org/ abs abs/2404. 04475
Zahra Fatemi, Chen Xing, Wenhao Liu, and Caimming Xiong. Improving gender faimess of pre-
trained language models without catastrophic forgetting. Proceedings of the 6l Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1249-1262, 2023.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck. arXiv preprint arXiv:2002.07017 2020.
Isabel O Gallegos, Ryan Aponte, Ryan A Rossi, Joe Barrow, Mehrab Tanjim, Tong Yu, Hanieh Deilamsalehy, Ruiyi Zhang, Sungchul Kim, Franck Dernoncourt, et al. Self-debiasing large language models: Zero-shot recognition and reduction of stereotypes. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Human Language Technologies ( (Volume 2: Short Papers), pp. 873-888, 2025.
Linguistics:
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 10835-10866. PMLR, 2023.
Gemini. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL UKL/httpss///rxxvvv.or//// abtp///a..22222... 2256b006610 06261
Aaron Grattafiori and Meta Llama Team. The llama 3 herd of models, 2024. URL https
7/arxiv. www.ivv.orgaaaoo 240/ 240.221780
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL
http::/ http://mm.r..r/pppprrr.eme
/ 2a, html
He He. Sheng Zha, and Haohan Wang. Unleam dataset bias in natural language inference by fitting the residual, 2019. URL htt.:/ttss//aaxv..cr//aab///////
10763
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding, 2021. URL httts:///axxi
org/ http//bb/200000 03300
R Devon Hjelm, Alex Fedorov, Samuel Mavoie-Marcchllonn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representationns by mutual information estimation and maximization, 2019. URL httpt:htt//ee
wrw/abs. 808 06670
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
7/ //arxiv.orr/aaa www.ivvvvrraabbb///ae.oo 09685
13

---

Published as a conference paper at ICLR 2026
Jinpeng Hu, Ao Wang, Qianqian Xie, Hui Ma, Zhuo Li, and Dan Guo. Agentmental: An interactive multi-agent framework for explainable and adaptive mental health assessment, 2025. URL https?
//arxiv. www.iii.rrreaeee abs / 2508. 11567
Zeyu Huang, Zihan Qiu, Zili Wang, Edoardo M. Ponti, and Ivan Titov, Post-hoc reward calibration: ptt.tp:://axxxv.oo///aaaaa
A case study on length bias, 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL hwtttttt////xxi//////
abs/1705.03551
Masahiro Kaneko and Danushka Bollegala. Gender-preserving debiasing for pre-trained word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1641-1650, 2019.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. 2024.
Angang Kimi, Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, al. Kimi-vl technical report. arXiv preprint arXiv:2504.07411, 2025.
Solomon Kullback. Information theory and statistics. Courier Corporation, 1997.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations, 2017. URL https:://axiv.cet 04683
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. http::/::
org/abs/2403 13787
Lauro Langosco, Jack Koch, Lee Sharkey, Jacob Pfau, Laurent Orseau, and David Krueger. Goal misgeneralization in deep reinforcement learning, 2023. URL https::/aaxiv..rrg/////// org.aaxiiiirrggo/ooo////o//bb
2105.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hare and benchbuilder pipeline, 2024. HRRLhttps:
org abs / /200 222.6.11999. 11939
Zhuo Li, Yuhao Du, Jinpeng Hu, Xiang Wan, and Anningzhe Gao. Self-instructed derived prompt generation meets in-context learning: Unlocking new potential of black-box LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1884-18577 Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN doi: 10. 10.18653/v1/2025 2a12255.clloooo lon.loong.2. URL lttps://aclatthoooggy
acl0022.cclll0 927
Zhuo Li, Yuege Feng, Dandan Guo, Jinpeng Hu, Anningzhe Gao, and Xiang Wan. Aplot: Robust reward modeling via adaptive preference learning with optimal transport, 2025b. URLhttps
//arxiv. wexxiv.or//abl. wbwprg//b//////ee 2510. 10051111190000 10963
Paul Pu Liang, Irene Mengze Li, Emily Zheng, Yao Chong Lim, Ruslan Salakhutdinov, and Louis- Philippe Morency. Towards debiasing sentence representations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang. Shuicheng Yan, Yang Liu, and Yahui Zhou, Skywork-reward: Bag of tricks for reward modeling in llms, 2024a. URL
wtt. http:://aarrvv.orrr//////110 18451
hitpps:/aaae
14

---

Published as a conference paper at ICLR 2026
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, et al. Skywork-reward Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01332, 2025.
Dugang Liu, Pengxiang Cheng, Hong Zhu, Zhenhua Dong, Xiuqiang He, Weike Pan, and Zhong Ming. Debiased representation learning in recommendation via information bottleneck. ACM Transactions on Recommender Systems, 1(1):1-27, 2023.
Siyang Liu, Trisha Maturi, Bowen Yi, Siqi Shen, and Rada Mihalcea. The generation gap: Exploring age bias in the value systems of large language models. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 19617-19634, 2024b.
Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style, 2024c. URL https://arxiv
org/abs/ 2410. 16184
Do Xuan Long, Hai Nguyen Ngoc, Tiviatis Sim, Hieu Dao, Shafiq Joty, Kenji Kawaguchi, Nancy F Chen, and Min-Yen Kan. Lims are biased towards output formats! systematically evaluating and mitigating output format bias of Ilms. arXiv preprint arXiv:2408. 08656, 2024.
Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 615-621, 2019.
Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao, Inform: Mitigating reward hacking in rlhf via information-theorete reward modeling. Advances in Neural Information Processing Systems, 37:134387-134429 2024.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier, 2020. URL
2007.02550
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1eXXiv:807.03744 2018
OpenAI. Gpt-4 technical report, 2024. URL
08774
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022a.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022b.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecificaation Mapping and mitigating misaligned models, 2022. URL httth:tp://
hwapg///b////// /ttp://///...2......... 03544
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in
direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 4998-500 5017, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/633/122222 http://///0....mmnnnnaaaa....99 URL htt.hhtttps:/aclaattooooooo .wg.gy..orroooo org//024..0000000 2024. f findi ins.nnnnnnnnll.....m 2977]
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:23004.03277, 2023.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International conference on machine learning, pp. 5171-5180. PMLR, 2019.
15

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model, 2024a.
/ttsstsss////xi..ooro wttp:rrg.aaaa....a////a/ 18290
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. URL https://arxiyy http://ttai...aaaaaa//aaaaaaaaa/ wwt/aas/ga///ooooooo...oo.ae
02054
Andrew Saxe, Yamini Bansal, Joel Dapello, Madhu Advani, Artemy Kolchinsky, Brendan D Tracey, and David D Cox. On the information bottleneck theory of deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12)1224000, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL
170770677777077
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379-423, 1948.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li, Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL
orw.orgggaao.... /abs//
24C2. 03300
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310. / 2023.
Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. 2023.
arXiv preprint arXiv:2310.055
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking, 2025. URL htttps://axxi... org/ hatp://aab/222.11122
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thoughe can solve them, 2022. URL https: httpp:::///xxxv.ororo
Enzo Tartaglione, Carlo Alberto Barbano, and Marco Grangetto. End: Entangling and disentangling deep representations for bias correction. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13503 13512. IEEE, June 2021. doi: 10. 109/cvp446437.2021. 01330. URL httpt:ttp//..../// doi httt::://aa.dal..org100000000000 wtt.://g/g...11////////..3.
2021. 01330
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, 2000.
URL httts://aaxii: org/ abs wwtp///yssss//aaaaaaaaapaaaaes/o 00044057
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Zhibin Wan, Changqing Zhang, Pengfei Zhu, and Qinghua Hu. Multi-view information bottleneck representation learning. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 10085-10092, 2021.
clommation-bbttleneckk
16

---

Published as a conference paper at ICLR 2026
Chaoqi Wang. Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language model alignment. arXiv preprint 3xiv:2501.096200 2025a.
Haoxiang Wang, Wei Xiong. Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objectivv reward modeling and mixture-of-eppers.. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582-10592, Miami, Florida, USA. November 2024, Association for Computational Linguistics. doi: 10. 10.18653/1120244 10o.8866//1/2224.oodnggo-goooo Saddings-emnlp.66 URL http:://aelantto.oogg
org/2024. f find wiwiiinngneeiinn
6207
Rui Wang. Pengyu Cheng, and Ricardo Henao. Toward fairness in text generation via mutual information minimization based on importance sampling. In International conference on artificial intelligence and statistics, pp. 4473 1485-4485. PMLR, 2023.
Zhilin Wang, Jiaqi Zeng. Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, and Oleksii Kuchaiev. Helpsteer3-preferenee Open human-annotated preference data across diverse tasks and languages, 2025b. Uttptt :/ htt/aaxiv.orrg//aaa
22555.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 151 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang. Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL httttsss//aaii...t.//////tt org. abs aw05 09388
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint 2rX1v:2503.14476 2025.
Siyang Yuan, Pengyu Cheng, Ruiyi Zhang, Weituo Hao, Zhe Gan, and Lawrence Carin. Improving zero-shot voice style transfer via disentangled representation learning. In International Conference on Learning Representations, 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. /ttps:////// wwtpa/wwwwww..//////////./// 07830
Dun Zeng, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu. On diversified preferences of large language model alignment. In Findings of the association for computational linguistics: EMNLP 2024, pp. 9194-9210, 2024.
Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, and Tong Zhang. From lists to emojis: How format bias affects model alignment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties \Volume I: Long Papers), pp. 26940-26961, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979 0779--889918 7-7---89976--51--- doi: 10. 1/8866611/22255 2w..22225..clloog... 1308. URL http:p:///ccaaatooooooorg///////
acl long
1308/
Yuze Zhao, Jintao Huang. Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Hong Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, and Yingda Chen. Swiftta scalable lightweight infrastructure for fine-tuning, 2025. URL htt.t:://///// arxiv. /ttp:p/////ax....r////bb... 2408. 75517
17

---

Published as a conference paper at ICLR 2026
Chujie Zheng, Zhenru Zhang, Beichen Zhang. Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jin- gren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical reasoning, 2025. URL DRLpttps:: http:p::/aaxiv..rrte org. abs http//ab////aa..// 24552.6222.6
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging juddge with mt-bench and chatbot arena, 2023. URL /tR/attps///ar....00//// abs/2305 05685
Fan Zhou, Yuzhou Mao, Liu Yu, Yi Yang, and Ting Zhong. Causal-debias: Unifying debiasing in pretrained language models and fine-tuning via causal invariant learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers). pp. 4227-4241, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. (0.18663/1112023acclloo 232. URL htt.thttts///eetmm //cl:
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma. Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction evaluation for large language models, 2023b. URL https: / wtt///a.vvvv..////v wttgg/as/////e 2311 07911
Banghua Zhu, Michael I Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024.
Ethem Yağız Çali and Talha Rüzgar Akkuş, Enhancing human-like responses in large language models, 2025. URL httpps::/aaxii..cttttt wttp://wwrg.bbb........///
18

---

Published as a conference paper at ICLR 2026
A USAGE OF LLMS
In the preparation of this paper, we utilized Large Language Models (LLMs) solely for the purpose of grammatical polishing and text refinement of the manuscript content. Specifically, the LLMs were only used to optimize the clarity, fluency, and grammatical accuracy of the written text.
All content polished by LLMs underwent thorough manual review and verification by the authors. We carefully checked the polished text to ensure its consistency with the original research intent, accuracy of scientific facts, and compliance with academic integrity standards. We confirm that we take full responsibility for all contents of the paper under our names, including the parts that underwent LLs-sssisted grammatical polishing.
B BOUND PROOF
B. PROOF THE BARBER-AGARE (BA) BOUND
We aim to prove that for any variational distribution go (y(x), the mutual information [(x; is lower-boute bounded by:
y(x;y)= 2 (oog (o(y|>)] + H [p] =: TBA
(15)
where [p) is the entropy of the ground-trum distribution p(x, 3). By the definition of mutual information, we have:
T(x;y) H(y) - H(y Hyy) + (ooxx))loopyyar p(y)))]
(16)
Consider the expected Kullback-Leibh (KL) divergence between the true conditional distribution p(y x) and the variational approximation qe (y [x), we have:
DxL (p(v|x) qo ( Ep(x,)) log p(y[x) qo (
px())
2
(17)
By the linearity of expectation, the inequality 17 can be rearranged as:
Ep(xy) (ly xx] 2 Ep(xyy (log go (y//x)).
(18)
Then substitute equation 18 into the definition of I(x: y), yielding:
T(x:y) 2 H [p] + (p((x.) (log (y|x)).
(19)
which completes the proof. The bound is tight if and only if go x) p(y)x) almost everywhere with respect to p(x,y). y).
B.2 PROOF OF THE CLUB UPPER BOUND
We aim to prove that for any variational distribution qo (1y (z). the mutual information I (æx; y) is soper-bounded by (club(x; y). We begin with the definition of mutual information:
I (æ; y) (.p(()))) [log m] Ep(r)) [log p())
(20)
Let's focus on the second term, which is the negative marginal entropy + (z)). We can express the marginal distribution p(y) by marginalizing out a::
p(y) Ep(ar) [p [x)]
(21)
where x' is random variable drawn from the same distribution æ, but is independent of the æ in the first term of equation 20 Substituting equation 21 into the entropy term:
Epig) [log p(y) = - -Ep(s) [log Ep(c') [p(r]>=)]]]
(22)
Since the logarithm is a concave function, we can apply Jensen's inequality, which states that [2)) < (o](E]2)) and implies [2] - [[]o)(Z)) Applying this, we get:
- Ep(x) [log Expcr [pl)(=))]]]] < - EEp()) [Bp(a') [log p(y)|==)]] (ppe (og (logp(y/a")) (23)
19

---

Published as a conference paper at ICLR 2026
Now, substituting equation back into original MI expression equation 20 obtain upper bound on the mutual information:
I (a;; y) Ep(a.y) [logp(ylae [[y]))) (p(e))p())) p(yla()
(24)
Note that the second expectation is over the product of marginals p(a))((): The inequality equa- tion 24lholds for the true conditional distribution p(y) æ). The CLUB bound replaces p(y (x) with the variational approximation go ( (z). The key insight from Cheng et al. (2020) is that the difference between the true bound and the variational bound is an expectation of KI.-divergences, and the pro-
posed variational form serves as a practical, sample-based upper bound for minimization. Therefore, we use the variational form our tractable objective:
[x: y) I Ep(a. qo [ Ep(rr)(t)) [lo (o(())))] =: ICLUB (2; y);
(25)
which completes the justification for using ICLUB as an upper bound for mutual information mini- mization.
C EXPERIMENT
C.I LENGTH BIAS
Training Seftings. For our PPO experiment, we fine-tune two distinct models using 20,000 samples from the alpaca-gpt4-dee gpta-data-n dataset (Peng et al. (203)). The first model, Llama3. Instruct (Grattafiori Team 2024), has undergone prst-training that includes both DPO and RLHF. The second, DDenR--H--Laama--8---
(Dong et al. 2024), is an instruction-folloiii built upon Llama3-8B- 8B-Base, without the RLHF post-training stage. We conduct the PPO training using the ms-swift (Zao et al. 2025) framework with its default PPO training configuration.
version
Baselines. We mainly consider the following baselines due to the reproducibility: (1) Vanilla BT Baseline and popular open-soures RM
1.t.maaa...------- 1-88-v0.2 (Liu al. 2024a); (2)
Length Debiased RMs, including PoE (Shen et al. 2023) and ALBM (Bu et al. 2025); (3) Length
Penalty that directly resharps the reward during PPO by F(ar, y) r(æ, y) 0.001 len(y) (y) (Dong
ctt al. 2024); (4) InfoRM (Miao et al. 2024) that is also designed from the information theory perspective.
Performance on RM-Bench. We further evaluate our debiased reward models on RM-Bench Luu al. (024c), a comprehensive benchmark that assesses model capabilities across four distinct domains, including Chat, Math, Code, and Safety with three difficulty levels: Hard, Normal 1, and Easy. As shown in Table our DIR framework consistently outperforms several strong baselines in terms of overall performance. Our primary variant, denoted Ours-1.0, corresponds to the optimal trade-off point identified in our ablation study (A = 1.0), which achieves the second- highett aggregate score of 69.35, reflecting well-calibrated balance between debiasing and generalization and indicating that DIR enhances the reward model's 's discriminative capacity on core reasoning tasks without substantially degrading its general-purpose alignment.
When we increase the debiasing strength to À = 10.0, the resulting model Ours- 10.0 achieves the highest overall score of 70.18. The most pronounced improvement occurs on the Hard subset, where performance surges to 64.41, surpassing the next-best method by over 16 points. Such a substantial performance improvement suggests that by explicitly suppressing reliance on superficial bias through the DIR mechanism, the reward model is better able to attend to nuanced, content- based indicators of response quality, particularly those that are critical for evaluating complex or challenging prompts. Moreover, Ours- 10.0 achieves the top scores in both the Chat and Code domains. However, this stronger debiasing comes at a cost: performance on the Easy subset declines relative to weaker debiasing settings. On such instances, where simple heuristics often suffice for accurate judgment, the aggressive removal of bias signals appears overly restrictive and counterproductive In summary, these results demonstrate that DIR not only enhances the overall capability of the reward model but also offers a tunable mechanism to prioritize robustness on challenging tasks over simpler ones, howcasing the flexibility and effectiveness of our approach.
20

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison on RM-Bench. Best results are in bold. Second-performancee is underlined.
Method
Chat
Math
Code
Safety
Hard
Normal
Easy
Total
BT
64.69
61.21
51,41
95.11
42.76
72.30
89.24
68. 10
PoE
67.70
61.23
51,51
95.51
44.94
73.17
88.86
68, 99
ALBM
64.57
58.48
52.34
95.21
47.88
71.50
90.32
67.40
Ours- 1.0
68.91
61.81
$1.56
95.13
47.88
73.59
88.93
69.35
Ours-10.0 71.23
61.59
52.73
94,91
64.41
71.29
74.85
70.18
Table 7: Win rate (%) performance comparison on MT-Bench.
Win Rute (%) (vs. Base)
Base Model
OpenRLHH--Loma---0----
Meta-Llama3..1 1-1timmm...----mmmm -nSstrstructt
Ours PoF
56.25
56.88
48.75
51.25
Skywork
49.38
51.25
ALBM
53.75
50.63
InfoRM
46.88
51.88
Performance on MTT-Bench and AlpaceEval. For MT-Bench (Zheng et al. 2023), we report the win rate of each RM-guided policy against its own base model, using the standard MT-Bench LLM-as-a-judge setup. As shown in Table our method ("Ours") achieves the highest win rates on both backbones (56.25% vs. 48.75-53.75%4 for OpenkLLHF--Laaa----B-- 50.63-51.888 for Meta-Llama3. 1-8--...-------tteeee indicating more improvements open-ended, multi- turn dialogue quality.
and 56. 56.88% vs.
For Length Controlled AlpacaEval, we follow the length-controlled protocol of Dubois et al. (0255 and report both raw win rate and length-controlled win rate over the base model. On Meta-Llama., I- 8B-Instruct, Ours achieves the highest scores on both metrics. On DpenRLHFF-Llama---9B- Skywork attains a slightly higher raw win rate, but Ours achieves the best length-controlled win rate, which is consistent with our goal: once the confounding effect of response length is controlled for, our debiased RMs yield policies that are preferred more often, demonstrating better alignment that is not driven by verbosity. We will include these M'T-Bench and AlpacaEval results and their analysis in the revised version.
PPO Training Monitoring. Figure presents three key metrics for monitoring the PPO training process. The left plot (RLHF Reward) evaluates the final quality score of the model's outputs. with higher values being better. The middle plot (KL Divergence) measures how much the learned policy has deviated from the initial reference model, indicating the extent of exploration. The right plot (Approx. KL) shows the magnitude of each policy update, serving as a critical indicator of training stability. Our policy model demonstrates a better balance across these metrics by achieving top reward score that significantly outperforms all baselines. Concurrently, our KL divergence is maintained at a moderate level, suggesting effective exploration without catastrophic deviation from the base model's 's capabilities. Most importantly, our method exhibits the lowest and most stable Approx. KL, which proves that the training process is exceptionally smooth and reliable. In summary, our approach successfully boosts performance while ensuring unparalleled training stability.
C.2 RM TRAINING COST ANALYSIS.
We analyze the computational overhead in terms of GPU memory consumption and training time, with detailed comparison presented in Table 9 We use 8 GPU cards with full parameter training and DeepSpeed Zero- Z (Rajbhandari al. 2020). Our approach demonstrates highly comparable resource efficiency to existing methods. Specifically, the GPU memory usage of our method (57.22GB) is only marginally higher than the baseline (56.80GB) and on par with other techniques like ALBM (56.88GB). Regarding training time, while method (67.09 minutes) requires moderate increase compared to the simpler baseline (50.46 minutes), DIR remains competitive and aligns closely with other advanced methods such ALBM (68.21 minutes), which shows that the significant
21

---

Published as a conference paper at ICLR 2026
Table 8:
Performance
comparison
Length
Controlled
AlpacaEval
against
apt4-1106 peee-epreee.....
Base Model: Meta Llama3. 1 -Ss-nuct
Methods
Raw Win Rate (%)
Length Control Win Rate (%)
Ours
31.30
PoE 26.58
11.41
Skywork
29.38
13.21
ALBM
26.83
10.61
InfoRM
25.22
11.02
Base Model: OpenRIHH Laam--3-8--SFT
Methods
Raw Win Rate (%)
Length Control Win Rate (%)
Ours
9.50
5,46
PoE
7.14
3.28
Skywork
10.19
3.93
ALBM
InfoRM
5.84
3.65
RLHF Reward
KL Divergence
Approx. (Update Step)
Klppllcc II raferance)
20
0.0014
60
10
40
Approx.
0.0012
20
0.5k Training Steps
0.5k Training Stops.
Training Steps
Ours
InfoRM
PoE
5K
ALBM
Figure 4: PPO training dynamics across key metries. Our RM obtains higher policy score and demonstrates better training stability.
performance improvements offered by our approach do not come at the expense of prohibitive computational costs, establishing our approach as practical and efficient solution.
C.3 ABLATION STUDIES UNDER LENGTH DEBIAS
Ablation Study on Representation for Debiasing, In this section, we investigate the influence of difference-based representation Ah. h" h as input to the variational debiasing network ly. compared with the representation of the concatenating form nh h']. We compare both vari- ants on RewardBench-vi and RM-Bench. Results in Table show that the difference-batt variant slightly outperforms the concatenation-based one across most domains and difficulty levels. accuracy on Chat Hard improves from 78.9% to 83.6%, and on Reasoning from 88.8% to 90.0%. On RM-Bench, the Chat score increases from 63.9% to 66.8%. Performance on other subsets remains stable, indicating trade-off in generalization. From an efficiency standpoint, the difference operator preserves the embedding dimensionality. whereas concatenation doubles The reduced input size lowers both parameter count and GPU memory usage during training. Given its empirical advantage, theoretical grounding, and computational efficiency, we adopt representation difference as the default input formulation in DIR.
Empirically, difference-based approach yields clear gains. On Rewa RewardBench
Ablation Study on Debiasing Coefficient X. The hyperparameter in Equation 14 governs the trade-off between the standard preference learning objective (Epreference) and our information- theoretic debiasing objective (LDebiusing). To analyze its sensitivity, we tested range of values: E {0.1, 0.3, 0.5, 1, 2, 2, 103. The results, visualized in Figure 5 reveal a clear trade-off.
As shown in the figure, when À is too small (c.g.. 0.1). the debiasing signal is insufficient. The model behaves similarly to a standard BT model. exhibiting a high bias metric (c.g.. high Pearson correlation with bias attribute) while achieving good performance RewardBench. Conversely, when À is
22

---

Published as a conference paper at ICLR 2026
Table 9: Training cost comparison.
Method
GPU Memory
Training Time
Baseline
55.08GB
50.46m
PoE
56.80GB
55.35m
ALBM
57.22GB
78.21m
InfoRM
$7.99GB
75.21m
Ours
56.88GB
67.09m
Table 10: Ablation study on the representation format for the debiasing module. We report accuracy (%) on RewardBench-vl and RM-Bench. The difference-based approach consistently outperforms concatenation, especially on challenging conversational and reasoning tasks. Best results are in bold.
RewardBench-vl (Ace %))
RM- RM-Bench (Ace %%
Method
Chat
Chat Hard
Sufety
Reasoning
Chat
Math
Code
Safety
Concat (ih"); : h'))
93.3
78.9
90.9
88.8
65.9
60.8
52.6
95.0
Difference (Ah)
94.1
83.6
89.7
90.0
67.8
61.1
52.4
95.2
too large (e.g., 10), the debiasing objective dominates the training. The resulting "over-correction" successfully minimizes the bias metric but severely compromises the reward model's ability to learn true preference signals, leading to significant drop in RewardBench accuracy. Moreover, we observe that À 1 strikes optimal balance. At 1, the bias metrie is substantially reduced, while the preference learning performance on RewardBench is maximized. The behavior demonstrates that the proposed method can effectively neutralize spurious correlations without damaging and fact while enhancing the reward model's core capabilities. Therefore, all main experiments in this paper
use
Trade-off Analysis for Debiasing Coefficient
94
Performance (Acc %)
0.550
RewardBench Accuracy (%)
93
Length Correlation
0.533
92.1
0.525
82
91.5
95.0
0.810
91.2
0.500
Length Correlation
0.495
0.475
90
0,480
89.5
0.468
89
0.450
88
Optimal À
0.440
0.425
87
0.415
0.400
0.406
(Baseline)
0.1
0.3
0.5
2
5
10
Debiasing Coefficient (A)
Figure 5: Ablation study on the debiasing coefficient X. The plot shows the trade-off between preference learning performance (RewardBench Accuracy, blue) and the bias metric (e.g.. Pearson green). achieves the best balance.
C.4 EXPERIMENT ON DPO
Specifically, we adopt the ms-swift framework with its default DPO training configuration on Liiman-Like-DPO-attass (Çalik & Akkuş, 2025), based both OpenRRLHF-Llmma-------
and Meta-Llama3.1 Instrucct models, where DPO 3 = 0.1, debias factor X = 1. We train epoch and evaluate the performance the final checkpoint. LLMs toward generating more human-like responses, which includes 10,884 samples across 256 topics, covering technology, daily Life, science, history and arts. We evaluate the performance
is created to fine-tune
23

---

Published as a conference paper at ICLR 2026
ArenaHard-vO.. and several popular benchmarks. For baselines, also compare with the conghh-oottollled DPO method (Park et al. (02222 which disentangles the length from the quality to explicitly avoid the policy model from preferring the longer response DPO training.
C.5 CONCURRENT MULTI-BIASES BIASES
Real-world datasets frequently exhibit multiple concurrent biases. Hence, we investigate whether DIR can effectively mitigate such co-occurring biases, specifically, length bias and sycophancy bias, simultaneously. Concretely, we extend DIR to length and sycophancy biases by introducing two independent mutual information terms, each with its own debiasing head Wiength and Wsyco- respectively:
(To = Preeference (0) + Alength Debinsinn (0, thhnngh) + Asyoo Lechissing (Ø, Vssyoo)
(26)
where Agen and Auyco are set to 1.0. We follow the setting in Table 4 training -lama-3.1-88-Inaa --Istuuttt on the HelpSteer3 dataset under two sycophancy contamination configurations (y 40% and 80%, = 70%), where a larger indicates a more challenging setting. All models (B'T, length-only DIR, and length+syco DIR) trained for epoch on the same data, and we report results using the final checkpoint for fairness and convenience.
As shown in Table 11 on RM-Bench, the joint model (Ours-Len-Syco) achieves the best overall performance and the largest gains on the hardest subset (e.g., at 7 = 40%, Total: 67.25 - 69.96, Hard: 39.88 + 46.85 vs. BT), while still clearly reducing the Pearson correlation with length relative to BT, confirming that length bias is mitigated even when sycophancy is also debiased. We also observe that the length-only model (Ours-l.en) attains the lowest length-reward Pearson coefficient, but somewhat surprisingly, the joint length+-yyoophnccy debiasing (Qurs-LLe-SSyen yields the best overall R.M-Bench performance, suggesting that debiasing multiple biases together could also help lead to more balanced reward model. As shown in Table 12 sycophancy stress tests, debiasing only sycophancy (Ours-Syco) gives the strongest syco robustness, as expected, but the joint model (Ours-Len-Syee -Syoo still substantially outperforms BT all sycophancy metrics (All/Nat/Adv.) across both y settings, while additionally reducing length bias. summary, a second debiasing term leads controlled trade-off, not conflicting gradients: both biases improved over BT, and overall RM quality remains strong.
Table 11: Performance comparison of concurrent multi-bias experiments on R.M-Bech. Best results are in bold.
40% 70%
Chat
Math
Code
Safety
Hard
Normal
Easy
Total
Pearson Coefficient
66.58
64.17
53.70
84.53
39.88
72.82
89.04
67.25
0.4807
Ours-Len
66.93
64.59
53.12
89.14
44.25
72.43
88.65
68
0.4235
Oury-Len-Syco
70.80
65.07
55.26
88.71
46.85
75.08
87.96
69.96
0.4446
= 80%, a 70%
Chat
Math
Code
Safety
Hard
Normal
Easy
Total
Pearson Coefficient
BT
65.37
63.71
53.12
80.85
37.58
70.96
88.75
65.76
0,4666
Ours-Len
68.91
64.25
53.75
87.23
44.78
73.09
87.72
68.
0,4081
-ury-Len-Syyo
68.39
64.92
54.04
88.06
45.15
72.92
88.50
68.85
0.4235
Table 12: Reward model accuracy (%) on the concurrent multi-bias under different contamination
settings.
All
Adv.
BT
Ours-Syco
Ours-Len-Syco
BT
Ours-Syco
Ours-Len-Syeo
BT
Ours-Syco
Ours-Len-Syco
40%
70%
83.6
88.0
85.6
84.4
87.4
86.4
82.6
88.6
84.4
80%
70%
81.2
86.2
85.9
86.4
79.7
86.2
85.1
D PROMPT-BASED JUSTIFICATION PROMPT
We provide a Swee3-235BB. A-2B- -ased pair-wise wise justification prompt shown below, which is adopted from ArenaHard's official implementation.
24

---

Published as a conference paper at ICLR 2026
Please impartial judge and evaluate the quality of the responses provided by two Al assistants to the user prompt displayed below. You will be given assistant A's answer and assistant B's answer. Your job is to evaluate which assistant's 's answer is better. Begin your evaluation by generating your own answer to the prompt. You must provide your answers before judging any answers. When evaluating the assistants' answers, compare both assistants' answers with your answer. You must identify and correct any mistakes inaccurate information. Then consider if the assistant' 's answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note that when a user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate ask for clarifications more in information from the user than providing an answer based on assumptions. Relevant means appropriate what is being asked. Concise
all parts of the response closely connect means the response is clear and not verbose or excessive. Then consider the creativity and novelty of the assistant's 's answers when needed. Finally, identify any missing important information in the assistants' answers that would be beneficial to include when responding to the user prompt. After providing your explanation, you must output only one of the following choices as your final verdict with label:
1. Assistant A is significantly better: [[A >> B]]
Assistant A is slightly better: [[A>B]]
3. Tie, relatively the same: [[A=B]]
Assistant is slightly better: [[B>A]]
5. Assistant B is significantly better: [[B>>A]]
Example output: "My final verdict is tie: [[A=B]]"
[User Prompt]
[Assistant A's Answer]
{answer A}
[Assistant B's Answer] {answer B}
E
CASE STUDY
Our model demonstrates its ability to generate not just correct, but a more professionally help- ful response compared to strong baselines like GPT-4o. As visualized 95bb5011 4.00242288 ad7830 of ffffff55999 5992), when asked how to set up passwordless SSH, our model provided a guide that was praised by the LLM-based judge for its clear structure and organiza- tion. In this instance, GPT-4o's response is judged to be better to all other baselines except for our response, which provides more secure, modern, and concise solution, highlighting its advanced capabilities.
Figure (Data UID:
Crucially, the judge noted that our model successfully avoids unnecessary verbosity, resulting in a response that is of higher quality and more concise. This is merely stylistic advantage; the substance of our model's 's response was also technically useful, which recommended the more modern and secure "ed25519" encryption algorithm over the RSA suggested by GPT-4o. Furthermore, our model included the critical security best practice of disabling password authentication entirely via the sshd config file-a vital step for server hardening that GPT-4o and other models omitted.
25

---

Published as a conference paper at ICLR 2026
Prompt: How do set up ssh into a server such don't require password when ssh into it?
ALBBM (2486 chars):
InfoRM (2422 chars):
eaciaa bupalte
the putli key tu lirets
eveii bey trwomic
por of SSH Yee on clemse the typ key (RSA E42551
heype
Public day ts Senvrio feall and oey public kay to sovan You
will
with "eetfe
seneer venty
ow Copp the apei ligitee
Ross
passond yoei Ssitaaaii
up
ligetta doand cass "aadrondd -iaie oot huklts publo key to
Sop ie Aud the lin key the Server Mtlora Mlanedmnnd sorti eeeg
th you capid car Borie aciss inpional
Reyk or
key by Tevi falk inf-cipped
benges
do
sladhie buttnd bey' bnn be pvving bust a en yom po beyy--y--
kope' taahh
cratc -caal dircatt en
doarc appcad the
kerp prtveriig key
thyy
5511 pertiambae the 500 key on lott, yee'l nood
liw died all liac Skay Spkes E00l dimation
peivat kay precuat an as bubbed acoic a ya clandd sire
Les
takos hil
opdix 5S1 leys stuar pettre araplte fooo venuue Hop
bust aeal cavise at avasse one "badt trade soreed
locigh
and Bls doi
sccvar cali Sllviigg soerard dins badirack avolllecnans "aal"
desiomam strpen di pall onga more pmm availla ape ren
serves plicas byy
Make
quane git walker in apowce
Skywork (2274 chars): misiing 58B wil on
PoE (2172 chars):
yeel
and
hibloohira key Pai Local
This
keypl
dofast floaas bus
ki Capy kiy
dee
adseaaretiige ttep the hele kee Sonv
bin
hi puino agis the semert
Canah esiiia "balee lak pueaada on wil lay
addras saruiia tahhrrate Melhac bluada Cogynn the hull
pullee les the sanol's as hustend be caing Sdiviigg depinial
of bey
the
uar Appud the pull
boblodd buy
uacomace the Adleviag lachtig unt iesiepiiaaaceeeeeeee 1ana.
your tii kry hort with ihe conaors of pepptte
2.
sdat Savon
yiver
accoical loo ihs dincery to provet bettol
and wich
twe wile
peosic en
ueeedy vally evinng you 1009
5511
key
soal kr po oog latdd ie viddy
OURS (2052 chars):
rideoriiig 5SH Keys foi
GPT4o (2008 chars):
tuss Cot Your fifilley
bey be actid
5511
you boroe tinaaly in strminal th fdlovi "ia
much of
35Ai orng to aipetity You
bookenll indt this dlaetii
Heylicr
with your
rabtl atloc
uiv kuy pul: Pre feor dela dndiiia incaiii suily
we
intervattos Copy the purlic boy
uecaan de severr and firmer vith the IP the
sevec
bey
suii wolee wiete werd bey the
liad food loyela 5iil ming
scre's tradecos er bomarr of oaboipped en pvilabe.
Rois New,, modke
shoel lepic in libuat biing proppa
and
dircscy bax remeee lidbe
This
wed wime
aerrac log add cer of thyelt ariga your in endiiiien siltlee euge
whe unun of
tott Now wr
de piseeiiiees with
le
skodlle by nen with pattz
y' pan liau
Weeoeeedee mater in her The retirn
siloiing
oup to recont server
quas conveiiia and hosas wap saken our sarue
**Mlodem Alpustha Mecormendane 1a225 key efficerrta thas BSA (Ausinant BS cheketl vn. *Soaurity Bea Praties Exgiially menices diabiing paswodd aufecriction via wil jonlig coliicl socurity eez Assetan il senin and Cr & Strectott Clea stp by denatore with aepeu ups pusphraace peonder checio) wathest cane vatheast "*Perzaanee tiaelnyg tnchades oechic steaed corenand ccurpla. auante proper dicekeppill permiedres
Judgreen
(OURSSS (127F40) [Txa "opplaaaateee
sempp. Ulowerer Amsuuur ermpppee expetor
ronved kays nethed
ctcele.neceeeeeeeee wourty grobein
lechissllly
Figure 6: A case study on vrenaHard-vO.I Benchmark where our model's response significantly outperforms GPT-4o technical question about setting up passwordless SSH.
26


Reviews:
[{"review_id": "EGtAa7csX8", "reviewer": "Reviewer_cNct", "summary": "This paper proposes DIR, an information-theoretic framework for debiasing reward models in RLHF. The method maximizes MI between predictions and true preferences while minimizing MI between internal representations and predefined bias attributes. Using data processing inequality and variational bounds (BA and CLUB), the theoretical objective becomes a tractable loss. Experiments on length, sycophancy, and format biases show improvements in both RM metrics and downstream RLHF performance.", "questions": "1. How would DIR handle concurrent biases? Would you use $\\mathcal{L}\\_{\\text{reward}} + \\sum\\_i \\lambda\\_i \\mathcal{L}\\_{\\text{debias}}^{i}$? What optimization challenges arise from negative interactions between debiasing signals?\n2. Have you studied the architecture sensitivity of $q_{\\psi}$? Could an overly powerful $q_{\\psi}$ discard legitimate preference-informative correlations along with spurious biases?\n3. How do debiased RMs interact with direct alignment methods like DPO? Could altered reward landscapes complicate implicit differentiation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The explicit use of data processing inequality to justify representation-level debiasing, combined with dual variational bounds (BA for information retention, CLUB for bias suppression), provides an elegant and principled solution.\n\n2. Experiments cover three diverse bias types with end-to-end assessment (RM performance + downstream PPO policies). Strong ablations on representation choice (Table 6) and hyperparameter $\\lambda$ (Figure 4) validate design decisions.\n\n3. Zero inference overhead and moderate training cost (~33% increase). Demonstrated versatility across bias types suggests broad applicability.", "weaknesses": "1. Method requires knowing bias types *a priori* and labeling $b_{\\mathrm{rel}}$ for every pair. No mechanism for unsupervised bias discovery limits real-world applicability.\n2. Experiments isolate single biases. Real datasets likely contain concurrent biases (e.g., lengthy + sycophantic responses). Unclear how to extend DIR—multiple debiasing terms with separate $\\lambda$ values? Potential optimization conflicts?\n3. Sycophancy evaluation uses fixed prefix injection (“*Yes, you are right.*”). Real biases are more subtle and contextually integrated, raising generalization concerns.\n4. Unexplored Representation Alternatives: Exclusively uses final hidden state without comparing alternatives (e.g., mean-pooling). Global representations may better capture stylistic/format biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CV9yKDrzb8", "reviewer": "Reviewer_jfKf", "summary": "The paper proposes an information-theoretical viewpoint on reward modeling and the Bradley-Terry model. Specifically, the proposed method, DIR, focuses on mitigating inductive biases, such as verbosity bias and stylistic biases, by maximizing the mutual information between preference prediction and input-response pairs. By demonstrating the performance of the reward model itself and as a preference proxy in RLHF training, the paper shows that DIR could be an effective debiasing objective for reward modeling.", "questions": "- Can DIR be expanded to the implicit reward models like direct alignment algorithms?\n- On the official RM-Bench leaderboard, “Skywork-Reward-Llama-3.1-8B-v0.2” (“BT” in Table 5) has mostly higher numbers compared to the results stated in Table 5. For example, “Hard” score for Skywork-Reward-Llama-3.1-8B-v0.2 is 52.6 and 69.3 for “Chat”, while the reported scores in the paper are 42.76 and 64.69, respectively. Given that the numbers on the leaderboard could change the trend in Table 5 (e.g., the overall score of Skywork-Reward-Llama-3.1-8B-v0.2 on the leaderboard is higher than “Ours-10.0”), this part needs clarification.\n- On the RM-Bench scores, why does the “Ours” model experience a notable drop in the “Easy” accuracy? By debiasing, is the model experiencing a trade-off in easy stylistic differentiation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents an intuitive yet theoretically reasonable scope of understanding reward modeling as aligning the preference distribution and preference prediction from the reward model.\n2. The benchmark analysis on the biases in the reward model benchmark, RM-Bench, comes before the actual debiasing evaluation of the proposed method, which strengthens the experimental rigor of the paper.\n3. Alongside the well-known length bias, the paper studies multiple types of biases and demonstrates that DIR can be an effective learning objective across different biases.", "weaknesses": "The main weakness of the paper is in the clarity of writing. The clarity of mathematical notations and experimental details in the paper can be improved. Other points that could either be clarified or stated as weaknesses are listed in the questions. Overall, the clarity in Sections 2 and 3 should be improved for better clarity. While there are multiple cases where the notational consistency/clarity is lacking, these are a few examples:\n- Section 3.1 starts by saying that $\\mathcal{L}\\_\\text{total}$ consists $\\mathcal{L}\\_\\text{pref}$ and $\\mathcal{L}\\_\\text{debias}$, while Equation (12) uses $\\mathcal{L}\\_\\text{reward}$ instead.\n- $\\mathcal{L}_\\text{debias}$ is not explicitly defined in the paper and appears in Equation (12).\n\nThese inconsistencies and missing definitions prevent a clear understanding of the paper, even though the paper's theoretical soundness should be highlighted as its strength.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jOo5yEOa6U", "reviewer": "Reviewer_6hPr", "summary": "The paper introduces a method called DIR (Debiasing via Information Optimization for Reward Models) aimed at mitigating inductive biases, such as response length, sycophancy, and format, that plague reward models in RLHF settings. It frames training as an information-theoretic problem: the method maximises the mutual information (MI) between model preferences and input response pairs, while simultaneously minimising the MI between the model’s internal representation and specified bias attributes. The debiasing objective is realised via an adversarial variational network applying the CLUB MI-estimator to ensure the learned representation is less correlated with bias attributes.", "questions": "1. The DIR framework's primary practical limitation is that it requires a \"pre-defined bias attribute b\". This means a human must identify and label the specific bias (length, format, etc.) they want to remove. How does this method fare against unknown or unlabeled biases, which are common in real-world data?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper propose a new method and tackles well-documented biases in reward models\n2. the approach provides a  structure that could extend to multiple bias types.\n3. The proposed DIR method shows improvements over several baselines", "weaknesses": "1. The paper presents itself as introducing a “novel information-theoretic framework,” but its core components are repurposed versions of existing methods. The preference loss is simply the standard Bradley-Terry ranking loss, reinterpreted post hoc as a mutual-information maximization objective. The debiasing term also relies on a conventional adversarial setup using the CLUB estimator [1], a technique already established in prior work. Although the implementation is sound and practically useful, it does not represent a genuinely new theoretical contribution.\n2. The paper’s analysis of the key hyperparameter, $\\lambda$, is self-contradictory. Figure 4 claims that performance peaks at $\\lambda = 1$ and drops sharply at $\\lambda = 10$ , calling the latter an “over-correction.” However, Table 5 shows the opposite: the $\\lambda = 10$  model achieves the highest overall and “Hard” subset scores. These results directly conflict, undermining the credibility of the authors’ claims about the optimal debiasing strength and the rigor of their tuning methodology.\n3. Although the results are generally strong, they are not consistently superior across benchmarks. In the Length Bias test, the proposed method’s correlation (0.468) only slightly outperforms the Skywork baseline (0.498), showing minimal real improvement. In the Sycophancy Bias test, the InfoRM baseline even outperforms the proposed method in the 20%/70% adversarial setting (86.6 vs. 85.1). These mixed outcomes weaken the claim of “most consistent and robust performance” and suggest the improvements may be context-dependent rather than universal.\n4. The method’s dependence on a pre-defined and labeled bias attribute ($b$) is a practical limitation. It requires the user to already know what the bias is (e.g., length, sycophancy) and be able to label it for every data pair. This makes it useless against unknown or hard-to-quantify biases, a limitation that more \"indirect\" methods like InfoRM (which the paper critiques) [2] would not have.\n\n[1] Club: A contrastive log-ratio upper bound of mutual information.\n[2] Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S7R2XKm1eb", "reviewer": "Reviewer_UoY4", "summary": "This paper targets to the issue of inductive bias of reward models. This paper proposes an information-theoretic debiasing method, called DIR, which maximizes the mutual information between preference prediction and input pairs, while minimizing the mutual information between outputs and predefined bias attributes to debiase these biases. Experimental results demonstrate its effectiveness in debiasing biases of length, sycophancy and format, while achieving competitive alignment performance.", "questions": "- Do you compare performance with DPO directly trained using the same preference data?\n- What is your implementation details of the variational network $\\psi$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and clear to read.\n- Denoising reward model is very important for RLHF to improve alignment performance of LLMs.\n- The experimental results are solid, covering three types of biases.", "weaknesses": "- In the experimental setup, the authors address the biases of length, sycophancy, and format individually. However, in practice, we would expect a reward model to debias all these biases simultaneously. It seems that the paper lacks an experimental setup that assesses and discusses the ability of DIR to mitigate all these biases simultaneously.\n- There are two commonly used benchmarks for preference alignment, i.e., AlpacaEval [1] and MT-Bench [2], which should be included to evaluate the model ability of open-ended and multi-turn generations.\n- I think that a figure to illustrate your methods will help readers intuitively understand your proposed method.\n- The proposed method DIR relies on predefined inductive biases, such as length, sycophancy and format. However, if the reward model has some potential biases that are not intuitive such as length perceived by human, can the proposed DIR handle such scenario?\n\n[1] Length-controlled alpacaeval: A simple way to debias automatic evaluators.  \n[2] Judging llm-as-a-judge with mt-bench and chatbot arena.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)