Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking
Authors: Paria Rashidinejad
Yuandong Tian
Venue: ICLR 2025 Poster (2025)
Keywords: RLHF
Preference Optimization
Reinforcement Learning
Alignment
Large Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: We define Type I and II Reward Hacking in RLHF, prove existing algorithms are vulnerable to reward hacking, and propose RLHF methods that provably mitigate hacking and consistently outperforming current approaches on LLM alignment and benchmarks.

Abstract: Aligning AI systems with human preferences typically suffers from the infamous *reward hacking* problem, where optimization of an imperfect reward model leads to undesired behaviors. In this paper, we investigate reward hacking in offline preference optimization, which aims to improve an initial model using a preference dataset. We identify two types of reward hacking stemming from statistical fluctuations in the dataset: Type I Reward Hacking due to subpar choices appearing more favorable, and Type II Reward Hacking due to decent choices appearing less desirable. We prove that many (mainstream or theoretical) preference optimization methods suffer from both types of reward hacking. To mitigate Type I Reward Hacking, we propose POWER, a new preference optimization method that combines Guiasu's weighted entropy with a robust reward maximization objective. POWER enjoys finite-sample guarantees under general function approximation, competing with the best covered policy in the data. To mitigate Type II Reward Hacking, we analyze the learning dynamics of preference optimization and develop a novel technique that dynamically updates preference labels toward certain "stationary labels", resulting in diminishing gradients for untrustworthy samples. Empirically, POWER with dynamic labels (DL) consistently outperforms state-of-the-art methods on alignment benchmarks, achieving improvements of up to **13.0** points on AlpacaEval 2 and **11.5** points on Arena-Hard over DPO, while also improving or maintaining performance on downstream tasks such as mathematical reasoning. Strong theoretical guarantees and empirical results demonstrate the promise of POWER-DL in mitigating reward hacking.

Published as a conference paper at ICLR 2025
SAIL INTO THE HEADWIND: ALIGNMENT VIA ROBUST REWARDS AND DYNAMIC LABELS AGAINST REWARD HACKING
Paria Rashidinejad, Yuandong Tian Fundamental AI Research (FAIR) @ Meta AI (pariard, yuandong) 8meta. com
ABSTRACT
Aligning AI systems with human preferences typically suffers from the infamous reward hacking problem, where optimization of an imperfect reward model leads to undesired behaviors. In this paper, we investigate reward hacking in offline preference optimization, which aims to improve initial model using prefer- ence dataset. We identify two types of reward hacking stemming from statistical fluctuations in the dataset: Type Reward Hacking due to subpar choices appear- ing more favorable, and Type II Reward Hacking due to decent choices appearing less favorable. We prove that many (mainstream theoretical) preference op- timization methods suffer from both types of reward hacking. To mitigate Type preference optimization method
Reward Hacking, we propose POWER, that combines Guiaşu's weighted entropy with a robust reward maximization ob- jective. POWER enjoys finite-sample guarantees under general function approx- imation, competing with the best covered policy in the data. To mitigate Type II Reward Hacking, we analyze the learning dynamics of preference optimiza- tion and develop novel technique that dynamically updates preference labels toward certain "stationary labels", resulting in diminishing gradients for untrust- worthy samples. Empirically, POWER with dynamic labels (POWER-DI.) consis- tently outperforms tfate-of-thh-an art methods on alignment benchmarks, achieving improvements of up 13.0 points AlpacaEval 2.0 and 11.5 points Arena- Hard over DPO, while also improving or maintaining performance on downstream tasks such as mathematical reasoning. Strong theoretical guarantees and empirical results demonstrate the promise of POWER-DL in mitigating reward hacking.
INTRODUCTION
Aligning AI systems with human values a core problem in artificial intelligence (Russell, 2022). After training vast datasets through self-supervised learning, large language models (LLMs) typically undergo an alignment phase to elicit desired behaviors aligned with human values (Ouyang al., 2022). A main alignment paradigm involves leveraging datasets of human preferences, with techniques like reinforcement learning from human feedback (Christiano et al., 2017) or preference optimization (Rafailov et al., 2024b). These methods learn an (implicit from human preferences, which guides the decision eerriiion-maiingge process of the Al system. This paradigm has been instrumental today's 's powerful chat models (Achiam al., 2023; Dubey et al., 2024).
explicit) reward model
However, these alignment techniques are observed to suffer from the notorious reward hacking prob- lem (Amodei al., 2016; Tien et al., 2022; Gao et al., 2023; Casper et al., 2023), where optimizing imperfect learned reward leads to poor performance under the true reward assuming an underlying true reward exists (Skalse et al., 2022). One primary cause of the discrepancy between the learned and true rewards arises because preference data do not encompass all conceivable choices, making the learned reward model vulnerable to significant statistical fluctuations in areas with sparse data. Consequently, the AI system might be swayed toward choices that only appear favorable under the learned reward but are, in reality, subpar, or the system might be deterred from truly desirable choices that do not seem favorable according the learned rewards.

---

Published as a conference paper at ICLR 2025
Unknown Trae Rewards
Initial Model
Preference dataset
Learned Model
pubdiiiiic
RLMF
profuitinis
(a) Type
Reward Hacking
ciorices
0
chuices
C-0
choices
Unknown Tiue Rewards
Initial Model
Preference dataset
Learned Model
RLHF
(b) Type
Reward Hacking
choices
choices
Q+-
3+0
C+@
chuices
Unknown Tree Rewards
5
Choices (responses)
8
Preferance pair
(a) Example of Type Reward Hacking. The initial model has a uniform distribution over choices while the dataset has a high coverage on the high-reward choice and low coverage on a low- reward choice. With a decent chance, the poorly-covered, low-reward choice is labeled as preferred, causing PO methods to erroneously assign a high weight to it (Proposition 1). (b) Example of Type Reward Hacking. The initial model is aligned with the true rewards while dataset has a low coverage on the high-reward choice. With a decent chance, the poorly-coveree high- reward choice is labeled rejected, leading deterioration of the model post alignment (Proposition 2).
Figure 1:
In this paper, investigate reward hacking in offline preference optimization (PO), in which we provided with an initial AI system (initial model) and a preference dataset. We do not assume that the preference dataset is necessarily constructed through sampling from the initial model, allowing to leverage existing datasets collected from other models. Our objective is to dissect the roots of reward hacking from statistical standpoint, analyze current methods, and introduce theoretically sound and practically strong methods to mitigate reward hacking. Our contributions are as follows.
Types of reward hacking. We describe two types of reward hacking in preference optimization that stem from high statistical fluctuations in regions with sparse data; see Figure for an illustration. Type Reward Hacking manifests when poorly covered, subpar choices appear more favorable than they truly are, leading the model to assign high weights to these subpar choices. Type II Reward Hacking arises when decent choices with insufficient coverage appear worse than their true value and that leads to deterioration of the initial model. While reward hacking in offline preference optimization is related the challenge partial coverage in offline RL, the setting we consider here faces two sources of distribution shift: between the learned model and data, and between the initial model and data, This differs from offline RI., which typically considers access to an offline dataset alone (Levine et al., 2020) and is concerned with single source of distribution shift.
PO methods provably suffer from reward hacking. We prove that several PO methods suffer from both types of reward hacking (Propositions I and 2). A common countermeasure against reward hacking is keeping the learned model close to the initial model through minimization of divergence measures (Rafailov et al., 2024b; Azar et al., 2024; Huang al., 2024). Yet, our analysis reveals that divergence minimization does not induce sufficient pessimism to prevent Type Reward Hacking, nor does it mitigate deterioration of the initial model caused by Type II Reward Hacking. Notably, reward hacking can occur even when divergence from the initial model is small.
Against Type I and Type II Reward Hacking, To mitigate reward hacking, we inte- grate a robust reward maximization framework with Guiaşu's weighted entropy (Guiaşu, 1971). We transform this objective into a single-step optimization problem (Proposition 3) leading to Prefer- ence Optimization via Weighted Entropy Robust Rewards (POWER). We prove that POWER enjoys umite-sample guarantees for general function approximation. improving over the best covered policy and mitigating Type I Reward Hacking (Theorem 1). Due to the weighted entropy, POWER effec- tively learns from well-covered choices the dataset, even those with a large divergence against the initial model, countering potential underoptimization in divergence-basee methods. We next de- velop Dynamic Labels to mitigate Type II Reward hacking, whereby preference labels are updated in way that diminishes gradients for untrustworthy data (Theorem 2). Our final algorithm combines POWER with Dynamic Labels (POWER-DL), which interpolates robust rewards with maintaining closeness to the initial model, allowing to trade off between reward hacking types.
POWER-DL:
POWER-DL consistently outperforms other methods across various settings. For aligning L.Mss, we implement POWER-DI. and compare it against other preference optimization methods
2

---

Published as a conference paper at ICLR 2025
across different datasets and two scenarios: using existing preference dataset and another with preference data generated through sampling from the initial model. POWER-DI consistently outperforms state-of-the-an methods in alignment benchmarks, achieving improvements over DPO of up to 13.0 points on AlpacaEval 2.0 and 11.5 points on Arena-Hard. Additionally, POWER-DI. improves maintains performance downstream tasks such truthfulness, mathematical rea- soning, and instrrction-follloingg demonstrating robustness against reward hacking and achieving more favorable bias-variance trade-off compared other methods.
BACKGROUND AND PROBLEM FORMULATION
2.1 LEARNING FROM HUMAN PREFERENCE
Contextual bandit formulation. We adopt the contextual bandits formulation described by a tu- ple (X),, where X' is the space of contexts (e.g., prompts), is the space of actions (e.g.. responses), and r X' x y 7 R is a scalar reward function. A stochastic policy (e.g., model or language model) X + A()) takes in a context E X' and outputs action according to (-) We denote the set of all stochastic policies by II = {T X' - A(V)).
Performance metric. We assume that there exists an underlying (unknown) true reward function r* : x'xx y + R. Given the true reward function r* and target distribution over contexts æ ~ p(-), performance of policy æ is the expected true reward over contexts and actions
((x) = (amp))v< [r" (zz v)]
(I)
The Bradley-Terry model of human preferences. Consider a prompt x E X' and a pair of re- sponses y y¹ E V. For any reward function the Terry (BT) model characterizes the probability of preferring y² over y", denoted by = 1, according to:
(l = 1 x, y vv = a (r(z,v')
(2)
where (2) = 1/(1 + exp(-2)) is the sigmoid function.
Offline preference optimization. We consider an offline learning setup, where we start from an initial reference policy (model), denoted by Te Tref. and an offline pairwise preference dataset D = {(x, y (1),, comprising of N iid samples. Prompt and response pairs are sampled ac- cording data distribution: x, v/ y' μl, and preferences label is sampled according to the BT model corresponding to true rewards: I ~ P-12.. أ² ,v"). Importantly, we do not assume that the preference dataset is necessarily constructed through sampling from the initial model. To simplify notation, we define yt ly + 11 l)y² and y (1 - -Iyy* +ly to denote the chosen and rejected responses in the dataset, respectively. Appendix A presents additional notation.
2.2 DIRECT PREFERENCE OPTIMIZATION
A classical approach to learning from human preferences involves learning reward model from dataset, followed by finding a policy through maximizing the learned reward typically regularized with (reverse) KL-divergence keep the learned policy closed to initial policy:
f E arg min (r) = -Ep [log a (r(2, y*) r(r, )))
(3)
A E arg max Ex [f(x, v)) 3DKa mell
and (gT(F) is the negative log- ligeeloodo ac- cording to the BT model. Rafailov et al. (2024b) observed that the policy maximization step in (3) can be computed in closed form and thus simplified the two-step process into a single minimization objective. This method is called direct preference optimization (DPO) and has inspired series of works; see Tables and for several examples.
Here, DKL[ [ee Ezuup [D (-
Some representative variants of DPO that we theoretically analyze are IPO (Azar et al., 2024), which applies a nonlinear transformation to preferences to reduce overfitting, and SimPO (Meng et al., 2024), which removes the reference policy from the DPO objective. We also analyze two recent theoretical methods that come with finie--sampl guarantees and aim at mitigating overoptimiza- tion: xO (Huang al., 2024), which replaces the KI. divergence in DPO with stronger x *
3

---

Published as a conference paper at ICLR 2025
Table I: Preference optimization objectives given data D = {) and initial model Tee--
Method
Objective
DPO (Rafailov et ol., 2024h)
Apee E ary min, -F Ingg (18 Inn Talai (e)
log
Duly led)
DFO-SFT (Liu al., 2024)
RipeosFF arg miin, EEp llog (8)
(lug Nola"
log sxly
Ep (lug maly yy (x)]
(Azar al., 2024)
iip arg ming Fo
(log Isly' [x]
log
Jola ls) "oetto
+)
SimPO (Meng et al., 2024)
Asinro arg miny -En [log a (o) (IoTT log moly"" ( (ax) w log moly [t)) -7)
XPO (Huang et al., 2024)
Ayeo E argming -Ep llogor (cliPan [3 (3(
malx' lxi Teefla 183)
raly
d(s) log(e)
divergence, and DPO+SFT (Liu et al., 2024; Cen et al., 2024), which adds a supervised finetuning term that increases log- log-liiklihood of chosen responses in the preference dataset.
3 REWARD HACKING IN PREFERENCE OPTIMIZATION
In this section, we investigate reward hacking in preference optimization. One driver of reward hacking is statistical enrors present in the dataset. Typically, preference datasets suffer from partial coverage, lacking extensive samples across all possible options. As a result, preferences for poorly covered choices subject to high levels of statistical fluctuations, given the fact that preference labels are Bernoulli random variables with probabilities described by the Bradley-Tery model (2). Subsequently, we describe two types of reward hacking, both originating from the presence of poorly covered choices (actions) in the dataset.
3.1 TYPE I REWARD HACKING
Type Reward Hacking occurs when poorly covered, subpar choices in the dataset appear more favorable due to statistical errors, and that leads to learned policy á with low expected true reward J(+). In the following proposition, we prove that even in the favorable scenario that the high-reward actions are well-covered in the dataset, the existence of a single sample on low- reward action can overwhelm many preference optimization algorithms,, causing them to learn highly suboptimal policies. Figure (a) provides an illustration of one of the failure instances analyzed in this proposition and proof is presented in Appendix C.I.
Proposition 1 (Type I Reward Hacking in xPO). Consider multi-armed bandits with bounded rewards r* (a) [0. 1] and the softmax policy class, defined
11. (Te(V) exp(0())) Zo Zo
Σ eep(((())) E 10.113
Define the best-in-class policy TT. = max (TT). There exist three-armed bandit instances with I1 parameterization, high coverage of the optimal arms p(a arg max, r* (a)) 1/2, and
bounded KL-divergence (KL(TT- Ted)) such that for any N 2 2, 3 > 0. 7 T > 0, policy
E {Apro, huro, (tmmP or ît Exro for < 3 S 1/3, suffers from a constant suboptimality (mT-) (ft) > 0.15 with a constant probability of at least (e(1+ +e))
Type I Reward Hacking and the failure result in Proposition are closely connected to the challenge of partial data coverage in offline RL (Levine et al., 2020), which can be robustly addressed through be applied in various ways
the principle of pessimism in the face of uncertainty. Pessimism such as reducing the rewards (values) of poorly covered actions (Kumar et al., 2020; Cheng et al., 2022) keeping the learned policy close to data collection policy (Nachum et al., 2019). Although divergence-basse -based methods such as DPO and IPO aim at keeping the learned policy close to the initial policy, Proposition shows that maintaining small KL divergence from initial model does not induce sufficient amount of pessimism to prevent Type Reward Hacking.'
Proposition does not contradict guarantees of Huang al. (2024) this work assumes that preference data are collected from the initial policy. However, this assumption is restrictive, as it prevents using existing
4

---

Published as a conference paper at ICLR 2025
Remark 1. Failure result in Proposition 1 is rigorous and constructed under a realistic setting close to practice: the policy class is a softmax with bounded rewards and the KL divergence between ini- tial and overoptimization in DPO, which rely on unbounded rewards (Azar et al., 2024), updates to model parameters despite receiving no samples (hence, conclusion breaking gradient-based optimiza- tion) (Huang et al., 2024), or events with probabilities approaching zero (Song et al., 2024).
policy is bounded. This makes Proposition 1 stronger than prior arguments
3.2 TYPE II REWARD HACKING
Type II Reward Hacking can occur when poorly covered, good choices in the dataset appear to be less favorable than their true value due to statistical errors, leading to the deterioration of the initial model after preference optimization. Figure 1(b), where the initial policy has high probability on the high-reward choice. Yet, due to the low coverage of the high-reward choice in the data, by mere chance this choice can appear unfavorable, resulting in the deterioration of the initial model. In the following proposition, we prove that many preference optimization methods are susceptible to Type II Reward Hacking. Proof of this proposition can be found in Appendix C.2.
example of this type of reward hacking is illustrated in
Proposition 2 (Type II Reward Hacking in +PO). Consider the multi-armed bandits setting with the softmax policy class Ill, as defined in (4). Let πι- maxxeT, (T) represent the best-in-class policy. There exists a three-armed bandit problem with HI parameterization and Too = Tg. such that for any N 2 3, 3 > 0, 17 2 0, and policy èt E {ppro ADPO+SPT, Aimmo) } or î E {āxro, Aro for O < B r S 1, the following holds with constant probability of at least (e(1 + +e))
) - J(T) > 0.1.
Proposition 2 states that even with a strong initial model, a preference dataset that poorly covers high-reward actions can lead to substantial deterioration of the initial model in cxisting approaches, even in methods such as DPO, IPO, and xPO that incorporate deveerence-minimiie the above setting is beyond the guarantees of traditional pessimistic offline RL, as these techniques typically do not consider access to an initial model and guarantee competing with the best covered policy in the data. Despite this, as we see in Section 5, there may be hope mitigate the degradation of the initial model and better control the trade-off between Type and Type II Reward Hacking.
We note that
4 AGAINST TYPE REWARD HACKING: WEIGHTED ENTROPY ROBUST REWARD MAXIMIZATION
4.1 WEIGHTED ENTROPY REWARD MAXIMIZATION
We demonstrated that approaches involving divergence minimization remain vulnerable to reward hacking. Moreover, maintaining small divergence can inadvertently lead to underoptimizing the preference dataset, as it may risk overlooking policies that, although well-covered, deviate signifi- cantly from the initial policy. These reasons motivate to explore alternative route and consider regularizing the reward maximization objective with the concept of weighted entropy.
Definition (Weighted Entropy; Guiaşu (1971)). The weighted entropy of a (discrete) distribution p(-) with non-negative weights w(y) is defined as Hp) == - -E,, w w(v)p(y) p(y)
Weighted entropy extends Shannon's entropy by incorporating weights associated with each out- come, reflecting attributes such as favorableness or utility toward specific goal (Guiaşu, 1971). Building on this, we consider a weighted-entrooy reward (WER) maximization objective:
max Ea-ppye (r v) + (T),
(5)
FEll
where (T) = Egrup [ (=((((()))) This objective expresses the principle of maximum (weighted) entropy (Jaynes, 1957; Guiasu & Shenitzer, 1985), promoting the selection of policies with maxi- mum entropy-thus thus favoring the most uniform or unbiased policies-among those compatible with
preference datasets collected from other models, which common approach in practical pipelines such as Wang et al. (2024c) and Tunstall et al. (2023).
?Simply reducing 3 to alleviate underoptimization may not always be viable. For example, reducing 3 may reduce underoptimization in one while inadvertently amplify overoptimization in another state.
5

---

Published as a conference paper at ICLR 2025
the constraints, such achieving high rewards. Objective (5) extends the well-established maxi- mum entropy framework in RL, used in various settings such as exploration (Haarnoja et al., 2018), inverse RL (Ziebart et al., 2008), and robust RL (Eysenbach & Levine, 2022).
4.2 POWER: PREFERENCE OPTIMIZATION WITH WEIGHTED ENTROPY ROBUST REWARDS
To mitigate reward hacking, we integrate the WER objective (5) with a robust (adversarial) reward framework, inspired by favorable theoretical guarantees (Liu al., 2024; Cen et al., 2024; Fisch et al., 2024). Specifically, we find a policy that maximizes WER (5) against an adversarial reward, which seeks to minimize WER while fitting the preference dataset:
max min TEl rER.
(BT(T) nogative log-likelihood
+1 (Exnp.ynr [r(2, y)] E (r(z, (/ +/ (H)(T))
(6)
WER minus baseline
where 1) 2 O, is a reward function class, and (ry(r) is the negative log-likelihood of the BT model. We subtracted a baseline Ez=pay'>>' [r( y²) that computes the average reward with respect to some policy n', as the preference data under the BT model only reveal information on reward differences (Zhan et al., 2023a). This baseline plays a crucial role in simplifying the objective and establishing finite-sample guarantees, and we subsequently discuss reasonable choices for n².
Under some regularity conditions (detailed in Appendix D.2), the objective (6) can be equivalently expressed as a minimax problem, which leads to a single-step preference objective presented in the proposition below; Appendix D.3 for the derivation and proof.
Proposition 3 (POWER Objective). Let w(y) > denote the weights in the weighted entropy (T)( and nT, denote the policy that maximizes the objective (5). Under certain regularity condi- tions on R (Assumption 1) and for any B > O, solving the maximin objective (6) is equivalent to
solving the following optimization problem:
min LaRT (3 ((w(y)) To (ulx)) + +(w)) ammpgn (ur(s)) log ,. (v/1x)] PER
(7)
We call the above objective preference optimization via weighted entropy robust rewards, or POWER. The first term in the above objective the Bradley-Terry loss with rewards set to w(y) log (y(zz) + w(y), resulting in reward gap expressed as weighted difference of log probabilities of chosen and rejected responses. The second expectation a weighted negative log- likelihood ka. supervised fine-tuning, or SFT) regularizer over the baseline policy x'.
Liu et al. (2024) propose an adversarial objective similar to (6) that uses KL divergence instead of weighted entropy. From theoretical perspective, our approach with weighted entropy improves over this work, such as through mitigating underoptimization; see Section 4.3 for details. Moreover our final algorithm presented in Algorithm 1 considerably different from the DPO+SFT objective (Liu et al., 2024; Pal et al., 2024) and significantly outperforms empirically (Section 6).
Remark 2.
4.3 FINITE-SAMPLE GUARANTEES AND THEORETICAL BENEFITS OF POWER
The following theorem shows that POWER enjoys knite-sample guarantees on performance.
Theorem 1 (Fimite-Sample Performance Guarantees of POWER). Given a competing policy < oo as defined in Definition 2.
E II, assume a bounded concentrability coefficient Ch (R, x') Furthermore, assume realizability of the true reward function r* E R, boundedness of rewards or E R r(x, y) E [0, RJ, and that the reward function class has a finite e-covering number Ne under the infinity norm. Define R == 1 + exp( R), = (NNN)-, + = vTog(N.)75. Set x 1/ 1/(f and B = L/VN in the objective (6). Then with probability of at least 1 - 6, policy TrOOER that solves (6) satisfies the following
((TPOWER) 2 J(z)
VN (ICI(R,='((+1) Re Hm)(
1
Furthermore, let L denote the maximum response length. Selecting w(y) = 1/1y| to be the inverse response length, one has
1 J ((PPOEER J(T) - VV ((IC(RR + +1) RP, 1og(LIV))

---

Published as a conference paper at ICLR 2025
Proof of the above theorem can be found in Appendix E.2. Below, discuss the implications of the above theorem and the guidelines it offers for practical choices.
Guarantees against Type Reward Hacking. Theorem shows that the policy learned by POWER competes with the best policy covered in the dataset, where the notion of coverage is characterized by single-policy concentrability, considered the gold standard in offline RL (Rashidinejad et al., 2021; Xie et al., 2021; Zhan et al., 2023a). This implies that as long as a favorable policy is covered in the preference data, POWER is robust existence of poorly covered, subpar policies and thus mitigates Type I Reward Hacking. Moreover, as Theorem does not impose a parametric form on the class R. guarantees hold for general function classes.
Benefits of weighted entropy and choice of weights. A non-zero weighted entropy term (3 > 0) essential in obtaining the one-step optimization problem in (7) and establishing equivalence to the maximin problem, as this term induces strict concavity in the objective (5). Moreover, using KL- regularization leads rates that grow with the divergence of competing policy and initial policy (Liu et al., 2024), which can be large or even unbounded. However, weighted entropy ensures bounded rates and thus mitigates underoptimization. Theorem 1 suggests that a particularly appealing choice for weights is the inverse response length wzy) 1/ly). which intuitively discourages learning poli- cies that generate lengthy responses. Theoretically, while using Shannon entropy (w(y) = 1) results in convergence rate that grows linearly with response length, with weights w(y) = 1/lyl conver- gence rate only depends on the logarithm of the vocabulary (token) size and logarithm of response length. Other choices for weights include response preference scores and per-sample importance weights.
Choice of the baseline policy. The rate in Theorem 1 is influenced by the concentrability coefficient C% (R, T'), which is impacted by the baseline policy T'. Inspecting the definition of concentrabil- ity coefficient in Definition 2, a reasonable choice make this coefficient small is selecting the distribution of chosen responses in the dataset (Zhan et al., 2023a; Liu et al., 2024).
With the above choices applied objective 7, a practically appealing version of POWER becomes:
log (z) log To (y 1y+
[z)
1 1u
log (o
(x)
max En
log o
+
1 + ngED
(8)
ly
Remark 3. Objective (8) shares similarities to SimPO (Meng el al., 2024) but has important differ-
ences. First, objective (8) includes length-normalized SFT term, which is key in mitigating Type I Reward Hacking (Theorem 1, Proposition 4). from which SimPO suffers (Proposition 1). Second, our approach analytically leads to the margin 1/1y*1-1//1 while SimPO uses a fixed hyperpa- rameter. Lastly, our objective is rooted in theory and f finite-sample guarantees.
4.4 POWER FACED WITH HARD INSTANCES AND THE ROLE OF PARTITION FUNCTION
In the following proposition, analyze the POWER objective (8) in the hard reward hacking instances of Proposition and Proposition 2. Proof is presented in Appendix C.3.
Proposition 4. (I) Consider the three armed bandit instance in Proposition 1. Then, for any 3 > 0 and 17 N2(2+cj POWER policy POWER that solves the objective (8) is the best-in-class policy:
(2+e)
fPOWER kg. (II) Consider the three-armed bandit instance in Proposition Then, for any 3 > 0, 2 0, policy POWEER that solves the objective (8) suffers from a constant suboptimality J J(TO-) ) J (#rOWE)) > 0.2.
Proposition 4 confirms that POWER robustly (for any 3 > 0 and v7 2 1/N) prevents Type I Reward Hacking in the hard instance of Proposition 1, where other preference optimization algorithms DPO, SimPO, IPO, and XP fail. Yet, the above proposition shows that POWER suffers from Type II Reward Hacking, which the design dynamic labels in the following section.
5 AGAINST TYPE II REWARD HACKING: DYNAMIC LABELS
We now turn our focus to mitigating Type II Reward Hacking, based on the following intuition: keeping the model's internal preferences close to initialization in the low-coverage regions (trust the

---

Published as a conference paper at ICLR 2025
preference labels less) while learning from the data in the high-coverage regions (trust the preference labels more).
For this purpose, we analyze the learning dynamics of preference optimization in the bandits setting with softmax parameterization (y(y) x exp(f(y)). We denote the dataset by D = {(2", y² []} with labels 1 ~ P- (-lu"; y'). We use Po,1 to indicate the empirical probability of comparing y with y', and o the empirical probability of preferring 3² over y². To simplify presentation, we consider POWER with w(y) = 1, = 3 = 1; a similar analysis can be extended to other objectives.
Reverse engineering label updates based erence labels, we allow the labels l to evolve across gradient updates. Denote the parameter gap corresponding to two actions yº y² by dg (y', .y²) = (vy')) - 0.()"). We show in Appendix F.1 that isolated (batch) gradient updates on
learning dynamics. Rather than using static pref-
and y is:
d (v' .80) = de, (v',8) afo.1 (P Po--)44 - (a (d', .v0) PO-1i]
(9)
We design labels le so that gradient updates are rapidly diminished for poorly covered preference pairs, ensuring that preferences for such pairs remain close to initialization. To achieve this, we first directly set the gradient in (9) to zero and derive a "stationary" label 4.
(i1>a- - 1001))4 - (do, (o".y")) 1 =0 = 1, (do, (2))) Po>-1
(10)
PiLn - Po-1
1, represents the ratio between a learned preference gap and the empirical preference gap. and we have 4, = 1 when learned and empirical preferences are equal. To implement dynamic preference labels, we employ the following update rule for 4, where ranges between 0 and I:
4+1 = (1 - 7)4 + Th, to 1.
(11)
In the following theorem, we analyze the coupled dynamical systems described by equations (11) and (9); Appendix F.2 for the proof.
Theorem (Learning Dynamics with Label Updates), Consider the following set of differential equations with initial values lo and any do:
de afoo =ajo. (10100- - -00 -001010--000000 (o((d)) (
(12)
i,
7
(1 - P 14, (o(de) - -(wz1)
PI20 Po-1
Assume Pi-- 1/2 and let c min (ond (dd)) P For any e < fix -1/4jam S and añnyoT 2 1.
Mt, ph: T, fo.1 such that ap//a S 7 S
1. (Low Coverage Case) When Po.1 S μ4, we have (o(rr - (do)ls ldr dols El.
2. (High Coverage Case) When Po, 2 ph we have (o(d)) p)0o)? 1 exp T)).
The above theorem shows that for poorly covered pairs (small (1,,)). learned preferences a (dr) re- main close to initialization, while for high coverage pairs (large (1.,), learned preferences converge to empirical preferences. In sense, y determines the level of conservatism. adjusting the threshold of what considered poor coverage. Moreover, the convergence rate in the high-coverage case is im- pacted by empirical preferences through c. In the of nearly equal preferences áiyo R 1/2, the convergence rate is faster, whereas in the case of strong preference with P1.0 -> 1, the convergence rate is slower suggesting that more updates are required to further distinguish the two choices.
Remark 4 (Related work on soft labels in RLHF). In preference oplimization, Mitchell (2024) considers noisy preference labels and incorporates constant soft labels through linear interpolation. Concurrent work by Furuta et al. (2024) develop a geometric averaging approach, in which samples are weighted according to the preference gap using scores from a reward model. the context of reward learning, Zhu et al. (2024) propose iterative data smoothing that updates labels toward learned preferences. In contrast to these methods, our approach is rooted in updating labels to shrink gradients of poorly covered pairs via a general recipe whereby dynamic labels are smoothly updated toward labels that set the gradient to zero. This approach goes beyond constant soft labels and does not require scores from extra reward or preference model. Moreover, label updates in prior works do not guarantee remaining close to a (non uniform) initial model in the low coverage areas, which aims at mitigating Type II Reward Hacking in the offline alignment setting.
8

---

Published as a conference paper at ICLR 2025
POWER with Dynamic Labels. Our final algorithm POWER-DI SIWER-DL-R (Algorithm 1) integrates the POWER objective with dynamic labels against reward hacking. In untrustworthy regions, POWER- DL interpolates between the initial model and robust rewards, allowing to trade off the two types of reward hacking through adjusting conservatism parameters 1/ and y, reflecting relative quality of the initial model compared to preference data and up to removing conservatism completely by setting = y= O. We highlight the fact that divergence-basee methods aim at keeping the learned model close the initial model wherever the learned model has a decent probability, regardless of data coverage. In contrast, the dynamic label procedure aims at keeping the learned model close to the initialization only in the untrustworthy regions while leamnng from the data in high coverage region, which can alleviate potential over-pessimism. All these factors can lead to a better performance, as supported by our empirical evaluations Section 6, See Appendix B.3 for further discussion.
EXPERIMENTS
6.1 EXPERIMENTAL SETUP
We conduct experiments assess different PO methods on aligning LLMs across four settings, varying in dataset size and level of distribution shift between the initial model and data. We fol- low two pipelines: Helpsteer2 (Wang et al., 2024e), which employs smaller datasets, and Zephyr (Tunstall et al., 2023) with significantly larger datasets. We implement two distinct setups similar to Meng et al. (2024): the base setup that uses an existing preference dataset and the instruct setup that constructs preference dataset by sampling from the initial model. These two setups allow evaluating across different levels of distribution shift between the initial model and preference data.
Helpsteer2 setups. In the base setup, train Llama-3-8B on the OpenAssistant2 dataset (Köpf' et al., 2024) to create the initial model. We conduct preference optimization using the Helpsteer2 dataset (Wang et al., 2024e), selecting responses based helpfulness scores and discarding ties, yielding about 7K samples. In the instruct setup, we use Maama-3-8B-Instruu as the initial model and generate a preference dataset from Helpsteer2 prompts. Following Wang al. (2024e), we generate 10 responses per prompt with temperature 0.7. We then score them with Armo reward model (Wang al., 2024c) and select the highest and lowest score responses as yt and y respectively.
Zephyr setups. In the base setup, we obtain the initial model by training Llama-3-8D base model on the Ulra dataset (Ding et al., 2023). We then perform preference optimization on the UltraFeedback dataset (Cui et al., 2024), comprising approximately 61K samples. In the instruct setup and following Meng et al. (2024), we start from 1ama-3-8BB InSt-unst and generate responses with temperature 0.8 per prompt in the UltraFeedback dataset. As before, the highest and lowest score responses are selected as preference response pairs.
Evaluation benchmarks. We primarily assess preference methods by evaluating the trained models on standard instruction-follovin benchmarks: AlpacaEval 2.0 (Li et al., 2023a; Dubois al., 2024) and Arena-Hard (Li et al., 2024), which evaluate the quality of the model responses. Following standard guidelines, for Hardd-Hard, we report the win rate (WR) of the model's responses against responses from GPr-4-Turbo. For AlpacaEval 2.0, in addition to the WR against GPT-4-urbo, report the length-controller (LC) win rate, designed to mitigate bias toward verbosity. We further evaluate the performance of models on MT-Bench (Zheng et al., 2023) and downstream tasks such as mathematics, reasoning, truthfulness, and cndrruction-olllowiii (Beeching et al., 2023).
Preference optimization methods. We compare POWER-DI against various baselines; see Ap- pendix H.1 for details. These include divergence-base methods DPO (Rafailov et al., 2024b), IPO (Azar al., 2024), offline SPPO (Wu et al., 2024b), and xPO (Huang et al., 2024), along with robust variants such as conservative DPO (cDPO) (Mitchell, 2024), robust preference optimization (ROPO) (Liang et al., 2024), R-DPO (Park al., 2024), and DPO+SFT (Pal et al., 2024; Liu al., 2024). We also evaluate against reference- free methods CPO (Xu et al., 2024a), SLIC-HF (Zhao et al., 2023), RRHF (Yuan et al., 2024a), ORPO (Hong et al., 2024), and SimPO (Meng et al., 2024).
6.2 BENCHMARK RESULTS
POWER-DL outperforms So methods on alignment benchmarks. Table 2 presents the results on alignment benchmarks, POWER-DI. consistently outperforms other methods in both Helpsteer2
LWWER-DL-outperfomm

---

Published as a conference paper at ICLR 2025
Table 2: AlpacaEval and Arena- results Helpsteer2 and Zephyr settings.
Method
Helpsteer2
Zephyr
Llamu3-8B-Baas
LJama3-SB-Instee
Llama3-BB-Bam
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
AlpacaEval
Arena-Hard
LC(S)
WR(S)
WR(S)
LC(S) WR(S)
WR(S))
LC(%) WR(S)
WR(S)
LC(%) WR(S)
WR(S)
Initial Model
8.02
5.42
2.4
33.41
32.40
23.0
4.76
2.83
2.0
33.41
32.40
23.0
DPO
18.52
14.99
10.0
40.87
39.05
29.6
22.53
17.84
13,3
44.20
43.63
38.4
DPO+SFT
18.33
12.93
7.9
39.85
37.51
27.0
19.11
14.69
9.5
45.98
44.07
39.0
eDPO
19.06
8.5
42.27
34.4
21.06
11.4
R-DPO
11.03
15.20
8.3
33.67
33.89
25.7
18.66
17.88
9.5
44.13
44.94
37.5
IPO
20.11
14,60
9.4
42.95
40.76
30.8
10.55
8.04
7.2
36.63
35.30
24.5
XO
11.06
7.67
5.1
42.10
39.65
35.8
13.16
10.87
8.9
44.25
42.41
34.7
SPPO
26.23
39.46
CPO
15,07
16,78
8.3
35.90
35.20
26.8
7.01
6.84
3.0
36.39
35,40
22.8
RRHF
8.25
7.15
5.8
35.15
34.07
25.7
6.61
6.39
3.0
35,56
34.56
23.1
SLiC-HI
15.19
18.77
10.1
37.76
39.68
32.2
19.35
21.81
11.2
41.74
45.05
38.2
ORPO
23.99
16.91
11.2
43.01
35.68
27.1
23.20
19.43
14.7
45.51
40.95
33.3
SimPO
25.35
19.30
13.7
43.23
36.89
32.6
24.38
21.21
16.4
43.24
37.34
26.8
ROPO
21.24
17.66
9.5
41.03
36.32
31.5
22.91
19.67
10.9
45.55
45.58
33.7
POWER-DL
31.52
31.44
21.5
47.16
43.08
34.8
27.00
22.57
17.3
48,97
43.75
41.5
POWER
29.57
31.5
16.0
and Zephyr pipelines and across base and instruct settings. These improvements can largely be attributed to the integration of weighted entropy, which effectively counters underoptimization, and mitigation of reward hacking. Notably, POWER-DL surpasses other robust methods such as cDPO and ROPO demonstrating its efficacy in handling poorly covered samples. Additionally, POWER- DL improvements are more pronounced in the base setting, which more susceptible to reward hacking due to higher levels of distribution shift. Comparing POWER-DL with POWER shows that incorporating dynamic labels further improves performance. See Appendix and Appendix J for results the Mistral family, MT-Bench, sample responses, and hyperparameter robustness.
POWER-DL improves or maintains performance on downstream tasks. One of the challenges of the alignment step is possible degradation of performance on downstream tasks, which can be attributed to reward hacking (Xu et al., 2024b). We evaluate the trained models on the LLM Leaderboard (Beeching al., 2023), which encompass variety of tasks, including language under- standing and knowledge benchmarks MMLU (Hendrycks et al., 2020), MMLU-PRO (Wang et al., 2024d), and ARC-Challema laSwag (Zellers et al., 2019) and Winogrande (Sakaguchi et al., 2021), factual accuracy evaluations on TruthfulQA (Lin et al., 2022), ietrructionfollowwe capabilities measured on IFEval (Zhou al., 2023), and mathematical reasoning evaluated on the GSM8K dataset (Cobbe et al., 2021).
(Clark et al., 2018), commonsense reasoning assessments like Hel-
Tables and 5 present the downstream task results, POWER-DL consistently improves or maintains performance across all tasks, effectively mitigating reward hacking. Notably, while PO methods vary in results on the GSM8K benchmark, with some like SimPO significantly degrading the initial model. POWER-DE -D consistently maintains enhances performance, achieving up to 7.0 point gain. Other tasks with notable variation include IFEval and TruthfulQA benchmarks. In TruthfulQA, POWER-DI. DiifER--Diignniiaaaaaa significantly outperforms DPO, with up to 12.8 point improvement over initial model. In the IFEval, methods like DPO and SLiC-HF sometimes degrade performance of the initial model, whereas POWER-DL consistently maintains or improves by up 11.7 points.
7 DISCUSSION
We studied reward hacking offline preference optimization. We identified two types of reward hacking stemming from statistical fluctuations in preference data. We demonstrated that many exist- ing methods are vulnerable to both types of reward hacking, despite maintaining a small divergence from the initial model. To mitigate reward hacking, we introduced POWER-DL. a practical algo- rithm based on a weighted entropy robust reward framework augmented with dynamic preference labels. POWER-DL enjoys theoretical guarantees and achieves strong empirical performance. Fu- ture research directions include applications of dynamic labels to oft-of distrribbutoon robustness and investigating the interplay between statistical errors and reward misspecification in reward hacking.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in Al safety. arXiv preprint arXiv:1606.0m 16Xi::6606606665, 06565, 2016.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models. In The Twelfth International Conference on Learn- ing Representations, 2024.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Ra- jani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard. Hugging Face, 2023.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307. 15217, 2023.
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang. Dale Schu- urmans, Yuejie Chi, and Bo Dai. ielue-inceetivized preference optimization: A unified approach to online and offline RLHF. arXiv preprint 1rXiv:2405.193320) 2024.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Am- rit Singh Bedi, and Mengdi Wang. MaxMin-RLHF: Towards equitable alignment of large lan- guage models with diverse human preferences. arXiv preprint arXiv:2402.089925 2024.
Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. In Forty-first International Conference on Machine Learning, 2024a.
Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang- Human-in-the-loop: Provably efficient preference based reinforcement learning with general function approximation. In International Conference on Machine Learning, pp. 3773-3793, PMLR, 2022.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024b.
Ching-An Cheng, Tengyang Xie. Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pp. 3852- 3878. PMLR, 2022.
Eugene Choi, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, and Mohammad Gheshlaghi Azar. Self-improving robust preference optimization. arXiv preprint arXiv:2406.0 2406.00666000 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803. 180.00557.. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1446. 2021.
11

---

Published as a conference paper at ICLR 2025
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafedbbckk: Boosting language models with high-quality feedback. International Conference on Machine Learning. 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversa- tions. In Conference on Empirical Methods in Natural Language Processing, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Hanze Dong, Wei Xiong, Bo Pang. Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. arXiv preprint arXiv:2405.07880 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXi::2407.2 1783, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04477 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? Reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.092444 2023.
Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022.
Ky Fan. Minimax theorems. Proceedings of the National Academy of Sciences, 39(1):42 47, 1953.
Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.1331 2024.
Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, and Izzeddin Gur. Geometric-averager preference optimization for soft preference labels. arXiv preprint arXiv:2409.0669) 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu, Eu- gene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, al. Averaging log-likelihoods in direct alignment. arXiv preprint 2rXiv: 2024.
Silviu Guiaşu. Weighted entropy. Reports on Mathematical Physics, 2(3):165-17) 1971.
Silviu Guiasu and Abe Shenitzer. The principle of maximum entropy. The mathematical intelli- gencer, 48, 1985.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel. Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
Jiwoo Hong, Noah Lee, and James Thome. ORPO: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024.
Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405./ 2024.
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of KL-regularization: Direct alignment without overpa- rameterization via Chi-squared preference optimization. arXiv preprint 2:3 2024.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:23 10.06825, 2023.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodee Nova DasSarma, Eli Tran-Johnson, et al. Language mod- els (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022.
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis)design for autonomous driving. Artificial Intelligence, 316:19, 2023.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richárd Nagyfi, al. Openassistant aonverrstions-dmmoorattiiii large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006. 2020.
Nathan Lambert and Roberto Calandra. The alignment ceiling:: Objective mismatch in reinforcement learning from human feedback. arXiv preprint urXiv:2311.00fF5 2023.
Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in LLMs: Reward calibration in RLHF, arXiv preprint arXiv:2410.09724, 2024.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643, 2020.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal no deal? End- to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2443-2453 2017.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. AlpacaEval: An automatic evaluator of instruction-follovinn models, 2023a.
Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learn- ing dynamic choices via pessimism. arXiv preprint arXiv:2305. 134488888. 2023b.
Xize Liang. Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi. Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for LLMs. arXiv preprint arXiv:2404.044 14 2024.
13

---

Published as a conference paper at ICLR 2025
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, 2022.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405. 16436, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. arXiv preprini arXiv:2405. 14734, 2024.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Geoffrey Irving, et al. Teaching lan-
Mia Glaese, Susannah Young, Lucy Campbell-Gillingee guage models to support answers with verified quotes. arXiv preprint arXiv:2203. 2022.
Eric J Michaud, Adam Gleave and Stuart Russell. Understanding learned reward functions. arXiv preprint arXiv:20/2.05882. 2020.
Eric Mitchell. A note on dpo with noisy preferences and relationship to IPO. 2024. URL https: a cdpo. pdf.
Ted Moskovitz, Aaditya Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dra- gan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. In The Twelfth International Conference on Learning Representations, 2024.
Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang. Matthieu Geist, Thomas Mesnard, Côme Fiegel, al. Nash learning from human feedback. In Forty-first International Conference on Machine Learning, 2023.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Bchavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pp. 2315-2325, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35; 27730-27744, 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO-Positive. arXiv preprint arXiv:2402. 13228, 2024.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. 19159, 2024.
Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. International Conference on Learning Representations, 2018
Baolin Peng, Linfeng Song. Ye Tian, Lifeng Jin, Haitao Mi, and Dong Yu. Stabilizing RLHF through advantage model and selective rehearsal. arXiv preprint arXiv:2309.1020. 2023.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Nickum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.022 02900, 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
14

---

Published as a conference paper at ICLR 2025
Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In Forty- first International Conference Machine Learning, 2024.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline rein- forcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:1 1702-11716, 2021.
Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimizaton in LLM with demonstratioon-guided reinforcement learning. arXiv preprint arXiv:2404. 19409, 2024.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct Nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.0377 037/15 2024.
Paul Rötteer, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 5377- 5400, 2024.
Stuart Russell. Human-compatible artificial intelligence., 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge scale. Communications of the ACM, 64(9)999-106, 2021.
John Schulman. Proxy objectives in reinforcement learning from human feedback. Invited Talk at cc/
the International Conference on MuchineLearming (ICML), 2023, URL
virtual/ /taiaaa///2///// http:///wwweeeeaaaaaaa2255////// //t.uuua////333/atttaaaaaa222112
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward inconsistency on RLHE. In The Twelfth Inter- national Conference Learning Representations, 2024.
Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias reinforcement learning from human feedback. In Conference on Empirical Methods in Natural Language Processing. 2023.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF, arXiv preprint (oXiv:3310003/111 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. ICML Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021 2020.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. In Forty-first International Conference on Machine Learning. 2024.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Row- land, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: unified approach to offline aligmment. In Forty-first International Conference on Machine Learning, 2024.
15

---

Published as a conference paper at ICLR 2025
Jeremy Tien, Jerry Zhi- Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference- reward learning. In The Eleventh International Conference Learning Representations, 2022.
Jeremy Tien, Jerry Hang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. In The Eleventh International Conference on Learning Representations, 2023.
Lewis Tunstall, Edward Beeching, Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang. Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of LM alignment. arXiv preprint arXiv:2310.. 16944, 2023.
Tyler VanderWeele. Controlled direct and mediated effects: Definition, identification and bounds. Scandinavian Journal of Statistics, 38(3):551-56)) 2011.
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of RLHF in large language models part II: Reward modeling, arXiv preprint vrXiv:2401.06080 2024a.
Chaoqi Wang, Yibo Jiang, Chenghao Yang. Han Liu, and Yuxin Chen. Beyond reverse KL: Gener- alizing direct preference optimization with diverse divergence constraints. In The Twelfth Inter- national Conference Learning Representations, 2024b.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406. 12845, 2024c.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? Exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, (6:74774-74778 2023a.
Yuanhao Wang. Qinghua Liu, and Chi Jin, Is RLHF more difficult than standard RL? arXiv preprint arXiv:2306.1411/ 2023b.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. MMLU-PRO: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406. 01574, 2024d.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. HelpSteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.0866 2024e.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46. 2017.
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109. 10862, 2021.
Junkang WU, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. 3-dpo: Direct preference optimization with dynamic B. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint 200500505 2024b.
Tengyang Xie, Ching An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Dellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-6694, 2021.
16

---

Published as a conference paper at ICLR 2025
Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit Q pa-appoxximaiion for sample-efficient RLHF. arXiv preprint arXiv:2405.21046 2024.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sam- pling from human feedback: A provable KL-constrained framework for RLHF. arXiv preprint arXiv:2312. 14456, 2023.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, 2024a.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.1668,, 2023.
Tengyu Xu, Fryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shao- liang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, et al. The perfect blend: Redefining RLHF with mixture of judges. arXiv preprint arXiv:2409.20370 2024b.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHE: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning. 2024b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. pnncertainty-peaaliinn reinforcement learning from human feedback with diverse reward LoRA ensembles. arXiv preprint arXiv:2401.0000 240.00033., 2023.
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D) Lee, and Wen Sun. Provable offline preference-based reinforcement learning. International Conference on Learning Representations, 2023a.
Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. How to query human feedback efficiently in RL? In JCML 2023 Workshop The Many Facets of Preference-Bast Learning. 2023b.
Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming ward overoptimization via adversarial policy optimization with lightweight uncertainty estima- tion. arXiv preprint arXiv:2403.05171, 2024.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. SLiC- HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM audduee with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595 46623, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. instruction-followin evaluation for large language models. arXiv preprint arXiv:2311.079 2023.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feed- back from pairwise or k-wise comparisons. In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.
17

---

Published as a conference paper at ICLR 2025
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfit- ting and overoptimization RLHF. Forty-first International Conference on Machine Learning, 2024.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1438. Chicago, IL. USA, 2008.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1999.08593, 2019.
18

---

Published as a conference paper at ICLR 2025
A ADDITIONAL NOTATION
We use calligraphy letters to denote sets, e.g., V. Given a response y, we write ly| to denote the length of the response in the number of tokens. We denote by V the vocabulary set and write [V] to denote the cardinality of the token space. We write S y when there exists a constant such that x < cy and similarly, write z x y when there exists a constant c such that æ cy. We write y² > n6 denoting that y² is preferred over v² in the dataset. For any two discrete probability distributions and n' over V, we define the KL divergence (TT
Ey~x (log =(y)1 J- The
<'yy]
probability simplex over a set is denoted by A(X). We write 1 = c} to denote the indicator function, which is equal to when x and zero otherwise. We write Ep to denote the empirical average over data.
B RELATED WORK
B.I RLHF AND PREFERENCE OPTIMIZATION
Earlier works on reinforcement learning from human preferences mainly focused on the continu- control domain (Wirth et al., 2017) such Atari games (Christiano et al., 2017). Recently, RLHF has been extensively applied in the natural language domain (Ziegler et al., 2019) to im- prove alignment of LLMS with human preferences in various areas such as summarization (Stiennon et al., 2020; Wu et al., 2021), information accuracy (Menick et al., 2022), and instruction following (Ouyang et al., 2022).
Classical RLHF pipeline includes two steps of reward learning and policy optimization using RL, commonly using variants of proximal policy optimization (PPO) algorithm (Schulman et al., 2017) that involves on-policy sampling. Direct preference optimization (Rafailov et al., 2024b) simplifies the two-step process into a single-step offline optimization of the policy, reducing computational burden and training instabilities of PPO. DPO has inspired development of new preference opti- mization objectives from a practical perspective such as IPO (Azar et al., 2024), RRHF (Yuan al., 2024a), SLiC-HF (Zhao et al., 2023), CPO (Xu et al., 2024a), ORPO (Hong et al., 2024), R-DPO (Park et al., 2024), SimPO (Meng et al., 2024), and general preference optimization (Tang et al., 2024) and theoretical perspective such as xPO (Huang et al., 2024) and RPO (DPO+SFT) (Liu et al., 2024). Our approach also falls under the category of offline preference optimization. We theoretically analyzed several of the mentioned methods and demonstrated theoretical benefits of- fered by our approach POWER-DL. We also showed that POWER-DL outperforms prior methods empirically across variety of settings.
Going beyond the Bradley-Terry model of human preferences, some works consider general pref- erence models (Munos et al., 2023; Swamy et al., 2024; Rosset et al., 2024: Choi al., 2024; Wu et al., 2024b), and develop algorithms aiming at finding the Nash equilibrium. Recently Huang et al. (2024) showed that this generalization comes statistically efficient algorithm exists to solve RLHF with general preferences under single-policy concentrability. Another line of work focuses on iterative, online improvement of language models through self-play (Chen et al., 2024b; Wu et al., 2024b; Xu et al., 2023; Yuan et al., 2024b).
cost of an anformationntheorrtt limit, where no
B.2 UNDERSTANDING REWARD HACKING
The phenomenon of reward hacking in training Al models has been observed in variety of domains, ranging from games (Ibarz al., 2018) to natural language (Paulus et al., 2018) to autonomous driv- ing (Knox et al., 2023). In the language modeling domain, existing RI.FF algorithms are observed to be susceptible to reward hacking (Gao et al., 2023; Casper et al., 2023; Amodei et al., 2;16; Lambert & Calandra, 2023). Reward hacking in LLMs manifests in different ways such verbosity (Shen et al., 2023; Singhal et al., 2023; Wang et al., 2023a), refusing to follow instructions (Röttger et al., 2024), lazy generations (Lambert & Calandra, 2023), emergence of language (Lewis et al., 2017), degradation of performance on downstream tasks such as reasoning (Xu et al., 2024b), and other problems such hedging and self-doubt (Schulman, 2023).
Origins of reward hacking, In RL/RLHF, reward hacking can originate from various factors such reward misspecification (Amodei et al., 2016; Hadfield-Menell et al., 2017; Knox et al., 2023),
19

---

Published as a conference paper at ICLR 2025
diversity and inconsistencies in human preferences (Chakraborty et al., 2024), labeling noise (Wang et al., 2024a), human labeler bias (Bansal et al., 2024), and statistical errors (Liu et al., 2024). Pan et al. (2022) study reward hacking due to human misspecification of the reward model and empiri- cally assess the impact of model size, optimization, and training on reward hacking, given synthetic misspecified reward models. Wei et al. (2024) attribute failure modes of LLM safety training to conflicts between model's 's capabilities and safety goals. Bansal et al. (2024) study mismatch arising from annotator bias in different types of human rating data. Peng et al. (2023) explain that variations of reward distribution across different tasks can lead to reward hacking. Rame et al. (2024) attribute reward hacking classical RLHF to distribution shift and human preference inconsistencies. Tien et al. (2023) conduct an empirical study, revealing that non-causal distractor features, human bias and noise, and partial observability exacerbate reward misidentification.
Lambert & Calandra (2023) argue that objective mismatch in RLHF originates from learning reward model, policy training, and evaluation, and links between each pair, and suggest further research is needed to understand objective mismatch in preference optimization due to entanglement of policy and reward. Rafailov et al. (2024a) conduct an empirical study of reward hacking in direct pref- erence optimization methods, showing that in larger KL regimes, preference optimization methods suffer from degradations reminiscent of overoptimization in RLHF. In contrast to the above works, focus reward hacking in offline preference optimization that originates from statistical fluc- tuations due to partial data coverage. Furthermore, the mentioned works conduct empirical studies, whereas here we present statistical learning theory characterizations of reward hacking.
B.3 REWARD HACKING TYPES AND COMPARISON WITH PESSIMISM IN OFFLINE RL
We now highlight the differences between the setting considered in this paper and conventional of- fline RL, explaining usefulness of defining two types of reward hacking. In the practice of RLHF fine-tuning of I.MMs, we typically have access an initial model, which already has decent perfor- mance many downstream tasks, and crcviously-oollecte preference data, which may not have been sampled from initial model (Xu et al., 2024b; Wang et al., 2024c). In this setting, we face two sources of distribution shift: one between the final model and data distribution, and the other between the initial model and data distribution. This setting is different from conventional offline RL, which considers access to an offline dataset (with possibly known data collection policy) and is only concerned with distribution shift between final model and data distribution (Levine et al., 2020).
Due to the existence of two sources of distribution shift, we find it useful to define Type I and Type Reward Hacking. These definitions motivate the design of our algorithm that achieves strong empirical performance. Furthermore, our empirical results removing dynamic labels (POWER vs. POWER-DL) as well as removing the SFT term (Appendix I.4) show that the two components contribute achieving the best empirical performance. Liu et al. (2024) and Huang et al. (2024) also consider reward hacking due to partial data coverage. However, Huang et al. (2024) assume preference data are collected from the initial model, which eliminates the distribution shift between initial model and data distribution. Liu et al. (2024) propose DPO+SFT to handle the distribution shift between the final model and data distribution, yet: reward hacking due to degradation of the initial model is not considered. In this paper, we present separate analysis for POWER (Theorem 1) and dynamic labels (Theorem 2). Combining these two results into unified analysis of POWER-DL. is challenging due to extending the analysis of dynamic labels to general function approximation, which we leave for future work.
B.4 MITIGATING REWARD HACKING
Various approaches have been proposed to mitigate reward hacking from applied and theoretical perspectives. Michaud et al. (2020) propose using interpretability techniques for probing whether learned rewards are aligned with human preferences. To mitigate reward hacking, several methods leverage multiple reward models. Moskovitz et al. (2024); Xu et al. (2024b) develop constrained policy optimization frameworks that leverage multiple reward models and assign weights to each of the reward models to mitigate reward hacking. Reward model ensembles (Coste et al., 2024; Zhai et al., 2023) aim to characterize uncertainty and can alleviate reward hacking; however, empir- ical investigations observe that they are not sufficient (Fisenstein et al., 2023). Rame et al. (2024)
20

---

Published as a conference paper at ICLR 2025
propose averaging the weights of multiple trained reward models instead of ensembles improve efficiency and performance. In contrast to these methods, our approach does not require access to training multiple reward models. To reduce the computational costs of ensemble methods, Zhang et al. (2024) construct lightweight uncertainty estimation via linear approximation that yields pessimistic reward model. However, such uncertainty quantification requires restrictive neural tan- gent kernel and approximately linear assumptions while the guarantees for our approach hold under general function approximation.
Other works use additional data to reduce reward hacking. Rita et al. (2024) leverage additional human demonstrations to calibrate the reward model and Shen al. (2024) propose methods that leverage data augmentation to improve consistency of the reward model. RaFT (Dong et al., 2023) reduces instabilities of RLHF through iterative supervised finetuning that only keeps the highest ranked responses from reward model. Peng et al. (2023) propose using advantage models instead of values. We propose seteeticall--ooundee methods to mitigate reward hacking, focusing on the offline setting. Our approaches are implemented with simple modifications to objective of DPO and directly leverage previously-collected dataset; without requiring training any additional models, generating or augmenting data, computationally expensive operation.
Among preference optimization methods, the most common approach is divergence regularization that aims at keeping the learned model close to the initial model, through metrics such as KL diver- gence (Rafailov et al., 2024b), f-divergence (Wang et al., 2024b), or Chi-squared divergence (Huang al., 2024). /BPOO (Wu et al., 2024a) calibrates (3, which is the strength of divergence minimiza- tion, based on the implicit reward gap and dynamically subsamples each batch to increase robust- ness with respect outliers. Other methods such as conservative DPO (Mitchell, 2024) and ROPO (Liang et al., 2024) design robust variants of DPO. We proved that many divergence- bassed methods still suffer from reward hacking and showed that our proposed methods outperform divergence- -based and prior robust methods empirically.
B.5 MITIGATING SPECIFIC MANIFESTATIONS OF REWARD HACKING
Several works focus on mitigating specific artifacts of reward hacking such as verbosity through var- ious designs. Design of length-controll winrate Alpaca-Eval (Dubois et al., 2024) aims at making the evaluation more robust against length exploitation, through estimation of controlled direct effect (VanderWeele, 2011) 1). ODIN (Chen et al., 2024a) enforces disentanglement of preference estima- tion and response length by using two linear heads. In preference optimization, norgalioomiiiiooioon (Meng et al., 2024; Grinsztajn et al., 2024; Yuan et al., 2024a) and length regularization (Park et al., 2024) are used mitigate length exploitation.
In this paper, we consider reward hacking due to partial data coverage that can manifest in many different ways and not just length. Our weighted entropy approach provides general framework for handling specific manifestations of reward hacking by selecting weights w(y) that are smaller for undesirable response properties, such inverse response length or preference scores. Furthermore, our approach provides a theoreticalll-sound way of incorporating such weights into preference opti- mization objective (Proposition 3), that is different from previous methods. For example, compared to SLiC-HF and SimPO, our approach results in a weight gap in the preference optimization objec- tive and a weighted SFT term. In our practical implementation, we used inverse response length as weights, which we show in Theorem prevents sample complexity to depend linearly on the response length.
Another potential benefit of our weighted entropy approach is disentangling entropy (controlled through 39) and conservatism components (controlled through 1) and 7). Adjusting the level of stochasticity of final learned model through 3 may result in alleviating the notorious overconfidence challenge, in which finHtfinctuned models become overconfident and have sharpened output prob- ability (Leng et al., 2024; Kadavath et al., 2022). In contrast, in DPO, 3 is the coefficient of the KL divergence, which impacts both pessimism and stochasticity of the learned model.
B.6 THEORY OF RLHF AND PREFERENCE OPTIMIZATION
A series of works study theoretical foundations for RLHF and preference optimization under dif- ferent settings (Zhu et al., 2023; Xiong et al., 2023; Zhu et al., 2024; Liu et al., 2024; Huang al.,
21

---

Published as a conference paper at ICLR 2025
2024; Song et al., 2024; Fisch et al., 2024). Xiong al. (2023); Zhu et al. (2023) propose provable pessimistic offline RLHF algorithms either through confidence regions or lower confidence bounds, but are restricted to the linear family of models. Zhu al. (2023) show that maximum entropy IRL is similar to the maximum likelihood under the Plackett-Lauc models; however, entropy in maximum entropy IRL is different from our use of weighted entropy in objective (6), which goes beyond opti- mizing the Bradley- Terry loss. Other works that develop provable algorithms with general function approximation (Chen et al., 2022; Zhan et al., 2023a;b; Wang et al., 2023b; Li al., 2023b) involve intractable computation. Exceptions include xPO (Huang et al., 2024) and DPO+SFT (Liu et al., 2024; Cen et al., 2024), which we have compared with POWER-DL from theoretical and empirical fronts.
C PROOFS FOR REWARD HACKING
C.I TYPE REWARD HACKING: PROOF OF PROPOSITION
We first construct two multi-armed bandit (MAB) instances and then analyze each algorithm.
C.I.1 MAB INSTANCES FOR TYPE REWARD HACKING
O. and all actions having length one |y/= 1. We consider a preference data distribution that has high coverage on the high reward arms, where the probability of comparing arms 1 and 2 11,2 probability of comparing arms and 3 is f1.3 1/N. In this scenario, there is constant probability that arm is compared with arm exactly once. To demonstrate this, let (i. denote the number of comparisons between arms i and j. We have (LN((1,) = 1) = N(1 m1,3) N
We construct three-armed bandit problem, with rewards r" -1)
I/N and the
#1,3 N
(1 - 1/N) For any N 2 2, the above probability is bounded below according to
((N(1.3) = 1) = (1 - 1/N) 2 1/e.
(13)
Conditioned on the event N(1,3) = 1, there is a constant probability that arm 3 is preferred over arm I according to the Bradley- Terry model:
P.- - 1) (r (3)
- - (1) = =0(-))=11(1+
(14)
Throughout the rest of the proof, condition the event & (1,1) 1 and 3 > 1} which occurs with a probability of at least 1/e(1 + +). We further consider two special instances of the above MAB problem, with the following specifications for the initial model parameters and the reward of the second arm:
Instance 1: True reward of the second arm is r (2) = 0 and initial parameters are (o(1) = (o(2) (3) = 1. In this case, the best-in-class softmax policy has parameters 0*
(2) (3) = O.
Instance True reward of the second arm is r* (2) 1 and initial parameters are (1)
(2) = 1, (3) 0. In this case, the best-in-class softmax policy has parameters 0* (1)
(2) 1, A* (3) = 0.
We additionally consider favorable scenario, in which oracle reveals the best-in-class values of the first two arms (1), (2).. This simplifies the preference optimization objectives as remains for the preference optimization algorithm to find 0(3).
C.1.2 ANALYSIS OF +PO METHODS IN THE MAB INSTANCES IN SECTION C.
We first record the following expression for the difference of the log probabilities of any two arms in the softmax policy class:
log (v) log (v') log exp(f(u)) log Zg log logexp(((s')) log Zo
(15)
0(y) 0(y').
Suboptimality of DPO. We show that DPO fails in both instances constructed Section C.I.1. Since parameters 0(1) and (2) revealed by the oracle, the optimization problem solved by DPO
22

---

Published as a conference paper at ICLR 2025
simplifies to
log (B (log T²(3) To (1) To (3)
max (0
log 10 T00 (1)
N
max 8(3)
log a (3 (0(3) - 60(3) ((1) - (o())))
max log a (3 (0(3) (o())) 8(3)
The first equality is due to (15) and the second equality is because (1 = 0(1) = 1. For 3 > 0 and regardless of (3). the above function increasing in ((3), and thus the maximum occurs at 0(3) 1. As result, DPO fails in both instances constructed in Section C.I.1, learning policy with constant suboptimality:
Instance 1: J(TT) ) (+DDO)) Ey (r* (v)] Eyrnturo [r'(
0.15
+ e 2c
1+ 2e
2
Instance 2: J(TT) (ftDPo) Ey (r" (v)] Eyrotuoo [r" (y))
> 0.15
1 2e
Suboptimality of IPO. We show that IPO fails in Instance constructed in Section C.1.1. Leverag- ing uniform initialization and the logit gap expression (15), the IPO objective can be simplified as follows:
min ED (108 To(y*) log TT, (yr)
Toly)) To, (v))
1 )
( 0(v - 2T
1
= min ED
2T
Since ((1) = ((2) are revealed by oracle, the IPO objective can be further simplified to
1
(0(3)-1-
1
min (3) N ((3) 0(1)
min [(3)
2T
2T
Over the interval of ((3) E [0, 1] and for any T >> O, this objective is decreasing in 0(3) and therefore the optimum is found at 0(3) 1. Thus, the policy found by IPO suffers from the following subpoptimality:
J(TT)) - (ffrr)) = Eyrarr [r'( - Ey (r-) = 2+e 1 2e
6
> 0.15
Suboptimality of SimPO. We show that SimPO suffers from Type Reward Hacking in both in- stances detailed in Section C.I.1. Following the same steps in our analysis of DPO and since 0(1) 0(2) = are assumed to be revealed by an oracle, SimPO objective simplifies to
max log (3 (0(3) ((1) 7))
((3)
N
Regardless of the value of y and for any 3 > O, the function log a (3 (0(3) 0(1) - 7)) is increas- ing in 0(3) and thus optimizing this objective over 0(3) E [0, 1] finds 0(3) 1. Therefore, policies found by SimPO in both instances in Section C.1.1 suffer from constant suboptimality:
Instance J(x²)= J (ftsimro) E Eyn-Tr [r (y)] + (v)
> 0.15
2
1+ 2e
2e
Instance J(x²) (fiSimoo) Eyr-TTo [r (v)) F (r [v))
> 0.15
+ 2e
Suboptimality of xPO. We analyze XPO for Instance 2 constructed in Section C.I.1. The XPO objective is given by
max (1 0(3)E(0.1]
1 NV
1
i log (a (cripp)
T1 no. (1)
(2)
(1)
(2)
B log
log
To, (2) To (1) Te (2)
(2)
To
T2
To
Pizz log
clip2 18
llo TTO (2)
loog
T8, (1)
T8, (2)
Te,
(a <lipp
(<llipp
T3)
log T)
(3)
To(1)
log
3
log
+
Too (3)
no. (1) Te (3) Te (1)
(16)
23

---

Published as a conference paper at ICLR 2025
By construction, (1( 0u (2), Tel (1) = no, (2), and 8(1) ((2) 1 are known, and therefore
the first two terms in (16) are equal to zero:
T& (1)
TTa (2)
nt (1)
T2(2)
log
log
+
Te, (1) Te, (2)
Te (1)
Teo (2)
exp((((1)
exp(0(2))
= 8(1) - (1) ( 0) (2) +
Zo = O.
Too (1)
Zo
Thus the maximization problem in (16) is simplified to
To (3)
log TT(1)
TT(3)
To(1)
max [(3)
log oclip,
log
T0 (3)
70 (1)
Teo
Too (1)
(exp(((3)
exp(O(1))
max (3)
log 3
((3) - (3(3) - 0(1) + 00(1) + +1 2e)
Ze
eZo
[exp(O(3)) 1/e exp(((3)) + 2e
max log 9 ((3) + (1 2e) 6(3)6/0.1]
It is straightforward to check that the above function is strictly increasing over 0(3) [0, 1] and the maximization step finds ((3) = when 0 < 3 < 1/3. This results in xPO finding the uniform policy (11 (2) (3) 1 and thus suffering from the following suboptimality:
2e
2
((To)) x(x = Ey (v)] Eyntiro (r'
0.15.
2e +1
C.2 TYPE II REWARD HACKING: PROOF OF PROPOSITION 2
C.2.1 MAB INSTANCE FOR TYPE II REWARD HACKING
We construct a three-armed bandit problem with the following true reward structure: r*(1) = (2) (r(3)) This reward function implies the following parameters for the best-in-class policy: 0* (1) = 8* (2) = O, 0* (3) = 1, leading to the following policy:
e
n& (1) = :g. (g) = 2 +
To TT- (3)
2+c
(3) (3) = c/{+te), We further consider the following initialization: (n(1) 0n 0, (3) = 1. Suppose that the comparison probabilities between the arms are μ4,2 = - 1/N and 11,3 = 1/N. Following the same argument as Section C.I.1. there is constant probability that arm 3 is compared with arm exactly once and that arm is preferred to arm 3. Throughout the rest the proof, we condition on the event (1, = 1, 3). We further consider favorable where the optimal parameters corresponding arms 1 and
The performance of the above policy (
2 are revealed by an oracle ((1) 0(2) = O, leaving only 0(3) to be estimated.
C.2.2 ANALYSIS OF +PO METHODS IN THE MAB INSTANCE C.2.
Suboptimality of the DPO+SFT policy. We show that DPO+SFT suffers from reward hacking for any 7) 2 and hence the argument also shows reward hacking in DPO as a special case. The objective of DPO+SFT is given by
max E log ( (10g To (V(+t) log Te, (2/
(3
))
+ SE [log To (y+)]
Te, (x+)
Since ((1) ((2) 0 are revealed by an oracle, we focus on the terms that involve 6(3) in the objective:
1 logo (3 (1g log To(1) log (3)
max (3)E(0011
Ti
N
Te, (1)
Te (3)
n/ (1 N [D1 log (1) + pir log (2)) 17 "N
1 log To = =T2
24

---

Published as a conference paper at ICLR 2025
Applying the softmax policy parameterization and using the fact that ((1) (u = 0 and (o 1, term Ti simplifies to
1 Ti N
1 N
log (a (0()) 0o(1) ((3()))))())()))))))
log (o (3 (1-3())))
The term 72 can also be simplified by substituting values ((1) = 0(2) = 0:
1 ( log Zo) + pzz (0(2) log Zn)] + 1188 N (0(3) - log Zo)
T2 1
N
n/8
N 0(3)
log 70
n/8
N ((3) log(exp(0(1))) + exp(O(()) exp(O(3)))
= n/ 0(3) log(exp (3)) + 2)
Combining terms T1 and T2, the objective becomes
1
log (o (3 (1 (()))) n
N (3) log(cxp(((3)) + +2)
8(3) N
We show that for any N > the above function decreasing in 0(3) for any 8, 17 and 0(3) E [0, 1]. Since the first function N log (ar (3 (1 0(3)))) is decreasing in 0(3), it is sufficient to show N (3) log(expi log(exp(0(3)) 2) decreasing in 0(3). Derivative of this function is over 0(3) E [0
and 2 3 is bounded by
1 exp(((3)
<.
N exp()(3)) +2
N 3
Therefore, optimizing over 0(3) E [0.1] finds 0(3) = 0. This leads DPO+SFT to find uniform policy, which suffers from a constant suboptimality:
( ) J ((TPOOSST) 2 2 e
0.2.
Suboptimality of the IPO policy, Similar to the analysis of DPO+SFT, for the IPO objective, we only focus the terms that include 0(3):
min 1 log T1( log (3) ((3)
1 )
(1 1
min 0(3)
0(3)
N Te, Te, (1)
To (3)
2T
2r
Since 0, solution to the above minimization is
0(3) max { {0
1 2r
Therefore, the suboptimality of the IPO policy is given by
exp(max (0, 1/(2r)))
J( ) J (ffro) 2+e 2 cmax 0, 1/(2r)))
And for the regime with have J(TT)) (firr) 0.1.
Suboptimality of the SimPO policy. The objective optimized by SimPO over the term that involve 033 is given by
1 log (3 (log (1) log T3(3) 7))
max 0(3)E(0.1]
N
log a (B (-0(3) -7))
8(3)c(0.1]
25

---

Published as a conference paper at ICLR 2025
The above function is decreasing in 8(3) therefore SimPO finds 0(3) 0 and suffers from the followingg suboptimality
e 1 > 0.2.
J( ) J (ásinoo)) 2+e 3
Suboptimality of the xPO policy. The objective optimized by xPO is given by
(1 (1- N
1
11022088 log (o (<clipp/
(1)
To (2)
To (1)
Te (2)
max 1[
3 log
log
+
Too (1)
To (2)
To (1)
T0 (2)
log (2)
To (1)
T2
Tp (1)
+ P log
clip2
log
186 (2)
16o (1)
Too (2)
Too (1)
(a
To (1)
To (3)
3o (1)
To (3)
log
(cipp
clip2
B log
log
N
Teb (1)
To (3)
Te (1)
Tea (3)
(17)
By construction, Teo (1) = Too (2) 1/(2 + e). and 0(1) ((2) revealed by an oracle, and
therefore the first two lines in (16) are equal to zero:
log (1) log T (2)
To (1)
To (2)
To, (1) To (2)
Te (1)
Too (2)
= =(1 (1(1) ((2 + (2) + (2 + (exp(((1)) exp(O(2))
= 0.
Zo
Zo
The objective (17) can therefore be simplified
(1) log To (3)
+ TT(1) - T3)
max (3)) N
log
clip2
_ log
To, (1)
Top (3)
Tep (1)
Too (3)
To
To (3)
max log ((3)
clip2
B
0(1) (1()) - 0(3) + (3( +
10. (1)
T0 (3)
1 0(3) + Te
33)
max log ((3)
clip2
TBo (1)
Too (3)
+ exp()(3)) (1 - exp(((((3)) - 11) 1))l)
2 + e
max logar 0(3)E(U.1]
clip2
1 0(3) +
The last equation applies the definition of no and substitutes values for (1()), (o(3), 0(1). The function
3 (1-0 [(3) + 2+ exp(((33) (1 - exx(((()) 11)
2+e
decreasing in 0(3) for 0 < B < 1 and that for 0(3) E [0, 1]. the above function remains between and 2. As a result, the maximization problem leads to 0(3) = O. Thus, XPO finds the uniform policy xxr = Unif ((1, which suffers from a constant suboptimality:
J(Zo-) [(xx00)= 2+c
e
1
> 0.2.
3
C.3 PROOF OF PROPOSITION
Policy learned by POWER in the MAB instances in C.I.1. We show that when faced with the Type Reward Hacking MAB instances of C.I.I, a more general variant of POWER mitigates reward hacking. Let g(x) be increasing function with bounded derivative: g'(x - 1) < By for any x E [0, 1]. We consider the following objective:
max ED [g (w(y+) log To(y (x) - w(y log (o (x) + w(y+) - w(y )) + 3 (w(y) log ne (v))
The POWER objective is a special case of the above objective with g(-) = loo (.). Note that we have g'(x - 1) = a--(a (x 1)) < 1, and therefore the bounded derivative assumption is satisfied.
26

---

Published as a conference paper at ICLR 2025
For the MAB instances in C.1.1, the optimization problem simplifies to
max (3
(o(0(3) - 1) + m)(3)) log (exp(0(3)) + + 1)-
N
We show that for any > B.())) (24e) the derivative of above function is negative. This is because for
any (33 [0, 11. have
1 N (g' (0(3) 1) 1)) exp(0(3)) +
exp(O(3))
1
[Bg +
11)
N
2+e
Bg
1
2 N
N
B (2(2+e)
N
N (2+
2
0.
Because the function is decreasing in ((3) the optimum is at 0(3) = and thus the algorithm finds Tg. Finally, the conclusion also holds for POWER special case and thus fTOWER = Tg-
Policy learned by POWER in the MAB instance in C.2.1. The POWER objective with response lengths equal one is given by
max E [log (8 mo(t)) log TV }))] ME [log (y(+))]
With similar argument as in our analysis of DPO+SFT, we obtain the following objective:
max log (a (3(-0(3))) + nB 0(3) log(cxp(0(3)) +2)
63)) N
easy to check that the above function is decreasing in 0(3). Therefore, optimization leads to (33 0 and the policy learned by POWER suffers from constant suboptimality
( (APOWER) 2 3 > 0.2.
D POWER OBJECTIVE DERIVATION
This section is organized as follows. In Section D.1 we record a useful proposition that captures properties of the optimal policy the WER objective. This result comes in handy for deriving our preference optimization abjective-which relies on the equivalence between minimax and maximin objectives as well as finding closed-form solution for the inner maximization problem. Section D.2, we prove that under certain regularity conditions on reward class R. the maximization and minimization steps in objective (6) can be interchanged. With these two results hand, we prove Proposition 5 in Section D.3, which gives the POWER objective.
D.1 OPTIMAL POLICY FOR WEIGHTED ENTROPY REWARD MAXIMIZATION
For the WER objective, we have the following proposition which shows the uniqueness of the op- timal WER policy the support of prompt distribution, and connects this policy to the reward gap.
Proposition 5 (WER Policy). For any 3 > O, reward function r, any I. E X with p(x) > O, and any action pairs y, y E e,, the policy ny that maximizes the WER objective (5) satisfies the following statements:
1. For any B > 0, policy n, is unique on the support of p.
2. Policy क, satisfies the following equation:
r(x,y) r(x, y²) = 3 ((w()) log n, (u]2z) - w(y/ log n. (c) + +wo()) - w(v())) (18)
27

---

Published as a conference paper at ICLR 2025
Proof of Proposition S. The WER objective solves the following optimization problem:
max Exup
Σ
(1)2)) [r(x,)) Swiy) log ग (yylz)]
8
(19)
Σ
<y/l)) =1 VIE
We find the optimal policy for each context in the support of p independently. For any such æ, we rewrite the constrained optimization problem using Lagrange multipliers:
Σ (ulx)) [r(,, v) -Bw(y) log (ulo:)] - Arr Σ [
Notice that the above function is concave in <y]x) due to w 3 > 0 and thus the solution is unique on the support of p and the stationary point the maximizer. Taking the derivative with respect to ( .r) and setting it to zero finds an equation governing the optimal policy n,
r(z, v) Bw(y) log ग. (g(x)) + Bw(y)
Ax
Thus for any y, y', one has
r(a, v) r(x, (x, 3 woy))oog (,lı,(ulz) + wly) wryy) mr which concludes the proof.
Tr (v1x) - v())))
D.2 MINIMAX OBJECTIVE EQUIVALENCE TO MAXIMIN OBJECTIVE
In this section, we show that that the maximin objective (6) can be written as a minimax objective under certain regularity conditions. Define the following notation to denote the weighted-entropy robust reward objective for any x E II and r E R:
((T, = LBT(F) + (Eorppyn [r(z, y)] Earp,y'<r'
(H
(20)
We follow the approach of Liu et al. (2024) and impose regularity conditions on the class R, which contingent upon our definition of Φ), to show the maximin and minimax equivalence. Formally, we make the following assumption.
Assumption (Regularity of the Reward Class). We assume that class R satisfies the the following:
1. The space R a non-emply compact topological space;
2. The function & defined (20) is convex-like in R; that is, for any F1,72 E R. π II, and a E [0, 1), there exists E R such that
rs) S (o((,r))) (1 a))(T.r2).
The above condition is satisfied in several special cases. For example, it is satisfied when R. is convex such linear class (Xiong et al., 2023; Fisch et al., 2024). As a more general case, if R is a Lipschitz continuous class, we can conclude function ((T, -) to be convex over R as Φ(T, :) is sum of linear term in and a convex term (r)
Under this assumption, we have the following proposition showing the equivalence between max- imin and minimax objectives.
Proposition 6 (Equivalence of Maximin and Minimax Algorithms). For the policy class II {I X + A(Y)} and reward class R satisfying Assumption define policy ne to be the optimal policy corresponding the minimax reward function, i.e.,
Te E arg max ((T,f) where
E arg min max ((π, PE well
TEII
Then, policy ne is also the optimal solution to the maximin objective, i.e.
T... E arg max min ((T, r).
REll
PER.
28

---

Published as a conference paper at ICLR 2025
Proof. We begin by recording the following lemma that shows the equivalence of the maximin and minimax objectives under the assumptions on R. Proof of this lemma is deferred to the end this section.
Lemma (Equivalence of Maximin and Minimax Objectives), Given the policy class 11: : { : X +
A())) and reward class R satisfying Assumption 1, the following statement holds for & defined in (20):
max min ((T,)) = min max ((T, rER well
Tell rER
Denote the policy solving the maximin problem by & arg max min ((T, The duality gap of f, ft is given by
dual(f, #) =
max (iTT, f)
min ((f, rER
#ell
max ((T,
min max ((T,T) + min max ((T,T) TER TEll
- min p r) rER
FEll
TER #ell
(21)
max (( min max ((T,)) + max min ((T,))
min ((f,r) r)
Fell
rER. #Elf
sell rER.
rR
In the penultimate equation, we applied Lemma 1 and the last equation uses the definition of f and Ā. The duality gap is also equal to
dual #) = max ( f) - 6(f. + o(ā. f) min @(t, r) rER
(22)
wEll
Comparing (21) and (22), we conclude that max; (x f)
Φ( which means E
arg ea (f,, x). Recall that by definition, we also have E arg max ((F,x). By the
uniqueness of the WER optimal policy over P as established in Proposition 5, we conclude that
TT(-1x)) +-|[z) for any with p(z) O. Since ((, r) depends π only through its value on the
support of p. we conclude that πę arg max minpeR R @TT, r), which completes the proof. O
Proof of Lemma 1. This result relies on minimax theorem by Fan (1953) presented in Lemma 2. We prove that all the requirements of this theorem are satisfied. First, by definition, policy class Il is a non-empty convex set and by Assumption I, the reward class R non-empty compact topological space. Second, function o(T,T) is concave on II because it is a sum of a linear function in π and (weighted) entropy of æ. Lastly, by Assumption 1, function @(T,r) r) is continuous and convex-like on R. Therefore, we apply Lemma 2 to conclude the equivalence of maximin and minimax problems on ø.
D.3 PROOF OF PROPOSITION 3
We start by deriving the objective (8) by changing the order of maximization and minimization in the maximin objective (6), which valid on the account of Proposition 6. Writing the minimax objective and rearranging some terms yields
min (rT(r) + 17 max (Ex=pyy>T (r(x, v) - r(a,, + (H),
(23)
The inner maximization problem over is the same the weighted entropy reward maximization objective (5) minus a baseline term, which is independent of ा.
We apply the reward gap expression provided by Proposition 5 that governs the maximizer policy Tr as well as the definition of weighted entropy in Definition to find the maximum value of the inner optimization problem:
max Ex-par<s mn [r u) - r rzz v/ 3
p....rrr y' (w(y) log ग. (g|=) w(v') log ( (x:) (w(y) - w(y')) w(y) log क. (ylx))
-8Ermpara w'xx! (w('') log (v/ |z) (w(y) -0w')w(y)
We substitute the above expression back in the minimax objective (23):
min (T) - p.E.pp.wrn. [w()') log o, [z) - (w(y) - w(3
(24)
min LRT(r) - 11 (w(v') log म, (1/ [=)]
(25)
29

---

Published as a conference paper at ICLR 2025
The above equation uses the fact that (wr( - ()))) is independent of r. To obtain the final ob- jective, we replace the reward gap expression from Proposition 5 in LBr(r), which cocludes the proof.
D.4 AUXILIARY LEMMAS
Lemma 2 (Minimax Theorem; Fan (1953)), Let X be nonempty (not necessarily lopologized)
set and y be nonempty compact topological space. Let f X x be lower semicontinuous on Y. Suppose that f is concave-like on X' and convex-like on v, i.e., for any 21, 22 E X, y E V [0, 1), there exists 23 E X such that
f(13, v) 2 a. f(11, + - o.) f(22.
and for any Un U2 V x E X, [0. 1], there exists yss E Y such that
f(z va) S ((x (1 (1-18) B) [xx 92).
Then the following holds:
max min (x, v) = min max ((x, ((, y). ZEA yEy
yEy ZE
E FINITE-SAMPLE ANALYSIS OF POWER
This section is organized as follows. We begin by presenting the definition of single-policy con- centrability for offline preference optimization, which characterizes the coverage of the competing policy in the dataset, in Section E.I. In Section E.2, we prove the sampee guarantees for POWER.
E. SINGLE-POLIES CONCENTRABILITY IN PREFERENCE OPTIMIZATION
Definition 2 (Single-Policy Concentrability; Zhan et al. (2023a)). Given a policy and ground truth reward r", the concentrability coefficient of offline data distribution is with respect to the reward model class R and the baseline policy x! is defined
Eamperrte (r' v) - (x,1)) (r(x, r(x,))
CR, (R, = max 0, sup PER
(26)
[ZN.N'' [(r (x. r" (x, (r(z, ((,,)))))
Single-policy concentrability coefficient in offline RL quantifies the extent to which target com- peting policy æ is covered by an offline data collection distribution μ. the offline RLHF setting, dingle-policy concentrability defined in the work Zhan et al. (2023a) also depends on baseline policy T'.
E.2
PROOF OF THEOREM
To prove simite-sample guarantees, we use similar argument Liu et al. (2024), adapted to the weighted-entropy objective and combined with the bounds on weighted entropy and the special weights as inverse response lengths. Suboptimality of the learned policy á sTOWER with respect to competing policy π can be decomposed into three terms:
((T) J(#) Ez~py>> [r* (s,, v)) Ex-pyr (x, = T2 + T3.
where Ti defined as
Ti = Exnp. (I, (2. - (Hw
(27)
min / [r(x, 9) r(x, v/) (T)) LBr(T)}
FER
T2 defined as
To = - min I (x(x,)) - r(x, v/) - 3 (T)] + [BT(T))
PER
(28)
-E (r, (xx -
30

---

Published as a conference paper at ICLR 2025
and T3 is defined
T3 (H (+)] We will prove the subsequent section that for weighted entropy Hm)(T) with general weights, terms Ti + T2 and T3 are bounded according to:
(29)
T. + T2 (R, T²) + 1)R2
(30)
VN
T3 (T)
(31)
Summing the above bounds, we conclude the first claim:
- - VN ((ITTR, x'1 +1) R8 HT)(
Furthermore, in the special case of wyy) = 1/1b/. we have the following bound on T3:
log/V|
T3 5
(32)
VN
The above bound combined with (30) leads to the following rate
J( s 1 VN ((C)(R,*')* + +1) R + log/V))
E.2.1 PROOF OF THE BOUND (30) ON Ti +122
Bounding Ti. the maximizer to the following objective
* E arg max min 7 DE (r(r, - (k, -H + Lpr(r)
#Ell rER
We use this fact to bound the term 7i according to
Ti < E (r" r -3H (T)
min { EER ( (Taappraa [r v) r(x, s/) 311 + LBT(T)}
max PER
E Exuperae [r (x. - - (z, v²) (r(z, 3) r(æ, y )) '7 - LBT(r)}
(33)
Bounding T2. By realizability of the true reward function r* E R we bound the term T2:
Th S Ez-py~r (x, 3) r" (z, v/) ,H (T) + +1 7 Lwr(r")
- (x, - -rx - 3H (T)] -Br(r")
(34)
Bounding Ti + Tz. Combing the bound (33) on Ti and bound (34) on T2, it remains the bound the following:
T + T2 < max
E Ex-p.y wor.owow [r' (x, v) (x, (r(z, r(z, [/)] Ti,1
PER
(35)
+ n - (LBr((r)) LBI(T)) = -T.2
Define the following notation:
Exorrr (r, - r (x, - (r(x. u) - r(r, u²))
(36)
Term Ti.1 is directly bounded by C (R, x A'A,, based on the Definition 2 of single-policy concen- trability, and we subsequently prove bound on Ti, according to:
T1, - C. (R, A')A,
(37)
T1,2 < -2 A 31
(38)
R2
31

---

Published as a conference paper at ICLR 2025
where R = exp(R) and v maximum over r, the bound on T1 + T2:
Adding the bounds T1,1 and T1,2 and taking the
Ti + T2 S max { ( CT (R, n "',, -
2
3
(39)
Tj
ny.
(R,
2 'RR²2
3L
(40)
8
TN
The last inequality uses the fact that az /6t/ (It² /N), the above bound becomes:
bz S a² /4b for any E R. By the choice of
Ti T (R, T') + 1)
VN
Proof of the bound (38) on Ti,2. In the view of the uniform concentration result in Liu et al. (2024, Lemma A. with probability least ō setting (6/tN)- the following bound holds for any E R
3c N
LBT(r*) - LBT(r)= - - Ditellnnger (Pr, (-12, v.
(41)
where Pr(-|z, y') is the Bradley-Terry preference probability given a reward model r as defined in (2). The Hellinger distance can be bounded by total variation (TV) distance according
D Piellinger 8'
2 Div (Pr.((....))))))PP(((
-r(()) r(r,
2 H (x, (x)
r (I,
(()"(1,)) r (z.1/)) ((r(2, r(z, (e,,)))) v/))
2 (r (2. v) r" (x, (x, 1/) - (r(x, 3) rlz, v/))
(42)
The penultimate equation uses the fact that (-x) = - o (xx) and the last inequality is due to bi- Lipschitz continuity of the sigmoid function over [-R, R, Rl; see e.g. Liu al. (2024, Lemma A.2). Applying the bound in (42) to (41), we have
((r"(,,)-->((r, (r(a, r(x, -(),
3
LBT(r*) LaT(r)= S -2 -F ~M
2A,
31
R2
N
where the last equation uses the definition of A, provided in (36), completing the proof.
E.2.2 PROOF OF THE BOUNDS (31) AND (32) ON Ta
In this section, we prove the bounds on T3 as delineated in inequalities (31) and (32) through bound- ing weighted entropy. The key bounds are encapsulated in the following lemma which asserts that weighted entropy is non-negative and for special case of weights w(y) = 1/lyl it can be bounded from above. The proof of this lemma is presented at the end of this section.
Lemma (Bounds on Weighted Entropy). For any weight function w(y) 2 0 and any probability distribution p(y). the weighted entropy satisfies (p) 2 0. Furthermore, when weights signed according to w(y) 1/ly), with y denoting the response length, the weighted entropy is bounded by H. (p) < log) Vl. where [V] the size of the vocabulary.
32

---

Published as a conference paper at ICLR 2025
Based on Lemma weighted entropy is nonnegative. Setting B x 1/VN immediately gives the bound (31) on T3:
T3 3= (T) (w) = (TT)
(43)
N
Moreover, when w(v) = 1/lul by Lemma 3, we have
Ta S Hw (x)
log V
VN
Proof of Lemma 3. First consider the case for any general non-negative weight function w(y) > 0. This ensures that the weighted entropy is non-negative because:
Hp) =
E wy))p(v) log
Σ w())p(y) log
2 0. p(y)
1
Next, we provide an upper bound on the weighted entropy when w(y) 1/ly/. Define to be random variable denoting the length response. The weighted entropy can be decomposed as follows
Σ =
Σ
| Pz(2 = 10))(( 1Ø/= 2) lo =10))(( 10|= 2)
y
= Σ
|yl" Pz(z = 10))p( - 2) log pz = (ul) - logply |8/= 2)
y
Σ
|ul" Pz(z (2 = (Bl) Σ
p(y | 2) log by|= 2) = T3.1
a.s.t 1=|=
Σ Pz(= = bol) lo lvl) Σ
plu lul= 2) = T3.2
s.t. lu/=
For the term T3.1, we have
Ta.1 Epe = (ul)
Σ
Py)=(y 12 (v/) log (y)=(1 2 111))
The sum - Pyl = (ol) log P (y | lvl) is the Shannon entropy of conditional distri- bution, which reaches its maximum when the distribution is uniform. Consequently, the maximum of this conditional entropy for a fixed length lyl is given by log Dilil |y|log/V]. Substituting this bound back to Ti gives:
T3.1 S Σ p=(z= = (y]) |v/log|V|= log |
For the term T3,2, first note that Ey ply |yy= 2) 1. Therefore, with the maximum response length denoted by and since |y| 2 1, we have
Ta Σ
(ul) log lol) S Pe Ep((= lv/ (z = (vl) < log L. 1 p=
Combining the bounds on T3,1 and T3.2, the upper bound on weighted entropy when using inverse response length weights is log|V|+ V/+ log L, which concludes the proof.
F DERIVATIONS AND PROOFS FOR DYNAMIC LABELS
F. DERIVATION OF THE LEARNING DYNAMICS
In this section we compute the learning dynamics POWER over preference dataset D ((x, yo y² I)) with the label notation defined in 2.1. Here, 1 indicates that y was preferred and
33

---

Published as a conference paper at ICLR 2025
1 indicates that = 1 was preferred. With this notation, the POWER objective from Proposition (3) is given by
max Ep llg of ( (w((o)) log (o( - w(v") log (x())x)) + (w(s*) - w((r)))
(1 - 1) log o (w((e) log (o x)) w(v') log To (z) + (w()") - -000--w(()) (w()")-((v)))
m(bn)]
+ngED (w(y") log To (z) + (1 - ()) log To To (v
To simplify presentation, we consider the softmax MAB setting with B 1, w(y) 1, 0. In this case, the objective simplifies to
min -ED log a (log (logTo(y') log () + (1 1) log o no (v²) - log To to(
= min Ep log
log Z - log cxp( (1 (1))) + log Zo)
(1 -1-1) 1) log o (log exp(O(y")) log Zo log (0gexp(((()))) )) + log Zo
min
Ep (Blg))(((())) - ((v")) (1 log (0()") - -
To understand the updates to the model parameters, consider the empirical probabilities derived from dataset comparisons: fo,1 the empirical probability of comparing y and y', and Í is the empirical probability of preferring y' over yº, conditioned their comparison. Isolate the updates to the parameters f(y') and ( based on comparisons between y and y' and through batch gradient descent. We allow the preference labels change across gradient steps and thus updates to the parameter gap at step and with a learning rate a is given by
(141 - (1+1
( + poo 10- (10-(111)))(00)))-- 0c('))
= -c('²) 0rlv") aaua.1
- -mpo-th p 4.) a(br(r') -
We used the fact that M(og (1)) o--x). Since a<x) = - -o(=x) and Pi1 + 1 1. the learning dynamics simplify to:
0 (v))
= - + asaa (0000 po 1)1 (o - (0((6))))(({)) -
F.2
PROOF OF THEOREM 2
The proof organized as follows. We start by establishing a lower bound on the dynamic labels. We subsequently use this lower bound to prove bounds on the parameter gap in low-coverage and high-coverage cases separately.
Lower bound on dynamic labels. The dynamics of labels are described by the following equation:
((()) (1 (1100) -1c) مilo - (1 - f1100)
((d))
i, =
(44)
Without loss of generality, we assumed that Āi1o 1/2. This condition is easily met by appropri- ately ordering the responses. Define:
1 - 1i120
(45)
Pıyu (1 (1100)
34

---

Published as a conference paper at ICLR 2025
Given that 0 S (de) I 1 and the assumption P1>0 (1 - 11100) 201100 O, find the following lower bound on Te:
( - (1- -1>0) -4) h)) 2 ?
o(de)
-(1 - p120)
1 = ( P1~0 (1 -110))
M1>0 p1>0) (1) =-(==-100 ())
The subsequent lemma establishes a lower bound on l using Grönwall's inequality, with its proof provided at the end of this section.
Lemma 4. Suppose that 4 satisfies the following inequality i, 2 7(-k - 44) with initial value Then, we have the following lower bound - (N + exp(-?t).
We proceed by separately analyzing the scenarios of low coverage and high coverage.
Low coverage case. The coupled dynamical system in (12) satisfies the following equation:
7 de aspo,14 = 0
201
Upon integrating the equation above and considering the initial condition 4o = 1, it follows that
2/100 - (de do) = a (1 - 4)
Applying the lower bound from Lemma yields:
20130 (de do) = afoo,1 (1 (1)
S ajoo, (1 + k- (1 + exp((7(()
Consequently, we conclude that
[d. dol S axoo (2 (1+ (1+ exp( exp((-tt) afio.111100 <
exx
onh0 S E
where we used the definition of x and the fact that by assumption axp//es S 7.
High coverage case. We extend the argument by Zhu et al. (2024) for general initialization do and establish the final convergence rate for proper choices of hyperparameters. Consider Lyapunov function 1 = (d.))) - 0 Derivative V2 is given by
V/ = 2(((d) ( o (d))
20/m..1 (o(db) - in (di)) ((2 - 1)4 +1 1-:1-~10---- o(()
22pp.. ( (((ch) - -mx0)) + 22mo 2amo (o(()) - (I (2 -)(4 1)
22 de) (d(s)) (2 - 1)(4 11)
Let T > 7T. We find upper bound V by applying the bound on given in
Lemma 4 and using the fact that Āiso, E [0;
Vi < apo1)(())((-d)) Vi + (I + 1)(1 - expl exp(->t)))
(46)
1/ (N 1)(1 - exp( (xxp(-e))) ())
(47)
Now consider two cases. The first case is that for any 0 < t < T, we have V/ 2 2(x + 1) exp(-e)). In such scenario, V, is a non-increasing function because
(- V/ + (n + 1) exp(-c))) < - s (48)
V <
Next, we analyze the term ((d))((-d)) We establish a bound on (d, (-d,) for the case of a (dg) S ji the case of (dn) 2 λaıyn can be proved with similar argument.
35

---

Published as a conference paper at ICLR 2025
We prove that when (do) I P1>0, then must have odd) S A1>o for any Assume otherwise that there exists some to where (dd)) > Piso. By continuity of o(de) and since o S Ā11-0= there exists I to such that add,) ) Pi However, this results in V1, 0 Via which contradicts the fact that V2 is non-increasing. Therefore, we have (d4) S f1>0. Moreover, since also have a (do) v (de). Thus, have (do) S (d,) S ii and
Vi is non-increasing, use this fact to find the following bound:
(dd))o (L (())))))))) 2 min {d(d)) o(do)), PIso(L - =c.
(49)
Applying bound (49) to (48) and integrating over t on both sides, we have
V S exp(-cojio,1 S exp(-coopo.1t).
(50)
We now consider the case where for some lo. have Vi, < 2(x 1)(1 exp(-e)). We show that in this case we must have VT < 2(x + 1)(1 - exp(-e)). Assume otherwise that 1/ > 2(K + 1)(1 exp(-e), then by continuity there must be t1 such that V4 2(x 1)(1 exp( (c)) However, inequality (48) implies that for any t E [T, V2 is non-increasing, leading to VT S 2(x + 1) exp(-e)), which contradicts our assumption. Therefore, we know that
VT S 2(k + 1)(1 exp(-c))
(51)
Combining the two bounds (50) and (51) on Vr, have:
VT E max (2 + 1) cxp(-c)), cxp(c)coo.0-
(52)
Subsequently, we show that 2(x (x1-cx(((c) s
provided that 6o Σ
exp(-1/4)
2(x+1)
Utilizing the inequality exp(-2) N 1 x, we have
exp( 1/4) exp(-1/4)
exp(-cn) 2 1 - c 2 1
2(n + 1)
Using the fact that 1/4,x 0 and that 1 1, we conclude
2(x 1) S 2(K + 1) cxp((()) + exp( 1//4)
< 2(x + 1) exp(-(n) + exp(-c)
< 2(x + 1) exp(-encrão, T) exp(-corpo,1 IT)
< 2(k + 1) exp((Enaroo, exp(- 1").
This leads the bound VT S exp(-corpa.11) 17).
Proof of Lemma 4. The result can be shown using Grönwall's inequality. Rewriting the inequality as l + 7he 2 -yy and multiplying both sides by exp(7l) gives:
exp(1t)l, + exp( 2 exp(t)))
Notice that the left-hand side is the derivative of 1, exp(1t). This yields
i (4 exp((tt) 2 -xxexp((1t) = L'" ōt (le (xxp()))) 2 f" -yxexp((t).
(53)
Ot
The left-hand side and using the fact that to is given by
S.
0 at (4 exp(rt)) = 4, exp(1t) to exp(7 0) exp(1t) - 1.
The right-hand side
/. -xxxxx((7t) -x(exp(1tt 1).
(54)
Substituting equations (54) and (54) in (53), we have exp(()) - -x(cxp(1t) - 1) + 44 exp(>t) 2 -xcxp(11)) + 1.
(55)
Multiplying both sides with exp(->t) gives the final lower bound on 4
0
36

---

Published as a conference paper at ICLR 2025
G POWER-DL PSEUDOCODE
Algorithm POWER with Dynamic Labels
1: Inputs: Dataset = { v. y2 ) hyperparameters 1, 7, learning rate a, weight func- tion w(y) (e.g.. v(()) = 1/ y),, initial model Te
2: Initialize 1 = 0 for all (2,, V,T v v,).
3: for f (l..... 7') do Objective function:
LPOWER-
,,,,,,,,,,, -1)) LO Vi v SHOWER-DL(';O0)) DL';; ED LP ,,,,,,
LPOWER( - log (o (a(((1(0)) log (x) way ) log Toly lar)
w(y+) wiy w(y 11) I n)w((y)) log (( (2))
5: Update policy with stop gradient labels:
0z-1 1 V sg(ry LPOWER DJ (P:
7: Update dynamic labels:
(w((t) log ne(s" (x) w/y5 log TOly (x) w(vt) w(v -yy
1 (1
1-)44--- +
Putroi Poirot
8:
Return: TO+
H EXPERIMENTAL DETAILS
H.1 PREFERENCE OPTIMIZATION OBJECTIVES
We compare POWER with a variety of preference optimization methods as baselines, summarized in Table Several of these methods include divergence minimization (typically KL divergence) against the reference models, such as DPO (Rafailov et al., 2024b), IPO (Azar et al., 2024), which considers a different loss function on preferences, and XPO (Huang et al., 2024), which combines Chi-Squared with KL divergence for stronger pessimism. We also implement an offline variant of SPPO (Wu et al., 2024b), derived based on a self- play mechanism to improve upon the initial model.
We additionally consider several objectives that do not include the reference model. These include CPO (Xu et al., 2024a), which considers DPO with a uniform initial model; SLiC-HF (Zhao et al., 2023), which uses hinge loss; RRHF (Yuan et al., 2024a), which applies hinge loss with length- normalization on the contrastive term; and ORPO (Hong et al., 2024), which proposes odd ratio terms without initial models to contrast the chosen and rejected responses. Finally, SimPO (Meng et al., 2024) that removes the reference model from DPO and adds length-normalizzttio and a mar- gin. We implement POWER-DL. by combining the objective in (8) with dynamic labels (11), where we estimate the empirical preferences with fi1-0 = l. We also compare against POWER, which corresponds to y removing the dynamic labels.
H.2 INSTRUCTION FOLOOONN BENCHMARKS
Benchmark details. We select the default choices in benchmark baselines. In particular, for AlpacaFvall 2.0 benchmark, we use GPT-4-Preview-1 106 as comparison baseline model, and GPT-4 the judge model. AlpacaEval 2.0 then compares responses generated by our PO trained models with responses from the baseline model, and length-controlled (LC) and raw winrate (WR) are the GPT-4-0314 14 the baseline model,
computed metrics. For Arena Hard benchmark, and GPT-4 as the judge model, and the reported metric is winrate againts the baseline model.
Decoding hyperparameters. We follow Meng et al. (2024) and sampling decoding strategy with a temperature 0.9 on AlpacaEval 2.0 for all methods. For Arena Hard, we use the default approach of greedy generation.
37

---

Published as a conference paper at ICLR 2025
Table 3: Different preference optimization objectives and hyperparameter search range.
Method
Objective (Min)
Hyperparameter Range
DPO
logg o log
mely"Is) may (v [2]
Glog
Kuly wny [2]
e (0.001, 0.005, 0.01, 0,05, 0.1)
c (0.001, 0.005, 0.01, 0.05, 0.1)
DPO+SFT
Iely"lm) log lug may
8 Blog (x) Alog (o(x"(>)
e (0.0005, 0.001, 0.01,0.1,1
cDPO
-11 - c) log (8 log Manly=ia)
malu"is))
,9 log ro(u
Fo.ty-1n)
/ E {0.001, 0.005, 0.0,0.05,000 0.13
-elog a (Slog Moly
wnyly [e] Bog
soly may (5r
c (0.05, 0.1, 0.15, 0.33
{0.005, 0.01, 0.05, 0.5)
R-DPO
log lug aag(u-ls) Blog
Rny (xy(xr (ally1 (oll"(-abb"1) alr
e (0.001, 0.005, 0.01, 0,05, 0.1)
IPO
[log [al Teptu log znly 1x)
#//
E (0.001,0.005, 0.1, 1.03
Ray (2))
XPO
-log (a (clipzn [so
-90
6 (0.001,0.0) 0.01, 0.01,
((z) = + loy(z)
Re (0.1,0.5, 1,5, 10)
SPPO (offline)
malu"le) - ))² + log malu (8 log (o,(u-1
vaglu-1a) + +))
B e 10.1,1, 10, 100, 1000, 10000)
1.0
CPO
- logg (3 log Tol""" (z)) log vol (w)) A Alog (or)))
B E (0.001, 0.01, 1,
max (,, |u-1 logro(""(z)) [ log moly [c)
RRIIF
A (0.01, 0.1, 0.5, 1, 10)
A AAlug (o()
A e (0.01, 0.05, 0.1, 0.5, 103
SLiC-HF
max (0, log mo(y)(ar)) log moly (yooollll [x]) Alog mo(v
B c (0.1,0.5, 1.0,
[og - A log (log
I po(o- log le) po(y|æ) := exp ( log No Ix))
poly [4)
ORPO
polotaat
Ae 0..1.0.5, 2.0)
SimPO
log 1 log mo(y"+(>> (x)) I log noly (ol 12)
Ae 2, 10,20) 20)
E [0.3, 0.8, L.0}
ROPO
-a logo o (9 log typ s) Bog molo lo)
molyTie)
$ E (0.001, 0.005, 0.01, 0.05, 0.1)
tgo ( log may [x)
my (m)
B log
xnly' any x)
7 = 0.1, 20.2, 20,20,200 200, 2000)
H.3 TRAINING AND HYPERPARAMETER DETAILS
Hyperparameters for training reference models in the base setting. We train initial reference models in the base setups. In the Helpsteer2 setting (Wang et al., 2024e), we train a model through supervised instruction finetuning on the (English only) OpenAssistant2 dataset (Köpf et al., 2024). In the Zephyr setting (Tunstall et al., 2023), we conduct supervised finetuning on the UltraChat- 200K dataset (Ding et al., 2023). We use the following hyperparameters for both cases: train batch size of 256, learning rate of 2e-5 with a cosine learning rate schedule with 10% warmup, right padding, and max sequence length of 2048. We train the models with Adam optimizer for epoch.
General hyperparameters for preference optimization. We use fixed batch size of 128 and a maximum sequence length of 2048 for all methods. For learning rate, we search over (3e-7,5e-7) separately for each method and use a cosine learning rate schedule with a 10% warmup. We use Adam optimizer for all the approaches. In the Helpsteer2 setting and following Wang et al. (2024e), train the models for up to 7 epochs, and in the Zephyr setting, train the models for up to epochs, and select the best number of epochs for each method according to validation. We use right padding for preference optimization following the recommendation of Hu al. (2024).
Specific hyperparameters for preference optimization. For the hyperparameters specific to each preference optimization objective, conduct hyperparameter search according to the values in Table 3. In each setting, we select the best model according to the ranking performance on the validation set. For POWER-DI, we conduct hyperparameter search over 3 E (1,2, 20), n
38

---

Published as a conference paper at ICLR 2025
{0.0005, 0.001), and y 0.31. For Helpsteer2, select / 10, 0.001, = 0.1, 5
epochs, and learning rate of 5e-7 in the base setting and | = 20, = 0.0005, = 0.11 epochs and learning rate of 3e-7 in the instruct setting. For Zephyr, we select 3 = 1, = 0.001, 0.1, epochs, and learning rate of 3 in the base setting, and / = 2, 0.0005, 7 = 0.3, 2 epochs, and learning rate of 3 in the instruct setting.
Computation environment. All experiments are conducted on 8 A100 GPUs based on the Open- RLHF repository (Hu al., 2024).
39

---

Published as a conference paper at ICLR 2025
I ADDITIONAL EXPERIMENTAL RESULTS ON THE LLAMA FAMILY
I.1 PERFORMANCE ON ACADEMIC BENCHMARKS
Tables 4 and 5 present the benchmark scores for the Helpsteer2 and Zephyr pipelines across a variety of downstream tasks. Considering the average benchmark score, POWER-DI consistently improves over the initial model and ranks within the top two or three methods across all four settings, despite significantly outperforming other methods in alignment benchmarks AlpacaEval 2.0 and Arena- Hard as detailed in Table 2. Achieving high score on instruction- followwng benchmarks AlpacaEval 2.0 and Arena-Hard while maintaining good performance downstream tasks is considered key empirical evidence for mitigating reward hacking in practice (Xu et al., 2024b).
Table Downstream task evaluation results of the models trained on the Helpsteer2 pipeline. The arrows show improvement degradation of performance with respect to the initial model. The top three average scores are shown in bold.
Task
MMLU ARC
HellaSwag TruthfulQA Winogrande GSM8K IFEval MMLU-PRO Average
Helpsteer2 Llama3-8B-Bars
Initial Model
62.6
58.4
80.0
49.5
77.4
37.1
33.6
29.8
53.5
DPO
61.2 114 57.5 Lo9
81.1 11.
51.0 1.5
75.6 .18
32.5 14.6
41.1
135
29.8
53.7 10.2
DPO+SFT
62,1 jas
60.5
12.1
81.9 119
52.9 1344
77.0 Jn4
42.1
15,0
42.5
189
30.5 10.7
56,2 12.7
eDPO
62.0 jas 60.9 +2.5
82.9 12.9
53.9 14.
76.6 Jos
37.9 10.8
42.6
19.
30.5 10.7
55.9 12.4
R-DPO
62,8 z02 58.2 Lo2
80.3 10.3
52.9 13.4
77.4
43.2 16.1
38.7 n5.1
30.1 10.3
55.5 12.0
IPO
62.7 70.1 60.6
722
81.7 11.
52.5 13.0
78.1 70.77
43.8 16.7
42.3
76.7
30.3 10.5
56,5 13.0
XPO
62,9 10.3
59.0 70.6
81.0 11.0
51.7 12.
78.1 10.7
41.7
14.6
38.0 14.4
30.1 10.3
55.3 11.8
SPPO
62.8 102
60.9
12.
83.0 13.0
54,2 14.7
77.1 J03
38.8 11.7
42.7
19.1
30.6 10.8
56.3 12.8
CPO
62.6
59.0 10.
80.2 10.2
54,2 14.7
77.0 J04
44.2 17.1
41.4
17.8
30.4 10.6
56. 12.6
RRHF
62.6
$7.7 Jo7
79.0 11.0
51.1 11.6
77.2 J02
37.2 10.1
34.2 pou6
29.8
53.6 10.1
SLiCHF
62.6
r06
79.9 1o.1
55.0 15.5
76.8 J06
43.7
16.5
40.1 16.5
30.2 10.4
55.9 12.4
ORPO
61.5 L1.1
57.6 Loa
79.0 11.0
61.4 111.
77.7 70.
121.5
40.4
16.
29.6 Jo.2
52,9 10.7
SimPO
61.3
106
80.6 10.6
59.6 110.1
77.7 10.3
113.7
40.5
169
30.2 10.4
$4.0 10.5
ROPO
61.5 11.11
+2.5
82.1 12.1
52.5 13.0
76.6
37.5 10.4
41.4 17.8
30.1 10.3
55.3 M.8
POWER-DL
62.0
10.
59.6
m1.2
82.0 12.0
61.0
111.
78.1 10.7
36.5 10.6
40.3
16.2
30.5 10.7
56,3 12.8
POWER
61.9 J0.7
60 11.6
82.0 12.0
61.0 -11.5
77.9 10.5
35.3
11.8
40.1
16.5
30.4 10.6
56. 12.6
Helpsteer2 Llama3-8B-Innttre
Initial Model 65.7
62.0
78.8
51.7
76.0
75.3
54.4
36.0
62.5
DPO
65,9 ru2
63.4 11.4
79.5 10.7
52.7 1u.
75.9 .0.1
76.4 11.1
54.1 U0.3
36.3 10.3
63.0 10.5
DPO+SFT
65.9 102
61.0 L1.0
73.6 15.2
54.7 13.0
71.2 .48
74.7
10.6
48.2
16.2
37.0 11.0
60.8 μ1.7
cDPO
66.0 103
65.4
13.4
79.5 10.7
$7.0 15.3
74.6 J1.4
76.9
11.6
49.9
14.5
36,9 10.9
63.3 10.8
R-DPO
65.8 10.1
62.8 108
74.8 14.0
54.6 12.9
72.6 J34
77.7
124
51.0
L3.4
36.6 10.6
62.0 1o.5
IPO
66.0 иιз
64.9 +2.
79.2 10.4
58.4 76.7
73.8 .22
75.8 10.5
49.8
LA.6
37.1 11.1
63.1 19.6
XPO
65.8 yul
63.7 01.7
75.6 13.2
59.1 17.4
72.4 J3.6
75.2 Ju.1
52.4
L2.0
37.2 11.2
62.7 10.2
SPPO
66.0 103
11.0
76.7 12.1
55.7 N4.
72.9
75.7 104
49.6
37.1 TL.
62.1 10:4
65.7
70.
10.8
$2.2 105
74.1
J19
75.7 10.4
49.9
LA.5
36.0
61.9 10.6
RRHF
65.9 102
11.3
51,5 10.
73.8
76.7 114
51.2
36.4
61.9 10.6
SLiCHF
65.77
62.9
78.0 Jn.
53.9 72.2
74,2
76.7
47.6
36.1
62.3 (0.2
ORPO
65.7
62.1 n0.1
74.0 14.8
56.7 15.0
71.5
75.9
51.3
37.2
61.9 10.6
SimPO
65.8
61.9 Jo.1
75.0 13.8
58.3 166
72.3
74.3
54.1
37.2
62.3 10.2
ROPO
65.4 J0.3
62.0
76.9 11.9
54.1 13.4
73.7
73.9
50.2
36.2
61.8 10.7
POWER-DL
66.0 10.
64.3
(2.3
79.5 10.7
53.1 11.
76.0
76.3
11.0
53.5 Lous
36.6 10.6
63.2 19.7
POWER
65.8 10.1
11.9
79.6 10.8
53.1 1L4
76.1 10.1
76.6 11.3
52.5
L199
36.4 10.4
63.0 ro.5
40

---

Published as a conference paper at ICLR 2025
Table 5: Downstream task evaluation results of the model trained the Zephyr pipeline. The arrows show improvement or degradation of performance with respect to the initial model. The top three average scores are shown in bold.
MMLU
ARC
HellaSwag TruthfulQA Winogrande GSM8K IFEval
MMLU-PRO Average
Zephyr Llama3-8B-Ban
Initial Model
61.6
58.2
78.6
52.1
75.9
47.3
38.1
29.4
55.2
DPO
61.9 70. 62.4 14.2
81.6 -3.0
111.0
74.4 114
52.8 15.5
50.6
1125
31.2 11.
59.7 14.6
DPO+SFT
62.0 70. 62.0 13.8
80,9 -23
61.6 19.5
75.1 10.7
55.0 17.7
48.7
1106
31.0 11.6
59.5 14.4
eDPO
61.9 70.3 61.9 13.7
81.4 -27
1102
10.5
54.7 174
49.3
1112
31.0 11.6
59.7 14.6
R-DPO
61.9 102
11.3
80.1 -1.
62.0 19.9
75.0 10.9
52.8 15.5
45.3 172
30.5 T1.1
58.4 13.2
IPO
61.8 f0.1 58.3
10.1
78.7 10.1
53.5 11,4
10.9
46.8 1o.5
11.3
29.7 10.3
55.6
XPO
62.1
60.8
12.6
80.2
57.9 15.8
75.3 10.5
54.8 17.5
48.8
1107
30.2
58.7 13.6
SPPO
61.5
61.9
81.2 #26
62.0 1100
73.4
51.9
50.7 1126
30.7
59.2
CPO
61.6
56.1 12.1
77.8 Joa
51.5 1ub6
75.9
40.2 U7.1
37.3
ju.s
29.3
53,4
RRHF
61.7
55.8
77.7 .09
51.3
75.6
39.1 J82
36,6
29.3
53,4 11.8
SLiCHF
61.8
57.9
79.4 708
60.2 18.2
75.6
49.4 12.1
42.3
14.2
30.3
57.1 12.0
ORPO
61.8
62.5
81.2 #26
63.9 1119
76.7
48.9 116
56.1
1180
30.8
60.2 15.1
SimPO
61.9
62.4
14.2
81.6 +300
63.0 111.0
74.4 114
42.8 LA.5
50.6
1125
31.2
58.5 13.3
ROPO
61,5 1o.
61.7
13.5
81,4 -28
64.8 1128
73.8 42.1
54.4 17.1
49.9
1118
30.9 11.
59.8 14.6
POWER-DL
61.5
61.7
13.5
81.4
64.8 1128
73.8
54.4 17.1
49.9
1118
30,9
59.8 14.6
POWER
61.8
61.8
13.6
80.6 +20
59.8
76.6
46.3 U1.1
53.7
1156
30.4
58.9
Zephyr Llama3-83 -B--aa-3---Innnu
Initial Model 65.7
62.0
78.8
51.7
76.0
75.3
54.4
36,0
62.5
DPO
66.0
63.0 10.9
76.7
55.7
72.9
75.7 10.4
49.6 44.8
37.1
62.1
DPO+SFT
66.0
66.6 14.5
79.2 -0.5
59.9 18.2
75.0
75.7 104
51.8 12.6
37.3
64.2
cDPO
66.0
67.8
80.5 -188
$9.0
75.1
76.9
51.2
13.2
37.3
64.2
R-DPO
65.7
66.0
78.2 .06
58.9 17.2
74.5 11.5
75.8 10.5
50.7
13.7
36,9
63.3 10.8
IPO
65.8
62.1 10.1
78.7 .0.0
51.8
75.9 10.2
75.8 10.5
54.0
1o.5
35.8
62.2 10.3
XPO
66.2
65.4 13.3
80.5 -1.7
54.2 12.6
76.2
76.8 11.
54.6
10.3
37.1
64.0 11.2
SPPO
66.1 10.4
65.8 13.8
78.7 .0.1
58.6 16.9
74.1 11.9
74.0 113
55.2
10.7
37.4 114
63.7 1.1
CPO
65.4 jo.3
61.9
1o.2
77.8 .09
52.3 10.6
75.5 10.5
75.4 10.1
53.8
40.6
35.9
62.2 10.2
RRHF
65.4 jo.3 61.9
1o.2
77.7 JI1
52.3 1066
75.3 J0.7
75.3
54.2
jo.2
35.8 L02
62.2 10.2
SLiCHF
65.6
63.1
11.0
79.1 +0.3
56.0 14.3
75.4 10.6
76.8 11.
48.8
15.6
36.4 no.
62.9 10.4
ORPO
65.9 70.2
64.5
12.5
78.5 .03
57.6 15.9
75.3 10.7
77.2 119
52.3
122
36.7 7o8
63.7 11.1
SimPO
65.8 10.1
62.1
10.1
74,2 .466
57.4 15.7
71.1 14.9
72.4 129
54.1
10.4
37.0 11.
61.8 10.7
ROPO
66.2 10.5
63.6
11.5
76.4 L24
58.1 16.4
72.9 13.1
73.0 12.3
55.6
37.5 11.5
62,9 10.4
POWER-DL
65.8
64.7 12.7
76.9
59.5 17.9
73.6
76.2 ra.
55.6 11.2
37.2
63.7 f1.1
POWER
65.7
63.1 11.1
75.0 .3.7
59.1 17.4
71.7 14.3
75.7 10.5
54.6 10.1
37.1 hL1
62.9 10.4
41

---

Published as a conference paper at ICLR 2025
1.2 PERFORMANCE ON MT-BENE BENCH
We evaluate the trained models on MT-Bench, which includes 80 questions across categories. We report the average score in Table evaluated by GPT-4 as the judge model. The highest score in trained models is shown in bold and the second highest score is underlined. POWER-DI. consistently outperforms other preference optimization methods. Similar to Meng et al. (2024), we observe that variations across different methods in MT-Bench scores are small compared to A AlpacaEval 2.0 and Arena-Hard.
Table 6: Llama3 MT-Bench results on Helpsteer2 and Zephyr settings.
Helpsteer2
Zephyr
Llama3-8B-Ber Base Llama3-81 Siamma------ssueee Llama3-8B 8B-Base
J-ma3-88- 3-88-Innstruet
Method
GPT- Score
GPT- Score
GPT-4 Score
GPT-4 Score
Initial Model
4.9
8.3
8.3
5.6
DPO+SFT
5.5
8.1
7.0
8.0
<DPO
5.8
R-DPO
5,4
8.2
6.9
8.1
IPO
4.9
8.0
8.0
XPO
5.5
8.0
SPPO
5.8
8,2
7.1
8.2
5.9
7.2
RRHF
5.7
7.9
6.2
8.1
SLiC-HF
5.9
7.8
7.9
ORPO
3.7
8.1
6.6
8,3
SimPO
6.0
8.0
8.2
ROPO
5.8
8.2
POWER-DL
6.0
8.2
7.1
8.2
POWER
5.9
8.2
7.0
1.3 EXPERIMENTS ON ITERATIVE PREFERENCE OPTIMIZATION
The multi-iteration experiments are conducted on the Zephyr pipeline that extends the single- iteration instruct setting into three iteration by splitting the dataset of prompts. We compare approach with iterative DPO (Dong et al., 2024), XPO (Xie et al., 2024), which adds negative SFT DPO, and an iterative variant SimPO. Table presents the results, demonstrating that the ben- efits of our approach that extend to the multi-iteratin setting.
Table 7: Instruction-followingg benchmark results in the multi-iteration setting.
Method
AlpacaEval 2.0 (LC%) AlpacaEval 2.0 (WR%) Arena-Hard (WR%)
Initial Model
33.41
32.40
23.0
DPO iter=
40.02
38.58
31.6
DPO iter=2
42.15
40.71
37.80
DPO iter=3
42.38
40.8
38.8
XPO iter=1
40.96
39.50
32.6
XPO iter=2
41.33
39.66
36,3
XPO iter=3
42.42
39.98
35.7
SimPO iter=]
28.7
SimPO iter=2
41.15
37.77
28.9
SimPO iter=3
41.97
38.32
32.6
POWER-DL iter=l
41.42
37.64
34.4
POWER-DL iter=2
43.48
40.17
POWER-DL iter=3
46.55
43.21
42.1
42

---

Published as a conference paper at ICLR 2025
1.4 HYPERPARAMETER ROBUSTNESS RESULTS
We assess the hyperparameter robustness of POWER-DL. by examining its performance across range of values for 7, V, and in the Helpsteer2 base setting. Figure 2 shows AlpacaEval winrate and length-
winrate for various hyperparameters. POWER-DL. exhibits a robust behavior with respect to all three hyperparameters, particularly considering the length- cength-controlled winrate.
Alpaca-Eval
Alpaca
Alpcz: Eval
32
90
26
06
26
26
24
24
24
LC
WR()))
T.0%)) WR(%)
WR(3)
22
22
00
02
0.3
7
7!
8
Figure 2: POWER-DI. hyperparameter robustness results the Helpsteer2 base setting.
43

---

Published as a conference paper at ICLR 2025
1.5 EXAMPLES RESPONSES
The following is an example question along with responses from various models from the Helpsteer2 base pipeline. POWER-DL response shows analytical thinking, deeper reasoning process, and consideration of implicit factors, characteristics reminiscent of System 2 thinking.
Question
One One was race horse Two Two was one too. One One won one race. Two Two won one too. How many horses are this story?
Responses from Different Models
Initial Model: There are two horses in this story, One One and Two Two.
GPT-4: There are two horses in this story. Their names are "One One" and "Two Two."
DPO: Based on the information given in the riddle, there are two horses involved in the story. "The One One" and "The Two Two" are both mentioned race horses, and each of them has won one race respectively. So, there is one "One One" horse and one "Two Two" horse this story. Together, they make up the total two horses.
cDPO: Based on the information given in the riddle, there are two horses involved the story. "The One One" and "The Two Two" are both designated race horses, indicating that there are two horses with those specific monikers. "One One won one race" refers to the horse with the name "One One" winning
too" refers the horse with the name "Two Two" winning one race as well, implying that both horses won one race each, ruling out the possibility of both being race horses. So, there is "One One" horse and "Two Two" horse in this story.
race. "And Two Two
xPO: There are two horses in this story.
SPPO: Based on the information provided in the riddle, there are two horses involved in the story. "One One" and "Two Two" are both race horses, and the sentence "One One won one race. Two Two won one too." indicates that both horses won one race each. So, there are two distinet horses that won a race each, namely One One and Two Two. Therefore, the number of horses in this story is 2.
Based solely on the information provided in the riddle, there could be as many as three horses and few one. The first two lines introduce two horses, "One-One" and "Two- Two." The next two lines indicate that each can win a race, but the information about the races themselves and the horses that competed in them is missing. If "One-One" and "Two-Two" raced each other, then just those two horses would exist this story. However, if either or both raced third horse, the number of horses in the story could be three. Therefore, the actual number of horses in this riddle's story is unknown.
SimPO:
POWER-DL (ours): The story mentions two horses, One One and Two Two. However, it's possible that there could be more horses involved. The statement "One One one race" does not specify whether the track was crowded with competitors, and it's possible that the horse had race against multiple opponents. Therefore, the number of horses in the story could be anywhere from two to many. The statement only specifies the race results of the two named horses.
44

---

Published as a conference paper at ICLR 2025
J EXPERIMENTAL RESULTS ON MISTRAL FAMILY
In this section, we provide experimental results on conducting preference optimization on Mistral- 7B (Jiang al., 2023). We follow the same experimental details described earlier and finetune mistralaaiMMittaal--B-00000 in the base setting and mistralaiMMistral----nsnsnee in the instruct setting.
Table AlpacaEvval 2 and Arena- results on Helpsteer2 and Zephyr settings for Mistral family. Zephyr
Helpsteer2
Mistral-7B-Bas
Mistral-7B-Instree
Mistral-7B-Base
Mistral-7B-Instruct
AlpacaFival
Trena-Hard
AlpacaFval Arena-
AlpacaEval
Arena-
AlpacaEval Arena-Hard
Method
LC(%)
WR(S)
WR(%)
LC(%)
WR())
WR(%)
LC(%)
WR(S)
LC(%) WR(())
WR(S)
Initial Model
5.47
4.16
1.5
27.70
22.26
14.8
2.85
2.11
0.6
27.70
22.26
14.8
DPO
11.89
10.39
4.6
36.47
17.1
15.67
13.26
5.9
36.55
36.28
24.3
DPO+SFT
10.57
8.21
3.4
35.28
27.88
16.9
12.30
10.68
5.4
35.31
35.70
25.1
cDPO
12.12
10,52
3,5
32.07
29.22
16.6
13.57
12.24
5.4
31,42
30.43
20.9
PO
10.88
8.64
4.1
38.90
37.02
22.5
9.80
8.35
3.4
34.79
35.53
17.0
SimPO
14.56
7.9
38.28
14.1
POWER-DL
19.83
15.40
42.26
34.72
23.7
20,34
19,50
12.1
42.57
42.53
POWER
19.72
16.04
6.5
39.23
33.06
20.1
17.09
15.25
10.2
38.13
36.85
26.2
Table 9: Mistral MT-Bench results on Helpsteer2 and Zephyr settings.
Helpsteer2
Zephyr
Hisstrl--B-Baase -istrral-7--Intttrrree II Mistral -eerrll--B-Baee Mistral 7B-Instruct
Method
GPT-4
GPT- Score
GPT-4 Score
GPT-4 Score
Initial Model
3.1
6.6
3.6
6.6
DPO
3.4
6.3
5.0
6.6
DPO+SFT
3.4
6,3
5,2
6.3
3.5
XPO
3.2
6,6
5,0
6.2
SimPO
2.9
6.3
POWER-DL
POWER
3,2
6.6
5.2
6.5
45


Reviews:
[{"review_id": "Hsvnt4jlT5", "reviewer": "Reviewer_sDff", "summary": "This paper studies the preference optimization problem for LLMs. This work points out the two types of reward hacking that could lead to inaccurate learned reward models. To solve the problem, the proposed algorithm makes use of the weighted entropy for robust estimation. Also, a dynamic labeling procedure has been illustrated in order to diminish gradients for untrustworthy samples. The theoretical and empirical studies are performed to validate the conjectures and benchmark the performance of the algorithm.", "questions": "Please see the questions in Weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The presentation of the paper is clear, which makes it easy to read. \n-  The paper investigates one of the most important problems in preference optimization, i.e., reward hacking.\n-  Some theoretical studies are conducted for analyzing the popular preference optimization, under the proposed conjectures. \n-  The algorithms are evaluated in some benchmarks and downstream tasks.", "weaknesses": "- There are many important and highly related literature that have not been reviewed in the paper. For example, \na. Chen, Zixiang, et al. \"Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models.\" ICML 2024\nb. Rame, Alexandre, et al. \"WARM: On the Benefits of Weight Averaged Reward Models.\" ICML 2024\nc. Pan, Alexander, Kush Bhatia, and Jacob Steinhardt. \"The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models.\"  ICLR 2022\nThe insufficient review of the related literature on reward-hacking and model alignment limits the credibility of the two types of reward-hacking in the paper. \n\n-  As claimed in the paper, the type 2 reward-hacking is due to the decent choices appearing less desirable. It is reasonable to think about that if the over-pessimism in solving the type 1 reward-hacking is the root cause of not well-leveraging the decent samples. However, in this work, there is not a sufficient discussion on this point. See, Chen, Lichang, et al. \"ODIN: Disentangled Reward Mitigates Hacking in RLHF.\" ICML 2024\n\n- The examples used in propositions 1 and 2 are very simple, which deviated from the real applications in the fine-tuning of the LLMs, e.g., the definition of the reward function and policy classes in the examples. Therefore, from the results of propositions 1 and 2, it is only sufficient to obtain that the discussed popular preference optimization algorithms do not work well in such restrictive settings. However, the conclusion cannot be extended to the general settings in practice. \n\n-  The principle of maximum entropy (principle of the weighted entropy in the paper) has been widely studied in the RL literature and this is not a new idea. In particular, the work by Zhu, Banghua, Michael Jordan, and Jiantao Jiao. \"Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.\"  ICML 2023 has theoretically studied the maximum entropy in RLHF/IRL settings. \n\n-  As shown in Huang, Audrey, et al. \"Correcting the mythos of kl-regularization: Direct alignment without overparameterization via chi-squared preference optimization.\" Arxiv, the KL divergence is not enough to induce pessimism in order to mitigate the reward-hacking. Following this argument, the weighted entropy may also encounter similar problems. \n\n-  The choice of the weights in the weighted entropy needs more details to elaborate; otherwise, the length-normalized technique has been utilized in the existing RLHF algorithms. \n\n-  Intuitively, the weighted entropy tends to yield a more uniform distributed policy. Therefore, this is a natural question about how to balance the exploration and exploitation tradeoff. \n\n-  A more pronounced discussion on ``favoring unbiased policies'' is needed. \n\n-  The \\beta controls the strength of the explicit pessimism and \\eta controls the strength of implicit pessimism. The two factors make the control over the pessimism complicated. In practice, how to balance the two types of pessimism, and in theory, how to mitigate over-pessimism? \n\n-  In resolving the type 2 reward-hacking problem, is there any theoretical guarantee/regret analysis on the algorithm with the dynamic labeling procedure? \n\n- There is no algorithm table for describing the details of the two proposed algorithms. \n\n-  In experiments, it would be better to present some results of the iterative extension of the current method. \n\n-  As the current method is much more complicated than the existing methods, e.g.,  DPO or SimPO, the latency of the algorithms should be presented. An efficient algorithm should balance performance and computational efficiency. \n\n-  The evaluation over the MT-bench is very important. Also, it would be better to compare the models in the instruction-tuned model Mistral as in SimPO. This could better position the work in the existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rfUvD1MoU8", "reviewer": "Reviewer_3mYa", "summary": "This paper addresses reward hacking in offline preference optimization (PO) methods by identifying two types of statistical errors that impact reward reliability: \nType 1: Poorly represented, suboptimal choices in the dataset appear more favorable than they are due to statistical inaccuracies. \nType 2: Due to poor statistical coverage, good choices are misrepresented as less favorable than their actual value. \nThe authors demonstrate that many current offline PO methods are vulnerable to both forms of reward hacking, leading to suboptimal policy learning. They propose a novel offline PO method named POWER-DL to counteract these effects. This method integrates robust reward maximization with weighted entropy and dynamic preference labeling. Experimental results show that POWER-DL outperforms state-of-the-art methods on benchmark alignment tasks, such as Arena-Hard and AlpacaEval 2.0.", "questions": "In Proposition 4 (I), POWER-DL is shown to recover the best-in-class policy for the constructive example given in Proposition 1. Does this imply that POWER-DL fully mitigates type 1 reward hacking? Or could there still be instances where type 1 reward hacking persists?\n\nIn Table 2, POWER-DL seems to perform better in the base setup (where it uses an existing preference dataset) than in the instruct setup (where a preference dataset is constructed by sampling from the initial model). Does this suggest that POWER-DL is more effective under a distribution shift between the initial model and the model(s) used to collect preference data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well-written. The related work is covered comprehensively, and theoretical statements are clearly articulated (although I haven’t verified the proofs in the appendix).\n\nThe theoretical analysis, particularly Theorem 1, helps guide and justify the design of POWER-DL. \n\nThe proposed method, POWER-DL, is both theoretically grounded and easy to implement. Its approach—combining robust reward maximization with dynamic label updating—has practical implications.\n\nThe authors present extensive empirical results, with broad baseline coverage, that effectively demonstrate POWER-DL's advantages in addressing reward hacking and achieving better alignment.", "weaknesses": "Wu et al. (2024) introduce a dynamic beta-based approach that could be a relevant baseline here, especially given its emphasis on preference data quality. \n\nIt would be helpful to see experiments on the robustness of POWER-DL to hyperparameters, particularly \\beta and \\eta. \n\n(Wu et al., 2024) \\beta-DPO: Direct Preference Optimization with Dynamic \\beta", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wYjDAfVbpu", "reviewer": "Reviewer_w7RZ", "summary": "The paper introduces a new approach to address two forms of reward hacking due to low-coverage pairs. First, they propose a robust reward maximization objective, which enables a regret guarantee that all(?) previous methods lack. Second, they utilize a gradient freeze technique to prevent overreliance on low-coverage pairs.", "questions": "1. Could you pls comment on weakness 1?\n2. Could you pls comment on weakness 2?\n3. What is the main reason you introduced the $\\textbf{weighted}$ entropy term? Did you have other weight in mind other than the inverse response length?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I believe it is a strong paper with both convincing theoretical analysis and great empirical results.", "weaknesses": "Overall I think it's a very nice paper. But I do have some conceptual problems awaiting to be clarified by the authors: \n1. I am not sure the two components of your method are defending against each hacking type separately ( based on your paper construction, you are designing the approach this way).\nIt seems that dynamic labelling simply gives low trust to low-coverage regions. Then it should help both stop the less-desired outcomes from being pushed to a highly-desired position; AND stop highly-desired outcomes from being pushed to a lowly-desired position where the pair has low coverage.  Is that correct? Then should it be a technique for both hacking types?\n2. It is nice that you have theoretical guarantee for POWER itself. And empirically, POWER-DL seems to work even better. But I feel the two moves could also interfere with each other due to the fact that highly-covered pairs will still influence the poorly-covered pairs and ultimately many outcomes are competing against each other --- hence even though your set the gradient to zero for poorly covered pairs, the final distribution of the actions can still be changed by other pairs. What are your thoughts on this?\n3. Ps. typo on line 092: Should be POWER-DL right?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HZLG3AYoy0", "reviewer": "Reviewer_fMHu", "summary": "This paper proposed two types of reward hacking in offline preference optimization . The Type I is when subpar choices appear more favorable due to sparse data, and the Type II is when good choices appear worse. They propose POWER(-DL), a new offline preference optimization method that (1) integrates weighted entropy with robust reward maximization to address Type I hacking and (2) use dynamically updating preference labels to resolve Type II hacking. Empirical results show that it outperforms state-of-the-art methods on benchmarks.", "questions": "Compared to Zhan et al., 2023a, what is the advantage of your method? I think the settings are pretty similar and the concentrability coefficient is defined in the same way (please correct me if I missed any subtitles). Since you claim to address both reward hacking issues by proving a finite-sample bound, as in Theorem 1, do Zhan et al. (2023a) address them as well?\n\nIf I simply set $w(\\cdot)=0$, what would go wrong in Theorem 1? I believe that you have hidden some $w$-dependence term in the bound, and I am curious to know what that is.\n\nCan you provide some intuitive explanations on why choosing $w(y)=1/|y|$ leads to $\\log|\\mathcal V|$ in Theorem 1?\n\nMinor problem: I don't see how \"Sail into the Headwind\" is informative in the title", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The use of weighted entropy and dynamic label is interesting and new as far as I am aware.\n\nThe reasoning is sound and convincing. For instance, for the first type of reward hacking, they established a lower bound (proposition 1) showing that regardless of the number of samples, certain existing algorithms will suffer from a constant suboptimality with probability of at least some constant. Then they show that their proposed algorithm avoids this issue (Theorem 1).\n\nThe experimental results are impressive. The proposed method consistently outperforms baselines across multiple models.", "weaknesses": "Both types of reward hacking issues arise from insufficient data coverage, a well-explored topic in bandits and reinforcement learning. Therefore, to me, the two new definitions of reward hacking don’t feel entirely novel to me. Although I admit that most prior work hasn’t focused on learning-from-preference settings. This consideration actually leads to some questions that I show in the Question section.\n\n\nIt might be beneficial to merge Theorem 2 with Theorem 1 to provide a unified theoretical result for the proposed algorithm. Currently, they are presented separately, so I wonder how the use of dynamic labels will affect the suboptimality bound shown in Theorem 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)