Robust Preference Optimization through Reward Model Distillation
Authors: Adam Fisch
Jacob Eisenstein
Vicky Zayats
Alekh Agarwal
Ahmad Beirami
Chirag Nagpal
Peter Shaw
Jonathan Berant
Venue: Accepted by TMLR ()

Abstract: Language model (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Direct Preference Optimization (DPO) is a popular offline alignment method that trains a policy directly on preference data without the need to train a reward model or apply reinforcement learning. However, the empirical evidence suggests that DPO typically assigns implicit rewards that overfit, and trend towards infinite magnitude. This frequently leads to degenerate policies, sometimes causing even the probabilities of the preferred generations to go to zero. In this work, we analyze this phenomenon and use distillation to get a better proxy for the true preference distribution over generation pairs: we train the LM such that its induced implicit reward, i.e., the scaled log-likelihood ratio of the model to the reference model, matches an explicit reward model trained on the preference data. Moreover, to account for uncertainty in the reward model we are distilling from, we optimize against a family of reward models that, as a whole, is likely to include at least one reasonable proxy for the preference distribution. Our results show that distilling from such a family of reward models leads to improved robustness to distribution shift in preference annotations, while preserving the simple supervised nature of DPO.

Published in Transactions on Machine Learning Research (2/2025)
Robust Preference Optimization through Reward Model Distillation
Adam Fisch Google Mind
fisch fisch@google.co com
Jacob Eisenstein" Google Deep
jeisensteinndggoogee.com
Vicky Zayats* Google Mind
vsazaas wngoless@google.com
Alekh Agarwal Google Research
alebbagarwal@@ggngge..com
Ahmad Beirami
beirami@@google.com
Google Mind
Chirag Nagpal
chingnaggal@ggooge..com
Google Research
Peter Shaw Google Mind
petershaufAgoogle.comm
Jonathan Berant*
joberent@google.com
Google DeepMind
Reviewed on OpenReview: https: opereevew.. net/ forum? id= E2zKNouNDc
Abstract
Language model (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Direct Preference Optimization (DPO) popular offline alignment method that trains policy directly on preference data without the need to train reward model or apply reinforcement learning. However, the empirical evidence suggests that DPO typically assigns implicit rewards that overfit, and trend towards infinite magnitude. This frequently leads to degenerate policies, sometimes causing even the probabilities of the preferred generations to to zero. In this work, we analyze this phenomenon and use distillation to get. a better proxy for the true preference distribution over generation pairs: we train the LM such that its induced implicit reward, i.e. the scaled log-likelihood ratio of the model the reference model, matches an explicit reward model trained on the preference data. Moreover, account for uncertainty in the reward model distilling from, optimize against family of reward models that, whole, is likely to include at least one reasonable proxy for the preference distribution. Our results show that distilling from such family of reward models leads to improved robustness to distribution shift. in preference annotations, while preserving the simple supervised nature of DPO.
*Core contributor.

---

Published in Transactions on Machine Learning Research (2/2025)
1 Introduction
Language model (TM) post-training (or alignment) aims to steer language model policies towards responses that agree with human preferences. Early state-of-the-art approaches have focused on reward learning from human foodback. In this paradigm, preference annotations are used to train reward models, which then guide the optimization of the language model policy through online reinforcement learning (an approach broadly referred to as RLHF). Recent research on offline "Direct Preference Optimization" (DPO; Rafailov et al., 2023) and extensions thereof (Azar et al., 2024; Tang et al., 2024b; Meng et al., 2024), however, has demonstrated that it is possible to directly optimize policies on the preference data, which (a) bypasses the need for a separate reward model, and (b) uses standard supervised techniques rather than online reinforcement learning, which can be more difficult to optimize. These advantages have led to the the adoption of offline aligmment, and in particular offiine DPO, as the post-training algorithm of choice in both smaller-scale academic settings as well as larger-scale projects such as Llama 3 (AI@Meta, 2024) and OLMo (Groeneveld et al., 2024).
While this direct approach to preference optimization is attractive in its simplicity and efficiency, it also raises questions about the effect and robustness of the resulting policies explicit reward model beyond online reinforcement learning. In this paper, argue that explicit reward modcling can, in fact, offer substantial practical and theoretical benefits. In particular, we theoretically show that relying solely on the preference data can be a precarious strategy, with few natural brakes in place to prevent policies trained under the DPO objective from careening off towards degenerate policies when the preference data exhibits certain idiosyncratic properties. On the other hand, explicit reward models can easily be regularized and understood regarsts of whether they are Tradley-Terry models (Bradley and Terry, 1952), margin-based ranking models (Zhao et. al., 2023); or any other function that correlates well with human preferences (Lee et al., 2023; Tang et al., 2024b; Swamy et al., 2024).
well as the broader utility of an
Taking a step back from pure direct preference optimization, we first explore a method that merges the best of both worlds for the offline setting: an efficient reward model distillation algorithm that (i) operates effectively in the offline setting, (ii) makes minimal assumptions about the truc, optimal reward we aim to maximize, and (ii) demonstratee greater robustness to the specific distribution of prompt/ /response data used for policy aligmment. prompt/response Drawing inspiration from prior knowledge distillation techniques (Hinton et al., 2015; Romero et al., 2015; Yang et al., 2019; Furlanello al., 2018), we use the same change of variables trick employed in DPO to express the language model policy in terms of its implicit reward model (Rafailov et al., 2023). We then train the policy's implicit reward model to match our desired, explicit reward via an Ta loss that directly regresses the pairwise differences in target rewards for any two generation pairs (I, and (z, 32)." Here we theoretically establish the equivalence between optimizing this simple distillation loss over a sufficiently diverse offline dataset of unlabeled examples, and optimizing the traditional online RLHF objective with reinforcement learning.
While this approach adds reward modeling and reward inference back into the pipeline, it still maintains much of the simplicity and efficiency of reward-model--ree DPO. Specifically, in our setting, rewards for the training data can be computed offline, once, ahead of time. This computation is completely parallelizable, and reward inference is significantly faster than the autoregressive generation (also known as model rollout) that is done in online settings. Consequently, this allows the policy training framework to be nearly identical to standard DPO, modulo the structure of the data that is fed in. This is true regardless of the size of the policy that is trained (as the policy's 's implicit reward model does not need to be the same size as the explicit reward model) or its
(as the policy's syper-parameters do not need to match those of the reward model).
Reward model distillation, however, is still subject to some of the same challenges facing DPO-style training. In particular, distillation requires having reliable reward model but having reliable reward still requires having a reliable method for extracting a reward model from a potentially noisy preference dataset. To address the uncertainty surrounding what the "right" reward model to optimize against is, we also introduce a pessimistic extension to our approach. This extension aims to maximize the worst-case improvement of our model across a plausible family of reward models (c.g., those sufficiently consistent with annotatod preference data). This strategy aligns with that of existing work in conservative offline reinforcement learning (Cheng
We also note that similar La reward alignment loss was recently explored in independent work by Mao al. (2024) and Gao et al. (2024h), giving further support for effectiveness. See a discussion related work $2.
2

---

Published in Transactions on Machine Learning Research (2/2025)
et al., 2022; Kumar et al., 2020). We show that this pessimistic objective can be equivalently expressed and optimized by adding simple additional KT-ddivegeecc regularization to the original distillation objective.
Empirically, we find that reward model distillation, particularly pessimistic reward model distillation, leads to similar performance to prior direct preference optimization methods when the preference datasets used are unbiased. When the preference datasets are biased, however, it leads to significantly beter performance when compared DPO and the Identity Preference Optimization (IPO) framework of Azar et al. (2024), which was introduced as more robust alternative to DPO. To further support these empirical observations, we provide extensive theoretical analysis that both (i) sheds more light on the degenerative tendencies of DPO and issues inherent to its objective, and (ii) highlights relative advantages of our explicitly regularized approaches.
2 Related work
Recent work in offline alignment has focused on DPO (Rafailov et al., 2023) as a simpler alternative for aligning language models from preference data. Subsequent work, however, has identified issues with DPO, including wcak regularization (Azar et al., 2024) and a tendency to decrease the probability of winning generations during training (Pal et al., 2024). Similar to some of the findings in this work, Rafailov et al. (2024a) theoretically showed the existence of minima to the empirical DPO loss that assign non-zero probability to outputs not present in the training data, which can lead to unexpected model behaviour. Here we further show that not only can such minima place non-zero probability on outputs not present in the training data, but they can also assign near-zero probability to preferved responses that do appear in the training data, A number of methods have since explored various avenues for combating these issues. These include analyzing the impact of noise on DPO alignment (Gao et al., 2024a), proposing to update the reference policy during training (Gorbatovski et. al., 2024), and suggesting a variant of IPO with a per-context. margin (Amini et al., 2024). Additional research has focused on token-level alignment methods (Zeng et al., 2024; Rafailov et al., 2024b; Mudgal et al., 2024; Chakraborty et al., 2024) and developing unified view of various offline alignment methods (Tang et al., 2024b). Similar to our contributions towards using pessimism with respect to the correct choice of reward model for robust reward model distillation, existing work on offline RLHF has also focused on encompassing various forms of conservative reward penalties (Zhu et al., 2023; Liu et al., 2024; Zhan et al., 2024). Most relevant to our work on a technical level, concurrent work by Mao et al. (2024) also uses an offline loss that leverages targets from a learned reward model for supervision (as opposed to binary preference labels) that has the same form of the simple "distillation loss" analyzed in $5.1 of this paper, but does not explore pessimism. Similarly, the REBEL algorithm of Gao et al. (2024b) also studies the same reward regression loss, but in the online setting. This work builds upon these findings, and provides further analysis, as well as a solution based on pessimism together with reward distillation.
As discussed in 11, offline settings are attractive since they allow for simple, efficient, and scalable training frameworks. At the same time, while offline alignment. methods are popular, recent evidence suggests that online alignment methods such as RLHF (Christiano et al., 2017: Stiennon al., 2020), can still lead to more favorable outcomes (Guo et al., 2024; Tajwar et al., 2024; Dong et al., 2024; Xu et al., 2024; Xiong et al., 2024: Calandriello et. al., 2024), especially when high reward outputs have low probability under the base policy. One of the advantages of online settings over offline settings is that many strategies for mitigating over-optimization that are simple to apply in online settings, such as reward shaping, are not as straightforward to apply in offline settings. For example, reward ensembles have been widely investigated recently as a mechanism for tackling reward hacking in online RL.FF (Fisenstein et al., 2023; Coste et al., 2023; Zhai et al., 2023; Ramé et al., 2024), and in the context of multi-objective optimization (Moskovitz et al., 2023; Rame et al., 2024). Other notable examples for combating reward over-fitting and optimization by training reward models with regularized training objectives include iterative data smoothing (Zhu et al., 2024), which uses trained model to softly label data during RLHF, and reward calibration from demonstrations (Rita el al., 2024). This work addresses some of the holooolloi and experimental gap that exists between online and offline methods for RLHF, by allowing for explicitly designed, trained, and regularized reward models (or pessimistic reward model ensembles) to be added back into the offline alignment setting without losing the practical benefits of the offline setting. Also relevant to our work, Moskovitz et al. (2023) focus on "composite" rewards in the online setting, with the goal of achieving high task reward while ensuring that every individual
3

---

Published in Transactions on Machine Learning Research (2/2025)
component is above some threshold- also by applying a Lagrangian relaxation. In this work, we also consider multiple reward models, but we only focus on cases where there is no known, obvious reward decomposition.
Finally, the question of using a small amount of offline data to learn high-quality policies, instead of online access to reward feedback, has also been widely studied in the offine reinforcement learning (RL) literature. The predominant approach here to use pessimism, that is, to learn a policy with the highest reward under all plausible environment models consistent with the data, with an extensive theoretical (Liu et al., 2020; Zanette et al., 2021; Xie et al., 2021) and empirical (Kumar et al., 2020; Cheng et al., 2022; Yu et al., 2021) body of supporting work. The key insight in this literature is that without pessimism, the RL algorithm learns undesir- able behaviors which are not explicitly ruled out in the training data, and pessimism provides robust way of preventing such undesirable extrapolations, while still preserving generalization within the support of the data.
3 Preliminaries
We begin with a brief review of Direct Preference Optimizaaiio (DPO) (Rafailov et al., 2023).
3.1 The preference alignment problem
Let x be an input prompt, ~ np(- | x)) be the language model policy ay's response to H, and Tyer x) a reference policy (such as a pretrained or finetuned language model that high-pprforming, but not yet aligned, and often used as the starting point for optimization). Given some reward function r" r"(2, y), the goal of alignment is to solve for the "aligned" policy To- (y a)) that maximizes the following RLHF objective, i.c.
T (y 1) argmax (p(a))
[Bmo(vl>] [rr (zz v)] [DkI [To ( []]err [)]]
(1)
where p(z) is a fixed distribution over prompts, and the KL-divergence term keeps the aligned policy close to the anchoring reference policy, Trer(y x). Here, the reward function r" is typically not known in advance, but rather inferred from collected human preference data in the form of (x, y/", yy), where x is the prompt, y" is the "winning", or preferred, response, and of the "losing", or dispreferred, response. A common approach is assume that (H1, (2) follow a Brarlyy-Terry model (Bradley and Terry, 1952), under which the probability that y1 is preferred to y/2 given the reward function r' and prompt is p" (31 y2 | x)) (r" y1) r r", U2)), where (-) is the sigmoid function and > denotes preference. Under this model, can use the preference data (z, 8" v Drref to estimate r* via maximum likelihood estimation, i.e.
i E argmin (y of pret (-ooga(((z, -(x=="(
(2)
With A in hand, Eq- (1) can be optimized using standard reinforcement learning algorithms (Schulman el al., 2017; Stiennon et al., 2020; Christiano et al., 2017).
3.2 Direct preference optimization
DPO is a simple approach for offline policy optimization that uses preferences directly align the language model policy, without training intermediate reward model. Specifically, DPO leverages the fact that the optimal solution to the KL-constrained objective in (1) takes the form (Korbak et al., 2022)
(re | 2)) exp (!r((a.))
To- (y x)) = Z(x)
(3)
where Z(x)= E, Tr x) eepp(tr((ar) (x, v)) is the partition function. DPO reparameterizes the true reward function r* in terms of the optimal policy σg- that it induces, i.e.,
(y x) To (y) x))
r" rxx,,) = log
+ Bog Z(x).
(4)
Under the Bradlcy-Torr Terry model, the likelihood that y1 > y2 can then be written as
p" (3) > U2 x)) (Alog
T (11 x) TT) x))
(5)
To- (U2 x ITTee
x)
4

---

Published in Transactions on Machine Learning Research (2/2025)
where now ze- can be directly estimated on D Drret following the objective in (2), in place of the intermediate reward model f, Lee, Tg(U x) E argminz, Ldpa (To; (To: Drret) where
Eapo (ooi D Ppret) = (y wf D
[-log a
(Blog
To Tn
x Tee (y x)) )
x (ro x3
(6)
Tol
As described in $1, optimizing Lapo offers two main advantages over using online RL for Eq. (1): (a) there is no need for a separate reward model, and (b) Lápo is a supervised objective that can be trained offline, which allows for a simpler training setup than online learning. Still, Lapo also has certain pitfalls, as we analyze next.
4 Pitfalls of direct preference optimization
As argued by Azar et al. (2024), DPO strongly relies on the Bradley-Terry assumption, which leads surprising and undesirable consequences when trained on finite preference data. The root issue is that if we have any two responses y1 and y/2 where p" (y1 1/2 x) 1, then the Bradley-Terrr model dictates that r" (11) - r" (12) = to0, and therefore TT- (32 | æ) 0 for any finite KL-regularization strength B.
We can illustrate this phenomenon on a broader level with the following example:
Assumption 1. Suppose we are given a preference dataset of (context-free) pairs Dpret= = {(ut" v =.)) the pairs (y)) are mutually disjoint in both the elements. Further suppose that we optimize the DPO objective on D pree with single parameter Oy for each y- Proposition 1. Under Assumplion 1, for any (y.yy) such that Vi and s' = vi for some i, we have + 00, for all global minimizers To. of the DPO objective in (6), for any B > O.
ay- (a) wg. (1"))))rly))
Corollary 1. Under Assumplion 1, further assume that 0 < [vo < for all y. Then ag. is a global minimizer of the DPO objective in (6) iff Tg- (C(g")") + 1 with he- (v(") > 0 vi E [n], where Clyfy is the complement of the set of all responses y that appear as a dispreferred vf for any i E [m].
Additional analysis the training dynamics of DPO is provided in $7. A significant implication of this result is that the set of global optima of the DPO loss includes policies that can shift nearly all probability mass responses that never appear in the training set and even assign near-zero probability to all of the training data responses that do in fact correspond to winning generations, y", phenomenon that has been observed empirically and analyzed theoretically (Pal et al., 2024; Rafailov et al., 2024a;b; Tajwar et al., 2024).²
Stated differently, Corollary 1 implies that any 0" merely satisfying T (uf) = 0 with mg- (v,") > 0 Vi E [m] is a global minimizer of the DPO objective in this setting. Though simplistic, the scenario in Assumption 1 is closer to reality than might first be appreciated: in many practical situations we can expect the finite-sample preference data to contain one (or at most a few) preference annotations per example (I, /1, y2), while the policies xo can have billions of parameters (> n). It is important to note that the supposed regularization term 3 does not help: it can limit the speed at which the optimizer reaches the degenerate solution, but it cannot alter the final destination. This issue could be viewed as a classic instance of overfitting but as opposed to overpredicting responses within the training set, we might overfit to almost never producing anything like the "good" responses that do appear within the training set. Furthermore, without additional regularization (beyond 3), we can expect this degeneration to occur in typical preference datasels.
Takeaways: Pitfalls of direct preference optimization
The DPO objective only requires that the likelihood of the preferred response is relatively higher than that of the dispreferred response. A peculiarity of this objective is that when preference data is disjoint (i.e., preferred responses never appear as dispreferred responses, and vice versa), certain types of policies (e.g,, over-parameterizer will learn to assign 0 probability to all dispreferred responses, with merely non-zero probability to all preferred responses. This includes policies that assign near-sero probability to preferred responses, and place all mass on (often degenerate) generations outside the training set.
PIn fact, the conclusions can also be inferred from Proposition in the concurrent work of Rafailov et al. (2024a). While their assumptions are slightly different, similarly to us, they also target scenarios which the training data exhibits certain (realistic) deficiencies.
5

---

Published in Transactions on Machine Learning Research (2/2025)
4.1 Case study: degenerate DPO optima in BoW model
For inluition on why the DPO global optima can include policies where T(y") may be nearly 0 for all yd in the training set, consider the simplified case where the policy is a bag-of-words words model, To (t) x exp (c(y) O) for c(y) representing a vector of counts in y and 8, representing the unnormalized log-probability of token i. Then we can formally show that DPO optimization monotonically decreases an upper bound on the probability of the preferred completion, Tga 1) (v²) 2 Agco (v²)) 2 (V
Proposition 2. Let y" . y be preferred versus dispreferred outpuls of length n, respeclively, with (y" (yer(y > 0 and corresponding count vectors e(y"). 2(0) = log E: ext with upper bound log #t(y) = c(y) - n max, Bg. Let g(t) represent the parameters of after steps of gradient descent on La v ,z), with g(0) 0. Then, we have that Tyeol (y²)) S agtes (v")) Ag(en1(")) for all t, with strict inequelity when ||c((") c(y")llo
) cly"). Let logno(y) cly) 0 nZ for
If Tote (v") decrcases in t, what other strings become more probable? In the following proposition, we show that under the bag-of-words model, DPO optimization moves probability mass away from y²d to sequences that contain only the tokens that maximize the difference between y" and y".
Proposition 3. Let y" and y be preferred versus dispreferred oulpuls of length n. Let A = cly ) -cly²) be the difference in unigram counts. Let i [i, for E argmax A, with ||c( = n. Then Toto (yr") Tows (0) T(t)k for some k and some non-decreasing Z+ - R.-
We have k- = O when cly")) c(0). and & < O when ||c(y") < lc(0)ll2 n (when c(v")) is dense) and || x |ا (when A is sparse). This implies that when y" and of are similar, To("") will degrade more rapidly. Early stopping will therefore have to trade off between reaching the degenerate solution on such cases, and underfitting other cases which y" and y are more distinct. Related findings are also reported in Razin et al. (2024).
Takeaways: BoW model case study
For a simple BoW model, we show a realized setting in which DPO causes the probabilities of preferred outputs to catastrophically plummet, and instead puts all probability mass on generations that simply maximize token-level differences between preferred and dispreferred examples.
5 Uncertainty-aware reward model distillation
As discussed in the previous section, a core issue in preference optimization is that the true preference distribution p² > ya æ) is not known. Attempting to infer it from finite-sample preference data (which may further be biased or out-of-distribution with respect to the target domain) can then result in failure to learn reasonable policies. In this section, we propose a regularized, pessimistic approach to direct preference optimization that brings explicil reward modeling back into the picture through model distillation objective, while still maintaining the simplicity and efficiency of offline alignment methods.
5.1 Reward model distillation
Suppose for the moment that the reward function Y" was in fact known, and did not have to be inferred from sampled preference data. Under this sctting, we can then construct a straightforward and efficient. offline optimization procedure that is similar in spirit to DPO, but no longer relies directly on a preference dataset. Concretely, given unlabeled samples (xr y1, U1) ~ p (where the number of samples be potentially unlimited), we can define a simple squared "distillation" " loss that matches the pairwise differences of the explicit reward r' with those of the implicit policy reward defined by no. i.c.,
v2) - -8 T x Frot (3/2 (1rto v1) - (x" (2)
21
x)
Latsum(r", no: p)= = Ep Eptarer.s))
(7)
To (22 Tret (V1 x))
Due to symmetry, here it does not matter if (v1, U2) (,"'." i.c., preferred vs. dispreferred, or vice versa. Notably, similar squared-loss objective has also been recently proposed and shown to be effective in
6

---

Published in Transactions on Machine Learning Research (2/2025)
independent work by Mao et al. (2024) and Gao et al. (2024b). We now first provide further motivation for this approach, before extending it to pessimistic variants in the next section.
Intuitively, the distillation loss in (7) seeks to exactly match differences in reward model scores across all generation pairs (z, 81, 32). It is easy to see that under the Bradley-Terry model, this is equivalent to matching the strength of the preference relationship, y1 > 92- Furthermore, by only matching differences, we can still conveniently ignore the log partition term, log [(x), in the implicit reward formulation for wæ as shown in (4), as it is constant across different y for any given z. Finally, similar to the motivation in DPO, we show that minimizing Laisum(r*, TT: p) indeed results in optimally aligned policy noa as long the data distribution p has sufficient support over the space of prompts and responses.
Theorem 1. Let y denote the set of all possible responses for any model no- Assume that supp( =)) = =, i.e., the reference policy may generale any oulcome with non-zero probabilily. Purther, let supp(p(z,, V1, 32)) = supp(u(zz) x V x y. Let xe- (y x) E argmin Laiauu(""maa over all possible policies, of the implicit reward distillation loss in (7), for which r" r"(x, y) is assumed to be determinislie, and finile everywhere. Then for any O, Tg. also marimizes the aligrement objective in (1).
be minèmizer
The theorem holds for a broad class of data distributions p(x; y1, 92), and makes no assumptions on r². For example, is no longer necessary for it to be defined using a Bradley- Terry model. Notably, it applies for reward models that are much larger and potentially better than the policy model, as the reward model is not used al test time (in contrast. to DPO, which ties the size of the policy model used for generation to the size of the implicit reward model that is trained on preferences). In fact, this result can also be seen strict generalization of the IPO framework of Azar et al. (2024), which corresponds to the special case r" (2, v) [ = }, if labeled pairs (z, Ve, 31) are provided instead of the unlabeled pairs (I, V1.31.22.
Of course, the true reward r" is usually not known in practice. Still, as in standard RLHF, we can construct good proxies by using the preference data to identify plausible target reward models Yigt, further guided by any amount of regularization and inductive bias that we desire. Moreover, prior work has found that such explicitly trained reward models are more accurate and generalize better than the implicit reward model defined in (4) that is learned by DPO (Tang et al., 2024a; Lin et al., 2024). A natural choice is to thus first learn Pegt the preference data Prref using standard methods, and then reuse D Ppref to distill va, which is similar to classical settings in teacher-based model distillation (Hinton et al., 2015; Romero et al., 2015). Furthormore, as Figs is a real-valued model, at a bare minimum it is guaranteed to induce a regularized Bradley- Terry preference distribution Piga > y22 x) > O, Yz, y1, 1/2 E X x y x V,, and thereby avoid the degeneracies identified $4 for the maximum likelihood estimate under DPO.
Takeaways: Reward model distillation
If the true reward y* is known, Eq. (7) provides an objective that distills r" directly into πa- where Tg. maximizes r* as in Eq. (1). When the true reward r* is unknown, we can use any form of approzimate reward model Figt. Furthermore, any real-valued Tigt naturally adds regularization, and avoids degenerate optima such as the 0-probability solutions identified in $4.
5.2 Pessimistic reward model distillation
Choosing a single reward model Figt for anchoring the LM policy can naturally still lead to degenerate behavior if rigt is a poor approximation of the true r" that accurately reflects human proferences. However, we can easily extend our Iramework to handle uncertainty the right target reward function by defining a confidence set of k 2 1 plausible target reward models, S friger..... .., rk r and training To- (y x) maximize the following "pessimistic" form of the objective in (1):
max min Ep(x) Pro(st))li (2. v)1 E a) (riee (x, v)] (BD (To(-) [ m))] we Ti ES
(8)
advantage over the baseline policy
In this pessimistic objective we are no longer optimizing To for single reward, but optimizing wo to produce generations that are scored favorably average, even by the worst-case reward model in the set S, relative
7

---

Published in Transactions on Machine Learning Research (2/2025)
Ps(S)
SPT
TSFT ()*) = 0.3
TSFT (v²) = 0.3
22
Ki (|ma)
TyLE(y") 1.0
3T3
TT
TMLE
TM)E(y)) = =.0
T
T3 T3(y")= = =0.5
433 TT(" 0.05
Figure 1: A toy illustration of Theorem 2, which states that the optimal Tg- for (8) is the policy in Ps(S) with the lowest. forward-KT. from TSFT- The set (S) contains a (potentially infinite) set of policies T1: T..-.. cor- responding to target reward models. Here, #SFT assigns equal mass to y" and y², TMLE the MLE solution for the DPO objective, which puts all probability mass on y",, and ny the policy in Ps(S) with lowest forward-KL.
to the generations of the baseline policy Tee.. 3 When the set S { consists of only the ground-trute reward, the objective (8) is equivalent to standard RLHF (1), up to constant offset independent of 0. More generally, whenever S includes a good proxy F for r", the pessimistic advantage evaluation ensures that the policy zi that maximizes eq- (8) still has large advantage over nhe under all S including F. This use of pessimism to handle uncertainty in the knowledge of the true reward is related to similar techniques in the oflline RL literature (Kumar et al., 2020; Cheng et al., 2022).
For the objective to be meaningful, however, the set S has to be chosen carefully. When S is small, it might not include any good proxy for r". Conversely, S is too rich, it forces Tg. to be nearly identical to Trefs since any deviations from Fref might be penalized by some reward model in S. Consequently, we want to design S be the smallest possible set which contains a reasonable approximation r*.
Finally, to solve (8), we reformulate it an equivalent constrained offline optimization problem, which conveniently admits similar loss form as (7), as shown below:
Theorem 2 (Pessimistic distillation). Define the constroined minèmizer
TT- (y x) E argmin (KL (Tre((: [x) x)), meEPa(S)
(9)
where PS(S) is the set of all possible policies with implicit reward models that are consistent with any target reward model riga E S, i.c., Pş(8) & (mo, ) where no, Tretly x)) exp g (2, v). Then for any B3 0, mn- also marimizes the pessimistic aligument objective in (8).
To unpack this result, Theorem 2 stipulates that the mo that maximizes the pessimistic objective in (8) is the policy in (S) that is closest in forward KT-divergence to Tref (see Figure 1).* In addition, this policy also maximizes the expected reward of one of the rig. E S (minus the additional weighted reverse KL-divergence penalty term). Intuitively, the forward KL-divergence term serves the role of biasing the model towards optimizing for reward models that are similar to the implicit reward that Tue already maximizes. Otherwise, there might exist a target reward model rig e S for which the advantage of no relative to 7ee will be low, or even negative (a solution that we would like avoid).
5.2.1 Optimization
The constraint in (9) can be relaxed and approximately optimized by introducing an objective with Lagrangian-style penalty with strength a >> 0 on a form of distillation loss as (7), i.e.
min SE DKr m))) x)) + a min Caistat Caissuuriig Toi p).
(10)
rig ES
"It is useful optimize the advantage it cancels the effects of constant differences between reward models in S. We also free to use any bascline policy Tbwei we pick Wye for simplicity and case of analysis in $7.
*Note that the abjective in (9) minimizes the forward KI-diiergeece DEL (Fref(- [)]] a)) even though the pessimistic
objective (8) is regularized with renerse KTedivereence ( [laa x)).
8

---

Published in Transactions on Machine Learning Research (2/2025)
where for convenience divide by and instead optimize
Lpdistin To; 0) min Laisti(ri T0: p) + p(x)) DeL (Free m)) x));
(11)
where y = B/a. In reality, minimizing (11) for y > 0 is equivalent to solving the constrained optimization problem in (9) with an implicitly larger set of possible reward models Sy 2 indexed by y. More specifically, Sy also contains all reward models F that are approximately consistent. with the anchoring reward models Fige contained in S, as the following result states.
Proposition 4 (Soft pessimistic distillation). Assume the same conditions as Theorem 1. Then for any O<* 7 < o0, there eaists a X 2 0 such that To. x) E argmin Epdistin (S, Te: p), where Tg. is a minimizer over all possible policies of the objective (9), for the effective reward model set.
Sy = U (F: (pta.nn.w)) ((rgg (x. - (2, v2) F(x, f(r, s {).. Tixe
(12)
As a result, optimizing (11) even when using the singleton S {regth yields an implicitly pessimistic objective, in which the pessimism is over all reward models that are consistent up to with Figt.
Takcaways: Possimistic reward model distillation
It can often be unclear what reward model should be used as target for policy distillation. A natural optimization objective in the face of uncertainly is to optimize for the worst-case reward, out of a set of plausible reward candidates. We show that this can be formulated in a very similar style to the original distillation loss in Eq. (7) by simply adding an additional forward KL penalty term.
5.3 Pessimistic DPO
Proposition 4 can also be leveraged to obtain an alternative, implicitly pessimistic, objective that uses DPO directly instead of distillation. Consider the following regularized DPO loss:
[pd Prret) Lapo (To: D Dpret) + y p( (Trelly x)) [=)[To(y) x)).
(13)
Following similar analysis in Proposition 4, we can derive that this implicitly corresponds to maximizing the pessimistic objective in (8) for the reward model set
S, {Tru: ( D prret) I min Lapo Dpret + 2].
(14)
where Tno (z. y) & 3 log To | x))TTeer( x) log ((x) is the implicit reward model defined by ne. S, then corresponds to the set of reward models Tsa that are approximate minimizers of the DPO loss. This includes not only the MLE, but also all other estimators that obtain nearly the same loss. In principle, this can be expected to help ameliorate some of the issues of $4: since driving the reward to too only marginally decreases the Lapo loss past a certain point, the set S will also include finite reward functions Irx, (x, (x.y)) for any > 0. These rewards would then be preferred if they induce policy with smaller (forward) KL-divergence to Jye than the degenerate, infinite rewards.
6 Experimental results
The main motivation for reward distillation and pessimism to increase alignment robustness in challenging settings where is difficult to learn good policies directly from the preference data. To demonstrate the effectiveness of our approach, we run experiments 2020; Völske et al., 2017), in which we simulate a scenario where the preference data has spurious correlation between the length of a summary and whether or not it is preferred." Additionally, we show results for an unbiased setting on TL:DR, as well for an unbiased setting on Authropic Helpfulness (Bai et al., 2022).
the popular TL;DR summarization task (Stiennon et al.,
iIn practice, we also compute and optimize the min over reward models per each mini-batch examples. 6 Length has been repeatedly shown the past to correlate with reward (Singhal al., 2023; Park et al., 2024).

---

Published in Transactions on Machine Learning Research (2/2025)
6.1 Experimental setup
We first Irain an "oracle" reward model on the TL;DR preference data training set (Stiennon et al., 2020) and relabel all preference pairs with this oracle. This enables us to use the oracle reward model for evaluation, without worrying about the gap to true human preferences. After relabeling, longer responses (where longer is defined as y/1 having at least 10% more tokens than y2) are preferred in 61% of the examples.
To test the effect of spurious correlation on preference- based policy optimization, we select a training set of 30K examples from the relabeled data such that the longer output is preferred in p fraction of examples, with p {0.2. 0.3, 0.4, 0.6, 0.7, 0.8}. Each such training set is denoted D,. At each Do, compare our approach to DPO (Rafailov et al., 2023) and TPO (Azar et al., 2024), which are currently the most commonly used offiine alignment methods. We test the following variants of distillation and pessimism:
Distilled DPO (d-DPO): Trains a reward model rр on Dp. and then optimizes Cdistil Tel p).
Pessimistic DPO (p-DPO): A pessimistic version of DPO as described in $5.3, trained on Dp-
Pessimistic Distilled DPO (dp-DPO): Combines the above two by training a reward model Top on D, and optimizing the pessimistic distillation objective (Eq. (11)) with confidence set S = {Fige).
Pessimistic Ensemble DPO (e-DPO): To create ensembles of reward models, we subsample from each D, five preference datasets, D,,b, at b E B {0.2, 0.4, 0.6, 0.8), such that the fraction of pairs where the longer response is preferred is b, and train reward models rp,s on those subsets. Consequently, sensitivity to length should vary across ensemble members. We then apply the same procedure as dp- dp-DPO above, with a confidence set Sp [rp.)
All reward models and policies are initialized from Palm-2-XS (Anil et al., 2023). Policies also go through a supervised finetuning step on Human-writted summaries from the original TL;DR training set (Völske et al., 2017) prior to alignment, and we term this policy TSFT. We evaluate performance by sampling summarics for test set prompts, evaluating the average reward according to the oracle reward model, and computing the advantage in average reward compared to #şFT (before alignment).. We train policies for 10, 000 steps with batch size 16 and learning rate 10-6, and reward models for 3k steps with batch size 64 and learning rate 4 x 10-6. We use the validation set for model selection during policy training and to choose the following hyperparameters. For all DPO variants, we sweep over , E [101,.1,1 1, 3, 10,30, 100). For IPO, we sweep over T E f0.01. 3,5, 10, 25}. For all pessimistic methods we anncal 7 == a/3 from 10-4 to 10-2 lincarly during the 10k training steps (however, in later experiments performed with e-DPO, we found annealing does not affect performance and a constant y also leads to similar performance, see Figure B.5).
6.2 Results
We present the results of our experiment in Figure As can be seen the plot, the more challenging setting when p < 0.5, which corresponds to a sample of preference annotations in which shorter outputs are generally preferred. This distribution shift is more difficult because as mentioned the oracle reward model (trained on human annotations) has bias in favor of longer outputs (Singhal et al., 2023). Nevertheless we get sizable improvements compared to the reference policy TSrT for all length bias values.
All approaches that invoke distillation (d-DPO, e- e-DOO dp-DPO) outperform IPO and DPO (p < .01 by a Wald test) for p S 0.5, where shorter responses are preferred. Pessimistic ensemble DPO (e-DPO) performs particularly well in these settings, generally outperforming all methods that use a single reward model. When longer responses are preferred (p > 0.6), single reward distillation (d-DPO) leads to the highest performance, significantly outperforming both DPO and IPO (p < .01 by a Wald test). Interestingly, P-DPO does not provide empirical benefits relative to the distillation based methods, indicating that the distillation loss itself is quite important. For the effect of hyper-parameter selection, see Figure B.4. In DPO-based methods, the optimal value of / is inversely correlated with the bias; in IPO the same holds for the t hyperparameter.
To better understand the utility of reward ensembles in c-DPO, in particular when p < 0.5, we examine the role of each reward model in the ensemble across different biases. Specifically, for e-DPO, we identify for each example, throughout training, the reward model rpb that best matches the implicit reward of the current.
10

---

Published in Transactions on Machine Learning Research (2/2025)
1.00
0.75
0.50
oracle reward
Method
0.00
DPO
-0.25
d-DPO
-0.50
dp-DPO
e-DPO
-0.75
p-DPO
0.2
0.3
0.4
0.5
0.6
0.7 0.8
length blas (p)
Figure Main results, showing the oracle reward compared to the initial finetuned policy (the oracle reward of the initial finetuned policy is -1). Error bars correspond to bootstrap 95% confidence intervals for finite sample variance. Ensemble DPO (e-DPO) is significantly better than DPO and TPO in the challenging setup where shorter responses are preferred (p s 0.5), and is generally the best-performin method overall in this regime. Distilled DPO (d-DPO) performs best when longer responses are preferred (p 0.6).
policy, i.e., for which reward model is Ldsstill minimized on that example (see Fq. (7) and (11)). We find that when the policy is trained on data where shorter preference are preferred (p < .5), the reward model that best matches the policy often has the opposite bias (b is high), and vice versa. Thus, the success of c-DPO may be explained by its ability to distill from reward models that do not suffer from the bias the policy training data, which particularly helpful when p S 5 as this bias also not shared by the oracle RMM. We provide the full distribution over reward models for all p and 3 in Appendix B.3. Overall, these results demonstrate the efficacy of training policy by distilling from reward model in the presence of distribution shifts, and that careful design of ensemble to mitigate spurious correlations can lead to further performance gains.?
6.3 Additional results in an unbiased setting
To test the ability of our method to perform well on preference tasks where no bias is present, we next run experiments on the Authropic Helpfulness dataset (Bai al., 2022). We use a Gemini 1.0 Ultra (Gemini Team, 2024) LLM-as-ad asdduge model for evaluating win-rates of the policies over both the SFT starting point and the best DPO baseline. As shown in Table 1, in this unbiased setting our distillation objectives provide modest gains. Concretely, e-DPO's win rate against the SFT policy is 65.8%, while DPO's win rate is 64.2%. Moreover, comparing eDDPO and DPO directly, e-DPO wins in 49.7% of the cases, while DPO wins in 46.9% of the cases (the rest are considered to be ties with no preference relation).
also
Takeaways: Experimental results
Reward model distillation, specifically pessimistic reward model distillation when ensemble of reward models is used, leads to improved robustness tasks where bias is present in the preference data. In addition, reward model distillation results in small improvements in performance even on unbiased settings, making it a simple but compelling algorithmic modification to offline training.
7 Theoretical analysis
This section characterizes solutions offered by pessimistic DPO and distillation to the issues identified in $4, focusing on the simplified scenario which we optimize with respoct a single preference pair (y"', y)
7 We note that we also experimented with ensemble where members are different checkpoints training reward model the preference data, however, did observe any empirical gains from this form ensemble.
11

---

Published in Transactions on Machine Learning Research (2/2025)
Method
%Wnss
%Ties
%ossss
Method
% Wins
%Ties
%Losses
c-DPO SFT
65.8
1.3
32.9
c-DPO vs. DPO
49.7
3.4
46.9
d-DPO vs. SFT
65.6
1.0
33.3
d-DPO DPO
48.2
3.6
48.2
DPO vs. SFT
64.2
1.0
34.8
Table 1: Side-by-side win rates on the Helpfulness dataset (with Gemini 1.0 Ultra evaluator).
7.1 Optima
In its Lagrangian formulation, possimistic DPO adds a forward KL term to the DPO objective ($5.3). Here we seek to better analyze how this additional term effects the optimal policy. For the sake of analysis, we assume that the preference annotations are sampled from the reference distribution, p(z) x Tro x)) x Teet x). Then a sinite-sample approximation of the forward KL term
2(())=
Σ
-(log ro('") + log ()))")).
(y" ()DDPil
By applying this finite-sample approximation, p-DPO has finite optimum, unlike DPO, as shown in Proposition 1. Note that this analysis is limited in two ways: (1) as mentioned, we compute the KT. term over the completions in the preference data; (2) we directly optimize the probability ratios the = no (y ) (y""(y") and uhy ( (vV"), rather than optimizing them jointly through the parameters. For sulficiently expressive πe, however, this approximation captures the behavior of the two algorithms reasonably well
Proposition 5. Let Epapo represent mmitte-sampe approzimation to Epdpo with the empirical forward KL term (2(@). For fixed Atly!") and a > 1, the argmin Epdpo is min (1 =o(v"")) fo with log Regee 1o log (a - 1) log (v")) log ".)e(";"))
The optimum in Proposition 5 corresponds to log tw/tee 3- log(a 1). Recall that IPO (Azar et al., 2024) seeks to assign a constant value to this ratio by minimizing (log the --- the (unconstrained) optima are identical for T- = 8- log(a - 1), but the loss surfaces are different (see further analysis of this in $7.2). DPO sets T - 0, as shown in Corollary 1; this is due not only to competition from ro(") but from DPO penalizing positive probability on of- Analysis of the distilled loss gives similar result:
Proposition 6. For any fixed ) and B > 0, the argmin of the diställed DPO objective in (7) is m.r) min(1 te (v.")) (e()))), with log (v (r.(x, vf) re(x, ).") + log to(ly")) + log Foat(?"")
While the setting is simplistic, the results are comforting: here the additional regularization effects of both distillation and pessimism (in the case of p-DPO) clearly help to avoid degenerale optima.
7.2 Transitive closure: p-DPO vs IPO
As pointed out in $7.1, both p-DPO and IPO target a constant ratio for log / Despite the similar form of optima, however, the loss surfaces of the two objectives differ in notable ways. To see this, we consider a simplified setting with three possible outputs, yı, V2: Y3- We observe either = {(y y2). (y2 x ys)} or multi-arm bandit, the goal is to assign weight to each arm, which denote ti; = log (Jr x) Zz, with Za underdetermined log- rag-partiiion function.
D D {M) < y. If we treat. this problem
Proposition 7. Let D= {(i, 1) i 1,2,..., for n 2. Let D be the dataset arising from the transitive closure of D. Assume Neet indifferent to all (yt. Let (D) = max, vP - min, wPP) Then
( = (1-1)7-1 > the 2n
Intuvvllly, the observation of y1 < ys should increase confidence that ys is superior to yı, but in IPO it has the opposite effect, drawing their scores closer together. While pessimistic DPO also has a target ratio between each preference pair, its loss surface is different: in particular, does not increase quadratically move away from the target. We find empirically that pessimistic DPO robust to the transitive closure
12

---

Published in Transactions on Machine Learning Research (2/2025)
of preference annotations in the multi-arm bandit setting, as shown in Figure B.2. As discussed above, DPO will set 1/1 + -00 because y1 is never preferred. Specifically, in our experiments we solve the p-DPO and IPO objectives for both D = {(y1 32). (12, }a)} and D = DU {(1, 8a)}, solving with respect to {Ta(M)) IPO is solved analytically as a quadratic program; for p-DPO we used projected gradient descent. We consider 3 E (1,3, 30) and a E (5, 10, 20, 50, 100, 1000). As shown in Figure B.2, there are significant differences in the TPO solutions with and without transitive closure, while for p-DPO these differences are imperceptible.
Takeaways: Theoretical analysis
Both pessimistic DPO and reward model distillation avoid the degenerate optima of DPO that were analyzed in 54-pmiiarlly through adding additional sources of regularization to the objective. In addition to these more stable optima, we also show how the loss surface of p-DPO can lead more favorable outcomes with respect to modeling challenging transitive preferences TPO.
8
Conclusion
LM alignment is crucial for deploying safe and helpful assistants, but is difficult due to lack of accoss to perfect preference oracles. We presented thorough theoretical analysis of some of the degeneracies that DPO is sus- ceptible to when learning from sampled human preference data. Furthermore, our findings suggest that explicit reward modeling remains a powerful vehiele for introducing regularization into post-training. By distilling the reward assigned by a single, explicit reward modcl-or a family of explicit reward models-directly into the implicil reward maximized by our policies using oflline data, we demonstrated that we can achieve improved robustness to variations in preference dataset quality, while maintaining the simplicity of offline alignment frameworks. Finally, reward model distillation also results in modest but consistent improvements in perfor- mance even unbiased settings, making it an overall compelling algorithmic modification to offline training.
Acknowledgements
We thank the anonympous reviewers, Alexander D' D'Amour, and Chris Dyer for helpful comments and feedback on the mamuscript. This research also benefited from discussions with Victor Veitch, Mandar Joshi, Kenton Lee, Kristina Toutanova, David Gaddy, Dheeru Dua, Yuan Zhang, Tianze Shi, and Anastasios Angelopoulos.
References
Al@Meta. The llama 3 herd of models, 2024. URL
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint wwXiv:2402.10577 2024.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barhan, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Chon Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sncha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lec, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pollat, Martin Polacck, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Sacta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha
Aakanksha Chowdhery, Clément
13

---

Published in Transactions on Machine Learning Research (2/2025)
Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/ hos/2204.005622, 2022.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 -455, 1952. ISSN 00063444. URL http://www.jest org/stable. 2334029.
Daniele Calandriello, Zhaohan Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 4 41sl International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 5409 5435. PMLR, 21-27 27 Jul 2024. URL https: pttps://proceedingg .mr.. www.mlrrrsssssee. (ress/v235/calandriee
html.
Souradip Chakraborty, Soumya Suvra Ghosal, Ming Yin, Dinesh Manocha, Mengdi Wang, Amrit Singh Bedi, and Furong Huang. Transfer q star: Principled decoding for Imm alignment. arXiv preprint ar arx::22405 20495, 2024.
Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for oflline reinforcement learning. In International Conference on Machine Learning, pages 3852 3878. PMLR, 2022.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXie preprint #xXi:2310.027453 2023.
Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.078633 2024.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.008 0924444, 2023.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learnóng Research, pages 1607 1616-1616. PMLR, 10-15 Jul 2018. URL https: httpp::s//prooeednggs mlr wttaa///////ttaaallllllll html.
Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. arXiv preprint or arXiv:2404.090 2024a.
14

---

Published in Transactions on Machine Learning Research (2/2025)
Zhaolin Gao, Jonathan Daniel Chang, Wenhao Zhan, Owen Ocrtell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Tee, and Wen Sun. REBEL; Reinforcement learning via regressing relative rewards. In ICML 2024 Workshop on Models of Human Feedback for Al Aligmment, 2024b. URL bttps://openrevan net r
Gemini Team. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv..org abs/2312.1180)
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. Xiviv preprint arxiv:2404 09656, 2024.
Dirk Grocneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodncy Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, lan Magmusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zeutlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings ( (Volume 1: Long Papers),
of the 62nd Annual Meeting of the Association for Computational Linguistics pages 15789 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1118665//112224.acc/ .ol.-......11.. 841. URL https: //tast/holaantooggo...oo. wwwpanooooyoooogggg/ee
841.
Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. Direct language model alignment from online ai feedback. arXiv preprint arXic:2402.00 04 792, 2024.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http://arxiv.orgg org/abs//////oeeooooo...o... 11///11033.1 02531.
Tomasz Korbak, Ethan Perez, and Christopher Buckley. RL with KL penalties is better viewed as bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1083-1091, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offiine reinforee- ment learning. Advances in Newral Information Processing Systems, 33:1 77-77911111199 2020.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint 2309 002677 2023.
Yong Lin, Skyler Seto, Maartje ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, and Tong Zhang. On the limited generalization capability of the implicit reward model induced by direct. preference optimization. arXiv preprint arXio: 2409. 030003350, 2024.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off- off-policy reinforcement learning without great exploration. Aduances in neural information processing systems, 33: 1264 1274, 2020.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint (xxx::2/05516436 2024.
Xin Mao, Lin Li, Huimin Xu, Wei Zhang, Wang Chen, and Anh Tuan Luu. Don't forget your reward values: Language model alignment via valuc-based calibration. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Miami, Florida, USA, November 2024. Association for Computational
Processing, pages
15

---

Published in Transactions on Machine Learning Research (2/2025)
Linguistics. doi: 10.18663/x1/2024.mmml mainnlm-.....76... URL http://////acaaathhoooog///// http:///rrooooooooo//2222222 emnlp- main. 976.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. Advonces in Neural Information Processing Systems, 2024.
Ted Moskovitz, Aaditya K Singh, DJ Strouse, Thomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04377, 2023.
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng- The Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. In Forty-first International Conference on Machine Learning, 2024.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.1 13228, 2024.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint orXie:2408.19150 2024.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a. reward model. Advances in Newral Information Processing Systems, 36, 2023.
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, W. Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: ///pennevvew.neee///eeeeeeeeeeee wwwte//oruuiddppt0ll..
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q": Your language model is secretly a q-function. arXiv preprint 244. 12358, 2024b.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal aligmment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, GeofTrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. arXiv preprint 47X117440.111877 2024.
Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjecv Arora, and Boris Hanin. Unintentional unalignment: Likelihood displacement in direct preference optimization. orXiv preprint arxiv: 4410088847 2024.
Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pielquin. Countering reward over-optimization LLM with demonstration guided reinforcement learning, In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, oditors, Findings of the Association for Computational Linguústics ACL 2024, pages 12447 12472, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguisties. doi: 10. 10.18653/v /200///////222...// /202/./22.//mmmmmm Sin24.indings-cl..4. URL https: https://aclanthology org/ 2024 foddffinningsacll .400.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deop nets. In In Proceedings of ICLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alce Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiu: 110.100770334.3 2017.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. Xio preprint or arXin:2310.03771 2023.
16

---

Published in Transactions on Machine Learning Research (2/2025)
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Newral Information Processing Systems, 33:3008 3021, 2020.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback, 2024.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of 1lms should leverage suboptimal, on-policy data. arXiv preprint Krr::22404.144 14367, 2024.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arx: 2405.08448. 2024a.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Danicle Calandricllo, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offiin alignment. arXiv preprint ar arXi::2402.00770 05749, 2024b.
Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the Workshop on New Frondiers in Summarizalion, pages 59 63, Copenhagen, Denmark, September 2017. Association for Computational Linguisties. doi: 10. 10.18653/v1 01/66///11/11-45/5 11/117-4508. org/W17-4508.
URL https: https://aclanthhrt hotoolllll.n
Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Minciro, and Alckh Agarwal. Bellman-consistene pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683-660 06036,66444- 2021.
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feodback: Bridging theory and practice for RLHF under KL-constraint. Forty-forst International Conference on Machine Learning, 2024. URL tttps://openreviies wttprreviiiieee/ffuurnte.. net/forum?
id=c1AKcA6ry1.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu,, and Yi Wu. Is DPO superior to PPO for LLM alignment? comprehensive study. orXiv preprint ar 204,11904,10119, 2024.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L. Yuille. Training deep neural net worksrks in generations: a more tolerant teacher educates better students. In Proceedings of the Thirty- Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAT Symposium on Educalional Advances in Artificial Intelligence, AAAT19/IAAT19/EAN A htt///ai.ooooooo orrg org/
AAAT Press, 2019. ISBN 978-1-57735-800-11 doi: 10.1609/ aaaaaaai.33333 .33015628. URL https: //toi.///o////// 10. 1609/ 1/609//aa..... aaai. v33101. 33015628.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for oflline reinforcement. learning. Advances in neural information processing systems, 34:13626 14:13666-136400, 2021.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct. preference optimization. arXiv preprint arXim:2404.11199, 2024.
Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty- penalized reinforcement learning from human feedback with diverse reward lora ensembles. arXiv preprint 0rX0022401.00243, 2023.
17

---

Published in Transactions on Machine Learning Research (2/2025)
Wenhao Zhan, Masatoshi Uchara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable offline preference- based reinforcemeett learning. In The Twelth International Conference on Learning Representations, 2024.
URL httpp://ppenreviev.nettt net/forum?id==WWWPP
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 43037-43067. PMLR, pteps/v202/ zhu23f .tmll.
23 29 Jul 2023. URL https: httpp://tppot http::::/ppoceeeingg..om
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoplimization in rlhf. Xiv preprint or 21iv:2(01.16333 2024.
18

---

Published in Transactions on Machine Learning Research (2/2025)
A Proofs
A.1 Proof of Proposition 1
Proposition (Proposition restated). Under Assumption 1, for any (y,y') v') such that y v." and y' - vf for some i, we have Nee (a) + s0, for all global minimizers we- of the DPO objective in (6), for any B > 0. (20 (er(())
Proof. Since all the preference pairs (y.0') are mutually disjoint, and By is specific to each y, the DPO objective over pref convex (Ai,.... An where
Tel (V, (T)) A 8 log Te (y, Feet (v,")
(15)
Furthermore, the different A; completely independent from each other due to the preference pairs being disjoint, they can be optimized over separately.
In particular, for every we have that
lim - log (or (A:)) = 0, 4,0
(16)
which implies that A = {oo}" is the unique global minimizer of the DPO loss over Dpret in the space of A's, and any Hr that is global minimizer must therefore satisfy
[e(" To (y() (y Feel (y!")
log
(17)
A.2 Proof of Corollary 1
Corollary (Corollary 1 restated). Under Assumption 1, further assume that 0 < Teet for all y. Then
Tg- is a global minimizer of the DPO objective in (6) iff me. (C(y²))) + with we- (y. ) > 0 Vi E [n]. where
Cly")o is the complement of the set of all responses y that appear dispreferved yf for any i [n].
Proof. Following the same argument of the proof Proposition 1, we have that all global minimizers 0" of the DPO satisfy A; = oo, which in turn implies that
TT- (v) Tref (y
(18)
TT (y) Teee (y
Since (1) is assumed to satisfy Tue(()) < 1 for all y, this implics that all 6* satisfy
TT (v²") Tg. (uf)
0o,
(19)
which further implics that Th- (vf) 0 and Tg- (v!") > 0 for all E [n]. as n (y-") I 1 for any w." Aggregating
C(y)) = {y: Di [m]] s.t 3 = #}
(20)
then gives that
T-- (C(ve)) = Σ
Te- (v) Te- (C(ye) =
(21)
NEC(()))
7
19

---

Published in Transactions on Machine Learning Research (2/2025)
To prove the converse, let Ter be policy that satisfies ne (C()))) = 1, with To (vl") 0, Vi [n]] As Wo (y) 2 0 for all y, this implies that (T(())) = 0 vi E [n]. Then, we have
T (y Tor (uf)
00,
(22)
which by Proposition implies that ze a global optimum.
A.3 Proof of Theorem 1
Theorem (Theorem restated). Let Y denote the set of all possible responses for any model TT. Assume that supp(Trerly z)) = = Le., the reference policy may generate any outcome with non-zero probobility. Further, let supp(p(x,, y1, 32)) = supp(μ(x)) x y x y. Let Te- (y x) E argming, Laisau(r,, ne: p) be a minimizer over all possible policies, of the implicit reward distillation loss (7), for which r" (xxy) is assumed to be deterministic, and finite everywhere. Then for any O, Tg. also marimizes the alignmmnt objective in (1).
Proof. We know that the optimal policy for the RLHF objective (1) is given by Te (ylz) [x) exp(r"(x. ()))). Plugging this policy into the distillation objective (7), Ldisum (r", .T (a" = 0 for all P. In fact, the loss is equal to pointwise, meaning that Tœ- is global minimizer of the distillation objective (7). Further, let be some other minimizer of Laimmu("" .p" Then и also has to attain a loss of 0 at all (y)),y y in the support of meaning that log (y]=) - log (T/ log ne- (ylzz) - log Tg. (y|z) for all (x, v.y) in the support of p. Consequently, the two policies coincide in the support of p (due
that
the normalization constraint, there is no additional offset term allowed as the support of
p covers all of V). Finally, noting that the support of the chosen p is such that To. puts no mass outside its support due to the KL constraint in (1), we complete the proof.
A.4 Proof Theorem 2
Theorem (Theorem 2 restated). Define the constrained minimizer
T x) argmin ( (Fre(((())) (x) (To( x:);
TeEPs(S)
where Ps(S)= is the set of all possible policies with implicit reward models that are consistend with any targel reward model rigs E S, i.e., Pş(S) A [ where To, o Trelly x) exp 3 (z. v). Then for any B > 0, no- also marimizes the pessimistic alignment objective in (8).
Proof. Consider the pessimistic objective:
max min Ep(a) (e v)] - (x (z, w)]] DDL (To (eer). ES
(23)
As it is linear in Tig and convex in we can switch the order of min and max:
min Plge ES
max E Em(s) mell
Eagyla (x. v)l Ex Irng (z, w|l] BDDr. ( (
(24)
Note that every Figt E S can be written in terms of the KL-constrriieed policy Thet it induces, i.e.,
T (3 x) Te (y) x) + Bog Z(z, rigt);
(x = log
(25)
where
= arrgaxx Eu(z) E ( (ige (2, f)1 BBDDL ((ee))
(26)
Fig
20

---

Published in Transactions on Machine Learning Research (2/2025)
which has the form where (ax, (egt) is the partition function:
1 (1x, rege)
Ever u)) exp (r)eep(( (3)es((r1))
Peg (y/ 2²)
(27)
(x,, Figt)
Σ
(ye x) exp (3T1a(2.)))
(28)
DEy
Substituting ihe
in for max, and writing Figt in terms of The get the simplified objective
min E5
max E Bx((r)
E (rtge (2,0)) y)] - Bn m/1| 5I.. ( (I)er)
sell
Tiee (y) x))
+ Blg Z (2x, ress)
min Pig
(x(x)
Briga (wlz) | log
Txel( x))
Trem
(y (x)x)
B log
+ B log (xx, (gt)
(29)
Trref (y x))
31D Trot ax)
= min ES
8 (())
DeL ( I[ee ()) + DELL (Trer (x)- DKL ( | I xx)
min E5
SE p(a)
DrIeered ll mh xx)
A.5 Proof of Proposition 2
Proposition (Proposition 2 restaled). Let y", E V" be preferred versus dispreferred oulpuls of length n, respectively, with ret (y (y)) O and corresponding count vectors c(y""), cly"). Let log To(y) = c(y) 8 - n0) for Z(t) = log Σi ctt, with upper bound log to(y) cly) 0 - n max; 8,. Let g(t) represent the parumelers of π after sleps of gradient descent on Lapa ((v .9t ,)),, with o()) = 0. Then, we have that
Toc (1)²) S Foac (2))) # for all t, with strict incquality when |Ic(y") c( 1.
Proof- Let A [e(y")) - c(u¹)] and p Tres Tree(1" \ (y'). The theorem assumes 1b" 1/ Then Lápo log (3(A 0) + 3 log p). The derivative with respect to 0 is,
( (1-0(8(A-0) Blog p)) -plo 0);0)33 0.
(30)
00
Let & = -ppl > n0 0(1)). Then,
# =0(() -(()") - maxx (/0)
(31)
-(0(1-1) +.,)) c(y²²) n max (t-1) (5AA))
(32)
-00-11 1) c(y00) n max 0 + S.A cly ) rad, max Aj
(33)
=ā0 1s + 6,
A A A(y(y n max 4,)
(34)
= + 5 Σ ()0 - max Ar) S aga 1)-
(35)
21

---

Published in Transactions on Machine Learning Research (2/2025)
We obtain max, (0f--1) + ر,4,) = max, 034 + max, 6,A, from the fact that ((0) 0 and therefore
arg max A implies j arg git') for all t' > O. The second-to-last step uses Σ cg((")) and the final step uses Aj < max' Ay- Finally, we have Tace (v) S Tgaes (1²) because Z log En exp z log max, exp 0j = max, fz.
A.6 Proof of Proposition 3
Proposition (Proposition 3 restated). Let v and y be preferred versus dispreferred outputs of length n. Let A = cly"*) - c(y²) be the difference in unigram counts. Let 9 = [i, i), for E any mnymax A, with |c(()]l1 = n. Then Taes (y1²) Tg(e)) (1) T(t)k for some S and some non-decreasing T - + R+.
Proof. Applying gradient descent with learning rate n, to the gradient from Equation (30), at each step t the parameters are,
g (t + n1ppyl > 1) Σ nopli > 301 A T(t) At)))
(36)
Plugging these parameters into the likelihoods,
(cly")) lotes (0) - c(yy")) (00) nZ((()) - -() 100) + -0((00)
(37)
=clly") c(ā)) 0(0) (c(y") di)) (T(1))
(38)
=T(+)(c()"") max A) = =(tkk
(39)
with k 0 0 by e(y"). A < ||(("")|1 x |1 maax A.
A.7 Proof of Proposition 4
Proposition (Proposition 4 restated). Assume the same conditions as Theorem 1. Then for any 0 < < xx, there erists a A 2 0 such that xe- (y | x) E argmin Epdistil(S, no: p), where ag- is a minimizer over all possible policies of the objeclive (9), for the effeclive reward model sel.
S,
U rige ES
{F:: Ep(n.s) [(rign (x. y1) rizs (z. (2) F(x, F(2, 82)) s $²}.
Proof. The proof is a standard Lagrangian duality argument, which we reproduce here for completeness. For two functions (() and g(2), let us define
argmin f(z) + a2()).
(40)
Let us also consider the constrained problem
= argmin f()) s.t. g(2) s g(2*).
(41)
Suppose by contradiction that 2" is not a minimizer of (41). Since x" is feasible for the constraint by construction, we get that f(z') < f(2)). Consequently, we further have
f(z') + ag(2)) f((") + ag(2").
where the inequality follows from the feasibility of in (41). This contradicts the optimality of =* in (40), mcaning that 2* must be minimizer of (41). Applying this general result with = 3E DKL (y x:) ()) a))) min, es Laistm(ri To: p): and = no completes the proof, since recognize the set S.,
in (12) to be equivalent to U es (rissun n8; p) < X.
22

---

Published in Transactions on Machine Learning Research (2/2025)
A.8 Proof of Proposition 5
Proposition (Proposition 5 restated). Let Lpdpo represent sinile-sample approzination to Lpdpo with the empirical forward KL term ((()). For a fixed #o(y!") and a > 1, the argmin Epdpo is min (1 - " " Ao()))), with log fo(( - - log (a - log ) + log B.rl
Proof. We differentiate Lddpo with respect to th = (T((")) /rer with implicit, obtaining,
pdpp Oie
B -- = -.55"
v/ a
=B
-
(42)
Ve + v/e
Ve + +
which is zero when,
a = = + +
(43)
1/3
1/ =
(44)
log the 3
log(a 1) +og v'.
(45)
log ग (y("") = log Tror( log (a - 1) + log m)"")) log Fref Fref(y") (y"
(46)
By the second-order condition, the critical point is a minimum. The objective Cpdpo the sum of components: the negative log sigmoid term for L, and the negative log probability for Ω. Because each component a convex function thi so Lpdpo- As a result, the local minimum log to (yy) also a global minimum.
A.9 Proof of Proposition 6
Proposition (Proposition 6 restated). For any fixed (vlv)") and 3 > O, the argmin of the distilled DPO objective in (7) min(1 (v.",,, aa((()))))))))))))))) with log (o(v() (rr(2, v/) r rx(,,, v(") +log *o("!") ) log (r
Proof. This follows directly from differentiating (7) with respect to mT(22).
A.10 Proof of Proposition
Proposition (Proposition restated). Let = {(i, + 1) 1,2...., for n > 2. Let be the dataset arising from the transitive closure of D. Assume nee is indifferent to all (y,, Vz). Let w max; (D min; (D)) Then w (n 1)r ( 2 2-17
(D)
Proof. For D, the IPO objective be minimized at zero so that will) (n 1)7-1 For D, each adjacent pair of completions is soparated by y, and the objective is Σt= (n i)(in 1117---112 The minimum is m(n+1)(n 1)/6
2 -1 so that w = (n 1)y = 2 2== T < (n - 1) 1 for n > 2.
=
n (n 1] 1)/12
Additional results
B.1 The effect of distillation and pessimism on likelihood collapse
Figure B.1 tests the effect of pessimism in both DPO (DPO vs p-DPO) and distilled DPO (d-DPO vs dp-DPO) likelihood collapse the preferred vs. dispreferred generations the offline data. Note that a subtle point arises from the distinction between regularizing the reference policy output distribution versus regularizing to the preference data distribution, which are only (asymptotically) identical if preferences annotated
23

---

Published in Transactions on Machine Learning Research (2/2025)
proferred log. likellhood
disprefarred likellhoed
nef
without
15.0
.00
-500
12.5
-0.0
1000
8-0
valee
-1000
1500
valeo
-1.0
-R- p-tPo, p-0.0,$-1.0
-1500
-200
5.0
-1.2
2.5
-2000
-1.4
3502
5000
1000
2000
3003
4003
1003
2003
3000
4200
5300
2000
4000
Step
5000
proferred log licellhood
disprefarred licellhoad
without
-0.5
2000
-1.0
Maw-0
value
-6000
40
X --
-#-
d 0.000
4200
-6900
8000
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
1000
2000
3300
4000
5000
2000
4000
186p
step
s8ap
step
Figure B.1: Pessimism mitigates likelihood collapse. In this case, we penalize the sample KL-divergence rather than the data KL-divergence, because the samples are not drawn from the preference data distribution. Thus the effect of regularization is primarily kl d__oss_reéd ref, which quantifies this divergence, rather than on the likelihood of the preference data itself, (dis)preferred loreedd___ooooooe lik_liiklioodd.
a sample from the reference policy (in our initial experiments we had found slightly better results overall when regularizing the reference policy output distribution). Figure B.1 reports results with respect to both distributions, showing that pessimism (a) mitigates the decrease in probability of preferred and dispreferred preference annotations, despite this data not being used in the regularizers (left-most subplots), and (b) mitigates the increase in KL divergence with respect to the reference distribution (third subplot from left), as expected due to the additional regularization term. As argued in $4 (see also Azar et al. (2024)), the B hyperparameter of DPO does not effectively regularize this KL distribution because the implicit DPO reward model assigns infinite- magnitude rewards.
B.2 Transitive closure
Figure B.2 shows the results of our multi-arm bandit experiments with p-DPO vs. IPO losses, as described in $7.2. In this synthetic setup, we solve the p-DPO and TPO objectives for both D = {(v 32), (12, Us)) and D = DD (o((11)3))) solving with respect to {πT()))}
B.3 Distribution over reward models for e-DPO
Figure B.3 investigates the reason for the success of c-DPO, especially when p << .5. For every length bias, we track during training for all training examples which reward model, Tnb, best matched the implicit reward of the currently trained eDPPO policy, and plot the distribution over reward models. The policy matches different reward models in different examples. Moreover, there is inverse correlation between the data bias for policy training (p) and the data bias for training the reward models (b). This suggests that the ensemble in e-DPO helps the policy is distilling from reward models that do not share the data bias of the policy training set.
B.4 Hyperparameters
Validation set performance across the range of hyperparameter settings is shown in Figure B.4. Figure B.5 also explores the impact of y (which is minimal). In pilot studies we found that these results were relatively robust variation in the random seed, but did not conduct extensive investigation of this effect across all methods and hyperparameters due to cost.
24

---

Published in Transactions on Machine Learning Research (2/2025)
P-DPO: n(y1)
P-DPO: n(ys))
P-DPO: m(ya)
0.6
beta
1.0
0.4
3.0
10.0
0.2
30.0
transitively_ _closed
False
0.0
True
IPO: nly1)
IPO: mly2)
IPO: n((ya)
1.0
0.8
0.6
0.4
0.2
transitively. closed
False - True
10-1
10
10-1
10
10-1 100
Figure B.2: Effect of transitive closure on p-DPO and IPO solutions to preference learning in multi-arm bandit. Fach column shows the learned policy probability for given arm, based on the preferences V2 < V/. The top row shows that in p-DPO, the probabilities are not materially affected by the transitive closure 1/1 < Va- The bottom row shows that in IPO, transitive closure causes the probabilities compressed In each subfigure, we sweep a range of elfective values of --1, shown on the x-axis.
C Compute resources
We train policies on 32 TPU v3 chips and reward models on 16 TPU v3 chips. We obtain roughly 0.1 steps per second when training, for both the policy and reward models.
25

---

Published in Transactions on Machine Learning Research (2/2025)
100
of time fp.a is picked
40
b=0.2
b=0.4
0=0,5
b=0.6
b=0.8
0.2 0.3
0.5
0.6
0.7
0.8
Length bias (p)
Figure B.3: For all training examples, we track which reward model during training best matches the implicit reward the current e-DPO policy and plot the distribution over reward models, for every length bias, p. We observe that the e-DPO policy matches different reward models across examples during training, Moreover, when the policy is trained with data biased towards preferring short responses, the reward model that was trained longer responses is by and large preferred and vice
DPO
IPO
1.0
beta
ipo tau start
0.01
0.1
0.5
reward
1.0
1.0
3.0
3.0
0.0
30.0
10.0
100.0
25.0
d-DPO
dp-DPO
1.0
beta
beta
0.01
+ 0.01
0.1
0.1
reward
0.5
1.0
1.0
3.0
3.0
0.0
10.0
30.0
30.0
100.0
100.0
e-DPO
P-DPO
beta
0.01
0.01
0.1
0.1
0.5
3.0
3.0
0.0
10.0
30.0
100.0
100.0
-0.5
bias (p)
bias (p)
Figure B.4: Validation set results across hyperparameters for each method. For all methods, different
values of p induce different optimal hyperparameters 3 and 1-
26

---

Published in Transactions on Machine Learning Research (2/2025)
dpo gamma
1.8
annealing w/o annealing
1.6
advantage
1.4
1.2
1.0
0.2
0.3
0.4
0.6
0.7
bias
Figure B.5: Comparing e-dpo with and without annealing of on the development set for all length biases using the best value 3 per bias. Annealing ? has minimal effect on performance and is not necessary for the success of o-dpo.
27


Reviews:
[{"review_id": "828fXXgB7z", "reviewer": "Reviewer_RV8o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses a known issue with DPO class of algorithms. The paper proposed a new pessismitic reward model distillation approach to handle the known issues of probability mass transfer in DPO.\n\n\nWeaknesses:\n1. In the abstract as well as further places in the text, the authors mention that they \"analyze the phenomena of probability of preferred generations going to zero in DPO\". It seems like prior work has shown this both theoretically and experimentally. Refer [1]. Proposition 1 and Corollary 1 of this paper follows from Proposition 1 in the paper [1].I think it might be more accurate to cite the relevant sources and make clear that this is not the contribution of this paper. \n\n2. The authors propose the algorithm in section 4.1 of reward model distillation. This algorithm matches exactly to the REBEL algorithm [2] accepted in NeurIPS 2024. Refer to equation 5 in their paper vs the equation 7 in this paper. I fail to understand if the authors are claiming this as their contribution or this is a interlude to the pessimistic approach they go towards in the next section. The theorems derived in this section also follow from the proofs of paper[2] where they show the reward model distillation approach is a principled way to solve the RLHF objective.\n\n3. While the [2] paper is purely in the online setting and I see the difference that authors aim to use this technique in the offline setting but the offline setting seems to be just one iteration of the online setting. \n\n4. For the implementation of pessimistic model distillation is the reward pessimism enforced per state (the worst reward per prompt) or it is the worst reward function over all prompts in an expected sense? The proofs indicate later but the empirical objective indicates former.\n\n5. Empirical results: The experiments on TLDR show a very slight improvement over IPO and it seems insufficient to conclude that this method actually results in consistent improvements. On the helpfulness dataset, other offline algorithms are not evaluated. The issues that motivate this paper plague all offline alignment algorithms (DPO, IPO, etc) as shown in [1] and the fixes here should result in consistent improvement over other offline baselines as well.\n\n\n[1]: Rafailov, Rafael, et al. \"Scaling laws for reward model overoptimization in direct alignment algorithms.\" arXiv preprint arXiv:2406.02900 (2024).\n[2]: Gao, Zhaolin, et al. \"Rebel: Reinforcement learning via regressing relative rewards.\" arXiv preprint arXiv:2404.16767 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a hybrid approach to fix the issues underlying DPO style methods. To combat the issue of arbitrary scale of reward functions learned by DPO and potentially OOD transfer of probability mass, authors propose to learn a seperate set of rewards and use it to distill policies. Furthermore, to make their method robust they use a pessimistic reward optimization procedure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I think the contribution of this paper is investigating the already published REBEL objective for the offline setting and figuring out ways to improve offline RLHF. This contribution is perfectely reasonable but the paper currently has over claims on multiple parts and from reading seems like reward model distillation + pessimism along with theoretical analysis is the novelty of the paper. Assumi that the novelty is investigation of REBEL in offline setting, a single dataset experiments with 1 seed seems insufficient to conclude the novelty of the method.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5MP1Tcc4Hz", "reviewer": "Reviewer_QDvm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The paper analytically studies the problem where-in DPO provides degenerate solutions (policies). The analysis uses a simplified setup to show that DPO exhibits a form of overfitting where-in the preferred output can be assigned a small non-zero probability which can lead to degenerate solutions. \n\n- The paper addresses the shortcoming identified above with a proposed explicit reward model that is used to update the model (LM) by distilling the outputs from the reward model either with single or multiple reward models (\"teachers\" if I understood this correctly) as well as a pessimistic version of DPO. The key contribution here is the proposed regularization term that seeks to match the distribution of the explicit reward model with that of the model (LM).\n\n- The paper conducts an empirical study to show that the proposed methods (distillation, pessimistic reward model and a combination of the two) show improved performance in cases where it is challenging to learn a good alignment model directly from preference data. \n\n# Weaknesses\n\n- The initial development of showing an analytical case where DPO can provide degenerate solutions is a strength of this paper. However, the development is spread of several seemingly disjointed sections within the main paper itself which may make it unnecessarily harder to follow for a general audience.\n\n- The paper does not make a connection to the issue uncovered via analysis to what is observed in practice even though the empirical shortcoming is mentioned in the paper. Specifically, it would be good to see how the probability mass being moved away from preferred solutions can be resolved with the methods proposed in the  paper. These ablations would be very beneficial to the reader\n\n- The paper studies one dataset that has a bias (length of summary) and shows how some of the newly proposed methods improve over DPO and IPO. Given the use of an explicit reward model, it may make sense to compare with other methods that propose explicit reward models. Or perhaps provide a discussion on why comparing to DPO/IPO is reasonable in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies a shortcoming of  Direct Preference Optimization (DPO) where-in DPO assigns infinite reward value that may lead to degenerate policies. Instead, the paper proposes an approach that updates the model (LM) to match the output distribution of an explicit reward model via Kullback-Liebler regularization term and shows that the regularized methods avoid degenerate solutions. A distillation based approach and a ``pessimistic'' optimization based approach are proposed to train the explicit reward model. The paper empirically compares the new approaches listed above as well as DPO and Identity Preference Optimization (IPO) on datasets that have certain biases as well as on an unbiased dataset. Results suggest that the newly proposed approaches improve performance over IPO and DPO.\n\n\nThe paper observes that", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Following proposed changes, if made, may significantly strengthen the work (\"critical\")\n- Please consider unifying portions of Section 3 and Section 6 as it pertains to analytical work that shows a form of degeneracy in DPO.\n- Consider updating caption for Figure 2 to include a small discussion on how a non-specialist can interpret the Y-axis values. ('Why is -0.5 better for \\rho = 0.2). I understand that there is a sentence in Page 9 in the draft but the figure may be easier to read if a clarifying sentence or two is included in the caption itself\n\nFollowing proposed changes, if made, may strengthen the work\n- Please consider empirical analysis with additional preference datasets. I am not an expert in post-training research but there appear to be additional datasets that is used to study distribution shifts, for e.g [1]\n- Why are other explicit reward model-based approaches not considered for comparison in the paper? A discussion would suffice in the paper.\n\n[1] Lin et al., https://arxiv.org/abs/2409.03650", "broader_impact_concerns": "I do not see a Broader Impact Statement in the paper which is fine.\n\nIf the authors feel that they want to share any concerns they are encouraged to do so . This work is very theoretical and as such needs no specific concerns to be noted in my opinion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HbxR2NECBr", "reviewer": "Reviewer_r5NJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper starts from rationale and clearly presented motivations. It shows comprehensive experimental setup and analysis that clearly demonstrate the method's efficacy. Finally, it provides strong theoretical backing that addresses and mitigates known issues with existing DPO techniques.\n\nThis is a good paper. Some minor weaknesses/questions are shown in the next part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a robust method of preference optimization that combines the simplicity of direct preference optimization with the benefits of explicit reward modelling. Key contributions include: (1) propose a reward model distillation approach to efficiently train language model policies offline, (2) theoretically and empirically demonstrate improved robustness to distribution shifts in preference data, and (3) introduce a pessimistic reward model distillation that optimizes against a family of reward models to handle uncertainty better.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The description of the reference policy presented in Section 3.1 lacks clarity and could benefit from a more detailed explanation. \n2. Could the authors clarify why, under Assumption 1, the ratio $\\frac{\\pi_{\\theta^*}(y) \\pi_{ref}(y')}{\\pi_{\\theta^*}(y') \\pi_{ref}(y)}$ tends towards infinity? Additionally, it would be helpful to discuss the prevalence of real-world datasets that meet Assumption 1 and the implications of this for practical applications.\n3. The impact of varying the coefficient $\\gamma$ should be studied in the experiment section or in the appendix. The manuscript mentions that both annealing and constant $\\gamma$ are shown in the experiments with e-DPO. Please refer the section near the text.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method that integrates reward model distillation with pessimistic optimization to address limitations in DPO. It aims to mitigate probability mass transfer issues and improve stability in preference learning. The experiments show the superioity of the proposed method over a few baselines. 
    A major concern from the reviewer QDvm is whether the relation to Gao et al. is clearly clarified. The authors explained that the submission focused on the offline setting and is quite different from Gao et al. After checking related works, I think the submission has a quite different motivation from Gao et al. and these two works can be viewed as concurrent works. In addition, another mentioned work Rafailov et. al. shall also be viewed as concurrent work. I think the submission can be accepted if the relation to these two works is clarified. The authors further modified the discussions on the relations to related works, and the reviewer QDvm is satisfied with the modifications. 
    Overall, I think the paper presents a meaningful contribution to RLHF and preference optimization. I recommend acceptance.
Decision: Accept (poster)