Robust Preference Optimization through Reward Model Distillation

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:50:26

Abstract: Language model (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Recently, Direct Preference Optimization (DPO) has gained popularity as an offline alignment method that is directly supervised with human preference annotations. However, DPO is overconfident about preference annotations, implicitly assigning them rewards of infinite magnitude. This frequently leads to degenerate policies, sometimes causing even the probability of the preferred output to go to zero. In this work, we propose to use distillation to combat overconfidence: we train the LM to match the reward distribution induced by a model trained on the preference data. Moreover, to account for uncertainty in the reward model we are distilling from, we optimize against a family of reward models, which may be instantiated either implicitly or explicitly. Our results show that both measures lead to improved robustness to distribution shift in preference annotations, while preserving the supervised nature of DPO.

Main paper: 
 introduction: 
 Language model (LM) post-training (or alignment) aims to steer language model policies towards responses that agree with human preferences. Early state-of-the-art approaches have focused on reward learning from human feedback. In this paradigm, preference annotations are used to train reward models, which then guide the optimization of the language model policy through online reinforcement learning (an approach broadly referred to as RLHF). Recent research on offline "Direct Preference Optimization" [DPO; 23] and extensions thereof [3; 31] , however, has demonstrated that it is also possible to directly optimize policies on the preference data, which bypasses the need for a separate reward model-and its offline nature also leads to faster, and simpler, training frameworks.While this direct approach to preference optimization is attractive in terms of its simplicity and efficiency, it also raises important questions about the effectiveness and robustness of the resulting policies-as well as the broader utility of using an explicit reward model. In this paper, we argue that explicit reward modeling can, in fact, offer substantial practical advantages that are not captured by DPO's formulation. In particular, we theoretically show that relying solely on the preference data can be a precarious strategy, with few natural brakes in place to prevent policies trained under the DPO objective from careening off towards degenerate policies when the preference data exhibits certain idiosyncratic properties. On the other hand, explicit reward models can easily be regularized and understood-regardless of whether they are Bradley-Terry models [4] , margin-based ranking models [40] , or simply any other kind of function that correlates well with human preferences [31; 17] .Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.Taking a step back from pure direct preference optimization, we propose a method that merges the best of both worlds: an efficient reward model distillation algorithm that (i) operates effectively in the offline setting, (ii) makes minimal assumptions about the true, optimal reward we aim to maximize, and (iii) demonstrates greater robustness to the specific distribution of prompt/response data used for policy alignment. Drawing inspiration from prior knowledge distillation techniques [14; 26; 35; 10] , we leverage the same change of variables trick employed in DPO to express the language model policy in terms of its implicit reward model [23] . We then train the policy to match our desired, explicit reward via an L 2 loss that directly regresses the pairwise differences in target rewards for any two generation pairs (x, y 1 ) and (x, y 2 ). We theoretically establish the equivalence between optimizing this distillation loss over a sufficiently diverse offline dataset of unlabeled examples and optimizing the traditional online RLHF objective.Our reward model distillation approach, however, is not immune to some of the same challenges facing DPO-style learning of policies. In particular, reward model distillation requires having a reliable reward model-but having a reliable reward requires having a reliable method for extracting a reward model from a potentially noisy preference dataset. To address the uncertainty surrounding the "right" reward model, we introduce a pessimistic extension to our approach. This extension aims to maximize the worst-case improvement of our model across a plausible family of reward models (e.g., those sufficiently consistent with annotated preference data). This strategy aligns with that of existing work in conservative offline reinforcement learning [5; 16] . Interestingly, we derive that this pessimistic objective can be equivalently expressed and optimized by adding a simple additional KL-divergence regularization to the original distillation objective.Empirically, we find that reward model distillation, particularly pessimistic reward model distillation, leads to similar performance to prior direct preference optimization methods in settings where the preference datasets used are unbiased, but significantly better performance in settings where the preference datasets are biased, when compared to DPO and the Identity Preference Optimization (IPO) framework of [3] , which was introduced as a more robust alternative to DPO. To further support these empirical observations, we provide an extensive theoretical analysis that both (i) sheds more light on the degenerative tendencies of DPO and issues inherent to its objective, and (ii) highlights relative advantages of our explicitly regularized approaches. 
preliminaries: 
 We begin with a brief review of Direct Preference Optimization (DPO) [23] and its analysis. Proofs of all theoretical results provided here, and in the rest of the paper, are deferred to Appendix A.Let x be an input prompt, and let y ∼ π θ (• | x) be the language model policy π θ 's response to x.Given some reward function r * (x, y) and another reference policy π ref (y | x), the goal of alignment is to solve for the "aligned" policy π θ * (y | x) that maximizes the following RLHF objective, i.e.,EQUATIONwhere µ(x) is a fixed distribution over prompts, and the KL-divergence term prevents the aligned policy from being dramatically different from the anchoring reference policy, π ref (y | x). Here, the reward function r * is typically not known in advance, but rather inferred from collected human preference data in the form of (x, y w , y ), where x is the prompt, y w is the "winning", or preferred, response, and y is the "losing", or dispreferred, response. A common approach is to assume that pairs (y 1 , y 2 ) follow a Bradley-Terry model [4] , under which the probability that y 1 is preferred to y 2given the reward function r * and prompt x is p * (y 1 y 2 | x) = σ(r * (x, y 1 ) -r * (x,With r in hand, Eq. ( 1) can be optimized using standard reinforcement learning algorithms [27; 29; 6] .DPO is a simple approach for offline policy optimization that uses preferences to directly align the language model policy, without training an intermediate reward model. Specifically, DPO leverages the fact that the optimal solution to the KL-constrained objective in (1) takes the form [15] EQUATIONwhereZ(x) = y π ref (y | x) exp( 1 β r * (x, y))is the partition function. DPO reparameterizes the true reward function r * in terms of the optimal policy π θ * that it induces, i.e.,EQUATIONUnder the Bradley-Terry model, the likelihood that y 1 y 2 can then be written asEQUATIONwhere now π θ * can be directly estimated on D pref following the objective in (2) , in place of theEQUATIONAs argued in [3] , the Bradley-Terry assumption that DPO strongly relies on for maximum likelihood estimation is sensitive to the underlying preference data. Specifically, if we have any two responses y 1 and y 2 where p * (y 1 y 2 | x) = 1, then the Bradley-Terry model dictates that r * (y 1 )-r * (y 2 ) = +∞, and therefore π θ * (y 2 | x) = 0 for any finite KL-regularization strength β.We can illustrate this phenomenon on a broader level with the following example.Assumption 1. Suppose we are given a preference dataset of (context-free) pairs D pref = {(y w i , y i )} n i=1 , the pairs (y w i , y i ) are mutually disjoint in both the elements. Further suppose that we optimize the DPO objective on D pref with a single parameter θ y for each y.Proposition 1. Under Assumption 1, for any (y, y ) such that y = y w i and y = y i for some i, we have π θ * (y)π ref (y ) π θ * (y )π ref (y) → ∞, for all global minimizers π θ * of the DPO objective in (6), for any β > 0.Corollary 1. Under Assumption 1, further assume that 0 < π ref (y) < 1 for all y. Then π θ * is a global minimizer of the DPO objective in (6) iff π θ * (C(y ) c ) → 1 with π θ * (y w i ) > 0 ∀i ∈ [n],where C(y ) c is 
uncertainty-aware reward model distillation: 
 As discussed in the previous section, a core issue in preference optimization is that the true preference distribution p * (y 1 y 2 | x) is not known. Attempting to infer it from finite-sample preference data (that may further be biased or out-of-distribution with respect to the target domain) can then result in a failure to learn reasonable policies. In this section, we now propose an inherently regularized approach to direct preference optimization that uses uncertainty-aware reward model distillation.Suppose for the moment that the reward function r * was in fact known, and did not have to be inferred from sampled preference data. Under this setting, we can then define an efficient offline optimization procedure that is similar in spirit to DPO, but no longer relies directly on a preference dataset. Concretely, given unlabeled samples (x, y 1 , y 2 ) ∼ ρ (where the number of samples can be potentially unlimited), we can define a simple "distillation" loss, L distill (r * , π θ ), as follows:EQUATIONIntuitively, the distillation loss seeks to exactly match differences in reward model scores across all generation pairs (x, y 1 , y 2 ). It is then easy to see that under the Bradley-Terry model, this is equivalent to matching the strength of the preference relationship, y 1 y 2 . Furthermore, by only matching differences, we can still conveniently ignore the log partition term, log Z(x), in the implicit reward formulation for π θ as shown in (4), as it is constant across different y for any given x. Finally, similar to the motivation in DPO, we can show that minimizing L distill (r * , π θ ; ρ) indeed results in an optimally aligned policy π θ * , as long as the data distribution ρ has sufficient support. Further, let supp(ρ(x,y 1 , y 2 )) = supp(µ(x))×Y ×Y. Let π θ * (y | x) ∈ argmin π θ L distill (r * , π θ ; ρ)be a minimizer over all possible policies, of the implicit reward distillation loss in (7) , for which r * (x, y) is assumed to be deterministic, and finite everywhere. Then for any β > 0, π θ * also maximizes the alignment objective in (1).The above result holds for a broad class of data distributions ρ(x, y 1 , y 2 ), and makes no assumptions on r * (e.g., it is no longer necessary for it to be defined using a Bradley-Terry model). In fact, this result can also be seen as strict generalization of the IPO framework of [3] when taking r * (x, y) 1{y = y w }, if labeled pairs (x, y w , y l ) are provided instead of the unlabeled pairs (x, y 1 , y 2 ).Of course, the true reward r * is usually not known in practice. Still, as in standard RLHF, we can go about constructing good proxies by using the preference data to identify plausible target reward models r tgt -further guided by any amount of regularization and inductive bias that we desire. A natural choice is to first learn r tgt on the preference data D pref using standard methods, and then reuse D pref to distill π θ , which is similar to classical settings in teacher-based model distillation [14; 26] .Furthermore, as r tgt is a real-valued model, at a bare minimum it is guaranteed to induce a regularized Bradley-Terry preference distribution p tgt (y 1 y 2 | x) > 0, ∀x, y 1 , y 2 ∈ X × Y, and thereby avoid some of the degeneracies identified in §2.3 for the maximum likelihood estimate under DPO.Choosing a single reward model r tgt for anchoring the LM policy can naturally still lead to degenerate behavior if r tgt is a poor approximation of the true r * that accurately reflects human preferences.However, we can easily extend our framework to handle uncertainty in the right target reward function by defining a confidence set of k ≥ 1 plausible target reward models, S = r 1 tgt , . . . , r k tgt , and training π θ * (y | x) to maximize the following "pessimistic" form of the objective in (1):max π θ min r i tgt ∈S E µ(x) E π θ (y|x) [r i tgt (x, y)] -E π ref (y|x) [r i tgt (x, y)]advantage over baseline policyEQUATION) P β (S) • π 1 • π 2 • π 3 = π * θ • π SFT • π MLE KL(πSFT||π3) π SFT (y w ) = 0.3 π SFT (y ) = 0.3 π MLE (y w ) = 1.0 π MLE (y ) = 0.0 π 3 (y w ) = 0.5 π 3 (y ) = 0.05Figure 1 : A toy illustration of Theorem 2, which states that the optimal π θ * for (8) is the policy in P β (S) with the lowest forward-KL from π SFT . The set P β (S) contains a (potentially infinite) set of policies π 1 , π 2 , . . . corresponding to target reward models. Here, π SFT assigns equal mass to y w and y , π MLE is the MLE solution for the DPO objective, which puts all probability mass on y w , and π 3 is the policy in P β (S) with lowest forward-KL.In this pessimistic objective we are no longer optimizing π θ for a single reward, but optimizing π θ to produce generations that are scored favorably on average, even by the worst-case reward model in the set S, relative to the generations of the baseline policy π ref .When the set S = {r * } consists of only the ground-truth reward, the objective ( 8) is equivalent to standard RLHF (1), up to a constant offset independent of θ. More generally, whenever S includes a good proxy r for r * , the pessimistic advantage evaluation ensures that the the policy π * θ that maximizes eq. ( 8) still has a large advantage over π ref under all r ∈ S, including r. This use of pessimism to handle uncertainty in the knowledge of the true reward is related to similar techniques in the offline RL literature [16; 5] .For the objective to be meaningful, the set S has to be chosen carefully. When S is small, it might not include any good proxy for r * . Conversely, if S is too rich, it forces π θ * to be nearly identical to π ref , since any deviations from π ref might be penalized by some reward model in S. Consequently, we want to design S to be the smallest possible set which contains a reasonable approximation to r * .To optimize (8) , it turns out that we can formulate it as an equivalent constrained offline optimization problem, that we will show to conveniently admit a similar loss form as (7) .Theorem 2 (Pessimistic distillation). Define the constrained minimizerEQUATIONwhere P β (S) is the set of all possible policies with implicit reward models that are consistent with any target reward model r i tgt ∈ S, i.e., P β (S){π θi } |S| i=1 where π θi ∝ π ref (y | x) exp 1 β r i tgt (x, y).Then for any β > 0, π θ * also maximizes the pessimistic alignment objective in (8) .To unpack this result, Theorem 2 stipulates that the π θ that maximizes the pessimistic objective in (8) is the policy in P β (S) that is closest in forward KL-divergence to π ref (see Figure 1 ). 1 In addition, this policy also maximizes the expected reward of one of the r i tgt ∈ S (minus the additional weighted reverse KL-divergence penalty term). Intuitively, the forward KL-divergence term serves the role of biasing the model towards optimizing for reward models that are similar to the implicit reward that π ref already maximizes. Otherwise, there might exist a target reward model r i tgt ∈ S for which the advantage of π θ relative to π ref will be low, or even negative (a solution that we would like to avoid).The constraint in (9) can then be relaxed and approximately optimized by introducing an objective with a Lagrangian-style penalty with strength α > 0 on a form of distillation loss as (7) , i.e., minEQUATIONwhere in practice we divide by α and instead optimize2 EQUATIONwhere γ = βα -1 . In reality, minimizing (11) for γ > 0 is equivalent to solving the constrained optimization problem in ( 9) with an implicitly larger set of possible reward models S γ ⊇ S indexed by γ. More specifically, S γ also contains all reward models r that are approximately consistent with the anchoring reward models r i tgt contained in S, as the following result states.Proposition 2 (Soft pessimistic distillation). Assume the same conditions as Theorem 1. Then for any 0 < γ < ∞, there exists a λ ≥ 0 such that π θ * (y | x) ∈ argmin π θ L pdistill (S, π θ ; ρ), where π θ * is a minimizer over all possible policies, is a solution to (9) for the effective reward model setEQUATIONAs a result, optimizing (11) even when using the singleton S = {r tgt } yields an implicitly pessimistic objective, in which the pessimism is over all reward models r that are consistent up to λ with r tgt .We can also observe that Proposition 2 can be leveraged to obtain an alternative, implicitly pessimistic, objective that uses DPO directly instead of distillation. Consider the following regularized DPO loss:EQUATION)Following a similar analysis as in Proposition 2, we can derive that this implicitly corresponds to maximizing the pessimistic objective in ( 8) for the reward model setEQUATIONwherer π θ (x, y) β log π θ (y | x)/π ref (y | x) + β log Z(x)is the implicit reward model defined by π θ . S γ then corresponds to the set of reward models r π θ that are all approximate minimizers of the DPO loss. This not only includes the MLE, but also all other estimators that obtain nearly the same loss. In principle, this can be expected to help ameliorate some of the issues of §2.3: since driving the reward to ±∞ only marginally decreases the L dpo loss past a certain point, the set S will also include finite reward functions |r π θ (x, y)| < ∞ for any γ > 0. These rewards would then be preferred if they induce a policy with a smaller (forward) KL-divergence to π ref than the degenerate, infinite rewards. 
experimental results: 
 The main motivation for reward distillation and pessimism is to increase alignment robustness in challenging settings where it is difficult to learn good policies directly from the preference data. To demonstrate the effectiveness of our approach, we run experiments on the popular TL;DR summarization task [29; 32] , in which we simulate a scenario where the preference data has a spurious correlation between the length of a summary and whether or not it is preferred. 3We first train an "oracle" reward model on the TL;DR preference data training set [29] and relabel all preference pairs with this oracle. This enables us to use the oracle reward model for evaluation, without worrying about the gap to true human preferences. After relabeling, longer responses (where longer is defined as y 1 having at least 10% more tokens than y 2 ) are preferred in 61% of the examples.To test the effect of a spurious correlation on preference-based policy optimization, we select as a training set 30K examples from the relabeled data such that the longer output is preferred in ρ fraction of examples, with ρ ∈ {0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}. Each such training set is denoted D ρ . At each D ρ , we compare our approach to DPO [23] and IPO [3] , which are currently the most commonly used offline alignment methods. We test the following variants of distillation and pessimism:• of pairs where the longer response is preferred is b, and train reward models r ρ,b on those subsets.Consequently, sensitivity to length should vary across ensemble members. We then apply the same procedure as pd-DPO above, with a confidence setS ρ = {r ρ,b } B b=1 .All reward models and policies are initialized from Palm-2-XS [2] . Policies also go through a supervised finetuning step on human-written summaries from the original TL;DR training set [32] prior to alignment, and we term this policy π SFT . We evaluate performance by sampling summaries for test set prompts, evaluating the average reward according to the oracle reward model, and computing the advantage in average reward compared to π SFT (before alignment). We train policies for 10 4 steps with batch size 16 and learning rate 10 We present the results of our experiment in Figure 2 . As can be seen in the plot, the more challenging setting is when ρ < 0.5, which corresponds to a sample of preference annotations in which shorter outputs are generally preferred. This distribution shift is more difficult because as mentioned the oracle reward model (trained on human annotations) has a bias in favor of longer outputs [28] . Nevertheless we get sizable improvements compared to the reference policy π SFT for all length bias values. To better understand the utility of reward ensembles in e-DPO, in particular when ρ < 0.5, we examine the role of each reward model in the ensemble across different biases. Specifically, given the final e-DPO policy per length bias, for each example we identify the reward model r ρ,b that best matches the implicit reward of this policy, i.e., for which reward model is L distill minimized on that example (see Eq. ( 7) and ( 11)). We find that when the policy is trained on data where shorter preference are preferred (ρ < .5), the reward model that best matches the policy often has the opposite bias (b is high), and vice versa. Thus, the success of e-DPO may be explained by its ability to distill from reward models that do not suffer from the bias in the policy training data, which is particularly helpful when ρ ≤ .5 as this bias is also not shared by the oracle RM. We provide the full distribution over reward models for all ρ and β in App. C. Overall, these results demonstrate the efficacy of training a policy by distilling from a reward model in the presence of distribution shifts, and that a careful design of an ensemble to mitigate spurious correlations can lead to further performance gains. 45 Theoretical analysisThis section characterizes problems with the DPO objective and solutions offered by pessimistic DPO and distillation, focusing on the simplified scenario in which we optimize with respect to a single preference pairs (y w , y ). Once again, all proofs are deferred to Appendix A.In its Lagrangian formulation, pessimistic DPO adds a forward KL term to the DPO objective ( §3.3).For the sake of analysis, we assume that the preference annotations are sampled from the reference distribution, µ(x) × π ref (y | x) × π ref (y | x).i ) = -1 β log (α -1) + log πθ (y w i ) + log π ref (y i ) π ref (y w i ) .The optimum in Proposition 3 corresponds to log ψ w /ψ = β -1 log(α -1). Recall that IPO seeks to assign a constant value to this ratio by minimizing (log ψw ψτ -1 ) 2 ; the (unconstrained) optima are identical for τ -1 := β -1 log(α -1), but the loss surfaces are different (see Appendix B). DPO sets π θ (y i ) → 0, as shown in Corollary 1; this is due not only to competition from π θ (y w i ) but from DPO penalizing positive probability on y i . Analysis of the distilled loss gives a similar result:Proposition 4. For any fixed πθ (y w i ) and β > 0, the argmin of the distilled DPO objective (eq. ( 7))is min(1πθ (y w i ), πθ (y i ), with log πθ (y i ) = 1 β (r t (x, y i )r t (x, y w i )) + log πθ (y w i ) + logπ ref (y i ) π ref (y w i ) .While the setting is simplistic, the results are comforting: here the additional regularization effects of both distillation and pessimism (in the case of p-DPO) clearly help to avoid degenerate optima.Why DPO can drive π(y w ) to zero. In §2.3 we pointed out a peculiarity of the DPO global optima:in certain cases, it can include policies where π(y w ) may be nearly 0 for all y w in the training set. This undesirable behavior has also been observed in practice [20; 22; 30] . For intuition on why this may happen, consider the simplified case where the policy is a bag-of-words model, π θ (y) ∝ exp (c(y) • θ)for c(y) representing a vector of counts in y and θ i representing the unnormalized log-probability of token i. Then we can formally show that DPO optimization monotonically decreases an upper bound on the probability of the preferred completion, πθ (t-1) (y w ) ≥ πθ (t) (y w ) ≥ π θ (t) (y w ).Proposition 5. Let y w , y ∈ V n be preferred vs. dispreferred outputs of length n, withπ ref (y w ), π ref (y ) > 0 and corresponding count vectors c(y w ), c(y ). Let log π θ (y) = c(y) • θ - nZ(θ) for Z(θ) = logV i e θi , with upper bound log πθ (y) = c(y)•θ -n max j θ j . Let θ (t) represent the parameters of π after t steps of gradient descent on L dpo ({y , y w , x}), with θ (0) = 0. Then π θ (t) (y w ) ≤ πθ (t) (y w ) ≤ πθ (t-1) (y w ) for all t.Where does the probability mass go? If π θ (t) (y w ) decreases in t, what other strings become more probable? In the following proposition, we show that under the bag-of-words model, DPO optimization moves probability mass away from y w to sequences that contain only the tokens that maximize the difference between y w and y . This is a concrete example of the type of undesirable optima described in §2.3, now shown here to be realizable. Then π θ (t) (y w )π θ (t) (ŷ) = τ (t)k for some k ≤ 0 and some non-decreasing τ : Z + → R + .We have k = 0 when c(y w ) = c(ŷ), and k 0 when ||c(y w )|| 2 ||c(ŷ)|| 2 = n (dense c(y w )) and ||∆|| 2 = ||∆|| ∞ (sparse ∆). This implies that when y w and y are similar, π θ (y w ) will degrade more rapidly. Early stopping will therefore tradeoff between reaching the degenerate solution on such cases, and underfitting other cases in which y w and y are more distinct. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Experiments are on publicly-available data, but it is not possible for us to share code. We believe that the implementation should be relatively straightforward, given the mathematical descriptions presented here.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Recent work in offline alignment has focused on DPO [23] as a simpler alternative for aligning language models from preference data. Subsequent work has identified issues with DPO, including weak regularization [3] and a tendency to decrease the probability of winning generations during training [20] . Other methods have explored various avenues for improvement. These include analyzing the impact of noise on DPO alignment [11] , proposing to update the reference policy during training [12] , and suggesting a variant of IPO with a per-context margin [1] . Additional research has focused on token-level alignment methods [38; 22] and on developing a unified view of various offline alignment methods [31] . This work builds upon several these findings, and provides further analysis, as well as a solution based on pessimism and reward distillation.While offline alignment methods are popular, recent evidence suggests that online alignment methods such as RLHF [6; 29] , may lead to more favorable outcomes [13; 30; 8; 34] . Notably, Zhu et al. [41] proposed iterative data smoothing, which uses a trained model to softly label data during RLHF.Whether online or offline, however, policies are still succeptible to overfitting to certain degenerate phenomena. To this end, reward ensembles have been widely investigated recently as a mechanism for tackling reward hacking in RLHF [9; 7; 39; 25] , and in the context of multi-objective optimization [19; 24] . We use an ensemble of rewards to represent the uncertainty with respect to reward models that are suitable given preference data. Moskovitz et al. [19] focus on "composite" rewards, with the goal of achieving high task reward while ensuring that every individual component is above some threshold-also by applying a Lagrangian relaxation. In this work, we also consider multiple reward models, but we only focus on cases where there is no known, obvious reward decomposition.Finally, the question of using a small amount of offline data to learn high-quality policies, instead of online access to reward feedback, has been widely studied in the offline reinforcement learning (RL) literature. The predominant approach here is to use pessimism, that is, to learn a policy with the highest reward under all plausible environment models consistent with the data, with an extensive theoretical [18; 37; 33] and empirical [16; 5; 36] body of supporting work. The key insight in this literature is that without pessimism, the RL algorithm learns undesirable behaviors which are not explicitly ruled out in the training data, and pessimism provides a robust way of preventing such undesirable extrapolations, while still preserving generalization within the support of the data.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: These details are provided in Section 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 LM alignment is crucial for deploying safe and helpful assistants, but is difficult due to lack of access to perfect preference oracles. We presented a thorough theoretical analysis of some of the degeneracies that DPO is susceptible to when learning from sampled human preference data.Furthermore, our findings suggest that explicit reward modeling remains a powerful vehicle for introducing regularization into post-training. By distilling the reward assigned by a single, explicit reward model-or a family of explicit reward models-directly into the implicit reward maximized by our policies using offline data, we demonstrated that we can achieve improved robustness to variations in preference dataset quality, while maintaining the simplicity of the DPO framework.Limitations. The empirical results in the paper are based on one dataset and form of distribution shift.For deeper understanding of pessimism and ensembling, additional settings should be explored. The theoretical aspects of the paper are sometimes based on restrictive assumptions and simplifications.Nonetheless, they provide potential explanations for phenomena observed in real-world settings.Broader impact. We introduce new ideas to the active field of research on preference-based posttraining, which we hope will help facilitate the alignment of large models, and improve understanding of current approaches-ultimately supporting the development of capable and reliable AI systems.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Section 4.2 includes bootstrap 95% confidence intervals on the main figure and hypothesis tests for specific comparisons between methods.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The research does not involve human subjects and does not introduce new data.Its main impact should be to improve effectiveness and understanding of preference-based post-training.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Section 7Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: No data or models are released. 
 
 Appendix: A.1 Proof of Proposition 1Proof. Since all the preference pairs (y, y ) are mutually disjoint, and θ y is specific to each y, the DPO objective over D pref is convex in ∆ = {∆ 1 , . . . , ∆ n }, whereEQUATION)Furthermore, the different ∆ i are completely independent from each other due to the preference pairs being disjoint, so they can be optimized over separately.In particular, for every i we have thatEQUATIONwhich implies that ∆ * = {∞} n is the unique global minimizer of the DPO loss over D pref in the space of ∆'s, and any θ * that is a global minimizer must therefore satisfyEQUATIONA.2 Proof of Corollary 1Proof. Following the same argument of the proof of Proposition 1, we have that all global minimizers θ * of the DPO satisfy ∆ * i = ∞, which in turn implies thatEQUATIONSince π ref (y) is assumed to satisfy 0 < π ref (y) < 1 for all y, this implies that all θ * satisfyEQUATIONwhich further implies that π θ * (y i ) = 0 and π θ * (y w i ) > 0 for all i ∈ [n], as π θ * (y w i ) ≤ 1 for any y w i .AggregatingEQUATION)To prove the converse, let π θ be a policy that satisfies π θ (C(y) c ) = 1, with π θ (y w i ) > 0, ∀i ∈ [n],.As π θ (y) ≥ 0 for all y, this implies that π θ (y i ) = 0 ∀i ∈ [n]. Then, we haveEQUATIONwhich by Proposition 1 implies that π θ is a global optimum.Proof. We know that the optimal policy for the RLHF objective (1) is given by π θ * (y|x) ∝ π ref (y|x) exp(r * (x, y)/β). Plugging this policy into the distillation objective (7) , we see that L distill (r * , π θ * , ρ) = 0 for all ρ. In fact, the loss is equal to 0 pointwise, meaning that π θ * is a global minimizer of the distillation objective (7) . Further, let π be some other minimizer of L distill (r * , •, ρ). Then π also has to attain a loss of 0 at all (x, y, y ) in the support of ρ, meaning that log π(y|x)log π(y |x) = log π θ * (y|x)log π θ * (y|x) for all (x, y, y ) in the support of ρ.Consequently, the two policies coincide in the support of ρ (due to the normalization constraint, there is no additional offset term allowed as the support of ρ covers all of Y). Finally, noting that the support of the chosen ρ is such that π θ * puts no mass outside its support due to the KL constraint in (1), we complete the proof.A.4 Proof of Theorem 2Proof. Consider the pessimistic objective:EQUATION)As it is linear in r tgt and convex in π, we can switch the order of min and max:EQUATIONNote that every r tgt ∈ S can be written in terms of the KL-constrained policy π * rtgt it induces, i.e.,EQUATIONwhereEQUATIONwhich has the formEQUATIONwhere Z(x, r tgt ) is the partition function:EQUATIONSubstituting π * rtgt in for max π and writing r tgt in terms of π * rtgt , we get the simplified objectiveEQUATION-βD KL (π * rtgt π ref | x) = min rtgt∈S β E µ(x) D KL (π * rtgt π ref | x) + D KL (π ref π * rtgt | x) -D KL (π * rtgt π ref | x) = min rtgt∈S βE µ(x) D KL (π ref π * rtgt | x) .Proof. The proof is a standard Lagrangian duality argument, which we reproduce here for completeness. For two functions f (z) and g(z), let us defineEQUATIONLet us also consider the constrained problemEQUATIONSuppose by contradiction that z * is not a minimizer of (31) . Since z * is feasible for the constraint by construction, we get that f (z ) < f (z * ). Consequently, we further havef (z ) + αg(z ) < f (z * ) + αg(z * ),where the inequality follows from the feasibility of z in (31) . This contradicts the optimality of z * in (30) , meaning that z * must be a minimizer of (31) . Applying this general result withf = βE µ(x) D KL (π ref (y | x) π θ (y | x)), g = min r i tgt ∈S L distill (r i tgt , π θ ; ρ), and z = π θ completes the proof, since we recognize the set S γ in ( 12) to be equivalent to r i tgt ∈S L distill (r i tgt , π θ ; ρ) ≤ λ.Proof. We differentiate L pdpo with respect to ψ = π θ (y )/π ref (y ) with i implicit, obtaining,EQUATIONwhich is zero when,EQUATIONEQUATIONEQUATIONEQUATIONBy the second-order condition, the critical point is a minimum. The objective L pdpo is the sum of two components: the negative log sigmoid term for L i and the negative log probability for Ω. Because each component is a convex function of ψ i , so is L pdpo . As a result, the local minimum log πθ (y ) is also a global minimum.Proof. This follows directly from differentiating eq. ( 7) with respect to π θ (y 2 ). Then L dpo =log σ (β(∆ • θ) + β log ρ) . The derivative with respect to θ is, increase quadratically as we move away from the target. We find empirically that pessimistic DPO is robust to the transitive closure of preference annotations in the multi-arm bandit setting, as shown in EQUATIONValidation set performance across the range of hyperparameter settings is shown in Figure D.1. In pilot studies we found that these results were relatively robust to variation in the random seed, but did not conduct extensive investigation of this effect across all methods and hyperparameters due to cost.We train policies on 32 TPU v3 chips and reward models on 16 TPU v3 chips. We obtain roughly 0.1 steps per second when training, for both the policy and reward models. • If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Note that the objective in (9) minimizes the forward KL-divergence DKL(π ref (• | x) π θ (• | x)) even though the pessimistic objective in (8) is regularized with reverse KL-divergence DKL(π θ (• | x) π ref (• | x)).In practice, we compute and optimize the min over reward models per each mini-batch of examples.Length has been repeatedly shown in the past to correlate with reward[28; 21].We also experimented with an ensemble where members are different checkpoints across training of a reward model on the preference data and did not observe any empirical gains from this form of ensemble. 


Decision: This paper addresses the limitations of DPO in value alignment by proposing a reward model distillation approach. The authors present a method that trains LMs to match the distribution from a reward model to improve robustness and performance, especially in biased preference datasets. Their approach also includes a pessimistic extension to handle uncertainty in reward models. The experiments additionally demonstrate the superiority of the method presented in this paper.

However, some concerns need to be addressed in this paper. All of the reviewers y8px and wkMh suggested that this paper currently lacks a comparison between offline methods and online methods. We recommend the authors compare the methods in this paper with more online methods, such as PPO, Iterative DPO [1,2], and other ensemble strategies. If the authors could demonstrate that these models outperform these methods or require less complexity in training to achieve comparable performance with these methods, they could expand the method's application scope and impact. Moreover, the methods should really be tested on various datasets with multiple popular models, such as Llama-3, Gemma-2, and so on. With these concerns, I believe the current manuscript is not ready for publication and I recommend rejection.

[1] Xiong, Wei, et al. "Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint."

[2] Yuan, Weizhe, et al. "Self-rewarding language models."

Reviews: review: paper_summary: paper_summary: The authors discuss and give formal results on the limitations of DPO that have been observed in practice, and investigate reward model objectives for 1) distilling reward differences into the generator (eq. 7), and 2) pessimistic "minimax" distillation over a family of reward models, to mitigate these limitations. The theoretical equivalance of the "forward" and "reverse" pessimistic formulations (eq. 8 and 9) of the standard RLHF objective (eq. 1) is shown, and an approximate pessimistic objective (eq. 10) with a minimum over distillation losses for the reward model family considered in a Langrangian term. Results on TL;DR preference data that is biased to varying degrees to prefer long responses (Figure 2) shows good results for distilled models generally, and that pessimistic optimization over a family of reward models  optimized for varying response lengths improves results in irregular settings (i.e. when shorter responses are preferred).
strength_weakness: strength_weakness: - The results, as the authors acknowledge, are currently quite limited. These methods should really be tested on at least one additional preference task, and for results utilizing multiple models, pessimism should be compared with other basic ensembling strategies. 
- The gamma parameter is annealed during training, suggesting that the setting is important and perhaps sensitive. Some ablations and discussion around this seems prudent.
questions: questions: See above sections.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper addresses the limitations of DPO in LM alignment by proposing a reward model distillation approach. DPO, while efficient, often leads to overconfident and degenerate policies due to limited preference annotations. The authors introduce a method that trains LMs to match the distribution from a reward model, improving robustness and performance, particularly in biased preference datasets. Their approach also includes a pessimistic extension to handle uncertainty in reward models.
strength_weakness: strength_weakness: The main concern about this paper is the evaluation. As discussed in Section 2.3, DPO can shift nearly all probability mass to responses that never appear in the training set, also called OOD extrapolation in other papers. This issue arises when there are few annotations per instance (x, y1, y2). Therefore, it is expected that the authors should demonstrate how their proposed approach mitigates this problem, maybe by presenting the log-probabilities of the winning and losing responses. Specifically, the log-probability margin between winning and losing responses should not keep increasing (I assume this is true?).

Additionally, I can see the paper presents results showing robustness against distribution shifts in preference data. But what is the factor that makes the proposed algorithm learn the policy whose underlying preference distribution closer to the true preference distribution in biased setting? I would be willing to increase my score if these concerns are solved.
questions: questions: Refer to weaknesses.
limitations: limitations: I have no concerns about the limitations.
paper_summary: paper_summary: Based on the analysis of the shortcomings of DPO (Section 2.3), the authors proposed a simple reward distillation approach (Section 3.1) to align language models and a pessimistic variant (Section 3.2). These approaches outperform the vanilla DPO.
strength_weakness: strength_weakness: I think that this paper is technically well done. However, it has a limited application scope.

The authors aim to address the shortcomings of offline alignment methods like DPO, while maintaining an offline approach. However, online alignment methods such as PPO or Reinforce-style algorithms naturally alleviate these offline learning issues. This raises a question: Given the authors' findings on the limitations of offline alignment, why not adopt an online alignment method like PPO directly?

Admittedly, the distilled DPO method and its pessimistic variants are less complex than PPO. However, these proposed offline variants introduce additional complexities to vanilla DPO by requiring separate reward models and separate training phases. This makes them more akin to online methods. Furthermore, the experiment section lacks a comparison against online method baselines, leaving it unclear if the distilled-DPO variants offer a better performance--compute tradeoff compared to vanilla online methods like PPO.

To be clear, the paper offers intriguing insights. However, these techniques seem niche. They benefit those who prefer robust generalization in preference optimization without using online data, an assumption not very well communicated in the paper.

In my opinion, there are ways to increase the method's applicability. While the proposed distillation approaches are positioned as offline methods, to my understanding, they are also applicable to an online regime. For example, in Equation (7), instead of sampling $(x, y_1, y_2)$ from an offline dataset, as the authors currently do, we can sample $x$ from an offline dataset and then sample $(y_1, y_2)$ from the model in training. The learned models, in this way, are directly comparable to those learned via online alignment methods like PPO, but in a simpler approach than these online baselines. If the authors could demonstrate that these models outperform PPO or require less complexity in training, they could expand the method's application scope and impact.
questions: questions: As mentioned in the "Weaknesses" section above, does it make sense to apply reward distillation to online policy optimization? Does this compare favorably with online policy optimization baselines, like PPO?
limitations: limitations: See "Weaknesses" section above.

Comments: title: Response
comment: A quick thank you to the reviewer for reading our author response, and for adjusting their score! We are happy to see that the reviewer believes that our work offers valuable insights to the community. We will be sure to include the new results presented here in the paper.
We thank the reviewer for thoroughly reading our author response, and for adjusting their score. We also appreciate and will consider the additional suggestions for improving its contribution. We will certainly incorporate a clear discussion of the computational benefits of our approach in a revision to the main paper.
I appreciate the authors for their response. I especially appreciate the authors' discussion regarding their method's computational complexity:

> The additional overhead of reward distillation is small, and rewards for the training data can be computed offline (once) ahead of time. This allows training to be nearly identical to the standard variants, modulo the structure of the data that is fed in. This is true regardless of the size of the policy that is trained, or its hyperparameters: the reward data is generated once, for a finite (and completely parallelizable) dataset. We therefore respectfully, but strongly, disagree that the proposed method is more akin to online methods, where the reward models score generations sampled online from an ever-shifting distribution.

This corrects my misunderstanding, leading me to adjust my score. I suggest the authors include a similar discussion like this in the paper, emphasizing the overhead is minimal.

The paper currently lacks a comparison between offline methods (like DPO and the authors' approach) and online alignment variants (such as PPO), which I think is a limitation. I understand that incorporating such a comparison within the short rebuttal period might be unrealistic, but I think including it could significantly enhance the paper's impact. Indeed, the core argument of the paper is to address the idiosyncrasies of implicit reward behavior. Since online methods like PPO don't involve implicit rewards, and as the authors noted, "explicit reward models [in offline methods] can easily be regularized and understood," a comparison would seem beneficial. As a reader, I anticipate this comparison. An imperfect analogy might be a paper proposing a vision transformer model robust to shift-translation but neglecting to compare it with a conventional CNN, which is likely to possess this property by design.
Thank you to the authors for their thorough rebuttal. My major concerns have been addressed, and with the inclusion of these results in the paper, I believe this paper offers valuable insights to our community. I have accordingly raised my score.
Thanks again for your hard work in reviewing our paper! We hope that you've had a chance to read our reply to your comments. If they've addressed your concerns, then we hope that the reviewer can consider increasing the score. If not, please let us know if you have any further questions or concerns. We are committed to ensuring that our contributions are clearly communicated.
Thanks again for your hard work in reviewing our paper! We hope that you've had a chance to read our reply to your comments. If they've addressed your concerns, then we hope that the reviewer can consider increasing the score. If not, please let us know if you have any further questions or concerns. We are committed to ensuring that our contributions are clearly communicated.
Thanks again for your hard work in reviewing our paper! We hope that you've had a chance to read our reply to your comments. If they've addressed your concerns, then we hope that the reviewer can consider increasing the score. If not, please let us know if you have any further questions or concerns. We are committed to ensuring that our contributions are clearly communicated.