Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both

Authors: Abhijnan Nath, Changsoo Jung, Ethan Seefried, Nikhil Krishnaswamy
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: preference optimization, reward distillation, llms

Abstract:
Reward modeling of human preferences is one of the cornerstones of building usable generative large language models (LLMs). While traditional RLHF-based alignment methods explicitly maximize the expected rewards from a separate reward model, more recent supervised alignment methods like Direct Preference Optimization (DPO) circumvent this phase to avoid problems including model drift and reward overfitting. Although popular due to its simplicity, DPO and similar direct alignment methods can still lead to degenerate policies, and rely heavily on the Bradley-Terry-based preference formulation to model reward differences between pairs of candidate outputs. This formulation is challenged by non-deterministic or noisy preference labels, for example human scoring of two candidate outputs is of low confidence. In this paper, we introduce DRDO (Direct Reward Distillation and policy-Optimization), a supervised knowledge distillation-based preference alignment method that simultaneously models rewards and preferences to avoid such degeneracy. DRDO directly mimics rewards assigned by an oracle while learning human preferences from a novel preference likelihood formulation. Our experimental results on the Ultrafeedback and TL;DR datasets demonstrate that policies trained using DRDO surpass previous methods such as DPO and e-DPO in terms of expected rewards and are more robust, on average, to noisy preference signals as well as out-of-distribution (OOD) settings.

Decision: reject
Reviewer scores: 6, 5, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1729613842058

thread_type:
official_review

<thread_object>
creator: reviewer_g9er

<content>
summary:
This work seeks to addresses challenges that arise in DPO when ‘non-deterministic’ preferences are present in the dataset. The authors provide a theoretical motivation behind why these points are challenging and propose DRDO, Direct Reward Distillation and policy-Optimization to address this issue. They test DRDO on subsets of the Tldr and Ultrafeedback datasets.

strengths:
The proposal of the paper to combine a reward difference loss that measures the strength of preferences with a contrastive loss term that uses the binary preference labels from preference datasets is novel to the best of my knowledge. The authors show the proposed approach produces an improved win rate over a DPO and e-DPO baseline.

The paper conducts varied experiments to understand the proposed method. They vary the size and temperature of the generated responses to explore a wide range of possible settings and analyse the win rate using suitable approaches and prompts with GPT 4.0.

weaknesses:
A key component of the paper is the theoretical analysis of the weaknesses of DPO, specifically Lemma 1 and Lemma 2. The proof of these Lemma in the appendices seems to have multiple mistakes, for example the statement at the end of Appendix A.2 appears wrong as the latter equation should approach 1, this issue is repeated in the derivation of Lemma 2, both are key for later motivating the proposed DRDO. If this is a misunderstanding on my behalf, the quality of the explanation needs improvement as both Section 3 and the writing in the appendix was hard to parse. 

Furthermore, it appears that Lemma 1 simply re-states results found within the existing literature, issues with deterministic preferences in DPO i.e. Lemma 1a) and 1b) are presented in [1] and the result of Lemma 1c) appear in [2], both works are cited by this paper. 
The motivation of the paper is also very unclear to me. The authors define noisy preferences as those with $p(y \succ y’) \approx ½$ and claim that “This (DPO) formulation is challenged by non-deterministic or noisy preference labels”. In the proposed setting it appears that these non-deterministic points are just weak learning signals. The authors should justify why these points matter, for example, pluralistic approaches argue that these weak signals stem from multiple different viewpoints being present in the dataset and thus work to learn models unique to specific viewpoints [3,4]. An argument that motivates the proposed approach against the pluralistic setting would be beneficial.

[1] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences, 2023.

[2] Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation, 2024. URL https://arxiv.org/abs/2405.19316.

[3] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Amrit Bedi, and Mengdi Wang. 2024. MaxMin-RLHF: Alignment with Diverse Human Preferences. In ICML. https://proceedings.mlr.press/v235/chakraborty24b.html

[4] Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, et al. Conditioned language policy: A general framework for steerable multi-objective finetuning. arXiv preprint arXiv:2407.15762, 2024.

questions:
1.	In Figure 2, is the DRDO Preference Loss $L_{kl}$ or the Contrastive Log-unlikelihood?

2.	What does an ablation on the two loss terms look like? If trained just with the contrastive loss how does the model perform?

3.	Whilst the paper compares to e-DPO how does the approach compare to IPO an alternative to DPO mentioned heavily in the first few sections of the paper that does not have a reward oracle component?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730264902466

thread_type:
official_review

<thread_object>
creator: reviewer_pipq

<content>
summary:
Preference-based fine-tuning is known as one of the most effective methods to boost the generation ability of the modern large-language model (LLM). Direct Preference Optimization (DPO) provides a reasonable solution to this problem. However, the author points out that due to the design of the objective function and the non-deterministic preference label, the trained policy may degenerate, which can be treated as a kind of underfitting. To this, the author suggests DRDO (Direct Reward Distillation and policy-Optimization), which models rewards and preferences simultaneously. They adopt an oracle model that reflects the true preference and distillate its knowledge to a student model. Their method significantly outperforms the common baseline, and their analysis also matches the intuition.

strengths:
Their concern is quite intuitive and handles the fundamental limitation of PbRL. Especially, they mainly point out that the human's preference label may be both noisy and stochastic; hence, the trained model may be confused by the ambiguous update signal. They formulate their intuition into a mathematical lemma and verify its validity. The experimental gap is also significant, and it emphasizes the effectiveness of the proposed method.

weaknesses:
Even though their mathematical formulation and lemmas, their core algorithmic procedure is mainly written in natural language rather than pseudo-code, which diminishes the understandability of the overall process. On the extent of this problem, the components, techniques they adopted, and their main concerns are not harmonic. For example, to the best of my understanding, their key claim is that the 0-1 binary label fails to reflect the uncertainty of the human's preference, and therefore, the trained policy model can be either overfitted or underfitted. Consequently, they suggest to modify the objective function into a regression form rather than the MLE form. However, the concept of the oracle model, knowledge distillation, was suddenly mentioned. My understanding is that they train an Oracle preference model based on the given preference label, distillate its information to the student model, and use the student model for policy optimization. I am not sure how this idea handles the pointed limitation of DPO.

questions:
1. In the common sense of machine learning research, the term 'oracle' usually sounds impossible to achieve, only used during the upper bound experiment or thought experiment. I understand that you 'trained' an Oracle model from the given preference dataset. Is the coverage of the dataset enough to say your trained model is an oracle? Further, isn't your concern mainly built on the low data coverage assumption (Assumption 1)? Please provide more explanation on this part.

2. You said that because the dataset coverage is not infinite but limited, OOD works as a critical drawback to the PbRL framework. How does your framework handle this issue? What feature of knowledge distillation allows us to overcome the low data coverage?

3. I understand that your work mainly tries to handle the fuzzy, unambiguous label setting. Indeed, you split the dataset into a relatively fuzzy subset and a clear subset (hc,he & lc,le). However, the performance gap provided in Table 1 is pretty analogous, regardless of the subset partition. In my intuition, it indicates at least one of these: the edit-distance and human-confidence-based partitioning was not proper, or the performance gap comes from the other intuition rather than fuzzy-label handling. Would you provide more clarification or other experimental results on this point?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730487597328

thread_type:
official_review

<thread_object>
creator: reviewer_adch

<content>
summary:
The paper introduces Direct Reward Distillation and policy-Optimization (DRDO), a novel approach to preference alignment that combines reward modeling and policy optimization into a unified framework. Traditional methods like Direct Preference Optimization (DPO) can struggle with preference ambiguity, leading to suboptimal policies. DRDO addresses these issues by directly distilling rewards from an Oracle into the policy model while simultaneously learning from diverse preference signals. DRDO is evaluated on multiple benchmarks (Ultrafeedback, TL;DR, and AlpacaEval) and shows improved robustness, especially in handling noisy or non-deterministic preferences, outperforming popular methods like DPO and e-DPO.

strengths:
* Sound Theoretical Foundation: The paper provides a clear theoretical grounding, with detailed proofs supporting DRDO’s robustness to varied preference strengths.

* Experimental Quality: The experiments are well-designed, covering both deterministic and non-deterministic preference data, and include benchmarks that highlight DRDO’s superiority across multiple settings and model sizes.

weaknesses:
* A bit concern on novelty and motivation: this paper seems to combine the ideas from three papers: eDPO, Oracle RM(https://arxiv.org/abs/2406.10216), and Focal. The main motivation and contribution need to be better justified in order to be a high quality paper in ICLR.

* Out-of-distribution (OOD) concern: For OOD experiment, it is a bit unclear to me on what the authors want to address. There are a few distribution shifts involved in alignment: 1. prompt shift among reward model training data, rlhf training prompt, and rlhf evaluation prompt. 2. response shift between the reward model training data and on-policy generation. As far as I understand, the TL;DR dataset also contains a few CNN Dailymail examples. It will be helpful if authors can show how robust the proposed approach can address the OOD issue systematically. Say show a plot with x-axis (the amount of OOD) and y-axis (the performance gap between baseline and proposed approach).

* Additional complexity of the system: focal loss introduces additional two hyperparameters, it is challenging to tune them, especially for OOD. What evaluation dataset to use? How to make sure the evaluation dataset is not overfitted and represent a good coverage of OOD samples during testing?

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692095519

thread_type:
official_review

<thread_object>
creator: reviewer_ae9x

<content>
summary:
This paper introduces Direct Reward Distillation and Policy Optimization (DRDO). The method combines explicit reward modeling with policy optimization by training an oracle model to provide reward signals, then the signals are directly distilled into the policy model. Additionally, DRDO employs a loss function that integrates a reward difference component and a contrastive log-unlikelihood term, adjusting the learning process based on the model’s confidence in assigning preferences.

strengths:
1. DRDO avoids dependence on a reference model, which could offer computational benefits. 

2. In experiments, the inclusion of out-of-distribution settings (e.g., using CNN Daily articles) aligns with the claims of improved robustness.

weaknesses:
1. The paper lacks a clear definition of key variables such as $\hat{r}_1$ and $\hat{r}_2$, which are crucial for understanding the proposed loss function. This lack of detail impedes understanding of how the reward signals are computed and integrated into the training process.

2. The authors seem to overstate their contributions; the distillation loss they employ appears to be derived from prior work, specifically the paper “Robust Preference Optimization through Reward Model Distillation” by Fisch et al. (2024). Proper attribution and a clearer differentiation from existing methods are needed. 

3. Key concepts, such as the handling of ambiguous preferences and the role of the modulating focal loss component, require more elaboration. Terminologies like “reward difference” and “contrastive log-unlikelihood” could be better clarified to aid understanding.

questions:
1. How are $\hat{r}_1$ and $\hat{r}_2$ defined within the loss function? Providing clear definitions would enhance understanding of the method’s implementation.

2. Given that the distillation loss resembles that in Fisch et al. (2024), how does DRDO differentiate itself from this existing method?

3. Why were ablation studies on the focal loss component or comparisons across a broader range of datasets or evaluation metrics not conducted?

4. Considering the emphasis on practical efficiency, how does the computational overhead of DRDO compare to DPO or e-DPO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730803866196

thread_type:
official_review

<thread_object>
creator: reviewer_z2ia

<content>
summary:
The paper proposes an extension of Fisch et al 2024 work on Reward Distillation involving a regularization objective and demonstrate marginal improvements in terms of empirical benchmarks. There contributions are primarily two fold: 

1) When optimizing the reward model (Oracle in their paper) they add an additional regularization objective that ensures the Oracle still assigns high likelihood to the winning generation. 

2) The authors include a gradient unlikelihood objective, (which is further explained mechanistically in the appendix). This objective enhances gradients in the direction of the winning response when the winning response is small (and vice versa for the loosing response).

strengths:
1) The paper is on a very relevant area and proposes a simple extension to existing approaches for preference optimization. 

2) The empirical experiments are extensive and instill faith in the reviewer of the proposed algorithms merits. 

3) The explanation of the unlikelihood objective in Eq 6 in terms of the gradient updates helps clarify better what the proposed algorithm achieves mechanistically.

weaknesses:
A) Notation is a bit hard to follow in the paper, for example L_O represents the Oracle loss, but is there shared parameterization between the Oracle and the Policy? In either case, this should be explicitly notated in the paper. Similarly the variable alpha is shared between the equation 5 and 6, are these the same or different? It is important to be explicit about these differences.

B) The major contribution in the paper over Fisch et al comes from the unlikelihood term in Equation 6. However this is explained only in passing in the main text, if indeed the authors want the focus to be on the importance of this term.

C) My most major concern of the paper is that it is hard to decouple if the improvements come from i) Improved reward Distillation ii) The unlikelihood objective in Eqn 6) . I would be curious to see if replacing the first term in objective in Eq 6 with a standard DPO objective still results in an improved policy. If so, this framework would be much more generalizable.

questions:
A) In Eq. 5 is the \log(y_w) term \log(y_w|x) or \log(y_w, x) ? This could  have potential ramifications for the quality of the resulting Oracle. In situations where both y_w and y_l are reasonably close generations, would you not want to also penalize the Oracle to be close to \log(y_l|x) ? 

B) Was the oracle initialized from an model trained by SFT on the winning responses? This is an important question to understand the oracle regularization and its effects.

C) The proposed objective involves multiple additional hyperparameters, including \alpha in 5, \alpha in 6 as well as \gamma in 6. Can the author's comment on the sensitivity of the model performance to these additional parameters?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732332171190

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for taking the time to closely read our paper and post their feedback. We have responded in depth to each review and we look forward to engaging in the author-reviewer discussion in the coming days.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732462586368

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
As there are just a few days left on the discussion period, we hope the reviewers have had a chance to read our response, and we look forward to having a productive discussion with you in the remaining days.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732602934922

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of revisions and additional results for DRDO

comment:
We thank the reviewers for their feedback and insightful comments. In particular we are delighted that they found DRDO to be novel (reviewer **G9Er**), very relevant wrt preference alignment in LLMs (reviewer **z2iA**), theoretically and mathematically sound (reviewers **aDcH**, **z2iA**) and robustly evaluated (reviewers **aDcH**, **Pipq**, **z2iA**) with well-designed experiments across datasets. Here we briefly outline the changes made to the manuscript (in red font and including both additional experimental results and presentation fixes) and some crucial points brought up in the reviews. We hope that in the remaining time we will have the opportunity to address any remaining concerns and have a fruitful discussion. 

**Main paper**
- **Page 1 (introduction)**: added clarifications on DRDO motivation (in light of existing methods) as suggested by **G9Er**
- **Lemma 1a) (theoretical motivation in sec.3)**: changed lemma 1a statement to reflect correction as suggested by  **G9Er**
- **Non-deterministic preferences (page 4)**: clarified definitions of non-deterministic preferences in typical preference datasets vs flipped-label based noise  (**Pipq**)
- **Oracle/student policy details (sec. 4, page 6)**: clarified reward estimate computations using oracle/student policy including explicit parametrizations including minor change in eq. 5 suggested by **aE9X**

- **Constrastive log-unlikelihood (Fig. 2)**: replotted Fig. 2 with correction on y-axis label (**G9Er**)
- **OOD specification (Page 7)**: added a line clarifying why CNN/Daily Mail prompts constitute an OOD or out-of-domain settings for our experiments (**aDcH**)

**Appendix**
We have slightly rearranged the order in the appendix and added new sections for more clarity.

- **Appendix A.2**: corrected proof of lemma 1a and 1b (**G9Er**)
- **Appendix A.4**: corrected proof of lemma 2 (**G9Er**)
-**Appendix A.5**: added figure for DRDO complete pseudo-code/algorithm  (**Pipq**)
- **Appendix E.1 and Table. 8**: added DRDO component ablations results along with IPO baseline with a larger oracle (GPT-4o) (suggested by **z2iA**, **aE9X** and **Pipq**)
- **Appendix E.2 and Fig. 5**: added DRDO comparison with baselines (DPO and e-DPO) with extent of OOD generalization (**aDcH**)
- **Appendix F**: added clarifications and comment about DRDO’s computational overhead vs baselines (**aE9X**)
- **Appendix G**: added clarifications on DRDO’s motivation vs ensemble/linear interpolation based pluralistic preference learning (**G9Er**)
- **Appendix E.1 and Fig. 8**: added evaluation prompt using a larger oracle (GPT-4o) to resolve concerns regarding capacity and data–sufficiency in oracle  (**Pipq**).
- **Appendix H and Table 9 and 10**: added further results and analysis (KL-divergence and expected reward accuracies over SFT baseline) comparing DRDO’s $\gamma$ sensitivity vs DPO’s $\beta$ hyperparameter as suggested by **z2iA**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732894271808

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

As the discussion period has been extended until December 2nd, we would appreciate it if you could review our revisions and additional comments on the discussion board, and please let us know if and how we can address your questions further. We thank you for your reviews and comments to date and are confident that our revisions accordingly have resulted in a stronger paper and would appreciate your further engagement for the remainder of the discussion period.

Many thanks,
The authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper provides a new loss for simultaneous reward distillation and preference learning. The main paper is well-written and the experimental evidence is strong further motivating the proposal. However, the theoretical derivations are hard to follow, containing steps that are not verifiable. In particular, Reviewer G9Er worked through several of the derivations with the authors and helped them fix issues. However, unfortunately there are still remaining concerns (verified by the AC) in the proofs (as per the last comment of Reviewer G9Er). Given the issues in theoretical correctness, the paper unfortunately has to be recommended for rejection at this point. We hope that the authors can revise the paper using this feedback for a future venue.