UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function

Authors: Zhichao Wang, Bin Bi, Can Huang, Shiva Kumar Pentyala, James Zhu, Sitaram Asur, Claire Na Cheng
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Alignment, Unified Alignment, RLHF, PPO, DPO, KTO

Abstract:
An LLM is pretrained on trillions of tokens, but the pretrained LLM may still generate undesired responses. To solve this problem, alignment techniques such as RLHF, DPO and KTO are proposed. However, these alignment techniques have limitations. For example, RLHF requires training the reward model and policy separately, which is complex, time-consuming, memory intensive and unstable during training processes. DPO proposes a mapping between an optimal policy and a reward, greatly simplifying the training process of RLHF. However, it can not take full advantages of a reward model and it is limited to pairwise preference data.

In this paper, we propose \textbf{UN}ified \textbf{A}lignment (UNA) which unifies RLHF/PPO, DPO and KTO. Firstly, we mathematically prove that given the classical RLHF objective, the optimal policy is induced by a generalize implicit reward function. With this novel mapping between a reward model and an optimal policy, UNA can 1. unify RLHF/PPO, DPO and KTO into a supervised learning of minimizing the difference between an implicit reward and an explicit reward; 2. outperform RLHF/PPO while simplify, stabilize, speed up and reduce memory burden of RL fine-tuning process; 3. accommodate different feedback types including pairwise, binary and scalar feedback. Downstream experiments show UNA outperforms DPO, KTO and RLHF.

Decision: reject
Reviewer scores: 3, 5, 3, 3

Reviews and discussion:
<thread>
creation date:
1729068664577

thread_type:
official_review

<thread_object>
creator: reviewer_qkms

<content>
summary:
This paper introduces a method called Unified Alignment (UNA), which aims to unify existing alignment techniques like RLHF, DPO, and KTO. The authors argue that RLHF, though effective, suffers from high complexity and instability, while DPO simplifies the process but is limited in certain aspects, such as not fully leveraging the reward model and being constrained to pairwise preference data. The proposed UNA framework claims to generalize and improve upon these methods by creating a novel mapping between reward models and optimal policies, allowing it to handle different feedback types and improve performance across downstream tasks. The paper suggests that UNA can simplify, stabilize, and speed up fine-tuning processes while reducing memory requirements, and it reports outperforming RLHF, DPO, and KTO in experiments.

strengths:
It is of interest that the authors try to narrow the gap between the RM-based and RM-free alignment methods.

weaknesses:
The paper is not solid enough for me and the contribution is limited. The reasons are concluded as follow:

First, It is not clear to me why the proposed method differs from DPO, thus the first contribution claimed is not novel at all. In my understanding, the DPO has already proved the equivalent between the optimal policy and the reward model's objective (under the BT model setting). Besides, it is also very confusing why the authors introduce f(x) and c.

Second, for the claimed contribution 2, the author used a objective introduced by DPO model to conduct a supervised learning, which is also confusing. What is the difference between using the proposed method to train an additional model and directly using a well-trained DPO model? And what if the DPO model is not well-trained, then the label would be very noisy? It is a very confusing step and I am open to more discussion.

Third, the proposed method is compared to some baselines and is claimed to be better, however I believe the comparisons are not fair and the experimental results are suspicious.  In Table 1, the authors use 4 UNA variants and only use KTO and Mistral as the baselines, and the best scores in each benchmark are distributed to all 4 variants, so which one is the best? The authors also claim UNA perform better than RLHF while they didn't provide the experimental detail about RLHF. The proposed method is not even a RL algorithm since it doesn't shown any potential in exploration but more like a distillation process, it is meaningless to announce that it perform better than PPO regarding speeds or hardware consuming.

The presentation of the paper needs to be largely improved. There are many compile Errors such as line 215 in the 4th page. Given all these reasons, I will not recommend the paper to be accepted.

questions:
Please refer to weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730521961248

thread_type:
official_review

<thread_object>
creator: reviewer_9knh

<content>
summary:
This paper presents an algorithm which they call UNified Alignment (UNA), by utilizing a simplified DPO implicit reward. UNA's loss minimizes the difference between the implicit reward and the explicit reward (provided by an existing reward model), hence can be generalized to different modes of feedback: pairwise, binary or scalar. Empirical results are presented to compare UNA with several baselines.

strengths:
1. There are math derivations for the proposed method.
2. The reward distillation loss is  general for different modes of feedback.

weaknesses:
W1. The paper's presentation does not meet the standard of ICLR. 
* Some references are missing (215,353); there are grammar and spelling errors (026,106,120,141). 
* Figures 1 and 2 are almost identical except for the different parts highlighted, which makes me feel they are redundant. You could refine the two figures and make them respectively focused on the topic you want to highlight: algorithm comparison & applications to different modes of feedback.
* The overall tune is not formal, and the writing is not easy to follow (too many pieces). If your goal is to develop a unified framework for LLM alignment, in your methodology you would start from defining the overall process of alignment, then the objectives of it, followed by your approach. Maybe at the end you can make connections with existing methods like RLHF with PPO, DPO, KTO and say they are some special cases. This suggestion is only one possible way for improvement. There are certainly more that you could consider to improve the current version.

W2. The technical novelty is limited. The implicit reward (eq6) of DPO is standard and well-known. One of the contributions is "simplifying" the implicit reward by setting f(x)=c=0. Could you provide insights behind it? Besides, the other contribution is a supervised loss of minimizing the difference between implicit and explicit rewards (eq7) is also not novel. For example, [1] proposed such a distillation-style loss for robust preference optimization.


[1] Fisch A, Eisenstein J, Zayats V, Agarwal A, Beirami A, Nagpal C, Shaw P, Berant J. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316. 2024 May 29.

questions:
It seems to me UNA also needs a reward model to learn the policy, which is not too different from RLHF. But in your abstract, "RLHF requires training the reward model and policy separately, which is complex, time-consuming, memory intensive and unstable during training processes." Can you clarify the advantage of UNA compared to standard RLHF? For example, it could be that UNA saves some computation, or converges faster, etc.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645551145

thread_type:
official_review

<thread_object>
creator: reviewer_cbvy

<content>
summary:
This paper provides a unifying framework for value alignment that has:
(i) explicit and implicit reward functions
(ii) can handle different types of preferences (pairwise, binary, score etc.)
and seems to provide better performance than state of the art approaches, RLHF, DPO, KTO.

strengths:
1. The unified framework does seem to have significant advantages.
2. The paper's ideas are strong.
3. Experimental results are detailed and strong.

weaknesses:
1. The biggest concern I have is that presentation is extremely poor, with many typos in notation, proofs, text and references. 
2. There are notational inconsistencies in proofs as well.
3. There is no intuitive explanation for why such a unified approach would provide better performance than any of the individual approaches. 

I find that the paper is written in a rushed manner and even though the ideas are interesting, it does not seem ready yet with typos all over the paper. Happy to revise my scores based on the responses of the authors.

questions:
1. What is the intuitive explanation for why a unified approach provides better results? DPO provides better results than RLHF as it does direct optimization of preferences, rather than first creating a reward function and then optimizing. In a similar vein, is there an intuitive explanation. 
2. There are errors in the main proof in Appendix B. While I seem to follow the broad outline, I am not entirely sure and would request the authors to provide more explanation and a corrected version. "y" is present on the left hand side, there is a "y" used in expectation. I am not sure how the expectation over "y" makes sense?
3. Are the experimental improvements significant? The improvements are there, but are they significant? How can it be quantified?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713551320

thread_type:
official_review

<thread_object>
creator: reviewer_kn8x

<content>
summary:
This paper introduces the UNA framework, which integrates three prominent alignment techniques for LLMs: RLHF/PPO, DPO, and KTO. The UNA framework seeks to unify these methods by using a generalized implicit reward function to align language model policies through supervised learning. UNA also accommodates diverse feedback types (pairwise, binary, and score-based) and aims to outperform each technique on downstream tasks, simplifying the RLHF fine-tuning process.

strengths:
1. By combining RLHF, DPO, and KTO, UNA supports a range of feedback types (pairwise, binary, and score-based), which enhances its versatility.
2. UNA replaces RLHF’s unstable, memory-heavy RL process with supervised learning, making the alignment process more straightforward and efficient. 
3. The paper provides a mathematical proof linking the RLHF objective to a generalized implicit reward function.

weaknesses:
1. Evaluation of the experiments is relatively small-scale. The authors only evaluate UNA over DPO & KTO when fine-tuning with LoRA and evaluate online RLHF using a 1.5B model. It may be difficult to prove whether UNA can take advantage when using large-scale models. I suggest the authors to try larger models like ~8B models since they claim that they use 8 x 80G A100 GPUs. 
2. The improvement of UNA over three baselines are not significant, which is considerate since they are somewhat equivalent. However, I think the authors should show the advantages of UNA, e.g., the time and computational cost, compared to baselines. 
3. The paper seems written in a rush with several clarity issues. For example, some references (Line 215 and 352) are missing.

questions:
1. The paper states that $f(x)=0$ when implicit and explicit reward models are exactly the same. However, when we optimize a model using UNA with the simplified implicit reward $f(x)=c=0$, will it guarantee that it will lead to the convergence?
2. DPO also use log pi / pi_ref as an indicator of the reward. However, this implicit reward cannot well reflect the training status. Can the UNA framework better stabilize the training process in the DPO manner?
3. In Table 3, maybe you can try other reference models rather than GPT in Alpaca-eval? The win rates against GPT-4 is to low and difficult to compare among different models.
4. Can you include the results of the original DPO?
</content>
</thread_object>

</thread>


Metareview: