Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

Authors: Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, Mingyuan Zhou
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM alignment, fine-tuning, preferences

Abstract:
In the field of large language models (LLMs), aligning models with the diverse preferences of users is a critical challenge. Direct Preference Optimization (DPO) has played a key role in this area. It works by using pairs of preferences derived from the same prompts, and it functions without needing an additional reward model. However, DPO does not fully reflect the complex nature of human learning, which often involves understanding contrasting responses to not only identical but also similar questions. To overcome this shortfall, we propose Relative Preference Optimization (RPO). RPO is designed to discern between more and less preferred responses derived from both identical and related prompts. It introduces a contrastive weighting mechanism, enabling the tuning of LLMs using a broader range of preference data, including both paired and unpaired sets. This approach expands the learning capabilities of the model, allowing it to leverage insights from a more varied set of prompts. Experiments in both paired and unpaired dataset settings, including tasks like dialogue, summarization, and general evaluation benchmarks, demonstrate RPO's superior ability to align LLMs with user preferences and enhance adaptability during training.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730629966203

thread_type:
official_review

<thread_object>
creator: reviewer_i7yc

<content>
summary:
The paper introducs Relative Preference Optimization (RPO), a novel method to enhance the alignment of LLMs with human preferences. RPO considers preferences across both identical and semantically related prompts and employs a contrast matrix to discern preferences between preferred responses via a contrastive weighting mechanism. So RPO enhancs model training with a broader range of data. Empirical tests on dialogue and summarization tasks showed that RPO outperforms existing methods like DPO, IPO, and KTO, especially in adapting to varied contexts.

strengths:
- Extends beyond traditional methods by considering semantically related prompts.
- RPO is adaptable to both paired and unpaired data scenarios, making it a flexible tool for different types of preference data.

weaknesses:
- RPO relies heavily on the semantic similarity of prompts to establish contrastive pairs. This approach might limit its effectiveness in contexts where prompts lack clear semantic relationships, especially in diverse general-purpose task datasets. As a result, the probability of relevant prompts appearing within a mini-batch is relatively low, and it is rather difficult to screen out sample pairs of high $\omega$ within a mini-batch. Related to Question 1.
- As an article submitted to ICLR, both the scientific problems distilled in the article and its methodologies are rather deficient in terms of inspiration and theoretical contributions.

questions:
1. It seems that the performance improvement of this method relative to DPO mainly stems from the combination of additional weighted training samples within the mini-batch, which expands the scale of the training data. Could you please count the proportion of samples with omega > 0.5 in the Anthropic-HH training set under different configurations of batch size?

2. If we construct new sample pairs of relative prompts (i.e. high $\omega$ values) based on the entire dataset and then add them to the original dataset according to the ratio obtained from Question 1, could DPO produce comparable or even better result?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641739164

thread_type:
official_review

<thread_object>
creator: reviewer_hl8x

<content>
summary:
This paper introduces Relative Preference Optimization (RPO), an enhancement to the Direct Preference Optimization (DPO) method for aligning large language models (LLMs) with user preferences. RPO extends DPO by considering both identical and related prompts, and it uses a contrastive weighting mechanism to handle a broader range of preference data. Experiments demonstrate that RPO outperforms baseline methods on dialogue, summarization, and general evaluation benchmarks.

strengths:
1. RPO addresses a limitation of DPO by incorporating both identical and related prompts, thereby expanding the model's ability to learn from a more diverse set of data.

2. The paper is well-written and clearly explains the methodology and experimental results, making it easy for readers to understand the contributions and their implications.

weaknesses:
1. The approach of using sentence embeddings to assess semantic similarity may not be sensitive to token-level differences. This could be a significant drawback in domains like code and mathematics, where even minor differences in tokens of prompts can lead to entirely different or even contradictory answers.

2. The proposed RPO may be seen as a form of data augmentation, where different question answer pairs are combined during the training process to generate more data pairs to improve model performance. This augmentation approach is common in traditional data augmentation fields.

3. The experiments could be further strengthened. For instance, it would be interesting to see if using the model being trained to compute sentence similarity would yield better results. Additionally, the paper should explore whether this simple method provides any benefits for larger models, such as 13B or more capable ones, to fully validate its effectiveness across different scales.

questions:
Please refer to the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689030322

thread_type:
official_review

<thread_object>
creator: reviewer_7wmz

<content>
summary:
The Relative Preference Optimization (RPO) method presents a novel approach to aligning preferences that significantly improves data efficiency and effectiveness.

strengths:
The proposed RPO method introduces an innovative approach to preference alignment, showcasing enhanced data efficiency and effectiveness. By integrating a contrastive weighting mechanism, RPO significantly broadens the learning capabilities of language models, allowing them to process a wider range of preference data. The experimental results across various tasks, such as dialogue, summarization, and general evaluation benchmarks, highlight RPO's superior adaptability and effectiveness in optimizing language model performance according to user preferences.

weaknesses:
The paper states that "DPO does not fully reflect the complex nature of human learning, which often involves understanding contrasting responses to not only identical but also similar questions." However, there appears to be a lack of explicit experimentation or detailed discussion in the manuscript to substantiate this claim. It would be beneficial for the authors to provide empirical evidence or theoretical elaboration to support this assertion, which would strengthen the argument for RPO's advantages over existing methods like DPO.

questions:
- How long does the embedding preprocessing step take?
- In my understanding, RPO utilizes a loss function similar to DPO's, augmented with a data reweighting mechanism. Could this mechanism be integrated with other approaches such as IPO, KTO, or SimPO? If so, what would the potential benefits or challenges be in combining these methodologies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731114609757

thread_type:
official_review

<thread_object>
creator: reviewer_2go9

<content>
summary:
The authors proposed new preference optimization methods to improve the performance of preference learning on various tasks. They explained that the current state-of-the-art methods cannot fully reflect the complex nature of human learning. To overcome this challenge, they proposed Relative Preference Optimization (RPO) to discern between more and less preferred responses derived from identical and related prompts. They showed that RPO outperforms DPO, KTO, and IPO on dialogue, summarization, and general evaluation benchmarks.

strengths:
The authors explored one of the important problems of direct preference optimization methods, and it's interesting. Another advantage of this paper is the comprehensive experiment and theory used to satisfy their hypothesis.

weaknesses:
I would appreciate that if the authors could clarify the following points:

- The results show that RPO outperforms other methods. I looked at the implantation they put in Appendix G and understood the main difference between RPO loss and DPO loss is the weight matrix. I am wondering how the authors can describe if a policy model has knowledge about the difference between the response of prompt i and j (assume prompt i is a reasoning question, and prompt j is a usual question that asks about the weather) will have a better performance?

- The Experiment section shows that the authors didn't explore the hyperparameters for other methods. To the best of my knowledge for the authors, direct methods like DPO, IPO, and KTO have a different performance on different hyper-parameters like Batch-Size, Beta, Regularization terms in IPO etc. Also, they didn't use beta=0.01 for DPO, which is the best hyper-parameter for this method. So, I suggest the authors compare their methods with the best DPO, KTO, and IPO hyperparameters. 

- With respect to the comprehensive experiments, it seems the authors didn't report the scores for all models. For example, the experiments on Summarization are on the Mistral model, but for Anthropic-HH, it is on LLaMA (7B, 13B) and Mistral. Also, for other benchmarks, they didn't report the score for other methods. For example, on Open LLM  Leaderboard benchmarks, they just compared with KTO; another comparison method can be DRO [1]. I am confused as to why the authors used the Binarized Capybara Dataset because Ultrafeedback_binarized [2] is available, and they could use this dataset.

- New methods like SimPO and CPO are better choices for comparison. I encourage the authors to compare the proposed method with these algorithms, too.

- Using new models like LLaMA-3 can improve the experiment section.

---
[1] DRO:  https://arxiv.org/abs/2405.19107

[2] Ultrafeedback_binarized: https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized

questions:
All the points are mentioned in the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732254801336

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to Reviewers

comment:
We deeply appreciate all reviewers for their valuable suggestions. In this section, we’ve updated additional tabular results of common interests.

**Table 1:** We fine-tuned Meta-Llama-3-8B-Instruct on a general chat task using the on-policy ultrafeedback dataset princeton-nlp/llama3-ultrafeedback-armorm. Each baseline method was trained with the optimal hyperparameters adopted in SimPO. RPO demonstrates superior performance across almost all benchmarks.

**Table 2:** Building on SimPO, we incorporated RPO's cross-prompt contrastive learning mechanism to fine-tune gemma-2-9b-it and Meta-Llama-3-8B-Instruct on the on-policy ultrafeedback dataset, respectively. Similarly, each baseline method was trained using the optimal hyperparameters from SimPO. The combined method, SimPO_RPO, achieves superior performance compared to SimPO alone.

**Table 1**:
| **Benchmark**    | **DPO** | **KTO** | **IPO** | **CPO** | **SimPO** | **RPO** |
|-------------------|---------|---------|---------|---------|-----------|---------|
| Arc Challenge     | 64.68   | 63.40   | 63.31   | 63.99   | 67.24     | 66.38   |
| TruthfulQA        | 55.82   | 55.62   | 60.77   | 56.28   | 64.71     | 61.17   |
| Winogrande        | 76.40   | 76.24   | 74.35   | 75.93   | 74.51     | 76.01   |
| GSM8k             | 76.95   | 75.36   | 75.21   | 75.74   | 70.74     | 74.83   |
| Hellaswag         | 79.70   | 79.56   | 76.45   | 79.06   | 78.15     | 79.88   |
| MMLU              | 65.91   | 65.95   | 65.52   | 65.72   | 65.09     | 65.95   |
| Average       | 69.91 | 69.36 | 69.27 | 69.45 | 70.07 | **70.70** |
|                   |         |         |         |         |           |         |
| AlpacaEval (LC)   | 41.59   | 41.46   | 43.90   | 41.37   | 42.19     | **44.20**   |
| MT-Bench          | **7.49**    | 7.43    | 7.33    | 7.47    | 7.21      | 7.45    |
| Arena-Hard        | 44.20   | 33.50   | 36.90   | 40.90   | 37.20     | **45.60**   |

| Method | lr    | beta  |
|--------|-------|-------|
| DPO    | 3e-7  | 0.01  |
| KTO    | 5e-7  | 0.01  |
| IPO    | 7e-7  | 0.5   |
| CPO    | 6e-7  | 0.05  |
| SimPO  | 1e-6  | 10    |
| RPO    | 7e-7  | 0.01  |



**Table 2**:

| **Benchmark**      | **Llama-3-8B-Instruct (SimPO)** | **Llama-3-8B-Instruct (SimPO_RPO)** | **gemma-2-9b-it (SimPO)** | **gemma-2-9b-it (SimPO_RPO)** |
|--------------------|---------------------------------|------------------------------------|---------------------------|------------------------------|
| Arc Challenge      | 67.24                          | 66.47                             | 67.83                     | 70.65                       |
| TruthfulQA         | 64.71                          | 64.35                             | 59.43                     | 60.49                       |
| Winogrande         | 74.51                          | 75.53                             | 75.14                     | 75.93                       |
| GSM8k              | 70.74                          | 70.51                             | 79.38                     | 82.11                       |
| Hellaswag          | 78.15                          | 77.85                             | 68.39                     | 70.38                       |
| MMLU               | 65.09                          | 65.27                             | 71.75                     | 71.97                       |
| Average       | 70.07                      | **70.16**                         | 70.32                 | **71.92**                   |
|                    |                                 |                                    |                           |                              |
| AlpacaEval (LC)    | 42.19                          | **44.72**                             | 50.05                     | **52.09**                       |
| MT-Bench           | 7.21                           | **7.31**                              | 8.02                      | **8.07**                        |
| Arena-Hard         | 37.20                          | **38.80**                             | 62.90                     | **63.50**                       |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Relative Preference Optimization (RPO) which is designed to discern between responses derived from identical and related prompts, which enables utilizing both paired and unpaired data. While the reviewers find the method to be interesting, they also raised significant concerns regarding the soundness of the method as well as the experiments/evaluations. The authors should consider reviewer's feedback when revising the manuscript.