On Extending Direct Preference Optimization to Accommodate Ties

Authors: Jinghong Chen, Guangyu Yang, Weizhe Lin, Jingbiao Mei, Bill Byrne
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Preference Optimization, Ties, Direct Preference Optimization, Langauge Model, Machine Translation, Summarisation.

Abstract:
We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without  the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. These findings motivate and enable the inclusion of tied pairs in preference optimization as opposed to  simply discarding them.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730267572162

thread_type:
official_review

<thread_object>
creator: reviewer_t7fn

<content>
summary:
This paper extends the Direct Preference Optimization (DPO) framework to handle tied preferences in pairwise comparisons by incorporating two established preference models, the Rao-Kupper and Davidson models. Unlike the original DPO, which discards ambiguous data points, the proposed variants, DPO-RK and DPO-D, allocate probabilities to ties, preserving more information from the data. The paper evaluates these DPO variants in machine translation and summarization tasks, showing that the inclusion of ties enhances regularization effects without compromising performance.

strengths:
1. The paper successfully integrates the Rao-Kupper and Davidson models to extend DPO for handling ties, addressing a known limitation in the original DPO framework where tied pairs were previously discarded. This extension makes the preference modeling process more inclusive and data-efficient.

2. The experimental analysis includes various statistical measures and visualization of training dynamics, such as reward margins and gradient scale factors, which add depth to the assessment of how well the models learn from tied data.

3. The paper is well-written and easy to follow. The adaptation of Bradley-Terry models for ties is systematically outlined, with clear mathematical formulations and derivations for the proposed DPO-RK and DPO-D models.

weaknesses:
1. The introduced method shows limited novelty, as it builds upon existing Bradley-Terry extensions to accommodate ties. The proposed DPO variants primarily involve substituting the standard Bradley-Terry model with these pre-existing extensions (Rao-Kupper and Davidson) for preference modeling, rather than presenting fundamentally new methodologies.

2. Although the proposed DPO variants improve regularization effects, they fail to demonstrate any clear enhancement in task performance relative to the original DPO. This raises questions about the practical value of the proposed modifications, particularly in performance-critical applications.

3. The experiments are limited to machine translation and summarization tasks, leaving more complex domains such as open-ended generation tasks (e.g., AlpacaEval, Arena-Hard, MT-Bench) and mathematical reasoning unexplored. Expanding evaluations to these challenging areas would provide a more comprehensive assessment of the proposed methods.

4. The paper introduces additional parameters, such as the probability adjustment factor $\upsilon$ for the Rao-Kupper and Davidson models, but lacks extensive discussion on the impact for tuning these parameters.

questions:
See weeknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687421547

thread_type:
official_review

<thread_object>
creator: reviewer_gv1j

<content>
summary:
The authors propose two methods to generalize DPO to accommodate tie (y_i = y_j).

However, it may seem a minor problem from my perspective. DPO cannot accommodate tie but it can tolerate when y_i is very close to y_j.

In addition, one problem is that the authors does not compare the proposed two method with DPO without considering tie. Will its performance be lower than DPO?

The author utilize a relative subjective method to compare models, while classification and generation tasks should be done.

strengths:
The authors propose two methods to generalize DPO to accommodate tie (y_i = y_j).

weaknesses:
However, it may seem a minor problem from my perspective. DPO cannot accommodate tie but it can tolerate when y_i is very close to y_j.

In addition, one problem is that the authors does not compare the proposed two method with DPO without considering tie. Will its performance be lower than DPO?

The author utilize a relative subjective method to compare models, while classification and generation tasks should be done.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715087776

thread_type:
official_review

<thread_object>
creator: reviewer_ehu6

<content>
summary:
The paper incorporates the pair-wise ties into DPO alignment framework, by replacing the BT model in DPO with Rao and Kupper and Davidson extensions. It conducted performance on machine translation and summarization tasks. Also, it provides some analysis on KL divergence and RM statistics. It provides potential insights to incorporating tie pairs for DPO training.

strengths:
1. The motivation is straight forward and easy to follow. It provides the straightforward idea that using the discarded tie pairs for optimizing the DPO framework.

2. The injection of DPO-RK and DPO-D remains similar task performance in comparison to original DPO, mitigating the problem of original DPO that will suffer from task performance degradation when injecting tied pairs.

3. The background is well presented and in a good shape.

weaknesses:
1. The paper experiments is not quite sufficient. It only conducted experiments on machine translation and summarization, rather than conducting commonly used alignment datasets such as HH-RLHF (https://huggingface.co/datasets/Anthropic/hh-rlhf).

2. Limited evaluation metrics. It only uses automatic evaluation metrics BLEURT and RM win rate, which is not often in consistent with human alignments. It is required to further report the results of gpt-4 evaluation and human evaluations.

3. It is recommended to report the case to better illustrate the effect of proposed methods.

4. The methods of obtaining tie pairs requires further evaluation. Is the automatic distance-based metrics reliable to obtain tie pairs? It is recommended to report the evaluation results.

questions:
1. What is the human evaluation results on the DPO-RK/DPO-D results compared to original DPO?

2. How is the results on commonly used benchmarks, such as HH-RLHF?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730819902812

thread_type:
official_review

<thread_object>
creator: reviewer_kszv

<content>
summary:
This paper introduces two variants of Direct Preference Optimization (DPO) designed to account for ties in pairwise preference data, addressing a limitation in the original DPO framework that requires strict winner-loser pairs. The authors replace the Bradley-Terry model traditionally used in DPO with two extensions from Rao-Kupper and Davidson models, both of which assign probabilities to ties in addition to clear preferences (and have a nice explanation of the different assumptions in each). 
The proposed DPO variants—DPO-RK and DPO-D—are then tested on neural machine translation and summarization tasks, using synthetic preference pairs (with ties) constructed metrics/reward models. The experiments show that incorporating ties into DPO-RK/D does not degrade task performance as it does in traditional DPO and, in fact, regularizes the policy with respect to the reference policy, as seen through KL divergence measurements. They also test the learned policies’ classification accuracy at  distinguishing between clear preferences and ties.

strengths:
- Overall, I like the paper’s motivation and proposed approach: DPO is very prevalent method to perform preference optimization, and the inability to handle ties is a definitely a drawback.
- The proposed variations are well motivated and the theoretical discussion on their differences was quite interesting to me.
- The (limited) results included hint that including ties and using one their proposed variants leads to better policies than the current dominant approach of discard ties.

weaknesses:
While overall like the idea and introduced approach, I think the paper is severely limited by the weak empirical evaluation. A couple of points:

- Reporting of results for both task are relatively limited, only reporting performance for preference aligned models on the metrics used to construct preference. I think having a table reporting baseline and best performing DPO models on multiple metrics is necessary to trust gains from DPO and its variants, as they could be over-optimizing for BLEURT. In MT for example, gains in COMET (or even BLEU/ChrF) would allow me to trust to overall conclusions alot more.
- For machine translation, a relatively weak model (BLOOMZ-mt-7b) is used. Why this model rather than many strong LLM for MT e.g. TowerLLM? Similarly, BLEURT is relatively outdated as a metric for MT, and I would have used a better metrics to construct preference pairs. In general, replicating the findings in more than one base LLM (for at least MT) would strengthen the paper.
- In general, there lack of comparison to some other baseline preference alignment algorithms (KTO, IPO, etc…). I would really like to see how DPO-RK/D compares one of them, especially if tie data is also used with it.

But in general I like the motivation of the paper, and would be happy to revise my score if authors address these comments

questions:
- What happens when the proportion of ties vs winning pairs changes (in terms of performance, KL, etc…)? Since the preference pairs are synthetic, this would be something relatively easy to explore, and could inform what kinds of preference datasets make DPO-RK/D work well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732290195537

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
References for Author Response

comment:
Please kindly see below for references used in authors' responses.

[1] Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding (https://arxiv.org/abs/2311.08380)

[2] Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation (https://arxiv.org/abs/2401.08417)

[3] Results of WMT23 Metrics Shared Task: Metrics Might Be Guilty but References Are Not Innocent (https://aclanthology.org/2023.wmt-1.51.pdf)

[4] CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task (https://aclanthology.org/2022.wmt-1.60.pdf)

[5] A Preference-driven Paradigm for Enhanced Translation with Large Language Models (https://aclanthology.org/2024.naacl-long.186.pdf)

[6] BLEURT: Learning robust metrics for text generation. (https://aclanthology.org/2020.acl-main.704/)

[7] COMET: A neural framework for MT evaluation. (https://aclanthology.org/2022.wmt-1.52/)

[8] COMET-22: Unbabel-IST 2022 submission for the metrics shared task. (https://aclanthology.org/2022.wmt-1.52/)

[9] xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection (https://arxiv.org/abs/2310.10482)

[10] A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models (https://arxiv.org/abs/2309.11674)

[11] Tower: An Open Multilingual Large Language Model for Translation-Related Tasks (https://arxiv.org/abs/2402.17733)

[12] Scaling up CometKiwi: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task (https://aclanthology.org/2023.wmt-1.73.pdf)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper extends the Direct Preference Optimization (DPO) framework by introducing two variants, DPO-RK and DPO-D, which incorporate tied preferences in pairwise data using existings models (Rao-Kupper and Davidson). Through experiments on neural machine translation and summarization tasks, this study shows that these variants preserve more information by accounting for ties, offering improved regularization effects while maintaining task performance. Additionally, the paper provides analyses on KL divergence and tie classification, highlighting the potential benefits of integrating tie-aware models into DPO.

Strengths:

* The paper addresses a clear limitation of the DPO framework, where tied preference pairs are discarded, by proposing two extensions (DPO-RK and DPO-D) that leverage ties effectively.
* The theoretical discussion and integration of Rao-Kupper and Davidson models provide a well-motivated foundation for handling ties in preference optimization.
* The paper is well-written, with clear mathematical formulations and derivations, making the proposed methods easy to follow.
* The experimental analysis, while limited, offers some insight into the performance improvements when incorporating tied pairs.

Weaknesses:

* The paper introduces methods based on existing extensions of the Bradley-Terry model, such as the Rao-Kupper and Davidson variants, but these modifications represent incremental advancements rather than fundamentally novel methodologies.
* The experimental validation is limited in scope, focusing solely on machine translation and summarization tasks without exploring broader, more challenging domains like open-ended text generation or mathematical reasoning.
* The work relies mostly on outdated or limited evaluation metrics.

While the authors addressed some of the issues raised in the original reviews, the reviewers remain mostly unconvinced about the novelty and the level of difficulty of the studied tasks, even after considering the authors' responses. I share the reviewers' concerns and recommend rejection.