Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

Authors: Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: large language models, alignment, preference optimization

Abstract:
Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data. However, the generation of the winning response and the losing response within pairwise data are typically isolated, leading to weak correlations between them as well as suboptimal alignment performance. To address this issue, we propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. Firstly, we increase the consistency and informativeness of the pairwise preference signals through targeted modifications, synthesizing a pseudo-winning response by improving the losing response with the winning response as a reference. Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model's confidence during training. Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730570181067

thread_type:
official_review

<thread_object>
creator: reviewer_wkka

<content>
summary:
The paper introduces a novel framework called **Bridging and Modeling Correlations (BMC)** to enhance Direct Preference Optimization (DPO) in aligning large language models (LLMs) with human preferences using pairwise preference data. The framework consists of two phases:

1. **Bridging Phase**: Enhances the correlation between winning and losing responses by generating a pseudo-winning response (ỹ_w) through targeted modifications of the losing response (y_l), referencing the winning response (y_w).

2. **Modeling Phase**: Improves token-level credit assignment by dynamically adjusting token rewards based on the policy model's confidence during training. This is formalized in their proposed objective function with adaptable weights.

Despite these new methods, the paper's experimental results show **modest performance improvements** compared to the computational and resource costs involved in the data modification process with stronger LLMs. I would like to understand these aspects further during the rebuttal period.

strengths:
1. **New Approach**: Introducing the Bridging Phase to enhance correlations in preference data is a new solution to a recognized problem in DPO methods.

2. **Dynamic Token-Level Adjustment**: Adjusting token rewards based on model confidence is a novel idea that could potentially improve fine-grained learning.

3. **Comprehensive Evaluation**: The authors conduct extensive experiments across multiple tasks, including question answering, mathematical reasoning, and instruction following.

4. **Versatility**: The framework is adaptable to various DPO variants, demonstrating its potential applicability in different contexts.

weaknesses:
1. **Marginal Performance Improvement vs. Cost**: The reported performance gains, while present, are relatively modest (e.g., 3.8 percentage points improvement in QA tasks), considering the significant effort and computational resources required for data synthesis and modification. Modifying 60-70% of the dataset introduces substantial overhead.

2. **Dependency on Large LLMs for Data Synthesis**: The Bridging Phase relies on powerful LLMs (e.g., GPT-4) for targeted modifications, which may not be feasible or cost-effective for many practitioners.

3. **Insufficient Analysis of Trade-offs**: The paper lacks a detailed discussion on the trade-off between the cost of data modification and the performance gains achieved.

4. **Limited Improvement Over Baselines**: Given the additional complexity and cost, the improvements over standard DPO and other baselines are insufficient to justify the added effort.

questions:
1. **Cost-Benefit Justification**: Can the authors provide a more detailed analysis of the computational and resource costs associated with the Bridging Phase relative to the performance gains? How do the authors justify the added complexity and cost?

2. **Impact of Data Modification Proportion**: Did the authors investigate the relationship between the proportion of data modified in the Bridging Phase and the resulting performance? Is there a point of diminishing returns?

3. **Baseline LLM Limitations**: Is the modest performance improvement partly due to inherent limitations of the baseline LLM used in the authors' experiments? Would using a more capable baseline model lead to greater improvements, or are there fundamental constraints in the proposed approach? It would be beneficial to elaborate on the capabilities of larger baseline models as well, if possible.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730600798670

thread_type:
official_review

<thread_object>
creator: reviewer_tygl

<content>
summary:
This paper identifies the important role of the correlation between chosen & rejected responses for LLM alignment with DPO-style algorithms. They propose a suite of solutions including (1): rephrasing the chosen response into the style closer to the rejected response to enhance their correlation; (2): adding a token-level weighting in the DPO loss based on the policy model's confidence on the different tokens between the (rephrased) chosen and rejected responses. Extensive experimental results demonstrate that the proposed BMC pipeline could improve the model's performance on QA, math reasoning, and instruction following benchmarks, as well as exhibiting better and more reasonable token-level and sequence-level reward behaviors.

strengths:
* This paper is written with high quality.
* The improvement to the current DPO-style alignment pipeline is well-motivated, with a simple and effective solution.
* The experiment part of this paper is well conducted, showing clear empirical significance with detailed ablation studies.

weaknesses:
* Regarding the "Bridging Phase", I feel it does not necessarily address every aspect of the initial annotations. For example, despite the semantic aspects (i.e., the chosen response is correct while the rejected one is not), the preferred syntax, or linguistic aspects, such as how to better organize the response, might be neglected. If the original chosen response is better in terms of aspects other than correctness, then the Bridging phase may "erase" such useful information.
* The scalability of rephrasing every chosen response is somewhat limited, which can be improved by investigating the effect of executing the BMC for a specific portion of the dataset or considering how to identify the crucial tokens during the DPO training (i.e., based on confidence pattern demonstrated in line 228-245 and Sec 5.3), i.e., designing a better token-level weighting criterion for DPO-MC.
* I think the overall computation overhead introduced by the BMC pipeline should be discussed.

questions:
* Regarding the first point in the weakness part, could the model still be able to learn some preferred linguistic aspect (such as organizing the answer with a CoT style) when the chosen & rejected responses overlap largely in terms of syntax? Does the shared linguistic structure matter or we can achieve a similar effect by identifying the token span that is critical to the semantic (which could be an improved version of DPO-MC with better token weighting criterion)?
* It would be interesting to see why LMs are bad at rephrasing a win response into a lose response. Do you have some observations other than Table 3, such as checking the difference of the result semantic equivalence between $(y_w, \tilde y_w)$ and $(y_l, \tilde y_l)$?
* Do you have some thoughts about why BMC can not boost SimPO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705374539

thread_type:
official_review

<thread_object>
creator: reviewer_pbh6

<content>
summary:
The paper proposes an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. It leverages a commercial LLM to rewrite the winning response and defines confidence-based token-level weighting to improve the original DPO. The experiments are extensive.

strengths:
- The idea of using commercial LLM to rewrite the winning response for DPO is interesting. 

- The experiments are extensive.

weaknesses:
- Regarding the motivation, the authors state that "yw and yl are generated isolatedly, the correlation between yw and yl can be weak during pairwise preference optimization", but this is not usually the case in practice. For example, a common practice is to let an LLM continue writing a prefix of the winning response to get the losing one. I mean, the correlation between these two responses can be easily established during data collection. 

- It is not well explained why "this weak correlation poses a challenge, as the winning response yw may not provide sufficiently informative gradients for adjusting the model’s parameters in relation to yl". This argument is too high-level. Can you delve into the details? Despite the results that have proven this point, I still wonder about the reason behind it. 

- The token confidence should not be always reliable and then the loss weighting trick hinges on intensive hyper-parameter tuning. This undermines the promise of the method. 

- To what extent does the final performance depend on the rewriting quality of the commercial LLM? Can some open-source ones be leveraged for this and what is their performance?

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732197517256

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers and Revision Submitted

comment:
We sincerely thank all the reviewers for their insightful and constructive comments, which have greatly contributed to improving our work. We have thoroughly revised the paper to address the reviewers' concerns. Below we summarize the major revisions (the main revisions are marked with *blue* text in the pdf, we also made some minor layout changes to fit the page limit), while we reply to the comments of each reviewer separately.

**Summary of Major Revisions**

1. We have supplemented a detailed cost analysis of the proposed BMC pipeline in Appendix B, which illustrates that the computational overhead introduced by our approach is minimal (Reviewer tYGL, WKKA).
2. We have relocated Appendix B from our initial manuscript to Section 5.1, emphasizing that a less powerful open-source model, Llama3-70B-Instruct, can serve as a substitute for GPT-4 in the Bridging Phase. This revision highlights the adaptability and cost-effectiveness of our approach (Reviewer pbH6, WKKA).
3. We have investigated the relationship between the proportion of data modified in the Bridging Phase and the resulting performance (Reviewer tYGL, WKKA).
4. We have expanded the explanation of why weak correlation between pairwise data poses a challenge to preference optimization (Reviewer pbH6), and why LLMs are bad at rephrasing a winning response into a losing response (Reviewer tYGL).
5. We have added more baselines in Section 5.1, including i) continue writing a prefix of the winning response to generate the losing one (Reviewer pbH6) and ii) leveraging an off-the-shelf LLM for external weighting of token-level reward (Reviewer tYGL). These additions provide a deeper understanding of the effectiveness of our method.
6. We have reported additional strong experimental results of our approach based on Qwen2.5-14B-Base in Appendix E, indicating that our method's effectiveness scales with model capability (Reviewer WKKA).

**Conclusion**

We deeply appreciate the reviewers' valuable feedback, which has enabled us to refine our work significantly. These revisions strengthen both the theoretical and practical contributions of our paper. We believe that the additional analyses, new baselines, and expanded experiments provide a comprehensive response to all concerns, making the work stronger and more impactful.

We welcome further questions or feedback and are committed to providing any additional clarifications or materials necessary to assist in your evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733295660704

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks to all Reviewers and Rebuttal Summary

comment:
Dear Reviewers and ACs,

We sincerely thank all the reviewers for their thoughtful feedback and constructive discussions! Below, we summarize our key contributions, responses to reviewer comments, and the outcome of our rebuttal process.

**Key Contributions**

Our submission proposes BMC, a novel framework for Bridging and Modeling Correlations in pairwise data, enabling more effective Direct Preference Optimization (DPO). Key contributions include:
1. We identify and address weak correlations in pairwise data with a unique framework for Bridging and Modeling Correlations.
2. The Bridging Phase of our framework effectively enhances the informative correlations between pairwise data through targeted modifications. Importantly, this phase is independent of commercial LLMs, ensuring broad applicability.
3. We further improve token-level correlation modeling in DPO by dynamically incorporating the policy model's confidence during training, an approach that has not been explored previously.
4. Our framework is **effective** (significantly outperforms DPO across 10 datasets), **efficient** (introduces minimal computational overhead), **robust** (validated through extensive quantitative and qualitative analyses), and **versatile** (seamlessly integrates with various DPO variants, consistently boosting their performance).

The novelty and soundness of our contributions have been recognized by reviewers as:
- "*interesting*", "*experiments are extensive*" (pbH6)
- "*well-motivated*", "*clear empirical significance with detailed ablation studies*" (tYGL)
- "*novel idea*", "*comprehensive evaluation*" (WKKA)

**Reviewer Comments and Our Responses**

We have thoroughly addressed the reviewer concerns, as summarized below:

| Comment | Response | Revision | Related Reviewers|
|:--------------|:--------------:|:---------:|:---------:|
| Computation cost of BMC should be discussed | Added a detailed cost analysis showing that BMC introduces minimal computational overhead | Appendix B | tYGL (W3); WKKA (W1, W3, W4, Q1) |
| Can open-source LLMs work for the Bridging Phase? | Highlighted Llama3-70B-Instruct as a viable substitute for GPT-4 | L.418-423 | pbH6 (W4); WKKA (W2) |
| Impact of data modification proportion in the Bridging Phase? | Added analysis showing higher modification proportions yield stronger performance | L.412-417 | tYGL (W2.1); WKKA (Q2) |
| Clarify motivation | Expanded explanation of weak correlation issues and challenges | L.150-155 | pbH6 (W1, W2) |
| Why are LLMs bad at rephrasing a winning response into a losing one? | Added qualitative and quantitative analyses | L.407-410 | tYGL (Q2) |
| Consider alternative methods for identifying crucial tokens in DPO | Added a competitive baseline to validate BMC's superiority | L.310-315 | tYGL (W2.2) |
| Does BMC scale with more capable models? | Added experiments with Qwen2.5-14B, showing BMC's effectiveness scales with model capability | Appendix E | WKKA (Q3) |

**Outcome of Rebuttal**

Our responses addressed all reviewer concerns, with no further questions raised. Notably, both Reviewer pbH6 and Reviewer WKKA increased their ratings from 5 to 6.

We deeply appreciate the reviewers' valuable insights, which has significantly enhanced our work. The new analyses, baselines, and experiments strengthen both the theoretical and practical contributions of our paper. We are confident that these revisions make our work more impactful and compelling. We will continue striving for excellence.

Thank you once again for your time and effort!

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a new framework for training models using DPO.  The core idea is to leverage large language models (LLMs) to refine responses, specifically by increasing the correlation between "winning" (preferred) and "losing" (less preferred) responses. This process enhances the training data and ultimately leads to improved model performance.  The authors demonstrate consistent improvements across various tasks.

Strengths:
- The proposed method is novel and the use of LLMs to manipulate response correlations is an innovative approach that yields performance gains.
- Additional results provided during the rebuttal phase, including experiments with different LLMs, showcase the generalizability of the method.

Weaknesses:
- While the LLM-based response regeneration can be performed offline, it still incurs a significant computational cost (over $200 for some datasets).
- The observed performance improvements are somewhat moderate.

Despite the identified weaknesses, the novelty and effectiveness of the proposed method warrant an acceptance recommendation.