Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

## Abstract

Abstract Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data.
However, the winning response and the losing response within pairwise data are generated isolatedly, leading to weak correlations between them as well as suboptimal alignment performance.
To address this issue, we propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC .
Firstly, we increase the consistency and informativeness of the pairwise preference signals through targeted modifications , synthesizing a pseudo-winning response by improving the losing response with the winning response as a reference.
Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model’s confidence during training.
Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO.
Additionally, our in-depth quantitative analysis reveals the reasons behind our method’s superior performance over DPO and showcases its versatility to other DPO variants.
We release our repository at https://github.com/YJiangcm/BMC .

## 1 Introduction

Direct preference optimization (DPO) (Rafailov et al., ) has emerged as a prominent alternative to reinforcement learning from human feedback (RLHF) (Christiano et al., ; Bai et al., ; Ouyang et al., ) for aligning large language models (LLMs) with human values.
Unlike the traditional RLHF approach, DPO bypasses training a reward model and avoids using any reinforcement learning algorithms.
Instead, it reparameterizes the reward function to directly learn a policy model from offline pairwise preference data, employing the Bradley-Terry ranking objective (Bradley & Terry, ).
Since the inception of DPO, numerous studies have sought to advance this method by refining its training objective (Wang et al., ).
For instance, IPO (Azar et al., ) introduces an alternative pairwise preference loss to mitigate overfitting to the preference dataset, while R-DPO (Park et al., ) incorporates a regularization term to prevent the exploitation of latent length bias in the training data.

However, relatively little attention has been given to enhancing DPO through advancements in the quality of preference data used for training.
In particular, the generation of winning and losing responses within preference data often occurs in an isolated manner, either through human annotation (Bai et al., ) or automated techniques such as RLAIF (Bai et al., ) and reject sampling (Liu et al., ; Pace et al., ).
This isolation implies that winning and losing responses are produced without mutual visibility, resulting in a lack of strong correlation or relevance between them.
Consequently, the model may struggle to identify nuanced yet significant distinctions that differentiate superior responses from inferior ones (Fürnkranz & Hüllermeier, ; Wirth et al., ), which can ultimately compromise optimization and alignment effectiveness.

In this work, we introduce an innovative framework, termed BMC, to Bridge and Model Correlations in pairwise data for direct preference optimization.
During the Bridging Phase, we enhance correlations by increasing the consistency and informativeness of pairwise preference signals.
By using the winning response as a reference, we synthesize a pseudo-winning response through targeted modifications of the losing response.
This pseudo-winning response offers two key advantages: (1) it preserves essential characteristics of the losing response, minimizing noise in preference signals (consistency); (2) it encapsulates all human-desired values from the winning response, enabling the model to better discern features that lead to superior performance (informativeness).
The nuanced differences between the pseudo-winning and losing responses are indeed what we expect the model to learn in the subsequent Modeling Phase.
Nonetheless, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
From the perspective of the token-level Markov Decision Process (MDP) (Rafailov et al., ), DPO aggregates rewards uniformly across all tokens, assuming equal contribution to sequence quality and neglecting token-specific importance.
To address this, we adjust the emphasis on rewards of different tokens between pseudo-winning and losing responses.
Unlike previous methods (Guo et al., ; Cao et al., ; Chan et al., ; Chen et al., ) that assign predefined values for fine-grained guidance, our adjustment is dynamically guided by the policy model’s confidence, i.e., the probability assigned to generated tokens during training.
This ensures the model focuses on learning challenging distinctions while reinforcing known patterns, resulting in a more nuanced and robust policy.

We conduct extensive experiments across three downstream scenarios: question answering, mathematical reasoning, and instruction following, utilizing a total of 10 datasets.
Our results demonstrate that our method consistently and significantly outperforms competitive offline optimization algorithms across various tasks. Notably, it exceeds vanilla DPO by up to 3.8 points in question answering and by up to 6.4 points on AlpacaEval 2.
Furthermore, we use in-depth
analyses to elucidate why our method outperforms DPO and show that our framework can be versatilely adapted to other DPO variants, confirming its potential for broad application.

## 2 Related Work

#### Preference optimization.

Preference optimization refers to aligning large language models with human preferences or specific desired outcomes.
A well-established method for this is reinforcement learning from human feedback (RLHF) (Christiano et al., ; Bai et al., ; Ouyang et al., ), which typically involves three stages: supervised fine-tuning, reward model training, and policy optimization.
Although RLHF produces highly capable models, its training process is often complex and unstable (Santacroce et al., ), requiring the training of multiple LMs and sampling from the LM policy.
To address these challenges, direct preference optimization (DPO) (Rafailov et al., ) introduces an alternative offline algorithm to optimize the regularized expected rewards without relying on RL.
Following the introduction of DPO, several studies have sought to enhance it in various ways. For instance, IPO (Azar et al., ) aims to prevent DPO from overfitting to the preference dataset;
ORPO (Hong et al., ) and SimPO (Meng et al., ) eliminate the need for a reference model;
and R-DPO (Park et al., ) incorporates a regularization term to prevent exploitation based on length.

#### Preference data construction.

Constructing high-quality pairwise preference data is essential for preference optimization.
Given the high cost of manually curating these datasets at scale, researchers have explored automated methods for producing preference data.
One notable approach, RLAIF (Bai et al., ) employs LLMs to label side-by-side response pairs, eliminating the need for human labeling.
Alternatively, winning and losing responses can be generated by utilizing models of varying quality (Kim et al., ) or through specific prompting techniques (Yang et al., ).
Recently, sampling-based methods such as Statistical Rejection Sampling (Liu et al., ) and West-of-N (Pace et al., ) have been introduced, generating preference pairs by selecting candidates sampled from the optimal policy. Nonetheless, these methods isolatedly generate winning and losing responses without accounting for the correlations between them.

#### Token-level preference optimization.

The majority of preference optimization strategies typically utilize trajectory-wise (sequence-level) rewards, while LM training and generation both occur at the token level (Yang et al., ).
To bridge this gap, FIGA (Guo et al., ) and DRLC (Cao et al., ) exploit external LLMs to pinpoint positive and negative token segments within responses, assigning fixed reward values (e.g., +1 for positive, -1 for negative) as guidance.
Meanwhile, ABC (Chan et al., ) and RLMEC (Chen et al., ) extract fine-grained credits from the reward model.
Despite their contributions, these methods rely on predefined values for fine-grained guidance, failing to account for the dynamic learning process of the policy model.

## 3 Methodology

In this section, we present the proposed BMC approach, which bridges and models correlations in pairwise data for direct preference optimization.
As depicted in Figure 1, our BMC framework is structured around two pivotal stages: (1) the Bridging Phase, where we enhance the correlations between pairwise data by increasing the consistency and informativeness of pairwise preference signals through targeted modifications (§3.1); and (2) the Modeling Phase, where we dynamically model the correlations during the optimization process by leveraging the confidence of the policy model (§3.2), alleviating the insufficient token-level credit assignment of DPO.

Figure: Figure 1: Overview of our proposed BMC framework. (1) In the Bridging Phase, we utilize an off-the-shelf LLM to make targeted modifications of losing response $y_{l}$ on undesired tokens, with the winning response $y_{w}$ serving as a reference. Therefore, the synthesized pseudo-winning response $\tilde{y}_{w}$ is highly correlated with $y_{l}$. (2) In the Modeling Phase, we model the correlations between $\tilde{y}_{w}$ and $y_{l}$ by dynamically emphasizing the rewards of their varied tokens ( $\mathit{diff}(\tilde{y}_{w}\mid y_{l})$ and $\mathit{diff}(y_{l}\mid\tilde{y}_{w})$), leveraging the policy model confidence (numbers indicated above tokens) during training.
Refer to caption: x1.png

### 3.1 Bridging Phase

In offline preference optimization, it is commonly assumed that we have access to a static pairwise preference dataset $\mathcal{D}=\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\}_{i=1}^{N}$, where $y_{w}$ and $y_{l}$ denote the winning and losing response, give the input prompt $x$.
However, since $y_{w}$ and $y_{l}$ are generated isolatedly, the correlation between $y_{w}$ and $y_{l}$ can be weak during pairwise preference optimization.
This weak correlation poses a challenge, as the winning response $y_{w}$ may not provide sufficiently informative gradients for adjusting the model’s parameters in relation to $y_{l}$ (Fürnkranz & Hüllermeier, ; Wirth et al., ).
Consequently, the optimization process struggles to effectively leverage the provided pairwise preferences to fine-tune the model, potentially resulting in suboptimal alignment performance.
To address this challenge, we enhance the alignment efficacy by improving the consistency and informativeness of pairwise preference signals.
As shown in the upper part of Figure 1, we utilize an off-the-shelf LLM to make targeted modification of $y_{l}$ by referring to $y_{w}$:

$$ $\text{LLM}(I,x,y_{w},y_{l})\rightarrow\tilde{y}_{w},$ (1) $$

where $\tilde{y}_{w}$ is the generated pseudo-winning response, $I$ is the instruction (see examples in Appendix A.2) that requires $y_{l}$ to be modified only on dispreferred tokens, using $y_{w}$ as a reference guidance.
In this way, $\tilde{y}_{w}$ preserves essential characteristics of the losing response $y_{l}$ while encapsulating all human-desired values in the winning response $y_{w}$.
The token-level differences between $\tilde{y}_{w}$ and $y_{l}$ highlight the core human expected and unexpected behaviors by decoupling from the inherent linguistic style and overall semantic distribution.
Thus, $(\tilde{y}_{w},y_{l})$ refines the original training data $(y_{w},y_{l})$ for more focused learning, shifting the optimization process to concentrate on the most critical differences in preference data.
The benefits of the Bridging Phase are further analyzed in §5.2.
Finally, we use the new dataset $\tilde{\mathcal{D}}=\{x^{(i)},\tilde{y}_{w}^{(i)},y_{l}^{(i)}\}_{i=1}^{N}$ for subsequent training.

An alternative approach that attempts to enhance the correlation between the winning
and losing responses is to degenerate $y_{w}$ to $\tilde{y}_{l}$ via targeted modification and utilize $(y_{w},\tilde{y}_{l})$ as the preference pair.
Nevertheless, our ablation study in Table 3 reveals that LLMs encounter challenges with this inverse operation, leading to a notable decline in performance.

### 3.2 Modeling Phase

After the Bridging Phase, the token-level differences between $\tilde{y}_{w}$ and $y_{l}$ can be obtained through dynamic programming algorithms like Levenshtein Distance (Yujian & Bo, ).
As depicted in the lower part of Figure 1, these nuanced variations guide LLMs to prioritize the reinforcement of optimal actions while discouraging suboptimal ones within a single response.
However, our findings below indicate that DPO alone is insufficient for capturing the nuanced variations, highlighting the necessity for supplementary techniques to comprehensively model these correlations.

#### Alternative interpretation of DPO.

DPO (Rafailov et al., ) introduced a novel framework for optimizing the equivalent KL-constrained reward function as in RLHF, without the need to learn an explicit reward model.
Instead, the problem is cast as a maximum likelihood estimation for the policy model $\pi_{\theta}$ on the preference dataset $\mathcal{D}$, resulting in the following training objective:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(x,y_{w}, y_{l})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w} \mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi_{\theta}(y_{l}\mid x )}{\pi_{\text{ref}}(y_{l}\mid x)}\right)\right],$ (2) $$

where $\pi_{\text{ref}}$ is the reference model, typically the supervised fine-tuned (SFT)
model, and $\beta$ is a regularisation term corresponding to the strength of KL-regularization in RLHF.

As shown in Eq. (2), DPO was originally conceptualized as a bandit problem, where the whole response of the model is treated as a single arm to receive a reward.
More recently, Rafailov et al. () extended the theoretical foundation of DPO, showing that it can also be derived in the context of token-level MDP.
The corresponding training objective at the token level is:

$$ $\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})=-\mathbb{E}_{(\tau_{w} ,\tau_{l})\sim\mathcal{D}}\left[\log\sigma\left(\beta\sum_{t=0}^{N-1}\log\frac {\pi_{\theta}(a_{w}^{t}\mid s_{w}^{t})}{\pi_{\text{ref}}(a_{w}^{t}\mid s_{w}^{ t})}-\beta\sum_{t=0}^{M-1}\log\frac{\pi_{\theta}(a_{l}^{t}\mid s_{l}^{t})}{\pi _{\text{ref}}(a_{l}^{t}\mid s_{l}^{t})}\right)\right],$ (3) $$

where $\tau_{w}$ and $\tau_{l}$ denote the win trajectory and the lose trajectory, respectively. $a$ indicates the action (current generated token), and $s$ signifies the state (all tokens generated so far).

#### Our solution.

It can be inferred from Eq. (3) that DPO, redefined as a token-level MDP, assigns rewards to each token generation by $\beta\log\frac{\pi_{\theta}(a^{t}\mid s^{t})}{\pi_{\text{ref}}(a^{t}\mid s^{t})}$, and simply add up the rewards of all tokens as the accumulated reward of the trajectory.
This uniform aggregation assumes that each token contributes equally to the overall quality of the sequence, without considering the varying importance of each token (timestep).
Therefore, nuanced differences between $\tilde{y}_{w}$ and $y_{l}$ that significantly influence the overall meaning or quality of the response might not be adequately emphasized (refer to Figure 6), leading to suboptimal performance.
To this end, we propose to emphasize the rewards of critical tokens, i.e., nuanced differences between $\tilde{y}_{w}$ and $y_{l}$.
The magnitude of the emphasis is determined dynamically by the policy model’s confidence, which refers to the probability assigned to the generated token during training.
Below, we detail our design choices for the pseudo-winning response and losing response, respectively.

- •
For varied tokens in the pseudo-winning response $\tilde{y}_{w}$, we adapt the reward factor based on the learning process of the policy model.
Lower policy confidence indicates underdeveloped learning of the target behavior, signaling the need for additional focus to help the model better capture these nuances.
Consequently, we adjust the reward factor to be inversely proportional to the policy model’s confidence, as formalized in Eq. (7).
Figure 2: We aggregate varied tokens in $\tilde{y}_{w}$ or $y_{l}$ into more coarser-grained spans. During the DPO training on $\tilde{\mathcal{D}}$, we compute the averaged $-\log(p)$ of tokens in different positions of spans.
- •
For varied tokens in the losing response $y_{l}$, we carefully adjust the reward factor by reinforcing already learned patterns of the policy model.
Intuitively, tokens in $y_{l}$ with higher confidence from the policy model may reflect inaccurate preference learning and therefore warrant stronger penalization.
However, our analysis reveals a distinct pattern of the policy model when processing $y_{l}$ compared to $\tilde{y}_{w}$.
Specifically, when grouping varied tokens in $y_{l}$ into coarser-grained spans, the model’s confidence is significantly influenced by the token’s position within these spans, as illustrated in Figure 2.
We observe that the probabilities assigned to the initial token of incorrect spans in $y_{l}$ are typically low, whereas the probabilities for subsequent tokens within the same span are notably higher.
Prior studies have identified token probability as a critical signal for detecting anomalous behaviors (Xiao & Wang, ; Fadeeva et al., ) and assessing generation quality (Yuan et al., ; Fu et al., ).
Consistent with these findings, our results indicate that during training, the policy model can effectively recognize the onset of undesired spans by assigning low probabilities to initial tokens.
Nonetheless, due to the autoregressive dependencies, subsequent tokens within these spans receive higher probabilities, reflecting the contextual coherence established by preceding tokens, even when the span as a whole is incorrect.
Thus, while it is crucial to penalize initial tokens, applying equally strong penalties to subsequent tokens might be suboptimal, as they often maintain local coherence within the flawed span.
Therefore, we adjust the reward factor to also be inversely proportional to the policy model’s confidence in Eq. (10).

In a nutshell, our approach dynamically modulates the emphasis placed on critical tokens based on the policy model’s confidence.
This adaptive reward mechanism ensures that the model focuses on learning challenging distinctions while reinforcing already learned patterns, ultimately fostering a more nuanced and robust policy (see our analysis in §5.2).
The formalization of our approach is encapsulated in Eq. (4), where $\lambda_{\tilde{y}_{w}^{t}}$ and $\lambda_{y_{l}^{t}}$ adjust dynamically based on the policy’s confidence, ensuring a tailored emphasis on critical tokens to improve the overall model performance.

$$ $\displaystyle\mathcal{L}_{\text{DPO-BMC}}(\pi_{\theta};\pi_{\text{ref}})=- \mathbb{E}_{(x,\tilde{y}_{w},y_{l})\sim\tilde{\mathcal{D}}}\left[\log\sigma \left(\beta\sum_{\tilde{y}_{w}^{t}\in\tilde{y}_{w}}\lambda_{\tilde{y}_{w}^{t}} \log\frac{\pi_{\theta}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}{\pi_{\text{ ref}}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}\right.\right.$ $\displaystyle\left.\left.-\beta\sum_{y_{l}^{t}\in y_{l}}\lambda_{y_{l}^{t}} \log\frac{\pi_{\theta}(y_{l}^{t}\mid y_{l}^{<t},x)}{\pi_{\text{ref}}(y_{l}^{t} \mid y_{l}^{<t},x)}\right)\right],$ (4) $$

where

$$ $\displaystyle\lambda_{\tilde{y}_{w}^{t}}$ $\displaystyle=\left\{\begin{array}[]{ll}1+\min\left(\mathit{sg}\left(\frac{1}{ \pi_{\theta}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}\right),\delta\right), &\text{if }\tilde{y}_{w}^{t}\in\mathit{diff}(\tilde{y}_{w}\mid y_{l})\\ 1,&\text{otherwise}\end{array}\right.$ (7) $\displaystyle\lambda_{y_{l}^{t}}$ $\displaystyle=\left\{\begin{array}[]{ll}1+\min\left(\mathit{sg}\left(\frac{1}{ \pi_{\theta}(y_{l}^{t}\mid y_{l}^{<t},x)}\right),\delta\right),&\text{if }y_{l }^{t}\in\mathit{diff}(y_{l}\mid\tilde{y}_{w})\\ 1,&\text{otherwise}\end{array}\right.$ (10) $$

The $sg$ denotes the stop-gradient operator, the $\delta$ is an upper limit threshold that controls the emphasis on the rewards of the critical tokens, preventing overly aggressive updates. The $\mathit{diff}(\tilde{y}_{w}\mid y_{l})$ and $\mathit{diff}(y_{l}\mid\tilde{y}_{w})$ signify using the Levenshtein Distance algorithm to find the varied tokens in $\tilde{y}_{w}$ and $y_{l}$, respectively.
In Appendix C, we provide a gradient analysis of DPO-BMC.
Unlike DPO, our approach harmonizes both sequential and token-level perspectives, effectively optimizing the overall sequence structure alongside crucial token choices for the desired outcome.

## 4 Experimental Setup

We conduct a comprehensive evaluation across three downstream scenarios, including question answering (QA), mathematical reasoning, and instruction following (IF).
The detailed data statistics as well as the evaluation metrics are listed in Table 6 of Appendix A.1.

#### Models and training settings.

For the QA and mathematical reasoning setup, we utilize Llama2-7B-base (Touvron et al., ) in our experiments.
Dealing with these tasks necessitates LLMs to possess domain-specific knowledge and engage in systematic, step-by-step reasoning to reach the ultimate answer.
Therefore, following prior works (Chen et al., ; ), we fine-tune Llama2-7B-base on the training set of ECQA (Aggarwal et al., ) and QASC (Khot et al., ) for QA, and fine-tune Llama2-7B-base on MetaMathQA (Yu et al., ) for mathematical reasoning. We denote the fine-tuned LLM as SFT and use it as the backbone for preference optimization.
In line with prior research (Chen et al., ; ), we construct preference pairs $(y_{w},y_{l})$ based on the training data, by using the ground truth as $y_{w}$ and the SFT model’s inference output as $y_{l}$.

For the instruction following setup, we utilize Llama3-8B-base (Dubey et al., ) and Mistral-7B-Base (Jiang et al., ) in our experiments.
Following the training pipeline of Zephyr (Tunstall et al., ) and SimPO (Meng et al., ), we train a base model on the UltraChat-200k dataset (Ding et al., ) to obtain an SFT model.
Then, we use the SFT model as the starting point and perform preference optimization on the UltraFeedback dataset (Cui et al., ), where $y_{w}$ and $y_{l}$ are collected from LLMs of varying quality.

During our Bridging Phase, we utilize gpt-4-0125-preview for targeted modification to obtain $\tilde{y}_{w}$, based on the prompt template in Appendix A.2.
We also demonstrate in Appendix B that a less powerful LLM, such as Llama3-70B-Instruct, can acquire comparable results.
During our Modeling Phase, we list the implementation details in Appendix A.3 for reproducibility.

#### Evaluation benchmarks.

In question answering, we adopt the test splits of ECQA (Aggarwal et al., ), QASC (Khot et al., ), OpenbookQA (Mihaylov et al., ), and StrategyQA (Geva et al., ) for evaluation.
In mathematical reasoning, we conduct the evaluation on four challenge datasets including GSM8k (Cobbe et al., ), MATH (Hendrycks et al., ), MAWPS (Koncel-Kedziorski et al., ), and TabMWP (Lu et al., ).
In instruction following, We assess our models using two of the most popular open-ended instruction-following benchmarks: AlpacaEval 2 (Li et al., ) and Arena-Hard v0.1 (Li et al., ).
Both benchmarks evaluate the models’ versatile conversational abilities across a diverse set of queries.
For each query, the evaluated model’s response and the reference model’s response are
compared head-to-head using an auto-evaluator.
We use the officially recommended configurations(^1^11AlpacaEval: [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). Arena-Hard v0.1: [https://github.com/lm-sys/arena-hard-auto](https://github.com/lm-sys/arena-hard-auto).) during the evaluation.

#### Baselines.

We compare our approach with various powerful offline preference optimization methods, including FIGA (Guo et al., ), DPO (Rafailov et al., ), and DPO variants (IPO (Azar et al., ), KTO (Ethayarajh et al., ), ORPO (Hong et al., ), TDPO (Zeng et al., ), R-DPO (Park et al., ), and SimPO (Meng et al., )).
The training objectives of these methods are listed in Table 7.

## 5 Experimental Results

In this section, we present the main results of our experiments, showcasing the superior performance of our method across various benchmarks and ablation studies (§5.1).
Next, we conduct in-depth quantitative analyses to elucidate why our method outperforms DPO (§5.2 and §5.3).
Furthermore, we demonstrate the versatility of our framework by adapting it to other DPO variants (§5.4).

### 5.1 Main results and ablations

#### Our method consistently and significantly outperforms baselines.

As presented in Table 1, our model DPO-BMC consistently achieves state-of-the-art results across all evaluated QA and math benchmarks.
Specifically, DPO-BMC outperforms DPO by 3.8 absolute points on QA tasks and by 1.3 points on math tasks.
Additionally, on instruction-following tasks (refer to Table 2), DPO-BMC secures the highest length-controlled win rate, surpassing DPO by over 5 points across various settings.
The length-controlled win rate (Dubois et al., ) serves as a robust metric that mitigates the effects of length bias, thereby providing a more reliable evaluation of LLM-based auto-annotation.
Notably, DPO-BMC generates responses that are significantly more concise than other baselines.
As highlighted in Table 2, the average response length of DPO-BMC and DPO-BC is approximately 75% of that produced by DPO and DPO-MC.
This attribute of length normalization is credited to the correlated preference data we constructed, which directs optimization towards critical desired behaviors rather than verbosity.
A detailed case study, shown in Table 10, further underscores the effectiveness and robustness of our approach.

**Table 1: Experimental results (based on Llama2-7B-base) on question answering tasks and mathematical reasoning tasks. “Avg.” is the average accuracy of all sub-tasks. In each column, the highest score is bolded and the second-highest is underlined.**
| Method | Question-Answering Tasks | Mathematical Reasoning Tasks |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ECQA | QASC | OBQA | StrategyQA | Avg. | GSM8k | MATH | MAWPS | TabMWP | Avg. |  |
| SFT | 72.8 | 54.5 | 51.8 | 56.9 | 59.0 | 55.8 | 11.6 | 80.3 | 42.8 | 47.6 |
| FIGA | 70.3 | 52.5 | 51.7 | 48.6 | 55.8 | 54.1 | 9.8 | 75.5 | 39.0 | 44.6 |
| IPO | 71.5 | 58.9 | 53.6 | 58.4 | 60.6 | 57.2 | 12.1 | 82.2 | 42.5 | 48.5 |
| KTO | 71.6 | 56.5 | 52.8 | 56.0 | 59.2 | 56.4 | 12.0 | 80.1 | 42.5 | 47.8 |
| OPRO | 69.8 | 55.1 | 51.4 | 57.2 | 58.4 | 56.0 | 12.4 | 80.8 | 41.3 | 47.6 |
| TDPO | 74.4 | 58.2 | 54.8 | 59.1 | 61.6 | 57.3 | 12.2 | 81.6 | 43.2 | 48.6 |
| R-DPO | 73.5 | 59.5 | 55.4 | 58.8 | 61.8 | 56.9 | 12.0 | 81.9 | 42.2 | 48.2 |
| SimPO | 71.9 | 56.7 | 52.2 | 55.4 | 59.1 | 57.5 | 12.7 | 81.8 | 43.5 | 48.9 |
| DPO | 73.1 | 58.8 | 55.6 | 57.8 | 61.3 | 56.3 | 12.3 | 81.2 | 43.4 | 48.3 |
| DPO-BMC | 75.9 | 63.0 | 60.4 | 61.0 | 65.1 | 58.4 | 13.0 | 83.1 | 43.8 | 49.6 |
| DPO-BC | 75.7 | 62.0 | 56.0 | 60.1 | 63.4 | 57.6 | 12.7 | 82.8 | 43.4 | 49.1 |
| DPO-MC | 74.8 | 60.0 | 56.4 | 58.8 | 62.5 | 57.2 | 12.5 | 82.4 | 43.0 | 48.8 |

**Table 2: Experimental results on instruction-following tasks. “LC” is the length-controlled win rate, and “WR” is the raw win rate. “Avg. len” denotes the average number of tokens in the responses.**
| Method | Llama3-8B-Base | Mistral-7B-Base |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AlpacaEval 2 | Arena-Hard | AlpacaEval 2 | Arena-Hard |  |  |  |  |  |  |  |
| LC (%) | WR (%) | Avg. len | WR (%) | Avg. len | LC (%) | WR (%) | Avg. len | WR (%) | Avg. len |  |
| SFT | 7.5 | 4.7 | 956 | 2.6 | 414 | 8.1 | 5.9 | 998 | 2.2 | 454 |
| FIGA | 8.4 | 4.2 | 1,199 | 5.1 | 416 | 7.0 | 4.9 | 1,378 | 2.5 | 461 |
| IPO | 13.4 | 9.8 | 1,430 | 14.0 | 477 | 12.5 | 10.8 | 1,588 | 8.5 | 522 |
| KTO | 13.7 | 12.2 | 1,633 | 12.9 | 547 | 13.3 | 10.6 | 1,487 | 8.2 | 543 |
| ORPO | 12.5 | 11.4 | 1,793 | 11.7 | 573 | 14.5 | 11.5 | 1,630 | 9.4 | 566 |
| TDPO | 16.5 | 15.3 | 1,766 | 16.4 | 564 | 14.8 | 12.8 | 1,740 | 11.2 | 551 |
| R-DPO | 17.1 | 14.4 | 1,801 | 17.6 | 582 | 16.0 | 12.3 | 1,521 | 10.4 | 529 |
| SimPO | 21.3 | 18.9 | 1,718 | 26.6 | 562 | 16.8 | 14.4 | 1,906 | 18.4 | 615 |
| DPO | 16.0 | 14.8 | 1,713 | 17.6 | 559 | 15.1 | 13.3 | 1,657 | 13.6 | 540 |
| DPO-BMC | 22.4 | 16.8 | 1,285 | 18.1 | 406 | 20.8 | 16.6 | 1,317 | 17.6 | 488 |
| DPO-BC | 20.6 | 14.4 | 1,269 | 16.8 | 422 | 18.6 | 13.8 | 1,489 | 15.9 | 502 |
| DPO-MC | 17.7 | 15.2 | 1,890 | 17.9 | 579 | 16.4 | 14.3 | 1,712 | 15.4 | 551 |

#### Both key designs in BMC are crucial.

In Table 1 and Table 2, we additionally present results from ablating each key design element of DPO-BMC:

- •
DPO-BC: Training using DPO’s original objective on our constructed preference data.
- •
DPO-MC: Training using our proposed objective in Eq. (4) on the original preference data.

Our examination reveals several key findings: (1) enhancing the correlation between preference pairs remarkably boosts model performance; (2) even when using the same training preference data, our designed optimization objective outperforms DPO, highlighting its superior ability to model fine-grained correlations; and (3) combining our constructed data with our designed objective yields the best results, affirming the inseparability of the Bridging Phase and the Modeling Phase.

Figure: Table 3: Ablation study on diverse data synthesis methods in the Bridging Phase. The average accuracy is presented for QA and Math. LC on AlpacaEval 2 is reported for instruction following (IF), based on Llama3-8B.
Refer to caption: extracted/5912495/figures/ablation2.png

#### Influence of data synthesis method.

Table 3 shows the effects of various data synthesis strategies during the Bridging Phase.
When generating $\tilde{y}_{w}$ without referring to $y_{w}$, LLMs potentially make erroneous modifications that misalign with the intended target, leading to a performance drop.
An alternative approach that attempts to enhance the correlation between winning and losing responses is to degenerate $y_{w}$ to $\tilde{y}_{l}$ and utilize $(y_{w},\tilde{y}_{l})$ as the preference pair.
However, this approach also falls short, yielding significantly lower results.
Our analysis suggests that LLMs struggle with this inverse operation, resulting in $\tilde{y}_{l}$ failing to accurately capture the error patterns present in the original $y_{l}$.

#### Influence of δ 𝛿 \delta italic_δ .

We conduct an ablation study to examine the influence of the threshold $\delta$ in the DPO-BMC objective on model performance, as shown in Figure 3.
Setting $\delta=1.0$ reduces our method to one that assigns fixed token-level rewards, leading to suboptimal accuracy.
As $\delta$ increases, the model performance improves, with the optimal setting observed around $\delta=3.0$.
However, further increasing $\delta$ results may degrade model performance due to excessively aggressive gradient updates on certain tokens.
Notably, across all tested values of $\delta$, our method consistently outperforms the DPO baseline, indicating its robustness and effectiveness in stabilizing the learning process.

### 5.2 Quantitative analysis of bridging and modeling phase

To rigorously assess the effectiveness of the two pivotal phases in our framework, we segment the 60k training data of UltraFeedback into six equal-sized splits, ordered by increasing edit distance between winning and losing responses.
For each split, we also construct its corresponding

Figure: Figure 4: KL divergence from the policy model to the reference model on winning responses of the held-out set of UltraFeedback.
Refer to caption: extracted/5912495/figures/kl_div.png

$(\tilde{y}_{w},y_{l})$ pair data through our Bridging Phase.
We then train four models—–(a) DPO, (b) DPO-MC, (c) DPO-BC, and (d) DPO-BMC—on each split based on Llama3-8B, and report LC on AlpacaEval 2 and the average gradient norm during training.
All models are trained with identical hyperparameters to ensure consistency and comparability.
It can be observed from Figure 5 that the Bridging Phase successfully decreases the edit distance between pairwise data through targeted modification, shifting the optimization process to concentrate on the most critical differences in preference data.
The results underscore the efficacy of the Bridging Phase in consistently enhancing the performance across all splits, signifying its role in refining the training data for more focused learning.
Another notable observation is the behavior of the average gradient norm during the training of DPO, which exhibits an increasing trend as the edit distance between pairwise data enlarges.
This trend highlights the sensitivity of DPO’s training process to individual data points, which can potentially lead to a large variance of gradient norm and inefficient learning.
However, the proposed learning objective in our Modeling Phase effectively mitigates the variance through a dynamic training process that adjusts according to the policy model’s confidence.
This adaptive mechanism not only ensures that the model places greater emphasis on learning challenging distinctions but also reinforces the patterns that have already been acquired, culminating in a more nuanced and robust policy.
Such an approach is crucial in achieving a balanced optimization landscape with diverse training data, leading to superior generalization and performance in downstream tasks.

In Figure 4, we present the KL divergence between the policy model trained with DPO, DPO-MC, DPO-BC, and DPO-BMC with identical hyperparameters and the reference model, measured on the winning responses from a held-out set of UltraFeedback during training.
The results also validate our above analyses: (1) the Bridging Phase fosters tailored learning toward critical differences in preference data, resulting in more efficient and “sharp” training with a larger KL divergence; (2) our meticulously designed loss function in the Modeling Phase effectively moderates the optimization intensity across diverse training data, thereby achieving a more controlled and steady KL divergence.

Figure: Figure 5: We segment the 60k training data of UltraFeedback into six equal-sized splits based on increasing edit distance between winning and losing responses. For each split, we report LC on AlpacaEval 2 and the average gradient norm during training.
Refer to caption: x2.png

### 5.3 Quantitative analysis of credit assignment

We compare the token-level and sequence-level credits assigned by DPO and DPO-BMC, assessing how well their final learned rewards align with preference labels on a held-out set of UltraFeedback.

#### Analysis on token-level reward.

Figure 6 depicts the token-level reward assignment for DPO and DPO-BMC on a response pair consisting of a winning response $y_{w}$ and a losing response $y_{l}$.
The reward of each token is computed as $r_{\theta}(x,y^{t})=\beta\log\frac{\pi_{\theta}(y^{t}\mid y^{<t},x)}{\pi_{
\text{ref}}(y^{t}\mid y^{<t},x)}$.
From the figure, we observe that: (1) DPO assigns nearly uniform rewards across tokens, failing to differentiate the importance of tokens to the overall response quality; and (2) although DPO can identify and assign lower rewards to several erroneous tokens in the losing response (e.g., “13”), it struggles to capture subtle distinctions between the winning and losing responses.
In contrast, DPO-BMC assigns higher rewards to critical tokens (e.g., “descending order”) and effectively penalizes incorrect tokens in the losing response.
These results demonstrate DPO’s limitations in providing precise token-level preferences on sentence quality, and our method can effectively alleviate this issue.

Figure: Figure 6: Visualization of token-level rewards assigned by DPO and our method. The preference pair is sampled from the held-out set of UltraFeedback, whose input prompt is “Arrange the numbers 5, 13, 99, 1, and 22 in descending order. What is the first number in the new arrangement?”
Refer to caption: x3.png

#### Analysis on sequence-level reward.

For a rigorous comparison, we calculate the sequence-level DPO reward expression by $r_{\theta}(x,y)=\beta\log\frac{\pi_{\theta}(y\mid x)}{\pi_{\text{ref}}(y\mid x)}$ (with the partition function excluded) and present the results in Table 4.
The reward margin is determined by $r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})$.
Reward accuracy is defined as the percentage of preference pairs where the winning response achieves a higher reward than the losing response, i.e., $r_{\theta}(x,y_{w})>r_{\theta}(x,y_{l})$.
Our findings demonstrate that DPO-BMC outperforms DPO in terms of average reward margin and reward accuracy.
This enhancement validates the superior ability of our method to discern subtle differences between preference pairs, thereby facilitating more effective generalization.

**Table 4: Sequence-level reward margin and reward accuracy of DPO and DPO-BMC. The results are computed on the held-out validation set (2,000 samples) of UltraFeedback.**
| Method | Avg. Reward Margin | Reward Accuracy (%) |
| --- | --- | --- |
| DPO | 0.54 | 72.19 |
| DPO-BMC | 0.74 | 73.60 |

Figure: Table 5: Versatility of our framework across various XPOs. The average accuracy is presented for QA and Math, while LC on AlpacaEval 2 is reported for IF.

### 5.4 Versatility of our framework

Our BMC framework is versatile and can be applied to various DPO variants.
Table 5 shows the performance comparison between the original XPO and XPO-BMC methods.
Given that KTO only requires a binary signal to indicate whether an output is desirable or undesirable for a given input, and TDPO is based on token-level MDP, we exclude these from our consideration.
The results indicate that XPO-BMC variants generally surpass their corresponding XPO methods across various tasks.
Specifically, IPO-BMC, ORPO-BMC, R-DPO-BMC, and DPO-BMC achieve higher average accuracies in QA and Math, as well as superior LC scores on AlpacaEval 2 for IF.
However, SimPO-BMC does not outperform SimPO on the IF task, showing a slight decrease from 21.3 to 19.7.
These findings underscore the robustness and efficacy of our BMC approach in enhancing the performance of different DPO methods, affirming its potential for broad application.

## 6 Conclusion

In this work, we propose BMC, an effective framework for bridging and modeling correlations in pairwise data for direct preference optimization.
BMC equips LLMs with better human value alignment through a two-phase process: a Bridging Phase that enhances correlations between pairwise data by explicitly manifesting fine-grained preference signals via targeted modifications, and a Modeling Phase that learns token-level correlations by dynamically leveraging the the policy model’s confidence during training.
Our framework exhibits superior performance in question-answering, mathematical reasoning, and instruction-following tasks, consistently surpassing the baseline DPO by a significant margin.
Extensive analysis highlights that the key designs in BMC are crucial and validates the effectiveness and versatility of BMC.

## Appendices

## Appendix A Detailed Experimental Setup

### A.1 Data statistics and evaluation metrics used for experiments

We list the detailed data statistics and evaluation metrics of our experiments in Table 6.
Our experiments comprise both closed-ended evaluation (QA and math) and open-ended evaluation (instruction following).

**Table 6: Statistics of the training and evaluation datasets.**
| Task | Train / Test | Dataset | Number | Evaluation Metric |
| --- | --- | --- | --- | --- |
| QA | Train | ECQA | 7,598 |  |
| QASC | 8,134 |  |  |  |
| Test | ECQA | 2,194 | Accuracy |  |
| QASC | 926 |  |  |  |
| OBQA | 500 |  |  |  |
| StrategyQA | 687 |  |  |  |
| Math | Train | MetaMathQA | 40,000 |  |
| Test | GSM8k | 1,319 | Accuracy |  |
| MATH | 5,000 |  |  |  |
| MAWPS | 2,065 |  |  |  |
| TabMWP | 1,000 |  |  |  |
| IF | Train | UltraFeedback | 61,135 |  |
| Test | AlpacaEval 2 | 805 | Win rate against GPT-4 Turbo |  |
| Arena-Hard | 500 |  |  |  |

### A.2 Prompt template for targeted modification

We demonstrate the prompt template of targeted modification for question answering and mathematical reasoning tasks in Figure 7.
Since the SFT model has been fine-tuned on the ground truth $y_{w}$ for QA and math tasks, the inferred output $y_{l}$ may be quite approximate to $y_{w}$ in some circumstances.
Therefore, we require the off-the-shelf LLM to filter out preference pairs where $y_{l}$ is good enough.
Finally, we filtered out 31% data and 43% data for the QA task and math task, respectively.
Note that for the training data of our baselines like DPO, we also use the filtered $(y_{w},y_{l})$ pairs for a fair comparison.
For instruction-following tasks, the prompt template we use is shown in Figure 8.

Figure: Figure 7: Prompt template of targeted modification for question answering and mathematical reasoning tasks.
Refer to caption: x4.png

Figure: Figure 8: Prompt template of targeted modification for instruction-following tasks.
Refer to caption: x5.png

### A.3 Implementation details

Our implementation is based on the alignment-handbook repo(^2^22[https://github.com/huggingface/alignment-handbook](https://github.com/huggingface/alignment-handbook)) using 4×A800 GPUs.
To ensure a fair comparison, we conduct thorough hyperparameter tuning for all methods compared in our experiments.

#### SFT training hyperparameters.

We train SFT models using the following hyperparameters: a learning rate of 2e-5, a batch size of 128, a max sequence length of 2048, and a cosine learning rate schedule with 10% warmup steps.
For QA and instruction-following tasks, we train the model for 1 epoch, whereas for mathematical tasks, we extend the training to 2 epochs.
All the models are trained with an Adam optimizer (Kingma & Ba, ).

#### Preference optimization training hyperparameters.

During preference optimization, we performed initial experiments to determine the optimal batch sizes in [32, 64, 128] and training epochs in [1, 2, 3].
Our results indicate that using a batch size of 128 and a single training epoch consistently produces the best outcomes across all methods. Consequently, we adopted these parameters for all subsequent preference optimization experiments. We also configured the maximum sequence length to 2048 and employed a cosine learning rate schedule with a 10% warmup period for training on the preference optimization dataset.
For method-specific training hyperparameters, we individually search the learning rates in the range of [3e-7, 5e-7, 6e-7, 1e-6] for each method.
Besides, we conduct a grid search according to Table 7 and report the best performance.
Table 8 shows the hyperparameters of our method used under each setting.

**Table 7: Various preference optimization objectives and hyperparameter search range.**
| Method | Objective | Hyperparameter |
| --- | --- | --- |
| FIGA | $-\sum_{\tilde{y}_{w}^{t}\in\mathit{diff}(\tilde{y}_{w}\mid y_{l})}\alpha\log<br>\pi_{\theta}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)$ | $\alpha\in[0.5,1.0,1.5]$<br>$\beta\in[0.5,1.0,1.5]$ |
| $+\sum_{y_{l}^{t}\in\mathit{diff}(y_{l}\mid\tilde{y}_{w})}\beta\log\pi_{\theta}<br>(y_{l}^{t}\mid y_{l}^{<t},x)$ |  |  |
| IPO | $\left(\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}-\log\frac{<br>\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}-\frac{1}{2\tau}\right)^{2}$ | $\tau\in[0.01,0.1,0.5,1.0]$ |
| KTO | $-\lambda_{w}\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}<br>(y_{w}|x)}-z_{\text{ref}}\right)+\lambda_{l}\sigma\left(z_{\text{ref}}-\beta<br>\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right),$ | $\lambda_{l}=\lambda_{w}=1.0$<br>$\beta\in[0.01,0.05,0.1]$ |
| where $z_{\text{ref}}=\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\beta\text{KL}(\pi_{<br>\theta}(y|x)||\pi_{\text{ref}}(y|x))\right]$ |  |  |
| ORPO | $-\log p_{\theta}(y_{w}|x)-\lambda\log\sigma\left(\log\frac{p_{\theta}(y_{w}|x)<br>}{1-p_{\theta}(y_{w}|x)}-\log\frac{p_{\theta}(y_{l}|x)}{1-p_{\theta}(y_{l}|x)}<br>\right),$ | $\lambda\in[0.1,0.5,1.0,2.0]$ |
| where $p_{\theta}(y|x)=\exp\left(\frac{1}{|y|}\log\pi_{\theta}(y|x)\right)$ |  |  |
| TDPO | $-\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|<br>x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right.$ | $\alpha\in[0.1,0.3,0.5,0.7]$<br>$\beta\in[0.01,0.05,0.1]$ |
| $-\alpha\bigl{(}\beta D_{\text{SeqKL}}(x,y_{l};\pi_{\text{ref}}||\pi_{\theta})-<br>sg(\beta D_{\text{SeqKL}}(x,y_{w};\pi_{\text{ref}}||\pi_{\theta})\bigr{)}\Bigr<br>{)}$ |  |  |
| R-DPO | $-\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|<br>x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}-(\alpha|y<br>_{w}|-\alpha|y_{l}|)\right)$ | $\alpha\in[0.05,0.1,0.5,1.0]$ |
| $\beta\in[0.01,0.05,0.1]$ |  |  |
| SimPO | $-\log\sigma\left(\frac{\beta}{|y_{w}|}\log\pi_{\theta}(y_{w}|x)-\frac{\beta}{|<br>y_{l}|}\log\pi_{\theta}(y_{l}|x)-\gamma\right)$ | $\beta\in[2.0,2.5]$ |
| $\gamma\in[0.3,0.5,1.0,1.2,1.4,1.6]$ |  |  |
| DPO | $-\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|<br>x)}-\beta\log\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}\right)$ | $\beta\in[0.01,0.05,0.1]$ |
| DPO-BMC | $\log\sigma\left(\beta\sum_{\tilde{y}_{w}^{t}\in\tilde{y}_{w}}\lambda_{\tilde{y<br>}_{w}^{t}}\log\frac{\pi_{\theta}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}{<br>\pi_{\text{ref}}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}\right.$ | $\beta\in[0.01,0.05,0.1]$<br>$\delta\in[1.5,2.0,2.5,3.0,3.5]$ |
| $\,\,\,\,\,\,\,\,\,\,\,\left.-\beta\sum_{y_{l}^{t}\in y_{l}}\lambda_{y_{l}^{t}}<br>\log\frac{\pi_{\theta}(y_{l}^{t}\mid y_{l}^{<t},x)}{\pi_{\text{ref}}(y_{l}^{t}<br>\mid y_{l}^{<t},x)}\right),$ where |  |  |
| $\lambda_{\tilde{y}_{w}^{t}}=\left\{\begin{array}[]{ll}1+\min\left(\mathit{sg}<br>\left(\frac{1}{\pi_{\theta}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}\right)<br>,\delta\right),\text{if }\tilde{y}_{w}^{t}\in\mathit{diff}(\tilde{y}_{w}\mid y<br>_{l})\\<br>1,\text{otherwise}\end{array}\right.$ |  |  |
| $\lambda_{y_{l}^{t}}=\left\{\begin{array}[]{ll}1+\min\left(\mathit{sg}\left(<br>\frac{1}{\pi_{\theta}(y_{l}^{t}\mid y_{l}^{<t},x)}\right),\delta\right),\text{<br>if }y_{l}^{t}\in\mathit{diff}(y_{l}\mid\tilde{y}_{w})\\<br>1,\text{otherwise}\end{array}\right.$ |  |  |

**Table 8: Hyperparameter values for diverse training settings in DPO-BMC.**
| Task | Model | Learning Rate | $\beta$ | $\delta$ |
| --- | --- | --- | --- | --- |
| QA | Llama2-7B-base | 5e-7 | 0.05 | 3.0 |
| Math | Llama2-7B-base | 5e-7 | 0.05 | 2.5 |
| IF | Llama3-8B-base | 5e-7 | 0.01 | 2.0 |
| Mistral-7B-base | 5e-7 | 0.01 | 2.0 |  |

## Appendix B Impact of LLMs on Targeted Modification in the Bridging Phase

In Table 9, we explore the impact of LLMs on targeted modifications during the Bridging Phase.
Our findings indicate that substituting the gpt-4-0125-preview model with a less powerful yet open-source alternative, such as Llama3-70B-Instruct, yields comparable performance while significantly surpassing vanilla DPO.
Numerous studies have highlighted the superior text modification capabilities of LLMs. For example, LLMs have been effectively employed in synthesizing high-quality data (Wang et al., ; Liu et al., ).
Additionally, Ji et al. () show that LLMs can transform initial outputs from upstream models into more helpful and benign responses, thereby aligning generated content with human intentions.
In conclusion, our framework demonstrates robustness in leveraging diverse LLMs for targeted modifications, confirming its adaptability and effectiveness.

**Table 9: Ablation study on diverse LLMs for targeted modification in the Bridging Phase. The average accuracy is presented for QA and Math. LC on AlpacaEval 2 is reported for instruction following (IF), based on Llama3-8B.**
| Method | LLM for Targeted Modification | QA | Math | IF |
| --- | --- | --- | --- | --- |
| SFT | – | 56.9 | 47.6 | 7.5 |
| DPO | – | 61.3 | 48.3 | 16.0 |
| DPO-BMC | Llama3-70B-Instruct | 64.6 | 49.4 | 21.8 |
| DPO-BMC | gpt-4-0125-preview | 65.1 | 49.6 | 22.4 |

## Appendix C Gradient analysis

For a mechanistic understanding of our method, we examine the gradients of the loss function $\mathcal{L}_{\text{DPO}}$ in Eq. (2) and $\mathcal{L}_{\text{DPO-BMC}}$ in Eq. (4).
Their gradients with respect to the parameters $\theta$ can be written as:

$$ $\displaystyle\displaystyle\nabla_{\theta}\mathcal{L}_{\text{DPO}}(\pi_{\theta} ;\pi_{\text{ref}})=-\beta\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[ \sigma\left(\Delta_{1}\right)\left[\underbrace{\displaystyle\nabla_{\theta} \log\pi_{\theta}(y_{w}\mid x)}_{\text{increase likelihood of }y_{w}}- \underbrace{\displaystyle\nabla_{\theta}\log\pi_{\theta}(y_{l}\mid x)}_{\text{ decrease likelihood of }y_{l}}\right]\right],$ $$

where
$\Delta_{1}=\beta\log\frac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\text{ref}}(y_{l}
\mid x)}-\beta\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}$.

$$ $\displaystyle\displaystyle\nabla_{\theta}\mathcal{L}_{\text{DPO-BMC}}(\pi_{ \theta};\pi_{\text{ref}})=-\beta\mathbb{E}_{(x,\tilde{y}_{w},y_{l})\sim\tilde{ \mathcal{D}}}\left[\sigma\left(\Delta_{2}\right)\left(\underbrace{ \displaystyle\nabla_{\theta}\log\pi_{\theta}(\tilde{y}_{w}\mid x)}_{\text{ increase likelihood of }\tilde{y}_{w}}-\underbrace{\displaystyle\nabla_{\theta }\log\pi_{\theta}(y_{l}\mid x)}_{\text{decrease likelihood of }y_{l}}\right.\right.$ $\displaystyle\left.\left.+\underbrace{\sum_{\tilde{y}_{w}^{t}\in\mathit{diff}( \tilde{y}_{w}\mid y_{l})}(\lambda_{\tilde{y}_{w}^{t}}-1)\displaystyle\nabla_{ \theta}\log\pi_{\theta}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}_{\text{ increase likelihood of desired tokens of }\tilde{y}_{w}}-\underbrace{\sum_{y_{ l}^{t}\in\mathit{diff}(y_{l}\mid\tilde{y}_{w})}(\lambda_{y_{l}^{t}}-1) \displaystyle\nabla_{\theta}\log\pi_{\theta}(y_{w}^{t}\mid y_{l}^{<t},x)}_{ \text{decrease likelihood of undesired tokens of }y_{l}}\right)\right],$ $$

where
$\Delta_{2}=\beta\sum_{y_{l}^{t}\in y_{l}}\lambda_{y_{l}^{t}}\log\frac{\pi_{
\theta}(y_{l}^{t}\mid y_{l}^{<t},x)}{\pi_{\text{ref}}(y_{l}^{t}\mid y_{l}^{<t}
,x)}-\beta\sum_{\tilde{y}_{w}^{t}\in\tilde{y}_{w}}\lambda_{\tilde{y}_{w}^{t}}
\log\frac{\pi_{\theta}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}{\pi_{\text{
ref}}(\tilde{y}_{w}^{t}\mid\tilde{y}_{w}^{<t},x)}$.

In contrast to vanilla DPO, which emphasizes sequence-level optimization exclusively, our proposed method integrates both sequence-level and token-level perspectives.
(1) At the sequence level, we promote preferred completions while penalizing those that are disfavored.
(2) At the token level, we further refine the rewards of critical desired and undesired tokens of $\tilde{y}_{w}$ and $y_{l}$, respectively.
This dual consideration ensures that both the overall sequence structure and the critical token choices are optimized for the desired outcome.

## Appendix D Case Study

In this case study, we analyze the performance of various methods, all based on the Llama3-8B-base model, using a sample instruction from AlpacaEval 2.
Table 10 displays the responses generated by four different methods.
The SFT model produces a concise and factually accurate answer, although it lacks detail and context regarding the name change.
In contrast, models further trained with SimPO and DPO yield incorrect responses, asserting that Facebook Corporation did not change its legal name and providing an inaccurate account of the rebranding process.
Our proposed method, DPO-BMC, demonstrates a notable improvement by providing a response that is both factually accurate and comprehensive.
Additionally, the response generated by our method is significantly shorter than those produced by DPO and SimPO.
This combination of detail, accuracy, and brevity highlights the superiority of our approach in delivering informative and precise answers.

**Table 10: Generated responses from different methods for a sampled instruction in AlpacaEval 2.**
| Instruction | Did Facebook corporation change its name? |
| --- | --- |
| SFT | Yes, Facebook Corporation changed its name to Meta Platforms, Inc. In 2021. |
| SimPO | No, the Facebook Corporation did not alter its legal name. Despite the fact that the company has introduced a diverse array of products and services beyond its primary social networking platform, which includes popular applications such as Instagram, WhatsApp, Messenger, and various others, the official name of the corporation continues to be Facebook, Inc.<br>Nevertheless, in October 2021, Facebook made an announcement indicating their intention to modify their corporate brand name to better reflect their expanding range of offerings. As of now, the new brand name has not been disclosed to the public, and it is anticipated that the complete transition to this new name will require a significant amount of time. |
| DPO | No, Facebook Corporation did not change its name. Facebook Corporation, also known as Facebook, Inc., is the parent company of the Facebook social media platform, and it has not gone through any significant name change. The company holds other online platforms and services such as Instagram and WhatsApp, but the name Facebook Corporation remains the same. There have been some changes in the branding, such as the rebranding of WhatsApp and Facebook Messenger with the Meta logo, but the parent company’s name has not changed. The announcement made by Facebook in October 2021 about rebranding its parent company as Meta did not materialize, and Facebook Corporation still holds the name Facebook Corporation. |
| DPO-BMC | Yes, Facebook Corporation changed its name to Meta Platforms, Inc. In October 2021. The rebranding aims to reflect the company’s expanded focus beyond social media and into building the ”metaverse,” a virtual space where people can interact and communicate in immersive, 3D environments. The new name ”Meta” represents the company’s ambition to drive breakthroughs in areas such as virtual and augmented reality. Despite the name change, Facebook remains the name of the company’s flagship social network. |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See above
soundness: 2
strengths: - The idea of using commercial LLM to rewrite the winning response for DPO is interesting. 

- The experiments are extensive.
summary: The paper proposes an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. It leverages a commercial LLM to rewrite the winning response and defines confidence-based token-level weighting to improve the original DPO. The experiments are extensive.
weaknesses: - Regarding the motivation, the authors state that "yw and yl are generated isolatedly, the correlation between yw and yl can be weak during pairwise preference optimization", but this is not usually the case in practice. For example, a common practice is to let an LLM continue writing a prefix of the winning response to get the losing one. I mean, the correlation between these two responses can be easily established during data collection. 

- It is not well explained why "this weak correlation poses a challenge, as the winning response yw may not provide sufficiently informative gradients for adjusting the model’s parameters in relation to yl". This argument is too high-level. Can you delve into the details? Despite the results that have proven this point, I still wonder about the reason behind it. 

- The token confidence should not be always reliable and then the loss weighting trick hinges on intensive hyper-parameter tuning. This undermines the promise of the method. 

- To what extent does the final performance depend on the rewriting quality of the commercial LLM? Can some open-source ones be leveraged for this and what is their performance?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: * Regarding the first point in the weakness part, could the model still be able to learn some preferred linguistic aspect (such as organizing the answer with a CoT style) when the chosen & rejected responses overlap largely in terms of syntax? Does the shared linguistic structure matter or we can achieve a similar effect by identifying the token span that is critical to the semantic (which could be an improved version of DPO-MC with better token weighting criterion)?
* It would be interesting to see why LMs are bad at rephrasing a win response into a lose response. Do you have some observations other than Table 3, such as checking the difference of the result semantic equivalence between $(y_w, \tilde y_w)$ and $(y_l, \tilde y_l)$?
* Do you have some thoughts about why BMC can not boost SimPO?
soundness: 3
strengths: * This paper is written with high quality.
* The improvement to the current DPO-style alignment pipeline is well-motivated, with a simple and effective solution.
* The experiment part of this paper is well conducted, showing clear empirical significance with detailed ablation studies.
summary: This paper identifies the important role of the correlation between chosen & rejected responses for LLM alignment with DPO-style algorithms. They propose a suite of solutions including (1): rephrasing the chosen response into the style closer to the rejected response to enhance their correlation; (2): adding a token-level weighting in the DPO loss based on the policy model's confidence on the different tokens between the (rephrased) chosen and rejected responses. Extensive experimental results demonstrate that the proposed BMC pipeline could improve the model's performance on QA, math reasoning, and instruction following benchmarks, as well as exhibiting better and more reasonable token-level and sequence-level reward behaviors.
weaknesses: * Regarding the "Bridging Phase", I feel it does not necessarily address every aspect of the initial annotations. For example, despite the semantic aspects (i.e., the chosen response is correct while the rejected one is not), the preferred syntax, or linguistic aspects, such as how to better organize the response, might be neglected. If the original chosen response is better in terms of aspects other than correctness, then the Bridging phase may "erase" such useful information.
* The scalability of rephrasing every chosen response is somewhat limited, which can be improved by investigating the effect of executing the BMC for a specific portion of the dataset or considering how to identify the crucial tokens during the DPO training (i.e., based on confidence pattern demonstrated in line 228-245 and Sec 5.3), i.e., designing a better token-level weighting criterion for DPO-MC.
* I think the overall computation overhead introduced by the BMC pipeline should be discussed.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. **Cost-Benefit Justification**: Can the authors provide a more detailed analysis of the computational and resource costs associated with the Bridging Phase relative to the performance gains? How do the authors justify the added complexity and cost?

2. **Impact of Data Modification Proportion**: Did the authors investigate the relationship between the proportion of data modified in the Bridging Phase and the resulting performance? Is there a point of diminishing returns?

3. **Baseline LLM Limitations**: Is the modest performance improvement partly due to inherent limitations of the baseline LLM used in the authors' experiments? Would using a more capable baseline model lead to greater improvements, or are there fundamental constraints in the proposed approach? It would be beneficial to elaborate on the capabilities of larger baseline models as well, if possible.
soundness: 3
strengths: 1. **New Approach**: Introducing the Bridging Phase to enhance correlations in preference data is a new solution to a recognized problem in DPO methods.

2. **Dynamic Token-Level Adjustment**: Adjusting token rewards based on model confidence is a novel idea that could potentially improve fine-grained learning.

3. **Comprehensive Evaluation**: The authors conduct extensive experiments across multiple tasks, including question answering, mathematical reasoning, and instruction following.

4. **Versatility**: The framework is adaptable to various DPO variants, demonstrating its potential applicability in different contexts.
summary: The paper introduces a novel framework called **Bridging and Modeling Correlations (BMC)** to enhance Direct Preference Optimization (DPO) in aligning large language models (LLMs) with human preferences using pairwise preference data. The framework consists of two phases:

1. **Bridging Phase**: Enhances the correlation between winning and losing responses by generating a pseudo-winning response (ỹ_w) through targeted modifications of the losing response (y_l), referencing the winning response (y_w).

2. **Modeling Phase**: Improves token-level credit assignment by dynamically adjusting token rewards based on the policy model's confidence during training. This is formalized in their proposed objective function with adaptable weights.

Despite these new methods, the paper's experimental results show **modest performance improvements** compared to the computational and resource costs involved in the data modification process with stronger LLMs. I would like to understand these aspects further during the rebuttal period.
weaknesses: 1. **Marginal Performance Improvement vs. Cost**: The reported performance gains, while present, are relatively modest (e.g., 3.8 percentage points improvement in QA tasks), considering the significant effort and computational resources required for data synthesis and modification. Modifying 60-70% of the dataset introduces substantial overhead.

2. **Dependency on Large LLMs for Data Synthesis**: The Bridging Phase relies on powerful LLMs (e.g., GPT-4) for targeted modifications, which may not be feasible or cost-effective for many practitioners.

3. **Insufficient Analysis of Trade-offs**: The paper lacks a detailed discussion on the trade-off between the cost of data modification and the performance gains achieved.

4. **Limited Improvement Over Baselines**: Given the additional complexity and cost, the improvements over standard DPO and other baselines are insufficient to justify the added effort.

Meta-review: This paper presents a new framework for training models using DPO.  The core idea is to leverage large language models (LLMs) to refine responses, specifically by increasing the correlation between "winning" (preferred) and "losing" (less preferred) responses. This process enhances the training data and ultimately leads to improved model performance.  The authors demonstrate consistent improvements across various tasks.

Strengths:
- The proposed method is novel and the use of LLMs to manipulate response correlations is an innovative approach that yields performance gains.
- Additional results provided during the rebuttal phase, including experiments with different LLMs, showcase the generalizability of the method.

Weaknesses:
- While the LLM-based response regeneration can be performed offline, it still incurs a significant computational cost (over $200 for some datasets).
- The observed performance improvements are somewhat moderate.

Despite the identified weaknesses, the novelty and effectiveness of the proposed method warrant an acceptance recommendation.

Decision: Accept (Poster)