Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
Authors: Yuxin Jiang
Bo Huang
Yufei Wang
Xingshan Zeng
Liangyou Li
Yasheng Wang
Xin Jiang
Lifeng Shang
Ruiming Tang
Wei Wang
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
alignment
preference optimization
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose BMC, a novel framework for Bridging and Modeling Correlations in pairwise data, enabling more effective direct preference optimization.

Abstract: Direct preference optimization (DPO), a widely adopted offline preference optimization algorithm, aims to align large language models (LLMs) with human-desired behaviors using pairwise preference data. However, the generation of the winning response and the losing response within pairwise data are typically isolated, leading to weak correlations between them as well as suboptimal alignment performance. To address this issue, we propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. Firstly, we increase the consistency and informativeness of the pairwise preference signals through targeted modifications, synthesizing a pseudo-winning response by improving the losing response with the winning response as a reference. Secondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level correlations by dynamically leveraging the policy model's confidence during training. Comprehensive experiments on QA, math, and instruction-following tasks demonstrate the effectiveness of our approach, significantly surpassing competitive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants.

Published as a conference paper at ICLR 2025
BRIDGING AND MODELING CORRELATIONS IN PAIR- WISE DATA FOR DIRECT PREFERENCE OPTIMIZATION
Yuxin Jiang' 22 Bo Huang' 12, Yufei Wang", Xingshan Zeng². Liangyou Li Yasheng Wang', Xin Jiang", Lifeng Shang'. Ruiming Tang', Wei Wang 1.2
'The Hong Kong University of Science and Technology (Guangzhou) The Hong Kong University of Science and Technology 3 Huawei Noah's 's Ark Lab
yj yjiangemé connect. ust. hk, weiwes@ust. .hk
ABSTRACT
Direct preference optimization (DPO), a widely adopted offline preference opti- mization algorithm, aims to align large language models (LLMs) with human- desired behaviors using pairwise preference data. However, the generation of the winning response and the losing response within pairwise data are typically iso- lated, leading to weak correlations between them as well as suboptimal align- ment performance. To address this issue, propose effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. Firstly, we increase the consistency and informativeness of the pairwise preference sig- nals through targeted modifications, synthesizing a pseudo-winning response by improving the losing response with the winning response as reference. Sec- ondly, we identify that DPO alone is insufficient to model these correlations and capture nuanced variations. Therefore, we propose learning token-level corre- lations by dynamically leveraging the policy model's confidence during train- ing. Comprehensive experiments on QA, math, and mnstruction-folloiing tasks demonstrate the effectiveness of our approach, significantly surpassing compet- itive baselines, including DPO. Additionally, our in-depth quantitative analysis reveals the reasons behind our method's superior performance over DPO and showcases its versatility to other DPO variants. We release our repository at https://githinn com Y wwoom//Jjangemmm aaiinnngm//te. BMC.
INTRODUCTION
Direct preference optimization (DPO) (Rafailov et al., 2023) has emerged as prominent alternative to reinforcement learning from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a; Ouyang et al., 2022) for aligning large language models (LLMs) with human values. Unlike the traditional RLHF approach, DPO bypasses training a reward model and avoids using any reinforce- ment learning algorithms. Since the inception of DPO, numerous studies have sought to advance this method by refining its training objective (Wang et al., 2024). For instance, IPO (Azar et al., 2024) introduces an alternative pairwise preference loss to mitigate overfitting to the preference dataset, while R-DPO (Park al., 2024) incorporates regularization term to prevent the exploitation of latent length bias in the training data.
However, relatively little attention has been given to enhancing DPO through advancements in the quality of preference data used for training. In particular, the generation of winning and losing responses within preference data often occurs in an isolated manner, either through human anno- tation (Bai et al., 2022a) or automated techniques such as RLAIF (Bai et al., 2022b) and reject sampling (Liu et al., 2024a; Pace et al., 2024). This isolation implies that winning and losing re- sponses are produced without mutual visibility, resulting in a lack of strong correlation or relevance between them. Consequently, the model may struggle to identify nuanced yet significant distinctions that differentiate superior responses from inferior ones (Fürnkranz & Hüllermeier, 2010; Wirth et al., 2017), which can ultimately compromise optimization and alignment effectiveness.
Work partially done during the internship Huawei Noah's Ark Lab.

---

Published as a conference paper at ICLR 2025
In this work, we introduce innovative framework, termed BMC, to Bridge and Model Correla- tions in pairwise data for direct preference optimization. During the Bridging Phase, we enhance correlations by increasing the consistency and informativeness of pairwise preference signals. By us- ing the winning response as a reference, we synthesize a pseudo-winning response through targeted modifications of the losing response. This pseudo-winning response offers two key advantages: (1) it preserves essential characteristics of the losing response, minimizing noise in preference signals (consistency); (2) it encapsulates all human-desired values from the winning response, enabling the model to better discern features that lead to superior performance (informativeness). The nu- and losing responses are indeed what we expect the model to learn in the subsequent Modeling Phase. Nonetheless, we identify that DPO alone insufficient to model these correlations and capture nuanced variations. From the perspective of the token-level Markov Decision Process (MDP) (Rafailov et al., 2024), DPO aggregates rewards uniformly across all tokens, assuming equal contribution to sequence quality and neglecting token- specific importance. To address this, we adjust the emphasis on rewards of different tokens between bseudo-winning and losing responses. Unlike previous methods (Guo et al., 2024; Cao et al., 2024; Chan al., 2024; Chen et al., 2024a) that assign predefined values for fine-gra ine-grained guidance, our adjustment is dynamically guided by the policy model's confidence, i.e., the probability assigned to generated tokens during training. This ensures the model focuses learning challenging distinc- tions while reinforcing known patterns, resulting in a more nuanced and robust policy.
anced differences between the pseudo-winnin
We conduct extensive experiments across three downstream scenarios: question answering, mathe- matical reasoning, and instruction following, utilizing a total of 10 datasets. Our results demonstrate that our method consistently and significantly outperforms competitive offline optimization algo- rithms across various tasks. Furthermore, we use in-depth analyses to elucidate why our method outperforms DPO and show that our framework can be versatilely adapted to other DPO variants, confirming its potential for broad application.
2 RELATED WORK
Preference optimization. Preference optimization refers to aligning large language models with human preferences or specific desired outcomes. A well-established method for this is reinforce- ment learning from human feedback (RLHF) (Christiano et al., 2017; Bai et al., 2022a; Ouyang et al., 2022). Although RLHF produces highly capable models, its training process is often complex and unstable (Santacroce et al., 2023). To address these challenges, direct preference optimization (DPO) (Rafailov et al., 2023) introduces an alternative offline algorithm to optimize the regular- ized expected rewards without relying on RL. Building on DPO, subsequent methods like IPO(Azar et al., 2024) address overfitting to preference data, while R-DPO (Park et al., 2024) introduces length-based regularization mitigate exploitation.
Preference data construction. Constructing high-quality pairwise preference data is essential for preference optimization. Given the high cost of manually curating these datasets at scale, re- searchers have explored automated methods for producing preference data. One notable approach, RLAIF (Bai et al., 2022b) employs LLMs to label side-by-side response pairs, eliminating the need for human labeling. Alternatively, winning and losing responses can generated by utilizing mod- els of varying quality (Kim et al., 2023) through specific prompting techniques (Yang et al., methods such as Statistical Rejection Sampling (Liu et al., 2024a)
2023a). Recently, sampling-base and West-of-N (Pace et al., 2024) have been introduced, generating preference pairs by selecting can- didates sampled from the optimal policy. Nonetheless, these methods isolatedly generate winning and losing responses without accounting for the correlations between them.
Token-level preference optimization. The majority of preference optimization strategies typi- cally utilize trajectory-wise (sequence-level) rewards, while LM training and generation both occur at the token level (Yang et al., 2023b). To bridge this gap, FIGA (Guo et al., 2024) and DRLC (Cao et al., 2024) exploit external LLMs to pinpoint positive and negative token segments within re- sponses. assigning fixed reward values (e.g.. +1 for positive, -1 for negative) as guidance. Mean- while, ABC (Chan et al., 2024) and RLMEC (Chen et al., 2024a) extract fine grained credits from the reward model. Despite their contributions, these methods rely on predefined values for fine- grained guidance, failing account for the dynamic learning process of the policy model.
2

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
In this section, we present the proposed BMC approach, which bridges and models correlations in pairwise data for direct preference optimization. As depicted in Figure our BMC framework is structured around two pivotal stages: (1) the Bridging Phase, where we enhance the correlations between pairwise data by increasing the consistency and informativeness of pairwise preference signals through targeted modifications ($3.1); and (2) the Modeling Phase, where we dynamically model the correlations during the optimization process by leveraging the confidence of the policy model ($3.2), alleviating the insufficient token-level credit assignment of DPO.
Bridging Phase
He folt pride when he lookted his new mile what did do? Options: (A) compette egainst (3) bell many yeople about (C) Improve yourse! (D) pass cless
Feeling paide about new mile time suggests he worked on getting better and improved hinseeff.
y
was proud his now mile time. peide when he looked mile timo improved himself. The answer is
He was proud of passing a mile race. He fellt pride when he looked at his mile time he passed class. The answer is
Targeted
V1
Yw as a reference
modification
LLM
Modeling Phase
He felt poide when be looked at his new mile time
He was of
y w
y He was proud of (hi newj mill (ime He feet yride when be looked at
as he improved himsei ff The answer [
time as proved himselfi answer
Ottmmization
diff
Leveraging the policy model confidence
0.1 0.8
0.3
He of passing mile Iic y1 pride he looked new mile time as he passed class The answer's
proud passing
He felt poide when looked
Y1 his new mile time as be assed 0
0.2 0.8
D.6
The answer is
Figure 1: Overview of our proposed BMC framework. (1) In the Bridging Phase, we utilize an off- the-shelf LLM to make targeted modifications of losing response y undesired tokens, with the winning response Vh serving as a reference. Therefore, the synthesized pseudo-winning response Du is highly correlated with yr. (2) In the Modeling Phase, we model the correlations between fw and yr by dynamically emphasizing the rewards of their varied tokens ( ddiff (N y1) and diff (y Fw.), leveraging the policy model confidence (numbers indicated above tokens) during training.
3.1 BRIDGING PHASE
In offline preference optimization, it is commonly assumed that have access static pairwise preference dataset = {a((0, you Vi (0),, J where yu and y denote the winming and losing re- sponse, give the input prompt æ. However, since yu and y are typically generated in isolation, the correlation between y., and 3 can be inherently weak during pairwise preference optimization. In the context of DPO, the Bradley-Terry objective (Bradley & Terry, 1952) computes gradients based on the relative likelihoods of yo and y/- When the correlation between Yor and yt is weak, the dif- ferences between these responses are often superficial (e.g.. stylistic or irrelevant variations) rather than substantive distinctions that reflect human- preeerree behaviors. Consequently, the optimization process may inadvertently focus on minor discrepancies rather than meaningful distinctions. This results in gradients that are less informative for guiding the model towards robust preference align- ment. To address this challenge, we enhance the alignment efficacy by improving the consistency and informativeness of pairwise preference signals. As shown in the upper part of Figure 1, we utilize an off-the-shelf LLM to make targeted modification of yn by referring to yu:
LLM(1, x. Ber 91) + Tw:
(I)
where Ew is the generated pseudo-winning response, I is the instruction (see examples in Appendix A.2) that requires yt to be modified only on dispreferred tokens, using toe as reference guidance. In this way, The preserves essential characteristics of the losing response yr while encapsulating all
3

---

Published as a conference paper at ICLR 2025
human-desired values in the winning response Ve- The token-level differences between The and yt highlight the core human expected and unexpected behaviors by decoupling from the inherent linguistic style and overall semantic distribution. Thus, (Fw. V1) refines the original training data (Um-)t) for more focused learning, shifting the optimization process to concentrate on the most critical differences in preference data. The benefits of the Bridging Phase are further analyzed in $5.2. Finally, we use the new dataset D = {x(*), you Vi N. for subsequent training.
An alternative approach that attempts to enhance the correlation between the winning and losing responses is to degenerate Yo to ÿt via targeted modification and utilize (Ywr ÿt) as the preference pair. Nevertheless, our ablation study in Table reveals that LLMs encounter challenges with this inverse operation, leading to notable decline in performance.
3.2 MODELING PHASE
After the Bridging Phase, the token-level differences between The and yr can be obtained through dynamic programming algorithmss like Levenshtein Distance (Yujian & Bo, 2007). As depicted in the lower part of Figure these nuanced variations guide LLMs to prioritize the reinforcement of optimal actions while discouraging suboptimal ones within single response. However, our findings below indicate that DPO alone is insufficient for capturing the nuanced variations, highlighting the necessity for supplementary techniques to comprehensively model these correlations.
Alternative interpretation of DPO. DPO (Rafailov et al., 2023) introduced novel framework for optimizing the equivalent KL-constrained reward function as in RLHF, without the need to learn an explicit reward model. Instead, the problem cast as a maximum likelihood estimation for the policy model Ta on the preference dataset D, resulting in the following training objective:
( Ret)) -E (n....)))) log (Blog
(1e x)
Blog
To (y) x
(2)
(ye yw xr)
Te Vt
x))
where Tret is the reference model, typically the supervised fine-tuned (SFT) model, and is regu- larisation term corresponding to the strength of Kl-regularizaaion in RI.HE
As shown in Eq. (2), DPO was originally conceptualized as bandit problem, where the whole response of the model is treated as a single arm to receive a reward. More recently, Rafailov et al. (2024) extended the theoretical foundation of DPO, showing that it can also be derived in the context of token-level MDP. The corresponding training objective at the token level is:
Wa (a
log To (af ss) Tet (af sf)
Coro(To: Ret) = (Tw log
B
log
B
Σ t=0
Tea (at,
(3)
where Tu and 7 denote the win trajectory and the lose trajectory, respectively. a indicates the action (current generated token), and signifies the state (all tokens generated so far).
Our solution. It can be inferred from Eq. (3) that DPO, redetined as a token-level MDP, assigns
rewards to each token generation by 8 log mofa' e¹) 1o and simply add up the rewards of all tokens as the accumulated reward of the trajectory. This uniform aggregation assumes that each token con- tributes equally to the overall quality of the sequence, without considering the varying importance of each token (timestep). Therefore, nuanced differences between Vw and y that significantly influ- ence the overall meaning or quality of the response might not be adequately emphasized (refer to Figure 6), leading to suboptimal performance. To this end, we propose emphasize the rewards of critical tokens, i.e., nuanced differences between Yor and yr. The magnitude of the emphasis is determined dynamically by the policy model's confidence, which refers the probability assigned to the generated token during training. Below, we detail our design choices for the pseudo-winning response and losing response, respectively.
For varied tokens in the useudo-winning response Tw. adapt the reward factor based on the learning process of the policy model. Lower policy confidence indicates underdeveloped learning of the target behavior, signaling the need for additional focus to help the model better capture these nuances. Consequently, we adjust the reward factor to inversely proportional to the policy model's confidence, as formalized in Eq. (5).
4

---

Published as a conference paper at ICLR 2025
(a) Pseudo-winning response
(b) Losing response
15
15
13.79
1ogie
10
1ogle
1.23
1.11
1st
w/o 1st
all
1st
w/o 1st
all
Token position
Token position
Figure 2: We aggregate varied tokens in Iw or y into more coarser-grained spans. During the DPO training on D. we compute the averaged - (p) of tokens in different positions of spans.
For varied tokens in the losing response yt. we carefully adjust the reward factor by reinforcing already learned patterns of the policy model. Intuitively, tokens in y with higher confidence from the policy model may reflect inaccurate preference learning and therefore warrant stronger pe- nalization. However, our analysis reveals distinct pattern of the policy model when processing compared to Tu. Specifically, when grouping varied tokens in yı into coarser- coarscr-grained spans, the model's confidence significantly influenced by the token's position within these spans, as illustrated in Figure 2. We observe that the probabilities assigned to the initial token of incorrect spans in yt are typically low, whereas the probabilities for subsequent tokens within the same span are notably higher. Prior studies have identified token probability
critical signal for detect- ing anomalous behaviors (Xiao & Wang, 2021; Fadeeva et al., 2024) and assessing generation quality (Yuan et al., 2021; Fu et al., 2024). Consistent with these findings, that during training. the policy model can effectively recognize the onset of undesired spans by assigning low probabilities to initial tokens. Nonetheless, due the autoregressive dependencies, subsequent tokens within these spans receive higher probabilities, reflecting the contextual coher- ence established by preceding tokens, even when the span as whole is incorrect. Thus, while is crucial to penalize initial tokens, applying equally strong penalties to subsequent tokens might be suboptimal, they often maintain local coherence within the flawed span. Therefore, we adjust the reward factor to also be inversely proportional to the policy model's confidence in Eq. (6).
results indicate
In a nutshell, our approach dynamically modulates the emphasis placed on critical tokens based on the policy model's confidence. This adaptive reward mechanism ensures that the model focuses on learning challenging distinctions while reinforcing already learned patterns, ultimately fostering a more nuanced and robust policy (see our analysis in $5.2). The formalization of approach is encapsulated in Eq. (4), where Àge, and An adjust dynamically based on the policy's 's confidence, ensuring a tailored emphasis on critical tokens to improve the overall model performance.
LDPO-BMC(TE) Fett) - -B (w.
log σ
Σ Hh Ein
Ape, log
To (1th
Mi x)) 1.5t xx)
To yi xx)
3 Σ
Xpp log
(4)
meerlyi y1 ; .x)
st
of Em
where
Agg =
1 1 min
8g
Rotiill xx)
.6).
if jle diff (Tro (m) otherwise
(5)
Agg
1 1 min
sg
Roluf ui .x)
of E diff (w)
(6)
otherwise
The sg denotes the stop-gradient operator, the ò an upper limit threshold that controls the emphasis on the rewards of the critical tokens, preventing overly aggressive updates. The diff (2h. w)) and diff (yn Ew) signify using the Levenshtein Distance algorithm to find the varied tokens in Fa and yi. respectively. In Appendix C, we provide gradient analysis of DPO-BMC. Unlike DPO, our approach harmonizes both sequential and token-level perspectives, effectively optimizing the overall sequence structure alongside crucial token choices for the desired outcome.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTAL SETUP
We conduct a comprehensive evaluation across three downstream scenarios, including question an- swering (QA), mathematical reasoning, and instruction following (IF). The detailed data statistics as well as the evaluation metrics are listed in Table of Appendix A.1.
Models and training settings. For the QA and mathematical reasoning setup, we utilize Llama2- 7B-base (Touvron et al., 2023) in our experiments. Dealing with these tasks necessitates LLMs to possess domain-specific knowledge and engage in systematic, step-by-step reasoning reach the ultimate answer. Therefore, following prior works (Chen et al., 2024a:b), we fine-tune Llama2-7B- base on the training set of ECQA (Aggarwal et al., 2021) and QASC (Khot et al., 2020) for QA, and fine-tune Llama2-7B--bse on MetaMathQA (Yu et al., 2024) for mathematical reasoning. We denote the fine-tuned LLM as SFT and use as the backbone for preference optimization. In line with prior research (Chen et al., 2024a;b), we construct preference pairs (Ve, 31) based on the training data, by using the ground truth as Vu and the SFT model's inference output as yt- For the instruction following setup, we utilize Alama3-8B-base (Dubey et al., 2024) and Mistral-7B-Bar (Jiang et al., 2023a) in our experiments. Following the training pipeline of Zephyr (Tunstall et al., 2023) and SimPO (Meng et al., 2024), train base model on the UltraChat-298 hhocttt0kkkkk dataset (Ding et al., 2023) to obtain an SFT model. Then, we use the SFT model as the starting point and perform preference optimization the UltraFeedback dataset (Cui et al., 2023), where Yo and y collected from LLMs of varying quality.
for targeted modification to obtain Yu. based on the prompt template in Appendix A.2. We also demonstrate in Table 4 that a less pow- erful open-source I.M,, such
During Bridging Phase, utilize
can acquire comparable results. During our Modeling Phase, we list the implementation details in Appendix A.3 for reproducibility. comprehensive cost analysis in Appendix B confirms that the computational overhead intro- duced by our BMC pipeline is minimal.
Evaluation benchmarks. In question answering, we adopt the test splits of ECQA (Aggar- wal et al., 2021). QASC (Khot et al., 2020), OpenbookQA (Mihaylov et al., 2018). and Strate- gyQA (Geva et al., 2021) for evaluation. In mathematical reasoning, we conduct the evaluation on four challenge datasets including GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), MAWPS (Koncel- Kedzziork et al., 2016), and TabMWP (Lu et al., 2023). instruction following, We assess our models using two of the most popular open-ended instruction-followin benchmarks: AlpacaEvall 2 (Li et al., 2023) and Arena-Hard v0.1 (Li et al., 2024). Both benchmarks evaluate the models' versatile conversational abilities across a diverse set of queries. For each query, the evaluated model's 's response and the reference model's response are compared head-to-head using an auto-evaluator. We use the officially recommended configurations! during the evaluation.
Baselines. We compare our approach with various powerful offline preference optimization meth- ods, including FIGA (Guo et al., 2024), DPO (Rafailov et al., 2023), and DPO variants (IPO (Azar et al., 2024), ORPO (Hong et al., 2024), R-DPO (Park et al., 2024), and SimPO (Meng et al., 2024)). The training objectives of these methods are listed in Table 7. Besides,
include two additional baselines: (1) DPO (CW): enhancing pairwise data correlation by prompting the SFT model to Continue Writing a prefix of the winning response to generate the losing one; (2) DPO (EW): lever- aging an off-the-shelf LLM for External Weighting of token-level reward (Lee et al., 2024), where L.M scores each token in the winning and losing responses based on how much it improves or decreases the overall quality.
5 EXPERIMENTAL RESULTS
In this section, we present the main results of our experiments, showcasing the superior performance of our method across various benchmarks and ablation studies ($5.1). Next, conduct in in-depth quantitative analyses to elucidate why our method outperforms DPO ($5.2 and $5.3). Furthermore, demonstrate the versatility of our framework by adapting it to other DPO variants ($5.4).
AlpacaFval: https:://gttiei com/tat.u 1ab/ai http/alpaca_seee paca_eval, Aard v0.1: https: /// gi hhub. . www/mmmm////eeee.o wwwssss//aareaa htt.b..cmmmmmsss//rrreaaaaaaaaaa duauto

---

Published as a conference paper at ICLR 2025
Table 1: Experimental results (based (ama2-7B-base) on question answering tasks and mathe- matical reasoning tasks. "Avg." is the average accuracy of all sub-tasks. In each column, the highest score is bolded and the second-highest underlined.
Question-Answering Tasks
Mathematical Reasoning Tasks
Method
ECQA
QASC
OBQA
Strategy QA
Avg.
GSM8k
MATH
MAWPS
TabMWP
Avg.
SFT
72.8
54,5
51.8
56.9
59.0
55.8
11.6
80.3
42.8
47.6
FIGA
70.3
52.5
51.7
48.6
55.8
54.1
9.8
75.5
39.0
44.6
IPO
71.5
58.9
53.6
58.4
60.6
57.2
12.1
82.2
42.5
48.5
OPRO
69.8
55.1
$1.4
57.2
58.4
56.0
12.4
80.8
41.3
47.6
R-DPO
59.5
58.8
61.8
56.9
12.0
81.9
42.2
48.2
SimPO
71.9
56.7
52.2
55.4
59.1
57.5
12.7
81.8
43.5
48.9
DPO
73.1
58.8
55.6
57.8
61.3
56.3
12.3
81.2
43.4
48.3
DPO (CW)
72.5
58.6
55.2
57.3
60.9
55.9
11.8
80.7
42.8
47.8
DPO (EW)
72.9
59.4
55.8
57.9
61.5
56.5
12.0
80.9
43.4
48.2
DPO-BMC
75.9
63.0
60.4
61.0
65.1
58.4
13.0
83.1
43.8
49,6
DPO-BC 75.7
62.0
56.0
60.1
63.4
$7.6
12.7
82.8
43.4
49.1
DPO-MC 74.8
60.0
56.4
58.8
62.5
57.2
12.5
82.4
43.0
48.8
Table 2: Experimental results on instruction-folloiing tasks. "LC" is the length-controlles win rate, and "WR" is the raw win rate. "Avg. len" denotes the average number of tokens in the responses.
LJama3-8B-Base
Mistral-7Bo- BB-Baal
Method
AlpacaEval
Arena-Hard
AlpacaEvall 2
Arena-Hard
LC (%)
WR (%)
Avg. len
WR (%)
Avg. len
LCC (%)
WR (%)
Avg. len
WR (%)
Avg. len
SET
7.5
4.7
956
2.6
414
8.1
5.9
998
2,2
454
FIGA
8.4
4.2
1,199
5.1
416
7.0
4.9
1,378
2.5
461
13.4
9.8
1,430
14.0
ORPO
12.5
11.4
1,793
11.7
573
14.5
11.5
1,630
9.4
566
R-DPO
17.1
14.4
1,801
17.6
582
16.0
12.3
1,521
10.4
$29
SimPO
21.3
1,718
26.6
14.4
1,906
615
DPO
16.0
14.8
1,713
17.6
559
15.1
13.3
1,657
13.6
540
DPO (CW)
15.2
14.0
1,756
17.1
570
14.5
12,9
1,647
13.0
532
DPO (EW)
17.2
18.2
13.4
13.9
549
DPO-BMC
22.4
16.8
1,285
18.1
20.8
16.6
1,317
17.6
488
DPO-BC
20.6
14.4
1,269
16.8
422
18.6
13.8
1,489
15.9
502
DPO-MC
17.7
15.2
1,890
17.9
14.3
1,712
15.4
551
5.1
MAIN RESULTS AND ABLATIONS
Our method consistently and significantly outperforms baselines. As presented in Table 1, our model DPO-BMC consistently achieves state-of-the-an results across all evaluated QA and math benchmarks. Specifically, DPO-BMC outperforms DPO by 3.8 absolute points on QA tasks and by 1.3 points on math tasks. On instruction- instruction-folloiingg tasks (Table 2), DPO-BN BMC secures the high- est length-controlled win rate, surpassing DPO by over 5 points across various settings, with even greater gains for larger base models (Appendix E). The length-controlled win rate (Dubois et al., 2024) serves as a robust metric that mitigates the effects of length bias, thereby providing a more reliable evaluation of LLM-based based auto annotation..n. Notably. DPO-BMC generates responses that are significantly more concise than other baselines. As highlighted in Table 2, the average re- sponse length of DPO-BMC and DPO-BC is approximately 75% of that produced by DPO and DPO-MC. This attribute of length normalization is credited to the correlated preference data we constructed, which directs optimization towards critical desired behaviors rather than verbosity. A case study in Table 12 further underscores the effectiveness and robustness of our approach.
Both key designs in BMC are erucial. In Table I and Table 2, we additionally present results from ablating each key design element of DPO-BMC:
DPO-BC: Training using DPO's original objective on our constructed preference data.
DPO-MC: Training using our proposed objective in Eq. (4) the original preference data.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on diverse data synthe- sis methods in the Bridging Phase. The average accuracy is presented for QA and Math. LC on AlpacaEval 2 is reported for instruction following (IF), based on Llama3-8B.
30-
Data Synthesis
Training Data
QA
Math
IF
Bu (burs)
(Tw Mr)
65.1
49.6
22.4
(Iw: yn)
64.3
49.2
19.8
100% (MPG-8MCI
Modification Proportion
20%
(2/0-MC)
Vw 50
(Shwr (i)
64.6
48.7
18.9
Figure 3: Ablation study on data modification proportion in the Bridging Phase.
No - 5e
(3/wr Bi)
63.9
48.6
17.6
Our examination reveals several key findings: (1) DPO (CW), the "Continue Writing" approach, slightly underperforms standard DPO, as it introduces superficial correlations that fail to capture the nuanced, task task-specific aligmments essential for effective optimization. In contrast, our Bridg- ing Phase explicitly enhances informutive corvelations-elune
fine-grained distinctions between desired and undesired behaviors through token-level variations. This targeted focus significantly improves model performance; (2) Even when leveraging identical training preference data, de- signed optimization objective consistently outperforms both DPO and DPO (EW), highlighting its superior ability to model fine-grained correlations based the dynamic of the policy model's 's con- fidence; and (3) Combining our constructed data with our designed objective yields the best results, affirming the inseparability of the Bridging Phase and the Modeling Phase.
Influence of data synthesis method. Table 3 shows the effects of various data synthesis strategies during the Bridging Phase. When generating For without referring to Yw. LLMs potentially make erroneous modifications that misalign with the intended target, leading to performance drop. An alternative approach that attempts to enhance the correlation between winning and losing responses is to degenerate Ye to ÿ and utilize (yw, Ht) as the preference pair. However, this approach also falls short, likely because LLMs are primarily trained to generate high-quality data, making it challenging for them to generate low-quality outputs that mimic the nuanced errors of losing responses. Semantic embedding model? supports this, showing a
similarity analysis using the a high score of 0.88 for (yw, Fw) but only 0.73 for (y1, 9t).
Influence data modification proportion. Figure 3 illustrates the impact data modification proportions during the Bridging Phase on performance. Increasing modifications from 0% to 20% yields the most substantial gains, highlighting the effectiveness of enhancing pairwise preference correlations. Performance plateaus beyond 80% modifications, indicating that extensive changes are heneficial but not essential, offering flexibility under computational or data constraints. These results demonstrate the scalability and adaptability of our framework for diverse applications.
Influence of LLMs for targeted modification. Table 4 explores the influence of diverse LLMs for targeted modification. Notably, substituting the gpt-4-0125-prevv/ powerful yet open-source alternative, such as L1 amaam-3- 3-taa---0BB-- IIwttIIIInutttt... yields comparable per- formance while significantly surpassing vanilla DPO. This finding underscores the adaptability of our method to varying levels of model sophistication, thereby reducing dependence on commercial LLMs without significant impact on final model performance.
model with less
Influence of 8. We conduct ablation study to examine the influence of the threshold d in the DPO-BMC objective on model performance, as shown in Figure 4. Setting ò = 1.0 reduces our leading suboptimal accuracy. As 6 increases,
method to one that assigns fixed loken-leel the model performance improves, with the optimal setting observed around = 3.0. However, further increasing 8 results may degrade model performance due to excessively aggressive gradient updates on certain tokens. Notably, across all tested values of ō, our method consistently outperforms the DPO baseline, indicating its robustness and effectiveness stabilizing the learning process.
Pttpps: http:///uggggnngg co/ sentence trans former s/all/all -mpnet -iteetti base-v2
8

---

Published as a conference paper at ICLR 2025
Table 4: Influence of diverse LLMs for targeted modifica- tion in the Bridging Phase. The average accuracy is pre- sented for QA and Math. LC on AlpacaEval 2 is reported for instruction following (IF), based on Llama3-8B.
QA Fer Perfromance
64
63
Method
LLM for Targeted Modification
QA
Math
IF
1.0
1.5
2.5
3.0
3.5
SFT
7.5
DPO
61.3
48.3
Figure 4: Ablation study in the Modeling Phase. The average accuracy is presented the QA performance.
DPO-BMC
Llama 70B- Instract
64.6
49.4
21.8
DPO-BMC
301-4-0112-preview
65.1
49.6
22.4
Edit Distance
LC (%)
Grad Norm
Ov
Edit Distance
LC (%)
Grad Norm
3.51
0,70
4.85
12.40
spit
4.36
13.27
3p
(a) DPO
(b) DPO-MC
(8)
().>)))
Distance
spit
0.45
10.2
3.47
0.45
11.21
5.26
spit 2
0.52
1087
4.30
spR
0.52
11499
7.30
spi 3
1356
12.54
5.20
spR
1156
11.47
7.70
spit
D.6
14.34
5.39
sp8-4
D161
14.38
8.17
spit.5
0.70
13.24
6.00
SpRt
0.00
15.28
7.05
spit
0/84
10399
9.57
3pR6
0.04
12:29
0.75
[c)
(d)
Figure 5: We segment the 60k training data of UltraFeedback into six equal-sized splits based increasing edit distance between winning and losing responses. For each split, we report LC on AlpacaEval 2 and the average gradient norm during training.
5.2 QUANTITATIVE ANALYSIS OF BRIDGING AND MODELING PHASE
To rigorously assess the effectiveness of the two pivotal phases in our framework, we segment the 60k training data of UltraFeedback into six equal-sized splits, ordered by increasing edit distance between winning and losing responses. For each split, we also construct its corresponding (Tur 31 pair data through our Bridging Phase. We then train four models- (a) DPO, (b) DPO-MC, (c) DPO-BC, and (d) DPO-BMC on each split based on Llama3-8B, with identical hyperparameters to ensure comparability. As shown in Figure 5, the Bridging Phase successfully decreases the edit distance between pairwise data through targeted modification, shifting the optimization process to concentrate on the most critical differences in preference data. This phase consistently enhances performance across all splits by refining training data for more focused learning. Another notable observation is the average gradient norm during DPO training increases as the edit distance between pairwise data enlarges, reflecting the sensitivity of DPO's training process to individual data points and potential gradient variance. Our proposed Modeling Phase mitigates the variance by dynami- cally adjusting the training process based on the policy model's confidence. This adaptive mech- anism prioritizes challenging distinctions while reinforcing learned patterns, promoting balanced optimization landscape with diverse training data (See Appendix D for further analysis).
5.3 QUANTITATIVE ANALYSIS OF CREDIT ASSIGNMENT
We compare the token-level and sequence-level credits assigned by DPO and DPO-BMC, assessing how well their final learned rewards align with preference labels held-out set of UltraFeedback.
Analysis on token-level reward. Figure depicts the token-level reward assignment for DPO and DPO-BMC on response pair consisting of winning response yu and a losing response yi-
The reward of each token is computed as ro(2. y²) = log (y" ly St
x. " "y From the figure, we observe that: (1) DPO assigns nearly uniform rewards across tokens, failing to differentiate the importance of tokens to the overall response quality; and (2) although DPO can identify and assign
lower rewards several erroneous tokens in the losing response (e.g., "13"). it struggles capture

---

Published as a conference paper at ICLR 2025
winning response
winning response
arranged
descending
The nummbers arrenged ii
descending order
mmxi
89
and
The Arst
number
the
now
first
arrangement
99
arrangement
99
response
losing response
Rew
The numbers arranged
descanding order
would
The numbers arranged
descondinn order would
So
first
number
Ute
So
(he
lirs:
number
dhe
attengement
engeement
-0.25
(a) DPO
(b) DPO-BMC
Figure 6: Visualization of leken-level rewards assigned by DPO and our method. The preference pair is sampled from the held-out set of Uttraeedbbcck, whose input prompt is "Arrange the numbers 13, 99, and 22 descending order. What is the first number the new arrangement? ?"
subtle distinctions between the winning and losing responses. In contrast, DPO-BMC assigns higher rewards to critical tokens (e.g., "descending order") and effectively penalizes incorrect tokens in the losing response. These results demonstrate DPO's limitations in providing precise token-level preferences on sentence quality, and our method can effectively alleviate this issue.
Analysis on sequence-level reward. For a rigorous comparison, we calculate the sequence- Te(y(x)
level DPO reward expression by ro(o, v) = 3log ))((y()) The reward margin determined by the percentage of preference pairs where the
(2 Mw) re 31). Reward accuracy is defined winning response achieves a higher reward than the losing response, i.e., ( Bv.) > ro(2; yr). Our findings show that DPO-BMC outperforms DPO in terms of average reward margin (0.74 vs. 0.54) and reward accuracy (73.60 vs. 72.19). This enhancement validates our method's superior ability to discern subtle differences between preference pairs, enabling more effective generalization.
5.4 VERSATILITY OUR FRAMEWORK
Our BMC framework demonstrates versatility and can be seamlessly integrated with various DPO variants. As shown in Table 5, the XPO-BMC methods consistently outperform their corresponding XPO baselines across a diverse set of tasks, including QA, Math, and Instruction Following (IF).
Table 5: Versatility of our framework across various xPOs..
Method
Math
SET
56.9
47.6
For instance, PPO-BMC achieves a significant improvement in QA accuracy (64.1 60.6) and IF score (15.7 vs. 13.4) compared to IPO. Similarly. ORPO-BMC, R-DPO-BMC, SimPO-BMC. and DPO-BMC exhibit higher performance in QA and Math, alongside notable gains in IF, such as R- DPO-BMC improving the IF score from 17.1 to 20.0 over R-DPO. These results highlight the robustness of our frame- work enhancing task-specific performance across vari- ous settings, reaffirming its potential as generalizable en- hancement to existing DPO methodologies.
IPO
60.6
48.3
13.4
IPO-BMC
48.6
ORPO
58.4
47.6
ORPO-BMC
62.3
48.4
15.7
R-DPO
61.8
48.2
17.1
D.O..
65.3
49.5
20.0
SimPO
59.1
48.9
21.3
SimPO BMC
61.6
49.0
21.9
DPO
61.3
48.3
16.0
BMC
CONCLUSION
In this work, we propose BMC, an effective framework for bridging and modeling correlations in pairwise data for direct preference optimization. BMC equips LLMs with better human value align- ment through a two-phase process: a Bridging Phase that enhances correlations between pairwise data by explicitly manifesting fine-grained preference signals via targeted modifications, and Mod- eling Phase that learns token-level correlations by dynamically leveraging the the policy model's confidence during training. Our framework exhibits superior performance in question-answentt mathematical reasoning, and cestruction-folloiiig tasks, consistently surpassing the baseline DPO by significant margin. Extensive analysis highlights that the key designs in BMC are crucial and validates the effectiveness and versatility of BMC.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Wei Wang was supported by the Guangdong Provincial Key Laboratory of Integrated Communica-
tion, Sensing, and Computation for Ubiquitous Internet of Things (Grant No. 2023B 1203B1222012 (212010007), the Guangzhou Municipal Science and Technology Project (Grant Nos. 2023 2023A03J0013, and 2024A03J0621J and the Institute of Education Innovation and Practice Project (Grant Nos. G0IRF000012 and G01RF000017).
REFERENCES
Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for commonsenseqa: New dataset and models. In Annual Meeting the Association for Computational Linguistics, pp. 3050-3065, 2021. https:
/ //aclanthooloy..o.oo 22tp://hooooooo.oo02201001211111 acl-long) accp://.....002....1oo.oo....... 238.
Mohammad Gheshlaghi Azar, Zhaoban Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. general theoretical paradigm to understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, volume 238, pp. 4447-4455, 2024. URL bhesslaahi-azarre ht..all..................
mlr. mml..ppees//233)
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. In arXiv, 2022a. URL https: //atxiv.org//aaaaaiaee a20/aa/aa.22.... 05B62.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from feedback. arXiv, 2022b. URL https: /tapss///axvvv.// wwtp:/i.wogg/a/////22// 08073.
Ralph Allan Bradley and Milton E Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika. 39(3/4):324 3454,,44,515 1952.
Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. Dric: Re- inforcement learning with dense rewards from Ilm critic. In arXiv, 2024. URL https: 07382.
Alex J Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. Dense reward for free in rein- forcement learning from human feedback. In arXiv, 2024. URL /atp:i///axvv.e/a org/obggg//... abs/ 2402. 00782.
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen. Improving large language models via fine-grained reinforcement learning with minimum editing constraint. Findings of the Association for Computational I Linguistics, pp. 5694-5711, 2024a. URL httph:ttp///.. acc anthology 2224 fi findin iningg- aas-incss.......a acl.3.3.3 338..
Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. Low-redundant optimization for large language model alignment. arXiv, 2024b. URL https://arxiv. orq/abs/200 6406. 12606.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing 2017.
Systems, pp.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. In arXiv, 2021. URL https: //arxiv. www.vvorg/aaae aaaaab//22 /aab//211... 14168.
Ganqu Cui, Lifan Yuan. Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. In arXiv, 2023. URL https: http///axxvvvve/ wbs 013.000177.

---

Published as a conference paper at ICLR 2025
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by sealing figh-quality instructional conversa- tions. In Conference Empirical Methods in Natural Language Processing, pp. 3029-3051, 2023. URL https:
anthoo logthooogyyoooooooooo.ooooooooot wtt..orgg/23333 emn p ma iiaiin.. 183//.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. In arXiv, 2024. URL htwp:: wwwp.arxivvv.... org/ abs/ 2407 21783.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. In arXiv, 2024. URL https: / //arxiv.org//////...../oo..... http:/abb/244..0000
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the output of large language models via token- level uncertainty quantification. Findings of the Association for Computational Linguistics, pp.
9367-9385, 2024. URL https: //aalaaaahhooooyyo..aaoa 2024 E findings-acl. 558/.
Jinlan Fu, Kee-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 6556-6576, 2024. URL bettss://ccaattiie
2zzg/22244 maacl-lont 365/.
Johannes Fürnkranz and Eyke Hüllermeier. Preference learning and ranking by pairwise compari- 8on. In Preference learning, pp. 65-82. Springer, 2010.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguisties, 9:346-361, 2021. URL https: //aclanthol orgg tacl 1itel-1.21..
Geyang Guo, Ranchi Zhao, Tianyi Tang. Xin Zhao, and Ji-Rong Wen. Beyond imitation: Lever- aging fine-grained quality signals for alignment. In International Conference Learning Representations, 2024. URL https: //penrevie/eee.viiee. net/ www..eeeeoooummm--L1L..eCC.C
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Advances in Neural Information Processing Systems, 2021.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. In arXiv, 2024. URL https: peattts://axxiie http///axxvv.r////./ abs///ss///// 2403. 07691.
Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang. Aligner: Achieving efficient alignment through weak-to-strong correction. In arXiv, 2024. URI. ht.ts:::::/ arxiv. org/ alss 2242. 02416 6.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. wttg/ab////........ 2230 06825.
Mistral 7b. In arXiv, 2023a. URL
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distilla- tion of proprictary large language models. In Conference Empirical Methods in Natural Language Processing, pp. 3154-3154, 2023b. URL https: htt///aaaanatcccca org/2023. emnlp-maing in. 189/.
Yuxin Jiang, Yufei Wang, Chuhan Wu, Wanjun Zhong, Xingshan Zeng, Jiahui Gao, Liangyou Li, Xin Jiang. Lifeng Shang, Ruiming Tang. Qun Liu, and Wei Wang. Learning edit: Aligning Iims with knowledge editing, In Annual Meeting of Association for Computational Linguistics, pp. 4689 4705, 2024a. URL. ht /aacaaaatthhoo wow.oooooooo...oooooooooooo....o wrwp/oo2/222222 112.4..cl-11 wwwp///.................. 258/.
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine- ine-grained constraints following benchmark for large language models. In Annual Meeting of Association for Computational Linguistics, pp. 4667-4688, 2024b. URL pttep:://aclattholoooo .org/ 2024.a8 acl.acll long. 2577.
12

---

Published as a conference paper at ICLR 2025
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. QASC: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8082-8090, 2020. URL https:/ v34105 6.19.
Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. In Conference Empirical Methods in Natural Language Processing, pp. 13677-13700, 2023. URL https:
/ //aclanthology org/ 2023. emnlp-maiie maan. 844.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In arXiv, 2014. URL
abs/ 1412. 6980.
Rik Koncel-Kedziorski, Subhro Roy. Aida Amini, Nate Kushman, and Hannanch Hajishirzi. MAWPS: math word problem repository. In Conference of the North American Chapter of the Association for Computational Linguistics, pp. 1152-1157, 2016. URL https:// aclanthology wwg.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In International Conference on Machine Learning, 2024. URL https: / /owpnroveewew.oce/oo//p/p...ewwe net forum?id=
uyd02w41KO.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data high-quality benchmarks: The arena-hard pipeline, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-followinn models, 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In International Conference on Learning Representations, 2024a. URL https: hatp::://penreeeee wpppreeeeeeieveee//e/ee net/ forum? ?idd= xb SSwrreee
Yilun Liu, Shimin Tao, Xiaofeng Zhao, Ming Zhu, Wenbing Ma, Junhao Zhu, Chang Su, Yutai Hou, Miao Zhang, Min Zhang, Hongxia Ma, Li Zhang, Hao Yang, and Yanfci Jiang. Coachlm: Automatic instruction revisions improve the data quality in Ilm instruction tuning. In International Conference on Data Engineering, pp. 5184 5197, 2024b.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference Learning Representations, 2023, URL attps:://ot //pprr///peeeeveew net/ 2eet//ddd?----iI/BBJJJ.. DHYHRBBJUTH.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In arXiv, 2024. URL htt.httts//axxiiii oorggii.ooo....oooooooooooo abs /2005.2 14734.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct elec- tricity? A new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, pp. 2381 2381-2391, 2018. URL https: http://aaaaaatthttoo/oo org/
D18-1260.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In Advances in neural information processing systems, pp- 27730-27744, 2022. URL https: http:///peeeeeiieee/eeeeeeeeeeee
Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. West-of- Synthetic preference generation for improved reward modeling. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024. URL https:
7/ //opereeview.. net. forum? w 1d=7kNWZhMefs.
13

---

Published as a conference paper at ICLR 2025
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In arXiv, 2024. URL 19159.
wrg.i.o..ii...........ooo..o..rt abs 2400.
Rafael Rafailov, Areaii Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In Advances in Neural Information Processing Systems, 2023.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q Your language model is secretly q-function. In arXiv, 2024. URL
aasaab///o /404 01.4441.2355 12358.
Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient rlhf: Reducing the memory usage of ppo. In arXiv, 2023. URL htt.t:t:/// htt.ttts:: http:::///axxxvvoo/// wwwp://iioor//s///////////// /2399 00754.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: / /qwenlm.. htt.buu..o/bllg//ppooop/ qwwne. 5/.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. In arXiv, 2023. URL httpt:t/t///...... /apps///axivvv aarxiv.ivvvv org ab3/0 09288.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Di- reet distillation of alignment. In arXiv, 2023. URL 4.
/ttp///axvvvv.// org/ abs/231/
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Annual Meeting of Association for Computational Linguistics, pp. 13484-13508, 2023. URL https://ach anthalathooooo oogy- otgg/0233 acl long. 754.
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment tech- niques: Rlhf, rlaif, ppo. dpo and more. In arXiv, 2024. URL pestps://arxte wwt.vborrg/abba abs/ 2407.162.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürmkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1 2017.
Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in condi- tional language generation. Conference of the European Chapter of the Association for Computational Linguistics, pp. 2734-2744 2021. URL https://aclanthh oorgggggo/ooooooooooooooo...o...
2021. waeeeaclmae 236/.
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Reinforcement learning from contrast distillation for language model alignment. In arXiv, 2023a. URL https:
/ //arxiv.org/ab/i abs /2377.127707..
Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference-grounded token-level guidance for language model fine-tuning. In Advances Neural Information Processing Systems, 2023b.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. In International Conference on Learning Representations, 2024. URL https: htttp:://peereet http//ppeeevieeeeeeeeeeeeeeeeeee net / forum? wtwprummid-nnnnhhhe N8ChhNNC..
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text gen- eration. In Advances in Neural Information Processing Systems, pp. 27277, 2021. URL https:/ openreview .et/ forrmmidddaao.oo.... 5y
Li Yujian and Liu Bo. A normalized levenshtein distance metric. IEEE transactions on pattern
analysis and machine intelligence, 199 2007.
14

---

Published as a conference paper at ICLR 2025
APPENDICES
A DETAILED EXPERIMENTAL SETUP
A.1 DATA STATISTICS AND EVALUATION METRICS USED FOR EXPERIMENTS
We list the detailed data statistics and evaluation metrics of our experiments in Table 6. Our experi- ments comprise both closed-ended evaluation (QA and math) and open-ended evaluation (instruction following).
Table 6: Statistics of the training and evaluation datasets.
Task
Train Test
Dataset
Number
Evaluation Metric
ECQA
7,598
Train
QASC
8,134
QA
ECQA
2,194
QASC
926
Test
Accuracy
OBQA
500
Strategy QA
687
Train
MetaMathQA
40,000
GSM8k
1,319
Math
MATH
5,000
Test
Accuracy
MAWPS
2,065
TabMWP
1,000
Train
UltraFeedback
61,135
IF
AlpacaEival
805
Test
Win rate against GPT-4 Turbo
Arena-Hard
500
A.2 PROMPT TEMPLATE FOR TARGETED MODIFICATION
We demonstrate the prompt template targeted modification for question answering and mathemat- ical reasoning tasks in Figure 7. Since the SFT model has been fine-tuned the ground truth yo for QA and math tasks, the inferred output z/ may quite approximate to Yo in some circumstances. Therefore, require the off- shel LLM to filter out preference pairs where y is good enough. Finally, we filtered out 31% data and 43% data for the QA task and math task, respectively. Note that for the training data of baselines like DPO, also the filtered (yue 3/1 pairs for fair comparison. For instruction-following tasks, the prompt template we use is shown in Figure 8.
A.3 IMPLEMENTATION DETAILS
Our implementation is based on the hiigment-handdboo repo* using 4xA800 GPUs. To ensure fair comparison, we conduct thorough hyperparameter tuning for all methods compared in our experiments.
SFT training hyperparameters. We train SFT models using the following hyperparameters: learning rate of 2e-5, a batch size 128, a max sequence length 2048, and a cosine learning rate schedule with 10% warmup steps. For QA and instruction-folloiing tasks, we train the model for epoch, whereas for mathematical tasks, we extend the training to epochs. All the models are trained with an Adam optimizer (Kingma & Ba, 2014).
During preference optimization, we per- formed initial experiments determine the optimal batch sizes in [32, 64, 128] and training epochs
Preference optimization training hyperparameters.
Phtpps://oo thub. www.huggiiggfacceeeeeeooooo / al iggeentttteeee -aaeeennt-andooo
15

---

Published as a conference paper at ICLR 2025
Prompt Template for Question Answering and Mathematical Reasoning Tasks
**Task:*
Given the Problem, the Correct Answer, and the Prediction, identify and correct any mistakes in the Prediction align the Correct Answer. Three rules you obey:
minimal modifications necessary (changing fewest words) correct the Prediction. Only output the complete Corrected Prediction without saying anything else.
3. If the Prediction is already good enough, simply output 'None'.
**Problem:**
(x)
**Correct *
(w
**Prediction:"*
(vr)
*orrrccte *
Figure 7: Prompt template of targeted modification for question answering and mathematical rea- soning tasks.
Prompt Template for Instruction Following Tasks
Task:**
Below question followed by two responses. Response more preferred by humans than Response 2. Please make the minimal necessary changes to Response to improve it, referring to Response 1. Maintain as much the correct parts of Response as possible. Output only the complete Revised Response
**Problem:**
(x)
*"Response (w
*Response 2:**
(yr)
*Revised Response 2:**
Figure Prompt template targeted modification for instruction-follwwii tasks.
in [1, 2, 3]. Our results indicate that using batch size of 128 and single training epoch consis- tently produces the best outcomes across all methods. Consequently, we adopted these parameters for all subsequent preference optimization experiments. We also configured the maximum sequence length to 2048 and employed a cosine learning rate schedule with 10% warmup period for training on the preference optimization dataset. For method-specine training hyperparameters, ually search the learning rates in the range of [3e-7, 5e-7, 6e-7, le-6] for each method. Besides, conduct grid search according Table and report the best performance. Table 8 shows the hyperparameters of our method used under each setting.
individ-
16

---

Published as a conference paper at ICLR 2025
Table 7: Various preference optimization objectives and hyperparameter search range.
Method
Objective
Hyperparameter
St. (o) log ( 15" steeu (ailbw) 3 log mo(yf Vi "1""," ,z]
00.5, 1.0, 1.5) 8 [0.5, 1.0, 1.5]
FIGA
IPO
(xile) log xaluula)) log (x)lz)
[0.01, 0.1, 0.5, 1.0
lo Mhe Alogo (log T-polBule) log Pe(prla)
ORPO
I-Paltn
A A (0.1, 0.5, 1.0,
where (o(()))) exp (T log inggooull (((())))
R-DPO
log (B
(Blog
nolvullet ke)) marlyulls
log matwrlt) mar(snl))
(o) Mull -
0.055, 0.1, 0.5, 1.0) e (0.01, 0.05, 0.1]
SimPO
log
8 [io]]
log Th (eo(Wll)) (n)
Tun log To( (u)) -7)
8 22.0, 2.5]
7 (0.3, 0.5, 1.0, 100
DPO
log
log nstvulks
log martvvl
matarls)
8 0.01,0.05,0.11
log σ
Eni ii eău Aga. log
mulil 1os (e))
Tellbe molaf
B Σ vie Aut log
where
| [0.01, 0.05, 0.1]
DPO-BMC
+ min 8g
5) if the e diff ())
8 E [1.5, 2.0, 2.5, 3.5
Age-
motbl IIS
1. otherwise
1 + min
sg Tata
if of e diff ūw)
Agt
1, otherwise
Table 8: Hyperparameter values for diverse training settings in DPO-BMC.
Task
Model
Learning Rate
B
5
QA
Llama2-7B-base
Se-7
0.05
3.0
Math
Llama2-7B-bras
5e-7
0.05
2.5
Llama3-8B-be
5e-7
0.01
2.0
IF
Mistral-7B-baat
5e-7
0.01
2.0
B COST ANALYSIS
B.1 COST OF BRIDGING PHASE
The Bridging Phase, responsible for synthesizing pseudo-winning responses, operates exclusively offline, meaning it incurs runtime cost during model training. The data synthesis process is designed to efficient, as it does not require iterative computations or model updates.
For context, estimated the budget for data synthesis using the based on the API's pricing of $0.01 per 1K input tokens and $0.03 per 1K output tokens. Table lists the breakdown of the estimated costs for our three evaluated tasks, which demonstrates that this a manageable expenditure.
API,
Table 9: Estimated budget for data synthesis using the 1ppt-40011 gptt-4-0125-preve API.
Task
# of Samples
Avg. Input Token Length
Avg.
Output Token Length
Cost ($)
QA 15,732
206
25
44.21
Math 40,000
429
47
228.00
IF
61,135
728
235
876.06
17

---

Published as a conference paper at ICLR 2025
Can an Open-Source LLM be Utilized Alternative? In Table 4, we explore the impact of LLMs on targeted modifications during the Bridging Phase. Our findings indicate that substi- tuting the
model with less powerful yet open-source alternative, such as I.l ama.3 --70----0----- 1o----nntrrteeeeeee--nn--eee ruct, yields comparable performance while significantly surpassing model can be deployed on only NVIDIA-3090 GPUs, with the option to further reduce hardware requirements through low-bit quantization", This provides an economical alternative for our Bridging Phase without compromising performance. Nu- merous studies have highlighted the superior text modification capabilities of LLMs. For example, LLMs have been effectively employed in synthesizing high-quality data (Wang et al., 2023; Jiang et al., 2023b; 2024a;b; Liu et al., 2024b). Additionally. Ji et al. (2024) show that LLMs can trans- form initial outputs from upstream models into more helpful and benign responses, thereby aligning generated content with human intentions. In conclusion, our framework demonstrates robustness in leveraging diverse LLMs for targeted modifications, confirming its adaptability and effectiveness.
vanilla DPO. The 1.1 Mama--30n-Inst-eee
B.2
COST OF MODELING PHASE
Our Modeling Phase adds minimal computational overhead compared vanilla DPO. Specifically:
Token Difference Identification: Using a dynamic programming algorithm (edit distance) to identify differing tokens between the pseudo-winning and losing responses. This is a lightweight operation and introduces negligible runtime cost.
Reward Weighting Calculation: We calculate a weighting factor based the policy model's probability of the identified tokens, which is already computed in the standard DPO setup. Be- cause we halt gradient backpropagation for the weighting factor, this operation does introduce additional computational costs.
Table 10 demonstrates the comparison of the training times between DPO and DPO-BMC on 4xA800 GPUs, illustrating that DPO-BMC increases training time by less than % across all evaluated tasks.
Table 10: Runtime usage for DPO and DPO-BMC during the Modeling Phase.
Task
Base Model
Runtime of DPO (s)
Runtime of DPO-BMC (s)
Increase (%)
QA
Llama2-7B
2,831
2,850
0.67%
Math
Llama2-77
9,586
9,641
0.57 %
IF
Llama3-8B
16,179
16,318
0.86% %
Overall, these results validate that the computational overhead introduced by BMC is minimal, and the approach is highly efficient in terms runtime, making it practical for real-world applications without significantly increasing resource requirements.
'https:// http://gggtuubb.ooooooo /ol
18

---

Published as a conference paper at ICLR 2025
C GRADIENT ANALYSIS
For a mechanistic understanding of our method, we examine the gradients the loss function LDO
in Eq. (2) and LDPO-BM. BMC in Eq. (4). Their gradients with respect to the parameters can be written as:
VoLDrO(TO; Wret) (2, wn (A1)
Vo To increase likelihood of the
Vo log mo (ytt (x)
decrease likelihood of yi
where A =Blog [2(1))))
log
(twll))) Tia [Vol(W/) [x]
-3B V VoLDFO.RMC(TA) Tret)
(a DDD o (A2)
V log (F x))
Va log To (3
increuse likelihood of gi
decrease likelihood gr
+
(gg - 1) log (e to aas,aa
Σ
(N - 1)Vo To 11
11r1 x0)
st. Ediff (8ma lun)
E Edilff (arel Bww)
increase likelihood
of Dh
decreuse likelihood undesired tokens of yr
where A2 = = EBE Em Auj log (wr(
3 DEe Ein Age, log
(V 1o
( 180
In contrast to vanilla DPO, which emphasizes sequence-level optimization exclusively, our pro- posed method integrates both sequence-level and token-level perspectives. (1) At the sequence level, we promote preferred completions while penalizing those that are disfavored. (2) At the token level, we further refine the rewards of critical desired and undesired tokens of f.. and yı, respec- tively. This dual consideration ensures that both the overall sequence structure and the critical token choices are optimized for the desired outcome.
D KL DIVERGENCE ANALYSIS DURING TRAINING
In Figure 9, we present the KL divergence between the policy model trained with DPO, DPO-MC, DPO- BC, and DPO-BMC with identical hyperparameters and the reference model, measured on the winning responses from held-out set of UltraFeedback dur- ing training. The results also validate our analyses in $5.2: (1) the Bridging Phase fosters tailored leamn- ing toward critical differences preference data, re- sulting in more efficient and "sharp" training with larger KL divergence; (2) our meticulously designed loss function in the Modeling Phase effectively mod- erates the optimization intensity across diverse train- ing data, thereby achieving more controlled and steady KL divergence.
0.1
DFO
DFO.MC
0.3
DPQ-SC
KL divergeenc on
DPO-9MC
0.2
0.1
0.0
Training step
Figure 9: KL divergence from the policy model to the reference model on winning sponses of the held-out set of UltraFeedback.
E EXPERIMENTS ON LARGER BASE MODELS
We conducted additional experiments using the more capable Qwen2.5-14PP 1---2-------Bass model (Team, 2024). As shown in Table 11, proposed DPO-BMC method delivers even greater performance improvements with this model, achieving a remarkable gain of +8.8 on AlpacaEval 2 and +7.3 on Arena-Hard compared to standard DPO. These results highlight the effectiveness of DPO-BMC scales with model capability, underscoring its potential to deliver even larger gains when applied to more powerful baseline models.
19

---

Published as a conference paper at ICLR 2025
Table 11: Performance comparison across different base models.
Method
Base Model
LC (%) of AlpacaEval 2
WR (%) of Arena-Hard
SFT
Mistral-7B-Ban
8.1
2.2
DPO
Mistral-7B-Batt
15.1
13.6
DPO-BMC
Mistral-7B-BB
20.8 (+5.7)
17.6 (+4.0)
SFT
Llama3-8B-Ban
7.5
2.6
DPO
Llama3-8B-Bann
16.0
17.6
DPO-BMC
Llama3-8B Baamaa-&B-Base
22.4 (+6.4)
18.1 (+0.5)
SFT
Bren2.5-14B-BB.
14.1
11.2
DPO
Swen2.5-14BB-ae.
36.3
33.6
DPO-BMC
Qwen2.5-144 1--B--Base
45.1 (+8.8)
40.9 (+7.3)
CASE STUDY
In this case study, we analyze the performance of various methods, all based on the Llama3-8B- model, using sample instruction from AlpacaEval 2. Table 12 displays the responses generated by four different methods. The SFT model produces concise and factually accurate answer, although lacks detail and context regarding the name change. In contrast, models further trained with SimPO and DPO yield incorrect responses, asserting that Facebook Corporation did not change legal name and providing inaccurate account of the rebranding process. Our proposed method, DPO- BMC, demonstrates a notable improvement by providing a response that is both factually accurate and comprehensive. Additionally, the response generated by method is significantly shorter than those produced by DPO and SimPO. This combination of detail, accuracy, and brevity highlights the superiority of our approach in delivering informative and precise answers.
Table 12: Generated responses from different methods for a sampled instruction in AlpacaEval 2.
Instruction
Did Facebook corporation change its name?
SFT
Yes, Facehook Corporation changed its name to Meta Platforms, Inc. In 2021.
No, the Facebook Corporation did not alter legal name. Despite the fact that company has introduced a diverse array of products and services beyond its primary social networking platform, which includes popular applications such as Instagram, WhatsApp. Messenger, and various others, the official name of the corporation continues be Facebook, Inc.
SimPO
Nevertheless, October 2021, Facebook made announcement indicating their intention to modify their corporate brand name to better reflect their expanding range offerings. As of now, the new brand name has not been disclosed to the public, and it is anticipated that the complete transition this new name will require significant amount of time.
No, Facebook Corporation did not change its Facebook, Inc., the parent company of the Facebook social media platform, and not gone through any significant name change. The company holds other online platforms and services such as Instagram and WhatsApp. but the name Facebook Corporation remains the same. There have been some changes in the branding, such as the rebranding of What- sApp and Facebook Messenger with the Meta logo, but the parent company's name has changed. The announcement made by Facebook October 2021 about rebranding its parent company as Meta did not materialize, and Facebook Corporation still holds the name Face- book Corporation.
Facebook Corporation, also known
DPO
Yes, Facebook Corporation changed its name to Meta Platforms, Inc. In October 2021. The rebranding aims to reflect the company's 's expanded focus beyond social media and into building the "metaverse," a virtual space where people can interact and communicate in immersive, 3D environments. The new name "Meta" represents the company's ambition to drive breakthroughs in areas such as virtual and augmented reality. Despite the name change, Facebook remains the of the company's flagship social network.
DPO-BMC
20


Reviews:
[{"review_id": "REtJkS3A0L", "reviewer": "Reviewer_pbH6", "summary": "The paper proposes an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. It leverages a commercial LLM to rewrite the winning response and defines confidence-based token-level weighting to improve the original DPO. The experiments are extensive.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of using commercial LLM to rewrite the winning response for DPO is interesting. \n\n- The experiments are extensive.", "weaknesses": "- Regarding the motivation, the authors state that \"yw and yl are generated isolatedly, the correlation between yw and yl can be weak during pairwise preference optimization\", but this is not usually the case in practice. For example, a common practice is to let an LLM continue writing a prefix of the winning response to get the losing one. I mean, the correlation between these two responses can be easily established during data collection. \n\n- It is not well explained why \"this weak correlation poses a challenge, as the winning response yw may not provide sufficiently informative gradients for adjusting the model’s parameters in relation to yl\". This argument is too high-level. Can you delve into the details? Despite the results that have proven this point, I still wonder about the reason behind it. \n\n- The token confidence should not be always reliable and then the loss weighting trick hinges on intensive hyper-parameter tuning. This undermines the promise of the method. \n\n- To what extent does the final performance depend on the rewriting quality of the commercial LLM? Can some open-source ones be leveraged for this and what is their performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HHXDtJx26b", "reviewer": "Reviewer_tYGL", "summary": "This paper identifies the important role of the correlation between chosen & rejected responses for LLM alignment with DPO-style algorithms. They propose a suite of solutions including (1): rephrasing the chosen response into the style closer to the rejected response to enhance their correlation; (2): adding a token-level weighting in the DPO loss based on the policy model's confidence on the different tokens between the (rephrased) chosen and rejected responses. Extensive experimental results demonstrate that the proposed BMC pipeline could improve the model's performance on QA, math reasoning, and instruction following benchmarks, as well as exhibiting better and more reasonable token-level and sequence-level reward behaviors.", "questions": "* Regarding the first point in the weakness part, could the model still be able to learn some preferred linguistic aspect (such as organizing the answer with a CoT style) when the chosen & rejected responses overlap largely in terms of syntax? Does the shared linguistic structure matter or we can achieve a similar effect by identifying the token span that is critical to the semantic (which could be an improved version of DPO-MC with better token weighting criterion)?\n* It would be interesting to see why LMs are bad at rephrasing a win response into a lose response. Do you have some observations other than Table 3, such as checking the difference of the result semantic equivalence between $(y_w, \\tilde y_w)$ and $(y_l, \\tilde y_l)$?\n* Do you have some thoughts about why BMC can not boost SimPO?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* This paper is written with high quality.\n* The improvement to the current DPO-style alignment pipeline is well-motivated, with a simple and effective solution.\n* The experiment part of this paper is well conducted, showing clear empirical significance with detailed ablation studies.", "weaknesses": "* Regarding the \"Bridging Phase\", I feel it does not necessarily address every aspect of the initial annotations. For example, despite the semantic aspects (i.e., the chosen response is correct while the rejected one is not), the preferred syntax, or linguistic aspects, such as how to better organize the response, might be neglected. If the original chosen response is better in terms of aspects other than correctness, then the Bridging phase may \"erase\" such useful information.\n* The scalability of rephrasing every chosen response is somewhat limited, which can be improved by investigating the effect of executing the BMC for a specific portion of the dataset or considering how to identify the crucial tokens during the DPO training (i.e., based on confidence pattern demonstrated in line 228-245 and Sec 5.3), i.e., designing a better token-level weighting criterion for DPO-MC.\n* I think the overall computation overhead introduced by the BMC pipeline should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HEXKVuCyNj", "reviewer": "Reviewer_WKKA", "summary": "The paper introduces a novel framework called **Bridging and Modeling Correlations (BMC)** to enhance Direct Preference Optimization (DPO) in aligning large language models (LLMs) with human preferences using pairwise preference data. The framework consists of two phases:\n\n1. **Bridging Phase**: Enhances the correlation between winning and losing responses by generating a pseudo-winning response (ỹ_w) through targeted modifications of the losing response (y_l), referencing the winning response (y_w).\n\n2. **Modeling Phase**: Improves token-level credit assignment by dynamically adjusting token rewards based on the policy model's confidence during training. This is formalized in their proposed objective function with adaptable weights.\n\nDespite these new methods, the paper's experimental results show **modest performance improvements** compared to the computational and resource costs involved in the data modification process with stronger LLMs. I would like to understand these aspects further during the rebuttal period.", "questions": "1. **Cost-Benefit Justification**: Can the authors provide a more detailed analysis of the computational and resource costs associated with the Bridging Phase relative to the performance gains? How do the authors justify the added complexity and cost?\n\n2. **Impact of Data Modification Proportion**: Did the authors investigate the relationship between the proportion of data modified in the Bridging Phase and the resulting performance? Is there a point of diminishing returns?\n\n3. **Baseline LLM Limitations**: Is the modest performance improvement partly due to inherent limitations of the baseline LLM used in the authors' experiments? Would using a more capable baseline model lead to greater improvements, or are there fundamental constraints in the proposed approach? It would be beneficial to elaborate on the capabilities of larger baseline models as well, if possible.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **New Approach**: Introducing the Bridging Phase to enhance correlations in preference data is a new solution to a recognized problem in DPO methods.\n\n2. **Dynamic Token-Level Adjustment**: Adjusting token rewards based on model confidence is a novel idea that could potentially improve fine-grained learning.\n\n3. **Comprehensive Evaluation**: The authors conduct extensive experiments across multiple tasks, including question answering, mathematical reasoning, and instruction following.\n\n4. **Versatility**: The framework is adaptable to various DPO variants, demonstrating its potential applicability in different contexts.", "weaknesses": "1. **Marginal Performance Improvement vs. Cost**: The reported performance gains, while present, are relatively modest (e.g., 3.8 percentage points improvement in QA tasks), considering the significant effort and computational resources required for data synthesis and modification. Modifying 60-70% of the dataset introduces substantial overhead.\n\n2. **Dependency on Large LLMs for Data Synthesis**: The Bridging Phase relies on powerful LLMs (e.g., GPT-4) for targeted modifications, which may not be feasible or cost-effective for many practitioners.\n\n3. **Insufficient Analysis of Trade-offs**: The paper lacks a detailed discussion on the trade-off between the cost of data modification and the performance gains achieved.\n\n4. **Limited Improvement Over Baselines**: Given the additional complexity and cost, the improvements over standard DPO and other baselines are insufficient to justify the added effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)