On Extending Direct Preference Optimization to Accommodate Ties
Authors: Jinghong Chen
Guangyu Yang
Weizhe Lin
Jingbiao Mei
Chenxu Lyu
Bill Byrne
Venue: NeurIPS 2025 poster ()
Keywords: Direct Preference Optimization
Bradley-Terry Extensions
Ties
Primary area: deep_learning
TL;DR: We derive and show the effectiveness of two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons.

Abstract: We derive and investigate two DPO variants that explicitly model the possibility of declaring a tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments in neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without  the degradation in task performance that is observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. We provide a theoretical explanation for this regularization effect using ideal DPO policy theory. We further show performance improvements over DPO in translation and mathematical reasoning using our DPO variants. We find it can be beneficial to include ties in preference optimization rather than simply discard them, as is done in common practice.

On Extending Direct Preference Optimization to Accommodate Ties
Jinghong Chen, Guangyu Yang, Weizhe Lin, Jingbiao Mei, Chenxu Lyu, Bill Byrne
Department of Engineering
University of Cambridge
Cambridge, United Kingdom CB2 1P% (jc2124, gy266, w1356, jm2245, c1927, wjb31 Jecam. uk
Abstract
We derive and investigate two DPO variants that explicitly model the possibility of declaring tie in pair-wise comparisons. We replace the Bradley-Terry model in DPO with two well-known modeling extensions, by Rao and Kupper and by Davidson, that assign probability to ties as alternatives to clear preferences. Our experiments neural machine translation and summarization show that explicitly labeled ties can be added to the datasets for these DPO variants without the degrada- tion in task performance that observed when the same tied pairs are presented to DPO. We find empirically that the inclusion of ties leads to stronger regularization with respect to the reference policy as measured by KL divergence, and we see this even for DPO in its original form. We provide a theoretical explanation for this regularization effect using ideal DPO policy theory. We further show performance improvements DPO translation and mathematical reasoning using DPO variants. We find it can be beneficial to include ties in preference optimization rather than simply discard them, is done in common practice.
Introduction
The original formulation DPO Rafailov et al. 2023 does not allow for ties. DPO requires training data consisting of paired options, Dw Yr, and each of these pairs should represent clear preference in judgment with no ambiguity as to which is the winner and which is the loser. From this data, the DPO learning procedure encourages the underlying policy prefer yu over y/- This formulation does not allow for any ambiguity or uncertainty in the comparison of the paired examples in the training data.
This certainty is not easy to achieve in practice. A common approach is simply to discard data. Dubey et al. 2024 Sec. 4.2.1] apply DPO in post-training of Ilama models and note that for "DPO, we use samples that are labeled as the chosen response being significantly better or better than the rejected counterpart for training and discard samples with similar responses." Similarly, Qwen2 developers (Yang et al. 2024a] Sec. 4.3] "sample multiple responses from the current policy model, and the reward model selects the most and the least preferred responses, forming preference pairs that are used for DPO." Over-generation followed by aggressive selection is effective in producing the strongly ordered judgments needed for DPO. However the process appears wasteful: many potentially useful, and expensively collected, preference judgments are discarded simply because they are ties. As Rao and Kupper [1667] note: "any model which does not allow for the possibility of ties is not making full use of the information contained in the no-preference class."
Motivated by this, investigate DPO variants that incorporate ties. We replace the Bradley-Tere preference model at the heart of DPO by two well-known extensions by |Roo and Kupper [1967 and by Davidson [970] that explicitly assign probability to tied judgments alongside winners and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

losers. Since these models are generalizations of the Bradley- model, we find that they are readily incorporated into the DPO modeling framework. experiments in neural machine translation and summarization, we find that ties can be added to the datasets for these DPO variants without the degradation in task performance that results from adding ties to the original DPO. We also observe improved regularization, in reduced KI--iieegencee to the reference policy. by adding ties. We provide a theoretical explanation based on ideal DPO policy theory by Chen et al. (2024). We further show DPO-RK and DPO-D improves performance over strong DPO baselines in translation and mathematical reasoning by including previously discarded data as tied pairs. These findings suggest it can be beneficial incorporate ties in preference optimization rather than simply discard them, as is done in common practice.
Methodology
2.1 DPO and the Bearley-Terry Preference Distribution
The Bradley-Terry model assigns probability that an item y/4 will be preferred to item yj in terms of their 'strength' parameters X. In the RLHF setting, strengths expressed rewards À e" [Eq. Rafailo et al. [202]], so that the preference distribution for item over item j depends on the difference in their rewards, dij = T; Tz
A DT (Ua Uj) = X; + eti +ee =o(r: ar(r--r)) o(dig)
e"
(1)
One of the enabling observations made by RRfaillvv al. [0033 is that when policy TTe is sought to maximize the KL-regularized objective max+, E (z, 8) 3 ((To(ylz) || Tref (v(z)), the reward
associated with the policy has the form re v) B log (ulr( expressing the difference rewards between hypotheses the and yt under a parameterized policy me as the reward margin
ma(ulz) Boog Zz(x). This allows
To To
To (x)
dø(x, yw 31) re Uw) - re( 31 Boo
B log Tree (w1(z))
(2)
Tres [x)
so that the corresponding Bradley-Terry probability that item you beats item yo
<0(coo B log Tree (Bwla)
To (wilz))
BT P6
(yu > v1) (do (2,, Vw. v)) (Blog
(3)
Tree (re(())
The DPO policy objective [Eq. 7 Rafailov et al. 202.3 follows by incorporating the parameterized form of the preference distribution into maximum likelihood objective
LDro(To: Tref) Er.y.. log (B >= yr)
(4)
-E log (3 log To 3 log To (x)
(5)
Tref (yor (x)
Tref (y) (r)
We note that Fq follows from the regularized risk optimization [Rafailov et al. 2023 A.1]. It does not rely any assumption that limits its use to the Bradley-Tery model.
2.2 Bradley-Terry Extensions that Accommodate Ties
An observed weakness of the Bradley- Terry model that it does not allow for ties. Unless two items have exactly the same strengths (so that d,j = 0). the model always assigns a higher probability of winning to the stronger item. This may be reasonable if one item much stronger than the other, but when items are relatively comparable it may be desirable to allow some probability for tied outcomes.
The Rao-Kupper Rao and Kupper 1967 model assigns win and tie probabilities
A
p RK (y, > (j) A; + VnK
Yi beats yj
(6)
(vīK - 1));A3
p RK (ys ~z) = (A: VRK " VRK
Yi ties yj

---

Balancing Wins and Ties. In the special case of two evenly matched players (A; Ag), we are interested in the probability of a tie p(y; ~ y;) versus clear win by either player, p(u: > 31) + P(y; > y.). It follows that PRK (tie) = - PRK (no tie) and (tie)) PD (no tie). This shows that the parameters determine the probability that equally-matched items are judged as while the Davidson Davidson
1970] model assigns win and tie probabilities as:
A
p py > Vi) = X; X3 + 2 Aidg
beats
(8)
2VDV Vidy
p" Ui)) 7 2
and Vy tie
(9)
The probabilities of the three outcomes sum one for both of these Bradley-Tery extensions:
ply: > W2) + p(y > y1) + p(u ~ y₁) =11. 1. For both models, p(y: Myy = p(y; ~ y1) and ply: yj) tends towards 0 if X > dr. Both variants have parameters that control how much probability is allocated to ties. Apart from VRK = 1 or up = O, when both variants agree with Bradley-Terry, some probability reserved for tied outcomes.
The Rao-Kupper and Davidson models arise from different considerations. Rao and Kupper 1967 begin with the formulation y (y: > 1/)) = 1 [ sech"(y/2)dt (y/2)dy [Bradley] 1953] and note its sensitivity to the difference in values r; ry. They note that some judges "may not be able to express any real preference" in paired-comparisonns if their "sense of perception is not sharp enough" to detect small differences. They reason that a "threshold of sensory perception" is needed such that if the observed difference less than the threshold, a judge declares a tie. They introduce the sensitivity threshold ORK as follows, RK (y: yj)
sech" (y/2)dy, and Eqs.
and] follow for MRK = =
Luce 1959a which states that a complete \//2g, which the Rao- Sao-Kupper model fails to do. Davidson [1970] observes that it is desirable for the probability of tie to "be proportional to the geometric mean of the probabilities of preference". AAdinng this requirement
Davidson [970] starts from Luce's "choice axiom" "
system of choice probabilities should satisfy p(y > y/) ply: > yi
p(y, ~ y/ = "VDypply > My yy > y:) to the choice axioms yields Eqs. and
preference
model that allows for ties and also satisfies the choice axiom.
The Rao- Kuo-Kupper win and tie probabilities can be written in a form more useful for DPO (Appendix B.I. with VRK = =RRRK as
Po
RK (yhr >xx y1) = (do Bw y1) - ank)
(10)
RK (yh ~s y1) = (vix 1) ( do(o, (I, yu v1) - (x pe RK (yo b 31)
(11)
and the Davidson win and tie probabilities can be written as
1
pe (Mh > 31) = do(2, M1) + 2upe- 30) pd (yw ~e W1) = 222 e do (2. pe (yw > v1)
(12)
22)=2pp
(13)
Although their parametric forms are different, their treatments of wins and ties are similar (Ap- pendix [B. Fig. 3). For pairs (z, Mws 31) treated as wins, higher likelihood is assigned for higher values of the reward margin do( yw: W. For the Rao-Kupper this is particularly clear, that the Maddeey-Terry preference distribution is simply shifted by aRK- Conversely, for pairs (x, ties v1) treated as ties, the probability of declaring a tie is high for small reward margins dg (I. Yw: M²)
tied or not. v can be tuned, but in our work, we assume that equally-matcher items will tie with probability of 1/2 and so we set Vnk and VDD 1.
2.3 Incorporating Rao-Kupper and Davidson Models into DPO
We extend the DPO policy objective (Eq. to include a binary flag / to indicate tie:
[(To; Tref) = - Ex the an t= log Po (V Ya Vr) - E gu. t=1 log ( ~2 v))
(14)
3

---

where (y > 31) and pa 3/ww
wi) are taken from either the Rao- suo-Ku model (Eqs. 10 1 or the
Davidson model (Eqs. 12 13).
Note that Eq. [4prfferneee pairs in the dataset are unambiguously either wins (t = 0) or ties = 1). The policy objectives for these two DPO variants are:
LRK (To: Tres) = -EE Nw 3n t=0 log o (dg( yw v1) - aRK)
[log al do Vhe, Vee aak) + log a (do (z,, Uw u1) (.RK) log(u - 1) - -Br.y.....t
RK
(15)
and
(D(TTi
Ex, 1=0 log
1 do da(a.wu.n)) + 2uDe du (2. wn)/2
2uDe do (wr. 3/1/2
Br log
(16)
1 e (2..v..8))) + 2uDe de..
We refer to these DPO variants as DPO-RK and DPO-D. Like DPO, these objectives depend on the policy through the reward margin dg (z. Yw 31) (Eq.2) (2. Unlike DPO, the training objective Eq. [4 consists of two competing terms. For pairs (I, Yu: y1) Tabeled as wins the objective to find my to increase the reward margin do(2, Yw: 11) However, for pairs labeled as ties the objective is find me to minimize dd (x, Yur, yt)( To simultaneously achieve both these objectives, the underlying policy should learn model both wins and ties.
2.3.1 DPO-RK and DPO-D Updates
Rafailov et al. 2023) show that DPO dynamically adjusts the gradient according to how well the preference objective is optimized for each sample
To (J xx) TT (yr (z)
V log BT (yu > v1) a(-do Viu 31)) V log
(17)
higher weight when reward estimate is wrong
DPO-RK and DPO-D also adjust their gradients dynamically (Appendix B.2). We define the gradient scale factors A Awn and Arse to illustrate the DPO-RK and DPO-D gradient updates on wins and ties:
(o
v loggp Pe (yw >= v1) o(a do(x, Yw 31) V o log n (3((())
(18)
A Aasn (1g)
To (0(yw(z)) (19)
Ve log p6
AK (yw ~r 41) [o(a dø(ar, ywu U1) aaa + do (Ix ywo vc)) 3V log
Te ( (z)
A (da)
do ve dg/2
no you (x)
V log Po (y >2 v1)
V log
(20)
+ e- du + 2ue do/2
To (2)12)
AD (do)
ne ()(l)z
Ve log pe (yw m1) = AD D
win (da)
V log
(21)
2
To(
A. (do)
V log po (Yer > W) For data labeled as wins, the DPO-RK gradient scale factor has the same form as DPO, but shifted by ORK (Fig. 4. DPO-D has symmetric scale factor that is not as steep as DPO-RK. All three methods work to increase the reward margin dg (z, Mos 91).
V log Po (yar ~r ): For data labeled ties, the DPO-D and DPO-RK gradient scale factors are odd and work to drive do (2, Mos U1) towards zero, although the DPO-RK scale factor more aggressive. This is mechanism not present in DPO.

---

Intuition for hyper-parameter aRK and VD: We note that DPO-RK and DPO-D each introduces one additional the Davidson variants use these pyppe--arameters to control the likelihood of a tie. The updates on ties above [9q[T9] and 21) offer another intuition for these hyper-parametery ORK and uD control the width of the band in reward margin where there is little gradient contributions from tied pairs whose difference in reward falls within the band. However, for tied pairs whose difference in reward fall outside the band, the gradient updates work to reduce the margin. We find experimentally that performance is not sensitive to the choice of ORK and up so long as they are in sensible range (Appendix F.I and so keep to aRK log 3 and VD 1 as motivated in Sec 2.2
relative to DPO. In Appendix B we discuss how the Rao-Kupper and
3 Experiments in Adding Ties to DPO, DPO-RK, DPO-D
DPO in its original formulation relies static dataset of comparisons D
yw y
(2)
(2)
where yic and v/ are preferred and dispreferred responses to prompt (++) Rafailov et al. 2023 These preferences are assumed to be sampled from some latent reward model and we refer this dataset as Clear Preference Pairs (CPs, for short) because they are typically selected to reflect a clear preference between winner and loser as assessed either by human judges or by some trusted automatic metric. We distinguish CPs from Tied Pairs (TPs). TPs also consist of a winner and loser, but are very similar in quality. Human judges might be less consistent, or have less confidence, in selecting the winner in a tied pair, and automatic metries will assign more similar or even conflicting quality scores to TPs than to CPs. We study conflicting preferences in See 3.5
3.1 Extending Preference Datasets to Include Ties
As noted, DPO datasets typically are constructed to include only CPs. We develop data selection procedures to generate TPs along with CPs so that we can investigate how DPO changes when Tied Pairs are included in the training data. We follow previous work [Wang et al. 2024 Xiong et al. 2024 Liu et al. 2024 to collect preference pairs by sampling and then ranking model responses. We pair the top- and bottom ranked responses as CPs, and select TPs as pairs that are close in the ranking. For Neural Machine Translation (NMT) on WMT-21 ZH-EN (Akhbardeh et al. 202T and IWSLT-177 FR-EN [Cettolo et al. [017]]. we rank responses by BLEURT, a widely-used reference- quality metric (Sellam et al. 2000 Freitag et al.] [023]. For Summarization on TL:DR [Stiennon et al.] 2020]. we rank responses using the implicit reward function learned by DPO itself, without external
reward model. Appendix E gives experiment details. Studies of these selection strategies can be
found in Appendix F.7 and F.8
3.2 Adding Ties to DPO The Regularizing Effects of Ties
Following prior work [Rafallov et al., 2023 Amini et al. 2024a Park et al. 2024], j. we evaluate DPO in terms of task performance versus KL. divergence to the reference policy. For each of the three tasks we form two training sets: CP, which contains the Clear Preference Pairs; and CP+TP, which contains both the Clear Preference Pairs and the Tied Pairs. We refer to DPO training these sets as DPO(CP) and DPO(CP+TP). We note IPO AAzr et al. 2024) yields similar results (Appendix [:5).
The obvious conclusion from these experiments (Figure D is that including tied pairs in DPO hurts task performance. All best performing systems are obtained by DPO(CP), with (PO(CP+TP) underperforming for nearly all values of KL relative to the reference policy. This performance degradation from including ties is consistent with common practice in the DPO literature which only keeps pairs with clear preference, filtering others to obtain the best pest-performing system (Yang et al. 2024a [ubeyy et al., 22224) Consistent with this, the TL;DR results show that removing tied pairs from the DPO dataset leads to improved summarization performance, even when ties are identified by a DPO model in an unsupervised manner. However these results also suggest that tied pairs the DPO datasets can enhance regularization. By this we mean that including TPs causes DPO to find models that are closer to the reference policy as measured by KI. divergence. The overall effect of the reduced task performance and more regularization is to shift the frontier 'down and to the left'.
5

---

37.2
76.
1013
75.0
F-1027
76.5
7700
66-2
65.0
66.2
60.0
66.0
20
10
10
16 20
50
150
Divergence
Divergence
Divergence
(a) WMT21 ZHIEEN.
IWSLT17 FR-EN. Baseline 75.7
(c) TL:DR PairRM Win-Rate; Baseline 37.5%
Baseline 66.0
Figure 1: Task Performance vs. KL to the reference policy for DPO systems trained on Clear Preference Pairs (DPO(CP), blue) and on Clear Preference Pairs and Tied Pairs (DPO(CPTTP), green). Task Performance is reported in BLEURT for translation tasks WMT21 ZH-EN and IWSLT17 FR-EN. Summarization performance is reported on TL;DR in terms PairRM win-rate against human-writte summaries. KL is estimated over 256 test set policy samples; 3 is noted for best performing systems. Full details are in Appendix
3.3 Explaining the Regularization Effect of Ties via Ideal DPO Policy Theory
Theorem 3.1 Chen et al. [0024 suggests how these regularization effects might arise. The ideal DPO policy x² should follow (Appendix G
π (2) nT" (3/12) (rer(y (z:) 1 7(2, Yu y1)
[er((yll ( 7x2, Mae M1
1/9
(22)
where "(x,, Mws M1) is the true preference probability of yu > under prompt x. If we assume that tied pairs have true preference probability (25, Yan yt) of 0.5, from Equation 22 we have
n (2) [z)() where n² is the ideal DPO policy By this analysis, the ideal DPO model
should maintain the same chosen/rejected likelihood ratio as the reference model on tied pairs, and this constraint serves as form of regularization. In our NMT experiments (Figures Ta lb). where half of the pairs are constructed to be ties, the regularization effect is especially pronounced as the DPO model should keep to the reference model likelihood ratio on 50% of the training data. Regularization is less pronounced on TL;DR (Figure Ic where only 1/8 of the pairs are ties.
Another way to view this to rearrange Eq 22 follows:
(1ullz) (Vwl
TT" (ui[z) Tref (y)
log 7(1, Yu 31)
d2, Nwo 31) B log
log
(23)
Tref
1 (xx, Vwe M1)
From this it follows that the reward margin on tied pairs should ideally be close to zero, which is form of regularization. We verify this experimentally in Table 3
3.4 Adding Ties to DPO-RK and DPO-D Regularization without Performance Degradation
Using the same data as in Sec 3.2 we now evaluate DPO-RK and DPO-D as DPO variants that explicitly model both ties and clear preferences. We use the DPO datasets CP+TP (Sec. 3.2) with the DPO-D and DPO-RK algorithms to produce models DPO-D(CP+TP) and DPO-RK (C(CCTTPP)) We follow the protocols of Sec. 3.. so that results are directly comparable to earlier DPO(CP) and DPO(CP+TP) results. We find that the choice of has only a minor effect on task performance (Appendix F.1 For all experiments we RK
and D = for DPO-RK and DPO-D as described in Sec. 2.2 Training dynamics are provided in Appendix D
We observe consistent patterns across three datasets: DPO-D(CP+TP) and DPO-RK(CP+TP) do not suffer the same drops in task performance that (PO(CP+TP) exhibits (Fig 2 orange and purple green). Both reach similar levels of task performance to DPO(CP), but do values than DPO (Fig. 2 orange and purple vs. blue). For a given level of KL to reference policy, DPO-D(CP+TP) and DPO-RK(DP+TP) yield higher task performance than DPO(CP). Compared to DPO as it is usually done, DPO-RK and DPO-D frontiers are shifted leftwards, showing similar task
smaller KL.
In Appendix G we show that the ideal policy can also be derived for DPO-D which includes the ideal DPO policy special case.

---

37.2
76.6
76.
7700
75.0
Foll 2007
66-2
65.0
76.4
66.2
76.3
60.0
66.0
762
15
10
50
150
Divergence
Divergence
Divergence
(a) WMT21 ZH-ENN frontiers. Baseline BLEURT= 66.0
(b) IWSLT FR-EN frontiers. Baseline BLEURT 75.7
(c) TL:DR frontiers.
Baseline win-rate 33.5%
Figure 2: KI--Performance frontiers with DPO(CP) in blue, DPO(CP+TP) in green, DPO-RK(CP+TP) purple, and DPO-D(CP+TP) in orange. Full details in Appendix E.4 For TL;DR, we additionally report win-rate as judged by GPT-4 in Table 13
performance but stronger regularization. Following the analysis in Sec 3.3 we empirically show the percentage of TPs in the preference dataset is proportional to the regularization effect in Appendix
F4
3.5
Performance Improvement through DPO-RK and DPO-D
In this section, we show two realistic scenarios in machine translation and mathematical reasoning where DPO-RK and DPO-D makes use of otherwise discarded data to achieve performance improve- ment over strong DPO baselines. We also find DPO-RK and DPO-D leads to overall higher reward accuracy compared to DPO (Appendix C).
3.5.1 Modeling Conflicting Preferences as Ties
In curating preference datasets, it is common employ multiple human or machine annotators to rank candidate responses. Conflicts between annotators easily arise. This is the case for the L.MA----referenne dataset [Xu et al. 2024), which contains 3 translations sampled from human, GPT-4, and a base model for each source sentence. These translations are ranked by the average of their XCOMET and KIWI-XXL scores, which are reliable reference-less metrics shown to correlate well with human expert judgment [Freitag al. 2023). In Xu et al.) [2024] 's original work, the pair of translations with the largest average score difference is selected as a CP and the third, middle-scoring, translation is simply discarded. This yields 3070 CPs, one pair for each source sentence.
We propose a simple scheme treat translations with conflicting preferences as ties. For pairs not selected as CPs, we identify pairs for which XCOMET and KIWI-XXL disagree about which translation better as pairs of conflicting preferences. For each source sentence where such pair exists, we add one to OPO-RK/DPO-D training as TP, resulting in 2302 TPs. Table DPO- RK(CP+TP) and DPO-D(CP+TP) yield better performance across all evaluation metries compared to DPO(CP). This shows the ability of our DPO variants to learn from pairs of confficting preferences.
Model
COMET
KIW1-22
XCOMET
KIWI-XXL
ALMA--B-LLRAAXA Xu al. [2024)
79.78
76.81
83.94
73.65
DPO(CP)
79.66
77.73
88.87
74.12
+ (PO-RK(CP+TP))
80.63
78.91
90.40
75.77
+ DPO-D(CP+TP)
80.38
78.27
90.09
75.54
Table ZH-EN translation performance on for DPO(CP), (P(PTT(CP+ and DPO-D(CP+TP) over beta sweep in [0.1, 0.3, 0.5, 0.7, 0.9].
test set. The best result is reported
3.5.2 Preserving good behaviour in the reference model through regularization
We have shown that including ties leads to strong regularization effect with respect to the reference model (.2c..2). We now exploit this property to preserve good performance observed in the reference model while leading stronger overall performance in mathematical reasoning.
Following the DPO-augmented Trainirain (DPO-ST) approach by Wang et al. (2024). we curate preference data by sampling model responses to questions in the GSM8K training set. For 2310 of

---

7470 questions, we find that the moderately-sizee Qwen2.5-3B Instruct model Yang et al. 2024b answers correctly in all of its 8 sampled responses (at temperature 0.7). In DPO-ST, these 2310 questions would be excluded from preference data, even though they are substantial portion (30.9%) of the training set. We show instead that correct responses to these instances can be treated as ties.
We label CPs and TPs as follows: for training instances where there is at least one correct and one incorrect model response, we randomly choose a vorrectvvrsus--nnooreee pair as CP: for examples where all model responses are correct, we randomly choose a pair as a TP. We then conduct DPO training on CP and DPO-RK and DPO-D training on CP+TP with range beta values. We also report SimPO and CPO training on the CP set for comparison.
B
SimPO (CP)
CPO (CP)
DPO (CP)
DPO-RK (CP+TP) DPO-D (CP+TP)
0.1
82.5%
82.6%
76.4%
83.5%
81.7%
0.3
81.8%
83.1%
83.7%
84.4% %
83.2%
0.5
81.8%
83.1%
83.6%
83.8%
84.5 %
0.7
81.6%
82.6%
83.3%
83.7%
84.5 %
1.0
82.2%
83.6%
83.5%
84.1 %
83.7%
Table 2: GSM8K test set performance with greedy decoding after one-epoch of preference opti- mization for a range of beta values, evaluated by exact match after "####". Tie-compatible variants yield better performance for every beta value. The base Qwen2.5-3BP --B--tuutttt model scores 70.9%. Standard deviation reported in Appendix E.5 E.5
Table shows that OPO-RK(CP+TP) and DPO-D(CP+TP) achieve better performance than the DPO(C)) model at every beta value. Intuitively, these TPs provide strong regularization signals for examples where the reference model already excels, thereby preserving good behaviour in DPO- RK/DPO-D training. Such mechanism is not available in the original DPO formulation. To verify this intuition, we define Preservation Rate (PR) as the percentage of questions where the policy continues answer the question correctly provided that the reference model gives the correct answer under greedy decoding. Lower PR indicates more instances where the answer flips from correct to incorrect after training. which is clearly undesirable. Table 3 shows PR along with KI. and overall accuracy on the GSMSK test set. We find that compared to DPO(CP), APO-RK(CP+TPP and DPO-D(CP+TP) has lower KL and higher PR which leads to higher overall accuracy.
Model (optimal ()
KL
Overall Acc%
PR%
Base model
0
70.9%
100%
DPO(CP), 3 = 0.3
2.258
83.7%
95.19%
OPO-RK/CP+TP),,5 0.3
1.762
84.4%
97.1 %%
DPO-D(CP+TP), 8 0.7
1.465
%
97.65 %
Table Preservation Rate (PR%) of the best-performing models.
4 Additional Supporting Experiments and Analyses
Due to space constraints, we briefly highlight additional findings and provide pointers to supporting experiments in the appendices.
DPO-RK and DPO-D yields higher reward accuracy than DPO Like DPO, DPO-RK and DPO-D yields an implicit reward function formed by the trained policy and the reference model, which can be used to classify preference pairs. In Table we show that DPO-RK and DPO-D yields overall higher reward accuracy compared to DPO classifying held-out clear preference pairs and ties. See Appendix C for details.
The proportion of TPs controls regularization We show empirically that the number of ties included in DPO-RK and DPO-D training is proportional to the strength of regularizaion (Table 14). This provides additional support for the regularization effect of ties shown in Sec 3.4
Ideal DPO-D policy theory Following the ideal DPO policy theory by Chen al. 2024 used in explaining regularization (Sec 3.3 we derive the ideal DPO-D policy theory starting from the

---

termary classification objective that accounts for ties in Appendix G This can be view as an extension to Chhhnn al. [2024] theory in that the ideal DPO policy emerges as special case.
Convergence behaviour In Appendix D we investigate how reward margins and gradient scale factors (defined in Eqs 21 evolve when tied pairs are included in training. We find that reward margins on TPs remain close to zero as desired, and that reward margins on CPs rise less sharply and lower magnitude compared DPO(CP). An interesting consequence is that, contrary to DPO(CP) training where gradient scale factor decreases to < 0.05 after 1/4 epoch and stabilizes, the gradient scale factors remain negligiie throughout the entirety of training for DPO(CP+TP), DPO- (PO-RK(CP+TP), and DPO-D(CP+TP), indicated prolonged learning.
Qualitative Analyses Example responses from DPO, DPO-RK, and DPO-D systems are provided and analyzed in Appendix
5 Related Work
Related Variants of Direct Preference Optimization ODPO [Amini et al. 2024b incorporates preference strength in the objective by introducing an offset parameter. In deriving ODPO, the offset parameter of Amini et al. [2024a] Theorem 31) plays a role similar to the sensitivity threshold of Rao and Kupper 19677. We note that the ODPO objective with fixed offset agrees with our proposed DPO-RK objective restricted to clear preference data, but does not extend to ties. We note DPO-RK is independently proposed by contemporaneooss work Guo et al. [2024] Compared to their work, we additionally introduce DPO-D, propose novel tie selection strategies, identify the regularization effects of ties and provide theoretical explanations. Our experimental setups complementary: while they primarily focus on general chatbots, we conduct experiments in translation, summarization and mathematical reasoning.
Frameworks for Pair-wise Preference Optimization Several works propose theoretical frame- works for understanding general Preference Optimization from which DPO can be obtained special case. Dumoulin et al. [2024) formulate learning from pair-wise preference as learning the im- plicit preference generating distribution of the annotators. In this formalism, DPO well-specified model for the implicit preference distribution assuming that the human preference generative process follows the Bradley-Terry model. Our work can be viewed as assuming an annotator preference gen- erating distribution that allows for the outcome of a tie (i.e. the Rao-Kupper or the Davidson model). Tang al. [2024] propose generalized approach deriving offline preference optimization losses through binary classification. In this work, we extend binary classification to ternary classification with the possibility of declaring a tie (Appendix G
Pair-wise Comparison Models Hamilton et al. 2023 review the Bradley- Terry model, including its relation to the logistic distribution [Bradley and Gart 1962] and Luce choice axiom Luce (959b).
The Rao- Mup-Kr [Rao and Kupper 1967] and the Davidson model [Dvii 1988) are two notable extensions Bradley-Terry (Sec.[2.7 (.2). See the review by David [1988] and bibliography by Davidsso and Farquhar [1976]. Modeling ties remains an active research topic in fields such as sport team ranking [Zhou et al. 2022], medical treatments [Gaohong Dong and Vandemnuleerroecke 2020 and
chatbots Ameli et al.) 2025
6 Conclusion
We have derived and investigated two tiemcompatible DPO variants, DPO-RK and DPO-D, by replacing the Bradley-Terry preference model with the Rao-Kupper and the Davidson models, respectively. Our experiments show that DPO-RK and DPO-D can accommodate tied pairs in preference data without the degradation in task performance that is observed when the same tied pairs are added to the original DPO. We find empirically that the inclusion of ties in preference learning leads stronger regularization with respect to the reference model and provide theoretical explanations based on ideal DPO policy theory. We further show our DPO variants can improve model performance over DPO by making fuller use of the available data on translation and mathematical reasoning. These findings motivate the use of tied pairs in available preference data as opposed to wastefully discarding them. We discuss limitations in Appendix

---

7 Acknowledgement
Jinghong Chen supported by the Warwick Postgraduate Studentship from Christ's College and the Huawei Hisilicon Studentship for the undertaking of the PhD in Engincering at the University of Cambridge.
Guangyu Yang is supported by Cambridge Commonwealth, European and International Trust for the undertaking of the PhD in Engineering the University of Cambridge.
Jingbiao Mei is supported by Cambridge Commonwealth, European and International Trust for the undertaking of the PhD in Engineering at the University of Cambridge.
Weizhe Lin is supported by a Research Studentship funded by Toyota Motor Europe (RG92562(24020)) for the undertaking of the PhD in Engineering at the University of Cambridge.
Prof. Bill Byrne holds concurrent appointments as Professor of Information Engineering at Cambridge University and as an Amazon Scholar. This publication describes work performed at Cambridge University and is not associated with Amazon.
We would also like to thank all the reviewers for their knowledgeable reviews that helped us strengthen the contribution.
References
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 16, 2023, 2023. URL http://papps nips. cc/ wwppappr_ppppeceepppeeeeee httppaapaapaaalllalllllalaeeelee hilles/paper/2002/ 2223/hash/ ltmm
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark. Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon. Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, lliyan Zarov, Imanol Arricta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis. Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji. Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla,
10

---

Stephane Collot, Suchin Gururangan. Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbucher. Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong. Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li, Yuning Mao. Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani. Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Fisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David. Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery. Elconora Presani, Emily Hahn. Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj. Igor Molybog, Igor Tufanov, Irina- Elena Veliche Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, -aptisee Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong. Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan. Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal I. Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal. Nick Egebo. Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji. Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto. Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shugiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterficld, Sudarshan Govindaprasad. Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao. Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The Llama 3 Herd of Models, 2024. URL httphttts://arre wwwpswwwwwrrr///oooooo
217833

---

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge. Xiaodong Deng. Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhemru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 Technical Report, 2024a. URL
10671
PV Rao and Lawrence L Kupper. Bradley-Terry model. Journal of the American Statistical Association, 52(317):194-206 1967.
Ties in paired-comparison experiments: generalization of the
Roger R. Davidson. On Extending the Bradley-Teryy Model to Accommodate Ties in Paired Comparison Experiments. Journal of the American Statisttical Association,, 65(329):317-328, 1970. ISSN 01621459, 1537274X. URL http: /wwww jstor 5r55
Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference Learning Algorithms Do Not Learn Preference Rankings. CoRR. abs/2405. 11024405.15544,, 1953,, 2024, doi: 10.48550/ARXXI.24005.0 arXiv 2405. 19534
Ralph Allan Bradley. Some Statistical Methods in Taste Testing and Quality Evaluation. Biometrics, 91):222-88, 1953. ISSN 0006-341X. doi: 10.2307/3001630, URL. hhttps://ww..jjstt .rrg/ stable/3001630 Publisher: [Wiley, International Biometric Society).
R Duncan Luce. Individual choice behavior, volume 4. Wiley New York, 1959a
Tianduo Wang, Shichen Li, and Wei Lu. Self-training with direct preference optimization im- proves ohain-of-thouuht reasoning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, edi- tors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pages 11917-11928, Bangkok, Thailand, August 2024. As- sociation for Computational Linguistics. doi: 10.18653/v1/2000 10.88655 URL httpss //aclanthology hww.ttthooogg.ooggggeeooo (ttp:/2222.cccccc.. long. 643/
Wei Xiong, Chengshuai Shi, Jiaming Shen, Aviv Rosenberg, Zhen Qin, Daniele Calandriello, Misha Khalman, Rishabh Joshi, Bilal Piot, Mohammad Salch, Chi Jin, Tong Zhang, and Tianqi Liu. Building math agents with multi-turn iterative preference learning. ArXiv, abs/2409.0239 2024. URL https://pp/ppp semanti wwt.aaaaiccco....rooooooooooooaa
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Salch, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
OpenReview.net, 2024. URL / http::/eee/ffo...o www..nte//orrmiiddbdsssrrrrr
Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina EspañaaBonet,, Angela Fan, Christian Feder- mann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyiiee Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi. Philipp Koehn, Nicholas Lourie. Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Mattco Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi. Valentin Vydrin, and Marcos Zampieri. Findings of the 2021 conference on machine translation (WMT21). In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Koemi, Andre Martins, Makoto Morishita, and Christof Monz, editors, Proceedings of the Sixth Conference on Machine Translation, pages 1-88, Online, November 2021, Association for Computational Linguistics. URL https://acllanthlllle wrw. wmt-1.17 17
12

---

Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Nichues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the IWSLT 2017 evaluation campaign. In Sakriani Sakti and Masao Utiyama, editors, Proceedings of the 14th International Conference on Spoken Language Translation, pages 2-14, Tokyo, Japan, December 14-15 2017. International Workshop on Spoken Language Translation. URL https::/acllanthollooo
201/ iwslt-1
17
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. BLEURT: Learning Robust Metrics for Text Generation. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2020. doi: 10. 10.18663/V1/2222AAALLL AAL-OI0N.CCL---...00 704. URL https:://oo.....o 000/10. 18653/v1/2020 acl main. 704
Association for Computational Linguistics.
Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George F. Foster. Results of WMT23 metrics shared task: Metries might be guilty but references are not innocent. In Philipp Koehn, Barry Haddon, Tom Koemi, and Christof Monz, editors, Proceedings of the Eighth Conference on Machine Translation, WMT 2023, Singapore, December 6-7, 2023, pages 778-628. Association for Computational Linguistics, 2023. doi: 10.18653/V 1/2023, WMT-1.51. URL URL https:///oo/ org/ 10. 18653/v1/2023 1m1
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01 01325,, 2020. URL hnttps://arxins http://awxi...rg/aaaaa/// www./ogg/////eeeeeeeeee 2ttp//2009.00..5...5 01325
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct Preference Optimization with an Offset. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 9954- 9972. Association for Computational Linguistics, 2024a. URL http::tttt///tt htttps://aclanthollla org/ 2024. findingsnninnnee acl. 592
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling Length from Quality in Direct Preference Optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 4998-5017, Association for Computational Linguistics. org 2024. fol.ffindda findings--dae aallggsssccllac acl. 297
2024. URL httphttss///caatttottto
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. General Theoretical Paradigm to Understand Learning from Human Preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, International Conference on Artificial Intelligence and Statistics, 2-4 May 2024, Pulau de Congressos, Valencia, Spain, volume 238 of Proceedings of Machine Learning Research, pages 4447-4455. PML.R. PMLR, 2024. URL https:: https://proccediing. mlr, wtt...ppees/v23e.8 hwtp/////ghhhhhhhhhege azar 24a. html
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of
LLM performance in machine translation. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL hhtps //openreview net w 11iiiiozpw
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li. Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412. 2024b. doi: URL https:///// o00/10 1....885//aaiii 2412. 15115
Afra Amini, Tim Vieira, and Ryan Cotterell, Direct Preference Optimization with an Offset. CoRR, Abs/2400.1057/ 2024b. doi: 1048550/ARRX133222.007..1 htll/httss://// //doi / 27Xiv.2402.182
13

---

Yuxiang Guo, Lu Yin, Bo Jiang, and Jiaqi Zhang. TODO: enhancing LLM alignment with ternary preferences. CoRR, abs/241 1.02442. 2024. doi: 1044550/ARKXV.2411.02444 URL https //doi.org//is .rgg10. 48550/arXiv.
.22442
Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, and Yann N. Dauphin. A density estimation perspective learning from pairwise human preferences. Trans. Mach. Learn. Res., 2024, 2024, URL
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Danicle Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized Preference Optimization: A Unified Approach to Offline Alignment. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL http:htps:://// hhttps::/openrevieeeee wwt.:e/.www..eeeeoourrr/rrraaaaa
Ian Hamilton, Nick Tawn, and David Firth. The many routes to the ubiquitous Bradley-Teny model, 2023. URL
13619
Ralph A. Bradley and John J. Gart. The Asymptotic Properties of ML Estimators when Sampling from Associated Populations. Biometrika, (9(1/2):0052211 214. 1962. ISSN 00063444, 14643510. URL httpp://www.jtteeet wrr.strr.rrr/saabee233333333
R Duncan Luce. Individual choice behavior, volume 4. Wiley New York, 1959b.
H. A. David. The Method of Paired Comparisons. Number No. 41 in Griffin's Statistical Monographs and Courses. Charles Griffin and Company Ltd., London, 2nd edition, 1988.
Roger R. Davidson and Peter H. Farquhar. A Bibliography on the Method of Paired Comparisons. Biometrics, 32(2):241-255 1976. ISSN 0006341X, 15410420. URL http: / http://ww..strrr /rg/ stable/2529495
Yuhao Zhou, Ruijie Wang. Yi-Cheng Zhang, An Zeng, and Marúš Medo. Improving Pagerank using sports results modeling. Knowledge-Based Systems, 241:108168, 2022. ISSN 0950-7051. doi: http:://oo..oggg0000006..../oo/
Scienee/arrcce///119999977222222 108168. URL fttts::/ww..scieneediiii com/
Junshan Qiu Roland A. Matsouaka Yu-Wei Chang Jiuzhou Wang Gaohong Dong, David C. Hoaglin and Mare Vandemeulebroecke. The Win Ratio: On Interpretation and Handling of Ties. 12 (1):99-106, 2020. doi: 10. 00.1000094466315219991555 19466315. 2019. 1575279
URL https://doi.onn //t:::///oo...////// 1080/
Siavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W. Mahoney, statistical framework for ranking LLM-based chatbots. In The Thirteenth International Conference on Learning Represen- tutions, 2025. URL httth:://oppereieeeeeeeeee
Guangyu Yang. Jinghong Chen, Weizhe Lin, and Bill Byrne Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 2: Short Papers), pages 391-398, Mexico City, Mexico, June 2024c. Association for Computational Linguistics. doi: 136653/v1/222.naagg naacl-short. 34
URL httttss:///clattthhtt www.aantooooooooooo///2/2///
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang. Dragomir Radev, Alham Fikri Aji. Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual Generalization through Multitask Finetuning. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings the 61 Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 15991-16111. Association for Computational Linguistics. 2023. doi: 10. No.11863// 1/2033//1/2222.0 ACL.LLLOAAAAALLLLGGOON...... 891. URI, htt.:::tt/s///t.. //doi org/100, 18653/v12223 acl-long. 891
14

---

Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. COMET: A Neural Framework for MT Evaluation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 2685-2700 2702. Association for Computational Linguisties, 2020. doi: emnlp-main. 213
http:ttts::/dd/ https://dooooo////// 18653/v1/2020
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR, 2023. URL mlr. www.rpress/vveee wttpss//222/bbidrmaaaaa 1
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender. Ensembling Large Language Models with Pairwise Comparison and Generative Fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023.
Matt Post. A Call for Clarity in Reporting BLEU Scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 191- Belgium, Brussels, October 2018. Associ- ation for Computational Linguistics. URL https://www org. httgg/ntholog///11-63333
Ricardo Rei, Marcos V. Treviso, Nuno Miguel Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte M. Alves, Luísa Coheur, Alon Lavie, and André F. T. Martins. Cometkiwi: Ist-unbabel 2022 submission for the quality estimation shared task. In Philipp Koehn, Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno- Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélic Névéol, Mariana Neves, Martin Popel. Marco Turchi, and Marcos Zampieri, editors, Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 634 645. Association for Computational Linguistics, 2022. URL httLitttss://acaathoooooo
www.:rrg/200 2022/022.2222 wmt-1. 1. 60
Hoang Tran, Chris Glaze, and Braden Hancock. Iterative DPO Alignment. Technical report, Snorkel AI, 2023.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu. Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Bleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Ieo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopss, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen Hc, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanic
15

---

Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne. Bob McGrew, Scott Mayer McKinney, Christine eaVeav Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' OeKeef Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond. Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam. Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv. http://orrgabb//////// https://axx..oorr//b/2223...
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple Preference Optimization with a Reference- Free Reward. CoRR, abs/2405.147344 2024. doi: 10.-5550/ARXX1222555.773. URL https
0rg/1..0g/10..444 4 2405. 14734
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model
Alignment as Prospect Theoretic Optimization. CoRR, abs/2402.01306 2024. doi: 10.48550/
ARXIV.2402.01066. URL htttp::://oi.orrrto 48550/arxxiv 2402. 01306
16

---

A Limitations
The effect of accommodating ties in preference learning be further investigated using human- annotated tied pairs. However, at the time writing, there is no substantial preference dataset with annotated ties; notably, current annotation guidelines We note that this enforcement win/lose judgments has likely conditioned the generative process of human preference towards the Bradley-Tene Terry model. A meaningful extension of this work would be to assess the effectiveness of DPO-RK and DPO-D on preference datasets where the annotators are asked to identify ties. As explained in Sec 2.2 the hyper-parameter VRK and un can be tuned which would require either grid search or estimation given ground-truth preference/tie probabilities. We find that the choice of VRK 3 and VD 1 as motivated in See 2.2 works well and we did not need to tune the parameter to obtain good performance. It is likely that better performance and more efficient frontiers can be obtained by tuning y to better fit the underlying preference generative process for both DPO-RK and DPO-D. Given our focus on accommodating ties from modeling perspective, we leave performance optimization future works concerning applications.
typically written to explicitly exclude ties.
B Mathematical Derivations
B.1 Rao-Kupper and Davidson Preference and Tie Probabilities
We derive the win and tie probabilities as functions of the reward margin do tee yr) ro Yoo)
ro (r, 31 (Eq 2 under the Rao- Kupper (Eq 10 and Davidson formulations (Eq 12
The Rao-Kupper win and tie probabilities can be obtained by substituting Au ( A e (s.a1) and VRK R into Egl6 and 7 respectively:
Au
e erolz,ya)
RK (yw 11 P8
Aw + vRKAN eto(a.y)) (RKe
(do Mw v1) oRR))
(VRK 1)2mnde
P6 (yo ~ ~y1)
(ww + HUR A0 + URK)W)
(UTK - (orleeewweerrlee
(cro(ziNw)) (rexeerllo))))(eerll cro(s)))))
VRKeeoll.na (ro(z.0w))
(WRx 1)( (To((r + VRKeTols.vu) erola,Mu) + URKKTolo..))
(VRx 1) [ (2, ywu y1) aRK) o (ol (do ywu V1) aRK)
(VR 1) oo-da( ywu y1) aRK) Po
RK (yw > y1)
The Davidson win and tie probabilities can be obtained with the same substitution into Eq 8 and Eq 9 respectively:
Au > 11) dw + X 2 Awn (yw (y
Pá
cro(n.wu) + (ro(an)) 2VD Vrolln
"1 e + 2uDe do(2.
2VD 2voiAudi
Pd (y ~x v1) Au + A 2VDV Xe
de + d + 2uDV/ (2nDdmt +1²))
2VD e =trol(aiww ph (Yw > 31)
-2ue -do(x.vo.))/2 (Ym 7 31)
17

---

In Figure we plot the preference and tie probabilities as a function of reward margin de under Bradley-Tery (as used DPO), Rao-Kupper (as used in DPO-RK), and Davidson used in DPO-D).
B.2 Gradients for DPO-RK and DPO-D
The gradients of the Rao-Kupper log probabilities (Eq 18 19 are as follows. For convenience, we use the short-hand de for de yu V1).
Ve log RK (yw >x y1) Volog (do - (RR)
olURK- do) vdda (2, Me 2/))
= - d V log To (ywl Ve log T (z)
(ds)
To (V
RK win
(dg) Vo log
TO (1nyy11))
Vo log lo Po
RK (yw ~r y1)
Vo ( -- aRK) + log (dds - aRK)
= (dg + (RR) Vada + o- + aRk) Vods
= (o(aRk - do) - o ooaak + do) Vodo
And (do)
To (Vonl (x)
A RK (dg) V log Te y1 x))
The gradients of the Davidson log prrbabbbliiiies (Eq 20 21
follow similarly.
Vo log pe (ymr >a w2) V Varg (Nh > 91)
P6 (yw >x 11
Vo(1 + edo + 2ve doo do/2)-1
pb x 31)
(-1) do 2ve 2ve-da/22 dol/2 14/22-2 -2
(-e 22 Vodg
P >2 11)
Pg (Yw > v1) (c do- + ve -o/22 -d
Do (No >x V1)
Pd (Nw >= (e)(e-d do ve do/2 -d/2 Vodo
e do ve ve-d"!" do/2
= + e de 2ve dal2 Vodg
To (Bw
A win (do) V log
T (r)
18

---

Ve log Ph (Ye ~x v1) = Ve log (2ve
do 2p (B 22 W))
Vo logrg (yw >o y1) da /
d ve-2
11 Vodo
+ do + 2ve do/2 do/2
D (dg)
V log (Y)llz) - Vo log (N)l
(do) V log (Y x))
A (do)
T0(2)
For illustration, we plot Awin and Aie function of reward margin dg in Figure 4
The quantities VaLp (T; Fref) and VaLRK Tres) follow by substituting the above results into the gradient of Eq 14
VoL(TO: Tres)) =
Vo tmo log (y >x y2) Volle, log po (Bh ~e 31)
(24)
19

---

10
Plyw Vi(x)
Plyw ~ yi(x)
0.6
0.6
0.2
0.2
0.0
10.0
7.5
5.0
2.5
0.0
2.5
5.0
7.5
10.0
10.0
7.5
5.0
0.0 2.5
7.5
10.0
dglx, Yw, yi)
de(x, Yw. vi)
Figure 3: The clear preference probabilities P(y > M [:) (left) and tie probabilities P(y y (z)
(right) function of reward margins dø(x, Bw. 31) for Bradley-Tery (as used in DPO) (blue), Rao-
Kupper (purple) (as used DPO-RK), and Davidson (orange) (as used in DPO-D). aRK = log and up = 1 are used in producing these plots.
1.0
0.8
Awin V.. Vi)
Aue Xx Yv.Yi)
0.2
0.4
.0.2
-0.5
-0.0
0.0
-1.0
2.5 do(x, Vw. Vi)
dolx, w. Yr)
Figure 4: The gradient scale factors for DPO (blue) and DPO-RK (purple) and DPO-D (orange) function of reward margins do (x, yw 31) on clear preference pairs (up) and tied pairs (down).a = log 3 and PD = 1 are used in producing these plots.
C Using DPO-RK and DPO-D model as Preference Classifier
C.1 Rao-Kupper and Davidson Classifiers
The DPO-RK and DPO-D variants yield probability distributions (o > 91) and (y ~2 y1) in terms of the policy and the reference model Kref- We can use these distributions as classifiers to label a pair (x,, y1, 3/2) as either win (y) > y/2 or y/2 >x 91 or (y) ~rx 1/2), whichever has
the highest probability under either the Rao-Kupper or the Davidson model (Eqs. 10 11 or 12 13).
We will evaluate classification performance on held-out data not used in training to policies produced by DPO variants learn to distinguish wins from ties.
20

---

C.2 Preference Pair Classification Accuracy
We assess the performance of the Rao Kupper and Davidson classifiers introduced in Sec C.T in terms of their ability to label preference pairs either clear preferences ties. Ideally, classification performance will improve: (1) as tied pairs are added to the clear preference data sets (CP vs CP+TP); and (2) with margins generated from models produced by DPO variants that emphasize the distinction between tied pairs and clear preference pairs (DPO-D(CP+TTP, DPO-RK(CP+TP))
We assess classifier performance the held-out set created by collecting CPs and TPs from the WMT18 ZH-EN test set as was done for WMT20 ZH-EN (Appendix [B; this yields pairs with gold labels as either clear preference pairs tied pairs. Classification and assessment proceeds follows: we generate reward margins for the WMT18 ZH-EN pairs using DPO(CP), DPO(CP+TP), DPO- RK(CP+TP), DPO-D(CP+TPP models;
these reward margins to label the unseen pairs using the Davidson and Rao- Kupper classification rules (Sec. C.T and finally compute the classification accuracy relative to the gold labels.
Results are shown in Table 4 We find that smaller beta in training consistently leads to better overall RK-classification accuracy (+10% overall Acc. from 3 = 1.0 to 3 0.1), suggesting heavy regularization with respect to the reference model impedes preference ranking. Classifiers based on reward margins generated from DPO(CP) models perform well in identifying clear preference pairs (Acc. 85%) but poorly in identifying tied pairs (Acc. 35%). This imbalance is likely explained by the DPO(CP) model never having seen tied pairs in training. Adding TPs the DPO datasets (DPO(CP+TP)) significantly improves the classification accuracy of tied pairs (+30%) with more balanced classification accuracies for CPs and TPs. The best overall classification accuracies (^= 73%) are obtained with reward margins generated by models trained to match its classifier. Across all beta values, OPPORRKCC+TTB and DPO-D(CP achieve better overall accuracy and more-balanced CP accuracy and TP accuracy under their respective decision rules.
Model
B = 0.1
3 0.5
B = 1.0
Rao- Kup-er Classifier
DPO(CP)
60.1% (87.1%, 33.1%%
52.8% (87.3%, 18.3%)
50.1 % (86.9% 13.3%)
TPO(CP+TP)
67.0% (72.0%, 62.1%)
57.5% (69.3%, 45.7%)
51.5% (71.2%, 31.9%)
DPO-RK(CP+TP)
73.1% % (74.5%, 71.7%) )
64.2% (73.2%, 55.3%)
58.5% (73.4%, 43.5%
Davidson Classifer 57.4% (83.7%, 31.0%)
DPO(CP)
65.3% (84.4%, 46.3%)
53.6% (84.6%, 22.6%)
DPO(CP+TP)
71.0% (59.1% %, 82.8%
62. 1% (58.3%, 65.8%)
57.2% (62.3% 52.2%)
DPO-D(CP+TP)
73.8% % (79.6%, 67.9%)
66.8% (75.9%, 57.8%)
62.7% (75.2%, 50.3%)
Table 4: Preference pair classification accuracies (Overall Acc. (CP Acc., TP Acc.)) for Rao- Hupper and Davidson classification rules based on reward margins computed using DPO(CP), DPO(CP+TP), DPO- IPO-RK(CP+TP), and DPO-D(CP+TP) models as evaluated on the WMT18 ZH-EN test set.
C.3 Empirical Reward Margin Distributions
We now look at the reward margins on held-out pairs to determine how the DPO objective generalizes to unseen data. Ideally, a post-DPO model should assign reward margins that are large for clear preference pairs but close to zero for tied pairs. We assess this the same held-out data in the previous section (Sec. C).
In Table 5] reward margins of DPO(CP+TP), DPO-RK(CP+TP), and DPO-D(CP+TP) similar and well-behaved, showing means close-to-zero on TPs (< 0.4) and farther from zero for CPs (> 2.3). Reward margin standard deviations are also similar and reasonably small. However the standard deviation for both tied pairs and clear preference pairs are much higher for DPO(CP) models (2 11.1 on CPs and 7.9 on TPs).
This can be explained by Figure 5 which shows that DPO(CP) models overwhelmingly assign preference probability values of either 1.0 or ~ 0.0 to tied pairs, corresponding to very positive and very negative reward margins, respectively. This contributes to the high standard deviation and shows that for tied pair (2), DPO(CP) model exhibits strong preference for either y1 y2
21

---

Model
0.1
0.5
B = 1.0
8 0.1
B 0.5
3 1.0
Clear Preference
Pairs
Tied Pairs
DPO(CP)
8.2 12.00
9.5 433.2
10.0 +1111
0.7 +13.2
0.6 +9,4
0,4 +7.9
DPO(CP+TP)
2.4 43.3
2.3 #3.2
2.5 433
0.4 44.8
0.3 #3.2
0.2 4:2.7
DPO-RKICP+TP)
2.9 +4.3
2.8 #3.3
3.0 #3.3
0.0 11.
0.0 #1.4
0.0 #1.7
DRO-D(CP+TP)
4.6 +5.8
4.8 =6.1
4.9 +6.3
0.0 +2.0
0.1 #2.3
0.0 =2.4
Table 5: Reward margin statistics (mean + std) for held-out Clear Preference Pairs and Tied Pairs collected from WMT18 ZH-EN.
7000
1200
6000
Frequency
Frequency
1000
5000
800
4000
600
3000
400
2000
1000
200
0
0.0
0.2
0.4
0.6 0.8 1.0
0
0.0
0.2
0.4
0.6
0.8
1.0
Estimated Ply1 > y2)
P (y1 > Y2)
(a) Held-out Clear Preferences Pairs (CP)
(b) Held-out Tied Pairs (TP)
Figure 5: Empirical distribution of preference probabilities under the Bradley-Tere model using the implicit reward function from the trained DPO policy on heldout CPs and TPs. DPO(CP) in blue, and DPO(CP+TP) in orange.
U/ V1, even though these are tied pairs by construction (y1 3/2) contrast, DPO(CP+TP) yields well-hehaved estimated preference probability distribution more centered around 0.5 for tied pairs.
The DPO(CP) model correctly assigns high preference probability to most of the held-out CPs. This is consistent with its high classification accuracy on clear preference pairs in Table 4 Similar to the estimated preference probability on held-out TPs, the DPO(CP) model tends give confident, clear preference judgment with > 0.8 probability in either direction. In comparison, the DPO(CP+TP) model is more conservative in making preference judgments, showing a less-sharp preference probability distribution over the held-out CP pairs. These results suggest that incorporating ties in DPO training leads to preference probability distributions that more evenly spread on both CPs and TPs opposed to one concentrated on the two ends.
For completeness, we also show the clear preference/tie probability distributions produced by models trained with DPO-RK(CP+TP) and DPO-D(CP+TP) on held-out clear preference pairs and tied pairs. Figure [6 show that these distributions are well-behaved in that most of the probability mass allocated to Po(y) > y/) > 0.5 on held-out clear preference pairs and to Pa(y y2) s 0.5 on held-out tied pairs. We note that under our hyper-parameter setting for the Rao-Kupper and Davidson models, the maximal tie probability is 0.5.
All models in this analysis trained with 3 0.1.
22

---

1750
10000
Frequency
1500
Frequency
8000
1250
1000
6000
750
4000
500
2000
250
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.1
0.2
0.3
0.4
0.5
Pa > y2)
Po ~ y22
(a) Preference probability under the models held-out clear preference pairs.
Tie probability under the models on held-out tied pairs.
Figure 6: DPO-D (orange) and DPO-RK (purple) preference/tie probability on held-out sets under the Davidson and Rao- Kup-er models, respectively.
D Training Dynamics and Convergence Behaviour
We analyse how the inclusion of tied pairs affects the training dynamics of DPO, DPO-RK, and DPO-D. We study the BLOOMZ-mt-76 datasets with 3 = 0.7 for WMT21 ZH-EN as these systems show both strong regularization effects and task performance degradation when tied pairs are added. Figure 7 shows the evolution of reward margins and gradient scale factors (Eqs) 21
10ly
(a) Reward margins CP
(b) Gradient scaling on CP
DRO
DRO DCC
(c) Reward margins on TP
(d) Gradient scaling on TP
Figure 7: DPO(CP) (blue), DPO(CP+TP) (green), (PO-RK(CP+TP) (purple), and DPO-D(CP+TP)
(yellow) training statistics on WMT21 ZH-EN.
DPO(CP) is well behaved: the reward margins the CP data increase the epoch; the DPO losses on the CP dataset decrease over the epoch; and the DPO gradient scale factor shows that learning slows and stabilizes after the th batch.
Adding tied pairs to the DPO dataset alters this behaviour for both tied pairs and clear preference pairs. DPO(CP+TP) does yield some gains in reward margins for clear preference pairs, but these
23

---

well below that of DPO(CP). By contrast, DPO(CP+TP) fails almost entirely to find any improvement the reward margins for its tied pair data. While this is less than ideal from a modelling perspective. we note that it provides empirical support for the observation in the previous section that the reward margins on tied pairs should ideally remain close to zero. Similar behaviour is observed in the DPO loss. Decreases in loss over clear preference pairs are offset by loss increases on the tied pairs. This reflected in the gradient scale factors. The gradient scale factors remain high as DPO(CP+TP) searches for better policy.
OPO-RK(CP+TP) and DPO-D(CP+TP) exhibit largely the same dynamics as DPO(CP+TP) except that the gradient scale factors on TPs have mean close to 0 instead of 0.5 and fluctuate between positive and negative values. This showcases that DPO-RK and DPO-D work to drive reward margin on TPs close to zero by possibly reversing optimization direction, a mechanism not present in the original DPO.
E Experimental Details and Full Results
We provide additional details of our experiments on Neural Machine Translation and Summarization with respect to the SFT models, the training configurations, and the decoding procedures. All experiments are run with the random seed set O. Codes are available at ththt::: http://..ttt......//e com/ EriChen@615/DPO-1
E.1 Neural Machine Translation
We largely follow Yang et al. obtained via sampling and BLEURT-basee ranking explained in Sec 3.1
2024c in our experimental setup for NMT where the preference dataset
CPs and TPs in NMT. We use DPO to improve translation quality similar to that done in Yang et al.) [2024c]. We apply DPO with BLOOMZ-mt-7b [Muennighoff et al. [023 as the baseline model. Translation quality is measured with BLBURT [Sellam et al., 2020 and COMET (Rei et al. 2020)
on the WMT21 ZH-EN and IWSLT17 FR-EN translation test sets. We note that the WMT-23 metric
overview paper reports high correlations (0.776 and 0.779, Table 1, Freitag et al. (203]) ) between BLEURT and COMET and human judgment. To construct DPO preference dataset for the WMT21 ZH-EN test set, we use BLOOMZ-mt-77b to generate 32 translations (via sampling) for each source sentence in the WMT20 ZH-EN test set. For each source sentence, the translations are ranked by their BLEURT scores computed with respect to the reference translations. The highest and lowest scoring translations form the Clear Preference Pairs; for each source sentence, these the two translations with the greatest difference BLEURT score. By contrast, we take the Tied Pairs as the two non-identical translations with the minimum absolute BLEURT difference; the translation with higher BLEURT is labeled as the winner of each Tied Pair. This yields ca. 16K CPs and TPs for use in DPO. The same procedure is applied the IWSLT17 validation set, yielding ca. 800 CPs and TPs for use as DPO preference datasets. We validate the selected TPs using GPT-4 in Appendix F.8
SFT Models On WMT-21 ZH-EN, performed supervised fine-tuning on the BLOOMZ-mt- 7b Muennighoff et al. 2023 using previous WMT test sets to obtain the SFT model from which we The clear preference pairs and tied pairs are generated by sampling from this SFT model. On IWSLT-17 FR-EN, we use the pretrained BLOOMZZ model directly in sampling clear preferences and tied pairs and in DPO fine-tuning, as we find further SFT leads to repetitive generation.
train with SPPODDOORK//POO.
Training Details We use the RMSProp optimizer with the learning rate set to 57 and the number of warm-up steps set to 150. All NMT experiments are run on two Nvidia A100-80G GPUs with an effective batch size of 4. We used FP32 for training the policy. The log-probabilities from the reference model are pre-computed with FP32 precision. Each training run takes z 2 hours on WMT20 ZH-EN CP+TP data and Rs 1 hour IWSLT17 FR-EN data.
Decoding Following Yang et al. 2024c we use beam search with a beam size to decode all models.
24

---

Held-out Clear Preference Pairs and Tied Pairs We curate held-out sets by generating translations by sampling on the WMT18 ZH-EN test set. Clear Preference Pairs and Tied Pairs are identified using their rankings under BLEURT exactly done for WMT21 ZH-EN. This gives 3980 CPs and 3980 TPs for held-out evaluation in Appendix
E.2 Experiments with Stronger Translation Systems
We conducted additional experiments, replacing the BLOOMZ-mI-7B model with the ALMA-7B- LoRA model reported in Xu et al. 2024 which has better translation performance.
We repeated the procedure described in Section 3.1 to collect 3000 Clear Preferences (CPs) and 3000 Tied Pairs (TPs) for ZH-EN translation. This done by sampling candidate translations using ALMA-7B-LORA on the ZH-EN training set of et al. [2024), followed by pair selection of used the average KIWI-XXL and XCOMET to rank responses. We evaluated our models on the ZH-EN test set of Xu et al. [2024), which consists of 2009 parallel sentences selected from FLORES-200 We report COMET, KIWI-XXL, and XCOMET as evaluation metrics. We also report KL. divergence with respect the reference model.
based scores from automatic metrics. Following Xu al. [2024,,
The table below shows the best-performing system evaluated on ZH-EN translation obtained from a B sweep in [0.1, 0.3, 0.5, 0.7, 0.9) under each configuration. For reference, provide the performance of the base HLLA--B-LoRRE represents the best 7B model reported in Xu et al. [2024] trained with Contrastive Preference Optimization (CPO) on a larger dataset than the one we used. While it not directly comparable, serves strong translation baseline.
model as reported in Xu et al. (002). Additionally, ALMA-7B-R
Model
KL
COMET
KIWI-XXL
XCOMET
Mean(X+K)
ALMA-7B-LORA Xu et al.
2024
79.8
73.7
83.9
78.8
ALMA-7B-RU Xu al. [2024]
81.0
75.7
90.7
83.2
Our Systems
DPO(CP)
101.64
80.5
75.2
91.1
83.2
DPO(CP+TP)
60.37
80.2
75.0
90.8
82.9
DPO-RKICP+TP)
75.57
80.2
75.0
90.8
82.9
DPO-D(CP+TP)
30.92
80.5
75.9
90.8
83.4
Table 6: Performance metrics for ZH-EN translation systems. Results include KI. divergence, COMET, KIWI-XXI, and XCOMET scores, along with the mean of XCOMET and KIWI-XXI.. ALMA-7B-LORA and ALMA-7B-H results are from Xu et al. [2024].
We achieve the best performance under the metrics with DPO-D(CP+TP), which also has the lowest KL divergence, top of very strong baseline. Consistent with all our other results, including Tied Pairs in training procedures yields a strong regularization effect.
E.3
Summarization
CPs and TPs in Summarization. We follow Amini et al. [2024b] in DPO fine-tuning of Pythia- 2.8B [Biderman al. 2023) on the TL:DR dataset [Stiennon et al. [020] with evaluation via win-rate against human- written summaries. Previous works use GPT-4 to compute the win-rate [Rafailoy] jet al., 2023 Amini et al. 2024a). We find that the judgments of PairRM [Jiang et al. 2023 agree well with those of GPT4 (Appendix(F66) and opt to use PairRM win-rate as cost-efiective automatic metric. In the TL;DR task, each prompt is associated with collection of paired summaries, with winner and a loser identified for each pair. There is no immediately obvious way to distinguish TPs from CPs in the collection and so we use DPO itself to select TPs. We first apply DPO with 3 3 0.1 on the full TL;DR training dataset. Using the reward model formed by this model and the reference model, we compute the reward margins of all pairs of summaries in the training split. For each prompt, the pair with minimal reward margin treated as TP, with all other pairs kept CPs, yielding ca. 14k (15.3% TPs. See Appendix F.7 for a study of this selection strategy.
We follow Amini et al. 2024b in experimental setups. The preference dataset is obtained via sampling and ranking with a DPO model without requiring an external reward model as explained in 3.c
25

---

We follow Amini et al. [2024b] to supervise-innttue finetune a Pythia-2.8B model Biderman a.. 2023 on the chosen responses in TL.DR train split for one epoch obtain the initial checkpoint for preference learning. We use the summarization prompt provided in Appendix D.2 by Rafailov et al. 2023
SFT Model
Training Details We use the RMSProp optimizer with the learning rate set to 5e and the number of warm-up steps set to 150, All summarization experiments are run on two Nvidia 100-400G GPUs with effective batch size of 64. We used FP32 for the policy and FP16 for the reference model. Each training run takes 7 hours on TL:DR CP+TP data.
Decoding We use greedy decoding for all models as we find it performs on-par or better than temperature sampling (Appendix F.)
E,4 Tabulated KI.-Performance Results on NMT and Summarization
We tabulate the KL-Performance results shown in Figure and Figure 2
E.4.1 Neural Machine Translation
In addition KL Divergence and BLEURT, we also provide COMET Rei et al. 2020 scores, BLEU Post 2018 scores and BLEU's Length Ratio.
We observe the "reward hacking" phenomenon identified by Yang al. 2024c both WMT21 ZH-EN and IWSLT17 FR-EN where systems achieve good BLEURT but have large length ratio (>1.5) and lower COMET than the pre-DPO system. These systems learn generate long, repetitive translations which BLEURT fails to recognize as low-quality. Yang et al. (024c) find that using small beta values (e.g. 0.1) in DPO training results in reward hacking models. Our results consistent with their findings and further suggest that large KI. divergence from the reference model is good indicator for reward hacking. On WMT21 ZH-EN, the only model that exhibits reward hacking is trained by DPO(CP) with beta=0.1 which also yields the highest KL divergence (174.13) among all models, greadl exceeding the hechnet KL divergence (68.12). On IWSLT17 FR-EN, Almost all models with KI. Divergence 30 (DPO(CP), 3 = 0.1, DPO-RK(CP+TP), 3 = 0.1 and (PO-D(CP+TP) B 0.1, 0.5) show reward hacking behaviours.
Reward hacking on NMT can be resolved by increasing regularization with respect to the reference model. We find that training with larger beta values incorporating ties in DPO-RK/DPO-D training can provide such regularization without performance degradation.
E.4.2 Summarization
Table Ishows the KL.-PairRM winrate TL;DR summarization.
26

---

System
beta
KL Divergence
BLEU
Length Ratio
COMET
BLEURT
Bloomz-mt-765
17.6
77.9
61.6
DPO(CP)
0.1
174.13
7.23
3.01
70.2
67.7
DPO(CP)
0.2
68.12
20.8
1.10
80.8
66.2
DPO(CP)
0.3
62.85
20.7
1.13
80.6
66.4
DPO(CP)
0.4
56.02
21.4
1.09
80.7
66.4
DPO(CP)
0.5
50.99
21.2
1.11
80.8
66.5
DPO(CP)
0.6
47.97
21.5
1.09
80.9
66.5
DPO(CP)
0.7
44.08
21.5
1.11
81.0
66.7
DPO(CP)
0.8
41.88
21.3
1.14
80.8
66.7
DPO(CP)
0.9
41.24
21.5
1.14
80.8
66.8
DPO(CP)
1.9
33.69
22.3
1.09
81.2
67.0
DPO(CP)
1.2
32.01
22.4
1.09
81.3
67.1
DPO(CP)
1.5
29.58
21.7
1.13
81.1
67.1
DPO(CP)
1.55
29.01
21.9
1.13
81.1
67.1
DPO(CP+TP)
0.1
51.29
20.3
1.16
80.0
66.0
DPO(CP+TP)
0.2
36.37
18.8
1.30
80.1
66.6
DPO(CP+TP)
0.3
26.15
19.5
1.24
80.2
66.6
DPO(CP+TP)
0.4
18.21
20.6
1.20
80.4
66.6
DPO(CP+TP)
0.5
15.47
21.2
1.15
80.4
66.4
DPO(CP+TP)
0.6
14.74
21.9
1.10
80.6
66.4
DPO(CP+TP)
0.7
13.29
22.1
1.11
80.5
66.4
DPO(CP+TP)
0.8
12.57
22.2
1.10
80.5
66.2
DPO(CP+TP)
0.9
12.10
21.9
1.10
80.5
66.3
DPO(CP+TP)
1.0
11.43
22.0
1.11
80.5
66.2
DPO-RK(CP+IP)
0.1
48.55
19.3
1.22
80.2
66.9
DPO-RK(CP+TP)
0.2
28.61
22.1
1.11
80.9
66.9
DPO-RK(CP+TP)
0.3
20.21
22.5
1.11
81.0
67.1
DPO-RK/CP+TP)
0.4
14.80
22,4
1.12
81.1
67.1
DPO-RK(CP+TP)
0.5
11.66
22.8
1.10
81.0
67.1
DPO-RK(CP+TP)
0.6
9.74
22.2
1.13
80.8
66.8
DPO-RK(CP+TP)
0.7
8.04
22.3
1.12
80.8
66.7
DPO-RK(CP+TP)
0.8
8.10
22.1
1.13
80.8
66.8
DPO-RK(CP+TP)
0.9
7.58
21.8
1.15
80.7
66.8
DPO-RK(CP+TP)
1.0
6.31
22.3
1.11
80.7
66.6
DPO-D(CP+TP)
0.2
42.74
21.4
1.13
80.8
66.6
DPO-D(CP+TP)
0.3
38.56
21.2
1.15
80.2
66.5
DPO-D(CP+TP)
0.4
17.01
22.5
1.11
81.0
67.1
DPO-D(CP+TP)
0.5
20.20
22.7
1.10
81.1
67.1
DPO-D(CP+TP)
0.6
26.85
22.3
1.10
81.1
66.9
DPO-DICP+TP)
0.7
14.97
22.6
1.11
81.1
67.1
DPO-DICP+TP)
0.8
13.33
22.7
1.11
81.1
67.1
DPO-D(CP+TP)
1.0
10.05
22.3
1.12
80.9
67.0
Table 7: KL-PPefformanc evaluated on WMT-21 ZH-EN.
27

---

System
beta
KI. Divergence
BLEU
Length Ratio
COMET
BLEURT
Bloomz-mt-7bl
17.6
85.4
74.8
DPO(CP)
0.1
53.60
25.8
1.40
82.3
74.7
DPO(CP)
0.3
30.80
23.7
1.60
83.6
76.5
DPO(CP)
0.5
16.70
36.8
1.00
86.1
76.2
DPO(CP)
0.7
13.80
38.5
1.00
86.4
76.4
DPO(CP)
1.0
12.40
38.6
1.00
86.5
76.5
DPO(CP)
1.2
11.80
38.8
0.98
86.5
76.5
DPO(CP)
1.5
10.70
38.9
0.99
86.5
76.5
DPO(CP+TP)
0.1
35.60
35.8
1.00
85.6
75.5
DPO(CP+TP)
0.3
25.80
35.7
1.10
85.4
75.9
DPO(CP+TP)
0.5
22.00
35.1
1.10
85.8
76.3
DPO(CP+TP)
0.7
17.00
38.7
1.00
86.3
76.3
DPO(CP+TP)
1.0
11.50
38,9
1.00
86.4
76.4
DPO(CP+TP)
1.2
8.50
39.1
0.98
86.5
76.4
DPO(CP+TP)
1.5
6.30
39.0
0.98
86.4
76.3
DPO-RK(CP+TP)
0.1
46.70
23.0
1.60
78.7
76.3
DPO-RK(CP+TP)
0.2
19.51
35,9
1.05
85.9
76.4
DPO-RK(CP-TP)
0.3
15.50
36.1
1.10
86.1
76.5
DPO-RK(CP+TP)
0.5
13.30
31.4
1.20
85.7
76.6
DPO-RK(CP+TP)
0.7
10.90
31.3
1.20
85.8
76.5
DPO-RK(CP+TP)
0.8
10.90
29.9
1.28
85.6
76.5
DPO-RK(CP+TP)
0.9
11.60
27.2
1.40
85.3
76.4
DPO-RK(CP+TP)
1.0
11.60
26.1
1.50
85.1
76.3
DPO-RK(CP+TP)
1.2
11.80
24.4
1.57
84.8
76.3
DPO-DICP+TP)
0.1
48.60
25.3
1.41
82.6
76.3
DPO-D(CP+TP)
0.3
19.90
35.4
1.07
85.8
76.5
DPO-DICP+TP)
0.5
51.90
8.4
4.35
75.1
76.1
DPO-D(CP+TP)
0.7
12.80
36.6
1.06
86.2
76.6
DPO-DICP+TP)
1.0
10.30
37.8
1.03
86.3
76.6
DPO-D(CP+TP)
1.2
10.90
32.1
1.20
85.9
76.6
Table 8: KL-Performance evaluated on IWSLT17 FR-EN
28

---

System
beta
KL Divergence
PairRM Winrate
Pythia-2.8B-SFFF Greedy
0.00
37.5
DPO(CP)
0.025
97.03
67.9
DPO(CP)
0.05
60.31
70.3
DPO(CP)
0.07
57.14
71.5
DPO(CP)
0.08
38.16
66.4
DPO(CP)
0.10
26.82
62.5
DPO(CP)
0.30
9.97
63.7
DPO(CP)
0.50
5.79
59.0
DPO(CP)
0.70
3.78
57.8
DPO(CP+TP)
0.025
87.66
63.7
DPO(CP+TP)
0.03
119.60
66.8
DPO(CP+TP)
0.04
70.69
69.5
DPO(CP+TP)
0.05
35.39
63.3
DPO(CP+TP)
0.10
17.21
57.4
DPO(CP+TP)
0.30
4.50
58.6
DPO(CP+TP)
0.50
7.61
57.8
DPO(CP+TP)
0.70
2.91
55.9
DPO-RK(CP+TP)
0.04
80.86
65.2
DPO-RK(CP+TP)
0.05
62.57
67.2
DPO-RK(CP+TP)
0.10
40.50
67.6
DPO-RK/CP+TP)
0.20
22.24
67.6
DPO-RK(CP+TP)
0.30
12.45
68.0
DPO-RK(CP+TP)
0.50
6.15
65.6
DPO-RK(CP+TP)
0.70
4.33
61.7
DPO-D(CP+TP)
0.05
82.35
64.8
DPO-D(CP+TP)
0.10
54.06
71.5
DPO-D(CP+TP)
0.20
39.23
66.0
DPO-DICP+TP)
0.30
22.46
68.8
DPO-D(CP+TP)
0.40
12.57
67.6
DPO-D(CP+TP)
0.50
9.92
67.2
DPO-D(CP+TP)
0.70
6.82
64.8
Table 9: KL-PairRM winrate against 256 human-written summaries on TL:DR summarization
29

---

E,5
Standard Deviation on GSM8K Mathematical Reasoning
B
DPO (CP)
DPO-RK (CP+TP)
DPO-D (CP+TP)
0.1
80.4% + 1.34%
82.9% 0.90%
82.2% + 1.95%
0.3
83.7% + 0.45%
84.3% 0.18%
83.7% + 0.68%
0.5
83.5% + 14%
84.1% + 0.08%
84.2% + 0.15%
0.7
83.6% + 0.53%
83.4% 0.19%
84.5% + 0.58%
1.0
83.8% + 0.29%
84.0% + 0.08%
83.7% + 0.42%
Table 10: Median accuracy (+ standard deviation) GSM8K test across five random seeds for
each 3 value. Tie-compatible variants (DPO-RK, DPO-D) consistently outperform the baseline DPO.
We experiments for five random seeds for each variant at each value (75 training in total). In Table TO we report the median accuracy and standard deviation. Consistent with our previous finding, tie-compatible variants achieve higher median accuracy all 3 values. Gains generally greater than one standard deviation.
Additional Supporting Experiments and Analyses
F1 Sensitivity Study on the Value of a for DPO-RK and DPO-D
In our early experiments with weaker translation model, we find that the choice of a does not have a strong impact on final task performance. In Table [T] we vary the hyperparameter a and report the KL divergence, KIWI-22, BLEURT, and COMET scores with the BLOOMZ-mt-77 setup evaluated on WMT-21 ZH-EN.
a
KL
KIWI-22
BLEURT COMET
DPO-RK
0.5
11.23
78.9
66.9
81.0
In3
14.80
79.0
67.1
81.1
2.0
21.84
79.1
67.1
81.1
3.0
30.41
79.0
66.9
80.9
5.0
76.66
69.9
68.9
74.6
DPO-D
-2.5
11.56
79.0
67.0
81.0
-1.0
15.25
79.0
67.1
81.1
0.0
11.66
79.2
67.1
81.0
1.0
29.72
79.0
67.0
81.0
2.5
47.20
79.2
67.1
81.1
Table 11: Effect varying a DPO-RK (3 = 0.4) and DPO-D (3 = 0.5) systems. For DPO-RK.
= ORK in [5] For DPO-D, a = In in [6 Results are reported for KL divergence, KIWI-22, BLEURT, and COMET metrics. Default values of are underlined.
We find that (1) The final task performance is stable over range of a values (note that a is illegal for DPO-RK) and (2) for large values of there is relatively little regularization as measured by KL divergence.
As explained in Sec 2.3.1 a controls the width of the band reward margins where there's little or no gradient contributions from the tied pairs whose difference reward falls within the band. This explains the lack of regularization under large Given large a, the gradient on TPs approach zero regardless of the reward margin of the pair under the current model. As show in the paper, tied pairs contribute to regularization effect. This explains why large and hence little gradient contribution from TPs leads relatively little regularization.
F.2
Grid search of 3 and WMT
We further conduct a grid search over 33 and values on WMT investigate their effects. The results shown in Table 22
30

---

Variant
KL
COMET
0.1
19.73
80.9
0.5
22.31
81.0
DPO-RK (CP+TP) 0.2
In3
28.61
80.9
2.0
37.05
81.1
3.0
49.00
81.0
0.1
9.54
80.9
0.5
14.80
81.0
DPO-RK (CP+TP) 0.4
In3
21.84
81.1
2.0
30.41
81.1
3.0
76.66
74.6
0.1
6.32
80.7
0.5
5.91
80.6
DPO-RK (CP+TP)
0.8
ln3
8.10
80.8
2.0
10.37
80.9
3.0
14.53
80.9
-25
27.46
80.9
-.0
34.80
80.6
DPO-D (CP+TP)
0.2
0
42.74
80.8
1.0
56.83
80.6
2.5
14.56
81.0
1.0
18.71
81.0
DPO-D (CP+TP)
0.4
0
17.01
81.0
1.0
33.15
81.0
2.5
56.18
80.7
11.56
81.0
1.0
15.25
81.1
DPO-D (CP+TP)
0.5
0
11.66
81.0
1.0
29.72
81.0
2.5
47.20
81.1
2.5
8.16
80.7
1.0
9.10
80.7
DPO-D (CP+TP)
0.8
0
13.33
81.1
1.0
19.54
81.0
2.5
30.48
80.8
Table 12: Grid search results for 3 and a on WMT showing the corresponding KL divergence and COMET scores.
Consistent with our findings in Appendix F.1 3 and values. These results add further empirical evidence supporting the role of B and as knobs for regularization strength. Controlled for fixed value of 3, increasing a and generally leads to higher KL divergence with respect to the reference model at the end of training.
find that task performance is stable over range of
Based these experiments, recommend the following hyperparameter tuning scheme for DPO- RK and DPO-D:
Run training across range values with the default a to identify the effective range of B.
2. Adjust (and B, if desired) those values, using the final KL divergence guide.
As shown in experiments, there is typically "sweet range" of KL where performance is optimal. This procedure allows practitioners to efficiently identify the best performance- KI. trade-off for given setup.
31

---

F.3 GPT-4 Evaluation on Summarization
For TL;DR, we additionally report the win-rate as evaluated by GPT-4 of the best- rest-performing systems in [3
Model
PaiRRM
GPT-
Pythia-2.8-DPOCCE
71.5%
62.1%
71.5%
64.1%
69.5%
62.1%
68.0%
57.8%
Pythia-2.6 -aaiaa-2.-SFT
37.5%
36.7%
Table 13:
Win-rate comparison of the best-performing systems evaluated using PairRM and GPT-4.
We note that the rank based on PairRM win-rate agrees with the rank produced by GPT-4 except that GPT-4 prefers Pythia228-DPO-D(CP-17
F,4 Varying the Percentage of Ties
In Table 14 vary the proportion of TPs and evaluate on the WMT-21 ZH-EN dataset, keeping the CPs unchanged. We train systems with three 3 values [0.2, 0.4, 0.6) and report the best-performing system in terms of BLEURT. We observe that, under the same / value, including the KI. divergence with respect to the reference model at the end of training while maintaining the same performance. This provides further empirical evidence for our analysis in Sec 3.3 where ties regularize training.
TPs reduces
TP%
KL
KIWI-22
BLEURT COMET
DPO-RK (CP+TP)
25%
10.16
78.4
66.5
80.7
50%
6.05
78.2
66.2
80.4
75%
4.40
78.5
66.4
80.7
DPO-D (CP+TP)
25%
17.26
78.5
66.4
80.8
50%
10.40
78.5
66.3
80.7
75%
6.63
78.5
66.5
80.8
Table 14: Regularization effects of varying TPs in DPO-RK and DPO-D systems. KIWI-22 Rei et al.
2022 reference-less metrie. A TP% of 50% means that keep 50% of all the TPs, resulting in CPs:TPs ratio of 2:1.
F.5 IPO Baselines and KIWI-22 Evaluation on WMT
We additionally evaluate systems with KIWI-22 (Rei et al. 2022 popular reference-less neural
metric used by recent works in evaluating state-of-ther (SoTA machine translation systems Xu
al. 2024].
Model
Best BLEURT
Best COMET
Best KIW1-22
Bloonz-mt-7bl-SFT
61.6
77.9
77.3
Bloomz-mt-7bl (CP)
67.1
81.3
82.0
Bloomz-mtt-bl-DDP (CP+TP)
66.6
80.6
78.5
Bloomz-mt--bl-DPa (CP+TP)
67.1
81.0
79.0
-DDDD-- (CP+TP)
67.1
81.0
79.2
Bloomz-mt-7bl-IPO (CP)
66.6
80.6
78.6
Bloomz-mt-7bl-IP- (CP+TP)
66.3
80.5
78.4
Table 15: Evaluation results under BLEURT, COMET, and KIWI-22 metrics for various models.
DPO and its variants show clear gains over the SFT baseline across all three metrics.
perform additional NMT experiments with IPO fine-tuning on CPs (IPO(CP+TP) and CPs+TPs (IPO(CP+PP)). We conduct IPO fine-tuning (using code from http::/githhb.com/rrrimmiiiii
32

---

seffreece-optimmzaaion with the range of beta values [0.025, 0.05, 0.1, 0.3, 0.5, 0.7, 1.0] and report the best performing system as evaluated on the WMT-21 ZH-EN dataset in Table We use identical training hyper-parameters
used in DPO, DPO-RK and DPO-D fine-tuning. We note that these metrics are strongly rated in the WMT-23 metric overview paper [Freitag et al. 2023]
We find that (1) both IPO(CP) and IPO(CP+TP) yield substantial gains compared to the baseline SFT IPO causes a small degradation (<-0.3) across all metrics. (3) IPO(CP) performs on par with DPO(CP+TP) but under-performs DPO(CP), DPO-RK(CP+TP), and DPO-D(CP+TP) on all metries.
model (+4.7 BLEURT. (2) Adding TPs to
F.6 PairRM as Proxy Evaluator for GPT-4
System
GPT-4
PairRM
DPO
T=1.0
23.4%
27.3%
T=0.75
40.2%
40.6%
T=0.5
52.3%
54.7%
T=0.25
46.9%
51.6%
T=0.0
50.4%
55.5%
SFT
T=1.0
22.3%
23.0%
T=0.5
37.5%
38.7%
T=0.0
36.7%
39.8%
Table 16: Win-rate of Pythia-2.8B model SFT/DPO on TL;DR train against 256 suman-writted summaries judged by GPT4-0613 and PairRM.
PairRM (Jiang et al. 2023 is a strong reward model that has been shown to be effective in curating preference datasets for iterative DPO training [Tran et al.) 2023). In our experiments on TL;DR sum- marization, we use the PairRM reward model instead of GPT-4 for comparing generated summaries against human references. In this appendix, we show that win-rate as judged by PairRM a good proxy for GPT4-0613 [O et al. 2024 win-rate on the TL:DR dataset Stiennon et al. 2202)).
We generate summaries from SFT pythia-2.8B model by sampling at temperature T [0.0, 0.5, 1.0] and the DPO model (3 = 0.1) trained on TL:DR's full training set at temperature T [0.0, 0.25, 0.5, 0.75, 1.0]. Their win-rates against the 256 human- written summaries in the TL:DR valid-2 split as judged by GPT-4 and PairRM are tabulated in Table 16 We find that the win-rates by GPT-4 and PairRM are similar and that system rankings are generally preserved. We opt to use PairRM as our evaluation metric which enables us to conduct experiments faster and at lower costs.
human-written
E.7 Verifying a Tied Pair Selection Strategy for TL;DR
As explained in Sec. 3.1 we use the reward model associated with the DPO model trained on TL;DR to identify summarizations that are similar in quality. Note that we are performing unsupervised labelling of ties in the DPO training data, which is somewhat more forgiving than the classification task discussed other sections which requires labelling ties in held-out data not seen in training. We do however assume that the reward model should perform well on the data it was trained on.
To investigate these assumptions, we swap the preferred and the dispreferred responses in all tied pairs to form "reversed Tied Pairs" (rTP). If the responses in TP are truly similar in quality (i.e., it acceptable reverse the preference direction), training with (PO(CP+TP) and DPO(CP+TP)) should yield similar performing models. Furthermore, the DPO-RK and DPO-D learning procedures which explicitly model tied pairs should yield better performing model. We conduct experiments on TL:DR. Table 17 shows that the performance relation DPO-D(CP+TP) ~ DPO-RK(CP+TP) > DPO(CP+TP) (PO(CP+TPP) indeed holds for TL;DR, which suggests that our Tied Pair selection strategy is reasonable.
33

---

System
PaiRRM
+ TP)
58.6%
DPO(CP+TTP) 60.9%
DPO-RK(CP+TP) 68.0%
DPO-DICP+TP) 68.8%
Table 17: Win-rates of Pythia-2.8B model DPO on TL;DR train against 256 human-written summaries judged by PairRM. Systems trained CP+TP or CP+rTP data with DPO, DPO-RK, DPO-D at fixed 3 = 0.3. For DPO-RK and DPO-D learning, rTP is equivalent to TP as there is no preference direction for ties.
F88 Using GPT-4 validate ties NMT
We conduct the following experiments with GPT-4 to evaluate whether the tied pairs selected by BLEURT reliable.
randomly select 128 source sentences on our training set sampled from WMT20 ZH-EN and collect their corresponding CPs and TPs as determined by BLERUT score's 's ranking. Then, for each source sentence, we present the two pairs translations to GPT-40-0024-08-06 and prompt it to decide which pair of translations is tied pair.
We find that on the 128 randomly selected examples from WMT20 ZH-EN, the GPT-4o model agrees 82.03% of the time with BLEURT's assigmment of CP and TP.
An example of our prompt and GPT-4's response below. In this case, GPT-4 chooses the pair (Pair A) as TP as BLEURT.
The prompt for GPT-4 and example responses provided below.
Pair A:
Better Translation: Beijing Youth Daily reporters learned that there were many important amendments in the first published drafts". Worse Translation: Beijing Youth Daily reporters learned that the "First Draft" has many important modifications.
Pair B:
Better Translation: The Beijing Youth Daily reporter learned that there were many important changes in the Draft for the first time published. Worse Translation: Beijing Youth Daily reporter learned that the first version the "Comments" had been revised many times.
You should provide short rationale and then decide which pair is tied pair. You should follow the format "DECISION: [[A]]" "DECISION: [[B]]" to give your decision.
Response:
To determine which pair represents a tied pair, let's evaluate the differences between the translations in each pair.
In Pair A:
The "better" translation is: "Beijing Youth Daily reporters learned that there many important amendments in the first published 'drafts'." The "worse" translation is: "Beijing Youth Daily reporters learned that the 'First Draft' has many important tant modifications."
Both translations in Pair convey similar meanings, describing modifications or amendments to a draft. The primary differences are stylistic, such as the placement and quotation of "draft"
In Pair B:
The "better" translation is: "The Beijing Youth Daily reporter learned that there were many important changes in the Draft for the first time published." The "worse" translation is: "Beijing Youth Daily reporter learned that the first version of the 'Comments' had been revised many times."
34

---

In Pair B, the translations have significant differences in meaning. The "better" translation accurately captures notion "important changes" and "first published," " whereas the "worse" translation introduces "first version of the "Comments'" and a focus on revisions rather than initial publication, which diverges more from the source sentence.
With this analysis, Pair has translations that are more similar meaning and are thus likely to be the tied pair, while Pair B shows a clearer distinction in translation quality.
DECISION: [[A]]
Listing 1: Prompt and Response for using GPT-4 to validate tie selection NMT
35

---

G Simulating the Perfect DPO-Davidson Policy
In Section 3.2 we make use the relationship derived by Chen et al. [2024] Appendix A.2] which specifies the optimal DPO policy to minimize the binary classification loss
minIP(y1 >= 1/2) log क (V1 >= V2)]
+ (1 - (v >a y2)) log(1 T >= v2))
where P(/1 > y2) is the human ground truth preference distribution.
We extend the analysis of et al. 2024 to include the Davidson model, noting that the binary maximum likelihood objective becomes ternary. We assume we have the ground- truth human preference distributions P111 >= 12). P(222 >~ = 1/1. and P(y1 ~ Y2) needed to define the objective. The resulting Theorem T can be viewed as a generalization of Theorem 3 of Chen et al.) [0224] that
allows for the observations of ties. Where ties not allowed (i.e. up 0), the Davidson model
simplifies to the Bradley-Terr, model and Theorem 3 of Chen et al. 2024 recovered as a special
case of Theorem
Theorem (Simulating Perfect DPO-D Policy). Assume
given an aggregated comparison datapoint (3r, y1, 1/2 and human ground-truth preference probabilities (P > y2), IP(y) >x y2 and P(y1 <e y2) which obey the Davidson model with hyper-parameter VD- Let the reference model be Tref- It follows that the perfect OPO-Davidson policy x on this aggregated comparison datapoint satisfies
(yi [2)
Tref (y1 (x)
P(yl >= 122)
1/8
(25)
(32(z)
Fref (2/2 x))
Plyz >x V11
or equivalently
n (yn (x) >' (3222)
(v1l())
P(1 = U2)
2/8
2VD
(26)
Tres (U2l2r)
P ~2 4/2
Proof.
The DPO-D policy objective optimizes the following three-way classification loss:
minP((y > 9/2 log π ( >= 1/2)
+ (P(22 = 31) log log (122 > M1) + PPy) ~a y2) log F(ツ ~> y/2
Let 0 denotes set of parameters such that væ- is an optimal policy for the above loss, then me- satisfies:
T- (y) >x y/2 P(11 >= V/2
(yz >x yn ) = P(yz > 3/1
TØ- (y) ~r 1/2 F ~ 422
Expressing the policy probability To- (Nw > vi) and wo- (ir x2 vho) in terms of the reward margins dg- (Ix you y:):
1
P > 82) + e do- (8. m1l + 2uDe do- (2)22
d- (w.v
P(yz > v1) =
1 + e do- (2.01.02) + 2u (2.21.82/22
Rearranging, we have
P(ya > 111
exp ( do- (2,
v1, ,12))
P( >x 122
exp (glog
Tg. (122 (z)
T (111 (x)
B log
Tree (22
Tre (v1 (z)
36

---

Taking logarithms on both side and divide by B.
P(yn >r W2) log
Th- (12 Tray (3/ (x)
log
8
P(yz >= v1)
Tre (V2 (T) [x)
Exponentiating both sides gives
T (y2/z)
Tree (12 (x)
P2y2 > v v1)) 1/9
TT- (y1 (x)
Trey (1i [x) P(y >= 82)
Taking the inverse yields Eq 25
To see the equivalence between Eq 25 and Eq
26 note that the ground-truth preference and tie probabilities which obey the Davidson model satisfy the following relation:
P(y: ~r y2) - 2 Plyl > 122 P(v2 >- v1)
Rearranging Eq 25
nT" (1/1 [xr)
(Vr (x:)
P(y1 >a 8/2) 1/8
nT* (3/2 (z)
/re (322 (z)
P(12 vx V1)
P(y1 x y22
2/3
Tree (3/2 (x)
P2(322 v 3/1)
Tref (3/1 (x:)
Plyl b 122)
2/3
Tree (3/2 (27)
Py/1 > U2 PVy2 x V/
Tre/ (3/1 (x)
2/D Ply1 > y2)
2/8
Tree (32
P(y) ~r 3/2
which is Eq 26
H Extended Related Work
Variants of Direct Preference Optimization A range of variants of Direct Preference Optimiza- tion have been proposed based problem-specific theoretical motivations. Park et al. 2024 tackle excessive response length by introducing explicit length normalization in the DPO objective. SimPO [Meng al.. 2024) modifies the DPO objective remove the need for reference model and to include length normalization. KTO EEhayarajh et al., 2024 is motivated by Kahneman and Tversky's 's prospect theory and learns from non-paired preference data. ODPO [Amini al. 2024b] incorporates preference strength in the objective by introducing an offset parameter. In deriving ODPO, the offset parameter of Amini et al. [2024a] Theorem 3]) plays a role similar to the sensitivity threshold of Rao and Kupper 1967]. We note that the ODPO objective with fixed offset agrees with our proposed DPO-RK objective restricted to clear preference data, but does not extend ties.
Frameworks for Pair-wise Preference Optimization Several works propose theoretical frame- works for understanding general Preference Optimization from which DPO can be obtained special case. Azar et al. 2024)) introduces the TPPO) formalism which allows alternative expression of the reward in terms of the model's predicted probability. IPO derived when the identity mapping used, and DPO arises under log-rat log-ratio mapping. Dumoulin et al. (2024) formulate learning from pair-wise preference as learning the implicit preference generating distribution of the annotators. In this formalism, DPO is well-specified model for the implicit preference distribution assuming that the human preference generative process follows the Bradley-Terry model. Our work can be viewed as assuming an annotator preference generating distribution that allows for the outcome of a tie (i.e. the Kupper or the Davidson model). Tang et al. (0024) propose generalized approach to deriving offline preference optimization losses through binary classification. In this work, we consider the ternary classification with the possibility of declaring a tie. In Appendix G we show that the 'perfect' DPO-D policy can be simulated starting from the ternary classification loss.
37

---

Pair-wise Comparison Models Hamilton et al. 2023 review the Bradley-Terry model, including
its relation to the logistic distribution |Badley and Gart 1962] and Luce choice axiom Luce [9999b].
The Rao-Kupoo Kupper [Rao and Kupper 1967 and the Davidson model [David 1988) are two notable extensions Bradley-Terry (Sec. (.2 See the review by David [988] and bibliography by Daviddsn
and Farquhar [1976]. Modeling ties remains an active research topic in fields such as sport team ranking [Zhou al. 2022], medical treatments [Gaohong Dong and Vandemeulcbroecke 2020 and
chatbots [Ameli et al. 2025
38

---

I Qualitative Cases from DPO, DPO-RK, and DPO-D Systems
We collect the responses on WMT-21 ZH-EN from the best performing model under each configura- tion. Below, we present two cases where systems trained with perform better than DPO(CP) and one case where they perform worse than DPO(CP) as judged by KIWI-22.
and (O-D((CP+TP)
The KIW1-22 score of each translation is shown in parentheses.
Case 1: ID 10
Source: N,T ズ
Reference: Hometowns are the key to moderate prosperity.
DPO(CP): It is not important whether you are rich or poor, as long as you have a good relationship with your neighbors. (71.6)
DPO(CP+TP): Wealthy or poor, it all depends on neighbors. (72.0)
DPO-RK(CP+TPP; Wealthy poor, it all depends on neighbors. (72.0)
. DPO-D(CP+TP): Whether you are rich or poor depends on your neighbors. (77.9)
In this case, the DPO(CP) system produces a long translation with incorrect meaning. The other three systems produce correct, literal translations. The translation from DPO-D(CP+TP) is more fluent. However, all systems fail to fully capture the meaning of "// "//> (i.e., "moderate prosperity") as done in the reference.
Case 2: ID 269
- BI LE 3T - S
Source:
The airline released a statement claiming that as its aircraft was flying from Seattle, Washington to Austin, Texas, one of the engines suddenly stalled, and the aircraft finally made an emergency landing in Austin.
Reference:
DPO(CP):
This airline operator said in statement that one engine on one of its aircraft failed midflight July 15 from Seattle, Washington to Austin, Texas, and the aircraft successfully made an emergency landing at the Austin airport. (84.5)
DPO(CP+TP): This airline operator said statement that one of its aircraft was on its way from Seattle, Washington to Austin, Texas when one of its engines suddenly went out order, and the aircraft successfully made an emergency landing at the airport in Austin. (81.4)
DPO-RK(CP+TP): This airline operator said in statement that one engine on one of its aireraft failed during the flight from Seattle, Washington Austin, Texas on July 15, and the aircraft successfully made emergency landing the Austin airport. (85.4)
This airline operator said in statement that one engine on one of its aircraft failed during the flight from Seattle, Washington Austin, Texas on July 15, and the aircraft successfully made an emergency landing at the Austin airport. (85.4)
DP)-D(CP+TP):
The translation produced by the DPO(CP+TP) system fails to capture the date of the event ("July 15"). Translations from DPO-RK(CP+TP) and DPO-D(CP+TP) are identical and differ from that from DPO(CP) only in rephrasing "midflight" with "during the flight."
Case 3: ID 91
Source: 3 (ΕEE) EAIRETERM
Reference: The inspiring competitive show for youth sports, We Are The One, was officially broadcast on Hunan TV on July 11.
39

---

The youth sports action show "Be Active, Be Sporty" will officially debut on Hunan Satellite TV July 11. (76.8)
DPO(CP):
DPO(CP+TP):
The youth sports competition show "Go, Boy!" officially landed the Changsha Satellite TV Station on July 11. (75.6)
DPO-RK(CP+TP): The youth sports competition show "Go Play Boys" will officially broadcast on the Hunan Satellite TV July 11. (72.1)
DPO-D(CP+TP): The youth sports competition show "Let's Go Boys" will officially debut on the Hunan Satellite TV on July 11. (74.2)
In this case, all systems produced more literal translation the name of the TV show compared to the reference. Although the KIWI-22 scores for DPO-RK(CP+TP) and DPO-D(CP+TP) are lower, the translation qualities are in fact similar.
Broader Impacts
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See abstract, introduction, and conclusion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
40

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Complete derivation for DPO-RK and DPo-D are provided in Appendix B Complete proof ideal DPO-D policy theory is provided Appendix G
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Hyper-pane parameters of our experiments are provided in E] where model versions, datasets, training hyper- parameters and computing platforms specified. We will publish code on publication.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility also be provided via detailed
41

---

instructions for how to replicate the results, access to a hosted model (e.g., g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code and scripts will be provided upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
cublic/guides/Codeeeee wwwbumsiiiooooiicy.....ooo.oo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https)
ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Appendix
Guidelines: The answer NA means that the paper does not include experiments.
42

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our main results reports grid search hyper-parameter and we report all available results. We note large incraase in computational cost is required to conduct multiple training runs at each hyper-parameter setting. We note that we have reported results on 5 different experimental setups (Appendix E and Sec(..5) with at least 15 training runs per setup (3 DPO variants times minimal beta values), and our findings are consistent across these experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma eгтor bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neeuriin http://///errrs..ccooooooottttta
43

---

Answer: [Yes]
Justification:
Our research has been conducted with strict adherence to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix J
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
This paper has such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
44

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have used open-sourced datasets and models that are widely-used in the research community and well-known at the time of writing. They have been cited where appropriate.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, ppperr swi wwirswiithhe thcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
45

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We do not use LLMs developing the core method in this research.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
for Please refer to our LLM policy http:s////eerrrosssccc////eeeres what should or should not be described.
46


Reviews:
[{"review_id": "guGrnrpQiJ", "reviewer": "Reviewer_Xv6R", "summary": "The author observed that the original DPO does not allow for ties for LLMs’ alignment, i.e., does not allow for ambiguity in the comparison of the paired examples in the training data. The author thereby investigate DPO variants that can incorporate ties during training. The authors proposed two tie-compatible DPO variants, DPO-RK and DPO-D, by replacing the BT preference model with the Rao-Kupper and the Davidson models.", "questions": "1. Are there any other ways to accommodate ties in DPO?\n2. Besides TPs, are there any other factors that have impact on the performance of DPO for LLMs’ alignment? Any discussions?", "limitations": "The main limitation of the paper is that: this is an extension of the standard DPO model, which makes the contribution of the paper sounds marginal.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths and weaknesses:\n1. The motivations of the proposed method is clear: to allow ties for LLMs’ preference alignment during training.\n2. The writing of the paper is easy to follow.\n3. Experiments were conducted to evaluate the performance of the proposed model.\n\nWeaknesses of the paper:\n1. The authors pair the top- and bottom-ranked responses as CPs and TPs as pairs that are close in the ranking. Not sure are there other way to construct CPs and TPs for training? Although the authors detailed the way they constructed the dataset for experiments, the constructed dataset was not publicly available, which making the reimplementation of the experiments impossible?\n2. The paper suffers from the lack of novelty. The paper sounds like an incremental work the extension of existing Direct Preference Optimization, DPO. The main difference lies in the fact that the authors considered TPs in the objective.\n3. Are there any other ways to accommodate ties in DPO? \n4. There are some DPO variants being proposed in the literature. Other DPO variants should be taken into account as baselines in the experiments.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive and careful review. In response:\n\n> The authors pair the top- and bottom-ranked responses as CPs and TPs as pairs that are close in the ranking. Not sure are there other way to construct CPs and TPs for training? \n\nThanks for your questions. We have demonstrated four ways to construct tied pairs: (1) via external automatic metrics (Sec.3.2, NMT); (2) via the trained DPO reward model (Sec.3.2; TL;DR); (3) via identifying pairs of conflicting preference annotations in existing pairwise datasets (Sec.3.5.1); and (4) via identifying pairs where both responses are correct when the label is binary (Sec.3.5.2). Though not exhaustive, we believe these offer a good range of approaches to tackle other downstream tasks. \n\nExpanding on (3) and (4) above, we show how to construct TPs other than based on ranking: by modeling conflicting preferences under multiple annotators as ties (ALMA-R-Preference, Machine Translation, Sec.3.5.1) and by modeling pairs that are both correct as ties (GSM8K, Mathematical Reasoning, Sec.3.5.2). We show that these approaches lead to improved performance over the DPO baseline without TPs. We will highlight these results in the introduction in revision to make clear we studied alternative ways to construct TPs. \n\n> Although the authors detailed the way they constructed the dataset for experiments, the constructed dataset was not publicly available\n\nThanks for raising this point. We will publish the constructed CP+TP datasets upon publication. All datasets used have been statically saved to disk to ensure reproducibility.\n\n> The paper suffers from the lack of novelty..... The main difference lies in the fact that the authors considered TPs in the objective.\n\nThanks for raising this. The DPO-RK/D formulation is one of our contributions. Other contributions include (1) To our knowledge, we are the first to show the regularization effect of including ties in DPO and its variants. We provide extensive empirical results and theoretical analysis to demonstrate this effect (Sec 3.1-3.4); (2) We show two novel ways that model previously discarded data as ties to improve model performance (Sec.3.5).  To our knowledge, this has not been investigated in the DPO literature. \n\n> Are there any other ways to accommodate ties in DPO?\n\nThanks for raising this. We think this is an open research question, for which this paper derives and investigates two tie-compatible variants based on the Rao-Kupper and the Davidson models. To our knowledge, there are no other established methods to accommodate ties in DPO.\n\n> Other DPO variants should be taken into account as baselines in the experiments.\n\nThanks for raising this point. We are specifically interested in the effect of replacing the Bradley-Terry model with tie-compatible comparison models in DPO to accommodate ties. Comparing directly to DPO in the main experiments allows us to study this replacement without interference from the modifications from other variants (e.g., length modifications and leaving out the reference model, etc.). \n\nWe do provide IPO results in Appendix F.4., where we find including ties also lead to a slight degradation in performance and overall inferior performance to DPO. Related variants are discussed in Related Work (Sec.5).\n\nWhile not directly relevant to our investigation on ties, we agree that results from other DPO variants could serve as point of comparison for interested researchers. We have run three established PO methods (IPO, SimPO, and CPO) on the GSM8K dataset using the same CP set as used by DPO. The results are tabulated below:\n\n | β   | SimPO (CP) | CPO (CP) | IPO (CP) | DPO (CP) | DPO-RK (CP+TP) | DPO-D (CP+TP) |\n|-----|------------|----------|----------|----------|----------------|----------------|\n| 0.1 | 82.5%      | 82.6%    | 20.2%    | 76.4%    | **83.5%**          | 81.7%          |\n| 0.3 | 81.8%      | 83.1%    | 57.1%    | 83.7%    | **84.4%**          | 83.2%          |\n| 0.5 | 81.8%      | 83.1%    | 69.1%    | 83.6%    | 83.8%          | **84.5%**          |\n| 0.7 | 81.6%      | 82.6%    | 78.3%    | 83.3%    | 83.7%          | **84.5%**          |\n| 1.0 | 82.2%      | 83.6%    | 76.1%    | 83.5%    | **84.1%**          | 83.7%          |\n\nWe find that in all but one case (CPO(CP), $\\beta=1.0$) these variants give lower accuracy compared to DPO. DPO-RK(CP+TP) and DPO-D(CP+TP) still yield the best performing system at every beta value with the added baselines. We believe these add to the strength of our empirical evidence.\n\n> Besides TPs, are there any other factors that have impact on the performance of DPO for LLMs’ alignment? Any discussions?\n\nRecent literature has studied many factors that could impact DPO performance, including length normalization, doing away with the reference model, token-wise formulation, etc. This is an on-going research topic and is more carefully discussed in recent survey papers (e.g., [Liu et al, 2025]). We note that the problem of accommodating ties remain under-investigated despite the large amount of literature. We discuss relevant DPO variants and frameworks in our Related Work section  (Sec. 5) to the best of our knowledge.\n\n**References**\n\nShunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, Yongbin Li, Dacheng Tao: A Survey of Direct Preference Optimization."}, {"review_id": "tGJpqic2s5", "reviewer": "Reviewer_eagA", "summary": "Direct Preference Optimization (DPO) effectively aligns large language models (LLMs) by maximizing the log-likelihood of preference samples, offering a simpler alternative to traditional reinforcement learning (RL) pipelines. However, the original DPO, which relies on the Bradley-Terry model, cannot handle \"tie\" outcomes common in real-world preference data, which are typically discarded, leading to information loss and performance degradation. To address this, this paper proposes two DPO extensions that support tie modeling: DPO-RK (based on the Rao-Kupper model) and DPO-D (based on the Davidson model), which explicitly model tie outcomes in pairwise comparisons. Experiments demonstrate that both methods significantly outperform the original DPO in translation, summarization, and mathematical reasoning tasks. Furthermore, the inclusion of tie data enhances model stability and generalization capability by providing stronger regularization.", "questions": "**Is Discarded \"Tie\" Data Necessarily High-Quality and Valuable Information?** The paper posits \"tie data is typically discarded, leading to information waste\" (e.g., Introduction, Lines 30-32) as one of its core motivations. However, the robustness of this premise is debatable: In real-world preference data, instances labeled as \"ties,\" beyond genuine preference ambiguity, might also include data that is inherently low-quality, difficult to distinguish in terms of preference, or stems from annotation noise. Could the authors provide stronger evidence or argumentation to support the claim that the \"tie\" data repurposed by their methods generally possesses high informational value? This would be necessary to substantiate the universality and urgency of the \"avoiding waste\" motivation, rather than potentially introducing unnecessary complexity or interference in certain scenarios.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n**Innovative and Effective DPO Extensions to Address Wasted Tie Preference Data**: Addressing the practical limitation of DPO's inability to directly handle \"tie\" preference data, the paper innovatively proposes two variants, DPO-RK and DPO-D. These variants explicitly model tie probabilities by integrating the Rao-Kupper and Davidson probabilistic models, respectively. Experiments demonstrate that these methods, when incorporating tie data, can prevent performance degradation and enhance model performance.\nThorough Validation of the Proposed Methods' Practical Value in Enhancing Model Performance and Utilizing Discarded Data: Through extensive experiments across multiple tasks such as machine translation, summarization, and mathematical reasoning, the paper demonstrates that DPO-RK and DPO-D can effectively utilize tie data previously discarded due to ambiguity or conflict. Consequently, they achieve superior performance compared to strong DPO baselines on these tasks and provide valuable strategies for handling and selecting tie data in practical applications.\n\n## Weaknesses\n**Neglect of β's Influence in α Parameter Sensitivity Analysis**: Although Appendix F.1 explores the sensitivity of α_RK and ν_D parameters, the experiments were conducted using a single β value (e.g., β=0.4 for DPO-RK and β=0.5 for DPO-D). The analysis does not investigate whether variations in β would affect the sensitivity results. Given that β is a core regularization strength parameter in the DPO framework, its value directly influences the model's learning from positive/negative preference samples. Therefore, when investigating the performance impact of α parameters (which control tie-handling strategies), neglecting β might obscure potential interaction effects between α and β. It could also lead to an incomplete understanding of the model's hyperparameter tuning mechanism.\n\n**Inconsistent β Values Between Sensitivity and Main Experiments without Justification**: The β values used in the α parameter sensitivity experiments in Appendix F.1 are inconsistent with those used in the main experiments (e.g., Figures 1 and 2) where the model achieved optimal performance. No reasonable explanation for this choice is provided. This inconsistency raises questions: Why wasn't the sensitivity analysis of α parameters conducted at the β values that yielded optimal model performance? Could different choices of β alter the optimal values of α? The paper does not discuss this, which reduces confidence in the conclusions drawn from the sensitivity analysis.\n\n**Ambiguous Definition of Hyperparameter α in Sensitivity Analysis**: In the main text, the paper explicitly substitutes the tie-control parameter ν_RK in DPO-RK with α_RK. However, it does not provide a similar substitution or equivalent expression for ν_D in DPO-D. Yet, in the sensitivity analysis in Appendix F.1, α is uniformly used to represent the tie-control parameter for both methods without necessary clarification or consistent symbolic linkage. This ambiguous definition and opaque symbol substitution can easily cause confusion.", "was_revised": true, "final_justification": "After the rebuttal, most of my concerns are well addressed. I think this paper can be accepted", "rebuttal": "Thank you for your positive and insightful review. In response:\n\n> Appendix F.1 explores the sensitivity of α_RK and ν_D parameters, the experiments were conducted using a single β value... Could different choices of β alter the optimal values of α?\n\nThanks for raising this and your careful review. We find $\\alpha_{RK}=ln3$ and $\\nu_D=1$ work well for the range of beta values (0.1-0.9) tested, but agree that there is an opportunity to finetune the hyper-parameter for optimal performance as we discuss in the Limitations section (Line 658-662). We have conducted a grid search of beta and alpha values on WMT to investigate this. Please see the table below for results:\n\n| Variant          | $\\beta$ | $\\alpha_{RK}$ | KL    | COMET |\n|------------------|---------|---------------|-------|--------|\n| DPO-RK(CP+TP)     | 0.2     | 0.1           | 19.73 | 80.9   |\n|                  |         | 0.5           | 22.31 | 81.0   |\n|                  |         | ln3           | 28.61 | 80.9   |\n|                  |         | 2.0           | 37.05 | **81.1**   |\n|                  |         | 3.0           | 49.00 | 81.0   |\n| DPO-RK(CP+TP)     | 0.4     | 0.1           | 9.54  | 80.9   |\n|                  |         | 0.5           | 14.80 | 81.0   |\n|                  |         | ln3           | 21.84 | **81.1**   |\n|                  |         | 2.0           | 30.41 | **81.1**   |\n|                  |         | 3.0           | 76.66 | 74.6   |\n| DPO-RK(CP+TP)     | 0.8     | 0.1           | 6.32  | 80.7   |\n|                  |         | 0.5           | 5.91  | 80.6   |\n|                  |         | ln3           | 8.10  | 80.8   |\n|                  |         | 2.0           | 10.37 | 80.9   |\n|                  |         | 3.0           | 14.53 | 80.9   |\n\n \n| Variant          | $\\beta$ | $\\alpha_{D}$ | KL    | COMET |\n|------------------|---------|--------------|-------|--------|\n| DPO-D(CP+TP)     | 0.2     | -2.5         | 27.46 | 80.9   |\n|                  |         | -1.0         | 34.80 | 80.6   |\n|                  |         | 0            | 42.74 | 80.8   |\n|                  |         | 1.0          | 56.83 | 80.6   |\n| DPO-D(CP+TP)     | 0.4     | -2.5         | 14.56 | 81.0   |\n|                  |         | -1.0         | 18.71 | 81.0   |\n|                  |         | 0            | 17.01 | 81.0   |\n|                  |         | 1.0          | 33.15 | 81.0   |\n|                  |         | 2.5          | 56.18 | 80.7   |\n| DPO-D(CP+TP)     | 0.5     | -2.5         | 11.56 | 81.0   |\n|                  |         | -1.0         | 15.25 | **81.1**  |\n|                  |         | 0            | 11.66 | 81.0   |\n|                  |         | 1.0          | 29.72 | 81.0   |\n|                  |         | 2.5          | 47.20 | **81.1**   |\n| DPO-D(CP+TP)     | 0.8     | -2.5         | 8.16  | 80.7   |\n|                  |         | -1.0         | 9.10  | 80.7   |\n|                  |         | 0            | 13.33 | **81.1**   |\n|                  |         | 1.0          | 19.54 | 81.0   |\n|                  |         | 2.5          | 30.48 | 80.8   |\n\nConsistent with our findings in Appendix F.1., we find that the task performance is stable over a range of $\\alpha_{RK}$ and $\\alpha_{D}$. These results add more empirical evidence to the role of $\\alpha_{RK}$ and $\\alpha_{D}$ as a knob for regularization strength. We see that, controlled for a fixed value of $\\beta$, increasing $\\alpha_{RK}$ and $\\alpha_{D}$ generally leads to higher KL with respect to the reference model at the end of training. The gradient-based analysis in Appendix F.1. could explain this result (Line 885-887): \"Given large α, the gradient on TPs approach zero regardless of the reward margin of the pair under the current model. As we show in the paper, tied pairs contribute to a regularization effect\". \n\nGiven these experiments, we suggest the following scheme in hyper-parameter tuning in DPO-RK and DPO-D: (1) run training with a range of beta values with the default $\\alpha_{RK}=\\ln 3$ and $\\alpha_{D}=0$ to identify the working range of beta; (2) increase or decrease $\\alpha_{RK}=\\ln 3$ and $\\alpha_{D}=0$ at those beta values, using the final KL as a guide. Note that as we show in experiments there is usually a \"sweet range\" of KL where performance is optimal. This procedure would allow developers to find the best performance-KL trade-off on a particular setup. \n\nThanks again for suggesting this. We believe this adds to the practicality of our methods. We will include these in Appendix F.1 and mention our finding in the main text in our revision. \n\n> in the sensitivity analysis in Appendix F.1, α is uniformly used to represent the tie-control parameter for both methods without necessary clarification or consistent symbolic linkage. \n\nThanks for raising this. We will make clear that $\\alpha_{RK}=\\log\\nu_{RK}$ and $\\alpha_D = \\log \\nu_D$ are defined for DPO-RK and DPO-D in Appendix F.1, respectively. Thanks for pointing this out. \n\n> The paper posits \"tie data is typically discarded, leading to information waste\" (e.g., Introduction, Lines 30-32) as one of its core motivations...... Could the authors provide stronger evidence or argumentation to support the claim that the \"tie\" data repurposed by their methods generally possesses high informational value? \n\nThanks for raising this. We take the view that the informational value of tied pairs depends heavily on how pairs are collected and constructed. In this paper, we experiment on three common scenarios where ties naturally occur: (1) when pairs are similarly ranked/scored; (2) when pairs show conflicting preference from different annotators; and (3) when pairs contain responses that are both correct. We show that tied pairs contribute to strong regularization effect and could lead to improved performance over system trained with clear preferences only. This supports that these otherwise discarded data contain useful information for the learning procedure. \n\nWhile it is difficult to determine whether ties generally possess high informational value, we believe our work offers useful methodology and empirical evidence to investigate this question on a case-by-case basis.\n\nOutside DPO, we know that modeling ties is becoming increasingly important as overall performance improves. In evaluating recent LLMs,  Ameli. et. al [ICLR, 2025] finds that \"the ability to handle ties—an integral aspect of human-judged comparisons—significantly improving the model’s fit to observed data\". We will include this discussion in the revised version.\n\n**References**\n\nSiavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W. Mahoney. A statistical framework for ranking LLM-based chatbots. In The Thirteenth International Conference on Learning Representations, 2025."}, {"review_id": "oljiITFNm8", "reviewer": "Reviewer_p1pE", "summary": "While Bradley-Terry model assumes preferences have clear winners and losers, this is not the case in practice. In fact, previous work ignored data points that represent ties. This paper extends DPO to accommodate ties using two existing approaches: Rao&Kupper and Davidson. These models differ in how they assign probabilities to win and tie cases. They also introduce additional parameters that control the mass assigned to ties. Analyzing the gradients, these two variants (DPO-RK and DPO-D) exhibit different scaling factors in gradient estimation. Experimental results on NMT and summarization suggest that (i) adding ties have a regularization effect, (ii) DPO-RK and DPO-D achieves better KL vs performance curves, (iii) while DPO reward estimation is better for win/loss, other variants offer better reward estimation for ties.", "questions": "1. Please add more preference learning baselines.\n\n2. Please develop simple DPO baselines to incorporate ties.\n\n3. Please clarify how TP dataset is created.\n\n4. Please run more experiments with DPO-RK (CP) and DPO-D (CP).\n\n5. Would DPO serve as a better reward modeling for near-ties based on Figure-3?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. The paper is well-written and easy-to-follow. It provides a clear motivation on why ties could help and successfully points to existing work in utility theory for the proposed problem.\n\n2. The authors introduce two extensions: DPO-RK and DPO-D. Both extensions are clearly explained and connected to existing baseline DPO model.\n\n3. Experimental results are conducted on real-world benchmarks (NMT and summarization) with KL-performance curves and ablations on $\\beta$ parameter.\n\n# Weaknesses\n1. **Lack of baselines.** There exists a large body of preference optimization objectives that add margins, different losses, etc. that are not studied in the paper. While the introduced variants show some promise, it is not clear if they are already captured by the existing DPO variants in the literature.\n\n2. **Simple DPO baselines for ties.** Can you run experiments with baseline DPO where different weights are assigned to win/loss or ties? Is it that introduced variants capture ties better or can a simple extension of DPO also work well in practice?\n\n3. **Clarification on TP dataset.** When you are creating the TP dataset, are you using only one direction for ties? I.e., given $y_1$ and $y_2$ is a tie, is only one of $y_1 > y_2$ or $y_1 < y_2$ present in the TP dataset for DPO? If it is the case, how would adding both directions impact DPO performance?\n\n4. **DPO-x ablations.** Can you train DPO-RK and DPO-D on CP-only data? It is not clear if extending the dataset with ties or the slightly different preference modeling is why these variants work better.\n\n5. **Clarification on Figure-3.** In Figure-3 (left), DPO achieves close to zero preference probability when $d_\\theta$ is also close to zero. Does this mean that, when used as a reward model to rank candidates, DPO performs better for \"almost ties\"?\n\n6. **Minor typos.** In Eq (23), $\\beta$ should be removed from the RHS.", "was_revised": true, "final_justification": "Clarifications and additional results.", "rebuttal": "Thanks for your careful and knowledgable review. In response:\n\n> it is not clear if they are already captured by the existing DPO variants in the literature.\n\nThanks for raising this. We discuss related variants (e.g., ODPO) and our theoretical contribution compared to previous work in Related Work (Sec.5). To our knowledge, there are no other DPO variants that explicitly model ties except for contemporaneous work by Guo et. Al. [2024], which we compare in detail in Sec.5. \n\n> Can you run experiments with baseline DPO where different weights are assigned to win/loss or ties? Is it that introduced variants capture ties better or can a simple extension of DPO also work well in practice?\n\nThanks for raising this. \n\nIn fact, the introduced variants do capture ties better from a modeling perspective. Considering the alternative which would be to assign different weights to win-loss and ties, the DPO objective would still try to increase the margins on ties, which is suboptimal from a modeling perspective because ties are taken to be equivalent. In Sec.2.3.1., we show that the DPO-RK and DPO-D gradients can work to decrease margins on ties, which is a mechanism not present in DPO, even with weights. A simple extension of DPO may work well in practice, but would struggle to accommodate ties. \n\n> Lack of baselines. There exists a large body of preference optimization objectives that add margins, different losses, etc. that are not studied in the paper. \n\nThanks for raising this. We choose to focus on a direct comparison with DPO in the main experiments. This is to allow us to directly study the effect of replacing Bradley-Terry with tie-compatible comparison models without interference from other modifications (e.g., length normalization, removing reference model, etc). However, we agree that results from other PO methods could provide useful points of comparison for interested researchers. We have run three established PO methods (IPO, SimPO, and CPO) on the GSM8K dataset using the same CP set as used by DPO. The results are tabulated below:\n\n | β   | SimPO (CP) | CPO (CP) | IPO (CP) | DPO (CP) | DPO-RK (CP+TP) | DPO-D (CP+TP) |\n|-----|------------|----------|----------|----------|----------------|----------------|\n| 0.1 | 82.5%      | 82.6%    | 20.2%    | 76.4%    | **83.5%**          | 81.7%          |\n| 0.3 | 81.8%      | 83.1%    | 57.1%    | 83.7%    | **84.4%**          | 83.2%          |\n| 0.5 | 81.8%      | 83.1%    | 69.1%    | 83.6%    | 83.8%          | **84.5%**          |\n| 0.7 | 81.6%      | 82.6%    | 78.3%    | 83.3%    | 83.7%          | **84.5%**          |\n| 1.0 | 82.2%      | 83.6%    | 76.1%    | 83.5%    | **84.1%**          | 83.7%          |\n\nWe find that in all but one case (CPO(CP), $\\beta=1.0$) these variants give lower accuracy compared to DPO. DPO-RK(CP+TP) and DPO-D(CP+TP) still yield the best performing system at every beta value with the added baselines. We believe these add to the strength of our empirical evidence. \n\n> When you are creating the TP dataset, are you using only one direction for ties? I.e., given $y_1$ and $y_2$ is a tie, is only one of  $y_1 > y_2$ or $y_2 > y_1$ present in the TP dataset for DPO? If it is the case, how would adding both directions impact DPO performance?\n\nIn our experiments on WMT, IWSLT and TL;DR, we always provide the higher scoring response as the chosen response when training with DPO (Line 786-787, Appendix E.1).  We did not add two training instances for one pair (i.e., provide both directions) because the gradients would cancel as we provide pairs for the same prompt in the same batch following the original implementation of DPO by Rafailov et al. [2023]. \n\nWe will include experiments where both directions are provided to DPO and not in the same batch. We have some intuitions from our findings so far: in Appendix F.6., we reverse the preference direction of Tied Pairs (rTP) and find that DPO(CP+TP) and DPO(CP+rTP) give similar performance (58.6% versus 60.9%). Both are substantially lower than DPO-RK(CP+TP) and DPO-D(CP+TP) (68.0% and 68.8%); In Appendix F.3., we find a higher proportion of TPs in training data leads to enhanced regularization effect. From these experiments, it is likely that providing both directions would not result in substantial performance gain but stronger regularization. \n\n> Can you train DPO-RK and DPO-D on CP-only data? It is not clear if extending the dataset with ties or the slightly different preference modeling is why these variants work better.\n\nThanks for raising this. We will include DPO-RK(CP) and DPO-D(CP) results in our final version for completeness. We emphasize that our variants are specifically designed to handle cases where Tied Pairs are present. As we have shown with the additional baselines, DPO remains a hard-to-beat method when only CPs are available and it is unlikely that replacing the Bradley-Terry model with Rao-Kupper or Davidson without providing Tied Pairs would enhance performance.\n\n> In Figure-3 (left), DPO achieves close to zero preference probability when $d_\\theta$ is also close to zero. Does this mean that, when used as a reward model to rank candidates, DPO performs better for \"almost ties\"? ... Would DPO serve as a better reward modeling for near-ties based on Figure-3?\n\nThanks for raising this. Figure 3 (left) shows the clear preference probabilities $P(y_w ≻ y_l |x)$ (left) as a function of reward margins $d_θ(x, y_w, y_l)$ for Bradley-Terry (See caption). When the reward margin is close to zero, the preference probability in either direction is close to 0.5. Note that different from the Rao-Kupper and the Davidson model, the Bradley-Terry model never explicitly assigns probability to the event of ties. In other words, a decision-rule based on the predicted margin is required for the DPO model to classify pairs as ties.\n\nIn Appendix C.2, we find that the DPO(CP) model, when used as a reward model, performs especially poorly at classifying tied pairs on held-out datasets compared to DPO-RK and DPO-D models (>20% difference in accuracy). This is true under two decision rules detailed in Appendix C.2. The DPO(CP+TP) model achieves higher accuracy at classifying TPs but at the cost of substantial degradation in CPs classification. We find that DPO-RK(CP+TP) and DPO-D(CP+TP) models achieve higher overall classification accuracies under both decision rules compared to DPO models.\n\n>  In Eq (23), should be removed from the RHS.\n\nThanks for catching this! We will fix this in our revision. Many thanks."}, {"review_id": "Jb3aFaxE5V", "reviewer": "Reviewer_87e1", "summary": "In this paper, the authors extend DPO with the ability to model ties (i.e., cases where the winner and loser are equally good or comparable). As the authors convincingly argue, their approach is a more principled alternative to simply removing such cases, which is a common pre-processing step in DPO due to known empirical disadvantages of having tied data instances. (though not mentioned, ties, or what are called \"indifference relations\" ~ are a well study theme in the broader literature on preference, see [Hansson, Sven Ove and Till Grüne-Yanoff, 2024]. The authors might consider citing this work as additional motivation). \n\nTo derive their variant of DPO, they rely on two established variants of Bradley-Terry (BT) that accomodate ties (named R(ao)K(upper) and D(avidson)). Both variants and very nicely summarized in Sec. 2.2 and (loosely speaking) define a distribution over w > l, l > w and a tie w ~ l via additional parameters (the difference in the two parameterizations of RK and D are discussed and explained from first principles in lines 74-87). They then use these two extensions to straightforwardly derive a general DPO variant (Eq. 14), one that factors into two loss terms that compete against one another, a term for clear winners with a binary weight and a term for ties with the same binary weight (both parameterized by either RK or D, giving rise to DPO-RK and DPO-D). Importantly, this general approach requires having an annotated set of tie and non-tie instances. (additional gradient analysis is provided in 2.3.1 that helps give intuition for how this differs from vanilla DPO). \n\nExperiments are performed for machine translation, text summarization and math reasoning. Ties and non-ties are obtained heuristically using outside models to score and rank outputs (the details vary depending on each task, though are easy to understand). The issue of DPO with tied instances is shown clearly in Figures 1-2 and compared against a filtering-based approach (both serve as natural baselines for their new loss). Modest improvements are obtained for both MT (Table 1) and math reasoning over DPO trained without tied instances (my main concern relatses to the narrowness of these tasks and the modesty of the improvements). Additional analysis is done (all in the appendix) to show that their approaches yield higher reward accuracy, enforce improved regularization (a technical argument for this is provided 3.3) and has improved convergence behavior (given limited time, I only had a cursory view of these results. I would encourage the authors to move the most important parts of this into the main text).  \n\n==== \n\nHansson, Sven Ove and Till Grüne-Yanoff, \"Preferences\", The Stanford Encyclopedia of Philosophy (Winter 2024 Edition), Edward N. Zalta & Uri Nodelman (eds.), URL = <https://plato.stanford.edu/archives/win2024/entries/preferences/>.", "questions": "-- Was significance testing done Tables 1-2? \n\n-- Did you extend your approach to any additional tasks? \n\n-- Have you considered making the binary `t` terms in Eq 14 real-valued? I ask because I'm left wondering how your approach relates to recent attempts at interpolating between multiple losses, e.g., in RainbowPO [Zhao et al. ICLR 2025]\n\n-- I'm also left wondering whether your approach could be easily extended to handle noise. Conceptually, ties seem to express a symmetry between the scores of the winner and loser, either they are equally \"bad\" (i.e., their probability should be driven towards zero) or equally \"good\" (i.e., their probability should be driven toward 1). From what I gather, your objective only accounts for the latter both not the former. Does this intuiton make sense?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# strengths\n\n-- A well motivated variant of DPO that directly addressed the important issue of ties in preference alignment. One that has clear and interesting connections with established variants of BT. I could imagine others in this area building on this novel direction and their approach. \n\n-- The paper is very well written and easy to follow. \n\n# weakensses \n\n-- (minor) The experiments are narrow, focusing only on MT, text summarization and math reasoning. I feel that these particular tasks are not sufficiently motivated and would like to see the authors address this. For example, it is not clear to me why MT is an interesting tasks wrt ties versus other tasks. \n\n-- The improvements reported in Tables 1-2 over standard DPO with filtering (as well as the baseline in row 1 for Table 1) are minimal and it is unclear whether they are significant. \n\n-- (minor) Their particular formulation in some ways makes the task more complicated by needing annotate tied data. It is unclear, outside of the tasks they chose, how such data can be reliably collected. I would like to see the authors address this, and also explain if they think that their approach could be extended so that this portion is learned.", "was_revised": true, "final_justification": "My biggest concern related to the limited empirical analysis and focus on MT, though these concerns were convincingly addressed by the authors. I also appreciated the additional feedback during the rebuttal.  I have therefore decided to keep my scor high.", "rebuttal": "Thank you for your careful and knowledgeable review, in response:\n\n> As the authors convincingly argue, their approach is a more principled alternative to simply removing such cases, which is a common pre-processing step in DPO due to known empirical disadvantages of having tied data instances. (though not mentioned, ties, or what are called \"indifference relations\" ~ are a well study theme in the broader literature on preference, see [Hansson, Sven Ove and Till Grüne-Yanoff, 2024]. The authors might consider citing this work as additional motivation).\n\nThanks for pointing out this work. We will include it in our discussion. \n\n> It is not clear to me why MT is an interesting tasks wrt ties versus other tasks.\n\nThanks for raising this. MT is actually a great task for studying ties, given how many ways there are to generate good translations.  The natural variability in translation means that ties are everywhere. Compared to summarization and general chat completion, Machine Translation also offers a range of well-established automatic metrics for assessing translation quality (e.g., BLEURT, XCOMET, KIWI-XXL), which we leverage to identify pairs of similar qualities as tied pairs reliably.    We will make this clear in our revision. We also note that applying DPO and its variants to MT is an established research topic with many published works. We follow them in designing our experiments.\n\n> Did you extend your approach to any additional tasks?\n\nWe conduct experiments on three Machine Translation datasets (WMT, IWSLT, ALMA-R), one summarization dataset (TL;DR), and one mathematical reasoning dataset (GSM8K). Our findings are consistent among these tasks experimented. We chose these tasks and datasets to cover different setups where ties may arise: in MT multiple reliable automatic metrics are available; in TL;DR an external metric is not used for identifying TPs and we use the base DPO reward model; in GSM8K the response labels are binary (correct or false). We believe our experiments so far cover a good range of scenarios and applications. \n\n> The improvements reported in Tables 1-2 over standard DPO with filtering (as well as the baseline in row 1 for Table 1) are minimal and it is unclear whether they are significant. Was significance testing done Tables 1-2?\n\nThanks for raising this. \n\nFor Table 1, our baseline is directly comparable to Table 10 (zh sub-table) in the original CPO paper [Xu et. al, 2024]. They established that a +1 gain in KIWI-XXL over the baseline achieves 80% accuracy with human judgement of improvements in quality. Our DPO-RK(CP+TP) and DPO-D(CP+TP) models give >+1.4 gain in KIWI-XXL compared to the DPO(CP) model. We will run additional experiments to report on the stability of these gains with different random seeds, but note that in Machine Translation consistent gains across multiple recent metrics are generally taken as significant.\n \nThe results from Table 2 are obtained by using the same random seed. We have run experiments for five additional random seeds for each variant at each beta value (75 training runs in total). In the table below, we report the median accuracy and standard deviation. We find that consistent with our previous finding, tie-compatible variants achieve higher median accuracy at all beta values. Gains are generally greater than 1x standard deviation. \n\n\n\n| $\\beta$ | DPO(CP)         | DPO-RK(CP+TP)    | DPO-D(CP+TP)    |\n|--------:|------------------|-------------------|------------------|\n| 0.1     | 80.4% $\\pm$ 1.34% | **82.9%** $\\pm$ 0.90% | 82.2% $\\pm$ 1.95% |\n| 0.3     | 83.7% $\\pm$ 0.45% | **84.3%** $\\pm$ 0.18% | 83.7% $\\pm$ 0.68% |\n| 0.5     | 83.5% $\\pm$ 0.14% | 84.1% $\\pm$ 0.08% | **84.2%** $\\pm$ 0.15% |\n| 0.7     | 83.6% $\\pm$ 0.53% | 83.4% $\\pm$ 0.19% | **84.5%** $\\pm$ 0.58% |\n| 1.0     | 83.8% $\\pm$ 0.29% | **84.0%** $\\pm$ 0.08% | 83.7% $\\pm$ 0.42% |\n\nWe have also run experiments with IPO, CPO, and SimPO using the same CP set as used by DPO. We find that these variants generally yield inferior performance to DPO under the same random seed used for Table 2. Please kindly see details in our response to Reviewer Xv6R. This shows that it is non-trivial to obtain consistent gains over DPO(CP), which remains a hard-to-beat baseline. \n\nWe also verify that the best-performing DPO-RK(CP+TP) and DPO-D(CP+TP) achieves a higher preservation rate compared to DPO(CP) (97.22%, 97.54% versus 96.04%). These numbers again agree with our results in Table 3. \n\nWe emphasize that the main contribution of Sec.3.5.2 is to show that ties can be constructed and used in DPO-RK/D training to preserve good behaviour in the reference model through regularization. It is a novel approach and in our experiments on GSM8K, we find small yet consistent gains in overall performance and higher overall preservation rate from tie-regularized systems. \n\n> Their particular formulation in some ways makes the task more complicated by needing annotate tied data. It is unclear, outside of the tasks they chose, how such data can be reliably collected. I would like to see the authors address this, and also explain if they think that their approach could be extended so that this portion is learned.\n\nThanks for your questions. We have demonstrated four ways to construct tied pairs: (1) via external automatic metrics (Sec.3.2, NMT); (2) via the trained DPO reward model (Sec.3.2; TL;DR); (3) via identifying pairs of conflicting preference annotations in existing pairwise datasets (Sec.3.5.1); and (4) via identifying pairs where both responses are correct when the label is binary (Sec.3.5.2). Though not exhaustive, we believe these offer a good range of approaches to tackle other downstream tasks. We will include the above discussion in our revision to improve clarity. We note Ameli et. al. [ICLR, 2024] has studied using ties in evaluating LLM chatbots.  They find that \"the ability to handle ties—an integral aspect of human-judged comparisons—significantly improving the model’s fit to observed data\". As we note in our introduction,  current development efforts collect and identify ties, but then discard them.  Our proposed approach simply requires keeping ties rather than throwing them out.\n\n> Have you considered making the binary t terms in Eq 14 real-valued? I ask because I'm left wondering how your approach relates to recent attempts at interpolating between multiple losses, e.g., in RainbowPO [Zhao et al. ICLR 2025]\n\nThanks for raising this insightful point. Our focus in this paper is accommodating ties in DPO when tied pairs are explicitly declared and so we keep the indicator $t$ binary-valued. Although we have not explored this, we note that the effect of a continuous $t$ on the gradients is relatively straight-forward: it would encourage a small margin between the winning and losing elements of tied pairs. \n\nInterpolating between multiple losses is an interesting topic and RainbowPO does offer a unified framework for many DPO variants.  However their formulation (their eq 18) assumes that expectation is carried out over clear preferences (i.e. no ties).    Conceivably, it would be possible to work from Eq.18 in the RainbowPO paper to extend the expectation to include ties, analogous to what we have done in Section 2.3 for plain DPO.\n\n> I'm also left wondering whether your approach could be easily extended to handle noise. Conceptually, ties seem to express a symmetry between the scores of the winner and loser, either they are equally \"bad\" (i.e., their probability should be driven towards zero) or equally \"good\" (i.e., their probability should be driven toward 1). From what I gather, your objective only accounts for the latter both not the former. Does this intuiton make sense?\n\nThanks for your question. Our objective accounts for both cases as it aims to assign a high probability of declaring a tie on labeled Tied Pairs (TP). That is, the objective does not distinguish between whether the two responses are good or bad in terms of absolute quality.\n\nIndeed, the approach could be extended to handle noisy comparisons. One such scenario is when multiple annotators give conflicting judgements, which we experiment in Sec 3.5.1 and  show including these data as tied give performance gain compared to discarding them. More generally, developers could choose to model pairs that are not certain in preference direction, owing to annotation noise, metric precision or otherwise, as ties. We believe our proposed variants and findings can inform future research in this direction.\n\n**References**\n\nSiavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W. Mahoney. A statistical framework for ranking LLM-based chatbots. In The Thirteenth International Conference on Learning Representations, 2025."}]

Author rebuttal:

Meta-review: (a) Summary: The paper addresses a significant limitation of the original Direct Preference Optimization (DPO) for aligning Large Language Models (LLMs) -- its inability to accommodate "tie" outcomes in training data, which leads to information loss because such data is typically discarded. The traditional DPO, relying on the Bradley-Terry model, assumes clear winners and losers, a premise often not met in real-world preferences. To overcome this, the authors propose two novel DPO extensions that explicitly model tie outcomes: DPO-RK (based on the Rao-Kupper model) and DPO-D (based on the Davidson model). These variants introduce additional parameters to control the probability assigned to ties.

(b) Strengths:
- The paper effectively highlights the practical problem of DPO's inability to handle ties, providing a strong and clear motivation for the proposed method.
- The paper introduces innovative DPO extensions (DPO-RK and DPO-D) by integrating established probabilistic models (Rao-Kupper and Davidson) to explicitly model tie probabilities. This is considered a more principled alternative than simply discarding tied data.
- Reviewers consistently praised the paper's clarity and readability, noting it is easy to follow and explains the extensions clearly.
- The paper conducts extensive experiments across multiple real-world tasks (machine translation, summarization, mathematical reasoning) to validate the practical value of the proposed methods. It demonstrates superior performance over strong DPO baselines and provides additional analysis on reward accuracy, regularization, and convergence.
- The work directly addresses an important, under-investigated issue in preference alignment, demonstrating clear and interesting connections with established utility theory.

(c) Weaknesses
- The constructed datasets used for experiments were not publicly available, raising concerns about reimplementability.
- Hyperparameter analysis issues:
- The paper's premise that discarded "tie" data necessarily possesses high informational value was debated, with questions about potential low-quality or noisy data included as ties.
- The formulation requires annotated tied data, and it was unclear how such data could be reliably collected outside the chosen tasks, or if this annotation portion could be learned.
- Questions were raised about whether a simpler extension of DPO (e.g., with different weights for ties) could also perform well, or if the benefit strictly came from the new modeling or the extended dataset.

(d) Reasons to accept 
- The paper tackles a fundamental practical limitation of DPO, which is the prevalent discarding of tie data in preference alignment. This problem is significant because ties are common in human-judged comparisons and represent valuable information.
- The introduction of DPO-RK and DPO-D provides a technically sound and novel approach to incorporate ties by building on established probabilistic models, demonstrating a deeper understanding of preference modeling beyond clear win/loss scenarios.
- Empirical demonstration of effectiveness across diverse tasks.

(e) Rebuttal:
- Reviewers questioned how tied pairs (TPs) were constructed and whether they held significant value. Authors explained four methods to construct TPs (e.g., using external metrics, conflicting annotations, or identifying pairs where both responses are correct). They provided empirical evidence that tied pairs contribute to a strong regularization effect and improved performance, supporting their utility, and committed to publishing the datasets.
- Concerns arose about the lack of novelty and the work being an incremental extension. Authors highlighted their contributions: the DPO-RK/D formulation, being the first to show the regularization effect of including ties in DPO, and proposing novel ways to model previously discarded data as ties. 
- Concerns were raised about the modesty of improvements and lack of significance testing. For Machine Translation, authors stated that a >+1.4 gain in KIWI-XXL is generally considered significant. For mathematical reasoning, they ran five additional random seeds, reporting median accuracy and standard deviation, which showed consistently higher median accuracy for tie-compatible variants, with gains generally greater than one standard deviation.
- Reviewers questioned the narrowness of tasks and MT's suitability. Authors explained MT is ideal for studying ties due to its natural variability and available automatic metrics. They also showed that vanilla DPO performs poorly at classifying held-out tied pairs, while DPO-RK/D models achieved higher overall classification accuracies for ties.
Decision: Accept (poster)